| *** tetsuro has joined #openstack-nova | 00:05 | |
| *** hiro-kobayashi has joined #openstack-nova | 00:20 | |
| *** yangyapeng has quit IRC | 00:25 | |
| *** yassine has joined #openstack-nova | 00:26 | |
| *** yassine is now known as Guest92842 | 00:27 | |
| *** salv-orlando has joined #openstack-nova | 00:40 | |
| *** salv-orlando has quit IRC | 00:45 | |
| *** hieulq has quit IRC | 00:46 | |
| *** hieulq has joined #openstack-nova | 00:47 | |
| *** psachin has joined #openstack-nova | 00:54 | |
| *** kumarmn has joined #openstack-nova | 00:54 | |
| *** hoangcx has joined #openstack-nova | 00:56 | |
| *** kumarmn has quit IRC | 01:00 | |
| *** hoangcx has quit IRC | 01:07 | |
| *** hoangcx has joined #openstack-nova | 01:07 | |
| *** StevenK has quit IRC | 01:08 | |
| *** rm_work has quit IRC | 01:08 | |
| *** huanxie has joined #openstack-nova | 01:08 | |
| *** rm_work has joined #openstack-nova | 01:09 | |
| *** yangyapeng has joined #openstack-nova | 01:10 | |
| *** StevenK has joined #openstack-nova | 01:12 | |
| *** phuongnh has joined #openstack-nova | 01:12 | |
| *** gouthamr has quit IRC | 01:21 | |
| *** trungnv has joined #openstack-nova | 01:22 | |
| *** namnh has joined #openstack-nova | 01:27 | |
| *** TuanLA has joined #openstack-nova | 01:31 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (1/2) https://review.openstack.org/430608 | 01:32 |
|---|---|---|
| *** trungnv has quit IRC | 01:34 | |
| *** trungnv has joined #openstack-nova | 01:34 | |
| *** Guest92842 has quit IRC | 01:35 | |
| *** namnh has quit IRC | 01:36 | |
| openstackgerrit | Wangpan proposed openstack/nova master: Handle glance exception during rotating instance backup https://review.openstack.org/523395 | 01:39 |
| *** hshiina has joined #openstack-nova | 01:39 | |
| *** TuanLA has quit IRC | 01:40 | |
| *** salv-orlando has joined #openstack-nova | 01:41 | |
| *** hui has joined #openstack-nova | 01:41 | |
| *** hoonetorg has quit IRC | 01:41 | |
| *** dave-mcc_ has joined #openstack-nova | 01:43 | |
| *** annp has joined #openstack-nova | 01:45 | |
| *** salv-orlando has quit IRC | 01:46 | |
| -openstackstatus- NOTICE: The freenode network is currently the target of automated spam attacks, we have enabled temporary restrictions on targetted OpenStack channels which requires users to be logged on to NickServ. If you see spam in your channel, please report it in #openstack-infra. Thanks. | 01:48 | |
| *** TuanLA has joined #openstack-nova | 01:48 | |
| *** hoonetorg has joined #openstack-nova | 01:49 | |
| *** baoli has joined #openstack-nova | 01:50 | |
| *** kumarmn has joined #openstack-nova | 01:55 | |
| *** jmlowe has quit IRC | 01:57 | |
| *** lei-zh has joined #openstack-nova | 01:57 | |
| *** kumarmn has quit IRC | 02:00 | |
| *** dave-mcc_ has quit IRC | 02:02 | |
| *** TuanLA has quit IRC | 02:02 | |
| *** yamahata has joined #openstack-nova | 02:04 | |
| *** kumarmn has joined #openstack-nova | 02:05 | |
| *** kumarmn has quit IRC | 02:10 | |
| *** nore_rabel has quit IRC | 02:11 | |
| *** liuzz has joined #openstack-nova | 02:18 | |
| *** Tom-Tom has joined #openstack-nova | 02:20 | |
| *** baoli has quit IRC | 02:28 | |
| *** huanxie has quit IRC | 02:31 | |
| *** huanxie has joined #openstack-nova | 02:35 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/python-novaclient master: Microversion 2.58 - Instance actions list pagination https://review.openstack.org/528601 | 02:41 |
| *** salv-orlando has joined #openstack-nova | 02:41 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: List/show all server migration types (2/2) https://review.openstack.org/459483 | 02:42 |
| *** salv-orlando has quit IRC | 02:46 | |
| *** gcb has joined #openstack-nova | 02:49 | |
| *** ChanServ sets mode: +r | 02:53 | |
| *** annp has quit IRC | 03:03 | |
| *** huanxie has quit IRC | 03:04 | |
| *** huanxie has joined #openstack-nova | 03:07 | |
| *** dmsimard has left #openstack-nova | 03:16 | |
| *** hiro-kobayashi has quit IRC | 03:16 | |
| *** yamamoto has joined #openstack-nova | 03:18 | |
| *** david-lyle has joined #openstack-nova | 03:25 | |
| *** hshiina has quit IRC | 03:35 | |
| *** Tom-Tom has quit IRC | 04:00 | |
| *** udesale has joined #openstack-nova | 04:12 | |
| *** lei-zh has quit IRC | 04:20 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova master: Use method validate_integer from oslo.utils https://review.openstack.org/528620 | 04:27 |
| vivsoni | Hi Team | 04:31 |
| vivsoni | when i was trying to attach a volume to an instance | 04:31 |
| vivsoni | using 'nova volume-attach <inst_id> <vol_id>' | 04:31 |
| vivsoni | the volume state went to 'reserved' status | 04:31 |
| vivsoni | and i could see my drivers 'initialize_connection()' is not called | 04:32 |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova master: Use method validate_integer from oslo.utils https://review.openstack.org/528620 | 05:02 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (resource_provider) https://review.openstack.org/528629 | 05:22 |
| *** yangyapeng has quit IRC | 05:31 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (allocation) https://review.openstack.org/528630 | 05:38 |
| *** abhishekk has joined #openstack-nova | 05:52 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Separate API schemas (allocation_candidate) https://review.openstack.org/528635 | 05:57 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a warning in 'nova-manage cell_v2 delete_cell' https://review.openstack.org/513771 | 06:00 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix missing marker functions https://review.openstack.org/514579 | 06:02 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add x-openstack-request-id in API ref https://review.openstack.org/523007 | 06:02 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix the order of target host checks https://review.openstack.org/526225 | 06:03 |
| *** daidv has joined #openstack-nova | 06:15 | |
| *** annp has joined #openstack-nova | 06:32 | |
| *** namnh has joined #openstack-nova | 06:37 | |
| *** namnh has quit IRC | 06:37 | |
| *** namnh has joined #openstack-nova | 06:38 | |
| *** zhurong has joined #openstack-nova | 06:43 | |
| *** trungnv has quit IRC | 06:44 | |
| *** hoonetorg has quit IRC | 06:45 | |
| *** rcernin has quit IRC | 06:46 | |
| *** hoangcx has quit IRC | 06:47 | |
| *** hieulq has quit IRC | 06:47 | |
| *** daidv has quit IRC | 06:47 | |
| *** phuongnh has quit IRC | 06:47 | |
| *** hieulq has joined #openstack-nova | 06:48 | |
| *** hoangcx has joined #openstack-nova | 06:48 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova stable/pike: Add 'delete_host' command in 'nova-manage cell_v2' https://review.openstack.org/513717 | 06:58 |
| *** hoonetorg has joined #openstack-nova | 07:01 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795 | 07:07 |
| *** phuongnh has joined #openstack-nova | 07:09 | |
| *** daidv has joined #openstack-nova | 07:11 | |
| *** daidv has left #openstack-nova | 07:11 | |
| *** daidv has joined #openstack-nova | 07:11 | |
| *** pooja_jadhav has joined #openstack-nova | 07:31 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/python-novaclient master: Microversion 2.58 - Instance actions list pagination https://review.openstack.org/528601 | 07:53 |
| *** bkopilov has quit IRC | 07:55 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406 | 07:56 |
| *** Dinesh_Bhor has joined #openstack-nova | 08:01 | |
| *** Dinesh_Bhor has quit IRC | 08:03 | |
| *** gcb has quit IRC | 08:04 | |
| *** rcernin has joined #openstack-nova | 08:06 | |
| openstackgerrit | Ahmed Zaid proposed openstack/nova master: Flavors with illegal names. https://review.openstack.org/528563 | 08:08 |
| openstackgerrit | Ahmed Zaid proposed openstack/nova master: Flavors with illegal names. https://review.openstack.org/528563 | 08:12 |
| *** zhurong has quit IRC | 08:21 | |
| *** yamamoto has quit IRC | 08:25 | |
| *** ircuser-1 has quit IRC | 08:26 | |
| *** huanxie has quit IRC | 08:40 | |
| *** markvoelker has quit IRC | 08:42 | |
| *** huanxie has joined #openstack-nova | 08:42 | |
| *** jpena|off is now known as jpena | 08:44 | |
| *** alexchadin has joined #openstack-nova | 08:58 | |
| *** Yingxin has quit IRC | 09:00 | |
| *** sbezverk has quit IRC | 09:01 | |
| openstackgerrit | Merged openstack/nova stable/pike: Fix CellDatabases fixture swallowing exceptions https://review.openstack.org/519702 | 09:01 |
| *** huanxie has quit IRC | 09:12 | |
| *** huanxie has joined #openstack-nova | 09:13 | |
| *** ttsiouts has joined #openstack-nova | 09:13 | |
| openstackgerrit | rahul bardia proposed openstack/python-novaclient master: nova limits ERROR (Exception): Field names must be unique https://review.openstack.org/527922 | 09:14 |
| openstackgerrit | Lajos Katona proposed openstack/nova master: WIP: Add nested resources to server moving tests https://review.openstack.org/527728 | 09:17 |
| *** lucas-afk is now known as lucasagomes | 09:23 | |
| *** giblet is now known as gibi | 09:23 | |
| gibi | happy Monday nova! | 09:23 |
| openstackgerrit | Merged openstack/nova stable/pike: Handle InstanceNotFound when setting password via metadata https://review.openstack.org/519659 | 09:24 |
| *** yamamoto has joined #openstack-nova | 09:26 | |
| maciejjozefczyk | gibi: I wish it could be happy :) | 09:31 |
| openstackgerrit | Lajos Katona proposed openstack/nova master: WIP: Add nested resources to server moving tests https://review.openstack.org/527728 | 09:31 |
| *** takashin has left #openstack-nova | 09:32 | |
| *** yamamoto has quit IRC | 09:33 | |
| *** andreykurilin_ has quit IRC | 09:36 | |
| *** andreykurilin has joined #openstack-nova | 09:36 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/python-novaclient master: Microversion 2.58 - Instance actions list pagination https://review.openstack.org/528601 | 09:37 |
| mdbooth | lyarwood: https://review.openstack.org/#/c/460243/6 interesting. | 09:41 |
| mdbooth | lyarwood: For the other recent encryptor bug, I was thinking of folding attach/detach encryptor into attach/detach volume | 09:41 |
| mdbooth | So the caller would never actually worry about encryptors at all, just volumes. | 09:42 |
| mdbooth | Which is nearly where you've gone with that. Any reason not to go all the way? | 09:42 |
| lyarwood | mdbooth: iirc just booting from a volume wouldn't call attach/detach volume | 09:43 |
| lyarwood | mdbooth: see _create_domain_and_network | 09:43 |
| *** huanxie has quit IRC | 09:44 | |
| mdbooth | lyarwood: That's a bug, though, right? | 09:44 |
| mdbooth | In that, that specifically was the cause of the other problem. We worked round it. | 09:45 |
| lyarwood | mdbooth: why? you can't attach to a domain that isn't running | 09:45 |
| gibi | maciejjozefczyk: only 3 working day left for me from this year so I this is my last work Monday, therefore I feel happy :) | 09:45 |
| mdbooth | lyarwood: How do we boot from volume? | 09:46 |
| *** huanxie has joined #openstack-nova | 09:46 | |
| mdbooth | Do we not call attach in that flow? | 09:46 |
| lyarwood | mdbooth: pretty sure we don't, we just connect the volume and encryptors before starting the domain | 09:49 |
| maciejjozefczyk | gibi: Good for you :) | 09:49 |
| * mdbooth looks it up | 09:50 | |
| mdbooth | Although I guess the binding would be specifically to connect/disconnect | 09:50 |
| kashyap | If anyone looking for a simple +2 to warm up their Monday - https://review.openstack.org/#/c/527691/ | 09:50 |
| mdbooth | Mounting a volume on a host without connecting its encryptor would always be a bug, I think | 09:50 |
| mdbooth | lyarwood: Anyway, I'm not saddling up for a -1 on your patch for this :) | 09:51 |
| mdbooth | lyarwood: I'd just been looking at combining this logic recently to make it harder to make mistakes. | 09:51 |
| mdbooth | And you're simplifying it somewhat here, which is good. | 09:51 |
| *** janki has joined #openstack-nova | 09:55 | |
| mdbooth | lyarwood: incidentally, I suspect all connect calls go through driver._connect_volume | 10:00 |
| mdbooth | So if the arguments lined up, I'd put the volume encryptor stuff in there, and remove all other calls. | 10:02 |
| mdbooth | That makes sense to me, anyway. May require some refactoring to make reality match theory. | 10:03 |
| lyarwood | mdbooth: urgh that's going to mess with the exception handling we currently have | 10:04 |
| *** liuzz has quit IRC | 10:05 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: update_cell allows more than once cell to have the same db/transport url https://review.openstack.org/518556 | 10:11 |
| mdbooth | lyarwood: Doesn't need to happen now, I'm just firming up a related idea. | 10:13 |
| mdbooth | Although I do think it would be much more robust. | 10:13 |
| lyarwood | mdbooth: kk, tbh my own reason for doing this refactor is to limit the number of conditionals when switching to native LUKS via QEMU | 10:15 |
| lyarwood | only* | 10:15 |
| *** phuongnh has quit IRC | 10:16 | |
| *** huanxie has quit IRC | 10:17 | |
| *** huanxie has joined #openstack-nova | 10:18 | |
| *** udesale has quit IRC | 10:27 | |
| *** udesale has joined #openstack-nova | 10:38 | |
| *** markvoelker has joined #openstack-nova | 10:42 | |
| *** moshele has joined #openstack-nova | 10:45 | |
| *** huanxie has quit IRC | 10:48 | |
| *** huanxie has joined #openstack-nova | 10:49 | |
| *** namnh has quit IRC | 10:52 | |
| *** sambetts_ is now known as sambetts | 10:54 | |
| *** moshele has quit IRC | 10:56 | |
| *** rcernin has quit IRC | 10:58 | |
| openstackgerrit | Chen Hanxiao proposed openstack/nova master: libvirt: don't call sync_guest_time if qga is not enabled https://review.openstack.org/524836 | 10:59 |
| *** udesale has quit IRC | 11:01 | |
| *** yamamoto has joined #openstack-nova | 11:12 | |
| *** moshele has joined #openstack-nova | 11:13 | |
| *** yamamoto has quit IRC | 11:15 | |
| *** claudiub has joined #openstack-nova | 11:15 | |
| *** markvoelker has quit IRC | 11:17 | |
| *** huanxie has quit IRC | 11:20 | |
| *** huanxie has joined #openstack-nova | 11:20 | |
| *** aloga has quit IRC | 11:29 | |
| *** derekh has joined #openstack-nova | 11:29 | |
| *** tetsuro has quit IRC | 11:38 | |
| *** cdent has joined #openstack-nova | 11:42 | |
| *** huanxie has quit IRC | 11:51 | |
| *** cdent has quit IRC | 11:51 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/python-novaclient master: Microversion 2.58 - Instance actions list pagination https://review.openstack.org/528601 | 11:54 |
| *** huanxie has joined #openstack-nova | 11:54 | |
| *** smatzek has joined #openstack-nova | 11:55 | |
| *** nicolasbock_ has joined #openstack-nova | 12:11 | |
| *** lucasagomes is now known as lucas-hungry | 12:13 | |
| *** markvoelker has joined #openstack-nova | 12:14 | |
| *** yamamoto has joined #openstack-nova | 12:16 | |
| *** yamamoto has quit IRC | 12:21 | |
| *** raildo has joined #openstack-nova | 12:22 | |
| openstackgerrit | Claudiu Belu proposed openstack/nova master: tests: autospecs all the mock.patch usages https://review.openstack.org/470775 | 12:22 |
| *** huanxie has quit IRC | 12:24 | |
| *** huanxie has joined #openstack-nova | 12:25 | |
| *** nicolasbock_ has quit IRC | 12:26 | |
| *** abhishekk has quit IRC | 12:28 | |
| *** mvk has quit IRC | 12:31 | |
| *** janki has quit IRC | 12:36 | |
| *** moshele has quit IRC | 12:37 | |
| *** jpena is now known as jpena|lunch | 12:38 | |
| openstackgerrit | Claudiu Belu proposed openstack/nova master: tests: autospecs all the mock.patch usages https://review.openstack.org/470775 | 12:42 |
| *** udesale has joined #openstack-nova | 12:46 | |
| *** markvoelker has quit IRC | 12:48 | |
| *** cdent has joined #openstack-nova | 12:50 | |
| *** claudiub has quit IRC | 12:51 | |
| *** claudiub has joined #openstack-nova | 12:53 | |
| *** moshele has joined #openstack-nova | 12:54 | |
| *** huanxie has quit IRC | 12:55 | |
| *** psachin has quit IRC | 12:56 | |
| *** huanxie has joined #openstack-nova | 12:57 | |
| openstackgerrit | sahid proposed openstack/nova master: libvirt: disconnect volume from host during detach https://review.openstack.org/515008 | 12:59 |
| *** igordc has quit IRC | 13:08 | |
| *** igordc has joined #openstack-nova | 13:08 | |
| *** lucas-hungry is now known as lucasagomes | 13:08 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: Test making column Text -> MediumText without migration https://review.openstack.org/528712 | 13:13 |
| *** yamamoto has joined #openstack-nova | 13:18 | |
| *** jpena|lunch is now known as jpena | 13:24 | |
| *** yamamoto has quit IRC | 13:26 | |
| *** huanxie has quit IRC | 13:28 | |
| *** rwellum has left #openstack-nova | 13:28 | |
| *** sri_ has quit IRC | 13:28 | |
| *** huanxie has joined #openstack-nova | 13:30 | |
| *** namnh has joined #openstack-nova | 13:32 | |
| *** moshele has quit IRC | 13:38 | |
| leakypipes | edleafe: mornin. we doing a scheduler meeting today? | 13:40 |
| cdent | leakypipes: do you want one, or are you more about trying to manage your schedule? | 13:41 |
| leakypipes | cdent: more managing my schedule | 13:41 |
| leakypipes | cdent: and just curious who's here :) | 13:41 |
| leakypipes | cdent: and good morning sir | 13:41 |
| cdent | mawrnin' | 13:41 |
| cdent | I am basically here, but still have not shed my cold. | 13:43 |
| bauzas | cdent: leakypipes: honestly, I'm working on implementing the VGPU change, so only if needed, I'd prefer to work | 13:46 |
| leakypipes | ack | 13:46 |
| bauzas | shit for libvirt btw | 13:46 |
| * cdent passes bauzas a cup of hot cocoa | 13:47 | |
| bauzas | cdent: <3 | 13:47 |
| cdent | this is what I'm going to do henceforth: hang out in irc and pass people nice stuff. | 13:48 |
| bauzas | cdent: thanks for the hot cocoa https://photos.app.goo.gl/qLBp2ZnFcJc2pvAp2 | 13:52 |
| *** takashin has joined #openstack-nova | 13:52 | |
| edleafe | leakypipes: yes, this is the last meeting of the year | 13:58 |
| edleafe | and plug your leak | 13:58 |
| stephenfin | bauzas, leakypipes: I've a question on the PCI NUMA policies spec for the two of you | 13:58 |
| bauzas | ack | 13:59 |
| leakypipes | stephenfin: yes>? | 13:59 |
| stephenfin | Per leakypipes' comments, I've been moving the storing of the resulting policy to the InstancePCIRequests object | 13:59 |
| alex_xu | leakypipes: morning, do we still need a separate microversionf or the traits parameter in allocationcandidates, or Eric's granular request's patch will cover that? | 13:59 |
| stephenfin | But that's thrown up a question cfriesen had asked me previously - should we be doing this policy configuration on an instance-level basis or on a PCI request-level basis? | 14:00 |
| *** namnh has quit IRC | 14:00 | |
| *** huanxie has quit IRC | 14:00 | |
| leakypipes | stephenfin: probably instance-level (i.e. the RequestSpec) | 14:00 |
| leakypipes | alex_xu: scheduler meeting... discuss in there? | 14:01 |
| *** namnh has joined #openstack-nova | 14:01 | |
| alex_xu | leakypipes: ok, got it | 14:01 |
| leakypipes | stephenfin: unless the policy knob can be toggled per-device, then of course it would be more appropriate to live in InstancePCIRequest I think. thoughts? | 14:02 |
| *** namnh has quit IRC | 14:02 | |
| *** namnh has joined #openstack-nova | 14:02 | |
| stephenfin | leakypipes: I've been avoiding that solely because it involves a lot of rewiring. We don't use RequestSpec objects on any of the code paths where we do NUMA'y scheduling | 14:02 |
| stephenfin | That was cfriesen's suggestion, yes | 14:02 |
| leakypipes | ack | 14:03 |
| *** huanxie has joined #openstack-nova | 14:03 | |
| leakypipes | stephenfin: InstancePCIRequest it is then. :) | 14:03 |
| leakypipes | stephenfin: it won't *hurt* to put them in there. | 14:03 |
| stephenfin | leakypipes: So that would suggest we don't use flavour extra specs + image metadata, and instead extend '[pci] aliases' to allow for a 'numa_policy' field, right? | 14:04 |
| *** gbarros has joined #openstack-nova | 14:05 | |
| leakypipes | stephenfin: unfortunately, since that is how PCI device requests are constructed. :( | 14:05 |
| stephenfin | leakypipes: Yeah, that thing is ugly as sin. Cool, I'll rework this as appropriate so and fix up the spec to reflect reality | 14:06 |
| *** ChanServ sets mode: -r | 14:08 | |
| leakypipes | stephenfin: danke | 14:09 |
| *** andreas_s has joined #openstack-nova | 14:10 | |
| *** salv-orlando has joined #openstack-nova | 14:16 | |
| *** jpena is now known as jpena|off | 14:19 | |
| *** andreas_s has quit IRC | 14:19 | |
| *** mriedem has joined #openstack-nova | 14:20 | |
| *** jpena|off is now known as jpena | 14:20 | |
| *** ralonsoh has joined #openstack-nova | 14:20 | |
| *** felipemonteiro has joined #openstack-nova | 14:20 | |
| *** artom has joined #openstack-nova | 14:21 | |
| *** andreas_s has joined #openstack-nova | 14:24 | |
| *** alexchadin has quit IRC | 14:26 | |
| *** smatzek has quit IRC | 14:28 | |
| *** smatzek has joined #openstack-nova | 14:28 | |
| *** andreas_s has quit IRC | 14:29 | |
| *** kumarmn has joined #openstack-nova | 14:29 | |
| *** andreas_s has joined #openstack-nova | 14:29 | |
| *** eharney has joined #openstack-nova | 14:30 | |
| *** kumarmn has quit IRC | 14:30 | |
| *** huanxie has quit IRC | 14:32 | |
| *** andreas_s has quit IRC | 14:32 | |
| *** andreas_s has joined #openstack-nova | 14:32 | |
| *** smatzek has quit IRC | 14:33 | |
| *** huanxie has joined #openstack-nova | 14:33 | |
| bauzas | leakypipes: crazy problem, since libvirt allows the possibility to create mediated devices on the fly and just assign them, I need to introspect all the mediated devices we have and see whether they are assigned to a guest | 14:34 |
| bauzas | leakypipes: in theory, a magical libvirt API would just tell me if it's assigned or not, but that's not existing | 14:34 |
| bauzas | leakypipes: so the only way I found would be to loop over all the instances and see their XML | 14:35 |
| bauzas | leakypipes: worst case scenario, one created mdev and 10K instances on a host, you can imagine the performance to find which guest is assigned to that mdev | 14:36 |
| bauzas | leakypipes: so, tbh, the only reasonable way I see is to persist the guest assignment information we do in nova | 14:36 |
| ildikov | mriedem: hi | 14:37 |
| bauzas | and consider that if for some specific mdev, we don't have the assignment in the Nova DB, then it means it's free to be allocated | 14:37 |
| bauzas | leakypipes: but you see my point, it would require some persistence model... | 14:37 |
| ildikov | mriedem: IIRC you were pointing me at some point to a review that added some locking similar to what I will need to add for multi-attach | 14:37 |
| bauzas | thoughts on that ? | 14:37 |
| ildikov | mriedem: in case it really is a memory and not a dream and you know what I'm talking about it would be great if you could point me to that one again :) | 14:38 |
| *** Nil_ has joined #openstack-nova | 14:39 | |
| ebbex | mdbooth: nova/virt/libvirt/imagecache.py function "_scan_base_images", should that have changes/deprecations in pike? | 14:39 |
| mriedem | ildikov: i think that's pseudocode in the multiattach spec | 14:40 |
| bauzas | leakypipes: wait, I think about a possible way to hack that without needing to persist that | 14:40 |
| mdbooth | ebbex: Yep, but mriedem wanted changes and I never followed it up. | 14:40 |
| mdbooth | ebbex: Lemme dig out the review, maybe you have time | 14:40 |
| ildikov | mriedem: oh, ok, I will re-read that one again then :) | 14:40 |
| bauzas | leakypipes: if we have a config opt that says "here are my already created mdevs", nova can use it and assume all of them as free | 14:40 |
| *** jmlowe has joined #openstack-nova | 14:41 | |
| mdbooth | ebbex: https://review.openstack.org/#/c/423392/ | 14:42 |
| ebbex | mdbooth: Cool, I'll have a look, thanks | 14:42 |
| mdbooth | ebbex: If you want to fix it up just go ahead, btw. No need to ask again. | 14:43 |
| mriedem | artom: are you adding the upgrade reno to https://review.openstack.org/#/c/528330/ or would you like me to wordsmith that? | 14:46 |
| artom | mriedem, yeah - question though about the migration being optional in the pike backport | 14:47 |
| mriedem | artom: i think it's fine | 14:47 |
| artom | mriedem, so I've pushed https://review.openstack.org/#/c/528712/ to test it | 14:47 |
| mriedem | we've done this before | 14:47 |
| artom | mriedem, ok, I just wanted to be sure | 14:48 |
| artom | Dropping kids off at daycare, back in a bit | 14:48 |
| *** artom has quit IRC | 14:48 | |
| *** awaugama has joined #openstack-nova | 14:48 | |
| *** artom has joined #openstack-nova | 14:49 | |
| stephenfin | bauzas: That sounds like something that could be libvirt really should have. If it's not present, could we do something with libvirt and carry the change locally til nova libvirt version > libvirt version with your fix? | 14:51 |
| *** felipemonteiro_ has joined #openstack-nova | 14:51 | |
| *** burt has joined #openstack-nova | 14:51 | |
| bauzas | stephenfin: WDYM ? | 14:51 |
| bauzas | hacking it with a TODO ? | 14:51 |
| stephenfin | bauzas: Aye. Figure out if it's something we could do with libvirt internals and propose the change upstream if possible | 14:52 |
| bauzas | I dunno | 14:52 |
| *** salv-orlando has quit IRC | 14:52 | |
| bauzas | lemme write something and we'll discuss | 14:52 |
| stephenfin | Neither do I. Just as a suggestion | 14:52 |
| *** salv-orlando has joined #openstack-nova | 14:53 | |
| stephenfin | Given that I imagine other people would benefit if libvirt had it too | 14:53 |
| *** felipemonteiro has quit IRC | 14:54 | |
| *** smatzek has joined #openstack-nova | 14:58 | |
| *** takashin has left #openstack-nova | 15:02 | |
| *** smatzek has quit IRC | 15:02 | |
| *** huanxie has quit IRC | 15:04 | |
| *** smatzek has joined #openstack-nova | 15:04 | |
| *** huanxie has joined #openstack-nova | 15:05 | |
| *** marst has joined #openstack-nova | 15:06 | |
| *** smatzek has quit IRC | 15:06 | |
| *** smatzek has joined #openstack-nova | 15:07 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Make request_spec.spec MediumText https://review.openstack.org/528330 | 15:07 |
| mriedem | artom: added the release note to the pike backport | 15:07 |
| mriedem | lyarwood: ^ | 15:07 |
| *** purplerbot has quit IRC | 15:10 | |
| *** purplerbot has joined #openstack-nova | 15:11 | |
| leakypipes | bauzas: sorry, went for breakfast... reading back now | 15:11 |
| bauzas | leakypipes: no worries, just trying a way to not have a huuuuuuge tech debt because $libvirt | 15:11 |
| *** mingyu_ has joined #openstack-nova | 15:11 | |
| bauzas | leakypipes: maybe a hangout would be cool for discussing | 15:11 |
| leakypipes | bauzas: ack. | 15:11 |
| leakypipes | bauzas: gimme about 10 minutes and I can do that, sure | 15:12 |
| bauzas | leakypipes: I'll need to get my girls in 15 but we can discuss that later | 15:12 |
| leakypipes | np | 15:12 |
| * bauzas will just get some paracetamol in the mean time | 15:13 | |
| bauzas | or prozac | 15:13 |
| leakypipes | lol :) | 15:13 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Make request_spec.spec MediumText https://review.openstack.org/528332 | 15:13 |
| *** tidwellr has quit IRC | 15:15 | |
| *** tidwellr has joined #openstack-nova | 15:15 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | 15:18 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | 15:18 |
| mdbooth | lyarwood: That second one ^^^^ | 15:18 |
| mdbooth | lyarwood: It's a bit disruptive in tests, but amongst its advantages is that it will expose both attachment_id and connection_info without further changes. | 15:21 |
| mdbooth | (Also bdm.uuid, which is what I want it for) | 15:21 |
| *** mlavalle has joined #openstack-nova | 15:22 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/newton: Make request_spec.spec MediumText https://review.openstack.org/528338 | 15:23 |
| artom | mriedem, ah, thanks | 15:24 |
| artom | mriedem, and https://review.openstack.org/#/c/528712/ passed Zuul, so I think we're good, yeah | 15:25 |
| mriedem | ok i've got the ocata and newton ones updated too | 15:26 |
| openstackgerrit | Mr Rambo proposed openstack/nova master: Fix the problems that volume-backed server rebuild https://review.openstack.org/528740 | 15:27 |
| *** gouthamr has joined #openstack-nova | 15:28 | |
| *** gouthamr has quit IRC | 15:29 | |
| artom | mriedem, I'll push a more complete reno if you don't mind, mentionning that the migration is optional unless you've been affected by the bug, and how to tell if you've been affected | 15:29 |
| mriedem | go nuts | 15:30 |
| *** tidwellr has quit IRC | 15:30 | |
| artom | http://www.risd.edu/news/stories/go-nads/ | 15:32 |
| *** kumarmn has joined #openstack-nova | 15:32 | |
| *** kumarmn has quit IRC | 15:32 | |
| *** kumarmn has joined #openstack-nova | 15:33 | |
| *** archit has joined #openstack-nova | 15:34 | |
| *** aarefiev has joined #openstack-nova | 15:35 | |
| *** huanxie has quit IRC | 15:36 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova stable/pike: Make request_spec.spec MediumText https://review.openstack.org/528330 | 15:37 |
| *** huanxie has joined #openstack-nova | 15:37 | |
| openstackgerrit | Hironori Shiina proposed openstack/nova master: virt/ironic: Implement rescue and unrescue https://review.openstack.org/416487 | 15:40 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova stable/ocata: Make request_spec.spec MediumText https://review.openstack.org/528332 | 15:40 |
| *** felipemonteiro_ has quit IRC | 15:40 | |
| *** armax has joined #openstack-nova | 15:41 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova stable/newton: Make request_spec.spec MediumText https://review.openstack.org/528338 | 15:41 |
| openstackgerrit | Claudiu Belu proposed openstack/nova master: tests: autospecs all the mock.patch usages https://review.openstack.org/470775 | 15:42 |
| *** moshele has joined #openstack-nova | 15:45 | |
| *** moshele has quit IRC | 15:49 | |
| *** moshele has joined #openstack-nova | 15:53 | |
| *** armax has quit IRC | 15:53 | |
| *** claudiub has quit IRC | 15:54 | |
| openstackgerrit | Jay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg https://review.openstack.org/526753 | 15:55 |
| *** damien_r has joined #openstack-nova | 15:55 | |
| *** r-daneel has joined #openstack-nova | 15:55 | |
| *** damien_r has left #openstack-nova | 15:56 | |
| *** moshele has quit IRC | 15:58 | |
| *** tidwellr has joined #openstack-nova | 16:03 | |
| *** lyan has joined #openstack-nova | 16:03 | |
| *** dtantsur|afk has quit IRC | 16:03 | |
| *** moshele has joined #openstack-nova | 16:04 | |
| *** _ix has joined #openstack-nova | 16:04 | |
| *** dtantsur has joined #openstack-nova | 16:04 | |
| *** jianghuaw has joined #openstack-nova | 16:06 | |
| *** felipemonteiro has joined #openstack-nova | 16:07 | |
| *** Apoorva has joined #openstack-nova | 16:07 | |
| *** chyka has joined #openstack-nova | 16:07 | |
| *** moshele has quit IRC | 16:08 | |
| *** huanxie has quit IRC | 16:08 | |
| *** huanxie has joined #openstack-nova | 16:09 | |
| openstackgerrit | Jianghua Wang proposed openstack/nova master: XenAPI: create vGPU for instance https://review.openstack.org/516899 | 16:09 |
| *** r-daneel_ has joined #openstack-nova | 16:11 | |
| jianghuaw | leakypipes, bauzas: ^ Please help to check if it has addressed your comments? Thanks. | 16:11 |
| *** r-daneel has quit IRC | 16:11 | |
| *** r-daneel_ is now known as r-daneel | 16:11 | |
| mdbooth | mriedem: I vaguely recall there's some reason we don't use docstrings in tests. Am I imagining things? | 16:12 |
| mriedem | mdbooth: that's no longer an issue | 16:13 |
| mriedem | it was because of limitations with nose | 16:13 |
| mriedem | but we don't use nose | 16:13 |
| mdbooth | mriedem: Ah, ok. Glad I didn't stick a -1 on lyarwood 's patch :) Figured that stuff should be in HACKING anyway if it's to be enforceable. | 16:14 |
| mdbooth | lyarwood: Ignore that one ^^^ | 16:14 |
| leakypipes | jianghuaw: +2 from me. :) | 16:14 |
| bauzas | jianghuaw: sorry, today I'm working on implementing the libvirt change, so tomorrow will look | 16:15 |
| jianghuaw | leakypipes, thanks:-) | 16:15 |
| jianghuaw | bauzas, no worries. tomorrow will be very good also. Thanks. | 16:15 |
| mdbooth | leakypipes: Is this now super Monday? | 16:15 |
| leakypipes | jianghuaw: no problem! hope I picked the correct Chinese characters from Google translate in my review comment ;) | 16:15 |
| leakypipes | oh, crap... | 16:15 |
| *** leakypipes is now known as jaypipes | 16:15 | |
| mdbooth | Hehe | 16:16 |
| jaypipes | thanks mdbooth :) | 16:16 |
| *** esberglu has joined #openstack-nova | 16:17 | |
| jianghuaw | jaypipes, haha. I was thinking why your Chinese was so good:-) | 16:18 |
| *** bfife has joined #openstack-nova | 16:18 | |
| kashyap | Folks using Ubuntu, there's some repo that provides newer packages than the system packages | 16:18 |
| kashyap | What's it called again? | 16:18 |
| mdbooth | jaypipes: https://boingboing.net/2008/07/15/chinese-restaurant-c.html | 16:18 |
| jaypipes | mdbooth: ha! that's awesome | 16:19 |
| mriedem | kashyap: UCA | 16:20 |
| mriedem | ubuntu cloud archive | 16:20 |
| jaypipes | lol | 16:20 |
| *** moshele has joined #openstack-nova | 16:27 | |
| *** tidwellr has quit IRC | 16:29 | |
| *** BryanS68 has joined #openstack-nova | 16:30 | |
| *** moshele has quit IRC | 16:34 | |
| *** namnh has quit IRC | 16:36 | |
| kashyap | mriedem: Ah, thank you. | 16:37 |
| *** gbarros has quit IRC | 16:37 | |
| * kashyap was helping another user on #kvm, and he was using newer functionality on Ubuntu w/ outdated packages. Couldn't recall UCA to recommend him. Thx. | 16:38 | |
| *** catintheroof has joined #openstack-nova | 16:38 | |
| *** andreas_s has quit IRC | 16:38 | |
| *** andreas_s has joined #openstack-nova | 16:39 | |
| *** huanxie has quit IRC | 16:40 | |
| *** gbarros has joined #openstack-nova | 16:40 | |
| *** penick has joined #openstack-nova | 16:40 | |
| *** huanxie has joined #openstack-nova | 16:42 | |
| *** jpena is now known as jpena|brb | 16:42 | |
| *** gbarros has quit IRC | 16:45 | |
| *** david_8 has joined #openstack-nova | 16:45 | |
| *** catintheroof has quit IRC | 16:45 | |
| *** andreas_s has quit IRC | 16:47 | |
| *** gbarros has joined #openstack-nova | 16:47 | |
| *** david_7 has quit IRC | 16:49 | |
| *** AlexeyAbashkin has joined #openstack-nova | 16:53 | |
| *** felipemonteiro has quit IRC | 16:55 | |
| *** AlexeyAbashkin has quit IRC | 16:57 | |
| *** andreas_s has joined #openstack-nova | 16:57 | |
| *** armax has joined #openstack-nova | 17:00 | |
| *** yamahata has quit IRC | 17:01 | |
| *** lucasagomes is now known as lucas-afk | 17:03 | |
| melwitt | stephenfin: hey, are you around? | 17:03 |
| stephenfin | I am | 17:03 |
| melwitt | I was just thinking about the xen ci failure on the patch and I realize, TLS is supposed to be disabled by default and you have to opt-in, and I don't think xen ci is doing anything to opt in. so I'm not sure yet why it's going through that path | 17:04 |
| *** gyee has joined #openstack-nova | 17:04 | |
| melwitt | so something is probably wrong there if it's going through the TLS path without being enabled. I'm still looking at it but just wanted to bring it up | 17:05 |
| stephenfin | Yeah, I saw that comment on Thursday. I don't know what it is either | 17:05 |
| melwitt | like, if they didn't enable it, their CI should be passing :\ | 17:05 |
| stephenfin | I know sross has some tests merged to DevStack way back related to this, but I don't know how that could possibly trigger it | 17:06 |
| stephenfin | Agreed | 17:06 |
| stephenfin | melwitt: I've been working on the PCI-NUMA policy code all day, but soon as that's done I'll take a look at that and see if I can suss it out | 17:06 |
| melwitt | yeah I'm not yet sure. in devstack you're supposed to have to use NOVA_CONSOLE_PROXY_COMPUTE_TLS=True to turn on TLS, which I have done in a DNM patch dependent on the devstack change https://review.openstack.org/#/c/513160 | 17:08 |
| melwitt | okay, cool. I'll look at it some more too | 17:08 |
| *** purplerbot has quit IRC | 17:10 | |
| *** purplerbot has joined #openstack-nova | 17:10 | |
| *** hemna_ has quit IRC | 17:10 | |
| *** imacdonn has joined #openstack-nova | 17:11 | |
| *** felipemonteiro has joined #openstack-nova | 17:11 | |
| *** huanxie has quit IRC | 17:12 | |
| *** huanxie has joined #openstack-nova | 17:13 | |
| *** felipemonteiro_ has joined #openstack-nova | 17:13 | |
| *** jianghuaw has quit IRC | 17:15 | |
| *** andreas_s has quit IRC | 17:16 | |
| *** mvk has joined #openstack-nova | 17:16 | |
| *** felipemonteiro has quit IRC | 17:17 | |
| *** Apoorva has quit IRC | 17:17 | |
| *** hemna_ has joined #openstack-nova | 17:20 | |
| melwitt | stephenfin: I think I see what it is, left a comment on the patch. I don't know what to do about it though, I'd want to ask danpb | 17:20 |
| melwitt | or sross/someone who knew how they wanted this to work | 17:22 |
| *** r-daneel_ has joined #openstack-nova | 17:22 | |
| *** r-daneel has quit IRC | 17:23 | |
| *** r-daneel_ is now known as r-daneel | 17:23 | |
| *** catintheroof has joined #openstack-nova | 17:23 | |
| *** jpena|brb is now known as jpena | 17:27 | |
| *** catintheroof has quit IRC | 17:30 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add PCI NUMA policy fields https://review.openstack.org/527470 | 17:30 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add PCI NUMA policies https://review.openstack.org/527472 | 17:30 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Modify signature of _filter_non_requested_pfs https://review.openstack.org/527473 | 17:30 |
| stephenfin | jaypipes: Ta-da! ^ | 17:31 |
| jaypipes | stephenfin: cool, will look in a bit. | 17:31 |
| stephenfin | jaypipes: Cheers :) I think that should satisfy most things. Only odd condition I had was what happened when two similar aliases requested different NUMA policies. I ruled that out as daft | 17:31 |
| stephenfin | and now I've to run home. Ciao o/ | 17:32 |
| jaypipes | ciao | 17:32 |
| *** BryanS68 has quit IRC | 17:34 | |
| *** penick has quit IRC | 17:35 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform missing delete notifications https://review.openstack.org/410297 | 17:37 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Send soft_delete from context manager https://review.openstack.org/476459 | 17:37 |
| *** mvk has quit IRC | 17:37 | |
| *** moshele has joined #openstack-nova | 17:38 | |
| *** moshele has quit IRC | 17:40 | |
| *** huanxie has quit IRC | 17:43 | |
| *** aarefiev has quit IRC | 17:44 | |
| *** huanxie has joined #openstack-nova | 17:46 | |
| *** sambetts is now known as sambetts|afk | 17:47 | |
| *** BryanS68 has joined #openstack-nova | 17:47 | |
| *** derekh has quit IRC | 17:52 | |
| *** andreas_s has joined #openstack-nova | 17:53 | |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add support for the 2.57 microversion https://review.openstack.org/528128 | 17:55 |
| *** andreas_s has quit IRC | 17:57 | |
| *** udesale has quit IRC | 17:58 | |
| openstackgerrit | Jay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg https://review.openstack.org/526753 | 18:00 |
| *** harlowja has joined #openstack-nova | 18:04 | |
| cdent | that's not what makes you evil mriedem | 18:06 |
| mriedem | |..|_ | 18:07 |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add support for the 2.57 microversion https://review.openstack.org/528128 | 18:13 |
| *** AlexeyAbashkin has joined #openstack-nova | 18:14 | |
| *** huanxie has quit IRC | 18:16 | |
| *** huanxie has joined #openstack-nova | 18:18 | |
| *** AlexeyAbashkin has quit IRC | 18:18 | |
| *** yamahata has joined #openstack-nova | 18:18 | |
| *** Apoorva has joined #openstack-nova | 18:21 | |
| *** Apoorva has quit IRC | 18:21 | |
| *** Apoorva has joined #openstack-nova | 18:22 | |
| *** mvk has joined #openstack-nova | 18:25 | |
| *** gbarros has quit IRC | 18:25 | |
| jaypipes | mriedem: how exactly can an instance group (server group) have >1 policy? :( | 18:26 |
| *** gbarros has joined #openstack-nova | 18:26 | |
| mriedem | can't | 18:27 |
| mriedem | https://review.openstack.org/#/c/415482/ | 18:27 |
| jaypipes | mriedem: then why is objects.InstanceGroup.policies a ListOfStringsField? | 18:27 |
| mriedem | ^ | 18:27 |
| mriedem | "legacy reasons" | 18:27 |
| jaypipes | uhg | 18:27 |
| jaypipes | k | 18:27 |
| mriedem | nice, we also have a separate dedicated table for the policies | 18:29 |
| mriedem | instance_group_policy | 18:29 |
| jaypipes | I know... it's just... sigh... | 18:29 |
| mriedem | we can microversion that out of the api | 18:30 |
| mriedem | and redo things in the data model if we actually cared | 18:30 |
| mriedem | looking at the old wiki for the bp https://wiki.openstack.org/wiki/InstanceGroupApiExtension#API | 18:31 |
| mriedem | it sounds like multiple policies was by design | 18:31 |
| mriedem | for affinity and QoS, as examples | 18:31 |
| *** penick has joined #openstack-nova | 18:31 | |
| *** ralonsoh has quit IRC | 18:33 | |
| jaypipes | mriedem: I'm just trying to get this stuff hacked up and working for Ironic... | 18:35 |
| jaypipes | mriedem: the real solution (distances and stuff like that) probably won't look like this. | 18:35 |
| jaypipes | mriedem: because I hate the server group functionality to begin with. | 18:36 |
| mriedem | i know, i've seen the tatto | 18:36 |
| mriedem | *tatoo | 18:36 |
| jaypipes | attoo? :) | 18:36 |
| jaypipes | tattoo? :) | 18:36 |
| mriedem | no | 18:36 |
| *** catintheroof has joined #openstack-nova | 18:36 | |
| mriedem | i've given up on trying to ever spell it correctly | 18:36 |
| jaypipes | lol | 18:36 |
| mriedem | so now it's just whatever the last thing i said | 18:36 |
| *** catintheroof has quit IRC | 18:36 | |
| *** catintheroof has joined #openstack-nova | 18:36 | |
| *** catintheroof has quit IRC | 18:36 | |
| *** namnh has joined #openstack-nova | 18:37 | |
| *** fragatina has joined #openstack-nova | 18:39 | |
| *** namnh has quit IRC | 18:42 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-ref: sort parameters for limits, quotas and quota classes https://review.openstack.org/528786 | 18:42 |
| jaypipes | mriedem: where is the latest microversion for the compute API found? | 18:44 |
| jaypipes | mriedem: in placement, it's a constant in the nova/api/openstack/placement/microversion.py file. | 18:45 |
| *** jmlowe has quit IRC | 18:47 | |
| artom | jaypipes, nova/api/openstack/api_version_request.py has _MAX_API_VERSION | 18:47 |
| jaypipes | artom: ty sir! | 18:47 |
| *** huanxie has quit IRC | 18:48 | |
| *** huanxie has joined #openstack-nova | 18:49 | |
| *** alee has joined #openstack-nova | 18:51 | |
| *** mingyu_ has quit IRC | 18:55 | |
| *** mingyu has joined #openstack-nova | 18:56 | |
| *** catintheroof has joined #openstack-nova | 18:56 | |
| *** itlinux has joined #openstack-nova | 18:58 | |
| *** fragatina has quit IRC | 18:59 | |
| *** mingyu has quit IRC | 19:00 | |
| *** jpena is now known as jpena|off | 19:01 | |
| openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM driver: ovs vif https://review.openstack.org/422512 | 19:02 |
| *** mingyu has joined #openstack-nova | 19:06 | |
| *** mingyu has quit IRC | 19:07 | |
| *** mingyu has joined #openstack-nova | 19:07 | |
| *** jmlowe has joined #openstack-nova | 19:08 | |
| *** mingyu has quit IRC | 19:12 | |
| *** smatzek_ has joined #openstack-nova | 19:16 | |
| *** smatzek has quit IRC | 19:19 | |
| *** huanxie has quit IRC | 19:20 | |
| *** huanxie has joined #openstack-nova | 19:20 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Remove microversion fallback code from report client https://review.openstack.org/528794 | 19:22 |
| cdent | that ^ may have some holes | 19:25 |
| *** gbarros has quit IRC | 19:27 | |
| *** penick has quit IRC | 19:30 | |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | 19:30 |
| *** penick has joined #openstack-nova | 19:31 | |
| *** penick has quit IRC | 19:39 | |
| *** mriedem has quit IRC | 19:40 | |
| *** mriedem has joined #openstack-nova | 19:41 | |
| kumarmn | Could these changes by mriedem get merged? https://review.openstack.org/#/c/527228/ and https://review.openstack.org/#/c/527468/ | 19:43 |
| kumarmn | They are holding me from tagging trove for queens-2. | 19:43 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Make request_spec.spec MediumText https://review.openstack.org/528332 | 19:43 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/newton: Make request_spec.spec MediumText https://review.openstack.org/528338 | 19:44 |
| mriedem | kumarmn: need to find another core to +W | 19:45 |
| mriedem | slim pickins right now | 19:45 |
| mriedem | jaypipes: ? ^ | 19:45 |
| mriedem | oomichi: ^ | 19:45 |
| melwitt | oh, stable cores. I was like, what about me!! | 19:45 |
| jaypipes | mriedem: I'm not a stable core. | 19:45 |
| mriedem | artom: i rev'ed the newton and ocata backports to fix the git commit hash to point at the updated ocata/pike commits, respectively | 19:46 |
| mriedem | jaypipes: no, not the stable patches | 19:46 |
| mriedem | https://review.openstack.org/#/c/527228/ and https://review.openstack.org/#/c/527468/ | 19:46 |
| mriedem | melwitt: you already +2ed them :) | 19:46 |
| jaypipes | ah, ok. yeah, just a sec. | 19:46 |
| *** armax has quit IRC | 19:46 | |
| melwitt | I too, thought they were the stable patches. too many links :D | 19:47 |
| * melwitt is not chopped liver | 19:47 | |
| mriedem | melwitt: what if you were blue wagyu chopped liver? | 19:48 |
| kumarmn | thanks mriedem, jaypipes, melwitt | 19:49 |
| melwitt | that is a conundrum | 19:49 |
| *** huanxie has quit IRC | 19:50 | |
| *** lyan has quit IRC | 19:52 | |
| artom | mriedem, ah, thanks | 19:52 |
| *** huanxie has joined #openstack-nova | 19:54 | |
| openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add support for the 2.57 microversion https://review.openstack.org/528128 | 19:54 |
| openstackgerrit | melanie witt proposed openstack/nova master: rbd: flatten images when creating/unshelving an instance https://review.openstack.org/457886 | 19:55 |
| *** nore__ has joined #openstack-nova | 19:58 | |
| jaypipes | mriedem: done | 20:00 |
| *** nore_rabel has joined #openstack-nova | 20:01 | |
| mriedem | thanks | 20:02 |
| *** windsn has joined #openstack-nova | 20:03 | |
| *** nore__ has quit IRC | 20:03 | |
| mriedem | edleafe: melwitt: i was thinking about something with the alternate hosts change during create server - https://review.openstack.org/#/c/511358/ - there is no functional test in that change that shows we're actually using the alternate hosts. we already have at least a couple functional tests which use 2 hosts and make the first fail to trigger a reschedule. but that worked before this because the compute and cell conductor c | 20:05 |
| mriedem | it the scheduler. i wonder if there is an easy way to make the cell conductor fail to hit the scheduler | 20:05 |
| mriedem | in our functional tests we're just running one conductor service | 20:05 |
| *** gbarros has joined #openstack-nova | 20:06 | |
| windsn | Hi, Im having issues with trying to use nova with ironic.... I see this error in the nova-compute logs, https://thepasteb.in/p/pghQo0OnW5vcR, and when i create an ironic node, i dont see the nova-hypervisor-list getting populated with an entry for the ironic node... I'm not too sure what seems to be the issue | 20:06 |
| mriedem | windsn: ocata, pike, master? | 20:06 |
| windsn | Pike | 20:07 |
| mriedem | windsn: have you run "nova-manage cell_v2 discover_hosts"? | 20:07 |
| mriedem | edleafe: melwitt: an easy think we could do is just make a select_destinations fixture which blows up if it's called more than once... | 20:08 |
| windsn | mriedem: No I haven't run that... I'm not sure what that does | 20:08 |
| mriedem | otherwise i think we'd need to do something where we start a 2nd conductor service with modified transport_url/db connection, and modify the compute service to point at the cell transport_url also | 20:08 |
| *** jose-phillips has joined #openstack-nova | 20:09 | |
| mriedem | windsn: https://docs.openstack.org/nova/latest/cli/nova-manage.html#nova-cells-v2 | 20:09 |
| edleafe | mriedem: I did think about doing something like that, but got tangled up in the multiple cell/conductor creation | 20:09 |
| mriedem | edleafe: by default the tests are running with 2 cells, cell0 and cell1 (unless the SimpleCellFixture or whatever it's called is used) | 20:10 |
| mriedem | so the trick would be running the cell conductor service and the compute service configured for the cell1 transport_url i think | 20:10 |
| mriedem | but we use a fake mq so i'm not sure it would actually help anything | 20:10 |
| edleafe | mriedem: ah, then I was trying for a third cell to show that it didn't touch the other cell | 20:10 |
| edleafe | I guess I don't think of cell0 as a "real" cell | 20:11 |
| mriedem | RPCFixture i think | 20:11 |
| edleafe | mriedem: ok, I'll start playing around with that in a bit | 20:12 |
| *** dave-mccowan has joined #openstack-nova | 20:12 | |
| mriedem | looks like dan left a breadcrumb https://review.openstack.org/#/c/396417/23/nova/tests/fixtures.py | 20:13 |
| cdent | dansel | 20:13 |
| jaypipes | gdi, these server group functional tests are inscrutable. :( | 20:14 |
| jaypipes | make me want to stab something. | 20:14 |
| mriedem | anyway, just thought about this before digging into the ironic multinode reschedule failures | 20:14 |
| jaypipes | AAAARRRRGGGGGGGGGGGGGGHHHHHHHHHHH I frigggin hate the integrate test helpers. | 20:16 |
| melwitt | jaypipes: one thing I learned about those tests is if you want to have scheduling consider > 1 compute host, you need a compute service and set_nodes per host | 20:17 |
| jaypipes | melwitt: just doing a simple copy/paste of one of the "negative" server group func tests results in completely random NotFound errors from the faked-out image service... | 20:18 |
| jaypipes | melwitt: makes no sense whatsoever. | 20:18 |
| jaypipes | melwitt: and the tests are completely unintelligible. | 20:18 |
| melwitt | hm, yeah I would not have expected that | 20:18 |
| *** BryanS68 has quit IRC | 20:18 | |
| melwitt | that, I do know from having used them as examples for other tests | 20:18 |
| * jroll puts the pugs away safe from keyboards being thrown | 20:18 | |
| melwitt | (the multiple compute host part I mean) | 20:19 |
| *** BryanS68 has joined #openstack-nova | 20:19 | |
| *** lyan has joined #openstack-nova | 20:20 | |
| openstackgerrit | Jay Pipes proposed openstack/nova master: WIP - Support aggregate affinity filters https://review.openstack.org/528804 | 20:20 |
| jaypipes | jroll: they are stowed away in padded rooms. not to worry. | 20:20 |
| jroll | excellent. | 20:20 |
| * edleafe wonders if the padded rooms have the right tenants | 20:21 | |
| jaypipes | melwitt: see the patch above... it will fail with this for no discernable reason whatsoever: http://paste.openstack.org/show/629242/ | 20:21 |
| jaypipes | edleafe: very good question... | 20:22 |
| jroll | ahem, projects | 20:22 |
| jaypipes | lol | 20:22 |
| jaypipes | jroll: well, disregarding my current functional test woes, your eyeballs on https://review.openstack.org/#/c/528804/1/nova/scheduler/filters/affinity_filter.py would be useful. | 20:23 |
| jroll | jaypipes: sure thing, on my list | 20:23 |
| jaypipes | ya, no rush.. | 20:23 |
| windsn | mriedem: I ran the command ""nova-manage cell_v2 discover_hosts", but i still dont see the hypervisor-list getting populated with the ironic node | 20:24 |
| *** armax has joined #openstack-nova | 20:24 | |
| *** huanxie has quit IRC | 20:24 | |
| windsn | mriedem: is the "nova-manage cell v2 discover_host" supposed to create a "host_mapping" entry in the nova db ? I don't see that happening | 20:26 |
| *** huanxie has joined #openstack-nova | 20:26 | |
| mriedem | edleafe: ah i think i know why the ironic multinode job is failing to reschedule | 20:28 |
| mriedem | "Re-scheduling is disabled" | 20:28 |
| mriedem | if the retryfilter isn't running, then max_attempts = 0 and we wouldn't get alternates yeah? | 20:28 |
| *** penick has joined #openstack-nova | 20:28 | |
| mriedem | jaypipes: likely using the latest microversion | 20:29 |
| mriedem | so we don't proxy for images | 20:29 |
| mriedem | you have to pass a specific image to _build_minimal_create_server_request i think | 20:29 |
| mriedem | something like that | 20:29 |
| mriedem | windsn: supposed to create a host_mapping entry in the nova_api db | 20:30 |
| mriedem | jaypipes: yeah "GET /v2.1/6f70656e737461636b20342065766572/images/detail" status: 404 len: 78 microversion: 2.59 | 20:30 |
| mriedem | that's capped at 2.35 | 20:31 |
| jaypipes | mriedem: sorry, I'm not following you... | 20:31 |
| melwitt | aha, yup. good thinkin mriedem | 20:31 |
| mriedem | jaypipes: your test is using the latest microversion by default | 20:31 |
| mriedem | or you've set self.microversion = 'latest' | 20:31 |
| mriedem | the /images proxy API is capped at 2.35 | 20:31 |
| jaypipes | mriedem: it's not *my* test :) it's a copy of one of the tests that is already in there... | 20:32 |
| melwitt | as of that microversion, we no longer proxy image API requests to glance, so you're getting not found | 20:32 |
| mriedem | anyway, you'll need to hard-code the image_uuid http://git.openstack.org/cgit/openstack/nova/tree/nova/tests/functional/integrated_helpers.py#n140 | 20:32 |
| mriedem | to one of the images in our fake image service | 20:32 |
| jaypipes | mriedem: ah... since the test I copied from sets microversion to 2.15, it didn't bomb it. | 20:32 |
| edleafe | mriedem: also, alternates are hosts, not nodes, so unless there are multiple computes per cell for ironic, there won't be alternates | 20:32 |
| mriedem | jaypipes: https://review.openstack.org/#/c/528804/1/nova/tests/functional/test_server_group.py@840 | 20:32 |
| jaypipes | mriedem: that is SO clear :/ | 20:32 |
| mriedem | you are setting the microversion to 2.59 | 20:32 |
| melwitt | right, it's setting microversion = '2.59' that does it | 20:32 |
| mriedem | edleafe: yeah i know, but in this case it's saying it's finding 12 hosts to start from | 20:33 |
| mriedem | edleafe: http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-sch.txt.gz#_Dec_12_22_23_47_783670 | 20:33 |
| jroll | "alternates are hosts, not nodes" O_O | 20:33 |
| jaypipes | mriedem: right, what I was saying was the test case I copied that from sets microversion to 2.15 and it just so happened that 2.15 < 2.35 | 20:33 |
| jroll | edleafe: that feels very wrong, is that code merged or is there still time to review it? :) | 20:34 |
| mriedem | jroll: yeah it's not completely baked for ironic yet obviously | 20:34 |
| jroll | right | 20:34 |
| mriedem | jroll: the code to use it isn't merged yet | 20:34 |
| mriedem | https://review.openstack.org/#/c/511358/40 | 20:34 |
| jroll | awesome, ty | 20:34 |
| mriedem | i have https://review.openstack.org/#/c/527289/ and am debugging it | 20:34 |
| jaypipes | mriedem: so I have to figure out how to adapt the functional test base so that only the _create_server_group() uses the microversion at 2.59 and everything else uses 2.1... | 20:34 |
| jaypipes | fun. | 20:34 |
| edleafe | jroll: yeah, we discussed updating that for unique host/node, but the first round it's just host | 20:34 |
| jroll | edleafe: gotcha, will take a look around | 20:35 |
| edleafe | jroll: the code to create the alternates has merged. | 20:35 |
| *** pchavva has joined #openstack-nova | 20:35 | |
| *** jmlowe has quit IRC | 20:36 | |
| *** namnh has joined #openstack-nova | 20:38 | |
| *** lbragstad has quit IRC | 20:38 | |
| *** lbragstad has joined #openstack-nova | 20:39 | |
| melwitt | jaypipes: it's ghetto, but you could just do what mriedem suggested and add something like this to _boot_a_server_to_group https://github.com/openstack/nova/blob/master/nova/tests/functional/test_servers.py#L1449 | 20:40 |
| *** _ix has quit IRC | 20:40 | |
| melwitt | I don't know who figured out that's always the image uuid of the first image but that's what it appears to me | 20:40 |
| mriedem | jaypipes: you can use the self.api client directly | 20:41 |
| mriedem | if you want to avoid the base class method to create the server group | 20:41 |
| *** namnh has quit IRC | 20:42 | |
| jaypipes | mriedem: I am looking at nova/tests/functional/api/client.py to try and understand how to pass a specific microversion... I don't see it. | 20:44 |
| mriedem | oh right... | 20:44 |
| mriedem | you can't, and the client specifically blows up if you try passing your own header | 20:44 |
| mriedem | jaypipes: please place your nuts here < > so i can kick you square in them | 20:44 |
| jaypipes | mriedem: looks like I will need to do somethin glike this: | 20:44 |
| mriedem | self.api.microversion = '2.59' | 20:44 |
| mriedem | self.api.post_server_group() | 20:45 |
| jaypipes | self.api.microversion = '2.59'; do something...; self.api.microversion = orig | 20:45 |
| mriedem | self.api.microversion = old_value | 20:45 |
| mriedem | yeah | 20:45 |
| jaypipes | heh, jinux | 20:45 |
| jaypipes | or even jinx. | 20:45 |
| mriedem | jinux is your personal linux distro? | 20:45 |
| jaypipes | I guess jinux is linux jinx. | 20:45 |
| mriedem | jay's linux | 20:45 |
| jaypipes | heh | 20:45 |
| * jaypipes goes to get the shotgun | 20:45 | |
| mriedem | this is the thing that blows up if you try to pass a header override https://github.com/openstack/nova/blob/master/nova/tests/functional/api/client.py#L179 | 20:45 |
| mriedem | however, i think that was added when this was all written by sdague and was very specific to how the tests worked with scenarios | 20:46 |
| jaypipes | I'll just add the microversion override as above with the reset right after | 20:46 |
| mriedem | i'd be cool with removing that so you can run tests at a default minimum microversion and then make certain calls with a higher microversion, or vice-versa | 20:46 |
| *** cdent has quit IRC | 20:47 | |
| *** raildo has quit IRC | 20:48 | |
| mriedem | btw, this is a useful code comment http://git.openstack.org/cgit/openstack/nova/tree/nova/scheduler/filters/retry_filter.py#n37 | 20:48 |
| jaypipes | mriedem: about equally as useful as the majority of code comments in the scheduler filters. | 20:49 |
| melwitt | lol | 20:49 |
| jaypipes | also, god help whomever needs to change "tenant" -> "project" when we no longer support keystone 2.0.' | 20:50 |
| mriedem | edleafe: ok this is really confusing http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-sch.txt.gz#_Dec_12_22_23_47_783964 | 20:50 |
| mriedem | we have 12 candidate hosts, | 20:51 |
| mriedem | initial select_destinations call for this create request | 20:51 |
| mriedem | and we log "Re-scheduling is disabled" 12 times, once per host, because of the RetryFilter, | 20:51 |
| *** fragatina has joined #openstack-nova | 20:51 | |
| mriedem | and i think it's all because we don't populate the 'retry' field in the RequestSpec before calling select_destinations the first time | 20:51 |
| mriedem | during create we call the scheduler here http://git.openstack.org/cgit/openstack/nova/tree/nova/conductor/manager.py#n1016 | 20:52 |
| mriedem | and we don't populate retry until http://git.openstack.org/cgit/openstack/nova/tree/nova/conductor/manager.py#n1108 | 20:52 |
| mriedem | i'm not sure if that was ever by design | 20:52 |
| *** ChanServ has quit IRC | 20:52 | |
| mriedem | looking at the old build_instances method, we'd populate retry before calling the scheduler http://git.openstack.org/cgit/openstack/nova/tree/nova/conductor/manager.py#n543 | 20:52 |
| mriedem | again, probably not our issue here since max_attempts=3 by default, and i verified that in this job, but it sure is confusing in the logs | 20:53 |
| * mriedem opens bug | 20:53 | |
| *** jackie-truong has joined #openstack-nova | 20:55 | |
| *** huanxie has quit IRC | 20:56 | |
| *** jackie-truong has quit IRC | 20:56 | |
| mriedem | https://bugs.launchpad.net/nova/+bug/1738876 | 20:57 |
| openstack | Launchpad bug 1738876 in OpenStack Compute (nova) "Scheduler logs "Re-scheduling is disabled" on initial server create scheduling even though max_attempts>0" [Medium,Triaged] | 20:57 |
| *** linkmark has joined #openstack-nova | 20:59 | |
| *** ChanServ has joined #openstack-nova | 21:01 | |
| *** barjavel.freenode.net sets mode: +o ChanServ | 21:01 | |
| *** mingyu has joined #openstack-nova | 21:08 | |
| edleafe | mriedem: with alternates, isn't the RetryFilter supposed to go away? | 21:09 |
| mriedem | i suppose it would over time | 21:09 |
| edleafe | since you won't try the same host more than once? | 21:10 |
| edleafe | IOW, it'll always be a noop | 21:10 |
| *** r-daneel has quit IRC | 21:11 | |
| *** r-daneel has joined #openstack-nova | 21:11 | |
| *** mingyu has quit IRC | 21:13 | |
| *** archit is now known as amodi | 21:13 | |
| mriedem | edleafe: this shows that we're getting duplicate alternates http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/22826a2/logs/screen-n-super-cond.txt.gz#_Dec_15_23_17_04_949754 | 21:17 |
| mriedem | Selected host: ubuntu-xenial-inap-mtl01-0001472200; Alternates: [u'ubuntu-xenial-inap-mtl01-0001472200', u'ubuntu-xenial-inap-mtl01-0001472200'] | 21:17 |
| mriedem | hmm | 21:19 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L361 | 21:19 |
| mriedem | host not in selected_hosts | 21:19 |
| *** felipemonteiro_ has quit IRC | 21:19 | |
| mriedem | we don't have a HostState.__eq__ override | 21:19 |
| edleafe | mriedem: wait - a HostState object isn't equal to itself? | 21:20 |
| *** catintheroof has quit IRC | 21:20 | |
| openstackgerrit | Merged openstack/nova stable/pike: Make request_spec.spec MediumText https://review.openstack.org/528330 | 21:21 |
| mriedem | edleafe: i'm not sure if that is the issue, | 21:22 |
| mriedem | but https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L361 is the place where we'd filter out duplicate alternates wouldn't it? | 21:22 |
| mriedem | oh wait, no | 21:23 |
| *** smatzek_ has quit IRC | 21:23 | |
| mriedem | maybe the logging you added to conductor is just incomplete because it doesn't also log the nodename? | 21:23 |
| edleafe | that could be the case | 21:24 |
| edleafe | I have some unit tests that verify that there aren't dupes added to alternates | 21:24 |
| mriedem | ok, so in the case of ironic, we could be getting alternate nodes properly from the scheduler | 21:25 |
| edleafe | I can add nodename and push a revision if you want to test that theory | 21:26 |
| mriedem | but the reschedule logic doesn't take that into account | 21:26 |
| mriedem | edleafe: yeah let's do that - i left some other comments in conductor manager about that log message | 21:26 |
| *** smatzek has joined #openstack-nova | 21:26 | |
| edleafe | ok | 21:28 |
| mriedem | this also explains why http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/22826a2/ passed the 2nd time i checked it, | 21:28 |
| mriedem | it's just totally random if we pick one of the alternates which is a different node on a different hose when we reschedule | 21:29 |
| mriedem | jroll: so to answer your question, reschedules with alternate ironic nodes might work, but it's not designed to work yet :) | 21:29 |
| jroll | heh, cool | 21:29 |
| *** huanxie has joined #openstack-nova | 21:30 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | 21:30 |
| *** smatzek has quit IRC | 21:30 | |
| edleafe | mriedem: ^^ now with improved logging! | 21:31 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: WIP - Support aggregate affinity filters https://review.openstack.org/528804 | 21:32 |
| *** openstackgerrit has quit IRC | 21:33 | |
| *** gbarros has quit IRC | 21:35 | |
| mriedem | edleafe: you changed one thing, broke one thing, and missed some other things :) | 21:35 |
| edleafe | about par for the course | 21:35 |
| mriedem | before you change, hold up | 21:35 |
| mriedem | so actually, this should work for ironic | 21:36 |
| *** nore__ has joined #openstack-nova | 21:37 | |
| mriedem | there is nothing in the scheduler or conductor retry loop code, that i can see, which is filtering alternates based on just the host | 21:37 |
| mriedem | so the failure we hit in the ironic job originally where it tried to upcall, is probably just do to not handling the final retry properly and killing it | 21:38 |
| mriedem | is_retry = host_lists is not None | 21:38 |
| mriedem | on the last reschedule, that could be None or an empty list yeah? | 21:38 |
| *** jose-phillips has quit IRC | 21:38 | |
| *** AlexeyAbashkin has joined #openstack-nova | 21:38 | |
| edleafe | how could it be None? | 21:39 |
| *** nore_rabel has quit IRC | 21:39 | |
| mriedem | # 'host_lists' will be None in one of two cases: when running | 21:40 |
| mriedem | # cellsv1, or during a reschedule from a pre-Queens compute. | 21:40 |
| edleafe | yeah | 21:40 |
| mriedem | i think i explain the problem scenario here https://review.openstack.org/#/c/511358/41/nova/conductor/manager.py@599 | 21:40 |
| edleafe | not on a reschedule | 21:41 |
| mriedem | we have 2 alternates, and let's say we're on the last one | 21:41 |
| mriedem | we pop that last entry from the host_list | 21:41 |
| mriedem | pass it to compute, which fails, and reschedules, and passes that empty list back to build_instances | 21:41 |
| edleafe | yup | 21:41 |
| edleafe | it won't be None | 21:41 |
| mriedem | i think i'm confusing myself | 21:42 |
| *** AlexeyAbashkin has quit IRC | 21:42 | |
| edleafe | it will be empty if there are no more alternates | 21:43 |
| edleafe | but it will be a list | 21:43 |
| edleafe | mriedem: https://review.openstack.org/#/c/511358/41/nova/compute/manager.py@1905 | 21:44 |
| mriedem | yeah the list of lists there really confuses me | 21:45 |
| mriedem | you said you're doing that because of [instance] | 21:45 |
| mriedem | but i don't think we need to follow that pattern here | 21:45 |
| edleafe | conductor gets [instance], so it need [host_list] | 21:46 |
| edleafe | compute gets instance, so it needs host_list | 21:46 |
| mriedem | so the zip works? | 21:46 |
| edleafe | that's one of many reasons | 21:46 |
| edleafe | it needs a host_list for each instance | 21:47 |
| *** jose-phillips has joined #openstack-nova | 21:47 | |
| mriedem | ok, not a major issue for me atm | 21:47 |
| mriedem | for that one ci job failure, this is the reason the first selected node failed | 21:47 |
| mriedem | http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cpu.txt.gz#_Dec_12_22_23_48_399346 | 21:47 |
| mriedem | ComputeHostNotFound: Compute host 5 could not be found. | 21:47 |
| edleafe | any idea why that host couldn't be found? | 21:49 |
| mriedem | nope | 21:49 |
| mriedem | we might not be using the proper cell context | 21:49 |
| *** BryanS68 has quit IRC | 21:51 | |
| mriedem | so we hit the first selected node, which fails because of compute node not found, | 21:52 |
| mriedem | and we cast up to the cell conductor to reschedule and hit this http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cond-cell1.txt.gz#_Dec_12_22_23_48_751837 | 21:52 |
| *** andreas_s has joined #openstack-nova | 21:54 | |
| mriedem | AHA | 21:55 |
| mriedem | i see the problemo | 21:55 |
| mriedem | oh ed | 21:55 |
| mriedem | ed ed ed | 21:55 |
| edleafe | you were getting me confused. How could the compute call back when the compute wasn't found? | 21:56 |
| *** burt has quit IRC | 21:58 | |
| mriedem | comments line | 21:58 |
| mriedem | *inline | 21:58 |
| mriedem | you were passing host_list using the wrong arg | 21:59 |
| edleafe | looking | 21:59 |
| *** andreas_s has quit IRC | 21:59 | |
| mriedem | that's why it was None when it came back up and we tried to hit the scheduler again | 21:59 |
| mriedem | use kwargs ftw | 21:59 |
| mriedem | so i think ironic reschedules should be totally fine | 22:00 |
| mriedem | jroll: ^ | 22:00 |
| mriedem | code as is is busted so it wasn't trying alternates | 22:00 |
| mriedem | once we fix that, things should be good | 22:00 |
| *** huanxie has quit IRC | 22:01 | |
| jroll | nice | 22:01 |
| *** huanxie has joined #openstack-nova | 22:03 | |
| *** openstackgerrit has joined #openstack-nova | 22:04 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | 22:04 |
| *** moshele has joined #openstack-nova | 22:06 | |
| edleafe | mriedem: ^^ updated | 22:06 |
| mriedem | edleafe: ok, that should fix the actual reschedule issue, but still missing comments from PS41 so re-applied my -1 | 22:07 |
| mriedem | if you're about end of day, | 22:07 |
| mriedem | then ill kick off the ironic job again | 22:07 |
| mriedem | and we can check results in the morn | 22:07 |
| edleafe | I just wanted to get you that fix first | 22:08 |
| edleafe | working on the other comments | 22:08 |
| mriedem | ok | 22:08 |
| edleafe | mriedem: re: your question on https://review.openstack.org/#/c/511358/41/nova/conductor/manager.py@578 | 22:11 |
| edleafe | Should that failure be handled at the compute level, or up here in the conductor? | 22:12 |
| mriedem | edleafe: conductor | 22:13 |
| mriedem | scheduler_utils.populate_retry is what raises MaxRetriesExceeded today | 22:14 |
| mriedem | so, maybe populate_retry just handles my issue on L578 | 22:14 |
| mriedem | since it comes before we'd get that far | 22:14 |
| *** klindgren has joined #openstack-nova | 22:15 | |
| edleafe | The question is at this point, if we have an empty host_list, should I raise a MaxRetriesExceeded exception? | 22:16 |
| mriedem | i don't think we'll get that far | 22:16 |
| mriedem | populate_retry should fail before that | 22:16 |
| mriedem | if you're asking about just being explicit... | 22:17 |
| mriedem | it's definitely confusing that populate_retry is also the "check if you've gone over the retry limit thing" | 22:17 |
| edleafe | that confused me too | 22:17 |
| mriedem | i'd be fine with an explicit check | 22:17 |
| edleafe | so if we reach 578 and have an empty host_list, something is seriously wrong | 22:18 |
| mriedem | yes | 22:18 |
| edleafe | what would be the best exception to raise there? Besides a YoureHosedException? | 22:19 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: Provide VGPU inventory for a single GPU type https://review.openstack.org/514763 | 22:20 |
| openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: libvirt: create vGPU for instance https://review.openstack.org/528832 | 22:20 |
| mriedem | edleafe: MaxRetriesExceedd? | 22:20 |
| mriedem | *exceeded | 22:20 |
| mriedem | that, or NoValidHost | 22:20 |
| mriedem | honestly L578 shouldn't be there | 22:20 |
| edleafe | Well, we found valid hosts, but just couldn't build on 'em | 22:20 |
| mriedem | the loop should assume we have a host per instance | 22:21 |
| mriedem | if we don't, something higher up in the method should have caught it | 22:21 |
| edleafe | ok, I'll remove it | 22:21 |
| mriedem | this is what should catch that case https://review.openstack.org/#/c/511358/42/nova/conductor/manager.py@545 | 22:21 |
| mriedem | it's definitely confusing to have the retry information spread and maintained across filter properties, request spec and the host list stuff, but that's what we have | 22:22 |
| mriedem | if for some reason populate_retry didn't catch it and host_lists was empty, then we have to handle it explicitly | 22:22 |
| *** pchavva has quit IRC | 22:22 | |
| mriedem | and, host_lists wouldn't be empty, because it'd be [[]] right? | 22:23 |
| mriedem | a list with a single entry, which is an empty list | 22:23 |
| edleafe | yeah, that's the plan | 22:23 |
| mriedem | >>> if [[]]: | 22:23 |
| mriedem | ... print 'empty lists are fun' | 22:23 |
| mriedem | ... | 22:23 |
| mriedem | empty lists are fun | 22:23 |
| edleafe | ok, what about this scenario: | 22:23 |
| edleafe | max_attempts=5 | 22:23 |
| edleafe | only 2 alternates were found | 22:23 |
| edleafe | the populate_retry thing won't catch that, will it? | 22:24 |
| mriedem | no it wouldn't, | 22:24 |
| mriedem | in that case, with the old flow, we'd call the scheduler and get NoValidHost | 22:24 |
| mriedem | ok so given that, we can't rely on populate_retry | 22:25 |
| edleafe | So I'll leave the check for an empty list, and raise MaxRetriesExceeded | 22:25 |
| mriedem | well, L578 isn't the place to do that | 22:25 |
| mriedem | we need it to fail in the try/except above | 22:25 |
| mriedem | so it sets the vm_state to ERROR | 22:25 |
| mriedem | so how about an else to this? "if not is_retry:" | 22:26 |
| mriedem | if not is_retry: select_dests elif not host_list[0]: raise MaxRetriesExceeded | 22:26 |
| edleafe | ok, I'll move the check there | 22:26 |
| mriedem | note that in that case, we shouldn' try to destroy the build request in the except block | 22:27 |
| edleafe | why not? | 22:28 |
| mriedem | because if we're in a cell conductor we don't have access to the api db | 22:28 |
| mriedem | and rather than BuildRequestNotFound, we'll get CantStartEngineError | 22:28 |
| mriedem | ala https://bugs.launchpad.net/nova/+bug/1733933 | 22:28 |
| openstack | Launchpad bug 1733933 in OpenStack Compute (nova) "nova-conductor is masking error when rescheduling" [Low,Triaged] | 22:28 |
| edleafe | ah. I don't know how you keep all that straight | 22:29 |
| mriedem | well, that's also what failed here http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cond-cell1.txt.gz#_Dec_12_22_23_48_751837 | 22:29 |
| mriedem | er this i mean http://logs.openstack.org/89/527289/1/check/ironic-tempest-dsvm-ipa-wholedisk-agent_ipmitool-tinyipa-multinode/406d4ab/logs/screen-n-cond-cell1.txt.gz#_Dec_12_22_23_48_940236 | 22:29 |
| mriedem | File "/opt/stack/new/nova/nova/conductor/manager.py", line 478, in _destroy_build_request | 22:29 |
| mriedem | i'll leave a note in the code so we don't forget | 22:30 |
| *** huanxie has quit IRC | 22:32 | |
| *** huanxie has joined #openstack-nova | 22:33 | |
| mriedem | edleafe: ok commented. after thinking through that, it might be best to leave L578 | 22:34 |
| mriedem | because host_lists is going to be per-instance right? | 22:34 |
| mriedem | although, if we're rescheduling, it will be a single entry yeah? | 22:34 |
| edleafe | yeah, always single on reschedule | 22:35 |
| edleafe | also, the populate_retry method pops the exc_reason from the filter_properties, so it isn't available later in build_instances | 22:36 |
| edleafe | why not check above as you mentioned before? | 22:36 |
| mriedem | we could - i left yet another comment | 22:37 |
| mriedem | i think the build request cleanup we have in here is likely latent broken | 22:37 |
| mriedem | if we're rescheduling | 22:37 |
| mriedem | if we get here https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L559 because populate_retry raised MaxRetriesExceeded, | 22:37 |
| *** rcernin has joined #openstack-nova | 22:37 | |
| mriedem | we shouldn't try to delete the build request | 22:37 |
| mriedem | melwitt: you might want to check my thinking here | 22:38 |
| *** awaugama has quit IRC | 22:38 | |
| *** AlexeyAbashkin has joined #openstack-nova | 22:38 | |
| *** namnh has joined #openstack-nova | 22:39 | |
| melwitt | yeah ... the way that code is written, it's assuming we have an instance record. and I thought we do indeed have an instance in ERROR state if max retries were reached | 22:41 |
| melwitt | I can't remember | 22:41 |
| *** AlexeyAbashkin has quit IRC | 22:42 | |
| *** eharney has quit IRC | 22:42 | |
| melwitt | also, that's cells v1 only code where I think the instance record is created right away in the compute/api. so a retries exceeded would have an instance record | 22:42 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Don't try to delete build requests on reschedules exceeded https://review.openstack.org/528835 | 22:42 |
| mriedem | melwitt: edleafe: ^ it's a start | 22:42 |
| mriedem | gotta run | 22:42 |
| edleafe | mriedem: ok, will post an update soon | 22:43 |
| *** namnh has quit IRC | 22:43 | |
| edleafe | and I renamed 'is_retry' to 'is_reschedule' | 22:43 |
| edleafe | just seems more accurate | 22:43 |
| melwitt | oh, cells v1 AND reschedules go through build_instances. guh | 22:46 |
| *** kumarmn has quit IRC | 22:47 | |
| edleafe | melwitt: fun, right? | 22:47 |
| *** moshele has quit IRC | 22:48 | |
| melwitt | totally | 22:48 |
| *** huanxie has quit IRC | 23:03 | |
| *** pchavva has joined #openstack-nova | 23:03 | |
| *** huanxie has joined #openstack-nova | 23:06 | |
| *** kumarmn has joined #openstack-nova | 23:06 | |
| *** catinthe_ has joined #openstack-nova | 23:08 | |
| *** kumarmn has quit IRC | 23:11 | |
| *** gouthamr has joined #openstack-nova | 23:12 | |
| *** catinthe_ has quit IRC | 23:13 | |
| rybridges | Hey guys | 23:14 |
| rybridges | Any idea why running suspending an instance would put it into an error state? Here is the log from nova-compute https://pastebin.com/jTedyZVJ | 23:14 |
| mgagne | can't wait for policy-in-code to land in all projects... =) | 23:15 |
| mgagne | I guess that would be one major reason I would upgrade | 23:16 |
| cfriesen | rybridges: is libvirtd running? | 23:16 |
| rybridges | here is the output of server show https://pastebin.com/8eQiz3hA | 23:17 |
| cfriesen | rybridges: check the libvirtd/qemu logs | 23:17 |
| rybridges | yes libvirtd is running | 23:17 |
| rybridges | this is happening intermittently | 23:17 |
| rybridges | sometimes it happens | 23:17 |
| rybridges | other times it is fine | 23:17 |
| cfriesen | your nova logs indicate that the call to virDomainManagedSave() failed with a connection reset...this implies an issue with libvirtd, I think. | 23:18 |
| cfriesen | I gotta run, suppertime | 23:19 |
| rybridges | right | 23:21 |
| rybridges | i figured that much | 23:21 |
| rybridges | why would that call be failing? | 23:21 |
| rybridges | i dont see anything in qemu logs | 23:21 |
| *** mlavalle has quit IRC | 23:21 | |
| rybridges | i am grepping for all sorts of stuff in /var/log/libvirt/qemu | 23:22 |
| rybridges | the hypervisor is rhel7 btw | 23:22 |
| *** tonyb has quit IRC | 23:26 | |
| *** nore__ has quit IRC | 23:27 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add functional tests for resource class API https://review.openstack.org/524506 | 23:30 |
| *** takashin has joined #openstack-nova | 23:31 | |
| melwitt | rybridges: you'll want to look at the libvirt daemon logs, libvirtd.log. also look at /var/log/messages | 23:31 |
| rybridges | by "libvirt daemon logs" do you mean the journalctl? | 23:31 |
| melwitt | I'm most familiar with just looking at /var/log/libvirt/libvirtd.log but journalctl should show the same thing I think | 23:32 |
| rybridges | i dont have anything in /var/log/libvirt/libvirtd.log | 23:35 |
| melwitt | meaning, you don't have that file? your libvirtd might be configured to log somewhere else | 23:35 |
| rybridges | the file is not there | 23:35 |
| rybridges | yea i can check the conf | 23:35 |
| *** tonyb has joined #openstack-nova | 23:36 | |
| clarkb | if it is going to journald you can do journalctl -u libvirtd (or whatever the unit name is) | 23:36 |
| *** huanxie has quit IRC | 23:36 | |
| clarkb | the -u restricts it to printing logs only belonging to the listed units | 23:37 |
| rybridges | right | 23:37 |
| rybridges | i am using that.. | 23:37 |
| rybridges | i will see if i find anything | 23:37 |
| *** kumarmn has joined #openstack-nova | 23:38 | |
| *** harlowja has quit IRC | 23:38 | |
| *** huanxie has joined #openstack-nova | 23:39 | |
| rybridges | o | 23:41 |
| rybridges | ok* | 23:41 |
| rybridges | libvirtd is showing: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error | 23:41 |
| *** kumarmn has quit IRC | 23:47 | |
| *** kumarmn has joined #openstack-nova | 23:48 | |
| rybridges | am also seeing the following in some places: | 23:51 |
| rybridges | virDBusCall:1558 : DBUS_METHOD_ERROR: 'org.freedesktop.machine1.Manager.TerminateMachine' on '/org/freedesktop/machine1' at 'org.freedesktop.machine1' error org.freedesktop.machine1.NoSuchMachine: No machine 'qemu-47-instance-000002e9' known | 23:51 |
| rybridges | not sure if that is related | 23:51 |
| rybridges | and i also see this: | 23:52 |
| rybridges | error : qemuDomainObjEnterMonitorInternal:4003 : operation failed: domain is no longer running | 23:52 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | 23:54 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | 23:54 |
| edleafe | mriedem: melwitt: ^^ moar updated | 23:54 |
| * edleafe heads out | 23:54 | |
| *** mingyu has joined #openstack-nova | 23:58 | |
| *** tetsuro has joined #openstack-nova | 23:59 | |
| *** kumarmn has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!