| *** gbarros has quit IRC | 00:59 | |
| *** yedongcan has joined #openstack-nova | 01:19 | |
| *** yikun has joined #openstack-nova | 02:02 | |
| *** zhubx has joined #openstack-nova | 02:04 | |
| *** BjoernT has joined #openstack-nova | 02:22 | |
| *** BjoernT_ has joined #openstack-nova | 02:26 | |
| *** BjoernT has quit IRC | 02:26 | |
| *** dklyle has quit IRC | 02:33 | |
| *** JamesBenson has joined #openstack-nova | 02:35 | |
| openstackgerrit | Fan Zhang proposed openstack/nova master: Fix exception translation when creating volume https://review.opendev.org/678991 | 02:40 |
|---|---|---|
| *** BjoernT_ has quit IRC | 02:42 | |
| *** BjoernT has joined #openstack-nova | 02:45 | |
| *** dklyle has joined #openstack-nova | 02:48 | |
| *** larainema has joined #openstack-nova | 03:00 | |
| *** mkrai has joined #openstack-nova | 03:19 | |
| *** ricolin has joined #openstack-nova | 03:20 | |
| *** dklyle has quit IRC | 03:20 | |
| *** psachin has joined #openstack-nova | 03:27 | |
| *** markvoelker has joined #openstack-nova | 03:54 | |
| *** markvoelker has quit IRC | 03:58 | |
| *** sapd1_x has joined #openstack-nova | 03:59 | |
| *** BjoernT has quit IRC | 04:06 | |
| *** BjoernT has joined #openstack-nova | 04:14 | |
| *** JamesBenson has quit IRC | 04:31 | |
| *** brault has quit IRC | 04:43 | |
| *** BjoernT has quit IRC | 04:46 | |
| *** dave-mccowan has quit IRC | 04:48 | |
| openstackgerrit | wangyue proposed openstack/nova master: Returns the max disk/cdrom unit used by scsi controller https://review.opendev.org/683845 | 04:56 |
| *** slaweq has joined #openstack-nova | 04:57 | |
| *** ircuser-1 has quit IRC | 04:58 | |
| *** Luzi has joined #openstack-nova | 05:00 | |
| *** slaweq has quit IRC | 05:05 | |
| *** ratailor has joined #openstack-nova | 05:10 | |
| *** slaweq has joined #openstack-nova | 05:10 | |
| *** sapd1_x has quit IRC | 05:19 | |
| *** macz has joined #openstack-nova | 05:26 | |
| *** slaweq has quit IRC | 05:28 | |
| *** slaweq has joined #openstack-nova | 05:30 | |
| *** macz has quit IRC | 05:31 | |
| openstackgerrit | hulina proposed openstack/nova master: Nova raise exceptions when extending volume fails https://review.opendev.org/680648 | 05:42 |
| *** jaosorior has joined #openstack-nova | 05:43 | |
| *** brault has joined #openstack-nova | 05:49 | |
| *** dpawlik has joined #openstack-nova | 06:00 | |
| *** luksky has joined #openstack-nova | 06:17 | |
| *** brault has quit IRC | 06:17 | |
| *** yikun has quit IRC | 06:21 | |
| *** luksky has quit IRC | 06:23 | |
| *** TxGirlGeek has joined #openstack-nova | 06:23 | |
| *** maciejjozefczyk has joined #openstack-nova | 06:26 | |
| *** slaweq has quit IRC | 06:27 | |
| openstackgerrit | Chason Chan proposed openstack/nova master: Note the ``hw_numa_nodes`` image property https://review.opendev.org/683849 | 06:29 |
| *** maciejjozefczyk has quit IRC | 06:30 | |
| *** slaweq has joined #openstack-nova | 06:34 | |
| *** rpittau|afk is now known as rpittau | 06:40 | |
| *** brault has joined #openstack-nova | 06:41 | |
| *** brault has quit IRC | 06:41 | |
| *** brault has joined #openstack-nova | 06:41 | |
| *** TxGirlGeek has quit IRC | 06:42 | |
| *** damien_r has joined #openstack-nova | 06:59 | |
| *** tesseract has joined #openstack-nova | 07:00 | |
| *** rcernin has quit IRC | 07:04 | |
| *** slaweq has quit IRC | 07:04 | |
| *** slaweq has joined #openstack-nova | 07:06 | |
| *** ivve has joined #openstack-nova | 07:10 | |
| *** cshen has joined #openstack-nova | 07:14 | |
| *** ccamacho has joined #openstack-nova | 07:16 | |
| *** xek has joined #openstack-nova | 07:17 | |
| *** pcaruana has joined #openstack-nova | 07:41 | |
| *** eharney has joined #openstack-nova | 07:44 | |
| *** slaweq has quit IRC | 07:47 | |
| *** zhubx has quit IRC | 07:54 | |
| *** boxiang has joined #openstack-nova | 07:55 | |
| *** ralonsoh has joined #openstack-nova | 07:59 | |
| *** toabctl has joined #openstack-nova | 08:02 | |
| *** brault has quit IRC | 08:21 | |
| openstackgerrit | pengyuesheng proposed openstack/os-resource-classes master: Update the constraints url https://review.opendev.org/683872 | 08:25 |
| openstackgerrit | pengyuesheng proposed openstack/os-vif master: Update the constraints url https://review.opendev.org/683873 | 08:27 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Move HostNameWeigher to a common fixture https://review.opendev.org/683874 | 08:28 |
| *** jangutter has joined #openstack-nova | 08:31 | |
| *** brault has joined #openstack-nova | 08:32 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Remove stale BDMs on reserve_block_device_name failure https://review.opendev.org/682594 | 08:38 |
| *** rcernin has joined #openstack-nova | 08:45 | |
| *** lpetrut has joined #openstack-nova | 08:48 | |
| *** derekh has joined #openstack-nova | 08:49 | |
| *** markvoelker has joined #openstack-nova | 08:57 | |
| *** CeeMac has joined #openstack-nova | 08:58 | |
| *** markvoelker has quit IRC | 09:02 | |
| *** dpawlik has quit IRC | 09:04 | |
| *** dpawlik has joined #openstack-nova | 09:04 | |
| *** ociuhandu has joined #openstack-nova | 09:05 | |
| *** martinkennelly has joined #openstack-nova | 09:05 | |
| *** dtantsur|afk is now known as dtantsur | 09:07 | |
| openstackgerrit | Silvan Kaiser proposed openstack/nova stable/stein: Exec systemd-run without --user flag in Quobyte driver https://review.opendev.org/660705 | 09:09 |
| *** yikun has joined #openstack-nova | 09:23 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Note use of 'nova-manage db sync --config-file' https://review.opendev.org/671298 | 09:36 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Correct 'nova-manage db sync' documentation https://review.opendev.org/677508 | 09:36 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Document global options for nova-manage https://review.opendev.org/677443 | 09:36 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: config: Explicitly register 'remote_debug' CLI opts https://review.opendev.org/677444 | 09:36 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: docs: Rewrite nova-manage docs to use proper directives https://review.opendev.org/677509 | 09:36 |
| *** dpawlik has quit IRC | 09:37 | |
| stephenfin | bauzas: Think you could send this trivial patch through? https://review.opendev.org/#/c/676898/ | 09:37 |
| bauzas | morning :) | 09:37 |
| *** dpawlik has joined #openstack-nova | 09:38 | |
| bauzas | and done | 09:38 |
| bauzas | stephenfin: congrats for your rugby team btw., nice play ;) | 09:38 |
| stephenfin | Heh, thanks :) | 09:39 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Rewrite host aggregate, availability zone docs https://review.opendev.org/667133 | 09:39 |
| *** awalende has joined #openstack-nova | 09:39 | |
| *** lpetrut has quit IRC | 09:45 | |
| *** lpetrut has joined #openstack-nova | 09:48 | |
| *** ricolin has quit IRC | 09:48 | |
| kashyap | aspiers: Hey, if you're back ... I'm feeling stupid ... you know why I'm getting the 'secure' enum value _three_ times here? -- http://paste.openstack.org/show/778857/ | 09:51 |
| aspiers | o/ | 09:52 |
| kashyap | Full "self-contained" WIP script in the pastebin :D | 09:53 |
| aspiers | ImportError: No module named libvirt | 09:55 |
| aspiers | I haven't woken up yet, help me | 09:55 |
| kashyap | aspiers: Err, I lied. | 09:55 |
| aspiers | the tox envs don't have libvirt? | 09:56 |
| kashyap | aspiers: For it to be "self-contained", I ran it from my Nova 'tox' directory sourced. | 09:56 |
| kashyap | aspiers: Yes, that's correct | 09:56 |
| aspiers | which tox env? | 09:56 |
| kashyap | aspiers: (Missing libvirt from your 'tox' env) | 09:56 |
| aspiers | There are many | 09:56 |
| aspiers | I've tried py27 and functional | 09:56 |
| kashyap | Hm, I'm using py27 | 09:57 |
| aspiers | what's the path to your libvirt module? | 09:58 |
| aspiers | pip install libvirt doesn't even work | 09:58 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Handle libvirt reporting incorrect 4k page quantities https://review.opendev.org/631038 | 09:59 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Make overcommit check for pinned instance pagesize aware https://review.opendev.org/631053 | 09:59 |
| *** ociuhandu has quit IRC | 09:59 | |
| *** ociuhandu has joined #openstack-nova | 10:00 | |
| kashyap | aspiers: Err, the module is called: 'libvirt-python' | 10:01 |
| kashyap | aspiers: The location of it is: `/home/kashyapc/.virtualenvs/nova/lib/python2.7/site-packages` | 10:01 |
| kashyap | (In my 'tox' env, obv) | 10:01 |
| aspiers | Yeah, you must have installed it yourself | 10:01 |
| aspiers | It's not in requirements.txt because nova doesn't have a hard requirement on it I guess | 10:01 |
| kashyap | Yes, installed it myself; and indeed Nova doesn't have a hard req. on it | 10:02 |
| *** cdent has joined #openstack-nova | 10:02 | |
| aspiers | I guess I need to check out one of your branches? | 10:02 |
| aspiers | git review -d ... ? | 10:03 |
| cdent | aspiers: I've given you days and days to respond to my response that you requested. /me stamps foot :) | 10:04 |
| aspiers | cdent: sorry, can't even remember what that was about - been on holiday :-/ | 10:05 |
| kashyap | aspiers: Not really; I'm just parsing existing domain_caps data | 10:05 |
| cdent | aspiers: :) me too. It was about openstack/opensource. Something in the tc election related threads | 10:05 |
| aspiers | kashyap: AttributeError: 'LibvirtConfigDomainCaps' object has no attribute '_os' | 10:05 |
| aspiers | cdent: rings a vague bell :-o | 10:06 |
| *** ociuhandu has quit IRC | 10:06 | |
| kashyap | aspiers: Bad me, you were right, get this one, please: https://review.opendev.org/#/c/673790/ | 10:06 |
| kashyap | aspiers: Okay ... I see it, was blind, it's due to the damn 'for' loop, for _each_ of the enums (readonly, type, secure), it is printing the value of 'secure' | 10:10 |
| aspiers | yeah | 10:11 |
| * aspiers wonders how much to charge for being a rubber duck | 10:12 | |
| kashyap | Hehe; top it with a drink | 10:12 |
| aspiers | kashyap: you may eventually want to use https://review.opendev.org/#/c/680777/6/tempest/scenario/test_server_sev.py as a basis for a SB tempest test | 10:14 |
| * kashyap clicks | 10:14 | |
| kashyap | aspiers: Ah, thank you | 10:14 |
| kashyap | aspiers: In your "copious free time", can you also please review these "sketch" methods (no tests yet, the first method is broken) for detecting SB: https://review.opendev.org/#/c/682627/1/nova/virt/libvirt/host.py | 10:17 |
| kashyap | aspiers: You'll see that they're "stolen" from how SEV detected (although, it requires check in the kernel support & host) | 10:17 |
| kashyap | (Note to self: maybe move these methods out from host.py --> to the grab-bag-of-all utils.py) | 10:17 |
| kashyap | aspiers: The commit message explains the design, with an example, even (of firmware auto-selection in action) | 10:18 |
| *** brault has quit IRC | 10:36 | |
| *** brault has joined #openstack-nova | 10:39 | |
| *** ociuhandu has joined #openstack-nova | 10:41 | |
| *** brault has quit IRC | 10:44 | |
| *** ociuhandu has quit IRC | 10:46 | |
| *** AdamMork has joined #openstack-nova | 10:46 | |
| *** brault has joined #openstack-nova | 10:49 | |
| *** ociuhandu has joined #openstack-nova | 10:50 | |
| *** brault has quit IRC | 10:53 | |
| *** ociuhandu has quit IRC | 10:54 | |
| *** dpawlik has quit IRC | 10:57 | |
| *** jaosorior has quit IRC | 10:58 | |
| *** zhubx has joined #openstack-nova | 11:01 | |
| openstackgerrit | Merged openstack/os-traits master: Add support for ppc64le platforms https://review.opendev.org/680580 | 11:02 |
| *** dpawlik has joined #openstack-nova | 11:04 | |
| *** boxiang has quit IRC | 11:04 | |
| *** dpawlik has quit IRC | 11:09 | |
| *** dpawlik has joined #openstack-nova | 11:14 | |
| *** ociuhandu has joined #openstack-nova | 11:19 | |
| *** cdent has quit IRC | 11:19 | |
| *** ociuhandu has quit IRC | 11:23 | |
| *** ccamacho has quit IRC | 11:24 | |
| openstackgerrit | Matthew Booth proposed openstack/nova stable/stein: libvirt: Fix service-wide pauses caused by un-proxied libvirt calls https://review.opendev.org/683922 | 11:31 |
| *** sean-k-mooney has joined #openstack-nova | 11:34 | |
| *** cdent has joined #openstack-nova | 11:38 | |
| openstackgerrit | Matthew Booth proposed openstack/nova stable/rocky: libvirt: Fix service-wide pauses caused by un-proxied libvirt calls https://review.opendev.org/683927 | 11:44 |
| *** zul has joined #openstack-nova | 11:47 | |
| *** rcernin has quit IRC | 11:49 | |
| openstackgerrit | Matthew Booth proposed openstack/nova stable/queens: libvirt: Fix service-wide pauses caused by un-proxied libvirt calls https://review.opendev.org/683930 | 11:50 |
| *** ociuhandu has joined #openstack-nova | 11:57 | |
| *** ociuhandu has quit IRC | 11:58 | |
| *** ociuhandu has joined #openstack-nova | 11:59 | |
| *** markvoelker has joined #openstack-nova | 12:03 | |
| *** brault has joined #openstack-nova | 12:20 | |
| *** brault has quit IRC | 12:22 | |
| *** brault has joined #openstack-nova | 12:22 | |
| openstackgerrit | Merged openstack/nova master: Rename 'nova.common.config' module to 'nova.middleware' https://review.opendev.org/676898 | 12:26 |
| *** rcernin has joined #openstack-nova | 12:33 | |
| *** ratailor has quit IRC | 12:35 | |
| *** lbragstad_ is now known as lbragstad | 12:39 | |
| *** psachin has quit IRC | 12:39 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Functional reproduction for bug 1844993 https://review.opendev.org/683947 | 12:45 |
| openstack | bug 1844993 in OpenStack Compute (nova) "migrate a server with qos port with compute RPC pinned to 5.1 fails and leaves the qos port in an inconsistent state" [Undecided,New] https://launchpad.net/bugs/1844993 - Assigned to Balazs Gibizer (balazs-gibizer) | 12:45 |
| *** brault has quit IRC | 12:45 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948 | 12:46 |
| *** nweinber has joined #openstack-nova | 12:53 | |
| *** ricolin has joined #openstack-nova | 12:59 | |
| *** francoisp has joined #openstack-nova | 13:01 | |
| *** larainema has quit IRC | 13:01 | |
| *** mloza has joined #openstack-nova | 13:02 | |
| *** gbarros has joined #openstack-nova | 13:02 | |
| *** efried_pto is now known as efried | 13:03 | |
| *** rouk has joined #openstack-nova | 13:04 | |
| efried | o/ nova | 13:06 |
| gibi | o/ efried | 13:09 |
| *** ociuhandu has quit IRC | 13:10 | |
| *** ociuhandu has joined #openstack-nova | 13:10 | |
| *** jaosorior has joined #openstack-nova | 13:11 | |
| *** Luzi has quit IRC | 13:14 | |
| *** mriedem has joined #openstack-nova | 13:16 | |
| *** ociuhandu has quit IRC | 13:17 | |
| sean-k-mooney | efried: o/ have a good weekend | 13:17 |
| *** mkrai has quit IRC | 13:18 | |
| efried | yeah, not bad, did much laziness. | 13:18 |
| *** artom has joined #openstack-nova | 13:19 | |
| sean-k-mooney | good. stephenfin's cpu patches finally merged on saturday too | 13:19 |
| sean-k-mooney | so thats a thing | 13:19 |
| efried | yeah, I was watching all of that, need to go update blueprints if that hasn't already been done... | 13:19 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Recalculate 'RequestSpec.numa_topology' on resize https://review.opendev.org/662522 | 13:20 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Cleanup of '_test_resize' helper test https://review.opendev.org/664245 | 13:20 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Add '_setup_compute_services' helper https://review.opendev.org/663102 | 13:20 |
| efried | done | 13:20 |
| sean-k-mooney | same, but it ment on sunday i coudl actully relax and not worry about them | 13:21 |
| bauzas | mriedem: I guess we can close https://bugs.launchpad.net/nova/+bug/1427772, right? | 13:21 |
| openstack | Launchpad bug 1427772 in OpenStack Compute (nova) "Instance that uses force-host still needs to run some filters" [Low,Confirmed] | 13:21 |
| *** mkrai has joined #openstack-nova | 13:21 | |
| bauzas | mriedem: because 1/ we no longer accept to force live-migrations | 13:21 |
| bauzas | 2/ we removed the CachingScheduler etc. | 13:21 |
| sean-k-mooney | bauzas: we still accept force for evaucation and maybe resize/cold migrate | 13:22 |
| bauzas | well, good point | 13:23 |
| sean-k-mooney | im not sure if that affect the bug or not | 13:23 |
| stephenfin | bauzas: It can be closed because it's possible to request a specific host without bypassing the scheduler filters | 13:23 |
| *** BjoernT has joined #openstack-nova | 13:23 | |
| sean-k-mooney | well at least the numa related stuff is going to be adress seperatly | 13:24 |
| *** beekneemech is now known as bnemec | 13:24 | |
| stephenfin | bauzas: Specifically, blueprint add-host-and-hypervisor-hostname-flag-to-create-server | 13:24 |
| sean-k-mooney | stephenfin: that wont prevent people form using the old way with --force | 13:25 |
| sean-k-mooney | using the az | 13:25 |
| stephenfin | sean-k-mooney: then the answer is "you're holding it wrong" | 13:25 |
| *** brault has joined #openstack-nova | 13:26 | |
| *** brault has quit IRC | 13:26 | |
| mriedem | osc by default forces live migrations to the specified host | 13:26 |
| sean-k-mooney | i guess we could close it because we have a new feature to replace it | 13:26 |
| sean-k-mooney | yes | 13:26 |
| sean-k-mooney | well | 13:26 |
| sean-k-mooney | you have to spcify a host | 13:26 |
| mriedem | given the age of that bug i'd say screw it, close it | 13:26 |
| sean-k-mooney | i dont know if it forces it | 13:26 |
| stephenfin | we can't remove the old way and the new way was specifically added to avoid this issue, so the answer is surely use the new feature | 13:26 |
| mriedem | sean-k-mooney: yes it does | 13:26 |
| mriedem | osc by default (1) requires you specify a host and (2) defaults to 2.1 | 13:27 |
| mriedem | which by default forces the host | 13:27 |
| sean-k-mooney | :( | 13:27 |
| sean-k-mooney | ok i know 1 but not 2 | 13:27 |
| mriedem | that is deprecated in osc 4.0 with something i added in train | 13:27 |
| *** ociuhandu has joined #openstack-nova | 13:27 | |
| mriedem | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-migrate | 13:27 |
| mriedem | --live <hostname> is deprecated | 13:28 |
| mriedem | but for rhosp customers using queens that's going to be an open issue for a long time | 13:28 |
| sean-k-mooney | unless we backport it | 13:28 |
| bauzas | anyway, nevermind | 13:28 |
| sean-k-mooney | that said i dont know if we can do that | 13:28 |
| stephenfin | we'll backport it, I imagine | 13:28 |
| mriedem | we == red hat | 13:29 |
| * bauzas doing some internal bug triage :( | 13:29 | |
| sean-k-mooney | well we cant backprot the api change downstream | 13:29 |
| mriedem | the api change is in queens, that's not your problem | 13:29 |
| mriedem | it's the client side tooling | 13:29 |
| stephenfin | well you were talking about rhosp, so yes, clearly Red Hat | 13:29 |
| sean-k-mooney | oh ok | 13:29 |
| sean-k-mooney | i guess we should add it to our backlog to consider. the question is can we do it without breaking people | 13:30 |
| * bauzas goes back working on https://review.opendev.org/#/c/670112/ | 13:30 | |
| sean-k-mooney | if we cant we cant backport | 13:30 |
| sean-k-mooney | that said just adding --live-migration shoudl be fine | 13:31 |
| stephenfin | efried, gibi, (dansmith): Interesting issue here https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@1060 | 13:31 |
| stephenfin | efried, gibi, (dansmith): As the comment suggests, that test is working by creating two hosts, one of which doesn't have the second node to which a given physnet is associated https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@855 | 13:32 |
| *** ociuhandu has quit IRC | 13:33 | |
| stephenfin | and we're doing that because it's not possible to have different configuration for different compute "services" in functional tests | 13:33 |
| stephenfin | however, there's a check that prevents us doing just this (configured a physnet for a NUMA node that doesn't exist) that the test simply wasn't triggering | 13:33 |
| stephenfin | because it's only hit as part of the calculation of the host NUMA topology object, which happens in the 'update_available_resource' periodic task, which our test wouldn't normally have time to trigger | 13:34 |
| *** gbarros has quit IRC | 13:35 | |
| efried | not having looked yet, just reacting to ^, we have a way to trigger that periodic | 13:35 |
| efried | but it's also triggered as part of instance creation fwiw | 13:35 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Error out interrupted builds https://review.opendev.org/666857 | 13:36 |
| stephenfin | it seems to be triggered when we query against the placement fixture https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@82 | 13:36 |
| stephenfin | and that's the issue - by adding that query, we end up in a situation when the compute service for that falls over | 13:36 |
| *** BjoernT has quit IRC | 13:36 | |
| *** dpawlik has quit IRC | 13:37 | |
| stephenfin | aaactually, maybe it was falling over already and we just never thought to check for that | 13:37 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pull up compute node queries to init_host https://review.opendev.org/682680 | 13:37 |
| stephenfin | which is the real reason we're getting NoValidHost - not because the compute service can't support the request but because it never actually started | 13:37 |
| * stephenfin deletes that test | 13:38 | |
| sean-k-mooney | stephenfin: you could check that by looking at the comptue service logs | 13:38 |
| *** BjoernT has joined #openstack-nova | 13:39 | |
| stephenfin | that's what I'm doing as we speak | 13:39 |
| stephenfin | :) | 13:39 |
| stephenfin | yup, it never even started \o/ | 13:40 |
| sean-k-mooney | i dont know the specific incantation to do that in the functional test but i have seen mriedem and gibi asserting log messages are emmited in the functional test form different agent in the past | 13:40 |
| * gibi was slow, running that test | 13:40 | |
| mriedem | self.assertIn('whatever', self.stdlog.logger.output) | 13:41 |
| gibi | sean-k-mooney: you can access the logs in from the logger fixture | 13:41 |
| gibi | sean-k-mooney: self.stdlog.logger.output | 13:41 |
| gibi | sean-k-mooney: but it is not per service | 13:41 |
| sean-k-mooney | cool ya its the self.stdlog.logger.output bit that i had seen but not used personally in the past | 13:41 |
| *** dklyle has joined #openstack-nova | 13:44 | |
| *** rcernin has quit IRC | 13:45 | |
| gibi | mriedem: do you think this qualifies as a rc candidate https://bugs.launchpad.net/nova/+bug/1844993 ? | 13:46 |
| openstack | Launchpad bug 1844993 in OpenStack Compute (nova) "migrate a server with qos port with compute RPC pinned to 5.1 fails and leaves the qos port in an inconsistent state" [Undecided,In progress] - Assigned to Balazs Gibizer (balazs-gibizer) | 13:46 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/683961 | 13:47 |
| stephenfin | gibi, sean-k-mooney: ^ (mriedem too, maybe) | 13:47 |
| gibi | stephenfin: looking | 13:47 |
| efried | gibi: was that a regression in train? | 13:47 |
| efried | (the qos one) | 13:48 |
| gibi | efried: not, it is a bug in the bandwith + migration code we merged to Train | 13:48 |
| *** jangutter has quit IRC | 13:48 | |
| *** jangutter_ has joined #openstack-nova | 13:48 | |
| mriedem | gibi: technically no because it was a regression in stein | 13:48 |
| mriedem | gibi: but it would be good to get it fixed in train regardless | 13:48 |
| mriedem | gibi: oh i guess that's a side effect of the other bug right? | 13:49 |
| *** dklyle has quit IRC | 13:50 | |
| *** tbachman has joined #openstack-nova | 13:50 | |
| sean-k-mooney | stephenfin: so the fact we cant confiugre different compute nodes with different config has come up afew times recently | 13:50 |
| *** cdent has quit IRC | 13:50 | |
| sean-k-mooney | we could remove that test but should we file a bug to add support for that in the future | 13:50 |
| gibi | mriedem: your migrate bug is visible if the rpc is pinned to 5.0 | 13:50 |
| gibi | mriedem: the qos migrate bug is visible even if the rpc is pinned to 5.1 | 13:51 |
| sean-k-mooney | stephenfin: i think there has to be a way to use mocking to make the CONF non global | 13:51 |
| sean-k-mooney | and in effect then allow us to have different confics in different agents althoug set_flags likely wont work in that context | 13:51 |
| mriedem | gibi: right so my bug, introduced in stein, is a problem if computes are pinned to rocky (5.0) and yours if computes are pinned to stein (5.1) | 13:52 |
| mriedem | gibi: then yeah it's probably worth tagging for rc1 | 13:52 |
| gibi | mriedem: OK, then we are on the same page | 13:53 |
| *** redrobot has quit IRC | 13:53 | |
| *** jangutter_ has quit IRC | 13:54 | |
| gibi | mriedem: I will respin the fix based on your rpc api method suggestion. I haven't seen the live migration way yet | 13:55 |
| mriedem | it's just a simple abstraction so the caller doesn't need to know the router.client or version internals | 13:56 |
| gibi | stephenfin: theortically you can go and mock things via self.compute['test_compute1'].manager.driver and that will be service and driver selective | 13:57 |
| *** JamesBenson has joined #openstack-nova | 13:59 | |
| *** ociuhandu has joined #openstack-nova | 14:00 | |
| *** JamesBenson has quit IRC | 14:01 | |
| *** JamesBenson has joined #openstack-nova | 14:01 | |
| *** slaweq has joined #openstack-nova | 14:01 | |
| *** ociuhandu has quit IRC | 14:04 | |
| *** Guest30550 has joined #openstack-nova | 14:05 | |
| *** macz has joined #openstack-nova | 14:05 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948 | 14:07 |
| *** brault has joined #openstack-nova | 14:08 | |
| *** brault has quit IRC | 14:08 | |
| *** Guest30550 is now known as redrobot | 14:08 | |
| *** brault has joined #openstack-nova | 14:09 | |
| *** brault has quit IRC | 14:09 | |
| *** BjoernT has quit IRC | 14:09 | |
| *** BjoernT has joined #openstack-nova | 14:13 | |
| mriedem | oh god schedule_and_build_instances is too big, have to write 100 LOC test case just to test one new line of code | 14:25 |
| *** gbarros has joined #openstack-nova | 14:27 | |
| *** dklyle has joined #openstack-nova | 14:31 | |
| *** TxGirlGeek has joined #openstack-nova | 14:35 | |
| *** macz has quit IRC | 14:37 | |
| *** mlavalle has joined #openstack-nova | 14:42 | |
| kashyap | stephenfin: Hey, about? | 14:43 |
| kashyap | stephenfin: When you are -- | 14:43 |
| stephenfin | on a call but yeah | 14:43 |
| kashyap | stephenfin: How much of this is possible _today_ in Nova? https://kashyapc.fedorapeople.org/NUMA-pinning.txt | 14:43 |
| kashyap | stephenfin: Sure, respond when you can | 14:43 |
| stephenfin | 1 and 2 | 14:44 |
| kashyap | stephenfin: Context is a QEMU dev was asking about it. | 14:44 |
| stephenfin | we don't provide a way to configure IOThreads | 14:44 |
| kashyap | Yeah, I see from the code that we don't do anything for IOThreads | 14:45 |
| efried | mriedem: what would it take to split tempest-slow-py3 into two parts? | 14:45 |
| stephenfin | and I don't know what a PXB device is so let's assume not for that too | 14:45 |
| *** JamesBenson has quit IRC | 14:46 | |
| kashyap | stephenfin: For that, `grep` for 'pci-expander-bus' here: https://libvirt.org/formatdomain.html | 14:46 |
| mriedem | efried: why? | 14:46 |
| efried | mriedem: That would make a run take ~1.5h rather than ~2.5h | 14:47 |
| efried | that last hour is always just waiting for tempest-slow-py3 to finish. | 14:47 |
| openstackgerrit | Merged openstack/python-novaclient master: Update master for stable/train https://review.opendev.org/683627 | 14:47 |
| kashyap | stephenfin: So the suggested IOThreads 'formula' for management tools is, _assuming_ we know the required info here: | 14:47 |
| mriedem | efried: what do you mean the last hour is waiting for the job to finish? | 14:47 |
| mriedem | for tempest to finish? | 14:47 |
| mriedem | or to collect logs? | 14:47 |
| kashyap | stephenfin: num_iothreads = min(num_devices, num_vcpus, num_host_cpus) | 14:48 |
| kashyap | Thanks for the answer so far | 14:48 |
| efried | mriedem: I mean that for a given patch, all the other jobs take <=1.5h to complete, but tempest-slow-py3 takes ~2.5h. Trying to reduce the wait time for a single patch to make it through CI. | 14:49 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Sanity check instance mapping during scheduling https://review.opendev.org/683730 | 14:50 |
| mriedem | efried: well, there are a few options. one is to go the way of tempest-integrated-compute and make a compute-specific job that only runs compute tests we care about (sort of like nova-next) but only runs slow tests (which tempest-integrated-compute does not), | 14:50 |
| mriedem | the other major difference is tempest-slow* is a multinode job where nova-next and tempest-integrated-compute are single node | 14:51 |
| *** ociuhandu has joined #openstack-nova | 14:51 | |
| mriedem | the multinode + slow combo is important to run some tests that we don't otherwise run | 14:51 |
| mriedem | one idea is just drop tempest-slow-py3 from nova's job list and make nova-next multi-node | 14:51 |
| mriedem | nova-next runs compute api and scenario tests, including slow | 14:51 |
| sean-k-mooney | one factor for multinode jobs is it only starts stacking the second node after the first completes | 14:52 |
| *** cdent has joined #openstack-nova | 14:52 | |
| efried | I'm sure I'm thinking of it way too simplistically, but what I was thinking was just, like, if tempest-slow-py3 currently runs 10 tests, make a tempest-slow-py3-1 that runs the first 5 and a tempest-slow-py3-2 that runs the second 5. | 14:52 |
| efried | i.e. one additional job, but since it runs in parallel, reduces the overall run time in half. | 14:53 |
| mriedem | efried: that would be a pain in the ass to manage from a tempest perspective i'd think | 14:53 |
| mriedem | i.e. managing what gets run per job | 14:53 |
| sean-k-mooney | efried: most of the time is spend on devstack | 14:53 |
| sean-k-mooney | efried: so you would not reduce the time by half | 14:53 |
| efried | mmph | 14:53 |
| mriedem | also note that the scenario tests are run in serial | 14:53 |
| sean-k-mooney | well that might not be quite true the slow test are slow | 14:53 |
| mriedem | which i tried to do something about but failed | 14:53 |
| sean-k-mooney | but devstack is still a signifcant portion | 14:54 |
| efried | "tempest_concurrency":2 | 14:54 |
| mriedem | https://review.opendev.org/#/c/650300/ | 14:54 |
| mriedem | no, | 14:54 |
| sean-k-mooney | efried: we need to run them serially | 14:54 |
| mriedem | because you want the api tests running at default concurrency, which is i think nproc/2 | 14:54 |
| sean-k-mooney | they can fail due to resouce constratins if we dont | 14:54 |
| mriedem | sean-k-mooney: we don't *need* to | 14:54 |
| efried | So the above line in the job def is... not doing anything? | 14:54 |
| efried | http://zuul.openstack.org/job/tempest-slow | 14:55 |
| sean-k-mooney | mriedem: we used to get intermitent failture when we didnt right? | 14:55 |
| mriedem | oh geez i didn't realize tempest-slow ran *everything* with only 2 workers | 14:55 |
| mriedem | sean-k-mooney: yes but that was more about ssh issues | 14:55 |
| mriedem | which might have been resolved by now | 14:55 |
| mriedem | because in the long ago, only the scenario tests ran ssh | 14:56 |
| sean-k-mooney | mriedem: oh ok | 14:56 |
| mriedem | but tempest-full has been running with ssh integration on for years now | 14:56 |
| mriedem | efried: so yeah that is definitely one reason that tempest-slow is slower, it's constrained to 2 workers | 14:56 |
| sean-k-mooney | the scenario tests still use ssh | 14:57 |
| mriedem | sean-k-mooney: yes i know | 14:57 |
| sean-k-mooney | but unfrotuetly some api tests also do... | 14:57 |
| mriedem | sean-k-mooney: i'm pretty sure ^ is intentional by the QA team | 14:57 |
| *** ociuhandu has quit IRC | 14:57 | |
| *** awalende has quit IRC | 14:58 | |
| sean-k-mooney | doing it via the connectivyt check is fine. but the api test were ment to work with any hyperviors and should work with the fake driver | 14:58 |
| sean-k-mooney | but i know that distinction has kind of faded over the years | 14:58 |
| mriedem | nova-next runs with full concurrency, api and scenario, and we don't have issues from that | 15:00 |
| mriedem | https://zuul.opendev.org/t/openstack/build/21573b9826664ec8a456f4e3007a91c4/log/job-output.txt#31633 | 15:00 |
| sean-k-mooney | dont we run the scenario tests with serially as a seperate step | 15:00 |
| mriedem | Ran: 579 tests in 2771.6131 sec. | 15:01 |
| sean-k-mooney | we do that in one of the jobs | 15:01 |
| mriedem | sean-k-mooney: that's what the full env does | 15:01 |
| mriedem | tempest-integrated-compute | 15:01 |
| mriedem | https://github.com/openstack/tempest/blob/master/tox.ini#L106 | 15:01 |
| efried | mriedem: if we made nova-next multinode, we could drop tempest-slow-py3, which would reduce the number of CI nodes consumed by one... but would likely inflate nova-next to 2.5h or more, wouldn't it? | 15:02 |
| sean-k-mooney | ok ya | 15:02 |
| sean-k-mooney | that is why we dont have issue with concurance 4 | 15:02 |
| mriedem | efried: nova-next would still be faster than tempest-slow i think b/c we'd be avoiding non-compute tests, | 15:02 |
| mriedem | but it's hard to say without just proposing | 15:02 |
| mriedem | efried: and we'd be running nova-next with 4 test workers rather than 2 | 15:02 |
| sean-k-mooney | i did a multinoe tempest full fun for one of the cpu pinning jobs | 15:03 |
| sean-k-mooney | it was just aboud 2 hours | 15:03 |
| mriedem | tempest-integrated-compute SUCCESS in 1h 38m 29s | 15:04 |
| mriedem | i'd say if we can stay within a reasonable comparison time-wise to tempest-integrated-compute it's a win | 15:04 |
| *** ociuhandu has joined #openstack-nova | 15:04 | |
| mriedem | so make nova-next multinode and drop tempest-slow-py3 from nova runs | 15:04 |
| *** ociuhandu has quit IRC | 15:04 | |
| sean-k-mooney | oh sorry tempest-full with concurrance:1 is just about 2 hours | 15:05 |
| *** ociuhandu has joined #openstack-nova | 15:05 | |
| *** pcaruana has quit IRC | 15:05 | |
| sean-k-mooney | so with concurrance:4 it should be closer to 1.5h as you suggested | 15:06 |
| *** eharney has quit IRC | 15:06 | |
| *** ociuhandu has quit IRC | 15:08 | |
| mriedem | stephenfin: was there any reason why you didn't remove this cellsv1 mention here? https://github.com/openstack/nova/blob/5a1c2d4ffa0815e874f373a87eb38b1833d03b24/nova/conductor/manager.py#L567 | 15:08 |
| *** ociuhandu has joined #openstack-nova | 15:08 | |
| *** udesale has joined #openstack-nova | 15:09 | |
| efried | sean-k-mooney: do you understand mriedem's suggestion enough to propose... whatever change(s) are necessary? | 15:09 |
| stephenfin | mriedem: I started doing it but got stuck because of highly coupled tests https://review.opendev.org/#/c/651316/2/nova/conductor/manager.py I'd prefer to leave it until that patch is finished | 15:10 |
| stephenfin | Or at least put in a TODO to remove all the cells v1 stuff | 15:10 |
| mriedem | stephenfin: that patch is wrong anyway | 15:11 |
| mriedem | you can't just drop compat code w/o a major rpc version bump | 15:11 |
| *** cfriesen has joined #openstack-nova | 15:12 | |
| *** ociuhandu has quit IRC | 15:13 | |
| stephenfin | Yeah, I don't get why we do that. I get that we can't change the signature but can't we start crapping out if a too-old client calls us? | 15:13 |
| stephenfin | i.e. 'if request_spec if None: raise Exception('too old')' | 15:14 |
| mriedem | i'll let dansmith answer that one | 15:14 |
| cdent | if cdent: raise Exception('too old') | 15:14 |
| *** TxGirlGeek has quit IRC | 15:14 | |
| mriedem | i ask b/c i want to drop ocata-era error handling in _populate_instance_mapping which would only be hit in the cells v1 case, which is no longer possible | 15:15 |
| dansmith | stephenfin: mriedem: Of course we *can* but we're breaking the contract/rules of that 5.x means vs 6.x. So yeah, we could go all wild-west and just deprecate minor versions by raising random exceptions in the receiving code, | 15:15 |
| dansmith | but then it becomes hard to reason about when we can remove things and what the impacts will be | 15:15 |
| dansmith | the version numbers and rules are there to make it easier for the humans to know what falls into what bucket, IMHO | 15:16 |
| mriedem | i know, i just pinged you since you're better with the words on this | 15:17 |
| mriedem | "we can but we shouldn't" | 15:17 |
| dansmith | okay | 15:17 |
| mriedem | bad habits etc | 15:17 |
| stephenfin | Would bumping the major version each release be too expensive? | 15:17 |
| mriedem | conductor doesn't need to change that often | 15:18 |
| dansmith | it used to be done almost every release | 15:18 |
| dansmith | but we don't change that much anymore, | 15:18 |
| mriedem | but we are definitely due for a conductor comptue task api 2.0 | 15:18 |
| dansmith | so yeah, I think it's too expensive | 15:18 |
| mriedem | b/c there is a lot of old shit in here | 15:18 |
| * dansmith nods | 15:18 | |
| stephenfin | maybe that's the solution so | 15:18 |
| mriedem | once we go to 2.0 that _populate_instance_mapping just gets dropped | 15:18 |
| stephenfin | I can't remove that stuff yet but I can go to 2.0 and then drop that stuff | 15:19 |
| openstackgerrit | Merged openstack/os-resource-classes master: Update the constraints url https://review.opendev.org/683872 | 15:19 |
| stephenfin | mriedem: While I have you - can we remove the os-networks entirely when we drop nova-net? | 15:19 |
| *** ociuhandu has joined #openstack-nova | 15:19 | |
| *** lpetrut has quit IRC | 15:20 | |
| mriedem | i had an etherpad with notes about the more complicated apis that involved networks b/c they aren't all nova-net only anymore | 15:20 |
| stephenfin | It seems some of the APIs work with neutron but most don't, and I'm trying to decide if we should selectively 404 them or 404 everything | 15:20 |
| mriedem | that's the issue | 15:20 |
| mriedem | if there are apis that work with neutron we can't just 410 those | 15:20 |
| stephenfin | 410, sorry | 15:20 |
| stephenfin | yeah | 15:20 |
| *** ociuhandu has quit IRC | 15:21 | |
| stephenfin | I was afraid you'd say that :( Time to rework again | 15:22 |
| stephenfin | Any idea where that old etherpad is? | 15:22 |
| mriedem | btw this is the last time we did an rpc api major version bump https://review.opendev.org/#/c/541005/ | 15:22 |
| *** ociuhandu has joined #openstack-nova | 15:22 | |
| mriedem | https://etherpad.openstack.org/p/nova-network-removal-rocky | 15:22 |
| *** KeithMnemonic has joined #openstack-nova | 15:22 | |
| stephenfin | I've a rough idea from converting all the API sample functional tests over but maybe I've missed some stuff | 15:22 |
| *** mrch_ has quit IRC | 15:24 | |
| stephenfin | gibi: Just in case, you haven't tried adding floating IP stuff to NeutronFixture or some subclass, have you? | 15:25 |
| gibi | stephenfin: let me check some notification tests | 15:25 |
| *** dtantsur is now known as dtantsur|afk | 15:26 | |
| * stephenfin is trying to convert 'nova/tests/functional/api_sample_tests/test_floating_ips.py' and getting complaints about things like 'create_floatingip' not being defined | 15:26 | |
| stephenfin | gibi: It would be local if so. We don't have it in tree | 15:26 |
| stephenfin | at least searching for 'create_floatingip' doesn't turn up anything but mocks/the actual call | 15:26 |
| gibi | stephenfin: unfortunately no, the IpPayload only contains fixed ips | 15:28 |
| stephenfin | damn | 15:28 |
| stephenfin | thanks for checking | 15:28 |
| gibi | np | 15:28 |
| *** dklyle has quit IRC | 15:36 | |
| *** dklyle has joined #openstack-nova | 15:36 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988 | 15:37 |
| mriedem | efried: i think this is what you're looking for ^ | 15:37 |
| * efried watches | 15:37 | |
| efried | thanks mriedem | 15:37 |
| *** damien_r has quit IRC | 15:46 | |
| dansmith | mriedem: so on your remap sanity check patch, | 15:48 |
| dansmith | mriedem: it looks to me like you're now letting whichever one comes last win, right? | 15:48 |
| dansmith | mriedem: really what we want is to let the non-cell0 one win, if possible | 15:49 |
| mriedem | dansmith: in the bury in cell0 case, if it's already mapped we don't bury it in cell0, | 15:49 |
| mriedem | in the non-cell0 case, we log an error and map it to the cell we just scheduled it to | 15:50 |
| mriedem | which would not be cell0 | 15:50 |
| dansmith | mriedem: oh I see, you created a helper but only call it in the non-cell0 case | 15:50 |
| mriedem | i think those are both letting the non-cell0 ones win | 15:50 |
| mriedem | yes | 15:50 |
| dansmith | I assumed you were calling it from both places | 15:50 |
| dansmith | okay | 15:50 |
| mriedem | b/c i didn't want to have to mock 100 LOC just to test 1 line change in schedule_and_build_instances | 15:50 |
| dansmith | sounds like cheating | 15:50 |
| mriedem | schedule_and_build_instances is a monster | 15:51 |
| *** rpittau is now known as rpittau|afk | 15:52 | |
| *** TxGirlGeek has joined #openstack-nova | 15:52 | |
| mriedem | i also noted that if we drop that InstanceMappingNotFound pre-ocata compat in _bury_in_cell0 the _bury_in_cell0 method can just call the new _map_instance_to_cell method | 15:52 |
| mriedem | we have a bunch of pre-ocata pre-cells v2 compat handlers all over the conductor task manager code which it'd be nice to remove | 15:52 |
| mriedem | since they shouldn't be possible anymore | 15:52 |
| dansmith | meh, I love piles of compat code | 15:53 |
| dansmith | makes me feel nostalgic | 15:54 |
| mriedem | all of the TODO(alaski)s to feel like an old sweater | 15:54 |
| mriedem | i'll admit | 15:54 |
| mriedem | s/to/do/ | 15:54 |
| *** ociuhandu has quit IRC | 15:56 | |
| *** ociuhandu has joined #openstack-nova | 15:57 | |
| dansmith | lol | 15:57 |
| *** ociuhandu has quit IRC | 16:01 | |
| *** ociuhandu has joined #openstack-nova | 16:02 | |
| stephenfin | I'm almost certain we've discussed this before, but why do we drop support for old microversion in novaclient? | 16:03 |
| stephenfin | Referring to the API part, rather than the CLI | 16:03 |
| stephenfin | Doesn't osc use that? | 16:03 |
| *** pcaruana has joined #openstack-nova | 16:06 | |
| *** ociuhandu has quit IRC | 16:06 | |
| mriedem | gibi: a few comments in https://review.opendev.org/#/c/683947/ but not worth holding it up, | 16:07 |
| mriedem | gibi: but we might want to reconsider the fault message that the user could see | 16:07 |
| mriedem | "oh hi your cloud provider is doing an upgrade and you shouldn't be trying to resize your instance right now bob!" | 16:08 |
| mriedem | stephenfin: where have we dropped support for old microversions in novaclient? | 16:08 |
| gibi | mriedem: the funny thing is that I think we cannot hit that error after the fix | 16:08 |
| gibi | mriedem: so we could even remove it | 16:09 |
| stephenfin | mriedem: e.g. 01fb16533bf562f39fe822bc12b9cc34b8580359 | 16:09 |
| mriedem | gibi: well, technically your fix is in conductor checking it's config but the computes involved in the resize/cold migrate could have different config with different pins | 16:09 |
| mriedem | stephenfin: that one broke osc | 16:09 |
| gibi | mriedem: can we pin our computes differently? how will they talk to each other? | 16:09 |
| mriedem | and they had to fix to avoid using novaclient and hit the api directly | 16:10 |
| *** ociuhandu has joined #openstack-nova | 16:10 | |
| mriedem | gibi: well i meant if your conductor was unpinned but the computes were, or not restarted yet or something | 16:10 |
| mriedem | after unpinning them | 16:10 |
| mriedem | i'm nto saying that's normal | 16:10 |
| *** slaweq has quit IRC | 16:10 | |
| mriedem | but i'd probably leave the error checking in place until we have a compute rpc api major version bump to 6.0 | 16:10 |
| gibi | mriedem: OK. I so it is a sort transinet. I can accept that | 16:10 |
| stephenfin | Yup, I recall that coming up now. Any reason we opted to do that instead of reverting the changes to 'novaclient/v2/client.py'? | 16:10 |
| stephenfin | that = call the API directly | 16:11 |
| mriedem | gibi: so if you wanted, you could write a separate patch which (1) logs the upgrade stuff but doesn't include it in the PortUpdateFailed message, (2) adds a TODO to that PortUpdateFailed code to say we can drop it after compute RPC API is bumped to 6.0 | 16:11 |
| gibi | mriedem: sure. I can do that | 16:11 |
| mriedem | stephenfin: it was released late in novaclient, end of the milestone before FF and i think osc found the problem too late | 16:11 |
| mriedem | stephenfin: so a mixture of releasing breaking / major version things in novaclient too late in the release to catch them and (2) osc historically not doing functional testing | 16:12 |
| mriedem | well, not doing good enough coverage with functional testing | 16:12 |
| *** mkrai has quit IRC | 16:12 | |
| mriedem | and (3) me not realizing osc was going to be broken | 16:13 |
| stephenfin | that makes sense | 16:14 |
| mriedem | that was probably before i cared more about osc and just figured, "this has been deprecated in novaclient since newton, we can surely drop it and no one will care" | 16:14 |
| mriedem | so having been burned before, and burning myself and others, that's why i'm less cavalier about just removing old stuff | 16:14 |
| stephenfin | Yeah, certainly true for something like this | 16:15 |
| mriedem | speaking of an upgrade-related burning sensation https://review.opendev.org/#/q/topic:bug/1843090+(status:open+OR+status:merged) | 16:15 |
| stephenfin | I was holding off on that til some people had looked at the oslo.messaging bug | 16:16 |
| stephenfin | though maybe that doesn't make sense | 16:16 |
| mriedem | you mean gibi's RequestSpecImageSerializationFixture in the functoinal test? | 16:17 |
| mriedem | it's an existing pattern / known issue | 16:17 |
| mriedem | could also summon dansmith to review those since it's rpc pin related | 16:18 |
| mriedem | and that's dan's middle name | 16:18 |
| mriedem | dan rpc-pin-and-sometimes-evacuate smith | 16:18 |
| stephenfin | If dansmith doesn't get to it by tomorrow, I will | 16:19 |
| *** mrch_ has joined #openstack-nova | 16:19 | |
| mriedem | gibi: i think we could drop the new and redundant unit test in https://review.opendev.org/#/c/683948/ | 16:29 |
| mriedem | the functional test covers it | 16:29 |
| *** BjoernT has quit IRC | 16:31 | |
| *** dtruong has joined #openstack-nova | 16:41 | |
| *** pcaruana has quit IRC | 16:46 | |
| *** BjoernT has joined #openstack-nova | 16:53 | |
| *** AdamMork has quit IRC | 16:57 | |
| *** udesale has quit IRC | 17:00 | |
| *** nweinber_ has joined #openstack-nova | 17:02 | |
| *** BjoernT has quit IRC | 17:04 | |
| *** nweinber has quit IRC | 17:04 | |
| *** BjoernT has joined #openstack-nova | 17:19 | |
| *** jmlowe has quit IRC | 17:19 | |
| openstackgerrit | Merged openstack/nova master: Add note about needing noVNC >= v1.1.0 with using ESX https://review.opendev.org/682946 | 17:24 |
| *** derekh has quit IRC | 17:24 | |
| *** ociuhandu has quit IRC | 17:27 | |
| *** lpetrut has joined #openstack-nova | 17:28 | |
| *** ociuhandu has joined #openstack-nova | 17:28 | |
| *** ociuhandu has quit IRC | 17:28 | |
| *** ociuhandu has joined #openstack-nova | 17:28 | |
| *** BjoernT has quit IRC | 17:30 | |
| bbobrov_ | hi! In https://opendev.org/openstack/nova/src/branch/master/nova/servicegroup/drivers/db.py#L99 messaging.MessagingTimeout is getting caught. My understanding is that service_ref.save() only interacts with the database. How can MessagingTimeout happen? | 17:30 |
| *** bbobrov_ is now known as bbobrov | 17:30 | |
| *** ralonsoh has quit IRC | 17:35 | |
| sean-k-mooney | bbobrov: if this is executing on the compute node then the save call would invoke the db update via rpc as the comptue services do not have database access | 17:38 |
| *** BjoernT has joined #openstack-nova | 17:39 | |
| sean-k-mooney | bbobrov: so on the concoctor,api and schduler this shoudl not raise a messaging timeout but it can in the compute agents | 17:41 |
| bbobrov | sean-k-mooney: understood, thanks | 17:41 |
| *** BjoernT_ has joined #openstack-nova | 17:43 | |
| *** jmlowe has joined #openstack-nova | 17:44 | |
| *** BjoernT has quit IRC | 17:45 | |
| *** davee_ has joined #openstack-nova | 17:45 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Rename Claims resources to compute_node https://review.opendev.org/679470 | 17:45 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add a prelude for the Train release https://review.opendev.org/683327 | 17:50 |
| *** rouk has quit IRC | 17:50 | |
| *** lpetrut has quit IRC | 17:52 | |
| bbobrov | #2 then. there is a `periodic_enable` option, and in nova codebase it seems to affect only https://opendev.org/openstack/nova/src/branch/master/nova/service.py#L205 . if true, self.periodic_tasks is scheduled to be run by add_dynamic_timer | 17:53 |
| bbobrov | But there is no implementation of periodic_tasks or run_periodic_tasks anywhere in Nova. The method in the base scheduler class does `pass`. Are there any implementation of run_periodic_tasks in Nova that i miss or they can come from somewhere else? | 17:54 |
| sean-k-mooney | we use the one form oslo service https://github.com/openstack/oslo.service/blob/master/oslo_service/periodic_task.py | 17:56 |
| *** yaawang has quit IRC | 17:57 | |
| *** yedongcan has quit IRC | 17:58 | |
| *** yaawang has joined #openstack-nova | 17:59 | |
| *** ricolin has quit IRC | 18:02 | |
| *** davee_ has quit IRC | 18:03 | |
| *** igordc has joined #openstack-nova | 18:06 | |
| *** luksky has joined #openstack-nova | 18:06 | |
| mriedem | melwitt: do you know if tripleo has any kind of tooling that pokes a cell mq/db on upgrade to make sure it's ok before saying everything is good to go in the api? | 18:07 |
| *** davee_ has joined #openstack-nova | 18:11 | |
| *** cdent has quit IRC | 18:14 | |
| *** ociuhandu has quit IRC | 18:19 | |
| bbobrov | sean-k-mooney: thanks, it makes sense now | 18:22 |
| *** jdillaman has joined #openstack-nova | 18:31 | |
| *** martinkennelly has quit IRC | 18:32 | |
| lyarwood | mriedem: we are only just introducing full support for multi cell deployments now in 16 but I can ask around about that in the morning. | 18:32 |
| lyarwood | owalsh: ^ unless you're around and know? | 18:33 |
| mriedem | lyarwood: yeah in grenade upstream jobs we don't have multiple cells, just cell0 and cell1 | 18:34 |
| mriedem | trying to debug https://bugs.launchpad.net/nova/+bug/1844929 | 18:34 |
| openstack | Launchpad bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] | 18:34 |
| mriedem | it looks like we upgrade code to train, restart scheduler, it starts up, even hits the cell1 database to pull compute nodes and instances for it's in-memory cache, and then on the first scheduling attempt after that we hang | 18:35 |
| dansmith | mriedem: scheduler doesn't need rabbit to do that, so the rabbit errors in your bug report would be unrelated I'd think | 18:37 |
| mriedem | true, didn't think of that | 18:39 |
| mriedem | obviously | 18:39 |
| mriedem | i just left a comment with a bunch of log links, | 18:39 |
| mriedem | but it looks like we start up ok, hit the cell1 db to pull compute nodes and instances, then 4 minutes later is the first scheduling attempt after the upgrade and at that point we're timing out | 18:40 |
| sean-k-mooney | dansmith: we do eventurlly need to adress the rabbit mq heartbeat issue even if it is unrelated to this issue | 18:41 |
| mriedem | it also looks like this goes back to 9/14 which is a few days before i originally thought | 18:41 |
| dansmith | mriedem: that timeout is related to the scatter | 18:41 |
| dansmith | mriedem: that's not new information, I'm just saying | 18:42 |
| dansmith | mriedem: I was expecting to see a trace, but I think the scatter squashes that | 18:42 |
| sean-k-mooney | i belive if any of the request raise an excpeiton we catch it and return the exception instead of raising it or something like that | 18:43 |
| mriedem | it just logs a warning without a trace | 18:43 |
| dansmith | mriedem: right | 18:43 |
| sean-k-mooney | so i dont think we actuly log the traceback the way we normally would | 18:43 |
| dansmith | sean-k-mooney: didn't I just say that? | 18:43 |
| *** slaweq has joined #openstack-nova | 18:44 | |
| mriedem | what i'm concerned about is if something in https://review.opendev.org/#/c/641907/ which merged and was released late is causing some side effect with the restart | 18:44 |
| sean-k-mooney | yes i was halfway through typing it when you did so i finished it and hit enter | 18:44 |
| dansmith | mriedem: you said it was able to prime its cache on startup right? so this is it getting all nodes from all cells on the schedule run? | 18:44 |
| mriedem | we're getting through this https://github.com/openstack/nova/blob/597b34cd87ac349c0f3702a872630f3c830b1483/nova/scheduler/host_manager.py#L413 | 18:45 |
| mriedem | and then 4 minutes later the first scheduling request comes, | 18:45 |
| mriedem | and it's going back to the cell to pull compute nodes by uuid | 18:45 |
| mriedem | and times out | 18:45 |
| dansmith | mriedem: I guess I thought we were stopping with systemd in that case, so no sighup involed yeah? | 18:45 |
| sean-k-mooney | https://github.com/openstack/grenade/blob/master/projects/60_nova/shutdown.sh#L21 | 18:47 |
| sean-k-mooney | i have not checked devstack but yes | 18:47 |
| sean-k-mooney | i think tha tdoes systemctl stop devstack@n=* effectivly | 18:48 |
| dansmith | right | 18:48 |
| sean-k-mooney | ya it does https://github.com/openstack/devstack/blob/master/lib/nova#L1047-L1059 | 18:49 |
| mriedem | Sep 22 00:37:32.126839 ubuntu-bionic-ovh-gra1-0011664420 systemd[1]: Stopped Devstack devstack@n-sch.service. | 18:49 |
| mriedem | Sep 22 00:45:55.359862 ubuntu-bionic-ovh-gra1-0011664420 systemd[1]: Started Devstack devstack@n-sch.service. | 18:49 |
| dansmith | mriedem: do we even stop/start mysql in grenade during the upgrade? | 18:50 |
| mriedem | Sep 22 00:37:27.786606 ubuntu-bionic-ovh-gra1-0011664420 nova-scheduler[25563]: INFO oslo_service.service [None req-91e88f0d-9b5c-4cb7-a5e9-e7309f922832 None None] Caught SIGTERM, stopping children | 18:50 |
| mriedem | yeah not a HUP | 18:50 |
| mriedem | dansmith: pretty sure we don't | 18:50 |
| dansmith | mriedem: yeah, so, pretty weird that it connects and works once and then times out later, because it's not like we restart mysql after that point or something | 18:51 |
| dansmith | mriedem: so I wonder if we're actually doing something other than what we think | 18:51 |
| *** igordc has quit IRC | 18:51 | |
| dansmith | and since we don't log a trace there we don't know where it's actually timing out | 18:51 |
| openstackgerrit | Dustin Cowles proposed openstack/nova-specs master: Spec: Provider config YAML file https://review.opendev.org/680471 | 18:52 |
| *** brault has joined #openstack-nova | 18:53 | |
| *** ircuser-1 has joined #openstack-nova | 18:54 | |
| *** jdillaman has quit IRC | 18:55 | |
| *** tbachman has quit IRC | 18:57 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Log CellTimeout traceback in scatter_gather_cells https://review.opendev.org/684118 | 18:59 |
| mriedem | dansmith: thinking like this? ^ | 18:59 |
| dansmith | mriedem: yeah, I mean we kinda specifically decided not to explode there so we're tolerant of transient failures | 19:00 |
| dansmith | I wonder if we should log debug with exc_info=True and warning without or something | 19:00 |
| dansmith | but yeah, something | 19:00 |
| *** tbachman has joined #openstack-nova | 19:00 | |
| mriedem | was thinking about passing down a kwarg to scatter_gather_cells to tell it what to do, but that's probably icky | 19:01 |
| mriedem | trace_on_timeout = kwargs.pop('trace_on_timeout', False) | 19:01 |
| dansmith | yeah, I don't like that | 19:01 |
| dansmith | I love **kwargs behavior in python, but hate kwargs.pop('arg', None) abuses of it | 19:02 |
| dansmith | which is why we can't have nice things in other languages :) | 19:02 |
| *** nweinber_ has quit IRC | 19:02 | |
| *** nweinber has joined #openstack-nova | 19:03 | |
| *** tbachman has quit IRC | 19:03 | |
| dansmith | mriedem: so you just need to recheck that a few times to get a repro on it yeah? | 19:05 |
| mriedem | maybe if we get lucky http://status.openstack.org/elastic-recheck/#1844929 | 19:06 |
| mriedem | it mostly only shows up on certain node providers | 19:06 |
| mriedem | so if i hit a rax node i likely won't see it | 19:07 |
| mriedem | and only grenade jobs for some reason | 19:07 |
| mriedem | something getting f'ed up in restarts | 19:07 |
| sean-k-mooney | mriedem: which node providers? | 19:07 |
| mriedem | fort nebula and ovh | 19:07 |
| dansmith | mriedem: you should put that in the bug | 19:07 |
| sean-k-mooney | we can target fort nebula using a specific node pool lable | 19:07 |
| dansmith | so people can know and don't have to ask | 19:08 |
| sean-k-mooney | im not sure about ovh | 19:08 |
| mriedem | dansmith: from the bug, "It also appears to only show up on fortnebula and OVH nodes, primarily fortnebula." | 19:08 |
| sean-k-mooney | but we coudl try and repoducice it via FN if you thing that was useful | 19:08 |
| *** zhubx has quit IRC | 19:08 | |
| dansmith | mriedem: I know :) | 19:08 |
| mriedem | i see what you did there | 19:08 |
| *** zhubx has joined #openstack-nova | 19:08 | |
| dansmith | haha | 19:09 |
| *** macz has joined #openstack-nova | 19:09 | |
| hemna | so I still can't seem to run tox -epy27 on anything less than stable/rocky | 19:11 |
| hemna | queens and pike both fail | 19:11 |
| *** slaweq has quit IRC | 19:11 | |
| hemna | http://paste.openstack.org/show/778953/ | 19:11 |
| hemna | 6444 failures, all the same type of failure. sqlalchemy.exc.NoSuchTableError: migration_tmp | 19:11 |
| dansmith | hemna: clean your local directory | 19:12 |
| hemna | I did | 19:12 |
| *** READ10 has joined #openstack-nova | 19:12 | |
| donnyd | Is this job CPU bound? | 19:12 |
| hemna | nuked all pyc files and .tox | 19:12 |
| dansmith | er, well, maybe not with that _tmp prefix | 19:12 |
| dansmith | hemna: yeah, normally that comes from stale migration pycs but I think this is something else | 19:12 |
| mriedem | looking a bit further between the n-sch restart and the timeout, verify_instance works | 19:14 |
| mriedem | 2019-09-22 00:47:29.331 | Instance 44d7efdc-c048-4dca-8b4b-3d518321eddd is in cell: cell1 (8acfb79b-2e40-4e1c-bc3d-d404dac6db90) | 19:14 |
| *** tbachman has joined #openstack-nova | 19:14 | |
| mriedem | donnyd: i think any job running nova and neutron is CPU bound :) | 19:14 |
| donnyd | Yea that is not a strength of FN | 19:15 |
| *** macz has quit IRC | 19:15 | |
| *** READ10 has quit IRC | 19:16 | |
| *** macz has joined #openstack-nova | 19:16 | |
| sean-k-mooney | mriedem: i assume you have seen the https://c0c3548b65f303ef6c0e-9dc5526a72bde5cc52e2c616e6a483fd.ssl.cf5.rackcdn.com/682061/5/check/grenade-heat/79209ac/logs/screen-n-cond.txt.gz i assume those are a result of the timeout | 19:16 |
| *** mgoddard has quit IRC | 19:17 | |
| mriedem | what, NoValidHost? yes that's the user visible failure | 19:17 |
| sean-k-mooney | ya | 19:17 |
| mriedem | the scheduler asks placement for providers for a build request, it gives back 1, we ask the cell db for the compute node for that one by uuid, timeout and then run an empty list of hosts through the filters | 19:17 |
| sean-k-mooney | donnyd: well for jobs that actully enable nested vert/kvm FN is proably faster then the qemu based jobs | 19:18 |
| openstackgerrit | Eric Fried proposed openstack/nova master: doc: attaching virtual persistent memory to guests https://review.opendev.org/680300 | 19:19 |
| sean-k-mooney | mriedem: ya i notice the timestamp in the conductor log was after the timeout. | 19:19 |
| donnyd | sean-k-mooney: for that case.. and anything with heavy IO should fly in FN - I am curious if the jobs that are failing on FN are related in any way to ipv6 | 19:19 |
| *** mgoddard has joined #openstack-nova | 19:20 | |
| dansmith | donnyd: could be, as we might be initiating a connection when this fails, | 19:20 |
| dansmith | which might be network-related | 19:20 |
| mriedem | i didn't think grenade jobs were doing anything with ipv6 | 19:20 |
| sean-k-mooney | donnyd: where you thinking about the reduced mtu | 19:21 |
| dansmith | well, maybe not, but just saying, if something network-y is the difference, that could explain flakiness | 19:21 |
| donnyd | I was thinking more about the other one where the node loses contact when the ipv6 public side network is created | 19:22 |
| donnyd | I recently tried to help that out by lowering my RA's so when it does it will pick it back up in time | 19:22 |
| sean-k-mooney | the vms are dual stack vms right. they get a public ipv6 and private ipv4 address | 19:23 |
| donnyd | correct sean-k-mooney | 19:23 |
| sean-k-mooney | in that case ill check what ip we actully use for devstack | 19:23 |
| donnyd | The mtu was lowered to 1450 | 19:23 |
| donnyd | not sure what the other providers are set at | 19:23 |
| sean-k-mooney | we are using the ipv4 addres | 19:24 |
| sean-k-mooney | 192.168.48.93 in this case | 19:24 |
| openstackgerrit | Merged openstack/nova master: Get pci_devices from _list_devices https://review.opendev.org/680674 | 19:24 |
| donnyd | can we replicate this job on a test instance that doesn't get kilt | 19:24 |
| sean-k-mooney | we can always ask infra to hold the node | 19:25 |
| donnyd | So we can go poke around and see what the real issue is.. if its just FN failing this job then it must be FN related... | 19:25 |
| dansmith | donnyd: it's not just FN | 19:25 |
| sean-k-mooney | donnyd: its also failing on both of the ovh clouds | 19:26 |
| dansmith | and for this, it matters what is in the cell mapping, not what is in the config file, just FYI | 19:26 |
| donnyd | so it only succeeds on rax then? | 19:26 |
| donnyd | what about limestone or vexxhost? | 19:27 |
| donnyd | i think limestone is setup like FN with ipv6 | 19:27 |
| sean-k-mooney | this is the logstash link for elastic search http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Timed%20out%20waiting%20for%20response%20from%20cell%5C%22%20AND%20tags:%5C%22screen-n-sch.txt%5C%22%20AND%20voting:1&from=864000s | 19:28 |
| donnyd | And I think maybe using the custom labels will make sure you can schedule the job to a provider known to fail.. | 19:28 |
| mriedem | this would be right around the time where we hit the scheduler for a tempest test to create a server and the timeout happens 1 minute after that | 19:29 |
| mriedem | https://zuul.opendev.org/t/openstack/build/d53346210978403f888b85b82b2fe0c7/log/logs/syslog.txt.gz#8269 | 19:29 |
| mriedem | idk if the netlink messages in there could be an issue | 19:29 |
| sean-k-mooney | mriedem: i just realised im looking at a single node greade-heat job which has the same issue | 19:29 |
| openstackgerrit | Merged openstack/nova master: [Trivial]Remove unused helper should_switch_to_postcopy https://review.opendev.org/679177 | 19:31 |
| *** tbachman has quit IRC | 19:31 | |
| openstackgerrit | Merged openstack/nova master: [Trivial]Removed unused helper _extract_query_params https://review.opendev.org/679174 | 19:31 |
| mriedem | looks like those syslog messages are 'normal' | 19:32 |
| dansmith | mriedem: yeah probably not related to this | 19:32 |
| dansmith | I dunno if OVS uses netlink for any datapath stuff, but in general, it's for setup of networking stuff, | 19:33 |
| *** nweinber_ has joined #openstack-nova | 19:33 | |
| dansmith | not related to moving actual packets | 19:33 |
| dansmith | although I guess if those indicated failure to set something up, then maybe, but.. | 19:33 |
| donnyd | I am also curious if this job was succeeding before I changed the MTU on FN | 19:33 |
| sean-k-mooney | it uses netlink for a few things | 19:33 |
| sean-k-mooney | donnyd: i dont think its MTU related | 19:34 |
| donnyd | Well did this job pass on FN before 12 Sep? | 19:34 |
| sean-k-mooney | as i said the logs i was looking at for the greade-heat job that failed was singlenode | 19:34 |
| donnyd | or has it always failed | 19:34 |
| *** nweinber has quit IRC | 19:35 | |
| mriedem | donnyd: this is recent | 19:36 |
| mriedem | logstash only goes back 10 days for us but i'm seeing it go to at least 9/14 | 19:36 |
| mriedem | i only noticed the failure while looking at some logs this weekend thouh | 19:36 |
| mriedem | *though | 19:36 |
| mriedem | so idk when it started, | 19:36 |
| donnyd | reason I ask is the MTU on FN was changed on the 12th here https://review.opendev.org/#/c/681951/ | 19:36 |
| mriedem | if you haven't noticed yet, when we get close to feature freeze and RC1, people recheck job failures like a chicken with its head cut off just to get their code merged | 19:37 |
| mriedem | Sep 22 00:14:19 ubuntu-bionic-ovh-gra1-0011664420 sudo[13369]: stack : TTY=unknown ; PWD=/opt/stack/old/devstack ; USER=root ; COMMAND=/sbin/ip link set mtu 1500 dev br-ex | 19:37 |
| donnyd | message:"sbin/ip link set mtu 1500 dev br-ex" AND node_provider:"fortnebula-regionone" | 19:41 |
| donnyd | yea that is probably not going to work well | 19:41 |
| *** jmlowe has quit IRC | 19:42 | |
| donnyd | Sep 23 16:41:40 ubuntu-bionic-fortnebula-regionone-0011712952 sudo[14580]: stack : TTY=unknown ; PWD=/opt/stack/old/devstack ; USER=root ; COMMAND=/sbin/ip link set mtu 1500 dev br-ex | 19:42 |
| *** pcaruana has joined #openstack-nova | 19:43 | |
| openstackgerrit | Merged openstack/nova master: Remove stubs from VolumeAttachmentsSample API sample test https://review.opendev.org/680834 | 19:43 |
| openstackgerrit | Merged openstack/nova master: Use multiple attachments in test_list_volume_attachments https://review.opendev.org/681618 | 19:43 |
| donnyd | best case it will just cause every packet to be retransmitted at a lower MTU... | 19:43 |
| sean-k-mooney | donnyd: again i dont think this is mtu related. none of the tests shoudl need to send packet that leave teh host | 19:44 |
| sean-k-mooney | donnyd: the way tempest executs zuul ssh's into the vm provided by FN and runs evertying from within that vm | 19:45 |
| sean-k-mooney | in a single node deployment that means that noting should leave the vm | 19:45 |
| donnyd | Well I don't think they would have to... | 19:45 |
| sean-k-mooney | so the MTU would cause any issues | 19:45 |
| donnyd | the interface itself that br-ex is connected to would have a lower MTU though wouldn't it | 19:46 |
| donnyd | and is the API connected to that same interface? | 19:46 |
| sean-k-mooney | it may but tempest wont be using that interface | 19:46 |
| donnyd | Ok | 19:47 |
| donnyd | I think we should just use the custom label and then hold the node | 19:48 |
| donnyd | so we can see what is in fact the what | 19:48 |
| hemna | hrmm, so the tox -epy27 issue goes away on ubuntu for stable/rocky | 19:48 |
| hemna | happens on opensuse Leap | 19:49 |
| hemna | :( | 19:49 |
| sean-k-mooney | what is the issue? | 19:49 |
| hemna | http://paste.openstack.org/show/778953/ | 19:49 |
| donnyd | yes what=issue | 19:49 |
| *** dave-mccowan has joined #openstack-nova | 19:50 | |
| mriedem | hemna: oh, i can't speak for suse, i only use bionic for unit test runs and local dev | 19:50 |
| mriedem | maybe some sqlite issue with the suse you're using? | 19:50 |
| hemna | hrmm yah I suppose so | 19:51 |
| owalsh | lyarwood, mriedem: AFAIK tripleo isn't doing anything for the cells (apart from the default cell). The cell has to be manually added and the api services need to be bounced | 19:52 |
| mriedem | owalsh: ack | 19:53 |
| owalsh | mriedem: and as lee mentioned support is recent so there isn't an upgrade path to worry about yet. Does something need poking? | 19:54 |
| mriedem | no, just started seeing some weirdness related to cell connection timeouts during scheduling after upgrade in our grenade jobs | 19:56 |
| mriedem | melwitt: this should be good to go now, your comments have been addressed https://review.opendev.org/#/c/541420/ | 19:57 |
| mriedem | and the patch on top is happy | 19:57 |
| openstackgerrit | Merged openstack/nova master: Tune up db.instance_get_all_uuids_by_hosts https://review.opendev.org/679627 | 19:58 |
| openstackgerrit | Merged openstack/nova master: Add reminder to update corresponding glance docs https://review.opendev.org/682012 | 19:58 |
| *** derekh has joined #openstack-nova | 20:01 | |
| *** derekh has quit IRC | 20:01 | |
| *** brault has quit IRC | 20:02 | |
| *** markvoelker has quit IRC | 20:07 | |
| *** jmlowe has joined #openstack-nova | 20:09 | |
| efried | mriedem: docs for vpmem if you please? https://review.opendev.org/#/c/680300/ | 20:10 |
| *** pcaruana has quit IRC | 20:12 | |
| mriedem | efried: one of the cores that reviewed that series can hit it | 20:17 |
| mriedem | i wouldn't know if any of it is true or not | 20:17 |
| efried | ack | 20:17 |
| *** brault has joined #openstack-nova | 20:18 | |
| openstackgerrit | Merged openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.opendev.org/641453 | 20:22 |
| *** tbachman has joined #openstack-nova | 20:30 | |
| *** gbarros has quit IRC | 20:35 | |
| *** gbarros has joined #openstack-nova | 20:36 | |
| melwitt | mriedem: ack, will look | 20:41 |
| *** nweinber_ has quit IRC | 20:59 | |
| *** tbachman has quit IRC | 21:03 | |
| *** xek has quit IRC | 21:05 | |
| *** tesseract has quit IRC | 21:23 | |
| *** igordc has joined #openstack-nova | 21:27 | |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: resubmit image metadata prefiltering spec for ussuri https://review.opendev.org/683258 | 21:32 |
| *** BjoernT_ has quit IRC | 21:34 | |
| openstackgerrit | Merged openstack/nova-specs master: Spec: Provider config YAML file https://review.opendev.org/680471 | 21:34 |
| *** rcernin has joined #openstack-nova | 21:38 | |
| *** rcernin has quit IRC | 21:40 | |
| *** rcernin has joined #openstack-nova | 21:40 | |
| *** takashin has joined #openstack-nova | 21:47 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Log error when volume validation fails during boot from volume https://review.opendev.org/684140 | 21:49 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional tests for [cinder]/cross_az_attach=False https://review.opendev.org/684141 | 21:49 |
| mriedem | dansmith: ^ i wrote some functional tests for that long-standing cross_az_attach api bug | 21:49 |
| mriedem | rather than rely on devstack wip'ery | 21:49 |
| mriedem | crap, i should actually add the file | 21:50 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional tests for [cinder]/cross_az_attach=False https://review.opendev.org/684141 | 21:50 |
| *** markvoelker has joined #openstack-nova | 21:53 | |
| *** mriedem has quit IRC | 21:55 | |
| *** markvoelker has quit IRC | 21:58 | |
| openstackgerrit | Merged openstack/nova-specs master: resubmit image metadata prefiltering spec for ussuri https://review.opendev.org/683258 | 22:02 |
| *** mlavalle has quit IRC | 22:15 | |
| *** gbarros has quit IRC | 22:18 | |
| *** adriant has quit IRC | 22:39 | |
| *** ivve has quit IRC | 22:40 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Update keypairs in saving an instance object https://review.opendev.org/683043 | 22:43 |
| *** tkajinam has joined #openstack-nova | 23:01 | |
| *** TxGirlGeek has quit IRC | 23:06 | |
| *** adriant has joined #openstack-nova | 23:12 | |
| *** tbachman has joined #openstack-nova | 23:40 | |
| *** alex_xu has joined #openstack-nova | 23:44 | |
| *** luksky has quit IRC | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!