Wednesday, 2019-03-13

*** betherly has joined #openstack-nova00:02
*** betherly has quit IRC00:07
*** slaweq has joined #openstack-nova00:11
gmannnova-next job is failing on Bionic migration. i think for TLS console proxy  - https://bugs.launchpad.net/nova/+bug/181979400:12
openstackLaunchpad bug 1819794 in OpenStack Compute (nova) "nova-next job fail on Ubuntu Bionic" [Undecided,New]00:12
gmannBionic migration will happen tomorrow EOD, either we need to make this job as n-v till fix the issue or fix it before that.00:13
gmannmelwitt: efried dansmith bauzas gibi stephenfin ^^ if any one of you know or figure out the issue.00:14
melwittgmann: thanks. I didn't realize the tls-proxy service was old and to not be used anymore00:15
*** IvensZambrano has quit IRC00:15
gmannmelwitt: it is enable and working fine for devstack based job but i think nova-next is the only legacy job (running on xenial) which enbale tls-proxy00:15
melwittwe added it as an "easy" way to get tls certs setup in the env, for testing console proxies with tls enabled. BUT, it's been several months that I noticed the job was no longer running console proxies with tls ("cannot find cert") so we're not gaining anything by enabling it for a long time now00:16
gmannyeah it run on xenial. my patch move the job to run on bionic.00:17
gmannrest all legacy job like nova-live-migration, nova-lvm etc run fine but nova-next fail00:19
melwittok, I think we need to stop enabling it on nova-next and leave a TODO that we need to find out what's the right way to configure tls certs in test env today00:19
gmannmelwitt: ok. let me give try with that if failure is only because of tls cert00:20
melwittgmann: to be clear, the only reason we were including tls-proxy service was to get the automatic setup tasks that devstack did when it was enabled00:21
gmannmelwitt: i see.00:22
melwittit used to copy the certs to right directories etc. but several months ago, maybe even a year ago, those setup tasks no longer worked and we found our console proxies running without tls (they couldn't find cert that they used to be able to find)00:22
melwittprobably related to the legacy/deprecated nature of tls-proxy service00:23
melwittso tls-proxy isn't really related directly to our testing in nova-next, so since it's causing a problem, we can take it out. it wasn't helping us anymore anyway00:23
gmannmay be, we need to check if it is working in tempest-full. if so then we can wait to move nova-next on devstack based job00:23
*** betherly has joined #openstack-nova00:23
gmannsure.00:24
*** slaweq has quit IRC00:24
melwittgmann: nova-next is the only place where we try to test console proxy with tls. because the tls certs etc has to be setup00:24
openstackgerritMerged openstack/nova master: Allow utime call to fail on qcow2 image base file  https://review.openstack.org/64131500:25
openstackgerritMerged openstack/nova stable/rocky: Handle missing exception in instance creation code  https://review.openstack.org/64140100:25
openstackgerritMerged openstack/nova stable/rocky: Fix WeighedHost logging regression  https://review.openstack.org/64135500:25
melwittwe just don't know the right way to do that nowadays. we used to piggyback on tls-proxy to do it00:25
gmannok00:26
openstackgerritGhanshyam Mann proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic  https://review.openstack.org/63901700:26
gmannmelwitt: ^^ if it work then, i will mark nova OK in https://etherpad.openstack.org/p/legacy-job-bionic00:27
melwittgmann: OK, sounds good. thanks00:27
gmannmelwitt: thanks for help and updates on tls thing.00:28
*** betherly has quit IRC00:28
*** luksky has quit IRC00:28
melwittgmann: np, thanks for the heads up about the failure on bionic00:28
*** gyee has quit IRC00:30
*** lbragstad has quit IRC00:40
openstackgerritMerged openstack/nova stable/rocky: ironic: stop hammering ironic API in power sync loop  https://review.openstack.org/64077100:42
openstackgerritMerged openstack/nova stable/queens: Add description of custom resource classes  https://review.openstack.org/61912500:42
*** wolverineav has quit IRC00:42
*** markvoelker has joined #openstack-nova00:43
*** betherly has joined #openstack-nova00:44
*** tetsuro has joined #openstack-nova00:44
*** wolverineav has joined #openstack-nova00:46
*** betherly has quit IRC00:49
*** wolverineav has quit IRC00:51
*** whoami-rajat has joined #openstack-nova00:51
*** wolverineav has joined #openstack-nova00:53
*** tbachman has quit IRC00:57
*** zhubx has quit IRC00:58
*** zhubx has joined #openstack-nova00:58
*** tkajinam_ has joined #openstack-nova00:58
*** tkajinam has quit IRC01:01
*** liuyulong has joined #openstack-nova01:03
*** betherly has joined #openstack-nova01:05
mnaserefried: mind joining #openstack-tc -- just have some questions about powervm state01:08
*** betherly has quit IRC01:10
*** slaweq has joined #openstack-nova01:11
*** marst has joined #openstack-nova01:14
*** tbachman has joined #openstack-nova01:16
*** markvoelker has quit IRC01:21
*** slaweq has quit IRC01:24
*** agopi has joined #openstack-nova01:24
*** Nel1x has joined #openstack-nova01:25
*** betherly has joined #openstack-nova01:26
*** betherly has quit IRC01:31
*** wolverineav has quit IRC01:31
*** wolverineav has joined #openstack-nova01:32
*** zbr has quit IRC01:34
*** tbachman has quit IRC01:34
*** wolverineav has quit IRC01:37
*** BjoernT has joined #openstack-nova01:38
melwittmriedem: I just noticed in #openstack-tc they're talking about jobs moving from xenial => bionic and project job failures. I don't understand most of what's in this email by fungi, but I think might know and be able to help: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003746.html01:39
melwitt*I think you might know01:39
fungii am happy to explain too01:40
fungiright now the only failure i see for nova is your functional-py35 job01:40
fungithe problem is that when master branch jobs all switch to running on bionic by default, py35 jobs will still need to run on xenial because bionic doesn't provide python3.501:41
fungiwe solved this for any of the shared/abstract py35-specific jobs by setting them to use an ubuntu-xenial nodeset unconditionally01:41
fungibut nova's functional-py35 job doesn't have any of those as a parent01:42
fungithe options there are to change the parent on the job, or set an explicit nodeset in it to use ubuntu-xenial01:42
fungieither fix should work and is only needed on master since the current stable branches already get their jobs set to use ubuntu-xenial01:43
*** lbragstad has joined #openstack-nova01:44
*** tetsuro has quit IRC01:47
melwittfungi: thanks01:54
fungione possible solution can be found in my edit to nova's .zuul.yaml in https://review.openstack.org/64258001:56
melwittthanks02:00
funginova gets this somewhat easier than other teams, since proposing changes to different branches of it is how i performed my preliminary testing02:01
* melwitt nods02:02
*** betherly has joined #openstack-nova02:08
*** tbachman has joined #openstack-nova02:11
*** betherly has quit IRC02:12
openstackgerritMerged openstack/nova stable/queens: Make host_manager use scatter-gather and ignore down cells  https://review.openstack.org/63759902:18
openstackgerritmelanie witt proposed openstack/nova master: Update nova-tox-functional-py35 parent job  https://review.openstack.org/64294002:20
*** lbragstad has quit IRC02:24
*** slaweq has joined #openstack-nova02:26
*** tbachman has quit IRC02:28
*** betherly has joined #openstack-nova02:29
*** tbachman has joined #openstack-nova02:29
*** betherly has quit IRC02:34
*** slaweq has quit IRC02:39
*** betherly has joined #openstack-nova02:50
*** betherly has quit IRC02:54
openstackgerritBoxiang Zhu proposed openstack/nova master: Trivialfix for help description of images_type  https://review.openstack.org/64294203:01
*** psachin has joined #openstack-nova03:03
*** Nel1x has quit IRC03:05
*** betherly has joined #openstack-nova03:11
*** slaweq has joined #openstack-nova03:12
*** betherly has quit IRC03:15
*** slaweq has quit IRC03:24
*** betherly has joined #openstack-nova03:31
*** betherly has quit IRC03:36
*** marst has quit IRC03:37
*** itlinux has quit IRC03:41
*** BjoernT has quit IRC04:01
openstackgerritMerged openstack/nova stable/queens: Don't emit warning when ironic properties are zero  https://review.openstack.org/60861104:08
*** slaweq has joined #openstack-nova04:11
*** udesale has joined #openstack-nova04:17
*** slaweq has quit IRC04:24
*** zhubx has quit IRC04:34
*** zhubx has joined #openstack-nova04:35
*** zhubx has quit IRC04:36
*** zhubx has joined #openstack-nova04:36
*** zhubx has quit IRC04:37
*** zhubx has joined #openstack-nova04:38
*** betherly has joined #openstack-nova04:44
*** zhubx has quit IRC04:44
*** zhubx has joined #openstack-nova04:45
*** betherly has quit IRC04:49
*** janki has joined #openstack-nova05:04
*** betherly has joined #openstack-nova05:05
*** betherly has quit IRC05:09
*** slaweq has joined #openstack-nova05:11
*** wolverineav has joined #openstack-nova05:12
openstackgerritGhanshyam Mann proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic  https://review.openstack.org/63901705:14
openstackgerritGhanshyam Mann proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic  https://review.openstack.org/63901705:15
*** udesale has quit IRC05:16
*** udesale has joined #openstack-nova05:17
*** jaosorior has joined #openstack-nova05:22
*** slaweq has quit IRC05:24
*** sridharg has joined #openstack-nova05:27
*** igordc has quit IRC05:30
*** liuyulong has quit IRC05:41
*** zhubx has quit IRC05:43
*** zhubx has joined #openstack-nova05:44
*** markvoelker has joined #openstack-nova05:47
*** udesale has quit IRC05:51
*** betherly has joined #openstack-nova05:57
*** betherly has quit IRC06:01
*** slaweq has joined #openstack-nova06:11
*** udesale has joined #openstack-nova06:11
openstackgerritChris Friesen proposed openstack/nova master: Add support for resize and cold migration of emulated TPM files  https://review.openstack.org/63993406:18
*** pcaruana has joined #openstack-nova06:19
*** slaweq has quit IRC06:24
*** slaweq has joined #openstack-nova06:26
*** jdillaman has quit IRC06:30
*** jdillaman has joined #openstack-nova06:35
*** slaweq has quit IRC06:36
*** cfriesen has quit IRC06:37
*** masayukig[m]2 is now known as masayukig[m]06:45
*** masayukig[m] is now known as masayuki406:45
*** masayuki4 is now known as masayuki606:48
*** masayuki6 is now known as masayukig[m]06:48
*** Luzi has joined #openstack-nova06:48
*** masayukig[m] is now known as masayukig06:48
*** wolverineav has quit IRC06:53
*** ivve has joined #openstack-nova06:53
*** slaweq has joined #openstack-nova07:12
*** tonyb is now known as tonyb_gone07:20
*** tonyb_gone is now known as tonyb07:21
*** dpawlik has joined #openstack-nova07:23
*** slaweq has quit IRC07:24
*** slaweq has joined #openstack-nova07:35
*** luksky has joined #openstack-nova07:41
*** wolverineav has joined #openstack-nova07:51
*** dpawlik has quit IRC07:54
*** Dinesh_Bhor has joined #openstack-nova07:56
*** wolverineav has quit IRC07:56
*** Dinesh_Bhor has quit IRC07:57
*** tssurya has joined #openstack-nova08:02
*** lennyb has joined #openstack-nova08:06
*** dpawlik has joined #openstack-nova08:06
*** awalende has joined #openstack-nova08:09
*** ralonsoh has joined #openstack-nova08:09
*** rcernin has quit IRC08:09
*** rpittau|afk is now known as rpittau08:09
*** tesseract has joined #openstack-nova08:14
*** zhubx has quit IRC08:17
*** zhubx has joined #openstack-nova08:17
*** tesseract has quit IRC08:18
*** tesseract has joined #openstack-nova08:18
*** zhubx has quit IRC08:21
*** ccamacho has joined #openstack-nova08:23
*** zhubx has joined #openstack-nova08:23
*** zhubx has quit IRC08:23
*** zhubx has joined #openstack-nova08:23
*** zhubx has quit IRC08:25
*** zhubx has joined #openstack-nova08:26
*** helenafm has joined #openstack-nova08:29
*** xek has joined #openstack-nova08:50
*** jsheeren has joined #openstack-nova08:51
*** ttsiouts has joined #openstack-nova08:53
*** zbr has joined #openstack-nova09:06
*** ttsiouts has quit IRC09:06
Kevin_Zhengping gmann09:07
*** ttsiouts has joined #openstack-nova09:07
*** ttsiouts has quit IRC09:11
*** dtantsur|afk is now known as dtantsur09:14
*** kukacz has quit IRC09:15
dtantsurefried: agreed!09:15
*** kukacz has joined #openstack-nova09:17
*** panda|rover|off is now known as panda|rover09:17
*** tkajinam__ has joined #openstack-nova09:22
*** FlorianFa has joined #openstack-nova09:23
*** tkajinam_ has quit IRC09:25
*** ttsiouts has joined #openstack-nova09:26
*** cdent has joined #openstack-nova09:32
*** IvensZambrano has joined #openstack-nova09:34
*** derekh has joined #openstack-nova09:35
*** mikal has joined #openstack-nova09:40
*** mikal has quit IRC09:45
openstackgerritzhaixiaojun proposed openstack/nova master: Fix bug: non-admin user can not filter instance by instance-uuid  https://review.openstack.org/64240309:51
gibistephenfin: I'm wondering about the interference between the bandwidth feature and the numa aware vswitches feature09:55
gibistephenfin: is it a supported case that there are a single physnet with two ovs bridges one bridge connected to a nic on numa0 and the other bridge is connected to a nic on numa1?09:56
gibistephenfin: if it is supported how nova selects which bridge a port shall be plugged for this physnet?09:57
*** jangutter has joined #openstack-nova09:59
openstackgerritMerged openstack/nova stable/pike: Lock detach_volume  https://review.openstack.org/63753610:16
*** zhubx has quit IRC10:19
*** zhubx has joined #openstack-nova10:20
*** jsheeren has quit IRC10:28
fricklerin case others have - like me - difficulties to decide between melwitt and efried for PTL, watch this ML thread before you vote: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003765.html10:43
openstackgerritNeil Jerram proposed openstack/nova stable/queens: Allow utime call to fail on qcow2 image base file  https://review.openstack.org/64299710:45
*** priteau has joined #openstack-nova10:45
openstackgerritNeil Jerram proposed openstack/nova stable/rocky: Allow utime call to fail on qcow2 image base file  https://review.openstack.org/64299810:47
cdentfrickler: you might have more success if you ask some specific questions. Their nomination statements amount to "some statements explaining why they would be the better PTL"10:49
cdentif you want them to differentiate themselves more, you need to ask them about a specific concern you have or present a scenario that you wonder how they would deal with10:49
*** tbachman has quit IRC10:52
*** maciejjozefczyk has quit IRC10:56
*** cdent has quit IRC11:09
*** tosky has joined #openstack-nova11:17
*** mriedem has joined #openstack-nova11:26
mriedem.11:26
*** Alon_KS has joined #openstack-nova11:29
*** belmoreira has quit IRC11:31
*** belmoreira has joined #openstack-nova11:32
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Allow utime call to fail on qcow2 image base file  https://review.openstack.org/64301111:33
*** yaawang has quit IRC11:33
*** IvensZambrano has quit IRC11:35
*** yaawang has joined #openstack-nova11:42
openstackgerritAdrian Chiris proposed openstack/nova master: Move get_pci_mapping_for_migration to MigrationContext  https://review.openstack.org/64302311:50
openstackgerritAdrian Chiris proposed openstack/nova master: Allow driver to properly unplug VIFs on destination on confirm resize  https://review.openstack.org/64302411:50
*** cdent has joined #openstack-nova11:51
*** ttsiouts has quit IRC12:06
*** ttsiouts has joined #openstack-nova12:06
*** zhubx has quit IRC12:07
openstackgerritMerged openstack/nova master: pass endpoint interface to Ironic client  https://review.openstack.org/64087912:08
*** ttsiouts has quit IRC12:11
openstackgerritLee Yarwood proposed openstack/nova stable/queens: libvirt: Add workaround to cleanup instance dir when using rbd  https://review.openstack.org/62872612:11
*** markvoelker has quit IRC12:12
*** markvoelker has joined #openstack-nova12:13
*** tbachman has joined #openstack-nova12:24
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: Allow utime call to fail on qcow2 image base file  https://review.openstack.org/63349312:24
*** ttsiouts has joined #openstack-nova12:28
*** wolverineav has joined #openstack-nova12:29
mriedemtonyb: what do you think about backporting this? https://review.openstack.org/#/c/641351/12:30
mriedemi asked in the ML as well, but no bites.12:30
mriedemit's an ABI change but does fix a long-standing bug so admins don't shoot themselves12:30
mriedemso i could go either way12:30
*** panda|rover is now known as panda|rover|eat12:30
openstackgerritchenker proposed openstack/python-novaclient master: Optimize limit option docs Description  https://review.openstack.org/64303512:32
*** wolverineav has quit IRC12:34
*** janki has quit IRC12:34
openstackgerritchenker proposed openstack/python-novaclient master: Optimize limit option docs string description for servers.py  https://review.openstack.org/64303512:35
*** mchlumsky has joined #openstack-nova12:38
*** awalende has quit IRC12:43
*** awalende has joined #openstack-nova12:45
*** luksky has quit IRC12:50
openstackgerritchenker proposed openstack/python-novaclient master: Optimize limit option docs string description for servers.py  https://review.openstack.org/64303512:57
*** janki has joined #openstack-nova12:58
*** panda|rover|eat is now known as panda|rover13:00
cdentthat's werid mriedem. is something perhaps sending allocations at the wrong microversion?13:01
mriedemwell it looks like it's sending a PUT /allocations request with no resources right?13:09
mriedemand it also looks like it's only hitting in queens13:09
mriedemhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Failed%20to%20save%20allocation%20for%5C%22%20AND%20message%3A%5C%22DestinationHypervisorTooOld%5C%22%20AND%20tags%3A%5C%22screen-n-super-cond.txt%5C%22&from=7d13:10
mriedemi'm not sure why the hell it's logging DestinationHypervisorTooOld13:10
mriedemMar 13 01:08:15.816855 ubuntu-xenial-inap-mtl01-0003740112 nova-conductor[25115]: DEBUG nova.scheduler.client.report [None req-a9345308-4e20-4976-ac6d-a1c529d14b16 tempest-TestNetworkAdvancedServerOps-840936673 tempest-TestNetworkAdvancedServerOps-840936673] Sending updated allocation [{'resource_provider': {'uuid': u'13509a14-0b25-4020-b1d6-2c56641a4d82'}, 'resources': {}}] for instance c2fd36f9-0b41-439c-9700-17aa0abe13c2 a13:11
mriedem removing resources for 13509a14-0b25-4020-b1d6-2c56641a4d82. {{(pid=26101) remove_provider_from_instance_allocation /opt/stack/nova/nova/scheduler/client/report.py:1616}}13:11
mriedemwtf, it thinks it's doing a same-host resize or something13:12
*** udesale has quit IRC13:16
*** udesale has joined #openstack-nova13:17
openstackgerritmelanie witt proposed openstack/nova master: DNM: Testing nova legacy jobs on bionic  https://review.openstack.org/63901713:19
*** IvensZambrano has joined #openstack-nova13:21
*** luksky has joined #openstack-nova13:22
*** tosky has quit IRC13:22
mriedemoh heh, qemu is at different versions in this job13:26
mriedemgmann: the tempest-slow job on stable/queens is running with different versions of ubuntu on the compute nodes13:26
*** priteau has quit IRC13:27
mriedemhttps://bugs.launchpad.net/tempest/+bug/181991013:28
openstackLaunchpad bug 1819910 in OpenStack Compute (nova) "test_server_connectivity_live_migration intermittently fails with NoValidHost due to PUT /allocations failure" [Undecided,Confirmed]13:28
*** marst has joined #openstack-nova13:28
*** lbragstad has joined #openstack-nova13:31
*** IvensZambrano has quit IRC13:33
mriedemcdent: ok i figured out where that weird same-host allocation goofiness happens13:33
cdentmriedem: sorry, only pay half attention, doing my taxes, which was a surprise gift for today13:34
mriedemthat is a fun gift13:34
mriedemanyway, the allocations thing was a red herring13:34
mriedemduring live migration we call the scheduler for dest hosts and then do some checks on them in the conductor task, like the hypervisor version comparison between the source and dest,13:35
mriedemif that fails, we call https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1527 to remove the allocations created on the dest provider by the scheduler,13:35
*** tbachman has quit IRC13:36
mriedemif we get to https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L1593 it assumes we're in a same-host resize situation, which is wrong13:36
mriedemwe end up sending {} for resource allocatoins and placement pukes on that https://github.com/openstack/nova/blob/stable/queens/nova/scheduler/client/report.py#L162013:36
mriedemwe log a warning, and b/c we're doing this from an except block, oslo.log is also logging the original exception DestinationHypervisorTooOld - which makes it all very confusing13:36
*** eharney has joined #openstack-nova13:37
fricklermriedem: https://opendev.org/openstack-dev/devstack/src/branch/stable/queens/tools/fixup_stuff.sh#L8613:38
*** tbachman has joined #openstack-nova13:38
fricklerseems ENABLE_VOLUME_MULTIATTACH is only set on controller and not on compute113:38
mriedemfrickler: gah that would do it13:39
*** tbachman has quit IRC13:43
*** zhubx has joined #openstack-nova13:43
*** marst has quit IRC13:44
*** zhubx has quit IRC13:48
*** cdent_ has joined #openstack-nova13:54
*** cdent has quit IRC13:54
*** cdent_ is now known as cdent13:54
*** awaugama has joined #openstack-nova13:56
gmannmriedem: is it?14:00
*** amodi has quit IRC14:00
*** mlavalle has joined #openstack-nova14:00
*** marst has joined #openstack-nova14:01
mriedemgmann: yeah https://bugs.launchpad.net/tempest/+bug/181991014:05
openstackLaunchpad bug 1819910 in tempest "test_server_connectivity_live_migration intermittently fails with NoValidHost due to DestinationHypervisorTooOld" [High,Triaged]14:05
*** zhubx has joined #openstack-nova14:07
mriedemgmann: i'll try pushing a fix to tempest14:09
*** irclogbot_2 has quit IRC14:09
gmannmriedem: thanks, is UCA thing will fix it.14:11
*** irclogbot_2 has joined #openstack-nova14:11
mriedemgmann: ok i was working a patch but are you doing it now?14:14
gmannmriedem: no, i am doing legacy job migration as of now. if you are working please push.14:15
mriedemhttps://review.openstack.org/64305214:20
mriedemi'll push a nova stable/queens change that depends on ^ to make sure it works14:20
*** bnemec has quit IRC14:22
*** eharney has quit IRC14:23
*** zhubx has quit IRC14:24
*** irclogbot_2 has quit IRC14:25
*** irclogbot_2 has joined #openstack-nova14:27
*** BjoernT has joined #openstack-nova14:27
*** bnemec has joined #openstack-nova14:27
*** eharney has joined #openstack-nova14:28
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Fix live migration _remove_host_allocations warnings  https://review.openstack.org/64305514:28
*** tbachman has joined #openstack-nova14:36
openstackgerritCorey Bryant proposed openstack/nova master: add python 3.7 unit test job  https://review.openstack.org/61069414:36
*** brtknr has joined #openstack-nova14:38
brtknrhello!14:38
brtknri have an instance stuck in BUILD which I do not seem to be able to delete...14:39
brtknrwhats the remedy14:39
brtknrit appears when i do `openstack server list` but when i run `openstack server delete UUID`, i get: No server with a name or ID of '1a141848-c91a-450b-b4ad-6d2f81e4125e' exists.14:39
mriedembrtknr: hmm, is there an entry for that instance_uuid in the nova_api.build_requests table?14:40
*** tbachman has quit IRC14:40
brtknrin the database?14:41
brtknrprobably...14:41
brtknrit exists as an entity somewhere...14:41
brtknrbut its not deletable14:41
mriedemcan you actually check the nova_api.build_requests table for a record with that instance_uuid?14:41
mriedembecause it should be temporary14:41
*** agopi has quit IRC14:42
*** agopi has joined #openstack-nova14:42
brtknrmriedem: im checking, one sec14:43
brtknrmysql> select * from build_requests;14:44
brtknrEmpty set (0.00 sec)14:44
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: DNM: testing tempest-slow fix for subnode config  https://review.openstack.org/64305914:46
*** mlavalle has quit IRC14:47
brtknrmriedem: ive just deleted another server and its gone into the same state... still shows up under server list but cannot be deleted...14:47
*** tbachman has joined #openstack-nova14:47
mriedembrtknr: so it's probably in a cell but the instance_mappings record in the nova_api database for it does not have cell_id set for some reason14:48
mriedemcheck the nova_api.instance_mappings table for the instance_uuid in question,14:48
mriedemthe cell_id column should be set, but i'm guessing it's not in your case14:49
mriedemthat would suggest something is failing in your nova-conductor service during scheduling where the instance mapping is not updated to point at the cell in which the instance record actually exists14:49
mriedemwhich is i'm guessing either cell0 or the 'nova' database (cell1) for you14:50
*** BjoernT_ has joined #openstack-nova14:56
openstackgerritMerged openstack/nova stable/pike: [pike-only] Fix resize_instance rpcapi call  https://review.openstack.org/60343914:57
openstackgerritMerged openstack/nova stable/pike: tox: Don't write byte code (maybe)  https://review.openstack.org/63691914:57
openstackgerritMerged openstack/nova stable/pike: PCI: do not force remove allocated devices  https://review.openstack.org/63507414:57
openstackgerritMerged openstack/nova stable/pike: Note the aggregate allocation ratio restriction in scheduler docs  https://review.openstack.org/62355214:57
dansmithwow, pike14:58
*** BjoernT has quit IRC14:58
*** psachin has quit IRC15:07
*** mriedem has quit IRC15:10
*** mriedem has joined #openstack-nova15:11
*** _alastor_ has quit IRC15:13
*** _alastor_ has joined #openstack-nova15:14
*** lpetrut has joined #openstack-nova15:14
*** Luzi has quit IRC15:14
mriedemdansmith: if you like those, mayhap you want to hit these other rocky +2'ed changes https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/rocky+label:Code-Review=215:15
dansmithno good deed15:16
*** BjoernT_ has quit IRC15:18
*** BjoernT has joined #openstack-nova15:19
*** dpawlik has quit IRC15:22
openstackgerritMerged openstack/nova stable/rocky: Handle templated cell mappings in nova-status  https://review.openstack.org/64261115:23
*** igordc has joined #openstack-nova15:23
*** _alastor_ has quit IRC15:24
*** _alastor_ has joined #openstack-nova15:24
dansmithmriedem: I got stopped yesterday with that bigger ironic change at the top there15:26
mriedemjust plow right through those concerns15:27
*** cdent has quit IRC15:28
*** awalende has quit IRC15:28
*** erlon has joined #openstack-nova15:29
openstackgerritBalazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance to make place for port healing  https://review.openstack.org/63795315:29
openstackgerritBalazs Gibizer proposed openstack/nova master: Refactor _heal_allocations_for_instance (2)  https://review.openstack.org/63795415:29
openstackgerritBalazs Gibizer proposed openstack/nova master: nova-manage: heal port allocations  https://review.openstack.org/63795515:29
openstackgerritBalazs Gibizer proposed openstack/nova master: cache neutron ports in heal allocation  https://review.openstack.org/63820715:29
*** zhubx has joined #openstack-nova15:32
*** mikal has joined #openstack-nova15:34
*** irclogbot_2 has quit IRC15:36
*** irclogbot_2 has joined #openstack-nova15:38
*** mikal has quit IRC15:39
openstackgerritMerged openstack/nova master: Remove duplicate cleanup in functional tests  https://review.openstack.org/63699615:43
*** luksky has quit IRC15:47
mriedemgibi: i went over the docs for bw providers https://review.openstack.org/#/c/642064/ some comments inline15:47
mriedemlet me know if you want to discuss my comments15:47
gibimriedem: thanks. looking15:47
*** IvensZambrano has joined #openstack-nova15:49
*** eharney has quit IRC15:49
*** irclogbot_2 has quit IRC15:49
gmannmelwitt: mriedem nova-next is failing with same reason  - http://logs.openstack.org/17/639017/8/check/nova-next/21327c4/logs/screen-n-cpu.txt.gz#_Mar_13_14_31_14_88612315:49
*** irclogbot_2 has joined #openstack-nova15:50
mriedemgmann: i don't think that's the same,15:51
mriedemand i don't think nova-next is multinode15:51
mriedemhttp://logs.openstack.org/17/639017/8/check/nova-next/21327c4/logs/libvirt/qemu/instance-00000001.txt.gz15:52
gmannmriedem: not same as tempest-slow. i mean cert issue which was occurring early starting from when migrating it to bionic15:52
mriedem-vnc 127.0.0.1:0,tls,x509verify=/etc/pki/libvirt-vnc: Failed to start VNC server: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.15:52
gmannyeah this one.15:52
gmannmelwitt: was trying it with 1024 bit.15:52
*** irclogbot_2 has quit IRC15:52
mriedemok i don't know about the vnc / cert stuff15:53
gmannmriedem: sorry for mixing it. it is for https://review.openstack.org/#/c/639017/15:53
*** cfriesen has joined #openstack-nova15:54
cfriesenI think this should be a fairly uncontroversial bugfix (has +2 from efried): https://review.openstack.org/#/c/641480/15:55
*** irclogbot_2 has joined #openstack-nova15:55
mriedemdansmith: you'll never believe it by the $11 lady won15:58
mriedem*but the15:58
mriedemi've been carole'ed15:58
dansmithmriedem: now you have to explain to maya why she's not an idiot after all?15:58
mriedemi think she's sleeping15:58
dansmithheh15:59
mriedemtoo much colonial penn commercials16:00
mriedemand oxygen tanks getting you down16:00
dansmithis the oatmeal guy still double-dipping on insurance commercials?16:01
dansmithoh I guess colonial penn was someone else16:02
mriedemalex trebek16:02
mriedemRIP16:02
mriedemoatmeal was wilford brimley16:02
mriedemfrom "the thing" fame16:02
mriedemhttps://www.youtube.com/watch?v=3qSbYEbtfoY16:03
dansmithyeah but he also did Liberty Mutual insurance, another daytime TV staple16:03
mriedemhere is now doing heavy computer work16:03
mriedemnote the (1) empty bottle of vodka on the desk and (2) .38 special in the drawer16:04
dansmithI have never seen this movie, but the highly accurate 1980s computer simulation makes me think I'm missing out16:05
mriedemi think that's a 38 special anyway, it's been awhile since i've listened to lynard skynard16:05
mriedemthe thing is one of my all time favorites16:05
dansmithI don't recall them providing details of .38 special identification16:06
mriedemthe 80s version of course16:06
mriedemi guess i'm thinking of saturday night special16:06
dansmithI believe there was something about technique.. perhaps holding on loosely.. but that wasn't skynard16:06
mriedemare those equivalent hand guns?16:06
*** walshh___ has quit IRC16:07
mriedemnobody likes the actual band "38 special"16:07
dansmithI believe that skynard was indeed making a reference to a .38 calibre handgun in that song yes16:07
mriedemI WIN!16:07
mriedemcfriesen: is that maybe due to a newer libvirt changing the type of error returned in that case?16:07
*** cdent has joined #openstack-nova16:09
*** erlon has quit IRC16:11
*** mikal has joined #openstack-nova16:12
cfriesenmriedem: as far as I can tell the libvirt line has been there since like 2012:  https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_monitor_json.c#L39216:12
gibimriedem: if I understand you correctly then in the bandwidth doc patch then a possible split would be to only describe the API limitation in the API guide and move the rest of the doc to the admin doc of nova16:13
mriedemcfriesen: yeah i saw the notes in the bug report, +W16:13
mriedemgibi: yeah more or less16:14
mriedemgibi: the api guide tends to be about the non-admin API end user experience, and the stuff about allocation candidates and granular group_policy stuff and flavors and virt drivers should probably be somewhere in the admin docs16:14
mriedemgibi: maybe in here https://docs.openstack.org/nova/latest/admin/adv-config.html16:15
gibimriedem: make sense, let me restructure the patch16:16
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: Add description for BDM volume_size  https://review.openstack.org/64125516:17
*** mikal has quit IRC16:17
*** b3nt_pin is now known as beagles16:23
*** priteau has joined #openstack-nova16:24
*** gyee has joined #openstack-nova16:25
openstackgerritguang-yee proposed openstack/nova stable/rocky: pass endpoint interface to Ironic client  https://review.openstack.org/64309816:30
*** wolverineav has joined #openstack-nova16:36
*** wolverineav has quit IRC16:39
*** wolverineav has joined #openstack-nova16:39
*** ivve has quit IRC16:40
*** helenafm has quit IRC16:40
mriedemdansmith: related to that az discussion we had yesterday, i realize we probably have another regression with resize revert,16:41
*** dave-mccowan has joined #openstack-nova16:41
*** FlorianFa has quit IRC16:41
mriedemwhen resizing, conductor will change the instance.az to whatever az the dest host is in,16:41
mriedembut on revert we don't set that back to whatever the source host is in, if it changed16:41
mriedemwhich technically it could if the server was created w/o a specific az16:42
*** tbachman has quit IRC16:42
*** udesale has quit IRC16:42
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif master: Refactor functional base test classes  https://review.openstack.org/64310116:43
*** lpetrut has quit IRC16:44
*** eandersson_ is now known as eandersson16:54
*** FlorianFa has joined #openstack-nova16:54
gmannmelwitt: mriedem nova-grenade-live-migration job also failing on bionic due to no ceph hammer for bionic - http://logs.openstack.org/17/639017/8/check/nova-grenade-live-migration/735242b/ara-report/result/8816413c-7053-418f-886b-e238fdea1ec5/16:56
gmannto proceed on bionic migration we can 1. disable tls-proxy  in nova-next 2. make nova-grenade-live-migration n-v till frickler finds out the reason if usage of ceph hammer16:57
gmannif ^^ is ok for you, i can update the patch.16:57
gmanni have planned the bionic migration ~ 2PM PT so we need to make nova change before that to avoid the gate block16:58
dansmithmriedem: yeah.. looking at the az stuff more recently has me thinking we've been neglecting that stuff too much in past years16:59
mriedemgmann: 1 and 2 are mutually exclusive problems correct? do we have bugs reported for those?17:04
*** lpetrut has joined #openstack-nova17:04
*** wolverineav has quit IRC17:04
mriedemgmann: is it not an option to keep these nova-specific jobs as voting and xenial until the issues are sorted out?17:04
mriedemthis is kind of a bad time to have to deal with this right before stein RC117:04
*** wolverineav has joined #openstack-nova17:06
openstackgerritBalazs Gibizer proposed openstack/nova master: Documentation for bandwidth support  https://review.openstack.org/64206417:07
gibimriedem: fyi ^^17:08
gibimriedem: don't rush to rereview as I'm leaving for today17:08
gibimriedem: btw, there is also a new version from the neutron doc https://review.openstack.org/64039017:09
gmannmriedem: https://bugs.launchpad.net/nova/+bug/1819794  https://bugs.launchpad.net/nova/+bug/181994417:09
openstackLaunchpad bug 1819794 in OpenStack Compute (nova) "nova-next job fail on Ubuntu Bionic" [Undecided,New]17:09
openstackLaunchpad bug 1819944 in OpenStack Compute (nova) "nova-grenade-live-migration job failing on ubuntu bionic" [Undecided,New]17:09
gmannmriedem: we should make n-v or fix. keeping them on xenial makes inconsistent that few job running on xenail and few on bionic17:09
* gibi will be off Thursday and Friday17:09
mriedemgmann: i don't have a fix if ceph doesn't have packages on bionic yet, or whatever the issue is there17:10
mriedemwe could temporarily disable the tls proxy stuff in the nova-next job on master17:10
mriedembut i'd need melwitt and/or stephenfin to help with that stuff17:10
mriedemand stephenfin is out this week17:10
*** wolverineav has quit IRC17:11
*** wolverineav has joined #openstack-nova17:11
mriedemdansmith: i'll write a functional test for that resize revert az thing because i got thinking about this with my cross-cell stuff so it'll be an issue there as well17:11
dansmithack17:11
mriedemgibi: ack, but we can't wait forever since we want those docs in the stein GA17:11
gmannyeah melwitt is trying the tsl-proxy with 2048 bit cert but that did not pass17:11
mriedemdoes she have a patch somewhere?17:12
mriedemis she around?17:12
gmannthis one (and deppends-on) - https://review.openstack.org/#/c/639017/817:12
mriedemok https://review.openstack.org/#/c/643045/17:12
mriedemhonestly melwitt probably has a better handle on the ceph CI stuff as well since i'm assuming she's got downstream ceph contacts17:15
*** dtantsur is now known as dtantsur|afk17:15
mriedemor eharney etc17:15
* mriedem jumps to cinder17:16
*** rpittau is now known as rpittau|afk17:16
mriedemcoreycb: actually you're probably the person to ask17:17
mriedemcoreycb: https://bugs.launchpad.net/nova/+bug/181994417:18
openstackLaunchpad bug 1819944 in OpenStack Compute (nova) "nova-grenade-live-migration job failing on ubuntu bionic due to missing ceph package" [Undecided,Confirmed]17:18
*** igordc has quit IRC17:19
*** gyee has quit IRC17:20
*** gyee has joined #openstack-nova17:20
coreycbmriedem: hi, we have luminous in our bionic /queensrepos17:23
coreycbbionic/queens repos17:23
mriedemyeah just saw http://download.ceph.com/debian-luminous/dists/bionic/17:25
coreycbi think they should be installing from ubuntu repos rather than http://download.ceph.com/debian-hammer/dists/bionic/main/binary-amd64/Packages17:25
mriedemi haven't looked into the actual devstack-ceph-plugin job yet to see where it's getting packages17:25
*** tssurya has quit IRC17:26
*** dave-mccowan has quit IRC17:27
*** ccamacho has quit IRC17:29
coreycbmriedem: i commented on the bug17:29
mriedemso apparently this isn't enough https://review.openstack.org/#/c/611594/17:29
coreycbthat should at least be sleeping for 6 seconds :)17:30
*** lpetrut has quit IRC17:30
mriedemumm, so https://review.openstack.org/#/c/611594/4/devstack/lib/ceph@94217:33
*** Swami has joined #openstack-nova17:33
mriedemfrickler: probably need your help here17:33
*** mikal has joined #openstack-nova17:34
mriedemgmann: is devstack-plugin-ceph-tempest-py3 ok with bionic? that's a single-node job17:36
mriedemi wonder if it's something with grenade in the nova-grenade-live-migration job17:36
gmannmriedem: yeah that is working fine17:36
mriedemhmm, ok, so then it's probably something in the nova setup scripts for that live migration job17:36
gmanni am figuring out on 611594 that why it is not picking luminous17:36
gmannseems so. in live_migration/hooks/run_tests.sh17:37
gmannas it is failing while post_test_hook - http://logs.openstack.org/17/639017/8/check/nova-grenade-live-migration/735242b/logs/devstack-gate-post_test_hook.txt.gz17:37
*** mikal has quit IRC17:38
mriedemgmann: how about the nova-live-migration job, is that ok with bionic?17:39
gmannmriedem: yes, that working fine on bionic17:39
*** eharney has joined #openstack-nova17:41
mriedemhmm, that post-test hook ceph script just runs functions from devstack-plugin-ceph,17:42
mriedemwhich is branchless17:42
gmannnova-live-migration has same issue with that ceph pckg but use old one and process further-  http://logs.openstack.org/17/639017/8/check/nova-live-migration/87084aa/logs/devstack-gate-post_test_hook.txt.gz#_2019-03-13_14_34_20_96717:43
gmannis something else stopping/failing  the nova-grenade-live-migration17:44
mriedemi think the nova-live-migration job just isn't failing properly17:45
mriedemand then goes on to run the evacuate stuff, which nova-grendae-live-migration doesn't run17:45
gmannyeah. not sure how d-g handle the failure on post_test_hook17:47
mriedemit must not17:47
mriedemgmann: to workaround bug 1819944 for now, we should probably just comment this out:17:49
openstackbug 1819944 in OpenStack Compute (nova) "nova-grenade-live-migration job failing on ubuntu bionic due to missing ceph package" [Undecided,New] https://launchpad.net/bugs/181994417:49
mriedemhttps://github.com/openstack/nova/blob/master/nova/tests/live_migration/hooks/run_tests.sh#L46-L5217:50
mriedemgmann: do you want to push a patch for that and i'll +217:50
mriedemthat way the job continues to run and be voting but the ceph stuff is disabled until someone fixes it17:50
gmannmriedem: ok thats even better, we run some stuff on that jobs17:50
*** wolverineav has quit IRC17:50
gmanni will push17:50
mriedemmake sure to push it real good17:51
mriedemhttps://www.youtube.com/watch?v=vCadcBR95oU17:51
dansmithmriedem: see, I was going to say that I had noted the reference,17:51
dansmithbut you keep going all self-serving with the youtube links17:51
gmann:)17:52
dansmithit's like laughing at your own jokes.. just.. so uncouth.17:55
*** wolverineav has joined #openstack-nova17:55
*** panda|rover is now known as panda|rover|off17:57
openstackgerritRodolfo Alonso Hernandez proposed openstack/os-vif master: Remove IP proxy methods  https://review.openstack.org/64311517:58
*** igordc has joined #openstack-nova17:58
*** derekh has quit IRC18:00
*** ttsiouts has quit IRC18:01
*** ttsiouts has joined #openstack-nova18:01
mriedemdansmith: i did that for gmann's benefit,18:02
mriedembecause i don't know if he grew up listening to TLC18:02
mriedemnot that i did, but i'm aware, it was hard to avoid18:02
dansmithDo you mean SnP ?18:03
dansmithwe're not discussing waterfalls here18:03
mriedemoh yeah heh18:03
mriedemor did i *really* know that but was trying to throw you off my trail?18:03
* dansmith scowls18:04
mriedemtwice in one day18:04
*** luksky has joined #openstack-nova18:04
dansmithI could do this all day18:04
*** ttsiouts has quit IRC18:05
openstackgerritGhanshyam Mann proposed openstack/nova master: Skip the ceph based live migration testing  https://review.openstack.org/64312218:06
gmannmriedem: ^^18:06
mriedemgmann: for bug 1819794 just do what you have here - disable tls https://review.openstack.org/#/c/639017/718:06
openstackbug 1819794 in OpenStack Compute (nova) "nova-next job fail on Ubuntu Bionic" [Undecided,New] https://launchpad.net/bugs/181979418:06
*** tesseract has quit IRC18:06
mriedemdansmith: want to +W https://review.openstack.org/#/c/643122/18:07
gmanni have not listen TLC or SNP  yet :)18:07
gmannmriedem: yes, doing on 64312218:07
mriedemgmann: well, something for you to bring home for dinner discussion with the mrs18:07
dansmithgmann: mriedem really likes 80s hip-hop18:07
gmannhumm18:08
dansmithmriedem: will look in a bit18:08
*** igordc has quit IRC18:08
*** xek_ has joined #openstack-nova18:08
mriedemthis is true https://www.youtube.com/watch?v=O4o8TeqKhgY18:08
*** xek has quit IRC18:10
*** igordc has joined #openstack-nova18:13
cfriesenis there any way to get detailed hypervisor/host resource information in nova?  Or is that a placement query now?18:14
cfriesen(for an admin, I mean)18:14
*** hongbin has joined #openstack-nova18:15
openstackgerritGhanshyam Mann proposed openstack/nova master: Disbale the tls-proxy in nova-next job for Bionic  https://review.openstack.org/64312918:15
openstackgerritGhanshyam Mann proposed openstack/nova master: Disbale the tls-proxy in nova-next job for Bionic  https://review.openstack.org/64312918:16
*** amodi_ has joined #openstack-nova18:16
*** amodi_ is now known as amodi18:16
gmannmriedem: ^^ doing it separately to revert those easily.18:16
*** priteau has quit IRC18:18
openstackgerritGhanshyam Mann proposed openstack/nova master: Migrate legacy jobs to Ubuntu Bionic  https://review.openstack.org/63901718:18
openstackgerritGhanshyam Mann proposed openstack/nova master: Migrate legacy jobs to Ubuntu Bionic  https://review.openstack.org/63901718:18
mriedemgmann: by "devstack based jobs" do you mean inheriting from the zuulv3 devstack job?18:20
gmannmriedem: yeah18:20
openstackgerritMatt Riedemann proposed openstack/nova master: Disable the tls-proxy in nova-next job for Bionic  https://review.openstack.org/64312918:21
mriedemok +2 on both of those18:21
mriedemefried: melwitt: fyi ^ to unblock bionic job migration18:21
openstackgerritMatt Riedemann proposed openstack/nova master: Migrate legacy jobs to Ubuntu Bionic  https://review.openstack.org/63901718:21
gmannmriedem: thanks, testing this on both of those - https://review.openstack.org/#/c/639017/1018:22
mriedemi'll throw these on the nova meeting agenda for gate bugs18:24
gmann+1. thanks18:24
*** pcaruana has quit IRC18:29
*** sridharg has quit IRC18:29
*** wolverineav has quit IRC18:30
*** ralonsoh has quit IRC18:31
*** wolverineav has joined #openstack-nova18:31
*** wolverineav has quit IRC18:31
*** wolverineav has joined #openstack-nova18:31
mriedemgmann: https://review.openstack.org/#/c/643052/ works18:36
mriedemhttp://logs.openstack.org/59/643059/1/check/tempest-slow/095dc55/compute1/logs/local_conf.txt18:36
mriedemENABLE_VOLUME_MULTIATTACH=True shows up in the subnode on the tempest-slow job in stable/queens now18:36
mriedemper https://review.openstack.org/#/c/643059/118:36
gmannmriedem: ok. i think it was not failing always. right18:39
melwittmriedem: ack. I think we'll also need this https://review.openstack.org/64294018:39
*** priteau has joined #openstack-nova18:41
gmanni saw you added the query for that bug in case. +2 on tempest patch.18:42
*** priteau has quit IRC18:44
*** wolverineav has quit IRC18:44
mriedemmelwitt: got it18:45
*** gmann is now known as gmann_afk18:48
*** amodi has quit IRC18:49
openstackgerritMatt Riedemann proposed openstack/nova master: Trivial: fix typo in reno  https://review.openstack.org/64313718:52
melwittgmann_afk: thanks for the heads up on the ceph thing. I'll look into how to fix it18:54
melwittdansmith: can you get this one too? need for bionic https://review.openstack.org/64294018:57
dansmithack18:57
melwittthanks18:58
*** jding1_ has joined #openstack-nova19:03
*** mchlumsky_ has joined #openstack-nova19:03
*** jding1__ has quit IRC19:04
*** mchlumsky has quit IRC19:04
*** zzzeek has quit IRC19:04
*** zzzeek has joined #openstack-nova19:07
*** wolverineav has joined #openstack-nova19:15
*** EmilienM has joined #openstack-nova19:19
EmilienMhello19:19
EmilienMI've been playing with ironic/nova on stein in tripleo environment (don't be scared :D) && I'm having an issue to register the nova compute as an hypervisor19:20
EmilienMhttp://paste.openstack.org/show/Bz6gl5oFokoah9Ug3VZc/19:21
EmilienMI'm wondering if this kind of error rings a bell to someone :)19:21
EmilienM(also asked on #openstack-ironic channel)19:21
mriedemjroll: ^19:21
mriedemthat RecursionError does not ring a bell19:22
mriedembut jroll added the virtdrivernotready stuff i think19:22
*** wolverineav has quit IRC19:22
tonybEmilienM: I hit that and was told to disable SSL in the undercloud19:24
dansmithyeah, interesting19:24
EmilienMtonyb: which I did19:24
tonybEmilienM: rats, but it's still using SSL19:24
EmilienMtonyb: well actually, not for public endpoint19:24
EmilienMdamn19:24
EmilienMtonyb: https://bugzilla.redhat.com/show_bug.cgi?id=168681719:25
openstackbugzilla.redhat.com bug 1686817 in openstack-nova "[osp15] nova is broken on the undercloud with ssl" [Urgent,Assigned] - Assigned to mbooth19:25
tonybEmilienM: I use: openstack-config --set undercloud.conf DEFAULT generate_service_certificate false19:25
*** gibi has quit IRC19:26
EmilienMtonyb: I'll try again without SSL and report back19:27
EmilienMmriedem: not sure if someone from Nova team is working on that bug19:27
EmilienMare you folks testing SSL (I guess yes?)19:27
tonybEmilienM: I htink it's tested in the gate but I don't know that ironic+nova is19:28
tonybI'll check later today19:28
*** BjoernT has quit IRC19:30
EmilienMtonyb: thanks!19:30
tonybEmilienM: I don't think I'll *fix* it but I can poke at least to see if it's a gap in our testing19:31
*** mikal has joined #openstack-nova19:35
tonybmriedem: I thinkI'm okay backporting that.19:35
tonybmriedem: I'll reply on list19:35
mriedemtonyb: ok thanks - bauzas is as well so wfm19:36
tonybmriedem: I need to check if any other data to those calls would cause a 400 status code19:36
tonyband I guess it will surprise operators that until now didn't nee to empty an aggregate to rename it19:37
*** gibi has joined #openstack-nova19:38
mriedemyou can rename an aggregate,19:39
mriedemjust no the az19:39
mriedem*not19:39
*** mikal has quit IRC19:39
openstackgerritKevin Rasmussen proposed openstack/nova master: created AggregateSubnetAvailabilityFilter  https://review.openstack.org/63317819:39
mriedemif they are not surprised, they were probably dealing with fallout maybe without even knowing it, i.e. rename az, resize a server and now it's back in the old az - or novalidhost19:39
mriedemdansmith: confirmed that revert resize az bug19:40
mriedemgot a test, will post with a bug shortly19:40
tonybYeah okay.19:40
dansmithmriedem: sweet19:41
dansmith"sweet"19:41
*** wolverineav has joined #openstack-nova19:45
mriedemhttps://bugs.launchpad.net/nova/+bug/181996319:45
openstackLaunchpad bug 1819963 in OpenStack Compute (nova) "Reverting a resize does not update the instance.availability_zone value to the source az" [Low,Triaged] - Assigned to Matt Riedemann (mriedem)19:45
mriedemmy test shows that the api hides the incorrect instance.availability_zone value19:46
mriedembecause of https://github.com/openstack/nova/blob/482f4fed654f384e8fb277c504a14a6407ba2e7b/nova/availability_zones.py#L179-L19419:46
mriedemmakes me wonder wtf we even have instance.availability_zone19:46
*** awaugama has quit IRC19:46
dansmithdo we have an az filter for listing instances?19:47
dansmithwe use it for metadata at least19:47
mriedemyeah we have an az filter when listing instances19:48
dansmithpresumably we use that there?19:49
dansmithobviously much of our model for everything has evolved since that was a thing though19:49
*** tbachman has joined #openstack-nova19:50
openstackgerritmelanie witt proposed openstack/nova master: WIP Use Ceph Jewel release for live migration testing  https://review.openstack.org/64315019:51
openstackgerritmelanie witt proposed openstack/nova master: WIP Use Ceph Jewel release for live migration testing  https://review.openstack.org/64315019:52
*** hongbin has quit IRC19:53
*** hongbin has joined #openstack-nova19:54
openstackgerritMatt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1819963  https://review.openstack.org/64315119:55
openstackbug 1819963 in OpenStack Compute (nova) "Reverting a resize does not update the instance.availability_zone value to the source az" [Low,Triaged] https://launchpad.net/bugs/1819963 - Assigned to Matt Riedemann (mriedem)19:55
mriedemdansmith: so i built ^ on top of my live migration functional test with azs, which may or may not be a problem19:55
openstackgerritmelanie witt proposed openstack/nova master: WIP Use Ceph Jewel release for live migration testing  https://review.openstack.org/64315019:55
mriedemand yeah i assume the GET /servers?availability_zone=foo filtering uses the instance.availability_zone value in the DB19:56
mriedemotherwise it would be crazy expensive to do that filtering in the API19:56
dansmithyeah19:56
dansmithmriedem: you should have been a lawyer19:59
dansmith"Well, your honor, as you can see I have reproduced the crime in a functional test, exhibit A, which shows that the defendant obviously committed said crime. Once sentenced, we can remove this assertion, exhibit B"20:00
mriedemi'd have to wear adult mens clothing then20:01
mriedemno thanks20:01
dansmithmaybe not if you were a lawyer in california, although that would have other downsides20:02
dansmithI hear they have silly magazine limits down there20:02
* mriedem alerts sdague20:03
dansmithheh20:05
*** dklyle has quit IRC20:09
mriedemin all seriousness, being a lawyer in MN would allow me to accept gifts of bars and hotdish from my defendents20:11
mriedemso it might be worth it20:11
*** ivve has joined #openstack-nova20:15
*** wolverineav has quit IRC20:15
*** dklyle has joined #openstack-nova20:16
*** wolverineav has joined #openstack-nova20:16
*** amodi has joined #openstack-nova20:17
melwittmriedem: what did you mean by ubuntu queens/bionic is using luminous? the version in the uca?20:18
mriedemyeah i think that's what coreycb meant20:19
mriedemwhen i asked him earlier today20:19
melwittbc I see luminous for queens xenial in the uca http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/xenial-proposed/queens/main/binary-amd64/Packages20:19
melwittok, cool20:19
coreycbmelwitt: yes and that's the same version that comes in bionic by default20:22
melwittcoreycb: ah, thank you20:22
coreycbnp20:23
openstackgerritMatt Riedemann proposed openstack/nova master: Update instance.availability_zone on revertResize  https://review.openstack.org/64315520:23
mriedemdansmith: exhibit B ^20:23
dansmiththat's an upcall no?20:23
* dansmith reads the comment20:23
dansmithoh sorry20:24
dansmithgod I'm stupid, sorry20:24
*** s10 has joined #openstack-nova20:24
mriedemi can optimize this20:25
mriedemleft a comment20:25
dansmithwas about to call your typo20:25
mriedemi know :)20:25
mriedemi could feel it20:26
dansmithso20:26
dansmithtalk me through this on your reqspec optimization20:26
dansmithlet's say...20:26
dansmithI have an instance with no specific az,20:27
*** lpetrut has joined #openstack-nova20:27
dansmithI move it to another host, and it's mid-resize20:27
dansmithor, waiting for confirm/revert I mean20:27
dansmiththe operator could change the az affiliation of that host in the middle right?20:27
mriedemof the source?20:27
dansmithof either20:27
mriedemnot the target,20:28
dansmithit wouldn't be smart, but..20:28
mriedembecause of avolkov's recent fix20:28
mriedembut they could rename the az of the source yeah20:28
dansmithbarring the source change would be hard, looking up instances that still have residue on hosts20:28
dansmithanyway,20:28
dansmithseems like maybe not worth the optimization,20:28
dansmithif it's still possible to be wrong but creates a much more obscure path to get there20:29
mriedemright his change only deals with the az that contains the instance based on instance.host https://review.openstack.org/#/c/641351/1/nova/objects/instance.py20:30
mriedemwhich would be the target at the time of the VERIFY_RESIZE status20:30
mriedemi'm fine with not trying to optimize it20:30
mriedemi could leave a comment to that effect as well if it's something we think we'll forget and care about in a year :)20:30
*** lpetrut has quit IRC20:31
dansmithI say just don't optimize it now, regardless of how you spin the comments20:31
mriedemmelwitt: you were +2 on this before https://review.openstack.org/#/c/567701/ if you can check it again, i think it was just rebased20:31
melwittmriedem: sure20:32
mriedemdansmith: ok if you're ok with it as is i can just leave it alone, or fix the typo in the commit message20:32
dansmithoh I'm not fine with the typo20:32
dansmithyou must fix that20:32
mriedemok then20:32
dansmiththere, made explicit20:32
dansmithI'll also explain why no opt I gess20:33
* dansmith notes his typing skills are on point today20:33
mriedemactually,20:34
mriedemno,20:34
dansmithmriedem: if I'm using default_availability_zone, I can change the target's affiliation with a restart right?20:34
mriedemif RequestSpec.availability_zone is not None, the server *has* to migrate within the AZ,20:34
mriedemmeaning while the server is in VERIFY_RESIZE, it's in the same az on the dest host as the source host, so the admin can't rename the az20:34
mriedemand if RequestSpec.availability_zone is None, then we'd end up updating the instance.availability_zone anyway20:35
mriedemi mean with my optimization20:35
dansmithyeah, but what about my CONF-based attack?20:36
dansmithtwo hosts, both in default_availability_zone=foo, then I can change that conf to =bar on one and restart it and you'll do the wrong thing20:36
dansmith(and instance.az will be wrong)20:36
mriedemyeah i suppose20:38
dansmithit just doesn't seem like enough optimization to worry about20:38
dansmithto me anyway20:39
openstackgerritMatt Riedemann proposed openstack/nova master: Update instance.availability_zone on revertResize  https://review.openstack.org/64315520:39
mriedemwhere is jaypipes when i want to hear someone say we should just delete availability zones as a concept from nova20:39
*** tbachman has quit IRC20:40
dansmithpfft, he'll say that about anything20:40
edleafe(uses my jaypipes imitation) Availability zones suck! Let's just get rid of them!20:40
*** gmann_afk is now known as gmann20:41
mriedem* suck! Let's just get rid of them!20:41
mriedemvolumes20:41
mriedemquotas20:41
mriedemeverything20:41
mriedem*root volumes20:41
dansmithinstances kinda suck too20:41
dansmithwe should nix those20:41
*** weshay is now known as other_guys20:46
*** other_guys is now known as other_guy20:46
openstackgerritMerged openstack/nova master: Trivialfix for help description of images_type  https://review.openstack.org/64294220:48
openstackgerritMerged openstack/nova stable/rocky: Fix an error when generating a host ID  https://review.openstack.org/64117320:48
openstackgerritMerged openstack/nova stable/rocky: Move legacy-grenade-dsvm-neutron-multinode-live-migration in-tree  https://review.openstack.org/64019120:48
openstackgerritMerged openstack/nova stable/rocky: Fix resource tracker updates during instance evacuation  https://review.openstack.org/64098220:48
*** other_guy is now known as weshay20:55
openstackgerritMerged openstack/nova master: Add missing libvirt exception during device detach  https://review.openstack.org/64148021:00
openstackgerritMerged openstack/nova stable/queens: Make service all-cells min version helper use scatter-gather  https://review.openstack.org/63759421:03
openstackgerritMerged openstack/nova master: api-ref: Add description for BDM volume_size  https://review.openstack.org/64125521:03
openstackgerritMerged openstack/nova master: docs: Fix nits in remote console guide  https://review.openstack.org/64109821:03
*** snevi has joined #openstack-nova21:04
*** IvensZambrano has quit IRC21:06
*** whoami-rajat has quit IRC21:11
*** wolverineav has quit IRC21:18
*** janki has quit IRC21:19
*** ivve has quit IRC21:20
*** wolverineav has joined #openstack-nova21:22
*** wolverineav has quit IRC21:24
*** wolverineav has joined #openstack-nova21:24
*** tbachman has joined #openstack-nova21:35
*** tbachman has quit IRC21:37
*** tbachman has joined #openstack-nova21:37
*** cdent has quit IRC21:37
*** cdent has joined #openstack-nova21:42
*** cdent has quit IRC21:42
*** tosky has joined #openstack-nova21:56
*** rcernin has joined #openstack-nova21:57
*** mikal has joined #openstack-nova21:57
*** s10 has quit IRC21:59
*** mikal has quit IRC22:03
gmannlegacy base jobs to Bionic are merged and nova-next job failing, hold the recheck until this workaround is merged (in gate) - https://review.openstack.org/#/c/643129/322:11
gmannsent on ML too.22:11
*** marst has quit IRC22:14
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: Update instance.availability_zone during live migration  https://review.openstack.org/64317322:17
*** mriedem is now known as mriedem_afk22:18
*** lifeless has quit IRC22:27
*** lifeless has joined #openstack-nova22:27
*** mikal has joined #openstack-nova22:34
fungimelwitt: oops, i missed that https://review.openstack.org/642940 had a depends-on to the demonstration patch. that commit message needs to be amended to remove the depends-on line22:45
fungimriedem_afk: dansmith: ^22:45
efriedfungi, melwitt, mriedem_afk, dansmith: I'll take care of it, stand by...22:48
openstackgerritEric Fried proposed openstack/nova master: Update nova-tox-functional-py35 parent job  https://review.openstack.org/64294022:49
fungithanks!!!22:49
fungiand sorry for missing that in review22:49
efriedfungi: done22:49
efriedI noticed that, but I thought it was intentional22:49
efriedlike, I thought the other patch was going to be non-DNM'd soon22:49
efriedor something22:50
efriedI didn't read the backstory closely at all22:50
funginope, it was entirely a demonstration. apologies for the confusion22:50
fungiwe couldn't actually show the effect of the change without a depends-on chain to an alternate universe base job22:50
fungiat least not until the actual base job change merged22:51
efriedack22:52
efriedand with that, I'm outta here o/22:52
*** mikal has quit IRC23:01
*** wolverineav has quit IRC23:03
*** slaweq has quit IRC23:05
*** sapd1 has joined #openstack-nova23:09
*** slaweq has joined #openstack-nova23:11
*** wolverineav has joined #openstack-nova23:15
*** slaweq has quit IRC23:15
*** wolverineav has quit IRC23:20
*** wolverineav has joined #openstack-nova23:22
*** tosky has quit IRC23:23
openstackgerritMerged openstack/nova master: Update instance.availability_zone during live migration  https://review.openstack.org/56770123:25
openstackgerritMerged openstack/nova master: Skip the ceph based live migration testing  https://review.openstack.org/64312223:25
*** hongbin has quit IRC23:26
*** dave-mccowan has joined #openstack-nova23:27
*** sapd1 has quit IRC23:30
*** xek_ has quit IRC23:33
*** dave-mccowan has quit IRC23:38
gmannfungi: efried we have cross-dependency to unblock the gate.  643129 cannot merge which need 642940 and vice versa.23:40
fungipossible we'll need to squash those23:41
gmannfungi: can we get this forcefully in - 642940? or i can make temp patch to unblock either one23:41
fungithey'll be properly tested if we squash the two commits together, but would need a nova-core on hand to approve the resulting revision of whichever change we squash into23:43
gmannok, let me squash first then i can catch someone alex_xu  might be coming online soon23:43
fungii'm just veeery hesitant to bypass ci for a change where it's not strictly necessary23:47
fungipretty much every time i do, i break something a job would have caught23:47
*** erlon has joined #openstack-nova23:48
openstackgerritGhanshyam Mann proposed openstack/nova master: Disable the tls-proxy in nova-next & fix nova-tox-functional-py35 parent  https://review.openstack.org/64312923:48
gmanndone ^^23:48
gmannefried: melwitt dansmith alex_xu bauzas stephenfin mriedem_afk ^^ if you are around.23:49
openstackgerritMerged openstack/nova master: Trivial: fix typo in reno  https://review.openstack.org/64313723:50
*** luksky has quit IRC23:51
*** wolverineav has quit IRC23:52
openstackgerritGhanshyam Mann proposed openstack/nova master: Migrate legacy jobs to Ubuntu Bionic  https://review.openstack.org/63901723:54
*** wolverineav has joined #openstack-nova23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!