| *** igordc has quit IRC | 00:04 | |
| *** jangutter has joined #openstack-nova | 00:06 | |
| *** samueldmq has quit IRC | 00:08 | |
| *** jangutter has quit IRC | 00:11 | |
| *** bbowen has quit IRC | 00:18 | |
| *** bbowen has joined #openstack-nova | 00:18 | |
| *** HW-Peter has joined #openstack-nova | 00:25 | |
| *** HW-Peter has quit IRC | 00:27 | |
| *** HW-Peter has joined #openstack-nova | 00:27 | |
| *** HW-Peter has quit IRC | 00:30 | |
| *** HW-Peter has joined #openstack-nova | 00:31 | |
| *** gyee has quit IRC | 00:31 | |
| *** HW_Peter has joined #openstack-nova | 00:37 | |
| *** markvoelker has quit IRC | 00:37 | |
| *** mlavalle has quit IRC | 00:47 | |
| openstackgerrit | Merged openstack/nova stable/queens: Update instance.availability_zone on revertResize https://review.openstack.org/648415 | 00:47 |
|---|---|---|
| *** mriedem has quit IRC | 00:59 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521 | 01:00 |
| openstack | bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem) | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Add instance hard delete https://review.openstack.org/650984 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | 01:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask https://review.openstack.org/638046 | 01:01 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method https://review.openstack.org/638047 | 01:01 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048 | 01:01 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268 | 01:01 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add archive_deleted_rows wrinkle to cross-cell functional test https://review.openstack.org/651650 | 01:01 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353 | 01:01 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269 | 01:01 |
| *** takashin has joined #openstack-nova | 01:08 | |
| *** nicolasbock has quit IRC | 01:09 | |
| *** ricolin has joined #openstack-nova | 01:09 | |
| *** brinzhang has joined #openstack-nova | 01:34 | |
| *** bbowen has quit IRC | 01:42 | |
| *** bbowen has joined #openstack-nova | 01:42 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Fix SynchronousThreadPoolExecutorFixture mock spec https://review.openstack.org/650171 | 01:55 |
| openstackgerrit | melanie witt proposed openstack/nova master: Use futurist.GreenThreadPoolExecutor in scatter_gather_cells https://review.openstack.org/650172 | 01:55 |
| openstackgerrit | melanie witt proposed openstack/nova master: Revert "Fix target_cell usage for scatter_gather_cells" https://review.openstack.org/653894 | 01:55 |
| *** jangutter has joined #openstack-nova | 02:07 | |
| *** jangutter has quit IRC | 02:12 | |
| *** tbachman has quit IRC | 02:21 | |
| *** mdbooth_ has joined #openstack-nova | 02:22 | |
| *** mdbooth has quit IRC | 02:23 | |
| *** dannins has joined #openstack-nova | 02:26 | |
| *** lbragstad has quit IRC | 02:37 | |
| *** tbachman has joined #openstack-nova | 02:59 | |
| *** irclogbot_2 has quit IRC | 03:05 | |
| *** edmondsw has quit IRC | 03:08 | |
| *** irclogbot_3 has joined #openstack-nova | 03:09 | |
| *** edleafe has quit IRC | 03:10 | |
| *** markvoelker has joined #openstack-nova | 03:33 | |
| *** d34dh0r53 has joined #openstack-nova | 03:36 | |
| *** brinzhang has quit IRC | 03:41 | |
| eandersson | efried, found something odd with the ironic code | 04:02 |
| eandersson | http://logs.openstack.org/39/653839/2/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/63a832a/controller/logs/screen-n-cpu.txt.gz#_Apr_19_03_21_25_243777 | 04:02 |
| eandersson | https://review.openstack.org/#/c/653839/2/nova/virt/ironic/client_wrapper.py | 04:02 |
| eandersson | If you provide a region_name, the get_endpoint does not seem to work | 04:02 |
| eandersson | It looks to be related to min_version, max_version. If I remove those options it works as intended. | 04:03 |
| *** brinzhang has joined #openstack-nova | 04:05 | |
| *** imacdonn has quit IRC | 04:08 | |
| *** jangutter has joined #openstack-nova | 04:08 | |
| *** imacdonn has joined #openstack-nova | 04:08 | |
| *** jangutter has quit IRC | 04:12 | |
| *** bhagyashris has joined #openstack-nova | 05:02 | |
| *** lpetrut has joined #openstack-nova | 05:12 | |
| *** tetsuro has joined #openstack-nova | 05:41 | |
| *** yedongcan has joined #openstack-nova | 05:49 | |
| *** jangutter has joined #openstack-nova | 06:09 | |
| *** jangutter has quit IRC | 06:13 | |
| *** whoami-rajat has joined #openstack-nova | 06:15 | |
| *** plestang has joined #openstack-nova | 06:26 | |
| *** markvoelker has quit IRC | 06:27 | |
| *** tetsuro has quit IRC | 06:28 | |
| *** lpetrut has quit IRC | 06:34 | |
| *** tetsuro has joined #openstack-nova | 06:35 | |
| *** rpittau|afk is now known as rpittau | 06:52 | |
| openstackgerrit | ya.wang proposed openstack/nova-specs master: Expose auto converge and post copy https://review.openstack.org/651681 | 06:56 |
| *** tetsuro_ has joined #openstack-nova | 07:08 | |
| *** tetsuro has quit IRC | 07:11 | |
| *** pcaruana has joined #openstack-nova | 07:12 | |
| *** tesseract has joined #openstack-nova | 07:18 | |
| *** markvoelker has joined #openstack-nova | 07:27 | |
| *** lpetrut has joined #openstack-nova | 07:34 | |
| lpetrut | Hi. Is nova already using nested resource providers for NUMA topologies? looking at the libvirt driver 'update_provider_tree' method it seems not but I may be wrong | 07:38 |
| *** liuyulong has quit IRC | 07:39 | |
| *** tetsuro_ has quit IRC | 07:48 | |
| *** tetsuro has joined #openstack-nova | 07:50 | |
| *** luksky has joined #openstack-nova | 07:52 | |
| *** slaweq has quit IRC | 07:53 | |
| *** helenafm has joined #openstack-nova | 07:53 | |
| *** tetsuro_ has joined #openstack-nova | 08:00 | |
| kashyap | cfriesen: I think you didn't read the Secure Boot spec in full. I did mention in the Work Items that we will re-use existing flavor/image properties. | 08:00 |
| *** markvoelker has quit IRC | 08:01 | |
| *** takashin has left #openstack-nova | 08:01 | |
| *** tetsuro has quit IRC | 08:04 | |
| *** jangutter has joined #openstack-nova | 08:10 | |
| *** jangutter has quit IRC | 08:14 | |
| *** tkajinam has quit IRC | 08:19 | |
| *** whoami-rajat has quit IRC | 08:24 | |
| *** tetsuro has joined #openstack-nova | 08:37 | |
| *** tetsuro_ has quit IRC | 08:39 | |
| *** whoami-rajat has joined #openstack-nova | 08:55 | |
| *** davidsha has joined #openstack-nova | 08:57 | |
| kashyap | cfriesen: Anyway, will respond to your questions on the change :-) | 08:57 |
| *** tetsuro has quit IRC | 09:02 | |
| *** luksky has quit IRC | 09:08 | |
| *** tetsuro has joined #openstack-nova | 09:12 | |
| *** luksky has joined #openstack-nova | 09:22 | |
| *** tetsuro has quit IRC | 09:23 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Specifying az when restore shelved server https://review.openstack.org/624689 | 09:38 |
| *** brinzhang has quit IRC | 09:43 | |
| openstackgerrit | ya.wang proposed openstack/nova-specs master: Expose auto converge and post copy https://review.openstack.org/651681 | 09:47 |
| openstackgerrit | Boxiang Zhu proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645520 | 09:52 |
| *** markvoelker has joined #openstack-nova | 09:58 | |
| *** bhagyashris has quit IRC | 10:09 | |
| *** pcaruana has quit IRC | 10:17 | |
| *** markvoelker has quit IRC | 10:31 | |
| *** nicolasbock has joined #openstack-nova | 10:45 | |
| *** luksky has quit IRC | 10:51 | |
| *** yan0s has joined #openstack-nova | 11:03 | |
| *** bbowen has quit IRC | 11:03 | |
| *** bbowen has joined #openstack-nova | 11:04 | |
| *** yedongcan has left #openstack-nova | 11:04 | |
| *** pvradu has joined #openstack-nova | 11:07 | |
| *** markvoelker has joined #openstack-nova | 11:29 | |
| *** pvradu has quit IRC | 11:30 | |
| *** lpetrut has quit IRC | 11:40 | |
| *** jmlowe has joined #openstack-nova | 11:41 | |
| yan0s | hello I'm having a problem with VNC in the web gui | 11:49 |
| yan0s | I can see it using the admin user | 11:49 |
| yan0s | but not with "Member" role users (also using SAML integrated login) | 11:50 |
| yan0s | error in nova-novncproxy.log is : code 400, message Bad request syntax ('\x88\x8fI¿cØJW7¹;Ø\x06¬iÜ\x0f·:Ú\x07') | 11:50 |
| *** edmondsw has joined #openstack-nova | 11:50 | |
| *** kaliya has joined #openstack-nova | 11:51 | |
| *** kaliya has quit IRC | 11:52 | |
| *** kaliya has joined #openstack-nova | 11:53 | |
| *** alex_xu has quit IRC | 11:56 | |
| *** markvoelker has quit IRC | 12:02 | |
| *** whoami-rajat has quit IRC | 12:05 | |
| *** jangutter has joined #openstack-nova | 12:08 | |
| *** jangutter has quit IRC | 12:12 | |
| *** HW-Peter has quit IRC | 12:17 | |
| *** dikonoor has joined #openstack-nova | 12:22 | |
| *** luksky has joined #openstack-nova | 12:33 | |
| *** ricolin has quit IRC | 12:41 | |
| *** lbragstad has joined #openstack-nova | 12:48 | |
| *** mriedem has joined #openstack-nova | 12:56 | |
| *** markvoelker has joined #openstack-nova | 12:58 | |
| *** jmlowe has quit IRC | 13:03 | |
| mriedem | seems like something exploded in infra over night and now everything has to be rechecked | 13:05 |
| yan0s | Actually, novnc breaks only when I launch the VM with a cloud-config file | 13:06 |
| *** dims has quit IRC | 13:21 | |
| *** efried is now known as fried_rice | 13:25 | |
| fried_rice | eandersson: I can't say I'm particularly surprised tbh. | 13:25 |
| fried_rice | do you have a way of seeing whether it broke recently? | 13:25 |
| fried_rice | mriedem: TestRPC.test_create_transport in py36 - is that the thing you were seeing yesterday? | 13:29 |
| *** edleafe has joined #openstack-nova | 13:29 | |
| mriedem | yes | 13:29 |
| mriedem | see ML | 13:29 |
| *** whoami-rajat has joined #openstack-nova | 13:31 | |
| *** markvoelker has quit IRC | 13:31 | |
| fried_rice | thanks mriedem. Recheckable or 100%? | 13:33 |
| *** jmlowe has joined #openstack-nova | 13:36 | |
| *** bbowen has quit IRC | 13:37 | |
| mriedem | i've been rechecking | 13:42 |
| mriedem | with 12 hour turnaround on ci times plus there is a gerrit maintenance scheduled today... | 13:43 |
| mriedem | looks like i should just quite early | 13:43 |
| fried_rice | mriedem: When I run this locally, I get 'oslo_config.cfg.NoSuchGroupError: no such group [oslo_messaging_notifications]' | 13:47 |
| fried_rice | Trying to find where that group is being registered... | 13:47 |
| *** dims has joined #openstack-nova | 13:48 | |
| mriedem | via oslo.messaging opts | 13:48 |
| mriedem | i think in nova.rpc | 13:48 |
| mnaser | mriedem: also, I had to turn down ovh-bhs1 (150 VMs) because it had networking issues | 13:49 |
| mnaser | and now I'm seeing a lot of RETRY_LIMIT failing jobs, which means that another provider is having issues (I suspect the other ovh region from what I was seeing) | 13:49 |
| mriedem | ah this https://github.com/openstack/nova/blob/master/nova/config.py#L50 | 13:49 |
| mriedem | mnaser: yeah i saw a lot of infra failures like that | 13:50 |
| mnaser | so we're pretty short on infra :\ | 13:50 |
| mnaser | but 100% relate on the "lets fix our gates that are on fire instead of making code pretty" comment | 13:50 |
| mriedem | mnaser: has there been something else going on all week because it's taken half a day to get a result on a change at times | 13:50 |
| mnaser | mriedem: afaik ovh has been having issues where a whole job will run and then fail to collect logs, restarting that job | 13:51 |
| mnaser | so in our case a 2-3h deploy job would run 3 times in a row and still fail | 13:51 |
| mriedem | ah | 13:51 |
| mnaser | and even then it'd re-queue after the fail so takes ages to get a node assigned again | 13:52 |
| mriedem | yeah i saw the POST_FAILURES which is generally failing to collect and publish logs | 13:52 |
| mnaser | because there's so many other things that are trying to also retry | 13:52 |
| mriedem | fried_rice: or it might be this that registers those options https://github.com/openstack/nova/blob/master/nova/config.py#L61 | 13:53 |
| mnaser | and today you're going to have an automated patch change all the references in your code to point to opendev and so | 13:53 |
| mnaser | enjoy that it in advance and hopefully that doesn't break your world too | 13:53 |
| * mriedem checks to see if there are any decent movies playing | 13:54 | |
| mriedem | which core can i pay off to look at these 2 simple changes? https://review.openstack.org/#/q/topic:bug/1823781+(status:open+OR+status:merged) | 13:58 |
| fried_rice | I'll look. But your credit score is shit with me. "The nickel is in my backpack," suuuure. | 14:00 |
| fried_rice | tonyb[m]: "Now would be a good time to start brainstorming Forum topics while some of the | 14:00 |
| fried_rice | PTG discussions are fresh. Just a couple months until the Summit and Forum in | 14:00 |
| fried_rice | Berlin." | 14:00 |
| fried_rice | You were just seeing if we were paying attention. | 14:00 |
| mriedem | fried_rice: it's still in there, i just can't ever remember to give it to you in person | 14:01 |
| mriedem | my offer to mail it still stands | 14:01 |
| fried_rice | Then it'll be, "the nickel's in the mail." I've heard that one before too. | 14:02 |
| fried_rice | I get that same NoSuchGroupError in stein and rocky. So I'm guessing there's a bug in the test setup where the test env doesn't run through the code path that registers the options if I'm just running those two tests. | 14:04 |
| mriedem | https://photos.app.goo.gl/giiB1e8RD8WztkaQ8 | 14:05 |
| mriedem | i've set a reminder | 14:05 |
| *** nicolasbock has quit IRC | 14:06 | |
| mriedem | fried_rice: yeah i get the same: $ tox -r -e py36 -- nova.tests.unit.test_rpc --until-failure | 14:06 |
| *** awalende has joined #openstack-nova | 14:07 | |
| fried_rice | so that's a red herring for our gate failure :( | 14:07 |
| *** jangutter has joined #openstack-nova | 14:09 | |
| fried_rice | If you actually pay me off, I won't be able to keep giving you crap about it. We need a different plan. | 14:12 |
| *** jangutter has quit IRC | 14:14 | |
| mriedem | what i'd like to know is what is calling oslo.config when the death spiral starts | 14:14 |
| *** nicolasbock has joined #openstack-nova | 14:18 | |
| *** dims has quit IRC | 14:20 | |
| mriedem | well i fixed that latent traceback at least :) | 14:23 |
| fried_rice | Running all of py36 locally I got these: | 14:23 |
| fried_rice | {2} nova.tests.unit.pci.test_utils.PciDeviceMatchTestCase.test_spec_extra_key [] ... inprogress | 14:23 |
| fried_rice | {7} nova.tests.unit.objects.test_instance_group.TestInstanceGroupObject.test_get_by_hint [] ... inprogress | 14:23 |
| fried_rice | oh, by the way, that ^ was on rocky | 14:24 |
| *** mlavalle has joined #openstack-nova | 14:24 | |
| fried_rice | what have we backported lately? | 14:25 |
| mriedem | shrug, lots of stuff | 14:26 |
| mriedem | what is getting an option with a namespace? https://github.com/openstack/oslo.config/blob/6.8.1/oslo_config/cfg.py#L2614 | 14:27 |
| mriedem | wtf is a config option namespace? | 14:27 |
| mriedem | stephenfin: ^? | 14:27 |
| mriedem | http://logs.openstack.org/45/649345/7/check/openstack-tox-py36/ba15c17/job-output.txt.gz#_2019-04-18_18_10_53_423952 | 14:27 |
| mriedem | that's where the stack overflow starts | 14:27 |
| fried_rice | and why am I suddenly getting | 14:28 |
| fried_rice | Error: pg_config executable not found. | 14:28 |
| fried_rice | when rebuilding my venv?? | 14:28 |
| mriedem | https://github.com/openstack/oslo.config/blob/6.8.1/oslo_config/cfg.py#L2183 | 14:28 |
| mriedem | might be due to newer psycopg2? | 14:29 |
| mriedem | are you on 18.04? | 14:29 |
| mriedem | bionic i mean | 14:29 |
| *** vishakha has joined #openstack-nova | 14:30 | |
| fried_rice | yes. Have been for three months. Apparently I suddenly needed libpq-dev | 14:30 |
| fried_rice | four months | 14:31 |
| fried_rice | time flies | 14:31 |
| fried_rice | On master I just got | 14:31 |
| fried_rice | {6} nova.tests.unit.objects.test_instance.TestRemoteInstanceObject.test_create_with_extras [] ... inprogress | 14:31 |
| fried_rice | '>' not supported between instances of 'NoneType' and 'datetime.datetime' | 14:31 |
| fried_rice | so it's not RPC I guess | 14:31 |
| mriedem | heh i can't recreate oslo_config.cfg.NoSuchGroupError: no such group [oslo_messaging_notifications] now | 14:32 |
| yan0s | So, novnc produces the error I mentioned earlier when both a key pair and a cloud-config (setting user password) are set | 14:32 |
| mriedem | fried_rice: that's what i saw yesterday when looking at this | 14:33 |
| mriedem | it's not in the logs everytime the stack overflow happens, but i know the cells 1 removal stuff touched some of that code | 14:33 |
| *** dims has joined #openstack-nova | 14:35 | |
| *** dr_gogeta86 has quit IRC | 14:41 | |
| *** awalende_ has joined #openstack-nova | 14:44 | |
| *** helenafm has quit IRC | 14:47 | |
| *** awalende has quit IRC | 14:47 | |
| *** awalende_ has quit IRC | 14:51 | |
| *** dims has quit IRC | 14:53 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Only set oslo_messaging_notifications.driver if using RPCFixture https://review.openstack.org/653954 | 14:55 |
| *** dims has joined #openstack-nova | 14:56 | |
| *** nicolasbock has quit IRC | 14:59 | |
| *** dims has quit IRC | 15:01 | |
| *** lpetrut has joined #openstack-nova | 15:04 | |
| -openstackstatus- NOTICE: Gerrit is offline for several hours starting at 15:00 UTC to perform the opendev migration; see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005011.html | 15:04 | |
| *** ChanServ changes topic to "Gerrit is offline for several hours starting at 15:00 UTC to perform the opendev migration; see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005011.html" | 15:04 | |
| *** nicolasbock has joined #openstack-nova | 15:05 | |
| *** gyee has joined #openstack-nova | 15:06 | |
| *** yan0s has quit IRC | 15:07 | |
| *** dims has joined #openstack-nova | 15:07 | |
| *** davidsha has quit IRC | 15:09 | |
| *** plestang has quit IRC | 15:34 | |
| *** dims has quit IRC | 15:35 | |
| *** tjgresha has joined #openstack-nova | 15:40 | |
| kashyap | Hmm, https://review.openstack.org/ is offline | 15:43 |
| *** boxiang has joined #openstack-nova | 15:43 | |
| dansmith | kashyap: see topic | 15:44 |
| *** dims has joined #openstack-nova | 15:44 | |
| *** mujahidali has joined #openstack-nova | 15:44 | |
| *** boxiang has quit IRC | 15:45 | |
| kashyap | dansmith: Yeah, noticed it. _Just_ when I was hitting send on a comment I saw it go down | 15:45 |
| *** boxiang has joined #openstack-nova | 15:45 | |
| kashyap | dansmith: But, I trained my muscle memory to copy it into an editor; before pasting it :-) | 15:46 |
| kashyap | dansmith: Shouldn't you be offline, too? | 15:46 |
| dansmith | no :) | 15:46 |
| dansmith | this day has no special meaning for me | 15:46 |
| kashyap | Hehe, nod. | 15:47 |
| edleafe | It *is* the 244th anniversary of the start of the American Revolution. Surely *that* has meaning for you? | 15:48 |
| *** sapd1_x has joined #openstack-nova | 15:56 | |
| mriedem | weeeee https://bugs.launchpad.net/nova/+bug/1825537 | 16:00 |
| openstack | Launchpad bug 1825537 in OpenStack Compute (nova) "finish_resize failures incorrectly revert allocations" [Medium,Triaged] | 16:00 |
| mriedem | mnaser: eandersson: ^ will add to the placement allocations are wrong woes | 16:01 |
| mriedem | i've got a recreate test locally which i'll push up once gerrit is back | 16:01 |
| *** jangutter has joined #openstack-nova | 16:10 | |
| mriedem | dansmith: interested in your opinion on the options i've laid out in https://bugs.launchpad.net/nova/+bug/1825537 for fixing it - comment 2 | 16:11 |
| openstack | Launchpad bug 1825537 in OpenStack Compute (nova) "finish_resize failures incorrectly revert allocations" [Medium,Triaged] | 16:11 |
| dansmith | mriedem: this is getting really old | 16:11 |
| dansmith | can't you do something positive to help the project? | 16:12 |
| mriedem | would you like auto-formatted code style? | 16:12 |
| mriedem | it will bring the new contribs | 16:12 |
| dansmith | yes please! | 16:12 |
| dansmith | mriedem: I'm highly allergic to changing the host/node assignment timing | 16:13 |
| dansmith | not because what we do today is good or right, but just because I'm afraid of breaking tons of other things that encode those assumptuons | 16:14 |
| dansmith | *assumptions | 16:14 |
| *** jangutter has quit IRC | 16:14 | |
| mriedem | dansmith: i.e. "I think this is probably not really an option because finish_resize has never done this on failure and we don't really know what state the instance is in" | 16:14 |
| dansmith | sure | 16:15 |
| mriedem | i don't think that's really a good option either, i just listed it for completeness | 16:15 |
| *** tesseract has quit IRC | 16:16 | |
| mriedem | an auto-revert is appealing but i very much doubt the revert code is graceful enough to handle it, b/c of baked in assumptions about the state of the world when a revert is started | 16:16 |
| mriedem | e.g. "cleaning up networking...wtf there is no networking on this host KABOOM!!!" | 16:17 |
| * dansmith nods | 16:19 | |
| fried_rice | mriedem: Did you already do the e-r thing for bug 1825435? | 16:22 |
| openstack | bug 1825435 in OpenStack Compute (nova) "TestRPC unit tests intermittently fail with "'>' not supported between instances of 'NoneType' and 'datetime.datetime'" - maybe due to "Fatal Python error: Cannot recover from stack overflow."" [High,Confirmed] https://launchpad.net/bugs/1825435 | 16:22 |
| mriedem | yeah | 16:22 |
| fried_rice | k | 16:22 |
| mriedem | http://status.openstack.org/elastic-recheck/#1825435 | 16:23 |
| *** wwriverrat has joined #openstack-nova | 16:26 | |
| mriedem | dansmith: ok option 1 fix is quick and easy and already done locally | 16:28 |
| mriedem | it's essentially just doing what confirm_resize does | 16:28 |
| dansmith | cool | 16:28 |
| *** sapd1_x has quit IRC | 16:31 | |
| *** igordc has joined #openstack-nova | 16:32 | |
| *** mgoddard has quit IRC | 16:34 | |
| *** mgoddard has joined #openstack-nova | 16:35 | |
| *** lpetrut has quit IRC | 16:41 | |
| *** rpittau is now known as rpittau|afk | 16:42 | |
| *** mujahidali has quit IRC | 16:43 | |
| *** mgoddard has quit IRC | 16:44 | |
| *** mgoddard has joined #openstack-nova | 16:45 | |
| *** igordc has quit IRC | 16:46 | |
| *** gyee has quit IRC | 16:48 | |
| *** gyee has joined #openstack-nova | 16:49 | |
| *** mriedem is now known as mriedem_lunch | 16:57 | |
| *** kaliya has quit IRC | 17:26 | |
| *** vishakha has quit IRC | 17:27 | |
| *** igordc has joined #openstack-nova | 17:28 | |
| *** jmlowe has quit IRC | 17:33 | |
| eandersson | Sweet mriedem_lunch | 18:00 |
| eandersson | fried_rice, We have this in Rocky as well. I don't think it ever worked. | 18:01 |
| eandersson | https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/hypervisor-ironic#L44 | 18:01 |
| eandersson | We don't add a region_name, so that path was never really tested. | 18:02 |
| *** jangutter has joined #openstack-nova | 18:11 | |
| *** mriedem_lunch is now known as mriedem | 18:15 | |
| *** jangutter has quit IRC | 18:15 | |
| *** jmlowe has joined #openstack-nova | 18:19 | |
| *** dims has quit IRC | 18:26 | |
| *** jaypipes_ has joined #openstack-nova | 18:28 | |
| *** dims has joined #openstack-nova | 18:29 | |
| *** jaypipes has quit IRC | 18:30 | |
| *** edmondsw has quit IRC | 18:33 | |
| *** edmondsw_ has joined #openstack-nova | 18:35 | |
| *** bryan_stephenson has joined #openstack-nova | 18:36 | |
| fried_rice | eandersson: Okay. It's nearly impossible to test with all the permutations of ksa opts, times all the permutations of service catalog and endpoint setups. Inevitable that we missed things. | 18:43 |
| eandersson | Yep | 18:43 |
| eandersson | For sure. | 18:43 |
| *** wwriverrat has quit IRC | 18:43 | |
| eandersson | I opened a PR to have it added. | 18:43 |
| fried_rice | eandersson: IIUC this is only reproducible when there's a real ironicclient listening on the other end? | 18:44 |
| eandersson | The problem is really that the real ironicclient hides this. | 18:44 |
| fried_rice | right | 18:44 |
| fried_rice | oh | 18:44 |
| fried_rice | wait, you mean the *real* real ironicclient? | 18:44 |
| eandersson | Since when you pass on a None value, the ironicclient tries to figure it out. | 18:45 |
| eandersson | Yep | 18:45 |
| fried_rice | so... where is it a problem? | 18:45 |
| eandersson | Well the problem is that ironicclient is smart, but not smart enough | 18:46 |
| eandersson | when you pass on a None endpoint it just gets the first endpoint in the catalog | 18:46 |
| eandersson | and does not take region into account | 18:47 |
| *** dims has quit IRC | 18:48 | |
| eandersson | There has been multiple bugs with this code pass, so been difficult to know exactly where things are failing | 18:48 |
| eandersson | e.g. https://github.com/openstack/python-ironicclient/commit/466be3b6568b643605d826e5aa26d9a344cc74ae | 18:49 |
| eandersson | *code path | 18:49 |
| fried_rice | I don't actually see where we're taking region into account at all. | 18:50 |
| eandersson | Yea - that was my first assumption. | 18:51 |
| fried_rice | oh, there it is. | 18:51 |
| *** dims has joined #openstack-nova | 18:51 | |
| eandersson | But it actually looks like it works. | 18:51 |
| eandersson | I assumed that > get_endpoint(region_name=bla) | 18:51 |
| fried_rice | eandersson: I'm still not understanding exactly what environment and setup it takes to make it not work, and also how it fails. | 18:51 |
| eandersson | had to look like this | 18:51 |
| eandersson | fried_rice, if you add region_name get_endpoint(..) in nova will always return None | 18:52 |
| fried_rice | no, by the time we hit get_endpoint(), region_name has already been consumed from the conf. That happens on load_adapter_from_conf_options. | 18:52 |
| eandersson | http://logs.openstack.org/39/653839/2/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/63a832a/controller/logs/screen-n-cpu.txt.gz#_Apr_19_03_21_25_243777 | 18:52 |
| eandersson | I just adding if not endpoint into the Nova code here and ran the test with region_name=regionOne set | 18:53 |
| eandersson | *added | 18:53 |
| fried_rice | And your service catalog has your ironic endpoint in regionOne? | 18:53 |
| eandersson | Yep | 18:57 |
| eandersson | Once review.openstack is up again I'll link you the changes I did for testing | 18:57 |
| fried_rice | Okay. I don't have a way to do ironic stuff locally. Is there a way to spin up the service with no nodes? | 18:59 |
| eandersson | They have a noop driver you can probably use. | 19:01 |
| *** bryan_stephenson has quit IRC | 19:04 | |
| eandersson | I was just doing something like this for testing http://paste.openstack.org/show/uANO7kLnU4NcTJrB2ywE/ | 19:07 |
| eandersson | and ironic can be set up with a super basic config to provide api only | 19:08 |
| eandersson | Since you don't actually need to create nodes to test this | 19:09 |
| eandersson | http://logs.openstack.org/39/653839/2/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/63a832a/controller/logs/etc/ironic/ironic_conf.txt.gz | 19:09 |
| fried_rice | eandersson: But you're not actually even getting to ironicclient by the time you hit the problem. | 19:15 |
| eandersson | Sorry, that was for a different issue. | 19:15 |
| fried_rice | I think you're loading the ksa auth, session, and adapter from conf... | 19:16 |
| eandersson | Just an example. | 19:16 |
| fried_rice | and then asking the adapter to get_endpoint()... | 19:16 |
| fried_rice | which is going to the service catalog looking for a matching endpoint | 19:16 |
| fried_rice | Either it doesn't find it there, or it finds it and then goes and queries that endpoint - the ironic API itself - to do the discovery. | 19:16 |
| fried_rice | One of those two steps is failing to yield resulds. | 19:16 |
| fried_rice | results. | 19:16 |
| *** bbowen has joined #openstack-nova | 19:17 | |
| fried_rice | eandersson: You have an env where you're able to reproduce this, or just in zuul? | 19:20 |
| eandersson | I do have an env yea | 19:21 |
| eandersson | I did it locally first, but as the way I have it set up is very specific to our env I wanted to confirm in zuul as well. | 19:21 |
| fried_rice | okay, can you throw some debug logs into... | 19:21 |
| eandersson | Sure | 19:21 |
| fried_rice | https://github.com/openstack/nova/blob/master/nova/virt/ironic/client_wrapper.py#L122-L133 | 19:22 |
| fried_rice | Specifically, after L122 to see if that's reached; vs at L127 to see if get_endpoint raised an exception - should be one or the other - and then after L130 to see what get_client returns. | 19:24 |
| fried_rice | eandersson: ^ | 19:24 |
| eandersson | So that is what I added into the Zuul logs | 19:24 |
| eandersson | ksa_adap.get_endpoint() returns None | 19:24 |
| eandersson | ironic.client.get_client(...) returns the endpoint | 19:25 |
| fried_rice | that's... pretty weird. | 19:25 |
| eandersson | If I remove line 120-121 it get_endpoint() returns the endpoint | 19:25 |
| fried_rice | What does the version document at your ironic endpoint say? | 19:25 |
| fried_rice | like if you just curl the root endpoint | 19:26 |
| *** jaypipes_ is now known as jaypipes | 19:26 | |
| eandersson | http://paste.openstack.org/show/eXVi99OZVrjJ6V9C496P/ | 19:31 |
| *** igordc has quit IRC | 19:36 | |
| *** bbowen has quit IRC | 19:38 | |
| fried_rice | eandersson: Oh. Ahem. That min_version/max_version thing is just wrong. No wonder it's borked. | 19:40 |
| fried_rice | Those are supposed to be major version numbers, not microversions. | 19:40 |
| fried_rice | yeah, those should just be removed entirely. | 19:40 |
| *** ttsiouts has joined #openstack-nova | 19:40 | |
| fried_rice | ye gods, who *wrote* this crap?? | 19:40 |
| *** whoami-rajat has quit IRC | 19:40 | |
| fried_rice | eandersson: Do you have a bug for this? | 19:41 |
| *** hongbin has joined #openstack-nova | 19:42 | |
| *** slaweq has joined #openstack-nova | 19:48 | |
| fried_rice | mriedem: I thought I could get rid of, "There is no script for 63 version" by rebuilding venv, wth? | 19:56 |
| mriedem | is that an api db version? | 19:58 |
| *** ttsiouts_ has joined #openstack-nova | 19:58 | |
| mriedem | do you have stale pycs or something? | 19:59 |
| eandersson | no bug yet fried_rice | 20:00 |
| fried_rice | eandersson: Okay, we're going to need one | 20:00 |
| fried_rice | eandersson: Because this fix will want to be backported | 20:00 |
| fried_rice | mriedem: Must be. | 20:00 |
| fried_rice | mriedem: seems to have cleared up now | 20:00 |
| *** ttsiouts has quit IRC | 20:00 | |
| eandersson | I'll create one in a bit | 20:01 |
| fried_rice | thanks | 20:01 |
| fried_rice | mriedem: I was on the second of those patches, 'Soft delete virtual_interfaces when instance is destroyed' -- test passes when fix is reverted. | 20:02 |
| mriedem | ok i'll look in a bit | 20:03 |
| *** ralonsoh has joined #openstack-nova | 20:08 | |
| *** jangutter has joined #openstack-nova | 20:12 | |
| fried_rice | eandersson: So this https://github.com/openstack-dev/devstack/blob/master/lib/nova_plugins/hypervisor-ironic#L44 isn't setting api_endpoint. Which means it still ought to be going through the code path that does get_ksa_adapter with the bogus min_version and max_version kwargs. Which should still be returning None. Which should still be attempting to call get_client with endpoint=None. So I don't get how adding region_name to | 20:12 |
| fried_rice | because region_name isn't in the kwargs sent to get_client | 20:13 |
| fried_rice | get_endpoint with min/max but no region in the conf somehow magically works, but adding region_name in the conf makes it not work until you remove the min/max version?? /me confused. | 20:15 |
| *** jangutter has quit IRC | 20:16 | |
| *** ralonsoh has quit IRC | 20:26 | |
| eandersson | Yea - not sure why it works without the region_name | 20:28 |
| mriedem | fried_rice: test_delete_virtual_interfaces_on_instance_destroy fails for me locally if i remove the fix | 20:30 |
| *** slaweq has quit IRC | 20:37 | |
| eandersson | I wrote a bug report but... launchpad error'd out :'( | 20:42 |
| fried_rice | eandersson: Yeah, I won't be able to post the fix until all the things come back anyway. | 20:45 |
| fried_rice | eandersson: One other data point that would be interesting: | 20:45 |
| fried_rice | The failure setup, i.e. with region_name and min_version and max_version, but change min_version from IRONIC_API_VERSION to the tuple (1, 0) | 20:46 |
| eandersson | Yea that seems to work | 20:49 |
| fried_rice | eandersson: Okay, good, theory confirmed. Soon as the dark, dark night is over, we can have this fix pushed. | 20:50 |
| fried_rice | though it's still a mystery why it doesn't break until you specify region_name. mordred, you around or chasing bunnies? | 20:51 |
| mordred | chasing finishing this opendev rollout | 20:51 |
| fried_rice | oh, you're involved in that? I'm sorry. | 20:52 |
| fried_rice | Please do carry on. | 20:52 |
| mordred | sorry - I saw the note from eandersson earlier -just havne't gotten to it. | 20:52 |
| mordred | we're having all the fun | 20:52 |
| fried_rice | yeah, I don't want to distract you from that for sure. | 20:53 |
| eandersson | btw fried_rice would be nice if we could error out if endpoint is None | 20:53 |
| *** hongbin has quit IRC | 20:53 | |
| fried_rice | eandersson: Well, if you look at the comment there, I deliberately didn't do that | 20:53 |
| eandersson | because at the moment if this gets fixed, and nova is misconfigured it will look like it's working | 20:53 |
| *** hongbin has joined #openstack-nova | 20:53 | |
| eandersson | Yea - saw your comment on that | 20:53 |
| eandersson | At the very least I believe we should log it | 20:53 |
| fried_rice | I was basically paranoid about changing the nature of legacy broken behavior. | 20:54 |
| fried_rice | Yeah, I can get behind a log for sure. | 20:54 |
| eandersson | because it can lead to some bad stuff | 20:54 |
| eandersson | or maybe we can pass on region to the ironic client | 20:54 |
| eandersson | because we are essentially telling the ironic client, hey if I fail, just make the best judgment you can.... but I will not give you all the info you need | 20:55 |
| fried_rice | eandersson: Well, I would rather rip out ironicclient altogether and use the sdk. | 20:55 |
| eandersson | I do like that idea :p | 20:55 |
| fried_rice | eandersson: https://blueprints.launchpad.net/nova/+spec/openstacksdk-in-nova | 20:56 |
| fried_rice | (I wasn't being entirely hypothetical) | 20:56 |
| eandersson | Yep - already commented on the very initial implementation :P | 20:56 |
| fried_rice | cool. I would have known that if I could have referenced the gerrit part of my extended brain. | 20:56 |
| eandersson | https://github.com/openstack/python-ironicclient/blob/master/ironicclient/client.py#L111 | 21:00 |
| eandersson | I still think we should pass on region_name to the client if we keep the current behavior with a possible endpoint = None | 21:00 |
| eandersson | That way at the least the ironicclient can make an educated decision | 21:01 |
| fried_rice | That can be done. I think I would prefer it to be a separate change though. It's clearly a different bug from the bogus min_version thing. | 21:02 |
| eandersson | For sure | 21:02 |
| fried_rice | ugh, this is such a mess | 21:03 |
| fried_rice | ironicclient is doing all this preliminary stuff that's almost (but not quite, and incorrectly) duplicating a subset of the logic in ksa itself. | 21:04 |
| eandersson | Yea - this took me hours to figure out | 21:05 |
| eandersson | Because there was actually 3 separate bugs | 21:05 |
| fried_rice | and for that matter, nova's client_wrapper is *also* doing more of that | 21:05 |
| fried_rice | mainly for backward compatibility | 21:05 |
| eandersson | https://github.com/openstack/python-ironicclient/commit/466be3b6568b643605d826e5aa26d9a344cc74ae | 21:07 |
| eandersson | This bug added A LOT of confusion for me | 21:08 |
| fried_rice | mriedem: Confirmed the test fails when the fix is removed. Something bizarre happening with cached .pyc files maybe, would explain a couple of the weird things that have befallen me today. So I'll be +A on that guy when the world comes back up. | 21:08 |
| fried_rice | eandersson: Oh yeah, that one had a knock-on effect resulting in several other bugs. | 21:08 |
| melwitt | mriedem: re: tripleo, I think it supports neither deploy nor upgrade with extracted placement based on the last I heard from lyarwood and EmelienM. this isn't going to work right now but the relevant patches are https://review.openstack.org/#/c/630644/ and https://review.openstack.org/#/q/topic:tripleo-placement-extraction+status:open and https://review.openstack.org/#/q/topic:tripleo-placement-upgrade-from-nova-placement+status:open | 21:09 |
| fried_rice | eandersson: like https://github.com/openstack/python-ironicclient/commit/ae1743d2c194c690c4d4629e51e860b5f5b84252 | 21:09 |
| melwitt | mriedem: I'm not 100% clear on whether deploy is supported in any capacity or if it's just that it can't deploy it by default yet | 21:09 |
| mriedem | melwitt: ok | 21:12 |
| melwitt | it's being worked on and supposed to ramp up soon. there's an additional person who's going to help who has been taken on the task recently | 21:16 |
| *** IvensZambrano has joined #openstack-nova | 21:19 | |
| *** IvensZambrano is now known as snevi | 21:19 | |
| imacdonn | who knows stuff about eventlet monkey-patching and WSGI? Re. the problem I reported on the ML, with nova-API not maintain the heartbeat on AMQP connections when running under uWSGI, I've discovered that the problem goes away when I remove the eventlet monkey-patching | 21:24 |
| fried_rice | imacdonn: isn't mdbooth_ our resident monkey-patching expert, and cdent our resident wsgi expert? I kinda think you might have poor luck raising the former, and I know the latter is on vacation. | 21:28 |
| melwitt | imacdonn: mdbooth_ landed a patch to change eventlet monkey-patching around in train, that you could try to see if it helps if you were up for an experiment, but I can't link it to you right now bc gerrit | 21:30 |
| melwitt | I guess I could find it in github | 21:31 |
| imacdonn | melwitt, Are you sure that didn't make Stein? i.e. this: https://github.com/openstack/nova/commit/3c5e2b0e9fac985294a949852bb8c83d4ed77e04 | 21:31 |
| imacdonn | problem started before that, and continues with that, though | 21:31 |
| melwitt | imacdonn: I'm sure. it's proposed for backport to stein but we wanted to let it bake longer and see if anyone else was having problems first. bc only RHOSP reported the issue | 21:32 |
| imacdonn | melwitt: hmm. Not finding anything beyond the above (which was definitely about "moving it around") | 21:33 |
| imacdonn | melwitt: maybe not merged to master yet? | 21:34 |
| melwitt | imacdonn: no, that's the one, it's just it's not in stein | 21:34 |
| fried_rice | definitely merged in master | 21:34 |
| melwitt | it's in train | 21:34 |
| melwitt | I was saying you could try applying that patch to see if it helps | 21:35 |
| imacdonn | hmm, that's odd .. I seemed to have that change ... which suggests either the RDO Stein packages are not actually Stein, or I'm doing something stupid\ | 21:36 |
| *** ianw_pto is now known as ianw | 21:37 | |
| imacdonn | # rpm -qf nova/monkey_patch.py | 21:37 |
| imacdonn | python2-nova-19.0.0-1.el7.noarch | 21:37 |
| melwitt | ohhh | 21:38 |
| melwitt | I bet we (redhat) backported it downstream only. but I wasn't sure we do that for RDO | 21:39 |
| fried_rice | Wow. That seems kinda crazy. | 21:39 |
| melwitt | I guess they did in this case at least bc otherwise it was totally broken for us | 21:39 |
| fried_rice | But I guess this was a special case because RH was busted. | 21:39 |
| fried_rice | yeah | 21:39 |
| fried_rice | but imacdonn you said the problem existed before this commit as well? Or is that erroneous in light of revelations above? | 21:40 |
| imacdonn | so, in any case, my problem is present both with and without that change | 21:40 |
| fried_rice | okay. | 21:40 |
| melwitt | ok, was worth a shot | 21:41 |
| melwitt | I have to run now, I'm off today because it's Spring Holiday™. see yall next week | 21:41 |
| imacdonn | k, have fun ;) | 21:41 |
| fried_rice | Pesach Tov. | 21:44 |
| fried_rice | I'm going to hit the metaphorical road as well. eandersson, I have your fix ready; please add me to that bug once you have it. | 21:51 |
| *** fried_rice is now known as efried | 21:51 | |
| imacdonn | have a good w/e, efried | 21:52 |
| efried | o/ | 21:52 |
| *** mriedem has quit IRC | 21:53 | |
| eandersson | Let me try to create it again | 21:55 |
| eandersson | efried, https://bugs.launchpad.net/nova/+bug/1825583 | 22:02 |
| openstack | Launchpad bug 1825583 in OpenStack Compute (nova) "Region name isn't respected when configuring ironic" [Undecided,New] | 22:02 |
| eandersson | I wish I would have saved the original one I wrote. | 22:02 |
| *** jangutter has joined #openstack-nova | 22:12 | |
| *** jangutter has quit IRC | 22:17 | |
| eandersson | I tried to keep the description searchable. So I didn't actually mention the min/max version stuff. | 22:18 |
| *** alex_xu has joined #openstack-nova | 22:27 | |
| alex_xu | gerrit is down? | 22:27 |
| *** snevi has quit IRC | 22:43 | |
| mnaser | alex_xu: see topic | 22:47 |
| alex_xu | mnaser: thanks | 22:48 |
| mordred | we're having a fun day | 22:51 |
| *** lbragstad has quit IRC | 23:11 | |
| *** ttsiouts_ has quit IRC | 23:22 | |
| *** ttsiouts has joined #openstack-nova | 23:37 | |
| *** luksky has quit IRC | 23:39 | |
| *** hongbin has quit IRC | 23:41 | |
| *** ttsiouts has quit IRC | 23:41 | |
| *** nicolasbock has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!