| *** ociuhandu has quit IRC | 00:17 | |
| *** ociuhandu has joined #openstack-nova | 00:17 | |
| *** TxGirlGeek has quit IRC | 00:20 | |
| *** ociuhandu has quit IRC | 00:23 | |
| mriedem | i've seen your minions in here | 00:25 |
|---|---|---|
| mriedem | but yes TPS reports are important | 00:26 |
| *** tbachman has joined #openstack-nova | 00:27 | |
| *** _mlavalle_1 has quit IRC | 00:31 | |
| *** slaweq has joined #openstack-nova | 00:34 | |
| *** ociuhandu has joined #openstack-nova | 00:36 | |
| *** slaweq has quit IRC | 00:38 | |
| *** tbachman has quit IRC | 00:41 | |
| * artom waits for the inevitable f-string injection attacks | 00:41 | |
| *** spatel has joined #openstack-nova | 00:41 | |
| mriedem | artom: i have a gift for you | 00:42 |
| mriedem | and it's not even this https://www.youtube.com/watch?v=GqY3bS6v6Wc | 00:42 |
| artom | Back pocket then? | 00:43 |
| *** macz has joined #openstack-nova | 00:44 | |
| *** ociuhandu has quit IRC | 00:45 | |
| mriedem | no, this: https://review.opendev.org/695609 | 00:46 |
| *** gyee has quit IRC | 00:46 | |
| mriedem | oh gdi | 00:47 |
| artom | I think you missed 'git add <new file>' | 00:47 |
| mriedem | yup, pushing | 00:47 |
| *** sapd1 has quit IRC | 00:47 | |
| mriedem | gerritbot must also be dead | 00:47 |
| artom | mriedem, thanks dude! Left some random thoughts after a quick read. Now must read bedtime story to daughter. | 00:54 |
| *** tosky has quit IRC | 00:56 | |
| mriedem | tell her the story about the dream where you thought you could change the image on a server during cold migrate | 00:58 |
| *** mriedem is now known as mriedem_afk | 00:58 | |
| *** abaindur has joined #openstack-nova | 01:02 | |
| *** abaindur has quit IRC | 01:03 | |
| *** abaindur has joined #openstack-nova | 01:03 | |
| *** openstackstatus has joined #openstack-nova | 01:03 | |
| *** ChanServ sets mode: +v openstackstatus | 01:03 | |
| *** Liang__ has joined #openstack-nova | 01:06 | |
| *** ociuhandu has joined #openstack-nova | 01:10 | |
| artom | mriedem_afk, but... you can? | 01:10 |
| artom | Isn't that why we went through that whole RUN_ON_REBUILD in the filters saga years ago? | 01:11 |
| artom | Or am I massively confused | 01:11 |
| artom | *facepalm* rebuild != cold migrate | 01:13 |
| *** jistr has quit IRC | 01:14 | |
| *** jistr has joined #openstack-nova | 01:15 | |
| *** abaindur has quit IRC | 01:19 | |
| *** macz has quit IRC | 01:23 | |
| *** ociuhandu has quit IRC | 01:28 | |
| *** brinzhang has joined #openstack-nova | 01:28 | |
| *** brinzhang has quit IRC | 01:29 | |
| *** brinzhang has joined #openstack-nova | 01:29 | |
| *** macz has joined #openstack-nova | 01:31 | |
| *** mriedem_afk has quit IRC | 01:40 | |
| *** brinzhang_ has joined #openstack-nova | 01:45 | |
| *** nanzha has joined #openstack-nova | 01:46 | |
| *** abaindur has joined #openstack-nova | 01:48 | |
| *** brinzhang has quit IRC | 01:48 | |
| *** abaindur has quit IRC | 01:48 | |
| *** abaindur has joined #openstack-nova | 01:49 | |
| *** tbachman has joined #openstack-nova | 01:49 | |
| *** zhanglong has joined #openstack-nova | 01:55 | |
| *** brault has joined #openstack-nova | 01:56 | |
| *** igordc has quit IRC | 01:57 | |
| *** openstackgerrit has joined #openstack-nova | 02:00 | |
| openstackgerrit | Wei Hui proposed openstack/nova master: bugfix device_type=type-PCI passthrough failed https://review.opendev.org/695416 | 02:00 |
| *** macz has quit IRC | 02:09 | |
| *** brinzhang has joined #openstack-nova | 02:10 | |
| *** brinzhang has quit IRC | 02:12 | |
| *** brinzhang has joined #openstack-nova | 02:12 | |
| *** brinzhang_ has quit IRC | 02:13 | |
| *** abaindur has quit IRC | 02:13 | |
| *** brinzhang has quit IRC | 02:14 | |
| *** dviroel has quit IRC | 02:21 | |
| *** ociuhandu has joined #openstack-nova | 02:28 | |
| *** zhanglong has quit IRC | 02:31 | |
| *** spatel has quit IRC | 02:37 | |
| *** ociuhandu has quit IRC | 02:39 | |
| *** tetsuro has joined #openstack-nova | 02:49 | |
| *** ricolin has joined #openstack-nova | 02:50 | |
| *** tetsuro has quit IRC | 03:11 | |
| *** ociuhandu has joined #openstack-nova | 03:14 | |
| *** awalende has joined #openstack-nova | 03:15 | |
| *** awalende has quit IRC | 03:19 | |
| *** ociuhandu has quit IRC | 03:19 | |
| *** macz has joined #openstack-nova | 03:30 | |
| *** yedongcan has joined #openstack-nova | 03:32 | |
| *** macz has quit IRC | 03:34 | |
| *** udesale has joined #openstack-nova | 03:41 | |
| *** tetsuro has joined #openstack-nova | 03:58 | |
| *** bhagyashris has joined #openstack-nova | 04:02 | |
| *** mkrai has joined #openstack-nova | 04:04 | |
| *** chenhaw has joined #openstack-nova | 04:05 | |
| *** shilpasd has quit IRC | 04:15 | |
| *** larainema has joined #openstack-nova | 04:23 | |
| *** ratailor has joined #openstack-nova | 04:30 | |
| *** tetsuro has quit IRC | 04:31 | |
| *** tetsuro has joined #openstack-nova | 04:35 | |
| *** tetsuro has quit IRC | 04:36 | |
| *** tetsuro_ has joined #openstack-nova | 04:36 | |
| *** tetsuro_ has quit IRC | 04:36 | |
| *** tetsuro has joined #openstack-nova | 04:38 | |
| *** dave-mccowan has quit IRC | 04:58 | |
| *** bhagyashris has quit IRC | 05:01 | |
| *** zhanglong has joined #openstack-nova | 05:09 | |
| *** bhagyashris has joined #openstack-nova | 05:09 | |
| *** ratailor has quit IRC | 05:13 | |
| *** ratailor has joined #openstack-nova | 05:18 | |
| *** ociuhandu has joined #openstack-nova | 05:30 | |
| *** ociuhandu has quit IRC | 05:36 | |
| *** macz has joined #openstack-nova | 05:53 | |
| *** zhanglong has quit IRC | 05:54 | |
| *** zhanglong has joined #openstack-nova | 05:56 | |
| *** macz has quit IRC | 05:58 | |
| *** links has joined #openstack-nova | 05:58 | |
| *** Luzi has joined #openstack-nova | 06:06 | |
| *** zbr has joined #openstack-nova | 06:07 | |
| *** zbr|ooo has quit IRC | 06:08 | |
| *** sridharg has joined #openstack-nova | 06:14 | |
| *** mmethot has quit IRC | 06:18 | |
| *** zhanglong has quit IRC | 06:20 | |
| *** zhanglong has joined #openstack-nova | 06:23 | |
| *** slaweq has joined #openstack-nova | 06:23 | |
| *** ociuhandu has joined #openstack-nova | 06:25 | |
| *** slaweq has quit IRC | 06:25 | |
| *** janki has joined #openstack-nova | 06:31 | |
| *** zhanglong has quit IRC | 06:34 | |
| openstackgerrit | Merged openstack/nova stable/stein: Don't delete compute node, when deleting service other than nova-compute https://review.opendev.org/695381 | 06:39 |
| *** zhanglong has joined #openstack-nova | 06:48 | |
| *** ociuhandu has quit IRC | 06:52 | |
| *** awalende has joined #openstack-nova | 06:58 | |
| *** awalende has quit IRC | 07:02 | |
| *** ociuhandu has joined #openstack-nova | 07:03 | |
| *** zhanglong has quit IRC | 07:14 | |
| *** macz has joined #openstack-nova | 07:25 | |
| *** macz has quit IRC | 07:29 | |
| *** ociuhandu has quit IRC | 07:32 | |
| *** slaweq has joined #openstack-nova | 07:32 | |
| *** janki has quit IRC | 07:52 | |
| openstackgerrit | Merged openstack/nova master: ZVM: Implement update_provider_tree https://review.opendev.org/694897 | 08:00 |
| *** slaweq has quit IRC | 08:07 | |
| *** ccamacho has joined #openstack-nova | 08:09 | |
| *** jawad_axd has joined #openstack-nova | 08:11 | |
| *** ccamacho has quit IRC | 08:13 | |
| *** tosky has joined #openstack-nova | 08:15 | |
| *** pcaruana has joined #openstack-nova | 08:21 | |
| *** zhanglong has joined #openstack-nova | 08:22 | |
| *** ccamacho has joined #openstack-nova | 08:27 | |
| *** awalende has joined #openstack-nova | 08:30 | |
| *** tesseract has joined #openstack-nova | 08:35 | |
| *** tetsuro has quit IRC | 08:36 | |
| stephenfin | artom: you is welcome. | 08:43 |
| *** yaawang has quit IRC | 08:49 | |
| *** yaawang has joined #openstack-nova | 08:49 | |
| *** rpittau|afk is now known as rpittau | 08:58 | |
| *** mlycka has joined #openstack-nova | 09:08 | |
| *** sridharg has quit IRC | 09:10 | |
| stephenfin | bauzas: Wanna take a look at https://review.opendev.org/#/c/686809/ today? | 09:12 |
| *** nanzha has quit IRC | 09:19 | |
| *** nanzha has joined #openstack-nova | 09:19 | |
| openstackgerrit | Alexandre arents proposed openstack/nova master: Do not update root_device_name during guest config https://review.opendev.org/670000 | 09:19 |
| *** zhanglong has quit IRC | 09:22 | |
| *** zhanglong has joined #openstack-nova | 09:23 | |
| * bauzas clicks | 09:23 | |
| bauzas | hah | 09:23 |
| bauzas | stephenfin: yeah you weren't around while we discussed between gibi, johnthetubaguy and others and when we agreed on a microversion as a signal | 09:24 |
| stephenfin | I've already replied to johnthetubaguy. It doesn't make any sense | 09:24 |
| bauzas | stephenfin: but your comment makes sense indeed | 09:24 |
| bauzas | I'm honestly not really opionated | 09:25 |
| *** martinkennelly has joined #openstack-nova | 09:25 | |
| stephenfin | Yeah, look at the other examples I referenced. This is what we've done in the past and I think it's correct | 09:25 |
| bauzas | let's just move the ball running and | 09:25 |
| bauzas | see what we can signal at the end | 09:25 |
| stephenfin | coolness :) | 09:25 |
| *** akshayhavile has joined #openstack-nova | 09:26 | |
| stephenfin | would be good to drag mriedem in when we get there since he's done this stuff in the past too | 09:26 |
| *** ralonsoh has joined #openstack-nova | 09:26 | |
| *** akshayhavile has quit IRC | 09:28 | |
| * stephenfin wonders how he missed that entire conversation yesterday /o\ | 09:28 | |
| bauzas | stephenfin: you were at the gym or feeding too much stuff in your mouth | 09:29 |
| stephenfin | that does sound like me alright | 09:29 |
| bauzas | btw. are we around Thanksgiving ? | 09:30 |
| bauzas | channel was quiet yesterday | 09:30 |
| gibi | sean-k-mooney: thanks for the bandwidth bug report. efried: thanks for the summary mail. I will dive in | 09:35 |
| *** tssurya has joined #openstack-nova | 09:36 | |
| *** Liang__ has quit IRC | 09:36 | |
| mlycka | Hello. What's the opinion on taking another stab https://specs.openstack.org/openstack/nova-specs/specs/mitaka/approved/expose-quiesce-unquiesce-api.html? | 09:37 |
| stephenfin | bauzas: I am | 09:38 |
| stephenfin | if that's who you're asking? | 09:38 |
| bauzas | stephenfin: you are what ? | 09:38 |
| bauzas | I was wondering whether it was Thanksgiving period | 09:39 |
| bauzas | (disclaimer, it's absolutely not a thing here) | 09:39 |
| stephenfin | Ah, I read that as are _you_ around | 09:39 |
| stephenfin | whoops | 09:39 |
| bauzas | but I can check the calender | 09:39 |
| bauzas | nope, it's next week | 09:39 |
| bauzas | sorted. | 09:39 |
| * bauzas notes asking google is way faster than asking colleagues over IRC | 09:40 | |
| * bauzas should consider AI as a good way to communicate | 09:40 | |
| *** ricolin has quit IRC | 09:43 | |
| *** derekh has joined #openstack-nova | 09:44 | |
| *** macz has joined #openstack-nova | 09:45 | |
| *** yedongcan has quit IRC | 09:48 | |
| openstackgerrit | Alexandre arents proposed openstack/nova master: Do not update root_device_name during guest config https://review.opendev.org/670000 | 09:48 |
| *** macz has quit IRC | 09:50 | |
| *** chenhaw has quit IRC | 09:50 | |
| *** yedongcan has joined #openstack-nova | 09:51 | |
| *** chenhaw has joined #openstack-nova | 09:59 | |
| *** zhanglong has quit IRC | 10:00 | |
| *** ociuhandu has joined #openstack-nova | 10:02 | |
| *** ociuhandu has quit IRC | 10:07 | |
| *** chenhaw has quit IRC | 10:09 | |
| *** chenhaw has joined #openstack-nova | 10:09 | |
| *** tssurya has quit IRC | 10:13 | |
| *** chenhaw has quit IRC | 10:15 | |
| *** dtantsur|afk is now known as dtantsur | 10:22 | |
| *** chenhaw has joined #openstack-nova | 10:38 | |
| *** nanzha has quit IRC | 10:38 | |
| *** nanzha has joined #openstack-nova | 10:40 | |
| *** chenhaw has quit IRC | 10:45 | |
| *** mkrai has quit IRC | 10:46 | |
| *** ratailor has quit IRC | 10:49 | |
| *** ociuhandu has joined #openstack-nova | 10:50 | |
| *** ociuhandu has quit IRC | 10:52 | |
| *** ociuhandu has joined #openstack-nova | 10:52 | |
| *** ociuhandu has quit IRC | 10:53 | |
| *** ociuhandu has joined #openstack-nova | 11:01 | |
| *** ociuhandu has quit IRC | 11:08 | |
| *** nanzha has quit IRC | 11:10 | |
| *** udesale has quit IRC | 11:10 | |
| *** nanzha has joined #openstack-nova | 11:24 | |
| *** luksky has joined #openstack-nova | 11:34 | |
| *** bhagyashris has quit IRC | 11:35 | |
| *** tbachman has quit IRC | 11:39 | |
| *** sridharg has joined #openstack-nova | 11:47 | |
| *** tssurya has joined #openstack-nova | 11:48 | |
| sean-k-mooney | melwitt: the non-ironic case is because they were following our scale in docs which direct them to delete the compute service however they skipped step 1 which is remove all instances form the node first. and i guess ensure that all allocations are removed before doing the service delete | 11:59 |
| sean-k-mooney | so they delete the compute service with vms running then the bug got filed to help them fix that and prefereabel keep the vms by migrating them. | 12:00 |
| sean-k-mooney | so its a self inflicted failure | 12:00 |
| *** rcernin has quit IRC | 12:01 | |
| sean-k-mooney | efried: gibi: by the way the downstream bug is just the bugzilla for validating support for bandwidth support. i think on ebug is being filed for triplo but we have not filed one for nova yet because i wanted to disucss what we found first to make sure what the behavior should be. | 12:04 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Include removal of ephemeral backing files in the image cache manager https://review.opendev.org/689422 | 12:04 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: Revert "nova shared storage: rbd is always shared storage" https://review.opendev.org/682523 | 12:04 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Rename _is_storage_shared_with to _is_path_shared_with https://review.opendev.org/693337 | 12:04 |
| gibi | sean-k-mooney: agree. Right now we need to find a backportable solution (see my mail on the ML about the problems with efried's proposal) | 12:05 |
| gibi | sean-k-mooney: for nova I will create a small spec to add the node uuid filtering capability to os-hypervisors | 12:05 |
| gibi | sean-k-mooney: for neutron I would like to wait for the neutron folks to react | 12:06 |
| sean-k-mooney | efried: gibi this is the downstream bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=1463838 fyi but let continue upstream | 12:06 |
| openstack | bugzilla.redhat.com bug 1463838 in openstack-nova "[RFE] Guaranteed minimum bandwidth" [Urgent,On_qa] - Assigned to smooney | 12:06 |
| sean-k-mooney | gibi: am what we need ist to be able to query it by host or hypervior host | 12:06 |
| sean-k-mooney | not by uuid | 12:06 |
| gibi | sean-k-mooney: my bad, yes, we need to be able to filter by service host name | 12:07 |
| gibi | to get the node uuid | 12:07 |
| sean-k-mooney | yep | 12:07 |
| sean-k-mooney | we might be able to do that already we can certenly just do a hypervior show | 12:08 |
| sean-k-mooney | and i think that will accpet the host but i need to check the hypervioru api | 12:08 |
| sean-k-mooney | im still catching up on the mail trhead so ill comment more once i have read it | 12:08 |
| sean-k-mooney | in terms of backport it would need to be backported to train and stein if we did backport a fix | 12:09 |
| sean-k-mooney | but for the sepecific case we found customer should not hit it | 12:10 |
| sean-k-mooney | the might unitll we fix the standalone mode but in a normal tipleo deployment it set the hostname to match the value it sets in the host config option | 12:11 |
| sean-k-mooney | in standalone mode the host os is installed manually and therfor cloud-init does not run and that step is not done | 12:12 |
| gibi | as a side track I filed bug as nova compute fails if the CONF.host is changed after initial deployment https://bugs.launchpad.net/nova/+bug/1853587 | 12:12 |
| openstack | Launchpad bug 1853587 in OpenStack Compute (nova) "nova compute fails to update inventory in placement after changing [DEFAULT]/host" [Undecided,New] | 12:12 |
| sean-k-mooney | ya that is basically the same issue not entirely but close enough | 12:14 |
| sean-k-mooney | gibi: ok so we use the host option to create teh compute service recored but the hypervior_hostname for the placement name | 12:16 |
| sean-k-mooney | so changing CONF.host after the fact causes us to creat a new service record and uuid | 12:17 |
| gibi | sean-k-mooney: yepp, which is logical if we have to support more than one hypervisors per service | 12:17 |
| sean-k-mooney | but then when we try to create the RP we get a colission on the hyperviour_hostname right | 12:17 |
| gibi | sean-k-mooney: right | 12:17 |
| gibi | sean-k-mooney: I guess if you need to change CONF.host then first have to cleanup the old service and placement | 12:17 |
| sean-k-mooney | that or if we dont find the service record by host check if its theyer by hypervior hostname and then upstat the host field? | 12:18 |
| sean-k-mooney | not sure that would be right in all cases however | 12:19 |
| sean-k-mooney | draning deleting and recreating seams safer | 12:19 |
| sean-k-mooney | funnily enough we have a customer that want to be able to do a backup and restor of the compute node and potentially chang ethe hostname since there hostnames are determine by the location of the server and they may need to restore to a different host | 12:20 |
| gibi | that sounds fun | 12:21 |
| sean-k-mooney | if they were not still running a pre placement version of openstack they would hit this issue | 12:21 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Take an instance.uuid lock when rebooting https://review.opendev.org/673463 | 12:22 |
| *** mdbooth has quit IRC | 12:26 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: virt: Provide block_device_info during a rescue https://review.opendev.org/270287 | 12:27 |
| sean-k-mooney | gibi: did i mention i also traced how neuton learns what hostname to use in the frist place | 12:27 |
| *** mdbooth has joined #openstack-nova | 12:28 | |
| gibi | sean-k-mooney: I think you did and I also looked at the code with the help of lajoskaton this morning | 12:28 |
| sean-k-mooney | its via the agent report | 12:28 |
| openstackgerrit | Merged openstack/nova master: Specify what RPs _ensure_resource_provider collects https://review.opendev.org/695429 | 12:28 |
| openstackgerrit | Merged openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests https://review.opendev.org/684345 | 12:28 |
| sean-k-mooney | so to use the actual hostname we would have to extend the report with a hypervior hostname field | 12:29 |
| sean-k-mooney | then the neutron server could use that if its present | 12:29 |
| gibi | sean-k-mooney: exaclty, which is an RPC change which might not be backportable | 12:29 |
| gibi | hence my reply asking neutron folks about it | 12:30 |
| sean-k-mooney | its not an object change | 12:30 |
| sean-k-mooney | its an unversioned dict of strings field | 12:30 |
| sean-k-mooney | at least it used to be | 12:30 |
| gibi | sean-k-mooney: neutron server still need to have a way to handle old RPC reports from not upgraded agents | 12:30 |
| sean-k-mooney | ya which could be to either first try to look up in placmene by its existing host value. | 12:31 |
| sean-k-mooney | and if it fails then fallback to the full hyperviour list and cache the result | 12:32 |
| *** mkrai has joined #openstack-nova | 12:32 | |
| sean-k-mooney | normally the CONF.host and hypervior_host will be the same in the libvirt case | 12:32 |
| *** nanzha has quit IRC | 12:38 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Use admin neutron client to query ports for binding https://review.opendev.org/694665 | 12:39 |
| gibi | elod: ^^ fixed your comment | 12:40 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova stable/stein: Use admin neutron client to see if instance has qos ports https://review.opendev.org/694668 | 12:40 |
| elod | gibi: thx, looking | 12:41 |
| *** nanzha has joined #openstack-nova | 12:42 | |
| *** slaweq has joined #openstack-nova | 12:42 | |
| gibi | oooo ma csovet is fognak ma vagni | 12:43 |
| gibi | az nagyon meno lesz | 12:43 |
| *** larainema has quit IRC | 12:50 | |
| *** macz has joined #openstack-nova | 12:55 | |
| gibi | wrong window obviously | 12:58 |
| *** macz has quit IRC | 12:59 | |
| *** priteau has joined #openstack-nova | 13:06 | |
| *** Luzi has quit IRC | 13:09 | |
| *** slaweq has quit IRC | 13:20 | |
| *** awalende has quit IRC | 13:22 | |
| *** tbachman has joined #openstack-nova | 13:22 | |
| *** awalende has joined #openstack-nova | 13:22 | |
| *** yedongcan has left #openstack-nova | 13:23 | |
| *** awalende_ has joined #openstack-nova | 13:27 | |
| *** awalende has quit IRC | 13:27 | |
| *** awalende_ has quit IRC | 13:31 | |
| *** priteau has quit IRC | 13:33 | |
| *** eharney has quit IRC | 13:37 | |
| *** jawad_axd has quit IRC | 13:42 | |
| *** jawad_axd has joined #openstack-nova | 13:43 | |
| openstackgerrit | Merged openstack/nova master: Remove 'os-security-group-default-rules' REST API https://review.opendev.org/686807 | 13:44 |
| *** mgoddard has quit IRC | 13:44 | |
| openstackgerrit | Merged openstack/nova master: nova-net: Remove unused '*_default_rules' security group DB APIs https://review.opendev.org/686808 | 13:44 |
| openstackgerrit | Merged openstack/nova master: Remove (most) '/os-networks' REST APIs https://review.opendev.org/686809 | 13:44 |
| *** mgoddard has joined #openstack-nova | 13:45 | |
| *** jawad_axd has quit IRC | 13:48 | |
| *** sridharg has quit IRC | 13:54 | |
| *** jaosorior has joined #openstack-nova | 14:00 | |
| *** ociuhandu has joined #openstack-nova | 14:08 | |
| *** awalende has joined #openstack-nova | 14:09 | |
| openstackgerrit | Merged openstack/nova stable/train: Add functional recreate test for bug 1852610 https://review.opendev.org/694544 | 14:10 |
| openstack | bug 1852610 in OpenStack Compute (nova) train "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Undecided,In progress] https://launchpad.net/bugs/1852610 - Assigned to Matt Riedemann (mriedem) | 14:10 |
| *** mmethot has joined #openstack-nova | 14:12 | |
| *** awalende has quit IRC | 14:14 | |
| *** ociuhandu has quit IRC | 14:16 | |
| *** damien_r has joined #openstack-nova | 14:17 | |
| *** damien_r has quit IRC | 14:18 | |
| *** damien_r has joined #openstack-nova | 14:19 | |
| *** mriedem has joined #openstack-nova | 14:19 | |
| mlycka | Hi again, quick question, is there no 'quiesced' state for VMs? | 14:35 |
| *** jawad_axd has joined #openstack-nova | 14:36 | |
| *** nweinber__ has joined #openstack-nova | 14:36 | |
| *** jawad_axd has quit IRC | 14:40 | |
| johnthetubaguy | mlycka: I don't think there is, but there is a state of snapshotting, or something like that, during which time it might be quiesced | 14:42 |
| mriedem | i think that's when creating a snapshot of an active volume-backed instance | 14:44 |
| mriedem | and only if the driver and image support it | 14:44 |
| mriedem | i.e. qemu guest agent | 14:44 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3213 | 14:44 |
| mlycka | Right right...I'm just looking through an older patch which added a new state as part of exposing quiescing/unquiescing in the API. I'm guessing that would still have to be done for that functionality to be implemented. | 14:44 |
| *** links has quit IRC | 14:45 | |
| mriedem | i don't see that we even set the task_state on the instance while creating a volume-backed snapshot, which seems bad | 14:46 |
| mriedem | means you could like migrate the server while creating the snapshot | 14:47 |
| mriedem | https://github.com/openstack/nova/blob/991d675675c1c6bb87a2b9d19327e2b4473f6c0b/nova/compute/task_states.py#L34 | 14:47 |
| mriedem | we uh don't even have a different server status during a snapshot https://docs.openstack.org/api-guide/compute/server_concepts.html | 14:48 |
| *** mns96 has joined #openstack-nova | 14:49 | |
| mriedem | anyway, that'd be an extremely latent issue | 14:49 |
| openstackgerrit | Merged openstack/nova stable/train: Add functional recreate revert resize test for bug 1852610 https://review.opendev.org/694545 | 14:50 |
| openstack | bug 1852610 in OpenStack Compute (nova) train "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Undecided,In progress] https://launchpad.net/bugs/1852610 - Assigned to Matt Riedemann (mriedem) | 14:50 |
| mlycka | mriedem: How do you mean latent? | 14:51 |
| *** eharney has joined #openstack-nova | 14:54 | |
| mriedem | meaning if it's a bug, it's always been there | 14:56 |
| *** mkrai has quit IRC | 14:56 | |
| mriedem | i just posted some of this to the ML since i think i come across this stuff like every year or so and wonder the same things | 14:57 |
| mlycka | Sorry, ML? | 14:57 |
| mriedem | mailing list | 14:58 |
| mlycka | Right, thanks | 14:58 |
| mriedem | http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011073.html | 14:58 |
| melwitt | sean-k-mooney: ah, thanks for the context | 14:58 |
| *** ratailor has joined #openstack-nova | 14:59 | |
| stephenfin | anyone want to hit this for me so I can keep the remove-nova-network train moving? https://review.opendev.org/#/c/686811/7 | 14:59 |
| mriedem | stephenfin: looking, | 15:01 |
| mriedem | stephenfin: you want to rebase your remove os-consoles change? then i can re-approve | 15:01 |
| *** ociuhandu has joined #openstack-nova | 15:04 | |
| mriedem | stephenfin: question https://review.opendev.org/#/c/686811/7/nova/tests/functional/wsgi/test_interfaces.py@a44 | 15:08 |
| mlycka | mriedem: Shouldn't quiescing have a state separate from snapshot though? | 15:10 |
| jroll | efried: dansmith: I'm back from an offsite thing and will catch up on vtpm discussion today, thanks for the pings | 15:10 |
| mriedem | stephenfin: nevermind, i answered my own question | 15:11 |
| mriedem | mlycka: if anything i think it would be a task_state as part of an overall snapshot "status" | 15:12 |
| mriedem | so if we had a SNAPSHOTTING status in the API, the task_state would be "quiescing" | 15:12 |
| mriedem | note that image-backed snapshot has 4 separate task_states | 15:12 |
| mlycka | mriedem: I'm sort of coming from an angle here and I'm still finding my feet with the nova state machine, so bear with me here: I'm looking at introducing the (un)quiesce operations into nova API for use by other entities (Cinder in my case). Would that not point to a quiesce state outside of the existing snapshot process? | 15:15 |
| mriedem | quiesce isn't a terminal state of a server though | 15:17 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support service_host filtering for os-hypervisor API https://review.opendev.org/695708 | 15:17 |
| mriedem | so you want to do something initiated in cinder and if the volume is attached to a server you want to call nova's rest api and tell nova to quiesce the instance, right? | 15:17 |
| stephenfin | mriedem: in a meeting but I'll rebase that soon as I'm done, yup | 15:17 |
| stephenfin | and thanks | 15:17 |
| mlycka | mriedem: Correct. | 15:17 |
| mriedem | mlycka: ok i think the api for that would be the os-server-external-events api https://docs.openstack.org/api-ref/compute/#create-external-events-os-server-external-events | 15:18 |
| mriedem | which cinder already uses when extending the size of a volume attached to a server | 15:18 |
| mriedem | it's still a compute api microversion change to change the schema of that api, but all of the plumbing already exists in the api layer, you just would need to implement the calls when the event is 'quiesce' or whatever | 15:19 |
| mlycka | Goodness me, I didn't even realize this existed | 15:19 |
| mriedem | i'm assuming you've seen the old huawei spec that was trying to expose quiesce/unquiesce out of the API for normal users? | 15:19 |
| mriedem | because that didn't get traction | 15:20 |
| mlycka | Yep, tried asking around earlier precisely for that reason but didn't get a respond | 15:20 |
| mriedem | working it into os-server-external-events as part of a larger orchestrated operation initiated in cinder is probably the way to go | 15:20 |
| mlycka | *response | 15:20 |
| mriedem | so just model it on the volume-extended event | 15:20 |
| mlycka | Well thank you kindly, I'll look into it | 15:21 |
| *** jawad_axd has joined #openstack-nova | 15:21 | |
| mriedem | mlycka: it will require a spec https://specs.openstack.org/openstack/nova-specs/readme.html | 15:21 |
| mlycka | I was just going to say | 15:21 |
| mriedem | so before implementing a ton of code and then saying "here it is!" | 15:21 |
| mriedem | i'd say get started on the spec to get initial feedback on the problem statement and the proposed change | 15:21 |
| mlycka | Right, I just need(ed) to get a feel for the way it currently operates and why it hit a brick wall previously. | 15:22 |
| mriedem | i can't remember all the details about that spec but i think they also wanted it to take a list of servers or something | 15:22 |
| mriedem | anyway, what i've said above is likely much more doable | 15:23 |
| mlycka | Yeah, I think that spec was more about snapshotting/quiescing multi-vm setups in a predictable order. | 15:23 |
| mlycka | Yep, gotcha'. I'll poke around the code for a bit and come up with a spec. Thanks. | 15:24 |
| mriedem | np | 15:24 |
| * mriedem takes the day off after completing one good deed | 15:24 | |
| *** ociuhandu has quit IRC | 15:25 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support service_host filtering for os-hypervisor API https://review.opendev.org/695708 | 15:27 |
| mriedem | stephenfin: what is neutron-network? https://review.opendev.org/#/c/686812/7/releasenotes/notes/remove-nova-network-c02953ba72a1795d.yaml is that something you're cooking up for nova v4? | 15:27 |
| stephenfin | damn it | 15:28 |
| stephenfin | I'll draft a follow-up once I'm done here | 15:28 |
| *** slaweq has joined #openstack-nova | 15:28 | |
| ayoung | once we merge Nova Network, we should merge Authentication in to Nova and decommision Keystone | 15:32 |
| *** slaweq has quit IRC | 15:32 | |
| *** ccamacho has quit IRC | 15:32 | |
| mriedem | stephenfin: while you're collecting FUPs https://review.opendev.org/#/c/686810/7/api-ref/source/os-tenant-network.inc@2 | 15:37 |
| mriedem | stephenfin: gibi: bauzas: dug a bit too deep here it looks like https://review.opendev.org/#/c/686810/7 | 15:40 |
| *** ayoung has quit IRC | 15:40 | |
| bauzas | dang | 15:40 |
| bauzas | we broke the world again. | 15:40 |
| gibi | ack | 15:41 |
| bauzas | who pushes it out of the gate ? | 15:41 |
| bauzas | stephenfin: ^ | 15:41 |
| stephenfin | we broke something? | 15:41 |
| stephenfin | huh? | 15:41 |
| bauzas | stephenfin: not yet (c) | 15:42 |
| bauzas | but mriedem provided (again) a good comment | 15:42 |
| bauzas | and we don't want the change to merge now | 15:42 |
| bauzas | https://review.opendev.org/#/c/686810/7 | 15:42 |
| mriedem | the GETs were not removed | 15:43 |
| stephenfin | ah, I see now | 15:43 |
| mriedem | but you removed them in docs, policy and tests | 15:43 |
| stephenfin | Go me \o/ | 15:44 |
| *** ratailor has quit IRC | 15:47 | |
| * mriedem fires off his armpits | 15:47 | |
| mriedem | if you're going to rebase, might as well fix that release note later in the series, | 15:47 |
| mriedem | and also a question here https://review.opendev.org/#/c/686813/8/nova/cmd/manage.py@a2673 | 15:48 |
| melwitt | why do we want to keep the GETs? | 15:49 |
| mriedem | they are proxies for neutron and work | 15:49 |
| mriedem | i mean, they'll work once nova-network is gone | 15:49 |
| melwitt | oh ok. I thought we didn't want to keep proxies | 15:50 |
| mriedem | the GETs are capped at 2.35 | 15:50 |
| mriedem | iow, proxies have already been deprecated for a long time | 15:50 |
| mriedem | newton i think | 15:50 |
| *** damien_r has quit IRC | 15:52 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove '/os-tenant-networks' REST API https://review.opendev.org/686810 | 15:52 |
| mriedem | pulled from the gate ^ | 15:52 |
| melwitt | oh right... but that wouldn't mean that all get/post/put stay and return http gone? I guess I never knew the detail behind how that was implemented | 15:52 |
| stephenfin | okay, think I know why I removed that - there's no functional test for it | 15:53 |
| *** mlavalle has joined #openstack-nova | 15:53 | |
| stephenfin | so it didn't exist as far as I, in functional test fixup mode, was concerned | 15:54 |
| *** slaweq has joined #openstack-nova | 15:54 | |
| mriedem | melwitt: can you rephrase? | 15:54 |
| mriedem | if a thing works after nova-net is gone but was just deprecated b/c it's a proxy, no we won't remove it, because it still works | 15:54 |
| mriedem | we're not removing our glance or volume proxies here either | 15:55 |
| melwitt | sorry, like in the 2.35 deprecation, were all apis except GET removed? or did all apis stay and change to return http gone, in version 2.35 | 15:55 |
| mriedem | they return 404 | 15:55 |
| mriedem | deprecation = 404 for us, gone = 410 | 15:55 |
| melwitt | and GET does not return 404? | 15:55 |
| mriedem | but you can still hit the proxies at 2.1 | 15:55 |
| mriedem | not below the deprecation microversion it shouldn't - not b/c of deprecation i mean; sure you could get 404 for trying to show details on a neutron network that doesn't exist | 15:56 |
| melwitt | just trying to parse why removing POST is ok but not GET | 15:56 |
| melwitt | should they all stay? | 15:56 |
| melwitt | *shouldn't | 15:57 |
| mriedem | POST is only implemented for nova-net | 15:57 |
| mriedem | nova-net is going to be gone | 15:57 |
| melwitt | ohhhh | 15:57 |
| mriedem | so it's a 410 | 15:57 |
| mriedem | if an API is only implemented for nova-net, it will be 410 | 15:57 |
| melwitt | yeah, ok. I see | 15:57 |
| melwitt | it all makes now | 15:58 |
| melwitt | make sense now, aaarrrggghh | 15:58 |
| stephenfin | mriedem: question on https://review.opendev.org/#/c/686810/7/api-ref/source/os-tenant-network.inc@2 | 15:58 |
| stephenfin | does that actually make sense, since some of the APIs have been removed but others have not | 15:59 |
| stephenfin | ? | 15:59 |
| * stephenfin just refreshed. Ignore me | 16:01 | |
| *** luksky has quit IRC | 16:04 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Filter hypervisors by service host https://review.opendev.org/695716 | 16:05 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove '/os-tenant-networks' REST API https://review.opendev.org/686810 | 16:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'USE_NEUTRON' from functional tests https://review.opendev.org/686811 | 16:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota https://review.opendev.org/686812 | 16:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove nova-manage network, floating commands https://review.opendev.org/686813 | 16:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove associate, disassociate network APIs https://review.opendev.org/686814 | 16:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-dhcpbridge' binary https://review.opendev.org/686815 | 16:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-network' binary https://review.opendev.org/686816 | 16:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: docs: Blast most references to nova-network https://review.opendev.org/686817 | 16:07 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP https://review.opendev.org/686818 | 16:07 |
| stephenfin | melwitt, gibi: fixed that silly typo in the reno if you want to re +2 https://review.opendev.org/686812 | 16:08 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support service_host filtering for os-hypervisor API https://review.opendev.org/695708 | 16:08 |
| *** jawad_axd has quit IRC | 16:08 | |
| *** gyee has joined #openstack-nova | 16:09 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'os-consoles' API https://review.opendev.org/687907 | 16:09 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'nova-console' service, 'console' RPC API https://review.opendev.org/687908 | 16:09 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'nova-xvpvncproxy' https://review.opendev.org/687909 | 16:09 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova-specs master: Filter hypervisors by service host https://review.opendev.org/695716 | 16:10 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Support service_host filtering for os-hypervisor API https://review.opendev.org/695708 | 16:10 |
| * bauzas bails early for the weekend | 16:12 | |
| bauzas | stephenfin: ask others for your series, sorry | 16:13 |
| stephenfin | have a good weekend o/ | 16:13 |
| melwitt | stephenfin: why did you update to say "which has been removed"? it's not going to removed until the end of the series right? I guess it makes sense to write it as thought it's already happened though since it will be by the time anyone reads that | 16:13 |
| melwitt | *though | 16:13 |
| stephenfin | yeah, I'm writing from the perspective of the end of the series | 16:14 |
| melwitt | yeah, ok that makes sense | 16:14 |
| stephenfin | and in reality, we removed nova-network support when we dropped cells v1 support https://github.com/openstack/nova/blob/master/nova/cmd/network.py#L46 | 16:14 |
| melwitt | oh yeah, I didn't think about that | 16:15 |
| gibi | stephenfin: sorry, I have to take that next week | 16:15 |
| stephenfin | gibi: 'sall good (y) | 16:15 |
| * gibi heads to a pub | 16:15 | |
| gibi | have a nice weekend folks! | 16:16 |
| melwitt | bye, happy weekend o/ | 16:16 |
| mriedem | stephenfin: a couple of thoughts/questions in the bottom change https://review.opendev.org/#/c/686810/ | 16:27 |
| stephenfin | looking | 16:27 |
| mriedem | i don't know if we have precedent for 410ing some methods in a route but not all | 16:27 |
| *** macz has joined #openstack-nova | 16:28 | |
| stephenfin | why would it be 500? | 16:28 |
| mriedem | i was thinking if you hit something that's not neutron, but i guess if you upgrade and the nova-network binary is literally gone, then yeah i guess that's not possible | 16:29 |
| mriedem | b/c you can only have neutron or you can't upgrade | 16:29 |
| *** tssurya has quit IRC | 16:29 | |
| stephenfin | yeah, the nova-network service literally can't be started since we dropped the cells v1 code | 16:29 |
| stephenfin | so people have to be using neutron in Train, never mind Ussuri | 16:30 |
| *** ricolin has joined #openstack-nova | 16:32 | |
| *** slaweq has quit IRC | 16:37 | |
| *** jmlowe has joined #openstack-nova | 16:41 | |
| mriedem | should update the reno here https://review.opendev.org/#/c/686812/8/releasenotes/notes/remove-nova-network-c02953ba72a1795d.yaml | 16:42 |
| stephenfin | damn it | 16:44 |
| stephenfin | yup | 16:44 |
| *** priteau has joined #openstack-nova | 16:44 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota https://review.opendev.org/686812 | 16:44 |
| stephenfin | done | 16:45 |
| mriedem | sorry one more in there - you forgot os-quota-class-sets in the reno | 16:48 |
| mriedem | the quota apis are super fun! | 16:48 |
| mriedem | because there are 100 of them | 16:48 |
| melwitt | this patch is a stats machine | 16:49 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota https://review.opendev.org/686812 | 16:49 |
| stephenfin | Do I get a prize if I win? | 16:50 |
| stephenfin | Can someone "win"? | 16:50 |
| stephenfin | I want the plush tiger | 16:50 |
| *** nanzha has quit IRC | 16:50 | |
| melwitt | haha | 16:50 |
| melwitt | I think we're all winning cause we're talking about quotas | 16:51 |
| melwitt | right?? | 16:51 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove nova-manage network, floating commands https://review.opendev.org/686813 | 16:52 |
| stephenfin | so long as you don't get more "winning" than you're paying for | 16:52 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: nova-net: Remove associate, disassociate network APIs https://review.opendev.org/686814 | 16:53 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: nova-net: Remove 'nova-dhcpbridge' binary https://review.opendev.org/686815 | 16:53 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: nova-net: Remove 'nova-network' binary https://review.opendev.org/686816 | 16:53 |
| melwitt | badum csh! | 16:53 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: docs: Blast most references to nova-network https://review.opendev.org/686817 | 16:53 |
| melwitt | man, I really can't type anything today | 16:53 |
| *** slaweq has joined #openstack-nova | 16:53 | |
| mriedem | have a kit kat | 16:54 |
| melwitt | hm, I could. still have some halloween candy | 16:54 |
| efried | stephenfin: did you see the conversation yesterday with johnthetubaguy about adding a "signal microversion" so clients can discover that nova-net is gone? | 16:55 |
| stephenfin | efried: I did. mriedem is probably the person you want to discuss that with, tbh | 16:55 |
| mriedem | -1 | 16:56 |
| efried | oh? | 16:56 |
| mriedem | oh. | 16:56 |
| mriedem | yeah. | 16:56 |
| *** mlycka has quit IRC | 16:57 | |
| stephenfin | I realise why he suggested it but I'm not sure about the idea of using a microversion when we can't actually provide the previous behavior on old microversions | 16:57 |
| mriedem | the deprecation microversions were the signals | 16:57 |
| stephenfin | we didn't do it for things like os-cells, os-fping, etc. | 16:57 |
| mriedem | if you want to add a microversion for each api we've removed now, you're talking about all of these https://docs.openstack.org/api-ref/compute/#obsolete-apis | 16:57 |
| mriedem | so -1 | 16:57 |
| efried | johnthetubaguy: ^ | 16:58 |
| efried | you have five minutes to respond | 16:58 |
| *** slaweq has quit IRC | 16:58 | |
| *** rchurch has quit IRC | 16:58 | |
| mriedem | i mean, i'm sure we can deal with someone that shows up and is like, "WTF HAPPENED TO CLOUDPIPE?!" | 16:58 |
| melwitt | the good old days | 16:59 |
| mriedem | https://docs.openstack.org/nova/latest/contributor/api.html#removing-deprecated-apis | 17:00 |
| mriedem | i think i wrote that after we did the first 410s | 17:00 |
| *** rpittau is now known as rpittau|afk | 17:01 | |
| efried | dig | 17:01 |
| *** rchurch has joined #openstack-nova | 17:01 | |
| openstackgerrit | Merged openstack/nova stable/train: Block deleting compute services with in-progress migrations https://review.opendev.org/694546 | 17:02 |
| openstackgerrit | Merged openstack/nova stable/train: Improve metadata server performance with large security groups https://review.opendev.org/694409 | 17:02 |
| *** jaosorior has quit IRC | 17:04 | |
| efried | mriedem: I got confused where it was suggested that GET /os-hypervisors*?service_host=CONF.host would ever return more than one entry. | 17:05 |
| efried | It would still return a list, but it would have (at most) one entry, right? | 17:05 |
| johnthetubaguy | mriedem: OK, I buy that. If you see the deprecation micro version, expect to be disappointed with HTTP Gone | 17:05 |
| mriedem | WRONG | 17:05 |
| efried | even for ironic | 17:05 |
| mriedem | efried: compute service : compute node (hypervisor) is 1:M with ironic | 17:06 |
| mriedem | please to be enjoying https://bugs.launchpad.net/nova/+bug/1852446 | 17:06 |
| openstack | Launchpad bug 1852446 in OpenStack Compute (nova) "Hypervisors in nova - no subpage details for ironic" [Undecided,New] | 17:06 |
| gregwork | is it expected that when an instance is booted into a non nova az, that volumes created by cinder in the nova AZ (successfully as part of the stack create) will fail to attach to the instance? | 17:06 |
| gregwork | also cinder doesnt seem to know about the az's i created in nova | 17:06 |
| gregwork | trying to pass availability_zone: mynovaAZ returns an unknown az error | 17:06 |
| mriedem | gregwork: see cross_az_attach in https://docs.openstack.org/nova/latest/admin/availability-zones.html | 17:07 |
| efried | mriedem: ack | 17:07 |
| gregwork | oh interesting | 17:07 |
| * gregwork configures | 17:07 | |
| mriedem | gregwork: also https://www.openstack.org/videos/summits/vancouver-2018/curse-your-bones-availability-zones-1 | 17:08 |
| mriedem | before you go too deep down the AZ rabbit hole you might want to watch that | 17:08 |
| gregwork | i was at that summit, i wish i attended that session | 17:08 |
| gregwork | ill check it out | 17:08 |
| gregwork | wait cross_az_attach defaults to true | 17:09 |
| gregwork | at least thats how it looks? | 17:09 |
| gregwork | "By default there is no availability zone restruction on volume attach." | 17:09 |
| mriedem | correct | 17:09 |
| * gregwork confused | 17:10 | |
| gregwork | i see in cinder that the volumes i wanted to attach were created, but they just didnt attach | 17:10 |
| gregwork | also heat didnt throw an error | 17:10 |
| mriedem | read https://docs.openstack.org/nova/latest/admin/availability-zones.html#resource-affinity | 17:10 |
| *** jmlowe has quit IRC | 17:10 | |
| gregwork | mriedem: i read that but since my cross_az_attach is unset (defaulted to true) that doesnt explain why the volume silently didnt attach. it seems to list the caveats with setting it to false | 17:12 |
| *** mns96 has quit IRC | 17:15 | |
| mriedem | are you talking about attaching volumes to an existing server or boot from volume where nova creates the volumes and attaches them to the server being created? | 17:17 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Resolve (most) flake8 3.x issues https://review.opendev.org/695732 | 17:17 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Switch to flake8 3.x https://review.opendev.org/695733 | 17:17 |
| *** ociuhandu has joined #openstack-nova | 17:17 | |
| mriedem | if cross_az_attach=True (default) then nova won't specify any AZ when creating volumes during boot from volume, | 17:19 |
| mriedem | if the attach is failing, it's likely due to something with the setup/storage backend etc | 17:19 |
| mriedem | so you'd have to dig into that | 17:19 |
| *** dtantsur is now known as dtantsur|afk | 17:21 | |
| *** ricolin has quit IRC | 17:21 | |
| * stephenfin -> 🍻 | 17:21 | |
| stephenfin | have a good weekend | 17:21 |
| *** awalende has joined #openstack-nova | 17:23 | |
| openstackgerrit | Merged openstack/nova stable/train: Added openssh-client into bindep https://review.opendev.org/691808 | 17:24 |
| *** awalende has quit IRC | 17:27 | |
| *** Guest24639 has joined #openstack-nova | 17:32 | |
| *** Guest24639 is now known as mgagne_ | 17:34 | |
| *** tesseract has quit IRC | 17:35 | |
| *** jaosorior has joined #openstack-nova | 17:44 | |
| openstackgerrit | Merged openstack/nova stable/stein: Use admin neutron client to query ports for binding https://review.opendev.org/694665 | 17:51 |
| openstackgerrit | Merged openstack/nova master: Abort live-migration during instance_init https://review.opendev.org/678016 | 17:51 |
| *** ociuhandu has quit IRC | 17:54 | |
| *** ociuhandu has joined #openstack-nova | 17:54 | |
| gregwork | mriedem: so you were right, it was something failing with cinder, however (and this is probably a bug) the volume attachment would fail because nova running "multipathd show status" failed (multipathd was not started on the compute node). the issue is that neither heat, nor nova, nor manually doing it in horizon caught the exception. | 17:59 |
| gregwork | it was only when i looked at nova-compute.log on the compute node that i saw the traceback | 17:59 |
| gregwork | im using queens so maybe this is fixed upstream | 17:59 |
| gregwork | just took the request and happily said it was done | 18:00 |
| *** ociuhandu has quit IRC | 18:00 | |
| *** derekh has quit IRC | 18:00 | |
| *** martinkennelly has quit IRC | 18:01 | |
| *** bnemec is now known as beekneemech | 18:04 | |
| *** mgariepy has quit IRC | 18:07 | |
| *** mgariepy has joined #openstack-nova | 18:10 | |
| sean-k-mooney | well on one hand managing the lifecycle of multipathd is out of scope of nova | 18:13 |
| sean-k-mooney | im not sure if os-brick would have detechted the fact that multipathd was not running or if nova would to be able to raise it and im not sure if we would want to expose that to a non admin if it did | 18:15 |
| sean-k-mooney | heat cant know this unless cinder/nova told it as heat has no acess to the compute node | 18:15 |
| gregwork | sean-k-mooney: well from the end user experience take horizon for example.. doing volumes -> manage attachments and picking my instance .. that should throw an error | 18:15 |
| gregwork | but it doesnt, it accepts the request and nothing happens | 18:16 |
| gregwork | a user should be informed what they asked for failed at least | 18:16 |
| gregwork | maybe not the precise details | 18:16 |
| sean-k-mooney | well it shoudl go to an error state or not be marked as attached unless it completes | 18:16 |
| * gregwork nods | 18:17 | |
| gregwork | but it doesnt which is why im mentioning it here | 18:17 |
| sean-k-mooney | so nova could do two things at that point | 18:17 |
| sean-k-mooney | it could put the instance in an error state | 18:17 |
| sean-k-mooney | or it could role back the attachment | 18:17 |
| gregwork | (to be clear it doesnt mark it as attached, it just returns an unmodified horizon screen) | 18:17 |
| sean-k-mooney | ok well that is different | 18:18 |
| sean-k-mooney | what is the volume status | 18:18 |
| gregwork | available | 18:18 |
| gregwork | it doesnt go into an error state | 18:18 |
| sean-k-mooney | then it sound like its doing the right thing | 18:18 |
| gregwork | its like you didnt do anything at all | 18:18 |
| *** cmurphy is now known as cmorpheus | 18:18 | |
| sean-k-mooney | yes because it tried to attach and failed | 18:19 |
| melwitt | did you check the instance actions api? :P | 18:19 |
| sean-k-mooney | i was just going to mention that | 18:19 |
| melwitt | I won! | 18:19 |
| melwitt | jk | 18:19 |
| sean-k-mooney | that is where the error if it was going to be reported would likely end up | 18:19 |
| gregwork | in cinder or nova ? | 18:19 |
| *** priteau has quit IRC | 18:19 | |
| melwitt | nova | 18:19 |
| melwitt | I don't know how it looks on horizon, but it's like 'openstack server event list' on the openstackclient | 18:20 |
| gregwork | so i cant openstack server event list <that instance> because we fixed it just now (configuring multipathd) and redeployed the stack | 18:21 |
| sean-k-mooney | in horizon there is the action log for the instace | 18:21 |
| gregwork | where would it be in the infra | 18:21 |
| sean-k-mooney | when you click into it | 18:21 |
| gregwork | compute or control plane | 18:21 |
| gregwork | the logs you are looking for | 18:22 |
| sean-k-mooney | gregwork: if you have not deleted the instnace it would still be in the instance event log | 18:22 |
| melwitt | by redeployed do you mean wiped the database? yeah, what sean-k-mooney said | 18:22 |
| gregwork | it was deleted when the stack was cleaned up | 18:22 |
| gregwork | so all i have are the overcloud logs on the various nodes | 18:23 |
| melwitt | I thought you could still see for deleted no? because one of the actions will be 'deleted' | 18:23 |
| melwitt | that's how you can check who deleted an instance and what time, for example | 18:23 |
| sean-k-mooney | if you have not done a db archive/purge it should still be there | 18:23 |
| *** igordc has joined #openstack-nova | 18:23 | |
| gregwork | id need the uuid of the instance wouldnt i ? the instances were redployed from a templated name | 18:23 |
| gregwork | or is there a list all events | 18:23 |
| sean-k-mooney | yes you would | 18:24 |
| melwitt | you need uuid | 18:25 |
| sean-k-mooney | you would be hitting this api endpoint https://docs.openstack.org/api-ref/compute/?&expanded=list-actions-for-server-detail#list-actions-for-server | 18:25 |
| sean-k-mooney | speaking of which is the volume attachmet api call blocking or async https://docs.openstack.org/api-ref/compute/?&expanded=attach-a-volume-to-an-instance-detail#attach-a-volume-to-an-instance | 18:26 |
| sean-k-mooney | it returns a 200 so i assume its blocking but i expected it to be async | 18:27 |
| *** irclogbot_1 has quit IRC | 18:27 | |
| melwitt | gregwork: ok, well, just fyi the "action log" 'openstack server event list' is a place to see failed actions that did not actually touch the instance. and there's a new change proposed to add more detail to errors that go in there, to aid in communication to the user https://review.opendev.org/694428 | 18:27 |
| efried | melwitt or mriedem or dansmith: Could I please get +A on three simple test-only tech debt reductions (johnthetubaguy just +2ed 'em so they're freshhhh): https://review.opendev.org/#/q/topic:use-base-test-code+(status:open+OR+status:merged) | 18:28 |
| efried | johnthetubaguy: thanks for that btw | 18:28 |
| melwitt | gregwork: this came up recently with a change in behavior in the resize and migration APIs too http://lists.openstack.org/pipermail/openstack-discuss/2019-November/011040.html | 18:28 |
| sean-k-mooney | melwitt: ya although in that context that is because we changed it to async so you have to check the action log going forward | 18:29 |
| sean-k-mooney | melwitt: if attach is currently blocking the we woudl expect the error to be returned there | 18:30 |
| *** ralonsoh has quit IRC | 18:30 | |
| melwitt | yeah. I'm just saying it brought the action log to my attention as a thing to use more regularly | 18:30 |
| sean-k-mooney | if its async then it would only be in the action log | 18:30 |
| sean-k-mooney | yep | 18:30 |
| melwitt | because prior to that, the only time I've seen action log used is when you're like "who deleted this instance and on what date" | 18:30 |
| *** irclogbot_2 has joined #openstack-nova | 18:31 | |
| sean-k-mooney | did that come up on the mailing list | 18:31 |
| sean-k-mooney | the fact the userid was missing or somthing like that | 18:31 |
| melwitt | and I know, some other clouds use it on a regular basis, just in yahoo when I was there, they didn't. so maybe other people don't | 18:31 |
| gregwork | so there is an instance id in the python traceback of nova-compute.log | 18:31 |
| gregwork | failed to attach xxx to for id | 18:32 |
| gregwork | let me try that | 18:32 |
| gregwork | im guessing the instance id in this line is not actually the instance uuid ? | 18:34 |
| gregwork | 2019-11-22 12:26:56.077 8 ERROR nova.compute.manager [req-a25aba4e-952f-431e-bb85-1e2a45f62e3b d16f2bc26e91f8221042c91c61d0c4f7e7ae57a09507d54809f0279e3d6ee29e f4e6856a85b14ecc99b494159f28e425 - 424dfc884bcd46e7915e62495d6b569c 424dfc884bcd46e7915e62495d6b569c] [instance: 1e65006f-58ee-470d-9111-522d5c63094d] Failed to attach 176cca17-3f02-4fcd-884d-a4bd74ad5943 at /dev/vdb: ProcessExecutionError: Unexpected | 18:34 |
| gregwork | error while running command. | 18:34 |
| gregwork | doing an openstack server even list 1e65006f-58ee-470d-9111-522d5c63094d | 18:34 |
| gregwork | says its unknown | 18:34 |
| gregwork | *event | 18:35 |
| sean-k-mooney | am you might need to pass a --deleted flag | 18:35 |
| sean-k-mooney | if we support that | 18:35 |
| gregwork | there is no --deleted flag | 18:36 |
| gregwork | at least not in my osc | 18:36 |
| sean-k-mooney | looking at the api i dont see one either | 18:36 |
| gregwork | melwitt: seemed to think this was a thing, hence the "show when something was deleted" | 18:36 |
| melwitt | yeah, I know it used to. that was the one use I saw in yahoo. let me take a quick look | 18:37 |
| melwitt | ah ok | 18:37 |
| melwitt | you have to do openstack --os-compute-api-version 2.21 openstack server event list | 18:38 |
| sean-k-mooney | i assume if do an openstack server show with that uuid i assume it give you nothing | 18:38 |
| melwitt | apparently ability to show deleted was added in 2.21 version of the api | 18:38 |
| melwitt | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-event.html#server-event-list | 18:38 |
| gregwork | is that api available in queens ? | 18:39 |
| sean-k-mooney | oh yes that is mentioned in the api ref too | 18:39 |
| melwitt | gregwork: yes https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id19 | 18:39 |
| melwitt | added in mitaka | 18:39 |
| gregwork | $ openstack --os-compute-api-version 2.21 | 18:41 |
| gregwork | (openstack) server event list 1e65006f-58ee-470d-9111-522d5c63094d | 18:41 |
| gregwork | No server with a name or ID of '1e65006f-58ee-470d-9111-522d5c63094d' exists. | 18:41 |
| melwitt | sean-k-mooney: I dunno, I see userid when I do openstack server event list abd23967-2dd5-42ea-b96b-a226ee07ad44 --long | 18:42 |
| melwitt | I'm gonna try on my devstack | 18:42 |
| sean-k-mooney | melwitt: yes but it was added i think in stien or rocky | 18:42 |
| sean-k-mooney | i think specificly for the how deleted my vm case you mentioned | 18:43 |
| melwitt | sigh, wtf it's not working for me either | 18:43 |
| melwitt | what am I missing here... | 18:44 |
| sean-k-mooney | same "openstack --os-compute-api-version 2.21 --os-cloud openstack server event list cef12383-3f43-44cf-be52-218a4da30c0a" fails | 18:45 |
| melwitt | gregwork: ok well the trusty legacy client works nova instance-action-list d7446288-688b-4a44-be78-008aa1ca524c | 18:45 |
| melwitt | sean-k-mooney: I bet it's some underlying osc thing where it can't get deleted servers. I don't know. have to dig into what's going on there | 18:46 |
| gregwork | | create | req-b76ac906-148b-497a-8d87-72a65be367bd | - | 2019-11-22T17:26:28.000000 | 2019-11-22T17:26:52.000000 | | 18:46 |
| gregwork | | attach_volume | req-a25aba4e-952f-431e-bb85-1e2a45f62e3b | Error | 2019-11-22T17:26:54.000000 | 2019-11-22T17:26:56.000000 | | 18:46 |
| melwitt | HALLELUJAH | 18:46 |
| gregwork | so there is an error being observed but nothing seems to react to it | 18:47 |
| gregwork | horizon/heat all just whistle by | 18:47 |
| gregwork | "volume error? cool story bro" | 18:47 |
| melwitt | yup. because the instance is still in a working state, it's not put into ERROR | 18:47 |
| melwitt | lol | 18:48 |
| melwitt | well, | 18:48 |
| melwitt | as you have noticed there's no task/status api and if there are any they're not unified, | 18:48 |
| melwitt | so our only way of communicating errors for asynchronous operations is basically ERROR state and 'server events' | 18:49 |
| sean-k-mooney | ya i would guess heat did not check the responce | 18:49 |
| sean-k-mooney | i think attach is blocking so it should have been return to heat | 18:49 |
| melwitt | yeah, I mean, heat could probably do something. but I don't know that much about heat | 18:49 |
| gregwork | the heat behavior is it sits there | 18:50 |
| gregwork | no errors, just waiting | 18:50 |
| gregwork | it eventually times out when its auth token expires | 18:50 |
| melwitt | we don't put the instance into ERROR because it's in a perfectly healthy state, it's just the attach didn't work, which sucks, but it doesn't mean the instance is broken or unusable. so that's why we don't do ERROR state. other than that, the API is async so return code will be 202 ACCEPTED but it can obviously fail after that point after the request goes async | 18:51 |
| sean-k-mooney | melwitt: which api is async? | 18:51 |
| melwitt | sean-k-mooney: is attach not async? | 18:52 |
| sean-k-mooney | its respocne code is listed as 200 | 18:52 |
| sean-k-mooney | which would imply not | 18:52 |
| melwitt | I assumed it must be based on the behavior. gregwork did you call volume attach or something else? | 18:52 |
| sean-k-mooney | https://docs.openstack.org/api-ref/compute/?expanded=attach-a-volume-to-an-instance-detail#attach-a-volume-to-an-instance | 18:53 |
| gregwork | i create a data volume with os::cinder::volume then attach it using ::volumeattach | 18:53 |
| sean-k-mooney | gregwork: are you attaching via nova or cinder | 18:53 |
| gregwork | cinder | 18:53 |
| sean-k-mooney | oh heat | 18:53 |
| melwitt | yeah it shows as a cast | 18:53 |
| gregwork | via heat | 18:53 |
| melwitt | attach_volume is a cast (async) | 18:53 |
| gregwork | i mean i assume os::cinder::herpderp means its talking to cinder ? | 18:53 |
| gregwork | (heat that is) | 18:54 |
| sean-k-mooney | the we are returning the wrong responce code it should be a 202 as you said | 18:54 |
| melwitt | fun | 18:54 |
| gregwork | i try and bring only quality bugs to these channels :) | 18:54 |
| melwitt | THANKS gregwork | 18:55 |
| gregwork | heh np :P | 18:55 |
| melwitt | sigh, yeah that sucks. I dunno why that's not a 202 | 18:55 |
| melwitt | gregwork: I think probably it's (heat) creating the volume in cinder and then calls nova to attach | 18:56 |
| sean-k-mooney | i guess its calling https://docs.openstack.org/api-ref/block-storage/v3/index.html?expanded=attach-volume-to-a-server-detail#attach-volume-to-a-server | 18:56 |
| sean-k-mooney | that one does return a 202 | 18:56 |
| melwitt | oh what? I didn't know that cinder had an api to attach to instance. it must just call us | 18:56 |
| melwitt | I don't remember any of this stuff | 18:57 |
| sean-k-mooney | all the proxy apis lol | 18:57 |
| melwitt | can we achieve a full mesh proxy to one another | 18:57 |
| sean-k-mooney | that is the dream | 18:57 |
| sean-k-mooney | also i grab the curl command form osc via --debug | 18:58 |
| sean-k-mooney | http://paste.openstack.org/show/786587/ | 18:58 |
| sean-k-mooney | i adde a valid token and that is what we get form the nova api | 18:58 |
| melwitt | yeah but you need to do that with the instance actions api | 18:59 |
| melwitt | it works fine in the legacy client | 18:59 |
| melwitt | so there's something buggy with how osc is calling instance actions api | 18:59 |
| sean-k-mooney | right so osc is first hitting the /server endpoing for some reason | 18:59 |
| melwitt | oh, yeah. that was my assumption that it does something like that across the board | 18:59 |
| melwitt | which will break nova commands that are supposed to work on deleted things | 19:00 |
| sean-k-mooney | right so its makeing an extra api call before trying to call the events api | 19:00 |
| melwitt | right | 19:00 |
| sean-k-mooney | but it does not pass the --delete flag or query arg | 19:00 |
| sean-k-mooney | so that fails and it never does the thing we asked | 19:00 |
| melwitt | I guess the bug fix might be to see if that extra call can be removed selectively for the server event commands | 19:00 |
| melwitt | right | 19:00 |
| melwitt | there might already be a storyboard open for this issue. I can't imagine we're the first to run into it | 19:01 |
| sean-k-mooney | when was the last time you tried to get server event? | 19:01 |
| sean-k-mooney | but ya it might already be open | 19:02 |
| sean-k-mooney | i would not be surpised if it was not however | 19:02 |
| melwitt | me? never (with osc). but I know that (was it huawei?) were making a lot of use of the apis. maybe their end users not using it through osc though. I dunno | 19:02 |
| melwitt | I'll chase it up though bc this is going to need to be a normal part of users workflow | 19:04 |
| *** eharney has quit IRC | 19:04 | |
| sean-k-mooney | oh we added user and projec to migration last cylce not instance actions but we did mention the isntace action as an alternitive in the spec | 19:05 |
| sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/train/approved/add-user-id-field-to-the-migrations-table.html | 19:05 |
| melwitt | it's already in instance actions. in osc --long | 19:06 |
| melwitt | I saw it | 19:06 |
| sean-k-mooney | ya it is | 19:07 |
| sean-k-mooney | i was just getting confused with that spec | 19:07 |
| melwitt | ok | 19:07 |
| sean-k-mooney | where we added to migration and said the alternitve was to find the instnace action | 19:08 |
| melwitt | ah | 19:08 |
| sean-k-mooney | have i mentioned lately i hate trying to find things in storyboard | 19:08 |
| melwitt | man I hate storyboard | 19:08 |
| *** jaosorior has quit IRC | 19:08 | |
| melwitt | the search is a lot worse than launchpad, it doesn't have per stable branch tracking ability | 19:09 |
| sean-k-mooney | i have set the list option to 1000 result and im now using ctrl-f in my brower instead | 19:09 |
| melwitt | launchpad forever | 19:09 |
| sean-k-mooney | as far as u can tell there isnt a story open | 19:10 |
| melwitt | ok. actually let me look for mriedem's mega etherpad of osc issues, I wonder if it's on there | 19:10 |
| sean-k-mooney | this seams related https://storyboard.openstack.org/#!/story/2006761 but its sperate | 19:11 |
| efried | I'm sure if you asked, they would say you can totally track stable branches, by creating a bunch of separate tags and tasks and ... | 19:11 |
| efried | ...in ways that are specific to your project... | 19:11 |
| melwitt | yeah not specifically mentioned https://etherpad.openstack.org/p/compute-api-microversion-gap-in-osc | 19:12 |
| sean-k-mooney | efried: im just happy you have not tried to adopt it for nova | 19:12 |
| melwitt | efried: yeah, true | 19:12 |
| efried | To be fair, I don't love LP either. But sb is nowhere near mature enough to replace it for something of the scope of nova. | 19:12 |
| sean-k-mooney | the other part of it are proably fine. its the search that kills me | 19:12 |
| efried | maybe it will be some day | 19:13 |
| melwitt | I used to complain about lp before storyboard | 19:13 |
| melwitt | and now I appreciate it a lot more | 19:13 |
| sean-k-mooney | i used to complain about lanchpad before bugzilla | 19:13 |
| melwitt | I actually love bugzilla. which I know is weird | 19:13 |
| sean-k-mooney | but serisly how is the colision in bugzilla actully a thing in 2019 | 19:14 |
| melwitt | yeah but meh, it never trashes what I wrote when that happens | 19:15 |
| melwitt | I guess if you change a bunch of metadata, then yeah true you have to do over | 19:15 |
| melwitt | so yeah, that's really annoying | 19:15 |
| sean-k-mooney | ya i usally am doing both so i end up copy pasting my comment then revert to the latest version set everything again and paste my comment and hope the bot does not change something inbetween | 19:16 |
| mriedem | what is the osc question? | 19:17 |
| sean-k-mooney | when you use osc to list the server events of a deleted server | 19:17 |
| sean-k-mooney | it first tries to get the server which failes | 19:17 |
| mriedem | --os-compute-api-version whatever | 19:18 |
| sean-k-mooney | yep even with that set | 19:18 |
| mriedem | b/c it's looking by name and not finding it, are you using the id? | 19:18 |
| mriedem | KeithMnemonic opened a similar osc bug recently | 19:18 |
| sean-k-mooney | yes useing the uuid | 19:18 |
| sean-k-mooney | http://paste.openstack.org/show/786587/ | 19:18 |
| sean-k-mooney | that is the curl command it produce with the token replaced | 19:19 |
| mriedem | https://storyboard.openstack.org/#!/story/2006761 | 19:19 |
| sean-k-mooney | that was form "openstack --debug --os-compute-api-version 2.21 --os-cloud openstack server event list cef12383-3f43-44cf-be52-218a4da30c0a" | 19:19 |
| melwitt | mriedem: https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-event.html#server-event-list claims that you can see events for a deleted server with 2.21 but you really can't bc osc is doing a lookup without --deleted essentially | 19:19 |
| melwitt | mriedem: ok yeah that story is similar | 19:20 |
| mriedem | yeah it's an osc bug with the server id lookup | 19:20 |
| melwitt | same problem, different command | 19:20 |
| mriedem | open an osc story bug thing but no one will work on it unless one of you do | 19:21 |
| mriedem | does https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-instance-action work? | 19:21 |
| sean-k-mooney | ya and there is no --deleted flag for event list so we cant signle what to do | 19:21 |
| sean-k-mooney | mriedem: yes | 19:21 |
| melwitt | I'm willing to work on it bc we need the action list to be part of a normal user workflow especially now | 19:22 |
| melwitt | or rather, I would prioritize it a lot higher than I usually would | 19:22 |
| mriedem | "especially now" is a bit heavy | 19:22 |
| sean-k-mooney | adding a --delete to the event list might just be the simpelest option | 19:22 |
| sean-k-mooney | mriedem: well we need it for resize not right | 19:22 |
| sean-k-mooney | welli guess not for deleted instnaces | 19:22 |
| mriedem | we changed one api from a call to a cast which might have failed with NoValidHost (MaxRetriesExceeded) after the api had already given you a 202 | 19:23 |
| mriedem | if resize rescheduling fails | 19:23 |
| melwitt | "especially now" is based on my own personal experience at the org where I worked and how openstack was used there. there was no need to look at action events unless you were the auditing | 19:23 |
| melwitt | -the | 19:23 |
| melwitt | I recognize that other orgs end users could/are already using the api heavily | 19:24 |
| mriedem | or not, | 19:24 |
| mriedem | my point is just that it's not like there was a sudden tectonic shift on how nova works that makes caring about instance actions a big deal now | 19:24 |
| mriedem | *in how | 19:25 |
| melwitt | maybe not, but it's an end user perception and I think that's what really matters | 19:25 |
| mriedem | what is an end user perception? | 19:25 |
| melwitt | and again, disclaimer, I don't know how every end user in the world uses nova | 19:25 |
| melwitt | I was an end user of nova on a daily basis at my last company. and I never needed the actions api for anything. if a server action failed, I looked at the server fault. that was it | 19:26 |
| mriedem | if users didn't care about instance actions before, nothing has really changed recently that makes them have to start caring | 19:26 |
| melwitt | that was my perception as an end user | 19:26 |
| mriedem | probably b/c back in havana/icehouse/juno we put the server into error status every time anything failed | 19:26 |
| melwitt | I think there is. if a resize used to fail with a 400, then how would you know to look at actions suddenly to know your resize failed scheduling? | 19:27 |
| melwitt | I think that's quite different. again, just MHO | 19:27 |
| melwitt | as someone who has used nova. that would have surprised me | 19:27 |
| mriedem | resize could have failed *before* after you got a 202 and the first host resize claim failed and rescheduled and got NoValidHost | 19:27 |
| mriedem | to know that, you'd have to know about instance actoins | 19:28 |
| melwitt | *shrug* | 19:28 |
| melwitt | whatever, just saying my opinion | 19:28 |
| melwitt | everyone else can think whatever they want | 19:28 |
| dansmith | my feeling is that things should be as async as possible because that involves the least amount of tying architecture to api behavior, | 19:31 |
| mriedem | i'm not trying to argue, i'm just saying nothing has dramatically changed - things have been this way for years wrt instance actions | 19:31 |
| dansmith | which bit us in the ass with the cells conversion | 19:31 |
| dansmith | right now we have a big mix, with some assumed behaviors from the users (i.e. that 200 from resize means something specific) | 19:31 |
| dansmith | actions have been there a long time, with lots of useful information | 19:31 |
| melwitt | no, I agree that more async is good. I'm just saying that to me, in my own perception, it's a shift from the past habit | 19:32 |
| dansmith | if the users don't know about it, then we probably have some doc gaps, but I don't think that really changes anything | 19:32 |
| melwitt | my habit was not to use instance actions. my habit will be to use instance actions from now on, now that I know | 19:32 |
| dansmith | and I also don't think that actions are some obscure corner of the nova world | 19:32 |
| mriedem | the shift happened years ago then, is my point | 19:32 |
| melwitt | ok, mine didn't, and I figured maybe I'm not the only one | 19:32 |
| dansmith | I think that some people might never go looking for it and just infer what they can from status, return codes, etc, but they're missing part of the story | 19:32 |
| mriedem | after yesterday's meeting i have a todo to write a doc about using actoins | 19:33 |
| mriedem | but i'm still working on this resize/cold migrate doc for artom | 19:33 |
| dansmith | I totes don't see how this is a shift | 19:33 |
| melwitt | I get that you guys don't | 19:33 |
| dansmith | when actions became a thing, there was more information available, that's not a shift, that's just more | 19:33 |
| artom | mriedem, I'll handle the sequence diagram, fwiw | 19:33 |
| dansmith | inferring things from api behavior is.. a thing I guess, and over time those inferences will have to change because they're based on assumptions | 19:34 |
| dansmith | so there is a shift in those inferences I guess, but not in like fundamental behavior of how one uses nova or anything | 19:34 |
| melwitt | it was a shift for me as a user, that's all I'm saying. and the only reason this came up is that, it's a shift for me, I see how important it is, so therefore I have a new interest in fixing the osc command for it | 19:34 |
| melwitt | it means I will change my habit in how I use nova and it might change others too | 19:35 |
| mriedem | artom: i'm about to push it | 19:35 |
| mriedem | artom: i don't want you putting rebuild in the diagram somewhere | 19:36 |
| mriedem | ZING | 19:36 |
| artom | *ded* | 19:36 |
| artom | mriedem, you saw my review, btw? | 19:36 |
| artom | Before you push | 19:36 |
| mriedem | nope, looking | 19:36 |
| artom | mriedem, nothing major, but probably better to see it before pushing again | 19:37 |
| sean-k-mooney | artom: by the way ill adress your comments on my patches on monday i didnt get to them this week but i still plan to adress them shortly | 19:37 |
| sean-k-mooney | but im ment ot be in limerick in 20 mins so i better go o/ | 19:37 |
| artom | sean-k-mooney, drive safe | 19:38 |
| dansmith | our docs should probably have something like this: | 19:39 |
| dansmith | https://support.metacloud.com/hc/en-us/articles/115003922347-Using-the-Instance-Action-List-for-Troubleshooting | 19:39 |
| melwitt | let's copy it | 19:39 |
| dansmith | which doesn't even say anything about error state or fault that I know of, | 19:39 |
| dansmith | but specifically is like "my instance is stopped, whyforisthat?" | 19:40 |
| melwitt | again, maybe it's just me, I'm the lone person who never used instance actions. like a rebel | 19:40 |
| dansmith | which is exactly what we're talking about I think | 19:40 |
| dansmith | melwitt: I think it's entirely possible that it may just be you :P | 19:40 |
| melwitt | uh huh | 19:40 |
| dansmith | heh | 19:40 |
| mriedem | dansmith: that's basically the kind of doc i was going to write, just cli though | 19:41 |
| mriedem | and using resize as an example where the claim on the dest fails | 19:41 |
| dansmith | mriedem: there's cli below | 19:41 |
| dansmith | yep | 19:41 |
| mriedem | using the nova clie | 19:41 |
| mriedem | *cli | 19:41 |
| mriedem | i'd use osc | 19:41 |
| mriedem | anyway, same idea | 19:41 |
| dansmith | I guess if we have no "how do I figure out why my instance is unhappy" doc, then people will make assumptions about how to (or not do) that, which might stop at the instance detail output | 19:41 |
| melwitt | well, if everyone else knows, then that's ideal. I'm glad to hear it | 19:41 |
| mriedem | what are instance actions, when are they useful, how to use them, give an example scenario using resize where the claim fails | 19:42 |
| dansmith | mriedem: well, I think starting with the use-case of debugging an unknown state of an instance is useful, | 19:42 |
| dansmith | because if people _don't_ know what actions are, they're not looking for a doc on actions | 19:42 |
| *** dviroel has joined #openstack-nova | 19:42 | |
| dansmith | melwitt: amirite/ | 19:43 |
| *** ociuhandu has joined #openstack-nova | 19:43 | |
| dansmith | maybe "unknown state" is the wrong term, but.. debugging why something failed or isn't expected on the instance, | 19:43 |
| melwitt | dansmith: yeah, I think so. case in point, gregwork was here earlier and a volume attach failed and there was no indication. I told him to look at instance actions based on the resize/migration thing from yesterday | 19:43 |
| dansmith | which might be a weird state and might be "resize never happend" | 19:43 |
| melwitt | and in there, it showed the volume attach with 'error' as the result | 19:44 |
| * mriedem deletes the todo | 19:44 | |
| dansmith | melwitt: I think gregwork is just lonely and wanted to talk to someone and not read docs, so not sure that would have helped him | 19:44 |
| * dansmith runs | 19:44 | |
| melwitt | so there's already a doc? what did I miss | 19:45 |
| mriedem | no | 19:45 |
| mriedem | i was joking, | 19:45 |
| * dansmith shakes his head | 19:45 | |
| melwitt | what | 19:45 |
| mriedem | how do i say politely, if you guys want to write the doc(s), go ahead | 19:46 |
| * gregwork ! | 19:46 | |
| melwitt | I'll copy metacloud's | 19:46 |
| gregwork | heh i was just reporting something strange that i observed. normally openstack is very vocal about errors that get thrown | 19:46 |
| gregwork | i thought it was unusual that not a lot of fuss was happening | 19:47 |
| gregwork | and ultimately heat would just fail to deploy the stack after some time (token expiry) | 19:48 |
| KeithMnemonic | mridem: melwitt: i can help as well and then abandon my hack sitting out there ;-) | 19:48 |
| gregwork | usually if a resource barfs heat catches it and also barfs | 19:48 |
| gregwork | this went off into space | 19:48 |
| melwitt | KeithMnemonic: sorry, what's your hack? | 19:49 |
| KeithMnemonic | using id only | 19:49 |
| melwitt | gregwork: sounds like heat needs to look at INSTANCE ACTIONS | 19:49 |
| KeithMnemonic | https://review.opendev.org/691045 | 19:49 |
| melwitt | oh that, ok | 19:49 |
| KeithMnemonic | gyee and i played with it a while trying to get the name to work and we never could | 19:50 |
| melwitt | that sounds hard then :) | 19:50 |
| melwitt | I'll try to help, it's related to the server event thing, so hopefully once figured out once it will work for all | 19:51 |
| melwitt | I don't have that much experience with osc | 19:51 |
| *** ociuhandu has quit IRC | 19:52 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Don't delete compute node, when deleting service other than nova-compute https://review.opendev.org/695382 | 19:52 |
| openstackgerrit | Merged openstack/nova master: tests: Use consistent URL regex substitution https://review.opendev.org/665949 | 19:53 |
| melwitt | gregwork: I'm joking but it's probably true. would need to open a bug with them and explain they need to use and parse instance actions | 19:53 |
| openstackgerrit | Merged openstack/nova master: Give the policy vision document a facelift https://review.opendev.org/644615 | 19:53 |
| openstackgerrit | Merged openstack/nova master: test cleanup: Remove skipIf test decorator https://review.opendev.org/685375 | 19:53 |
| melwitt | to be able to pick up the volume attach fail | 19:53 |
| KeithMnemonic | melwitt this is what we saw http://paste.openstack.org/show/785530/ | 19:59 |
| melwitt | KeithMnemonic: yeah, I was gonna say, (and gyee's comment says the same) that not even the legacy novaclient can lookup a deleted server by name | 20:00 |
| KeithMnemonic | and some where there is a case statement/loop that tried different lookups and that is where it never finds the server when a name is used | 20:00 |
| mriedem | since https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server-event.html doesn't take a --delete option, to fix the bug there i'd probably handle a NotFound when looking up the server by name or id, and if --os-compute-api-version >= 2.21 then just assume you were given an ID and pass it through | 20:01 |
| mriedem | it'll either work or the user will get a 404 anyway | 20:01 |
| melwitt | KeithMnemonic: yeah, like I said, I think even with the legacy client, the underlying binding that osc is using, there's no way to lookup deleted by name | 20:02 |
| melwitt | so if you want that, you'd have to add it to the novaclient python binding too, afaik | 20:02 |
| mriedem | i'm not sure that's true - if you're admin you can list servers filtering on deleted=True and name | 20:02 |
| melwitt | list you can, but I'm not sure about like nova show | 20:03 |
| mriedem | right, not in the api - it would be client side sugar | 20:03 |
| melwitt | yeah, that's what I mean. novaclient needs something to be able to. unless you're saying on the cli can't and that the python api can | 20:03 |
| mriedem | heh, "Filters the response by a server name, as a string. You can use regular expressions in the query. For example, the ?name=bob regular expression returns both bob and bobb. If you must match on only bob, you can use a regular expression that matches the syntax of the underlying database server that is implemented for Compute, such as MySQL or PostgreSQL." | 20:03 |
| mriedem | "if you know the cloud you're talking to is using mysql or postgres, just fashion your filtering that way" | 20:04 |
| melwitt | lol | 20:04 |
| melwitt | I'm super rusty on the clients, gotta get back in there | 20:05 |
| artom | mriedem, sql injection as a service | 20:05 |
| KeithMnemonic | and also the using "id" only started working after I applied mriedem's fix for the marker. Gentle plug, any change to get those last few you submitted merged? https://review.opendev.org/#/c/690721/4 | 20:07 |
| mriedem | fwiw i would not spend a bunch of time trying to build "find deleted service by name using list" for this action events osc bug - it's likely more of a corner case, only possible to do that with admin anyway (list deleted servers and filter by name), i would just make it work if an id is provided and not a name, like i said above | 20:11 |
| mriedem | *server | 20:11 |
| mriedem | you don't have to know if it's an id or a name, just: | 20:11 |
| mriedem | 1. if find by name or id returns NotFound | 20:11 |
| mriedem | 2. if microversion >= 2.21 | 20:11 |
| mriedem | 3. then pass through to the api and assume it works | 20:11 |
| mriedem | probably update the osc docs as well to say if you're trying to list actions for a deleted server, you must provide the server id | 20:12 |
| *** jcosmao has joined #openstack-nova | 20:13 | |
| KeithMnemonic | melwitt do you me to abandon my patch or do you want to add to it,. the code there does work with marker being an ID | 20:14 |
| mriedem | KeithMnemonic: she's talking about fixing a different bug | 20:15 |
| mriedem | different command | 20:15 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add contributor doc for resize and cold migrate https://review.opendev.org/695609 | 20:16 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add sequence diagrams to resize/cold migrate contrib doc https://review.opendev.org/695759 | 20:16 |
| *** eharney has joined #openstack-nova | 20:16 | |
| KeithMnemonic | ah ok | 20:21 |
| *** ociuhandu has joined #openstack-nova | 20:23 | |
| KeithMnemonic | i thought you were talking about the osc behavior sean-k-mooney mentioned above when you referenced my bug | 20:23 |
| *** abaindur has joined #openstack-nova | 20:23 | |
| melwitt | well, I thought maybe the two (not being able to get --deleted) had the same root cause but they might not. I didn't look that deeply at it yet nor do I already know that much about osc | 20:24 |
| melwitt | but yeah actually we can already tell they're not because KeithMnemonic's problem is only around the name | 20:26 |
| melwitt | mriedem: ack, that was my intention, make it work with id. because even the legacy novaclient can't do name for the instance action list | 20:27 |
| melwitt | for deleted servers | 20:27 |
| *** abaindur has quit IRC | 20:28 | |
| *** ociuhandu has quit IRC | 20:28 | |
| KeithMnemonic | my bug is with id also | 20:29 |
| melwitt | I thought you said it works with id earlier | 20:30 |
| KeithMnemonic | the nova cli, sorry the person who reported it to us used id | 20:30 |
| KeithMnemonic | this worked nova list --deleted --marker c244e139-98fe-44eb-a971-c067954da026 | 20:30 |
| KeithMnemonic | this did not openstack server list --delete --marker f1ed3399-b4d3-4c74-a8aa-4b5b935bc8d1 | 20:31 |
| *** luksky has joined #openstack-nova | 20:32 | |
| KeithMnemonic | but with that patch i have up, marker works | 20:32 |
| KeithMnemonic | sorry for the confusion | 20:32 |
| *** damien_r has joined #openstack-nova | 20:35 | |
| *** slaweq has joined #openstack-nova | 20:38 | |
| *** damien_r has quit IRC | 20:41 | |
| artom | mriedem, did you mean to not push the sequence diagram? | 20:45 |
| artom | Because... you didn't. | 20:46 |
| mriedem | artom: but i....did | 20:48 |
| mriedem | https://review.opendev.org/695759 | 20:48 |
| mriedem | "A sequence diagram is sorely needed for this as well but that will come in a separate change." | 20:48 |
| *** damien_r has joined #openstack-nova | 20:48 | |
| artom | gdi it's on top | 20:49 |
| artom | mriedem, sorry | 20:49 |
| *** rcernin has joined #openstack-nova | 20:49 | |
| *** damien_r has quit IRC | 20:55 | |
| *** slaweq has quit IRC | 20:56 | |
| openstackgerrit | Merged openstack/nova master: Remove '/os-tenant-networks' REST API https://review.opendev.org/686810 | 20:57 |
| openstackgerrit | Merged openstack/nova master: nova-net: Remove 'USE_NEUTRON' from functional tests https://review.opendev.org/686811 | 20:57 |
| openstackgerrit | Merged openstack/nova master: docs: Remove 'adv-config', 'system-admin' subdocs https://review.opendev.org/684402 | 20:57 |
| openstackgerrit | Merged openstack/nova master: docs: Replacing underscores with dashes https://review.opendev.org/685929 | 20:57 |
| openstackgerrit | Merged openstack/nova master: docs: Strip '.rst' suffix https://review.opendev.org/687264 | 20:58 |
| openstackgerrit | Merged openstack/nova master: Don't claim that CLI user data requires manual base64 encoding https://review.opendev.org/674665 | 20:58 |
| openstackgerrit | Merged openstack/nova master: Make it easier to run a selection of tests relevant to ongoing work https://review.opendev.org/677180 | 20:58 |
| melwitt | KeithMnemonic: if I were you I'd make it work with ID only and forget about name for now. the name requires a further change to python-novaclient (AFAIK) and it's already standard operating procedure that some APIs or options only work with ID. not ideal perhaps but it is what it is. just MHO | 21:05 |
| KeithMnemonic | ok so my patch does that ;-) | 21:06 |
| melwitt | yeah, you need to update it no? | 21:06 |
| KeithMnemonic | i can | 21:06 |
| melwitt | update the doc and everything mriedem pointed out on the patch | 21:06 |
| KeithMnemonic | as long as it seems everyone here is ok with the concept | 21:07 |
| melwitt | to make sure marker referenced by ID is consistent | 21:07 |
| KeithMnemonic | thanks for the help | 21:07 |
| melwitt | yeah I think so, unless I'm totally missing something, which I might be. people letting me know lately that I've missed a lot ;) | 21:07 |
| melwitt | I don't see why doing it ID only would not be ok | 21:08 |
| *** eharney has quit IRC | 21:08 | |
| KeithMnemonic | ok thanks again, any chance i can get some movement on that other patch by mriedem for rocky that fixes the marker? he made a ton of progress before the summit but it seems to have stalled | 21:10 |
| *** nweinber__ has quit IRC | 21:10 | |
| KeithMnemonic | it is merged through stein now | 21:10 |
| *** abaindur has joined #openstack-nova | 21:11 | |
| melwitt | yeah. my head's spinning with all the stuff I'm trying to do but I'll try to get those. they're not 100% easy bc there's differences and conflicts to look at | 21:11 |
| *** ociuhandu has joined #openstack-nova | 21:12 | |
| *** mgagne_ is now known as mgagne | 21:13 | |
| melwitt | KeithMnemonic: I guess the sticking point on the osc one is that it's a change in behavior? does that mean that marker by name works with non-deleted servers? | 21:15 |
| KeithMnemonic | yes it does | 21:16 |
| melwitt | ahhhh dammit | 21:17 |
| KeithMnemonic | so maybe some logic needs to go around the change to only do it if -deleted is passed | 21:17 |
| melwitt | yeah like, do some extra stuff (client gymnastics) to get the server by name first then call with marker if --deleted is passed | 21:18 |
| KeithMnemonic | that way it does not break anything but makes -deleted work in once case at least | 21:18 |
| melwitt | *with marker as ID if --deleted is passed | 21:18 |
| KeithMnemonic | yup | 21:18 |
| melwitt | yeah ok. I get it now | 21:18 |
| KeithMnemonic | let me spin up a fresh devstack and start hacking on it again | 21:19 |
| melwitt | yeah so I rewind from what I said earlier and I'd do the extra step if --deleted is passed. grab the server id and then pass marker as id under the covers | 21:19 |
| openstackgerrit | Merged openstack/nova master: Remove 'os-consoles' API https://review.opendev.org/687907 | 21:21 |
| *** awalende has joined #openstack-nova | 21:23 | |
| mriedem | heh, this is a bit old https://docs.openstack.org/nova/latest/reference/vm-states.html#create-instance-states | 21:26 |
| mriedem | api -> compute done! | 21:26 |
| *** awalende has quit IRC | 21:28 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-guide: flesh out BUILD and ACTIVE server create transitions https://review.opendev.org/695771 | 21:37 |
| *** spatel has joined #openstack-nova | 21:38 | |
| *** spatel has quit IRC | 21:39 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-guide: fix the file injection considerations drift https://review.opendev.org/695773 | 21:43 |
| openstackgerrit | Merged openstack/nova master: Avoid raise InstanceNotRunning exception https://review.opendev.org/541152 | 21:48 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-guide: remove empty sections about inter-service interactions https://review.opendev.org/695774 | 21:48 |
| openstackgerrit | Merged openstack/nova master: Start README.rst with a better title https://review.opendev.org/695025 | 21:48 |
| *** abaindur has quit IRC | 22:05 | |
| *** tbachman has quit IRC | 22:06 | |
| *** eharney has joined #openstack-nova | 22:06 | |
| *** spatel has joined #openstack-nova | 22:08 | |
| *** spatel has quit IRC | 22:13 | |
| *** spatel has joined #openstack-nova | 22:15 | |
| *** jcosmao has left #openstack-nova | 22:16 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-guide: flesh out flavor extra specs and image properties https://review.opendev.org/695776 | 22:16 |
| *** rcernin has quit IRC | 22:24 | |
| *** slaweq has joined #openstack-nova | 22:24 | |
| *** slaweq has quit IRC | 22:29 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-guide: flesh out networking concepts https://review.opendev.org/695777 | 22:33 |
| melwitt | efried: question on test cleanup https://review.opendev.org/685399 | 22:33 |
| efried | ... | 22:33 |
| efried | I think you're probably right. Lemme check one thing quick... | 22:35 |
| melwitt | ok. I wasn't sure how this is supposed to be | 22:36 |
| efried | Yup, I think that was just a miss melwitt, good spot, will remove. | 22:37 |
| *** slaweq has joined #openstack-nova | 22:37 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: test cleanup: Make base TestCase subclass oslotest https://review.opendev.org/685399 | 22:38 |
| efried | melwitt: done, thanks. | 22:38 |
| melwitt | kewl | 22:38 |
| efried | thanks for those reviews melwitt | 22:39 |
| efried | a day when my dashboard shrinks is a good day :) | 22:39 |
| efried | and johnthetubaguy ^ | 22:40 |
| melwitt | np, nice cleanups | 22:41 |
| *** slaweq has quit IRC | 22:42 | |
| openstackgerrit | Merged openstack/nova master: Remove 'nova-console' service, 'console' RPC API https://review.opendev.org/687908 | 22:43 |
| openstackgerrit | Merged openstack/nova master: nova-net: Remove 'networks' quota https://review.opendev.org/686812 | 22:44 |
| *** KeithMnemonic has quit IRC | 22:44 | |
| *** slaweq has joined #openstack-nova | 22:45 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: api-guide: flesh out todos in user doc https://review.opendev.org/695778 | 22:47 |
| *** slaweq has quit IRC | 22:50 | |
| *** ociuhandu has quit IRC | 22:51 | |
| *** spatel has quit IRC | 22:58 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: remove admin/manage-users https://review.opendev.org/695779 | 22:59 |
| *** mriedem is now known as mriedem_away | 23:00 | |
| *** luksky has quit IRC | 23:04 | |
| efried | dansmith: you said something about hard reboot being admin-only by default, but I can't find how that's the case in the code, where it looks like soft/hard are under the same umbrella. Did I misunderstand? | 23:13 |
| *** slaweq has joined #openstack-nova | 23:15 | |
| *** jmlowe has joined #openstack-nova | 23:15 | |
| openstackgerrit | Merged openstack/nova master: test cleanup: Use oslotest's Timeout fixture https://review.opendev.org/685389 | 23:17 |
| openstackgerrit | Merged openstack/nova master: test cleanup: Use oslotest's CaptureOutput fixture https://review.opendev.org/685392 | 23:17 |
| *** slaweq has quit IRC | 23:19 | |
| *** ociuhandu has joined #openstack-nova | 23:34 | |
| dansmith | efried: that wasn't me | 23:37 |
| dansmith | at least, not the hard reboot nova action | 23:38 |
| dansmith | I talked a lot about *host* reboot... | 23:38 |
| efried | dammit, that's probably where I got confused. | 23:38 |
| efried | thanks. | 23:38 |
| *** slaweq has joined #openstack-nova | 23:42 | |
| *** tosky has quit IRC | 23:45 | |
| *** slaweq has quit IRC | 23:52 | |
| *** tbachman has joined #openstack-nova | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!