Friday, 2019-11-22

*** ociuhandu has quit IRC00:17
*** ociuhandu has joined #openstack-nova00:17
*** TxGirlGeek has quit IRC00:20
*** ociuhandu has quit IRC00:23
mriedemi've seen your minions in here00:25
mriedembut yes TPS reports are important00:26
*** tbachman has joined #openstack-nova00:27
*** _mlavalle_1 has quit IRC00:31
*** slaweq has joined #openstack-nova00:34
*** ociuhandu has joined #openstack-nova00:36
*** slaweq has quit IRC00:38
*** tbachman has quit IRC00:41
* artom waits for the inevitable f-string injection attacks00:41
*** spatel has joined #openstack-nova00:41
mriedemartom: i have a gift for you00:42
mriedemand it's not even this
artomBack pocket then?00:43
*** macz has joined #openstack-nova00:44
*** ociuhandu has quit IRC00:45
mriedemno, this:
*** gyee has quit IRC00:46
mriedemoh gdi00:47
artomI think you missed 'git add <new file>'00:47
mriedemyup, pushing00:47
*** sapd1 has quit IRC00:47
mriedemgerritbot must also be dead00:47
artommriedem, thanks dude! Left some random thoughts after a quick read. Now must read bedtime story to daughter.00:54
*** tosky has quit IRC00:56
mriedemtell her the story about the dream where you thought you could change the image on a server during cold migrate00:58
*** mriedem is now known as mriedem_afk00:58
*** abaindur has joined #openstack-nova01:02
*** abaindur has quit IRC01:03
*** abaindur has joined #openstack-nova01:03
*** openstackstatus has joined #openstack-nova01:03
*** ChanServ sets mode: +v openstackstatus01:03
*** Liang__ has joined #openstack-nova01:06
*** ociuhandu has joined #openstack-nova01:10
artommriedem_afk, but... you can?01:10
artomIsn't that why we went through that whole RUN_ON_REBUILD in the filters saga years ago?01:11
artomOr am I massively confused01:11
artom*facepalm* rebuild != cold migrate01:13
*** jistr has quit IRC01:14
*** jistr has joined #openstack-nova01:15
*** abaindur has quit IRC01:19
*** macz has quit IRC01:23
*** ociuhandu has quit IRC01:28
*** brinzhang has joined #openstack-nova01:28
*** brinzhang has quit IRC01:29
*** brinzhang has joined #openstack-nova01:29
*** macz has joined #openstack-nova01:31
*** mriedem_afk has quit IRC01:40
*** brinzhang_ has joined #openstack-nova01:45
*** nanzha has joined #openstack-nova01:46
*** abaindur has joined #openstack-nova01:48
*** brinzhang has quit IRC01:48
*** abaindur has quit IRC01:48
*** abaindur has joined #openstack-nova01:49
*** tbachman has joined #openstack-nova01:49
*** zhanglong has joined #openstack-nova01:55
*** brault has joined #openstack-nova01:56
*** igordc has quit IRC01:57
*** openstackgerrit has joined #openstack-nova02:00
openstackgerritWei Hui proposed openstack/nova master: bugfix device_type=type-PCI passthrough failed
*** macz has quit IRC02:09
*** brinzhang has joined #openstack-nova02:10
*** brinzhang has quit IRC02:12
*** brinzhang has joined #openstack-nova02:12
*** brinzhang_ has quit IRC02:13
*** abaindur has quit IRC02:13
*** brinzhang has quit IRC02:14
*** dviroel has quit IRC02:21
*** ociuhandu has joined #openstack-nova02:28
*** zhanglong has quit IRC02:31
*** spatel has quit IRC02:37
*** ociuhandu has quit IRC02:39
*** tetsuro has joined #openstack-nova02:49
*** ricolin has joined #openstack-nova02:50
*** tetsuro has quit IRC03:11
*** ociuhandu has joined #openstack-nova03:14
*** awalende has joined #openstack-nova03:15
*** awalende has quit IRC03:19
*** ociuhandu has quit IRC03:19
*** macz has joined #openstack-nova03:30
*** yedongcan has joined #openstack-nova03:32
*** macz has quit IRC03:34
*** udesale has joined #openstack-nova03:41
*** tetsuro has joined #openstack-nova03:58
*** bhagyashris has joined #openstack-nova04:02
*** mkrai has joined #openstack-nova04:04
*** chenhaw has joined #openstack-nova04:05
*** shilpasd has quit IRC04:15
*** larainema has joined #openstack-nova04:23
*** ratailor has joined #openstack-nova04:30
*** tetsuro has quit IRC04:31
*** tetsuro has joined #openstack-nova04:35
*** tetsuro has quit IRC04:36
*** tetsuro_ has joined #openstack-nova04:36
*** tetsuro_ has quit IRC04:36
*** tetsuro has joined #openstack-nova04:38
*** dave-mccowan has quit IRC04:58
*** bhagyashris has quit IRC05:01
*** zhanglong has joined #openstack-nova05:09
*** bhagyashris has joined #openstack-nova05:09
*** ratailor has quit IRC05:13
*** ratailor has joined #openstack-nova05:18
*** ociuhandu has joined #openstack-nova05:30
*** ociuhandu has quit IRC05:36
*** macz has joined #openstack-nova05:53
*** zhanglong has quit IRC05:54
*** zhanglong has joined #openstack-nova05:56
*** macz has quit IRC05:58
*** links has joined #openstack-nova05:58
*** Luzi has joined #openstack-nova06:06
*** zbr has joined #openstack-nova06:07
*** zbr|ooo has quit IRC06:08
*** sridharg has joined #openstack-nova06:14
*** mmethot has quit IRC06:18
*** zhanglong has quit IRC06:20
*** zhanglong has joined #openstack-nova06:23
*** slaweq has joined #openstack-nova06:23
*** ociuhandu has joined #openstack-nova06:25
*** slaweq has quit IRC06:25
*** janki has joined #openstack-nova06:31
*** zhanglong has quit IRC06:34
openstackgerritMerged openstack/nova stable/stein: Don't delete compute node, when deleting service other than nova-compute
*** zhanglong has joined #openstack-nova06:48
*** ociuhandu has quit IRC06:52
*** awalende has joined #openstack-nova06:58
*** awalende has quit IRC07:02
*** ociuhandu has joined #openstack-nova07:03
*** zhanglong has quit IRC07:14
*** macz has joined #openstack-nova07:25
*** macz has quit IRC07:29
*** ociuhandu has quit IRC07:32
*** slaweq has joined #openstack-nova07:32
*** janki has quit IRC07:52
openstackgerritMerged openstack/nova master: ZVM: Implement update_provider_tree
*** slaweq has quit IRC08:07
*** ccamacho has joined #openstack-nova08:09
*** jawad_axd has joined #openstack-nova08:11
*** ccamacho has quit IRC08:13
*** tosky has joined #openstack-nova08:15
*** pcaruana has joined #openstack-nova08:21
*** zhanglong has joined #openstack-nova08:22
*** ccamacho has joined #openstack-nova08:27
*** awalende has joined #openstack-nova08:30
*** tesseract has joined #openstack-nova08:35
*** tetsuro has quit IRC08:36
stephenfinartom: you is welcome.08:43
*** yaawang has quit IRC08:49
*** yaawang has joined #openstack-nova08:49
*** rpittau|afk is now known as rpittau08:58
*** mlycka has joined #openstack-nova09:08
*** sridharg has quit IRC09:10
stephenfinbauzas: Wanna take a look at today?09:12
*** nanzha has quit IRC09:19
*** nanzha has joined #openstack-nova09:19
openstackgerritAlexandre arents proposed openstack/nova master: Do not update root_device_name during guest config
*** zhanglong has quit IRC09:22
*** zhanglong has joined #openstack-nova09:23
* bauzas clicks09:23
bauzasstephenfin: yeah you weren't around while we discussed between gibi, johnthetubaguy and others and when we agreed on a microversion as a signal09:24
stephenfinI've already replied to johnthetubaguy. It doesn't make any sense09:24
bauzasstephenfin: but your comment  makes sense indeed09:24
bauzasI'm honestly not really opionated09:25
*** martinkennelly has joined #openstack-nova09:25
stephenfinYeah, look at the other examples I referenced. This is what we've done in the past and I think it's correct09:25
bauzaslet's just move the ball running and09:25
bauzassee what we can signal at the end09:25
stephenfincoolness :)09:25
*** akshayhavile has joined #openstack-nova09:26
stephenfinwould be good to drag mriedem in when we get there since he's done this stuff in the past too09:26
*** ralonsoh has joined #openstack-nova09:26
*** akshayhavile has quit IRC09:28
* stephenfin wonders how he missed that entire conversation yesterday /o\09:28
bauzasstephenfin: you were at the gym or feeding too much stuff in your mouth09:29
stephenfinthat does sound like me alright09:29
bauzasbtw. are we around Thanksgiving ?09:30
bauzaschannel was quiet yesterday09:30
gibisean-k-mooney: thanks for the bandwidth bug report. efried: thanks for the summary mail. I will dive in09:35
*** tssurya has joined #openstack-nova09:36
*** Liang__ has quit IRC09:36
mlyckaHello. What's the opinion on taking another stab
stephenfinbauzas: I am09:38
stephenfinif that's who you're asking?09:38
bauzasstephenfin: you are what ?09:38
bauzasI was wondering whether it was Thanksgiving period09:39
bauzas(disclaimer, it's absolutely not a thing here)09:39
stephenfinAh, I read that as are _you_ around09:39
bauzasbut I can check the calender09:39
bauzasnope, it's next week09:39
* bauzas notes asking google is way faster than asking colleagues over IRC09:40
* bauzas should consider AI as a good way to communicate09:40
*** ricolin has quit IRC09:43
*** derekh has joined #openstack-nova09:44
*** macz has joined #openstack-nova09:45
*** yedongcan has quit IRC09:48
openstackgerritAlexandre arents proposed openstack/nova master: Do not update root_device_name during guest config
*** macz has quit IRC09:50
*** chenhaw has quit IRC09:50
*** yedongcan has joined #openstack-nova09:51
*** chenhaw has joined #openstack-nova09:59
*** zhanglong has quit IRC10:00
*** ociuhandu has joined #openstack-nova10:02
*** ociuhandu has quit IRC10:07
*** chenhaw has quit IRC10:09
*** chenhaw has joined #openstack-nova10:09
*** tssurya has quit IRC10:13
*** chenhaw has quit IRC10:15
*** dtantsur|afk is now known as dtantsur10:22
*** chenhaw has joined #openstack-nova10:38
*** nanzha has quit IRC10:38
*** nanzha has joined #openstack-nova10:40
*** chenhaw has quit IRC10:45
*** mkrai has quit IRC10:46
*** ratailor has quit IRC10:49
*** ociuhandu has joined #openstack-nova10:50
*** ociuhandu has quit IRC10:52
*** ociuhandu has joined #openstack-nova10:52
*** ociuhandu has quit IRC10:53
*** ociuhandu has joined #openstack-nova11:01
*** ociuhandu has quit IRC11:08
*** nanzha has quit IRC11:10
*** udesale has quit IRC11:10
*** nanzha has joined #openstack-nova11:24
*** luksky has joined #openstack-nova11:34
*** bhagyashris has quit IRC11:35
*** tbachman has quit IRC11:39
*** sridharg has joined #openstack-nova11:47
*** tssurya has joined #openstack-nova11:48
sean-k-mooneymelwitt: the non-ironic case is because they were following our scale in docs which direct them to delete the compute service however they skipped step 1 which is remove all instances form the node first. and i guess ensure that all allocations are removed before doing the service delete11:59
sean-k-mooneyso they delete the compute service with vms running then the bug got filed to help them fix that and prefereabel keep the vms by migrating them.12:00
sean-k-mooneyso its a self inflicted failure12:00
*** rcernin has quit IRC12:01
sean-k-mooneyefried: gibi: by the way the downstream bug is just the bugzilla for validating support for bandwidth support. i think on ebug is being filed for triplo but we have not filed one for nova yet because i wanted to disucss what we found first to make sure what the behavior should be.12:04
openstackgerritSurya Seetharaman proposed openstack/nova master: Include removal of ephemeral backing files in the image cache manager
openstackgerritLee Yarwood proposed openstack/nova master: Revert "nova shared storage: rbd is always shared storage"
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Rename _is_storage_shared_with to _is_path_shared_with
gibisean-k-mooney: agree. Right now we need to find a backportable solution (see my mail on the ML about the problems with efried's proposal)12:05
gibisean-k-mooney: for nova I will create a small spec to add the node uuid filtering capability to os-hypervisors12:05
gibisean-k-mooney: for neutron I would like to wait for the neutron folks to react12:06
sean-k-mooneyefried: gibi this is the downstream bugzilla fyi but let continue upstream12:06 bug 1463838 in openstack-nova "[RFE] Guaranteed minimum bandwidth" [Urgent,On_qa] - Assigned to smooney12:06
sean-k-mooneygibi: am what we need ist to be able to query it by host or hypervior host12:06
sean-k-mooneynot by uuid12:06
gibisean-k-mooney: my bad, yes, we need to be able to filter by service host name12:07
gibito get the node uuid12:07
sean-k-mooneywe might be able to do that already  we can certenly just do a hypervior show12:08
sean-k-mooneyand i think that will accpet the host but i need to check the hypervioru api12:08
sean-k-mooneyim still catching up on the mail trhead so ill comment more once i have read it12:08
sean-k-mooneyin terms of backport it would need to be backported to train and stein if we did backport a fix12:09
sean-k-mooneybut for the sepecific case we found customer should not hit it12:10
sean-k-mooneythe might unitll we fix the standalone mode but in a normal tipleo deployment it set the hostname to match the value it sets in the host config option12:11
sean-k-mooneyin standalone mode the host os is installed manually and therfor cloud-init does not run and that step is not done12:12
gibias a side track I filed bug as nova compute fails if the is changed after initial deployment
openstackLaunchpad bug 1853587 in OpenStack Compute (nova) "nova compute fails to update inventory in placement after changing [DEFAULT]/host" [Undecided,New]12:12
sean-k-mooneyya that is basically the same issue not entirely but close enough12:14
sean-k-mooneygibi: ok so we use the host option to create teh compute service recored but the hypervior_hostname for the placement name12:16
sean-k-mooneyso changing after the fact causes us to creat a new service record and uuid12:17
gibisean-k-mooney: yepp, which is logical if we have to support more than one hypervisors per service12:17
sean-k-mooneybut then when we try to create the RP we get a colission on the hyperviour_hostname right12:17
gibisean-k-mooney: right12:17
gibisean-k-mooney: I guess if you need to change then first have to cleanup the old service and placement12:17
sean-k-mooneythat or if we dont find the service record by host check if its theyer by hypervior hostname and then upstat the host field?12:18
sean-k-mooneynot sure that would be right in all cases however12:19
sean-k-mooneydraning deleting and recreating seams safer12:19
sean-k-mooneyfunnily enough we have a customer that want to be able to do a backup and restor of the compute node and potentially chang ethe hostname since there hostnames are determine by the location of the server and they may need to restore to a different host12:20
gibithat sounds fun12:21
sean-k-mooneyif they were not still running a pre placement version of openstack they would hit this issue12:21
openstackgerritLee Yarwood proposed openstack/nova master: compute: Take an instance.uuid lock when rebooting
*** mdbooth has quit IRC12:26
openstackgerritLee Yarwood proposed openstack/nova master: virt: Provide block_device_info during a rescue
sean-k-mooneygibi: did i mention i also traced how neuton learns what hostname to use in the frist place12:27
*** mdbooth has joined #openstack-nova12:28
gibisean-k-mooney: I think you did and I also looked at the code with the help of lajoskaton this morning12:28
sean-k-mooneyits via the agent report12:28
openstackgerritMerged openstack/nova master: Specify what RPs _ensure_resource_provider collects
openstackgerritMerged openstack/nova master: nova-net: Add TODOs for remaining nova-network functional tests
sean-k-mooneyso to use the actual hostname we would have to extend the report with a hypervior hostname field12:29
sean-k-mooneythen the neutron server could use that if its present12:29
gibisean-k-mooney: exaclty, which is an RPC change which might not be backportable12:29
gibihence my reply asking neutron folks about it12:30
sean-k-mooneyits not an object change12:30
sean-k-mooneyits an unversioned dict of strings field12:30
sean-k-mooneyat least it used to be12:30
gibisean-k-mooney: neutron server still need to have a way to handle old RPC reports from not upgraded agents12:30
sean-k-mooneyya which could be to either first try to look up in placmene by its existing host value.12:31
sean-k-mooneyand if it fails then fallback to the full hyperviour list and cache the result12:32
*** mkrai has joined #openstack-nova12:32
sean-k-mooneynormally the and hypervior_host will be the same in the libvirt case12:32
*** nanzha has quit IRC12:38
openstackgerritBalazs Gibizer proposed openstack/nova stable/stein: Use admin neutron client to query ports for binding
gibielod: ^^ fixed your comment12:40
openstackgerritBalazs Gibizer proposed openstack/nova stable/stein: Use admin neutron client to see if instance has qos ports
elodgibi: thx, looking12:41
*** nanzha has joined #openstack-nova12:42
*** slaweq has joined #openstack-nova12:42
gibioooo ma csovet is fognak ma vagni12:43
gibiaz nagyon meno lesz12:43
*** larainema has quit IRC12:50
*** macz has joined #openstack-nova12:55
gibiwrong window obviously12:58
*** macz has quit IRC12:59
*** priteau has joined #openstack-nova13:06
*** Luzi has quit IRC13:09
*** slaweq has quit IRC13:20
*** awalende has quit IRC13:22
*** tbachman has joined #openstack-nova13:22
*** awalende has joined #openstack-nova13:22
*** yedongcan has left #openstack-nova13:23
*** awalende_ has joined #openstack-nova13:27
*** awalende has quit IRC13:27
*** awalende_ has quit IRC13:31
*** priteau has quit IRC13:33
*** eharney has quit IRC13:37
*** jawad_axd has quit IRC13:42
*** jawad_axd has joined #openstack-nova13:43
openstackgerritMerged openstack/nova master: Remove 'os-security-group-default-rules' REST API
*** mgoddard has quit IRC13:44
openstackgerritMerged openstack/nova master: nova-net: Remove unused '*_default_rules' security group DB APIs
openstackgerritMerged openstack/nova master: Remove (most) '/os-networks' REST APIs
*** mgoddard has joined #openstack-nova13:45
*** jawad_axd has quit IRC13:48
*** sridharg has quit IRC13:54
*** jaosorior has joined #openstack-nova14:00
*** ociuhandu has joined #openstack-nova14:08
*** awalende has joined #openstack-nova14:09
openstackgerritMerged openstack/nova stable/train: Add functional recreate test for bug 1852610
openstackbug 1852610 in OpenStack Compute (nova) train "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Undecided,In progress] - Assigned to Matt Riedemann (mriedem)14:10
*** mmethot has joined #openstack-nova14:12
*** awalende has quit IRC14:14
*** ociuhandu has quit IRC14:16
*** damien_r has joined #openstack-nova14:17
*** damien_r has quit IRC14:18
*** damien_r has joined #openstack-nova14:19
*** mriedem has joined #openstack-nova14:19
mlyckaHi again, quick question, is there no 'quiesced' state for VMs?14:35
*** jawad_axd has joined #openstack-nova14:36
*** nweinber__ has joined #openstack-nova14:36
*** jawad_axd has quit IRC14:40
johnthetubaguymlycka: I don't think there is, but there is a state of snapshotting, or something like that, during which time it might be quiesced14:42
mriedemi think that's when creating a snapshot of an active volume-backed instance14:44
mriedemand only if the driver and image support it14:44
mriedemi.e. qemu guest agent14:44
mlyckaRight right...I'm just looking through an older patch which added a new state as part of exposing quiescing/unquiescing in the API. I'm guessing that would still have to be done for that functionality to be implemented.14:44
*** links has quit IRC14:45
mriedemi don't see that we even set the task_state on the instance while creating a volume-backed snapshot, which seems bad14:46
mriedemmeans you could like migrate the server while creating the snapshot14:47
mriedemwe uh don't even have a different server status during a snapshot
*** mns96 has joined #openstack-nova14:49
mriedemanyway, that'd be an extremely latent issue14:49
openstackgerritMerged openstack/nova stable/train: Add functional recreate revert resize test for bug 1852610
openstackbug 1852610 in OpenStack Compute (nova) train "API allows source compute service/node deletion while instances are pending a resize confirm/revert" [Undecided,In progress] - Assigned to Matt Riedemann (mriedem)14:50
mlyckamriedem: How do you mean latent?14:51
*** eharney has joined #openstack-nova14:54
mriedemmeaning if it's a bug, it's always been there14:56
*** mkrai has quit IRC14:56
mriedemi just posted some of this to the ML since i think i come across this stuff like every year or so and wonder the same things14:57
mlyckaSorry, ML?14:57
mriedemmailing list14:58
mlyckaRight, thanks14:58
melwittsean-k-mooney: ah, thanks for the context14:58
*** ratailor has joined #openstack-nova14:59
stephenfinanyone want to hit this for me so I can keep the remove-nova-network train moving?
mriedemstephenfin: looking,15:01
mriedemstephenfin: you want to rebase your remove os-consoles change? then i can re-approve15:01
*** ociuhandu has joined #openstack-nova15:04
mriedemstephenfin: question
mlyckamriedem: Shouldn't quiescing have a state separate from snapshot though?15:10
jrollefried: dansmith: I'm back from an offsite thing and will catch up on vtpm discussion today, thanks for the pings15:10
mriedemstephenfin: nevermind, i answered my own question15:11
mriedemmlycka: if anything i think it would be a task_state as part of an overall snapshot "status"15:12
mriedemso if we had a SNAPSHOTTING status in the API, the task_state would be "quiescing"15:12
mriedemnote that image-backed snapshot has 4 separate task_states15:12
mlyckamriedem: I'm sort of coming from an angle here and I'm still finding my feet with the nova state machine, so bear with me here: I'm looking at introducing the (un)quiesce operations into nova API for use by other entities (Cinder in my case). Would that not point to a quiesce state outside of the existing snapshot process?15:15
mriedemquiesce isn't a terminal state of a server though15:17
openstackgerritBalazs Gibizer proposed openstack/nova master: Support service_host filtering for os-hypervisor API
mriedemso you want to do something initiated in cinder and if the volume is attached to a server you want to call nova's rest api and tell nova to quiesce the instance, right?15:17
stephenfinmriedem: in a meeting but I'll rebase that soon as I'm done, yup15:17
stephenfinand thanks15:17
mlyckamriedem: Correct.15:17
mriedemmlycka: ok i think the api for that would be the os-server-external-events api
mriedemwhich cinder already uses when extending the size of a volume attached to a server15:18
mriedemit's still a compute api microversion change to change the schema of that api, but all of the plumbing already exists in the api layer, you just would need to implement the calls when the event is 'quiesce' or whatever15:19
mlyckaGoodness me, I didn't even realize this existed15:19
mriedemi'm assuming you've seen the old huawei spec that was trying to expose quiesce/unquiesce out of the API for normal users?15:19
mriedembecause that didn't get traction15:20
mlyckaYep, tried asking around earlier precisely for that reason but didn't get a respond15:20
mriedemworking it into os-server-external-events as part of a larger orchestrated operation initiated in cinder is probably the way to go15:20
mriedemso just model it on the volume-extended event15:20
mlyckaWell thank you kindly, I'll look into it15:21
*** jawad_axd has joined #openstack-nova15:21
mriedemmlycka: it will require a spec
mlyckaI was just going to say15:21
mriedemso before implementing a ton of code and then saying "here it is!"15:21
mriedemi'd say get started on the spec to get initial feedback on the problem statement and the proposed change15:21
mlyckaRight, I just need(ed) to get a feel for the way it currently operates and why it hit a brick wall previously.15:22
mriedemi can't remember all the details about that spec but i think they also wanted it to take a list of servers or something15:22
mriedemanyway, what i've said above is likely much more doable15:23
mlyckaYeah, I think that spec was more about snapshotting/quiescing multi-vm setups in a predictable order.15:23
mlyckaYep, gotcha'. I'll poke around the code for a bit and come up with a spec. Thanks.15:24
* mriedem takes the day off after completing one good deed15:24
*** ociuhandu has quit IRC15:25
openstackgerritBalazs Gibizer proposed openstack/nova master: Support service_host filtering for os-hypervisor API
mriedemstephenfin: what is neutron-network? is that something you're cooking up for nova v4?15:27
stephenfindamn it15:28
stephenfinI'll draft a follow-up once I'm done here15:28
*** slaweq has joined #openstack-nova15:28
ayoungonce we merge Nova Network, we should merge Authentication in to Nova and decommision Keystone15:32
*** slaweq has quit IRC15:32
*** ccamacho has quit IRC15:32
mriedemstephenfin: while you're collecting FUPs
mriedemstephenfin: gibi: bauzas: dug a bit too deep here it looks like
*** ayoung has quit IRC15:40
bauzaswe broke the world again.15:40
bauzaswho pushes it out of the gate ?15:41
bauzasstephenfin: ^15:41
stephenfinwe broke something?15:41
bauzasstephenfin: not yet (c)15:42
bauzasbut mriedem provided (again) a good comment15:42
bauzasand we don't want the change to merge now15:42
mriedemthe GETs were not removed15:43
stephenfinah, I see now15:43
mriedembut you removed them in docs, policy and tests15:43
stephenfinGo me \o/15:44
*** ratailor has quit IRC15:47
* mriedem fires off his armpits15:47
mriedemif you're going to rebase, might as well fix that release note later in the series,15:47
mriedemand also a question here
melwittwhy do we want to keep the GETs?15:49
mriedemthey are proxies for neutron and work15:49
mriedemi mean, they'll work once nova-network is gone15:49
melwittoh ok. I thought we didn't want to keep proxies15:50
mriedemthe GETs are capped at 2.3515:50
mriedemiow, proxies have already been deprecated for a long time15:50
mriedemnewton i think15:50
*** damien_r has quit IRC15:52
openstackgerritMatt Riedemann proposed openstack/nova master: Remove '/os-tenant-networks' REST API
mriedempulled from the gate ^15:52
melwittoh right... but that wouldn't mean that all get/post/put stay and return http gone? I guess I never knew the detail behind how that was implemented15:52
stephenfinokay, think I know why I removed that - there's no functional test for it15:53
*** mlavalle has joined #openstack-nova15:53
stephenfinso it didn't exist as far as I, in functional test fixup mode, was concerned15:54
*** slaweq has joined #openstack-nova15:54
mriedemmelwitt: can you rephrase?15:54
mriedemif a thing works after nova-net is gone but was just deprecated b/c it's a proxy, no we won't remove it, because it still works15:54
mriedemwe're not removing our glance or volume proxies here either15:55
melwittsorry, like in the 2.35 deprecation, were all apis except GET removed? or did all apis stay and change to return http gone, in version 2.3515:55
mriedemthey return 40415:55
mriedemdeprecation = 404 for us, gone = 41015:55
melwittand GET does not return 404?15:55
mriedembut you can still hit the proxies at 2.115:55
mriedemnot below the deprecation microversion it shouldn't - not b/c of deprecation i mean; sure you could get 404 for trying to show details on a neutron network that doesn't exist15:56
melwittjust trying to parse why removing POST is ok but not GET15:56
melwittshould they all stay?15:56
mriedemPOST is only implemented for nova-net15:57
mriedemnova-net is going to be gone15:57
mriedemso it's a 41015:57
mriedemif an API is only implemented for nova-net, it will be 41015:57
melwittyeah, ok. I see15:57
melwittit all makes now15:58
melwittmake sense now, aaarrrggghh15:58
stephenfinmriedem: question on
stephenfindoes that actually make sense, since some of the APIs have been removed but others have not15:59
* stephenfin just refreshed. Ignore me16:01
*** luksky has quit IRC16:04
openstackgerritBalazs Gibizer proposed openstack/nova-specs master: Filter hypervisors by service host
openstackgerritStephen Finucane proposed openstack/nova master: Remove '/os-tenant-networks' REST API
openstackgerritStephen Finucane proposed openstack/nova master: nova-net: Remove 'USE_NEUTRON' from functional tests
openstackgerritStephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota
openstackgerritStephen Finucane proposed openstack/nova master: Remove nova-manage network, floating commands
openstackgerritStephen Finucane proposed openstack/nova master: nova-net: Remove associate, disassociate network APIs
openstackgerritStephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-dhcpbridge' binary
openstackgerritStephen Finucane proposed openstack/nova master: nova-net: Remove 'nova-network' binary
openstackgerritStephen Finucane proposed openstack/nova master: docs: Blast most references to nova-network
openstackgerritStephen Finucane proposed openstack/nova master: WIP
stephenfinmelwitt, gibi: fixed that silly typo in the reno if you want to re +2
openstackgerritBalazs Gibizer proposed openstack/nova master: Support service_host filtering for os-hypervisor API
*** jawad_axd has quit IRC16:08
*** gyee has joined #openstack-nova16:09
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'os-consoles' API
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'nova-console' service, 'console' RPC API
openstackgerritStephen Finucane proposed openstack/nova master: Remove 'nova-xvpvncproxy'
openstackgerritBalazs Gibizer proposed openstack/nova-specs master: Filter hypervisors by service host
openstackgerritBalazs Gibizer proposed openstack/nova master: Support service_host filtering for os-hypervisor API
* bauzas bails early for the weekend16:12
bauzasstephenfin: ask others for your series, sorry16:13
stephenfinhave a good weekend o/16:13
melwittstephenfin: why did you update to say "which has been removed"? it's not going to removed until the end of the series right? I guess it makes sense to write it as thought it's already happened though since it will be by the time anyone reads that16:13
stephenfinyeah, I'm writing from the perspective of the end of the series16:14
melwittyeah, ok that makes sense16:14
stephenfinand in reality, we removed nova-network support when we dropped cells v1 support
melwittoh yeah, I didn't think about that16:15
gibistephenfin: sorry, I have to take that next week16:15
stephenfingibi: 'sall good (y)16:15
* gibi heads to a pub16:15
gibihave a nice weekend folks!16:16
melwittbye, happy weekend o/16:16
mriedemstephenfin: a couple of thoughts/questions in the bottom change
mriedemi don't know if we have precedent for 410ing some methods in a route but not all16:27
*** macz has joined #openstack-nova16:28
stephenfinwhy would it be 500?16:28
mriedemi was thinking if you hit something that's not neutron, but i guess if you upgrade and the nova-network binary is literally gone, then yeah i guess that's not possible16:29
mriedemb/c you can only have neutron or you can't upgrade16:29
*** tssurya has quit IRC16:29
stephenfinyeah, the nova-network service literally can't be started since we dropped the cells v1 code16:29
stephenfinso people have to be using neutron in Train, never mind Ussuri16:30
*** ricolin has joined #openstack-nova16:32
*** slaweq has quit IRC16:37
*** jmlowe has joined #openstack-nova16:41
mriedemshould update the reno here
stephenfindamn it16:44
*** priteau has joined #openstack-nova16:44
openstackgerritStephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota
mriedemsorry one more in there - you forgot os-quota-class-sets in the reno16:48
mriedemthe quota apis are super fun!16:48
mriedembecause there are 100 of them16:48
melwittthis patch is a stats machine16:49
openstackgerritStephen Finucane proposed openstack/nova master: nova-net: Remove 'networks' quota
stephenfinDo I get a prize if I win?16:50
stephenfinCan someone "win"?16:50
stephenfinI want the plush tiger16:50
*** nanzha has quit IRC16:50
melwittI think we're all winning cause we're talking about quotas16:51
openstackgerritMatt Riedemann proposed openstack/nova master: Remove nova-manage network, floating commands
stephenfinso long as you don't get more "winning" than you're paying for16:52
openstackgerritMatt Riedemann proposed openstack/nova master: nova-net: Remove associate, disassociate network APIs
openstackgerritMatt Riedemann proposed openstack/nova master: nova-net: Remove 'nova-dhcpbridge' binary
openstackgerritMatt Riedemann proposed openstack/nova master: nova-net: Remove 'nova-network' binary
melwittbadum csh!16:53
openstackgerritMatt Riedemann proposed openstack/nova master: docs: Blast most references to nova-network
melwittman, I really can't type anything today16:53
*** slaweq has joined #openstack-nova16:53
mriedemhave a kit kat16:54
melwitthm, I could. still have some halloween candy16:54
efriedstephenfin: did you see the conversation yesterday with johnthetubaguy about adding a "signal microversion" so clients can discover that nova-net is gone?16:55
stephenfinefried: I did. mriedem is probably the person you want to discuss that with, tbh16:55
*** mlycka has quit IRC16:57
stephenfinI realise why he suggested it but I'm not sure about the idea of using a microversion when we can't actually provide the previous behavior on old microversions16:57
mriedemthe deprecation microversions were the signals16:57
stephenfinwe didn't do it for things like os-cells, os-fping, etc.16:57
mriedemif you want to add a microversion for each api we've removed now, you're talking about all of these
mriedemso -116:57
efriedjohnthetubaguy: ^16:58
efriedyou have five minutes to respond16:58
*** slaweq has quit IRC16:58
*** rchurch has quit IRC16:58
mriedemi mean, i'm sure we can deal with someone that shows up and is like, "WTF HAPPENED TO CLOUDPIPE?!"16:58
melwittthe good old days16:59
mriedemi think i wrote that after we did the first 410s17:00
*** rpittau is now known as rpittau|afk17:01
*** rchurch has joined #openstack-nova17:01
openstackgerritMerged openstack/nova stable/train: Block deleting compute services with in-progress migrations
openstackgerritMerged openstack/nova stable/train: Improve metadata server performance with large security groups
*** jaosorior has quit IRC17:04
efriedmriedem: I got confused where it was suggested that GET /os-hypervisors*? would ever return more than one entry.17:05
efriedIt would still return a list, but it would have (at most) one entry, right?17:05
johnthetubaguymriedem: OK, I buy that. If you see the deprecation micro version, expect to be disappointed with HTTP Gone17:05
efriedeven for ironic17:05
mriedemefried: compute service : compute node (hypervisor) is 1:M with ironic17:06
mriedemplease to be enjoying
openstackLaunchpad bug 1852446 in OpenStack Compute (nova) "Hypervisors in nova - no subpage details for ironic" [Undecided,New]17:06
gregworkis it expected that when an instance is booted into a non nova az, that volumes created by cinder in the nova AZ (successfully as part of the stack create) will fail to attach to the instance?17:06
gregworkalso cinder doesnt seem to know about the az's i created in nova17:06
gregworktrying to pass availability_zone: mynovaAZ returns an unknown az error17:06
mriedemgregwork: see cross_az_attach in
efriedmriedem: ack17:07
gregworkoh interesting17:07
* gregwork configures17:07
mriedemgregwork: also
mriedembefore you go too deep down the AZ rabbit hole you might want to watch that17:08
gregworki was at that summit, i wish i attended that session17:08
gregworkill check it out17:08
gregworkwait cross_az_attach defaults to true17:09
gregworkat least thats how it looks?17:09
gregwork"By default there is no availability zone restruction on volume attach."17:09
* gregwork confused17:10
gregworki see in cinder that the volumes i wanted to attach were created, but they just didnt attach17:10
gregworkalso heat didnt throw an error17:10
*** jmlowe has quit IRC17:10
gregworkmriedem: i read that but since my cross_az_attach is unset (defaulted to true) that doesnt explain why the volume silently didnt attach. it seems to list the caveats with setting it to false17:12
*** mns96 has quit IRC17:15
mriedemare you talking about attaching volumes to an existing server or boot from volume where nova creates the volumes and attaches them to the server being created?17:17
openstackgerritStephen Finucane proposed openstack/nova master: trivial: Resolve (most) flake8 3.x issues
openstackgerritStephen Finucane proposed openstack/nova master: WIP: Switch to flake8 3.x
*** ociuhandu has joined #openstack-nova17:17
mriedemif cross_az_attach=True (default) then nova won't specify any AZ when creating volumes during boot from volume,17:19
mriedemif the attach is failing, it's likely due to something with the setup/storage backend etc17:19
mriedemso you'd have to dig into that17:19
*** dtantsur is now known as dtantsur|afk17:21
*** ricolin has quit IRC17:21
* stephenfin -> 🍻17:21
stephenfinhave a good weekend17:21
*** awalende has joined #openstack-nova17:23
openstackgerritMerged openstack/nova stable/train: Added openssh-client into bindep
*** awalende has quit IRC17:27
*** Guest24639 has joined #openstack-nova17:32
*** Guest24639 is now known as mgagne_17:34
*** tesseract has quit IRC17:35
*** jaosorior has joined #openstack-nova17:44
openstackgerritMerged openstack/nova stable/stein: Use admin neutron client to query ports for binding
openstackgerritMerged openstack/nova master: Abort live-migration during instance_init
*** ociuhandu has quit IRC17:54
*** ociuhandu has joined #openstack-nova17:54
gregworkmriedem: so you were right, it was something failing with cinder, however (and this is probably a bug) the volume attachment would fail because nova running "multipathd show status" failed (multipathd was not started on the compute node).  the issue is that neither heat, nor nova, nor manually doing it in horizon caught the exception.17:59
gregworkit was only when i looked at nova-compute.log on the compute node that i saw the traceback17:59
gregworkim using queens so maybe this is fixed upstream17:59
gregworkjust took the request and happily said it was done18:00
*** ociuhandu has quit IRC18:00
*** derekh has quit IRC18:00
*** martinkennelly has quit IRC18:01
*** bnemec is now known as beekneemech18:04
*** mgariepy has quit IRC18:07
*** mgariepy has joined #openstack-nova18:10
sean-k-mooneywell on one hand managing the lifecycle of multipathd is out of scope of nova18:13
sean-k-mooneyim not sure if os-brick would have detechted the fact that multipathd was not running or if nova would to be able to raise it and im not sure if we would want to expose that to a non admin if it did18:15
sean-k-mooneyheat cant know this unless cinder/nova told it as heat has no acess to the compute node18:15
gregworksean-k-mooney: well from the end user experience take horizon for example.. doing volumes -> manage attachments and picking my instance .. that should throw an error18:15
gregworkbut it doesnt, it accepts the request and nothing happens18:16
gregworka user should be informed what they asked for failed at least18:16
gregworkmaybe not the precise details18:16
sean-k-mooneywell it shoudl go to an error state or not be marked as attached unless it completes18:16
* gregwork nods18:17
gregworkbut it doesnt which is why im mentioning it here18:17
sean-k-mooneyso nova could do two things at that point18:17
sean-k-mooneyit could put the instance in an error state18:17
sean-k-mooneyor it could role back the attachment18:17
gregwork(to be clear it doesnt mark it as attached, it just returns an unmodified horizon screen)18:17
sean-k-mooneyok well that is different18:18
sean-k-mooneywhat is the volume status18:18
gregworkit doesnt go into an error state18:18
sean-k-mooneythen it sound like its doing the right thing18:18
gregworkits like you didnt do anything at all18:18
*** cmurphy is now known as cmorpheus18:18
sean-k-mooneyyes because it tried to attach and failed18:19
melwittdid you check the instance actions api? :P18:19
sean-k-mooneyi was just going to mention that18:19
melwittI won!18:19
sean-k-mooneythat is where the error if it was going to be reported would likely end up18:19
gregworkin cinder or nova ?18:19
*** priteau has quit IRC18:19
melwittI don't know how it looks on horizon, but it's like 'openstack server event list' on the openstackclient18:20
gregworkso i cant openstack server event list <that instance> because we fixed it just now (configuring multipathd) and redeployed the stack18:21
sean-k-mooneyin horizon there is the action log for the instace18:21
gregworkwhere would it be in the infra18:21
sean-k-mooneywhen you click into it18:21
gregworkcompute or control plane18:21
gregworkthe logs you are looking for18:22
sean-k-mooneygregwork: if you have not deleted the instnace it would still be in the instance event log18:22
melwittby redeployed do you mean wiped the database? yeah, what sean-k-mooney said18:22
gregworkit was deleted when the stack was cleaned up18:22
gregworkso all i have are the overcloud logs on the various nodes18:23
melwittI thought you could still see for deleted no? because one of the actions will be 'deleted'18:23
melwittthat's how you can check who deleted an instance and what time, for example18:23
sean-k-mooneyif you have not done a db archive/purge it should still be there18:23
*** igordc has joined #openstack-nova18:23
gregworkid need the uuid of the instance wouldnt i ? the instances were redployed from a templated name18:23
gregworkor is there a list all events18:23
sean-k-mooneyyes you would18:24
melwittyou need uuid18:25
sean-k-mooneyyou would be hitting this api endpoint
sean-k-mooneyspeaking of which is the volume attachmet api call blocking or async
sean-k-mooneyit returns a 200 so i assume its blocking but i expected it to be async18:27
*** irclogbot_1 has quit IRC18:27
melwittgregwork: ok, well, just fyi the "action log" 'openstack server event list' is a place to see failed actions that did not actually touch the instance. and there's a new change proposed to add more detail to errors that go in there, to aid in communication to the user
efriedmelwitt or mriedem or dansmith: Could I please get +A on three simple test-only tech debt reductions (johnthetubaguy just +2ed 'em so they're freshhhh):
efriedjohnthetubaguy: thanks for that btw18:28
melwittgregwork: this came up recently with a change in behavior in the resize and migration APIs too
sean-k-mooneymelwitt: ya although in that context that is because we changed it to async so you have to check the action log going forward18:29
sean-k-mooneymelwitt: if attach is currently blocking the we woudl expect the error to be returned there18:30
*** ralonsoh has quit IRC18:30
melwittyeah. I'm just saying it brought the action log to my attention as a thing to use more regularly18:30
sean-k-mooneyif its async then it would only be in the action log18:30
melwittbecause prior to that, the only time I've seen action log used is when you're like "who deleted this instance and on what date"18:30
*** irclogbot_2 has joined #openstack-nova18:31
sean-k-mooneydid that come up on the mailing list18:31
sean-k-mooneythe fact the userid was missing or somthing like that18:31
melwittand I know, some other clouds use it on a regular basis, just in yahoo when I was there, they didn't. so maybe other people don't18:31
gregworkso there is an instance id in the python traceback of nova-compute.log18:31
gregworkfailed to attach xxx to for id18:32
gregworklet me try that18:32
gregworkim guessing the instance id in this line is not actually the instance uuid ?18:34
gregwork2019-11-22 12:26:56.077 8 ERROR nova.compute.manager [req-a25aba4e-952f-431e-bb85-1e2a45f62e3b d16f2bc26e91f8221042c91c61d0c4f7e7ae57a09507d54809f0279e3d6ee29e f4e6856a85b14ecc99b494159f28e425 - 424dfc884bcd46e7915e62495d6b569c 424dfc884bcd46e7915e62495d6b569c] [instance: 1e65006f-58ee-470d-9111-522d5c63094d] Failed to attach 176cca17-3f02-4fcd-884d-a4bd74ad5943 at /dev/vdb: ProcessExecutionError: Unexpected18:34
gregworkerror while running command.18:34
gregworkdoing an openstack server even list 1e65006f-58ee-470d-9111-522d5c63094d18:34
gregworksays its unknown18:34
sean-k-mooneyam you might need to pass a --deleted flag18:35
sean-k-mooneyif we support that18:35
gregworkthere is no --deleted flag18:36
gregworkat least not in my osc18:36
sean-k-mooneylooking at the api i dont see one either18:36
gregworkmelwitt: seemed to think this was a thing, hence the "show when something was deleted"18:36
melwittyeah, I know it used to. that was the one use I saw in yahoo. let me take a quick look18:37
melwittah ok18:37
melwittyou have to do openstack --os-compute-api-version 2.21 openstack server event list18:38
sean-k-mooneyi assume if do an openstack server show with that uuid i assume it give you nothing18:38
melwittapparently ability to show deleted was added in 2.21 version of the api18:38
gregworkis that api available in queens ?18:39
sean-k-mooneyoh yes that is mentioned in the api ref too18:39
melwittgregwork: yes
melwittadded in mitaka18:39
gregwork$ openstack --os-compute-api-version 2.2118:41
gregwork(openstack) server event list 1e65006f-58ee-470d-9111-522d5c63094d18:41
gregworkNo server with a name or ID of '1e65006f-58ee-470d-9111-522d5c63094d' exists.18:41
melwittsean-k-mooney: I dunno, I see userid when I do openstack server event list abd23967-2dd5-42ea-b96b-a226ee07ad44 --long18:42
melwittI'm gonna try on my devstack18:42
sean-k-mooneymelwitt: yes but it was added i think in stien or rocky18:42
sean-k-mooneyi think specificly for the how deleted my vm case you mentioned18:43
melwittsigh, wtf it's not working for me either18:43
melwittwhat am I missing here...18:44
sean-k-mooneysame  "openstack --os-compute-api-version 2.21 --os-cloud openstack server event list cef12383-3f43-44cf-be52-218a4da30c0a" fails18:45
melwittgregwork: ok well the trusty legacy client works nova instance-action-list d7446288-688b-4a44-be78-008aa1ca524c18:45
melwittsean-k-mooney: I bet it's some underlying osc thing where it can't get deleted servers. I don't know. have to dig into what's going on there18:46
gregwork| create        | req-b76ac906-148b-497a-8d87-72a65be367bd | -       | 2019-11-22T17:26:28.000000 | 2019-11-22T17:26:52.000000 |18:46
gregwork| attach_volume | req-a25aba4e-952f-431e-bb85-1e2a45f62e3b | Error   | 2019-11-22T17:26:54.000000 | 2019-11-22T17:26:56.000000 |18:46
gregworkso there is an error being observed but nothing seems to react to it18:47
gregworkhorizon/heat all just whistle by18:47
gregwork"volume error? cool story bro"18:47
melwittyup. because the instance is still in a working state, it's not put into ERROR18:47
melwittas you have noticed there's no task/status api and if there are any they're not unified,18:48
melwittso our only way of communicating errors for asynchronous operations is basically ERROR state and 'server events'18:49
sean-k-mooneyya i would guess heat did not check the responce18:49
sean-k-mooneyi think attach is blocking so it should have been return to heat18:49
melwittyeah, I mean, heat could probably do something. but I don't know that much about heat18:49
gregworkthe heat behavior is it sits there18:50
gregworkno errors, just waiting18:50
gregworkit eventually times out when its auth token expires18:50
melwittwe don't put the instance into ERROR because it's in a perfectly healthy state, it's just the attach didn't work, which sucks, but it doesn't mean the instance is broken or unusable. so that's why we don't do ERROR state. other than that, the API is async so return code will be 202 ACCEPTED but it can obviously fail after that point after the request goes async18:51
sean-k-mooneymelwitt: which api is async?18:51
melwittsean-k-mooney: is attach not async?18:52
sean-k-mooneyits respocne code is listed as 20018:52
sean-k-mooneywhich would imply not18:52
melwittI assumed it must be based on the behavior. gregwork did you call volume attach or something else?18:52
gregworki create a data volume with os::cinder::volume then attach it using ::volumeattach18:53
sean-k-mooneygregwork: are you attaching via nova or cinder18:53
sean-k-mooneyoh heat18:53
melwittyeah it shows as a cast18:53
gregworkvia heat18:53
melwittattach_volume is a cast (async)18:53
gregworki mean i assume os::cinder::herpderp means its talking to cinder ?18:53
gregwork(heat that is)18:54
sean-k-mooneythe we are returning the wrong responce code it should be a 202 as you said18:54
gregworki try and bring only quality bugs to these channels :)18:54
melwittTHANKS gregwork18:55
gregworkheh np :P18:55
melwittsigh, yeah that sucks. I dunno why that's not a 20218:55
melwittgregwork: I think probably it's (heat) creating the volume in cinder and then calls nova to attach18:56
sean-k-mooneyi guess its calling
sean-k-mooneythat one does return a 20218:56
melwittoh what? I didn't know that cinder had an api to attach to instance. it must just call us18:56
melwittI don't remember any of this stuff18:57
sean-k-mooneyall the proxy apis lol18:57
melwittcan we achieve a full mesh proxy to one another18:57
sean-k-mooneythat is the dream18:57
sean-k-mooneyalso i grab the curl command form osc via --debug18:58
sean-k-mooneyi adde a valid token and that is what we get form the nova api18:58
melwittyeah but you need to do that with the instance actions api18:59
melwittit works fine in the legacy client18:59
melwittso there's something buggy with how osc is calling instance actions api18:59
sean-k-mooneyright so osc is first hitting the /server endpoing for some reason18:59
melwittoh, yeah. that was my assumption that it does something like that across the board18:59
melwittwhich will break nova commands that are supposed to work on deleted things19:00
sean-k-mooneyright so its makeing an extra api call before trying to call the events api19:00
sean-k-mooneybut it does not pass the --delete flag or query arg19:00
sean-k-mooneyso that fails and it never does the thing we asked19:00
melwittI guess the bug fix might be to see if that extra call can be removed selectively for the server event commands19:00
melwittthere might already be a storyboard open for this issue. I can't imagine we're the first to run into it19:01
sean-k-mooneywhen was the last time you tried to get server event?19:01
sean-k-mooneybut ya it might already be open19:02
sean-k-mooneyi would not be surpised if it was not however19:02
melwittme? never (with osc). but I know that (was it huawei?) were making a lot of use of the apis. maybe their end users not using it through osc though. I dunno19:02
melwittI'll chase it up though bc this is going to need to be a normal part of users workflow19:04
*** eharney has quit IRC19:04
sean-k-mooneyoh we added user and projec to migration last cylce not instance actions but we did mention the isntace action as an alternitive in the spec19:05
melwittit's already in instance actions. in osc --long19:06
melwittI saw it19:06
sean-k-mooneyya it is19:07
sean-k-mooneyi was just getting confused with that spec19:07
sean-k-mooneywhere we added to migration and said the alternitve was to find the instnace action19:08
sean-k-mooneyhave i mentioned lately i hate trying to find things in storyboard19:08
melwittman I hate storyboard19:08
*** jaosorior has quit IRC19:08
melwittthe search is a lot worse than launchpad, it doesn't have per stable branch tracking ability19:09
sean-k-mooneyi have set the list option to 1000 result and im now using ctrl-f in my brower instead19:09
melwittlaunchpad forever19:09
sean-k-mooneyas far as u can tell there isnt a story open19:10
melwittok. actually let me look for mriedem's mega etherpad of osc issues, I wonder if it's on there19:10
sean-k-mooneythis seams related!/story/2006761 but its sperate19:11
efriedI'm sure if you asked, they would say you can totally track stable branches, by creating a bunch of separate tags and tasks and ...19:11 ways that are specific to your project...19:11
melwittyeah not specifically mentioned
sean-k-mooneyefried: im just happy you have not tried to adopt it for nova19:12
melwittefried: yeah, true19:12
efriedTo be fair, I don't love LP either. But sb is nowhere near mature enough to replace it for something of the scope of nova.19:12
sean-k-mooneythe other part of it are proably fine. its the search that kills me19:12
efriedmaybe it will be some day19:13
melwittI used to complain about lp before storyboard19:13
melwittand now I appreciate it a lot more19:13
sean-k-mooneyi used to complain about lanchpad before bugzilla19:13
melwittI actually love bugzilla. which I know is weird19:13
sean-k-mooneybut serisly how is the colision in bugzilla actully a thing in 201919:14
melwittyeah but meh, it never trashes what I wrote when that happens19:15
melwittI guess if you change a bunch of metadata, then yeah true you have to do over19:15
melwittso yeah, that's really annoying19:15
sean-k-mooneyya i usally am doing both so i end up copy pasting my comment then revert to the latest version set everything again and paste my comment and hope the bot does not change something inbetween19:16
mriedemwhat is the osc question?19:17
sean-k-mooneywhen you use osc to list the server events of a deleted server19:17
sean-k-mooneyit first tries to get the server which failes19:17
mriedem--os-compute-api-version whatever19:18
sean-k-mooneyyep even with that set19:18
mriedemb/c it's looking by name and not finding it, are you using the id?19:18
mriedemKeithMnemonic opened a similar osc bug recently19:18
sean-k-mooneyyes useing the uuid19:18
sean-k-mooneythat is the curl command it produce with the token replaced19:19
sean-k-mooneythat was form "openstack --debug --os-compute-api-version 2.21 --os-cloud openstack server event list cef12383-3f43-44cf-be52-218a4da30c0a"19:19
melwittmriedem: claims that you can see events for a deleted server with 2.21 but you really can't bc osc is doing a lookup without --deleted essentially19:19
melwittmriedem: ok yeah that story is similar19:20
mriedemyeah it's an osc bug with the server id lookup19:20
melwittsame problem, different command19:20
mriedemopen an osc story bug thing but no one will work on it unless one of you do19:21
mriedemdoes work?19:21
sean-k-mooneyya and there is no --deleted flag for event list so we cant signle what to do19:21
sean-k-mooneymriedem: yes19:21
melwittI'm willing to work on it bc we need the action list to be part of a normal user workflow especially now19:22
melwittor rather, I would prioritize it a lot higher than I usually would19:22
mriedem"especially now" is a bit heavy19:22
sean-k-mooneyadding a --delete to the event list might just be the simpelest option19:22
sean-k-mooneymriedem: well we need it for resize not right19:22
sean-k-mooneywelli guess not for deleted instnaces19:22
mriedemwe changed one api from a call to a cast which might have failed with NoValidHost (MaxRetriesExceeded) after the api had already given you a 20219:23
mriedemif resize rescheduling fails19:23
melwitt"especially now" is based on my own personal experience at the org where I worked and how openstack was used there. there was no need to look at action events unless you were the auditing19:23
melwittI recognize that other orgs end users could/are already using the api heavily19:24
mriedemor not,19:24
mriedemmy point is just that it's not like there was a sudden tectonic shift on how nova works that makes caring about instance actions a big deal now19:24
mriedem*in how19:25
melwittmaybe not, but it's an end user perception and I think that's what really matters19:25
mriedemwhat is an end user perception?19:25
melwittand again, disclaimer, I don't know how every end user in the world uses nova19:25
melwittI was an end user of nova on a daily basis at my last company. and I never needed the actions api for anything. if a server action failed, I looked at the server fault. that was it19:26
mriedemif users didn't care about instance actions before, nothing has really changed recently that makes them have to start caring19:26
melwittthat was my perception as an end user19:26
mriedemprobably b/c back in havana/icehouse/juno we put the server into error status every time anything failed19:26
melwittI think there is. if a resize used to fail with a 400, then how would you know to look at actions suddenly to know your resize failed scheduling?19:27
melwittI think that's quite different. again, just MHO19:27
melwittas someone who has used nova. that would have surprised me19:27
mriedemresize could have failed *before* after you got a 202 and the first host resize claim failed and rescheduled and got NoValidHost19:27
mriedemto know that, you'd have to know about instance actoins19:28
melwittwhatever, just saying my opinion19:28
melwitteveryone else can think whatever they want19:28
dansmithmy feeling is that things should be as async as possible because that involves the least amount of tying architecture to api behavior,19:31
mriedemi'm not trying to argue, i'm just saying nothing has dramatically changed - things have been this way for years wrt instance actions19:31
dansmithwhich bit us in the ass with the cells conversion19:31
dansmithright now we have a big mix, with some assumed behaviors from the users (i.e. that 200 from resize means something specific)19:31
dansmithactions have been there a long time, with lots of useful information19:31
melwittno, I agree that more async is good. I'm just saying that to me, in my own perception, it's a shift from the past habit19:32
dansmithif the users don't know about it, then we probably have some doc gaps, but I don't think that really changes anything19:32
melwittmy habit was not to use instance actions. my habit will be to use instance actions from now on, now that I know19:32
dansmithand I also don't think that actions are some obscure corner of the nova world19:32
mriedemthe shift happened years ago then, is my point19:32
melwittok, mine didn't, and I figured maybe I'm not the only one19:32
dansmithI think that some people might never go looking for it and just infer what they can from status, return codes, etc, but they're missing part of the story19:32
mriedemafter yesterday's meeting i have a todo to write a doc about using actoins19:33
mriedembut i'm still working on this resize/cold migrate doc for artom19:33
dansmithI totes don't see how this is a shift19:33
melwittI get that you guys don't19:33
dansmithwhen actions became a thing, there was more information available, that's not a shift, that's just more19:33
artommriedem, I'll handle the sequence diagram, fwiw19:33
dansmithinferring things from api behavior is.. a thing I guess, and over time those inferences will have to change because they're based on assumptions19:34
dansmithso there is a shift in those inferences I guess, but not in like fundamental behavior of how one uses nova or anything19:34
melwittit was a shift for me as a user, that's all I'm saying. and the only reason this came up is that, it's a shift for me, I see how important it is, so therefore I have a new interest in fixing the osc command for it19:34
melwittit means I will change my habit in how I use nova and it might change others too19:35
mriedemartom: i'm about to push it19:35
mriedemartom: i don't want you putting rebuild in the diagram somewhere19:36
artommriedem, you saw my review, btw?19:36
artomBefore you push19:36
mriedemnope, looking19:36
artommriedem, nothing major, but probably better to see it before pushing again19:37
sean-k-mooneyartom: by the way ill adress your comments on my patches on monday i didnt get to them this week but i still plan to adress them shortly19:37
sean-k-mooneybut im ment ot be in limerick in 20 mins so i better go o/19:37
artomsean-k-mooney, drive safe19:38
dansmithour docs should probably have something like this:19:39
melwittlet's copy it19:39
dansmithwhich doesn't even say anything about error state or fault that I know of,19:39
dansmithbut specifically is like "my instance is stopped, whyforisthat?"19:40
melwittagain, maybe it's just me, I'm the lone person who never used instance actions. like a rebel19:40
dansmithwhich is exactly what we're talking about I think19:40
dansmithmelwitt: I think it's entirely possible that it may just be you :P19:40
melwittuh huh19:40
mriedemdansmith: that's basically the kind of doc i was going to write, just cli though19:41
mriedemand using resize as an example where the claim on the dest fails19:41
dansmithmriedem: there's cli below19:41
mriedemusing the nova clie19:41
mriedemi'd use osc19:41
mriedemanyway, same idea19:41
dansmithI guess if we have no "how do I figure out why my instance is unhappy" doc, then people will make assumptions about how to (or not do) that, which might stop at the instance detail output19:41
melwittwell, if everyone else knows, then that's ideal. I'm glad to hear it19:41
mriedemwhat are instance actions, when are they useful, how to use them, give an example scenario using resize where the claim fails19:42
dansmithmriedem: well, I think starting with the use-case of debugging an unknown state of an instance is useful,19:42
dansmithbecause if people _don't_ know what actions are, they're not looking for a doc on actions19:42
*** dviroel has joined #openstack-nova19:42
dansmithmelwitt: amirite/19:43
*** ociuhandu has joined #openstack-nova19:43
dansmithmaybe "unknown state" is the wrong term, but.. debugging why something failed or isn't expected on the instance,19:43
melwittdansmith: yeah, I think so. case in point, gregwork was here earlier and a volume attach failed and there was no indication. I told him to look at instance actions based on the resize/migration thing from yesterday19:43
dansmithwhich might be a weird state and might be "resize never happend"19:43
melwittand in there, it showed the volume attach with 'error' as the result19:44
* mriedem deletes the todo19:44
dansmithmelwitt: I think gregwork is just lonely and wanted to talk to someone and not read docs, so not sure that would have helped him19:44
* dansmith runs19:44
melwittso there's already a doc? what did I miss19:45
mriedemi was joking,19:45
* dansmith shakes his head19:45
mriedemhow do i say politely, if you guys want to write the doc(s), go ahead19:46
* gregwork !19:46
melwittI'll copy metacloud's19:46
gregworkheh i was just reporting something strange that i observed.  normally openstack is very vocal about errors that get thrown19:46
gregworki thought it was unusual that not a lot of fuss was happening19:47
gregworkand ultimately heat would just fail to deploy the stack after some time (token expiry)19:48
KeithMnemonicmridem: melwitt: i can help as well and then abandon my hack sitting out there ;-)19:48
gregworkusually if a resource barfs heat catches it and also barfs19:48
gregworkthis went off into space19:48
melwittKeithMnemonic: sorry, what's your hack?19:49
KeithMnemonicusing id only19:49
melwittgregwork: sounds like heat needs to look at INSTANCE ACTIONS19:49
melwittoh that, ok19:49
KeithMnemonicgyee and i played with it a while trying to get the name to work and we never could19:50
melwittthat sounds hard then :)19:50
melwittI'll try to help, it's related to the server event thing, so hopefully once figured out once it will work for all19:51
melwittI don't have that much experience with osc19:51
*** ociuhandu has quit IRC19:52
openstackgerritMerged openstack/nova stable/rocky: Don't delete compute node, when deleting service other than nova-compute
openstackgerritMerged openstack/nova master: tests: Use consistent URL regex substitution
melwittgregwork: I'm joking but it's probably true. would need to open a bug with them and explain they need to use and parse instance actions19:53
openstackgerritMerged openstack/nova master: Give the policy vision document a facelift
openstackgerritMerged openstack/nova master: test cleanup: Remove skipIf test decorator
melwittto be able to pick up the volume attach fail19:53
KeithMnemonicmelwitt this is what we saw
melwittKeithMnemonic: yeah, I was gonna say, (and gyee's comment says the same) that not even the legacy novaclient can lookup a deleted server by name20:00
KeithMnemonicand some where there is a case statement/loop that tried different lookups and that is where it never finds the server when a name is used20:00
mriedemsince doesn't take a --delete option, to fix the bug there i'd probably handle a NotFound when looking up the server by name or id, and if --os-compute-api-version >= 2.21 then just assume you were given an ID and pass it through20:01
mriedemit'll either work or the user will get a 404 anyway20:01
melwittKeithMnemonic: yeah, like I said, I think even with the legacy client, the underlying binding that osc is using, there's no way to lookup deleted by name20:02
melwittso if you want that, you'd have to add it to the novaclient python binding too, afaik20:02
mriedemi'm not sure that's true - if you're admin you can list servers filtering on deleted=True and name20:02
melwittlist you can, but I'm not sure about like nova show20:03
mriedemright, not in the api - it would be client side sugar20:03
melwittyeah, that's what I mean. novaclient needs something to be able to. unless you're saying on the cli can't and that the python api can20:03
mriedemheh, "Filters the response by a server name, as a string.  You can use regular expressions in the query. For example, the ?name=bob regular expression returns both bob and bobb. If you must match on only bob, you can use a regular expression that matches the syntax of the underlying database server that is implemented for Compute, such as MySQL or PostgreSQL."20:03
mriedem"if you know the cloud you're talking to is using mysql or postgres, just fashion your filtering that way"20:04
melwittI'm super rusty on the clients, gotta get back in there20:05
artommriedem, sql injection as a service20:05
KeithMnemonicand also the using "id" only started working after I applied mriedem's fix for the marker. Gentle plug, any change to get those last few you submitted merged?
mriedemfwiw i would not spend a bunch of time trying to build "find deleted service by name using list" for this action events osc bug - it's likely more of a corner case, only possible to do that with admin anyway (list deleted servers and filter by name), i would just make it work if an id is provided and not a name, like i said above20:11
mriedemyou don't have to know if it's an id or a name, just:20:11
mriedem1. if find by name or id returns NotFound20:11
mriedem2. if microversion >= 2.2120:11
mriedem3. then pass through to the api and assume it works20:11
mriedemprobably update the osc docs as well to say if you're trying to list actions for a deleted server, you must provide the server id20:12
*** jcosmao has joined #openstack-nova20:13
KeithMnemonicmelwitt do you me to abandon my patch or do you want to add to it,. the code there does work with marker being an ID20:14
mriedemKeithMnemonic: she's talking about fixing a different bug20:15
mriedemdifferent command20:15
openstackgerritMatt Riedemann proposed openstack/nova master: Add contributor doc for resize and cold migrate
openstackgerritMatt Riedemann proposed openstack/nova master: Add sequence diagrams to resize/cold migrate contrib doc
*** eharney has joined #openstack-nova20:16
KeithMnemonicah ok20:21
*** ociuhandu has joined #openstack-nova20:23
KeithMnemonici thought you were talking about the osc behavior sean-k-mooney mentioned above when you referenced my bug20:23
*** abaindur has joined #openstack-nova20:23
melwittwell, I thought maybe the two (not being able to get --deleted) had the same root cause but they might not. I didn't look that deeply at it yet nor do I already know that much about osc20:24
melwittbut yeah actually we can already tell they're not because KeithMnemonic's problem is only around the name20:26
melwittmriedem: ack, that was my intention, make it work with id. because even the legacy novaclient can't do name for the instance action list20:27
melwittfor deleted servers20:27
*** abaindur has quit IRC20:28
*** ociuhandu has quit IRC20:28
KeithMnemonicmy bug is with id also20:29
melwittI thought you said it works with id earlier20:30
KeithMnemonicthe nova cli, sorry the person who reported it to us used id20:30
KeithMnemonicthis worked nova  list --deleted --marker c244e139-98fe-44eb-a971-c067954da02620:30
KeithMnemonicthis did not openstack server list --delete --marker f1ed3399-b4d3-4c74-a8aa-4b5b935bc8d120:31
*** luksky has joined #openstack-nova20:32
KeithMnemonicbut with that patch i have up, marker works20:32
KeithMnemonicsorry for the confusion20:32
*** damien_r has joined #openstack-nova20:35
*** slaweq has joined #openstack-nova20:38
*** damien_r has quit IRC20:41
artommriedem, did you mean to not push the sequence diagram?20:45
artomBecause... you didn't.20:46
mriedemartom: but i....did20:48
mriedem"A sequence diagram is sorely needed for this as well but that will come in a separate change."20:48
*** damien_r has joined #openstack-nova20:48
artomgdi it's on top20:49
artommriedem, sorry20:49
*** rcernin has joined #openstack-nova20:49
*** damien_r has quit IRC20:55
*** slaweq has quit IRC20:56
openstackgerritMerged openstack/nova master: Remove '/os-tenant-networks' REST API
openstackgerritMerged openstack/nova master: nova-net: Remove 'USE_NEUTRON' from functional tests
openstackgerritMerged openstack/nova master: docs: Remove 'adv-config', 'system-admin' subdocs
openstackgerritMerged openstack/nova master: docs: Replacing underscores with dashes
openstackgerritMerged openstack/nova master: docs: Strip '.rst' suffix
openstackgerritMerged openstack/nova master: Don't claim that CLI user data requires manual base64 encoding
openstackgerritMerged openstack/nova master: Make it easier to run a selection of tests relevant to ongoing work
melwittKeithMnemonic: if I were you I'd make it work with ID only and forget about name for now. the name requires a further change to python-novaclient (AFAIK) and it's already standard operating procedure that some APIs or options only work with ID. not ideal perhaps but it is what it is. just MHO21:05
KeithMnemonicok so my patch does that ;-)21:06
melwittyeah, you need to update it no?21:06
KeithMnemonici can21:06
melwittupdate the doc and everything mriedem pointed out on the patch21:06
KeithMnemonicas long as it seems everyone here is ok with the concept21:07
melwittto make sure marker referenced by ID is consistent21:07
KeithMnemonicthanks for the help21:07
melwittyeah I think so, unless I'm totally missing something, which I might be. people letting me know lately that I've missed a lot ;)21:07
melwittI don't see why doing it ID only would not be ok21:08
*** eharney has quit IRC21:08
KeithMnemonicok thanks again, any chance i can get some movement on that other patch by mriedem for rocky that fixes the marker? he made a ton of progress before the summit but it seems to have stalled21:10
*** nweinber__ has quit IRC21:10
KeithMnemonicit is merged through stein now21:10
*** abaindur has joined #openstack-nova21:11
melwittyeah. my head's spinning with all the stuff I'm trying to do but I'll try to get those. they're not 100% easy bc there's differences and conflicts to look at21:11
*** ociuhandu has joined #openstack-nova21:12
*** mgagne_ is now known as mgagne21:13
melwittKeithMnemonic: I guess the sticking point on the osc one is that it's a change in behavior? does that mean that marker by name works with non-deleted servers?21:15
KeithMnemonicyes it does21:16
melwittahhhh dammit21:17
KeithMnemonicso maybe some logic needs to go around the change to only do it if -deleted is passed21:17
melwittyeah like, do some extra stuff (client gymnastics) to get the server by name first then call with marker if --deleted is passed21:18
KeithMnemonicthat way it does not break anything but makes -deleted work in once case at least21:18
melwitt*with marker as ID if --deleted is passed21:18
melwittyeah ok. I get it now21:18
KeithMnemoniclet me spin up a fresh devstack and start hacking on it again21:19
melwittyeah so I rewind from what I said earlier and I'd do the extra step if --deleted is passed. grab the server id and then pass marker as id under the covers21:19
openstackgerritMerged openstack/nova master: Remove 'os-consoles' API
*** awalende has joined #openstack-nova21:23
mriedemheh, this is a bit old
mriedemapi -> compute done!21:26
*** awalende has quit IRC21:28
openstackgerritMatt Riedemann proposed openstack/nova master: api-guide: flesh out BUILD and ACTIVE server create transitions
*** spatel has joined #openstack-nova21:38
*** spatel has quit IRC21:39
openstackgerritMatt Riedemann proposed openstack/nova master: api-guide: fix the file injection considerations drift
openstackgerritMerged openstack/nova master: Avoid raise InstanceNotRunning exception
openstackgerritMatt Riedemann proposed openstack/nova master: api-guide: remove empty sections about inter-service interactions
openstackgerritMerged openstack/nova master: Start README.rst with a better title
*** abaindur has quit IRC22:05
*** tbachman has quit IRC22:06
*** eharney has joined #openstack-nova22:06
*** spatel has joined #openstack-nova22:08
*** spatel has quit IRC22:13
*** spatel has joined #openstack-nova22:15
*** jcosmao has left #openstack-nova22:16
openstackgerritMatt Riedemann proposed openstack/nova master: api-guide: flesh out flavor extra specs and image properties
*** rcernin has quit IRC22:24
*** slaweq has joined #openstack-nova22:24
*** slaweq has quit IRC22:29
openstackgerritMatt Riedemann proposed openstack/nova master: api-guide: flesh out networking concepts
melwittefried: question on test cleanup
efriedI think you're probably right. Lemme check one thing quick...22:35
melwittok. I wasn't sure how this is supposed to be22:36
efriedYup, I think that was just a miss melwitt, good spot, will remove.22:37
*** slaweq has joined #openstack-nova22:37
openstackgerritEric Fried proposed openstack/nova master: test cleanup: Make base TestCase subclass oslotest
efriedmelwitt: done, thanks.22:38
efriedthanks for those reviews melwitt22:39
efrieda day when my dashboard shrinks is a good day :)22:39
efriedand johnthetubaguy ^22:40
melwittnp, nice cleanups22:41
*** slaweq has quit IRC22:42
openstackgerritMerged openstack/nova master: Remove 'nova-console' service, 'console' RPC API
openstackgerritMerged openstack/nova master: nova-net: Remove 'networks' quota
*** KeithMnemonic has quit IRC22:44
*** slaweq has joined #openstack-nova22:45
openstackgerritMatt Riedemann proposed openstack/nova master: api-guide: flesh out todos in user doc
*** slaweq has quit IRC22:50
*** ociuhandu has quit IRC22:51
*** spatel has quit IRC22:58
openstackgerritMatt Riedemann proposed openstack/nova master: doc: remove admin/manage-users
*** mriedem is now known as mriedem_away23:00
*** luksky has quit IRC23:04
efrieddansmith: you said something about hard reboot being admin-only by default, but I can't find how that's the case in the code, where it looks like soft/hard are under the same umbrella. Did I misunderstand?23:13
*** slaweq has joined #openstack-nova23:15
*** jmlowe has joined #openstack-nova23:15
openstackgerritMerged openstack/nova master: test cleanup: Use oslotest's Timeout fixture
openstackgerritMerged openstack/nova master: test cleanup: Use oslotest's CaptureOutput fixture
*** slaweq has quit IRC23:19
*** ociuhandu has joined #openstack-nova23:34
dansmithefried: that wasn't me23:37
dansmithat least, not the hard reboot nova action23:38
dansmithI talked a lot about *host* reboot...23:38
efrieddammit, that's probably where I got confused.23:38
*** slaweq has joined #openstack-nova23:42
*** tosky has quit IRC23:45
*** slaweq has quit IRC23:52
*** tbachman has joined #openstack-nova23:56

Generated by 2.15.3 by Marius Gedminas - find it at!