Friday, 2020-02-07

*** mriedem has joined #openstack-nova00:10
mriedemdansmith: a few comments in your reno00:11
*** mriedem has quit IRC00:20
*** tbachman has joined #openstack-nova00:20
openstackgerritSundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec.  https://review.opendev.org/63124300:36
openstackgerritSundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event.  https://review.opendev.org/69270700:36
openstackgerritSundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs.  https://review.opendev.org/63124400:36
openstackgerritSundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager.  https://review.opendev.org/69858100:36
openstackgerritSundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into domain XML in libvirt driver.  https://review.opendev.org/63124500:36
openstackgerritSundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted.  https://review.opendev.org/67373500:36
openstackgerritSundar Nadathur proposed openstack/nova master: Enable hard/soft reboot with accelerators.  https://review.opendev.org/69794000:36
openstackgerritSundar Nadathur proposed openstack/nova master: Enable start/stop of instances with accelerators.  https://review.opendev.org/69955300:36
openstackgerritSundar Nadathur proposed openstack/nova master: Enable and use COMPUTE_ACCELERATORS trait.  https://review.opendev.org/69955400:36
openstackgerritSundar Nadathur proposed openstack/nova master: Bump compute rpcapi version and reduce Cyborg calls.  https://review.opendev.org/70422700:36
openstackgerritSundar Nadathur proposed openstack/nova master: Add cyborg tempest job.  https://review.opendev.org/67099900:36
*** damien_r has joined #openstack-nova00:48
*** TxGirlGeek has quit IRC00:52
*** damien_r has quit IRC01:04
*** nweinber has joined #openstack-nova01:14
*** damien_r has joined #openstack-nova01:15
*** tbachman has quit IRC01:16
*** damien_r has quit IRC01:18
*** mlavalle has quit IRC01:57
*** gyee has quit IRC02:03
*** Liang__ has joined #openstack-nova02:12
*** vishalmanchanda has joined #openstack-nova02:18
*** Dinesh_Bhor has quit IRC02:21
*** tbachman has joined #openstack-nova02:26
*** brinzhang__ has joined #openstack-nova02:31
*** brinzhang has joined #openstack-nova02:34
*** brinzhang_ has quit IRC02:35
*** brinzhang__ has quit IRC02:36
*** nweinber has quit IRC02:54
*** nweinber has joined #openstack-nova03:06
*** mkrai has joined #openstack-nova03:12
*** yikun has joined #openstack-nova03:30
*** nweinber has quit IRC03:34
*** psachin has joined #openstack-nova03:35
*** mdbooth has quit IRC03:58
*** mdbooth has joined #openstack-nova04:00
*** tetsuro has quit IRC04:13
*** tetsuro has joined #openstack-nova04:13
*** damien_r has joined #openstack-nova04:47
*** damien_r has quit IRC04:47
*** damien_r has joined #openstack-nova04:47
*** udesale has joined #openstack-nova04:49
*** abhishekk|out is now known as abhishekk05:03
*** gmann has quit IRC05:27
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-nova05:34
*** ociuhandu has joined #openstack-nova05:53
*** ociuhandu has quit IRC05:57
*** owalsh has quit IRC06:01
*** owalsh has joined #openstack-nova06:04
*** Dinesh_Bhor has joined #openstack-nova06:08
*** ratailor has joined #openstack-nova06:15
*** Liang__ has quit IRC06:17
*** dtantsur|afk is now known as dtantsur07:27
*** dpawlik has joined #openstack-nova07:28
*** lpetrut has joined #openstack-nova07:29
*** lpetrut has quit IRC07:30
*** lpetrut has joined #openstack-nova07:31
*** ccamacho has joined #openstack-nova07:35
*** dpawlik has quit IRC07:44
*** dpawlik has joined #openstack-nova07:44
*** slaweq__ has joined #openstack-nova08:09
*** amoralej|off is now known as amoralej08:09
*** rpittau|afk is now known as rpittau08:10
*** tkajinam has quit IRC08:12
*** slaweq has joined #openstack-nova08:18
*** iurygregory has joined #openstack-nova08:19
*** slaweq__ has quit IRC08:19
gibiefried: ack, I will try to spend some time on the cyborg series today08:29
*** tesseract has joined #openstack-nova08:36
*** ivve has joined #openstack-nova08:37
*** ralonsoh has joined #openstack-nova08:41
*** xek has joined #openstack-nova08:44
*** tetsuro has quit IRC08:46
*** tosky has joined #openstack-nova08:48
*** Liang__ has joined #openstack-nova08:54
*** mlycka has joined #openstack-nova09:05
*** obondarev has joined #openstack-nova09:08
*** obondarev has left #openstack-nova09:08
*** david-lyle has quit IRC09:16
*** david-lyle has joined #openstack-nova09:16
*** martin_midolesov has joined #openstack-nova09:25
*** mvkr has joined #openstack-nova09:38
*** mlycka has quit IRC09:42
*** ociuhandu has joined #openstack-nova09:45
*** ratailor has quit IRC09:45
*** derekh has joined #openstack-nova09:46
*** ociuhandu has quit IRC09:51
*** ociuhandu has joined #openstack-nova10:02
*** abhishekk is now known as abhishekk|away10:11
*** slaweq has quit IRC10:13
*** slaweq has joined #openstack-nova10:15
stephenfingibi, lyarwood: real quick, do either of you see anything obviously wrong in https://zuul.opendev.org/t/openstack/build/5b196c5a7cf944df8857a08dc15aa79f/console ?10:22
stephenfinIt's saying "TypeError: create_port_binding() missing 2 required positional arguments: 'port_id' and 'data'"10:22
stephenfinBut line it's failing on is obviously passing those?10:22
stephenfin"binding = client.create_port_binding(port_id, data)['binding']"10:22
stephenfingibi: Also, low priority but I updated a commit message on a patch you'd previously approved, if you have time to revisit https://review.opendev.org/70565510:23
gibistephenfin: could you link the patch that produced such test result?10:24
stephenfingibi: https://review.opendev.org/#/c/706295/10:24
gibiinteresting :)10:25
stephenfinright?10:25
gibipulling it down10:25
gibitrying to reproduce locally..10:29
gibistephenfin: nova.tests.fixtures.NeutronFixture.get_port_binding has get_port_binding(self, context, client, port_id, host) signature so when the code pass two positional args it fills only client and context and not the port and host10:34
stephenfingdi10:34
* stephenfin hangs head in shame10:34
stephenfingibi++ Thanks /o\ :)10:34
gibino problem10:34
stephenfinI'd been staring at that for an hour10:34
gibithe error message was deeply missleading10:34
stephenfinYuuup. Bad coincidence :)10:35
gibiyepp10:35
*** priteau has joined #openstack-nova10:46
*** vishalmanchanda has quit IRC10:50
*** purplerbot has quit IRC10:50
openstackgerritStephen Finucane proposed openstack/nova master: Remove universal wheel configuration  https://review.opendev.org/70646610:50
*** purplerbot has joined #openstack-nova10:51
*** slaweq_ has joined #openstack-nova10:51
*** slaweq has quit IRC10:53
*** bauzas has quit IRC10:57
*** bauzas has joined #openstack-nova10:57
openstackgerritBrin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies  https://review.opendev.org/70647011:02
lyarwoodstephenfin: sorry missed that, still need me to take a look?11:05
stephenfinlyarwood: Nope, we sussed it11:05
lyarwoodwonderful11:05
openstackgerritStephen Finucane proposed openstack/nova master: WIP: Use neutronclient's port binding APIs  https://review.opendev.org/70629511:06
*** yikun has quit IRC11:09
*** rpittau is now known as rpittau|bbl11:18
*** mkrai has quit IRC11:31
*** tbachman has quit IRC11:35
*** udesale_ has joined #openstack-nova12:04
*** udesale has quit IRC12:07
*** gmann has joined #openstack-nova12:08
*** slaweq__ has joined #openstack-nova12:10
*** mkrai has joined #openstack-nova12:10
*** slaweq_ has quit IRC12:11
*** nicolasbock has joined #openstack-nova12:13
*** ociuhandu has quit IRC12:27
*** ociuhandu has joined #openstack-nova12:28
*** ratailor has joined #openstack-nova12:36
*** mkrai has quit IRC12:36
*** ociuhandu has quit IRC12:42
*** spatel has joined #openstack-nova12:43
*** spatel has quit IRC12:50
*** ratailor has quit IRC12:53
*** damien_r has quit IRC13:06
*** tbachman has joined #openstack-nova13:06
*** ociuhandu has joined #openstack-nova13:08
*** slaweq has joined #openstack-nova13:13
*** rpittau|bbl is now known as rpittau13:14
*** bbowen_ has joined #openstack-nova13:14
*** bbowen has quit IRC13:15
*** slaweq__ has quit IRC13:15
*** hongbin has joined #openstack-nova13:15
*** Luzi has joined #openstack-nova13:16
*** dtantsur is now known as dtantsur|brb13:26
*** davidsha has joined #openstack-nova13:27
*** damien_r has joined #openstack-nova13:37
*** gmann_ has joined #openstack-nova13:40
*** gmann has quit IRC13:40
*** gmann_ is now known as gmann13:40
*** gmann has quit IRC13:40
*** gmann has joined #openstack-nova13:41
*** nweinber has joined #openstack-nova13:44
*** martin_midolesov has quit IRC13:47
*** READ10 has joined #openstack-nova13:50
*** hongbin has quit IRC13:54
*** eharney has joined #openstack-nova13:59
*** yan0s has joined #openstack-nova14:00
*** amoralej is now known as amoralej|lunch14:07
*** lbragstad has quit IRC14:17
openstackgerritHuachang Wang proposed openstack/nova-specs master: Use PCPU and VCPU in one instance  https://review.opendev.org/66865614:19
*** tesseract-RH has joined #openstack-nova14:19
*** lbragstad has joined #openstack-nova14:19
*** tesseract has quit IRC14:21
*** spatel has joined #openstack-nova14:30
huaqiangstephenfin:sean-k-mooney:alex_xu spec https://review.opendev.org/668656 is updated, hope you guys have time to review it before freezing day14:30
*** jmlowe has joined #openstack-nova14:30
*** tesseract has joined #openstack-nova14:34
*** spatel has quit IRC14:34
*** ociuhandu has quit IRC14:35
*** tesseract has quit IRC14:43
*** tesseract-RH has quit IRC14:43
*** tesseract has joined #openstack-nova14:44
*** Luzi has quit IRC14:45
stephenfingibi: Comments left on https://review.opendev.org/#/c/704759/14:49
stephenfinhuaqiang: Ack. I'll hit that next Tuesday (spec review day)14:49
*** Sundar has joined #openstack-nova14:52
huaqiangstephenfin: appreciate!14:55
*** tbachman has quit IRC14:55
gibistephenfin: thanks. I will check14:56
*** amoralej|lunch is now known as amoralej14:56
openstackgerritBrin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies  https://review.opendev.org/70647015:01
*** rpittau is now known as rpittau|brb15:04
openstackgerritMartin Midolesov proposed openstack/nova master: Implementing graceful shutdown.  https://review.opendev.org/66624515:05
*** lpetrut has quit IRC15:08
*** jmlowe has quit IRC15:15
*** ociuhandu has joined #openstack-nova15:15
*** ivve has quit IRC15:15
*** ociuhandu has quit IRC15:16
*** ociuhandu has joined #openstack-nova15:17
*** rpittau|brb is now known as rpittau15:20
openstackgerritBrin Zhang proposed openstack/nova master: Add SYSTEM_READER role to servers actions API  https://review.opendev.org/70617915:22
*** udesale_ has quit IRC15:23
*** udesale_ has joined #openstack-nova15:24
*** spatel has joined #openstack-nova15:25
kashyapefried: Want to have a stab at this spec?  Already approved previously; and has a +2: https://review.opendev.org/#/c/693844/15:26
Sundardansmith, gibi, sean-k-mooney, efried: About https://review.opendev.org/#/c/706083/, this is a refactor that affects pci requests, numa topology etc. that are not related to the Cyborg patch. I see that some of you are noncommittal and, in any case, probably needs further discussion. The Cyborg series is already long in the tooth; if  we add this15:29
Sundargeneral refactor, keeping that entire context in our minds will only prolong our efforts and strain. Plus, some of us are hitting time constraints. What do you think of considering the series as it is now, and feel free to add refactors on top of that later? Both of them could even go concurrently, but the discussions on the refactor need not hold15:29
Sundarup the main series.15:29
*** mriedem has joined #openstack-nova15:29
SundarJust to be clear, I am not objecting to the refactoring. I am fine either way and can contribute to the reviews.15:31
sean-k-mooneySundar: im currently planing to redeploy with the code you pushed yesterday.15:33
Sundarsean-k-mooney: Thanks.15:33
*** tbachman has joined #openstack-nova15:33
sean-k-mooneyi found that with the previous version i was unable to delete the cyborg vm15:33
*** ociuhandu_ has joined #openstack-nova15:33
sean-k-mooneythe conductor was not able to delete the arp binding15:34
sean-k-mooneyi was able to manually do it but could still not delete the nova vm15:34
sean-k-mooneyill see if i hit the same issue with the new version and let you know15:34
SundarI see. In the worst case, for development purposes only, once could delete the ARQs using curl.15:35
sean-k-mooneyyes but you still cannot delte the nova vm15:36
SundarSure. You are also updating the Cyborg side, right?15:36
sean-k-mooneyyep15:36
sean-k-mooneyi delete all my repos and confirm it pulle the correct version if i also set CYBORG_REPO and CYBORG_BRANCH15:36
SundarCould you expand on what you mean by Nova VM? Different from a devstack all-in-1 setup?15:37
*** jmlowe has joined #openstack-nova15:37
*** ociuhandu has quit IRC15:37
sean-k-mooneyi mean if i do openstack server delete <uuid of vm with fake cyborg device> then you cant delete it15:37
*** igordc has joined #openstack-nova15:37
gibiSundar: I agree that such refactor could move the focus away from the goal to merge the cyborg integration code to nova. And I understand the time pressure here. I'm fine doing the refactor on top of the series15:37
sean-k-mooneythe conductor got an unexpected reponce code form cyborg and the delete failed15:38
gibiSundar: I'm reviewing the series as we speak15:38
sean-k-mooneySundar: this could be related to the error handeling you and dansmith were talking about. i did not dig into it as it was late when i tried to delete the vm15:39
*** ociuhandu_ has quit IRC15:40
sean-k-mooneySundar: im going to recreate teh env with the latest version and see if the issue is still there15:40
*** ociuhandu has joined #openstack-nova15:42
*** ociuhandu has quit IRC15:47
*** spatel has quit IRC15:48
*** david-lyle is now known as dklyle15:53
Sundargibi: Thanks15:54
Sundarsean-k-mooney: Please let me know how it goes. I may get on phone calls once in a while but I'll be around all morning PST.15:55
*** bnemec has quit IRC15:55
*** bnemec has joined #openstack-nova15:56
efriedSundar: The refactor is already "on top" of the series, not folded in the middle. It shouldn't block anything else.15:58
*** jmlowe has quit IRC15:58
efried(It's on top of a patch that's in the middle, but it's its own tip)15:58
efriedI just put it out there to get agreement with dansmith about what the refactor should look like, and to satisfy myself that it could be done without too much trouble, so that I could be okay with the preceding patch as it is.15:59
efriedkashyap: ack, in queue15:59
kashyapThank you.16:00
*** TxGirlGeek has joined #openstack-nova16:01
sean-k-mooneyefried: which patch is the refactor https://review.opendev.org/#/c/706083/1 you dont mean https://review.opendev.org/#/c/704227/11 right16:03
sean-k-mooneyim planning to deploy https://review.opendev.org/#/c/704227/1116:03
efriedthe former16:04
sean-k-mooneycool16:04
mriedemdansmith: +2 on your hidden=null fix in case efried or melwitt want to approve16:05
*** spatel has joined #openstack-nova16:06
spatelsean-k-mooney: hey! i have few more question related NUMA, I did lots of testing with erlang and have pretty good result with numa tuning16:07
sean-k-mooneythats good to hear16:08
sean-k-mooneyso memory seams to be your bottleneck then16:08
spatelwhen i create vm with 16vcpu core just running on single numa node then result is freaking good16:08
dansmithmriedem: thanks16:08
spatelbut if i create vm with 32vCPU and set numa_node=2 in flavor in that case result is 70% worst16:09
spatelnow question is how do i tell openstack to pin down vCPU with pCPU  1-0-1 ?16:09
spatelI do have hw:cpu_policy=dedicated policy but its randomly mapping vCPU <--> pCPU16:10
spatelsean-k-mooney: what do you suggest here?16:11
spatelIn short i want to make my VM fully NUMA map like bare metal has.16:11
*** sapd1_x has joined #openstack-nova16:13
sean-k-mooneyspatel: so it look like the erlang applciation is not numa aware16:14
*** N3l1x has joined #openstack-nova16:14
spatelsean-k-mooney: but if same erlang application i am running on bare metal it does work and result is good16:14
sean-k-mooneymeaning it is internally not numa affinitising its memory allocations16:14
sean-k-mooneyspatel: wehn you set hw:cpu_policy deicated it does a 1:1 mapping between cores on a virtual numa node to the a host numa node16:16
spateli thought if somehow openstack pin vcpu0 <--> pcpu0 , vcpu1 <--> pcpu1  so on... that would be great16:16
sean-k-mooneyso its not random  it is numa aware16:16
sean-k-mooneyspatel: no it does not by design16:16
spatelsean-k-mooney: look at this current mapping - http://paste.openstack.org/show/789225/16:17
sean-k-mooneybut it will map all fo the cores in one geust numa node to the a singel numa node on the host16:17
sean-k-mooneythats takeing a long time to load for some reason16:17
sean-k-mooneypaste.openstack.org is down for me it seams16:18
lyarwoodhttps://review.opendev.org/#/c/701430/ - trying to get my head around the required api samples for a microversion that changes behaviour but nothing in the request or response, does anyone know what I would need to add here?16:19
sean-k-mooneyok it finally loaded16:19
sean-k-mooneyspatel: how many cores to you have on the host16:20
sean-k-mooneyim assuming the are 16 core cpus and you have hypter treading enabled?16:20
sean-k-mooneyspatel: can you provide the host capablities xml and the full guest xml16:22
spatelGive me few min.. i am on phone..16:22
sean-k-mooneyspatel: ok that xml fragment does not really look correct.16:24
efrieddansmith: I can push https://review.opendev.org/#/c/706331/ if you don't plan to reword that reno.16:36
dansmithefried: it makes sense to me and nobody else has suggestions right?16:37
*** slaweq has quit IRC16:37
*** tesseract has quit IRC16:38
efrieddansmith: let me take a real swing at a reword. If I can't come up with something I like, I'll push it.16:38
*** xek has quit IRC16:39
efrieddansmith: do db migrations happen automatically as part of the upgrade process, or do users have to trigger them manually? (Or does it depend on the distro?)16:41
dansmithefried: depends a lot16:41
*** yan0s has quit IRC16:43
*** gyee has joined #openstack-nova16:44
*** udesale_ has quit IRC16:45
efrieddansmith: done, see what you think.16:46
*** udesale_ has joined #openstack-nova16:46
efrieddansmith: for me, the distinction of "apply the fix" versus "install a version that includes the fix" is where the confusion lies.16:46
dansmithI totes don't get the substantive difference between your sentences and mine, but as such I'll be glad to change it.. just a sec16:47
*** udesale_ has quit IRC16:48
efriedmriedem: if you're around -- does my suggestion improve anything IYO?16:48
*** udesale_ has joined #openstack-nova16:48
openstackgerritDan Smith proposed openstack/nova master: Fix instance.hidden migration and querying  https://review.opendev.org/70633116:48
efriedIt becomes hard to see anymore having discussed it to death16:48
efrieddansmith: coulda added the link while you were at it :P16:49
dansmithguh, will fix16:49
dansmithworking on the patch above this so I wanted to get back to that16:49
openstackgerritDan Smith proposed openstack/nova master: Fix instance.hidden migration and querying  https://review.opendev.org/70633116:50
dansmithefried: please check before I roll my tree forward again :)16:50
*** udesale_ has quit IRC16:50
*** udesale_ has joined #openstack-nova16:51
efrieddansmith: I'll build locally to make sure. You may need an extra newline between the paragraph and the link def. Also would be nice to link same from the `fixes` note (you don't need / can't include a second target def, they'll both use the same one)16:51
mriedemyou do16:52
mriedemit blows up without that16:52
openstackgerritDan Smith proposed openstack/nova master: Fix instance.hidden migration and querying  https://review.opendev.org/70633116:52
mriedemhttp://rst.ninjs.org/16:52
dansmithlike that?16:52
efrieddansmith: add trailing newline on the fixes one16:53
efrieds/newline/undescore/16:53
dansmith"for all you people who have now corrupted their database, I hope you appreciate that we waited quite a while to make sure the bug link was perfect"16:53
efriedYeah, these five minutes are really going to make the difference.16:53
dansmitha trailing newline where16:53
dansmith?16:53
efriedIf you want to go back to real work, I'll doodle with the formatting and fast approve it.16:53
efriednot a newline, the second bug link is backticked but not underscored.16:54
dansmithno, it's very important to me that I get this right16:54
mriedem.. _bug 1862205: https://launchpad.net/bugs/186220516:54
openstackLaunchpad bug 1862205 in OpenStack Compute (nova) "Instances not visible when hidden=NULL" [Critical,In progress] - Assigned to Dan Smith (danms)16:54
dansmithoh dammit16:54
mriedemyou need the ..16:54
openstackgerritDan Smith proposed openstack/nova master: Fix instance.hidden migration and querying  https://review.opendev.org/70633116:55
*** hongbin has joined #openstack-nova16:55
efriedthis would be a great time for that fast-reno tool I've been meaning to write. The releasenotes target takes forEVER to build.16:55
efriedthere, yeah, ps9 looks right to me.16:55
dansmithttfl16:56
stephenfinefried: We need to stop 'rm -rf' ing the built docs16:56
efriedis that what does it?16:56
efriedI'll try that16:57
*** tesseract has joined #openstack-nova16:57
mriedemefried: dansmith: so PS9 might work, but i've seen reno complain about links in weird ways between sections, so we'll see if the link defined in the first section is honored in the second16:57
dansmithmriedem: efried said it was only needed once16:57
mriedemi guess we'll know in a couple of hours :)16:57
efriedshould be the case, but it depends how reno does its build.16:58
stephenfinSphinx builds doctrees and reuses those when possible. Unfortunately it occasionally breaks if you e.g. remove a document that previously existed (like by changing to an older branch)16:58
stephenfinprobably less on an issue for renos16:58
*** Sundar has quit IRC16:58
*** dpawlik has quit IRC16:58
*** hongbin has quit IRC16:59
*** spatel has quit IRC17:00
efriedstephenfin: In the specs repo I wrote a tool that would only build one directory at a time. It was a fiddly thing to figure out exactly how to word the sphinx-build command. In this case... if I could restrict it to building just $release.rst it would probably be equivalent.17:01
*** mriedem has quit IRC17:02
*** rpittau is now known as rpittau|afk17:02
*** mriedem has joined #openstack-nova17:03
efrieddansmith, mriedem: local build is green, the link works in both spots as expected. Fast approving.17:09
*** jmlowe has joined #openstack-nova17:09
efried...mriedem do you agree with the new wording? Or at least it's not worse?17:10
efrieddansmith: we shouldn't have to backport the fixture poison beyond Train, right? because we wouldn't be backporting any future fix that hasn't gone through that fixture in Train+17:12
dansmithefried: we wouldn't backport that anyway17:12
openstackgerritDan Smith proposed openstack/nova stable/train: Fix instance.hidden migration and querying  https://review.opendev.org/70658217:14
dansmithbackport ^17:14
efried+117:15
efriedstephenfin: fwiw commenting out the 'rm' doesn't seem to be speeding anything up. It's still scanning the world, which seems to be the part that takes forever.17:17
stephenfindarn17:17
stephenfinI was thinking all that would be cached17:17
efriedstephenfin: considering a target that patches index.rst and (re)moves $not-this-release.rst before building.17:18
*** davidsha has quit IRC17:18
gibiefried, dansmith: do we have re-schedule handling implemented in the cyborg series?17:18
efried...and then of course it would have to undo that after17:18
dansmithgibi: unsure, I'm kinda depending on sean-k-mooney to poke at some of those things17:18
efriedgibi: How is that handled for bw? I would have thought it would be the same way.17:19
stephenfinefried: Have we closed any more branches yet? https://github.com/openstack/nova/commit/857b5003ccc0b37f4642ed77d9f0d08f9ee28dfb17:19
dansmithyeah, I think it should be the same for ports even, but actually showing that it works is different of course17:19
stephenfinI guess with EM we no longer do that17:19
efriedstephenfin: yeah, was thinking about that17:19
efrieddo we truly never EOL anything at this point?17:20
gibithe arqs needs to be unbound from the failed dest host and bound to the alternate host. the placement handling shoudl work out of the box, but we at least needs some test coverage to prove17:20
stephenfinMaybe we could just live with the lack of updated release notes for those branches?17:21
dansmithgibi: yup17:21
*** psachin has quit IRC17:21
sean-k-mooneydansmith: how would you like me to test reschdule? i can set up a multi node deployment i guess and maybe kill libvirt?17:22
efriedgibi: agree. At a glance, it's weird that delete_arqs_for_instance is first introduced here https://review.opendev.org/#/c/673735/17:22
dansmithsean-k-mooney: I dunno.. if that'd work then maybe that's easy, or just throw a raise into the code on one node?17:24
gibisean-k-mooney: I think if you kill libvirt then the compute service will go down and the scheduler will not select it as a target17:24
efriedstephenfin: I removed ocata-train rsts and from index, it's still taking bloody forever. It's the scanning that's slow.17:24
stephenfinefried: dhellmann told me there was a way to configure earliest-version to previous each one going back forever17:25
sean-k-mooneyif i do a multinode deployment then maybe i could kill ovs or the cyborg agent instead17:25
dansmithgibi: well, if you disable the compute filter you can make that work still I think17:25
gibiSundar, dansmith, efried: anyhow I left a -1 to get some answer / re-schedul test coverage from Sundar https://review.opendev.org/#/c/631244/61/nova/tests/functional/test_servers.py@762517:25
sean-k-mooneyso that the vif plugin would fail17:25
efriedstephenfin: I imagine it'd be a matter of fixing the reno.sphinxext code itself17:25
dansmithgibi: ack17:25
gibidansmith: true, that can be done17:25
gibianother question, does the series gracefully handle that move operations are not supported yet with accelerators?17:26
stephenfinefried: I think earliest-version would do the trick. Let me try with that17:26
dansmithgibi: probably not, but I'm trying to think of other examples of things we know don't work where we handle that gracefully17:29
gibidansmith: qos ports was implemented a check at the API level17:29
dansmithgibi: there are some cases where we don't really know until the virt driver(s) get involved, which is *kinda* this case17:29
gibidansmith: here we know that nova does not support migrating an instance with arq yet, so we should reject that17:30
dansmithgibi: okay but that's kindof a high-level thing because the coordination needs to be done above the compute (i.e. with neutron) anyway right?17:30
dansmithgibi: you mean libvirt I assume17:30
stephenfinefried: nope, still a couple of minutes and fans running at full blast :)17:31
dansmithwell, hmm, I was going to say the nova bits would still try, but maybe not because the conductor kicks the bind.. /me looks17:31
gibiI think not just libvirt but also nova needs to grow support for migration with arqs, like re-querying the resource request of the arq at the start of the migration to include them into the scheduling request17:31
*** dtantsur|brb is now known as dtantsur17:31
*** tbachman has quit IRC17:32
*** udesale_ has quit IRC17:32
dansmithgibi: yeah I guess you're right since we don't persist those in the reqspec17:32
gibiOK, left this as well as a comment in https://review.opendev.org/#/c/631244/6117:33
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-nova17:34
gibiI have to leave soon I will continue reading the series on Monday17:35
dansmithI guess we probably have cold migration handling for pci devices specifically,17:35
dansmithbut I was kinda thinking this would fail for live migration like I think it does for pci, which is.. late in the virt driver IIRC17:35
*** tesseract has quit IRC17:36
gibimy point is that we know we need to write some nova code to support these ops. So while we dont have that code we can reject such ops from the API.17:37
*** ivve has joined #openstack-nova17:38
dansmithgibi: yep, agree, I'm just talking out loud17:38
*** jcosmao has left #openstack-nova17:38
dansmithtalking out loud? thinking out loud :)17:38
* gibi tend to talk out loud while thinking alone 17:38
dansmithI don't want to put api-level barriers to something that is just a virt limitation but you're right, none of the paths where we do the arq stuff gets tickled in the, for example, resize paths17:39
*** tosky has quit IRC17:40
gibiyeah, I thin resize, migrate, live migrate, evacuate, unshelve (after offload) needs some code to recreate the proper resource request for the scheduling17:40
dansmithyup17:40
dansmithgibi: I was telling efried the other day that I had been so tunnel-vision on the bones of this that I wanted him to run through it again to break that up for me17:42
dansmithfor exactly this reason, so thanks for being that force :)17:42
gibiI saw that two of you already handling this seris so  I decided to focus energies elsewhere. but agree that fresh eyes helps to se things differently17:44
dansmithyup17:48
*** ociuhandu has joined #openstack-nova17:50
*** tbachman has joined #openstack-nova17:50
*** Liang__ has quit IRC17:52
*** Liang__ has joined #openstack-nova17:54
*** ociuhandu has quit IRC17:56
*** priteau has quit IRC17:56
*** jmlowe has quit IRC17:56
*** amoralej is now known as amoralej|off17:58
*** mlavalle has joined #openstack-nova18:04
*** derekh has quit IRC18:05
*** spatel has joined #openstack-nova18:10
spatelsean-k-mooney: I am back now, sorry was in back to back meeting18:10
*** igordc has quit IRC18:18
*** artom has quit IRC18:25
*** artom has joined #openstack-nova18:25
*** jmlowe has joined #openstack-nova18:30
sean-k-mooneydansmith: we have a check that blocks live migration if there are pci_request spec object that are not related to neutorn sriov ports18:38
sean-k-mooneydansmith: but that wont block cyborg devices18:38
dansmithsean-k-mooney: okay so something similar for cyborg I guess18:38
sean-k-mooneysince we also dont have pci_request spec objects18:38
sean-k-mooneywell for now we could just check if the flavor has accl:device-profile or whatever teh extraspec is18:39
dansmithright18:39
sean-k-mooneyim not sure we will be able to do that check in the api because we would need to check the embeded flaovr form the cell db. actully no we can. we can grab it from the request_spec in the api db18:40
sean-k-mooneyso ya we can reject the live migration in the api layer until we support that18:40
dansmithnot sure what the problem is.. the api can look at the instance's embedded flavor18:41
sean-k-mooneyyep it can so no problem18:41
sean-k-mooneyi was thinking we might need to do a down call to the cell db to get it but we dont18:41
sean-k-mooneyso all good18:41
dansmithwe do to get the actual embedded flavor, but that's fine of copurse18:42
dansmithanything that does anything on an instance gets the instance record from the cell in the api18:42
sean-k-mooneywell the embeded flavor is aslo stored in the api db in the request spec18:42
dansmithit's stored as it was at the time of creation, but not necessarily the same as what the instance has now18:42
dansmithand if we've done a data migration or something they could have diverged18:43
sean-k-mooneyoh that wont be update after reisze?18:43
sean-k-mooneyah ok18:43
dansmiththe instance's actual flavor is what we should use, and it's no more expensive to get18:43
sean-k-mooneysure makes sense18:43
*** dtantsur is now known as dtantsur|afk18:45
mriedemthe requestspec.flavor is updated as part of a resize, but .... there be bugs18:45
*** igordc has joined #openstack-nova18:49
spatelsean-k-mooney: here is the virsh capability - http://paste.openstack.org/show/789301/18:50
*** jmlowe has quit IRC18:50
spatelhere is the vCPU pinning map - http://paste.openstack.org/show/789302/18:51
sean-k-mooneyok so the first 15 cpus are pinned to host numa node 018:58
spatelYes18:58
sean-k-mooneyand the second 15 are all pinned to host numa node 118:58
sean-k-mooneyso this is correct18:59
spatelYes18:59
spatelif you see all looks correct from VM point of view..18:59
spatelVM correctly mapped its vCPU pins across physical numa nodes19:00
sean-k-mooneyyes so you left out one of the imporatn numa elelmnt form the vm19:00
*** imacdonn has quit IRC19:00
spatel?19:00
*** imacdonn has joined #openstack-nova19:01
sean-k-mooneyyou left out the numa element19:01
mriedemimportant numa elements from the vm19:01
mriedemtranslation-as-a-service19:01
sean-k-mooneyoh ya i miss typed talking downstream as well19:01
spatelI am not following you guys..19:01
sean-k-mooneyill slow down19:01
mriedemremember how this isn't a support channel?19:01
*** Sundar has joined #openstack-nova19:01
* mriedem leaves19:02
sean-k-mooneynova will map the first half of the cpus to numa node 0 in the guess so cores 0-15 and form what you have show it is working correctly19:02
*** mriedem has left #openstack-nova19:02
spatelYes.. that is what i am also saying.. CPU pins correctly mapped out to NUMA nodes19:03
*** martinkennelly has joined #openstack-nova19:03
spatelThis is what NUMA looks inside VM0 -> http://paste.openstack.org/show/789304/19:03
spatelnow question is when i run application why i am seeing poor performance?19:04
Sundargibi: Any pointers to how you handled reschedule for bandwidth provider?19:04
spatelDo you think i need to do something with -> hw:cpu_thread_policy ?19:04
*** tbachman has quit IRC19:05
sean-k-mooneyspatel: it woudl appear the aplication is not correctly optimising for the numa toplogy.19:05
sean-k-mooneyhw:cpu_thread_polciy wont help19:05
sean-k-mooneyyou can alther the cpu thread and socket toplogy in the guest which might but this does not seeam to be a nova/libvirt issue19:06
sean-k-mooneyspatel: my guess is that you have not toled the guest it is using hyper treads so it is miss allocationg threads internally to cores it think are independed but are not19:07
spatelI am just trying to understand if i run application on bare metal then performance is great but when i create VM with all CPU then performance is bad..19:07
sean-k-mooneyhw:cpu_thread_policy=isolate woudl help but it woudl reduce the available core count19:07
sean-k-mooneyyou shoudl set hw:cpu_treads=2 and hw:cpu_sockets=219:08
spatelhow may core count it will reduce ?19:08
sean-k-mooneybut you cant do that with 30 cpus19:08
sean-k-mooneyyou need either 28 or 3219:08
spateli will go with 28..19:08
Sundargibi: "does the series gracefully handle that move operations are not supported yet with accelerators?" There was an earlier attempt to explicitly block unsupported ops: https://review.opendev.org/#/c/674726/ But it was decided that it is not needed, and we should document the supported ops in Cyborg.19:08
sean-k-mooneyso that you have an even number of cpus per numa node/socket19:08
spatelyou are saying i should set flavor  -> hw:cpu_treads=2 and hw:cpu_sockets=2   and run benchmark right?19:09
sean-k-mooneyspatel: set  hw:cpu_treads=2 and hw:cpu_sockets=2 and do not set hw:cpu_thread_policy=isolate and see if that fixes the issue19:09
spatelsean-k-mooney: sounds good19:09
sean-k-mooneyspatel: actuly i misspelled the first one check the flavor docs to confrim they are right19:10
sean-k-mooneyspatel: https://docs.openstack.org/nova/latest/user/flavors.html19:10
spatelI will check doc don't worry all i need clue to try something :)19:10
Sundargibi: "nova needs to grow support for migration with arqs" Since Cyborg does PCI passthrough, live migration is not supported.19:11
spatelsean-k-mooney: you are saying that option will tell my VM you have two socket..19:11
sean-k-mooneyby default openstack emulates each gust cpu as a sperate socket which is incorrect if you have HT enabled on the host19:11
spatelI do have HT enabled19:11
sean-k-mooneyyep so you want to tell the vm it has 2 threads per cpu and in this case 2 socket 1 per numa node19:12
sean-k-mooneythat will help the guest kerenl make correct schduling decisions19:12
spatelsean-k-mooney: i think you are correct.. that could be the issue why erlang getting confused19:13
spatelbecause erlang also run inside VM and it should have own scheduler19:13
spatelsean-k-mooney: I will keep you posted about my testing.. (today isn't possible but Monday i will have some result.19:14
spatelsean-k-mooney: let me create i should set hw:cpu_treads=2 and hw:cpu_sockets=2 and hw:numa_node=2 right?19:15
spatelotherwise it won't let me run my VM on two numa nodes19:16
sean-k-mooneythe socket and numa nodes dont have to match but it general works better19:16
Sundarsean-k-mooney: Catching up on earlier discussion about instance ops with Cyborg. Explicit blocking of requests with a device profile name in the extra specs ( https://review.opendev.org/#/c/674726/) was not considered the way to go. Is that still your thinking?19:17
*** tbachman has joined #openstack-nova19:17
spatelsean-k-mooney: i will set all three option and give it a try.. thank you..19:18
sean-k-mooneySundar: am honestly i dont rember the full context of the discussin. i know we discussed this at lenght but cant recal what the decision was19:18
sean-k-mooneyi think we said skip the explcit check but document what works? but we can aslo block it if we want19:19
*** igordc has quit IRC19:19
sean-k-mooneySo the consensus is Option 2:19:20
sean-k-mooney> Gradually phase in the support for the server operations and document the limitations in the meantime but don't actively block them in the API like this change does.19:20
sean-k-mooneyWe will state the limitations, if any, in Cyborg documentation.19:20
sean-k-mooneythat was the last comment on that19:20
sean-k-mooneyoption 2 was "2. Gradually phase in the support for the server operations and document the limitations in the meantime but don't actively block them in the API like this change does. They either work (by chance) or they don't, but they aren't officially supported. Once they are supported, we patch them in without a new microversion as bug fixes (or just claim test support so they are no longer19:22
sean-k-mooneyconsidered experimental)."19:22
sean-k-mooneygibi: dansmith ^ are ye still ok with that regarding the livemigation check19:23
sean-k-mooneyso no check for now and document. and we can add one if we  want in the future19:23
dansmithsean-k-mooney: where was that? in the spec?19:25
dansmithit really depends on what the result is19:25
sean-k-mooneyhttps://review.opendev.org/#/c/674726/19:25
sean-k-mooneyits was a nova patch19:26
dansmithif it's data corruption, state intervention required, etc then it needs a check to be graceful19:26
dansmithif it fails in some reasonable way then I'm not so concerned19:26
sean-k-mooneydansmith: i think libvirt will raise an error19:26
sean-k-mooneyqemu will reject a migration if the domain has a hostdev that is not of type usb19:27
dansmithfor live migration I assume, but based on the changes we've made to the flow, I'm not sure what will happen on resize19:27
sean-k-mooneyso we will get to the migrate call and it will fail19:27
sean-k-mooneyyes19:27
dansmithwe might migrate the instance and ignore the fact that it's missing an accelerator19:27
sean-k-mooneyya i dont know what will happen for resize evacuate or shelve19:27
sean-k-mooneyill find out19:28
dansmithif live fails in a predictable and recoverable way, then I'm fine without a check on that one,19:28
dansmithwhich is what I said earlier that virt-specific limitations shouldn't be enforced in the api when we can help it19:28
Sundardansmith, sean-k-mooney: FWIW, IMHO, an explicit check for device profiles in extra specs, as in https://review.opendev.org/#/c/674726/, is probably the safest and clearest to the user.19:29
SundarWe did say that it should be documentation only. But I don;t know if folks will read Cborg dics, or any docs, before kicking off an op.19:30
sean-k-mooneySundar: the main issue with that patch was it was too agressive in what it blocked19:31
Sundar*Cyborg docs19:31
sean-k-mooneyand users never read docs until it breaks19:31
Sundarsean-k-mooney: The details of the patch can be adjusted.19:31
Sundardansmith, efried, gibi, sean-k-mooney: Would you all recommend to bring back https://review.opendev.org/#/c/674726/?19:32
*** dpawlik has joined #openstack-nova19:33
sean-k-mooneyi would not restore it as is.  if we add code to block it we shoudl only block the operation we know do not work.19:36
sean-k-mooneythat change blocks all snapshoting and backups, and interface and volume atache/removal and other operations like rescue and lock.19:38
sean-k-mooneyblocking resize,live-migrate and evacuate might make sense. the rest i think are questionable19:39
sean-k-mooneyalso blocking shelve might make sense but again we said document and fix as bugfixes so we could adress them one by one without api changes for each19:40
Sundar"Without API changes" -- are you thinking of microversion changes for blocking now and end every unblock in the future, if we support more ops?19:42
Sundar*and every19:43
sean-k-mooneyyes im saying we didnt want to do a microversion bump for evey one19:43
*** xek has joined #openstack-nova19:47
sean-k-mooneyspeaking of op i just booted a vm with your latest revision so ill go test some of them19:47
SundarSure. Thanks.19:48
sean-k-mooneymy isp broke my home network so its takeing longer then i hoped to test this. i might set up the multi node setup on my laptop instead of my home openstack on monday but ill do what i can on a singel node first19:50
*** dpawlik has quit IRC19:55
*** mriedem has joined #openstack-nova19:55
*** mmethot has joined #openstack-nova20:07
efriedSundar: I've always been in favor of that idea (blocking unsupported operations with a useful message rather than letting them fail "organically" and mysteriously), but I know others disagree.20:09
*** mmethot_ has quit IRC20:09
sean-k-mooneyfor what its worth i have just done boot,stop,start,reboot,add/remove volume, rescue/unrescue and so far no errors20:11
sean-k-mooneyi am not seeing any real interaction with cyborg during those operations20:11
sean-k-mooneywhich is more or less expected but im not sure if we would loss the acclerator when we regenerate the xml20:12
*** READ10 has quit IRC20:13
sean-k-mooneyi will look at this more closely next week but right now im just check to make sure the operation complette correctly20:13
*** mdbooth has quit IRC20:14
*** mdbooth has joined #openstack-nova20:15
*** ociuhandu has joined #openstack-nova20:16
*** xek has quit IRC20:19
*** spatel has quit IRC20:19
*** ociuhandu has quit IRC20:21
Sundarsean-k-mooney: Great. FWIW, I do most of these with FPGAs. Except rescue/unrescue with different images. The list of ops i have checked are in: https://review.opendev.org/67373520:29
sean-k-mooneyim still makeing my way through the list.20:29
sean-k-mooneyim hoping to get access to a real server with a rush creek or vista creek next week or the week after20:30
sean-k-mooneywhich one are you using again?20:30
sean-k-mooneyi have jsut done add/remove network interface, pause,unpause, suspend,resume, lock,unlock and rebuild20:31
SundarRush Creek, DCP 1.220:31
sean-k-mooneycool we have 1 server i think with one of each so ill ask for the rush creek system20:31
SundarOPAE version 1.1.2-120:31
SundarThe OPAE packages come with some sample bitstreams, sp. NLB modes 0 and 3. I use both.20:32
SundarI can also be reached at ns1.sundar AT gmail DOT com if there is a need for more detailed responses or file transfers.20:33
SundarI have some utility functions to create/delete device profiles, ARQs using curl. ould be happy to share them if you prefer.20:35
sean-k-mooneyim just using the openstack client and i have a test script20:37
SundarThe openstack client is WIP. There are patches to recast it to use openstacksdk etc. https://review.opendev.org/#/c/681391/20:38
sean-k-mooneyyep im using that20:38
sean-k-mooneywith the openstacksdk patch too20:39
sean-k-mooneythere are a bunch of design issue with it that should be adressed but its kind of useable20:39
sean-k-mooneylike when you create a device profile you can only spify its name not the uuid but it only allows you to show a device profile by uuid not name20:40
SundarYea, agreed. The APIs allow both name and uuid though.20:40
sean-k-mooneyyep so the patches are just incomplete20:41
sean-k-mooneyi have worked around that20:41
*** dpawlik has joined #openstack-nova20:41
SundarDo you have any pointers for me to look at the rescheduling question?20:42
sean-k-mooneyi have not read the question so no. i was going to try and force it by making the vm spawn fail20:43
sean-k-mooneyok so first bug. when i shelve the instance the arq is still bound to the host when the vm is shevle_offloaded20:44
SundarYup, I  have not added support for shelve. The Delete ARQ patch states what I support.20:45
sean-k-mooneyyep but we want to check them all anyway20:45
SundarBasically, I prioritized the basic ops. I asked some folks outside Intel what they do with FPGAs in their lab with Cyborg, and went with that.20:46
sean-k-mooneyya which is fine. unshevel "works" but i dont see any interacation with cyborg os i suspect it would not have the accellerator attached after unshelve.20:47
sean-k-mooneyagain thats fine we just need to document it20:48
*** lbragstad has quit IRC20:48
Sundarok20:49
sean-k-mooneyok so after unshelve the allocations do not container the fake device so ya that means the device woudl be lsot20:50
*** Liang__ has quit IRC20:51
sean-k-mooneyit looks like resize to a different flavor and then back does not fix the placement allcoation20:54
sean-k-mooneyit did however complete20:54
*** TxGirlGeek has quit IRC20:55
*** spatel has joined #openstack-nova20:55
SundarYes, I have only tested with resize to the same flavor, which is of course a no op. I was just making sure that there is no basic gotcha.20:58
sean-k-mooneythis is what i tested with the fake driver today20:59
sean-k-mooneyhttp://paste.openstack.org/show/789306/20:59
sean-k-mooneySundar: resize to same flavor should be blocked in the api20:59
sean-k-mooneyit is in the client21:00
sean-k-mooneyyou can migrate but resize to same flavor is invalid21:00
Sundarremove-vol/net: unrelated to Cyborg, right?21:01
sean-k-mooneyyes but you blocked them in your patch that check for the flavor extra spec21:01
sean-k-mooneythey seam to work fine21:01
SundarSuspend will not work with real FPGAs, because libvirt will error out with:  Domain has assigned non-USB devices.21:02
sean-k-mooneyat least with the fake dirver21:02
sean-k-mooneysuspend shoudl do a managed save which will detach all hostdev devices21:02
Sundardelete failed?21:02
sean-k-mooneyyes21:02
sean-k-mooneyit failed in the last versions too21:03
sean-k-mooneythe conductor explodes with an unexpected respoce form cyborg21:03
sean-k-mooneywhen it tries to delete /unbind the arqs21:03
SundarHmm, please send me the logs. I do deletes all the time, but your sequence of ops probably triggered something.21:04
sean-k-mooneyi found that was broken just with boot then delete two days ago21:04
sean-k-mooneyill need to unstack and stack to be able to test it again21:04
SundarI'll try boot + delete. Was it hard reboot?21:05
sean-k-mooneyno just boot then delete21:06
sean-k-mooneyhardreboot seams to be fine21:06
sean-k-mooneySundar: https://etherpad.openstack.org/p/sean-cyborg-testing-delete-logs21:08
sean-k-mooneypaste.openstack.org isnt loadign for me21:08
sean-k-mooneybut that is all the nova and cyborg logs21:08
sean-k-mooneyit look like the cyborg api returned a 40121:10
Sundardevstack@cyborg-api.service[26903]: 2020-02-07 20:56:05.719 .... Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed.21:12
sean-k-mooneyyep21:12
sean-k-mooneyso the call to cyborg either need an admin token which i dont think is correct or the token expired and you need to handel that21:13
sean-k-mooneydeleting the vm again does not fix it by they way21:14
sean-k-mooneybut i can create and delete non cyborg vms21:14
sean-k-mooneyi get the same 401 reponce on the second attempt21:15
sean-k-mooneybut i can delete hte arq myself21:15
SundarWhat does it take to handle an expired token? BTW: https://opendev.org/openstack/cyborg/src/branch/master/cyborg/common/policy.py#L86 applies defuault rule for ARQ deletes21:15
*** owalsh has quit IRC21:16
Sundar*default rule: rule:admin_or_owner21:16
sean-k-mooneyso there is basically two ways to try and resovle it or you can raise an error21:17
sean-k-mooneyif service tokens are configured youc an fall back to using those alternitvly you can have an admin token for the servcie in the nova conf and escalate21:17
sean-k-mooneyor you can fail21:17
sean-k-mooneybut in this can my token was not expired21:17
sean-k-mooneywhen i did it again i was issue a new token and it failed again21:18
*** dpawlik has quit IRC21:18
sean-k-mooneymy devstack also issue tokesn that last for an hour so it shoudl be fine21:18
sean-k-mooneyanyway im done for the day o/21:20
SundarOk. Not sure what is going wrong here. I'll try it out as non-admin.21:21
SundarThanks a lot for all your efforts. ;) Have a good weekend!21:21
*** derekh has joined #openstack-nova21:29
*** derekh has quit IRC21:30
*** owalsh has joined #openstack-nova21:34
Sundarefried: Re. "blocking unsupported operations", the objection seems to be that each change requires microversion changes.21:43
*** ociuhandu has joined #openstack-nova21:44
SundarFWIW, I think any further changes after this series will be few and far between. We could put in a microversion change now. Any further changes should be clumped together as much as possible, based on operator requirements.21:45
*** ociuhandu has quit IRC21:47
*** ociuhandu has joined #openstack-nova21:47
*** nweinber has quit IRC21:50
*** N3l1x has quit IRC22:02
*** ociuhandu has quit IRC22:04
*** ociuhandu has joined #openstack-nova22:05
*** jmlowe has joined #openstack-nova22:10
*** mriedem has left #openstack-nova22:18
*** jmlowe has quit IRC22:19
*** jmlowe has joined #openstack-nova22:20
*** jmlowe has quit IRC22:21
*** ociuhandu has quit IRC22:25
*** iurygregory has quit IRC22:29
*** tosky has joined #openstack-nova22:31
*** spatel has quit IRC22:32
*** tbachman has quit IRC22:38
*** spatel has joined #openstack-nova22:43
*** spatel has quit IRC22:48
*** Sundar has quit IRC22:51
*** dpawlik has joined #openstack-nova22:51
*** dpawlik has quit IRC22:59
*** damien_r has quit IRC23:09
*** bbowen has joined #openstack-nova23:22
*** bbowen_ has quit IRC23:23
*** mlavalle has quit IRC23:33
*** martinkennelly has quit IRC23:41
*** ralonsoh has quit IRC23:56
*** spatel has joined #openstack-nova23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!