Friday, 2019-11-01

« Thursday, 2019-10-31 Index Saturday, 2019-11-02 »
*** brinzhang has joined #openstack-nova00:10
*** ociuhandu has joined #openstack-nova00:10
*** brinzhang_ has quit IRC00:12
*** ociuhandu has quit IRC00:15
*** tetsuro has joined #openstack-nova00:15
*** spatel has joined #openstack-nova00:19
*** mvkr has joined #openstack-nova00:22
*** gyee has quit IRC00:26
*** ivve has quit IRC00:48
*** ab-a has quit IRC00:58
*** jkulik has quit IRC00:58
*** ab-a has joined #openstack-nova00:59
*** jkulik has joined #openstack-nova00:59
*** Liang__ has joined #openstack-nova01:00
*** dave-mccowan has quit IRC01:04
*** dave-mccowan has joined #openstack-nova01:09
*** ociuhandu has joined #openstack-nova01:12
*** ociuhandu has quit IRC01:17
*** brinzhang has quit IRC01:19
*** brinzhang has joined #openstack-nova01:20
*** brinzhang_ has joined #openstack-nova01:22
*** brinzhang has quit IRC01:24
*** brinzhang has joined #openstack-nova01:25
*** jdillaman has quit IRC01:26
*** brinzhang_ has quit IRC01:26
*** jdillaman has joined #openstack-nova01:27
*** Xuchu has joined #openstack-nova01:33
*** brinzhang_ has joined #openstack-nova01:44
*** nanzha has joined #openstack-nova01:44
*** brinzhang has quit IRC01:47
*** abaindur has quit IRC01:48
openstackgerritMerged openstack/nova stable/stein: Switch to opensuse-15 nodeset  https://review.opendev.org/69203301:51
openstackgerritMerged openstack/nova master: Add functional test for two-cell scheduler behaviors  https://review.opendev.org/45200601:51
*** spatel has quit IRC01:53
*** dave-mccowan has quit IRC01:56
*** ociuhandu has joined #openstack-nova01:59
*** ociuhandu has quit IRC02:07
*** markvoelker has joined #openstack-nova02:26
*** markvoelker has quit IRC02:30
openstackgerritBrin Zhang proposed openstack/nova-specs master: Allow specify user to reset password  https://review.opendev.org/68230202:37
*** ociuhandu has joined #openstack-nova02:55
*** ociuhandu has quit IRC03:02
*** mkrai has joined #openstack-nova03:05
*** brinzhang_ has quit IRC03:24
*** brinzhang_ has joined #openstack-nova03:25
*** mtreinish has quit IRC03:28
*** psachin has joined #openstack-nova03:35
*** factor has joined #openstack-nova03:47
*** sapd1_ has joined #openstack-nova04:02
*** sapd1 has quit IRC04:02
*** ociuhandu has joined #openstack-nova04:08
*** brinzhang has joined #openstack-nova04:09
*** brinzhang_ has quit IRC04:11
*** ociuhandu has quit IRC04:13
*** ociuhandu has joined #openstack-nova04:14
*** mkrai has quit IRC04:19
*** ociuhandu has quit IRC04:26
*** links has joined #openstack-nova04:31
*** tetsuro has quit IRC04:34
*** tetsuro has joined #openstack-nova04:36
*** udesale has joined #openstack-nova04:38
*** brinzhang_ has joined #openstack-nova04:58
*** brinzhang has quit IRC05:02
*** mkrai has joined #openstack-nova05:06
*** mdbooth has quit IRC05:06
*** brinzhang has joined #openstack-nova05:08
*** brinzhang_ has quit IRC05:11
*** mdbooth has joined #openstack-nova05:12
*** trident has quit IRC05:18
*** trident has joined #openstack-nova05:25
*** ociuhandu has joined #openstack-nova05:31
*** mkrai has quit IRC05:31
*** psachin has quit IRC05:35
*** ociuhandu has quit IRC05:35
*** nanzha has quit IRC06:00
*** nanzha has joined #openstack-nova06:01
*** threestrands has quit IRC06:02
*** mkrai has joined #openstack-nova06:20
*** mkrai has quit IRC06:26
*** mkrai has joined #openstack-nova06:26
*** jawad_axd has joined #openstack-nova06:30
openstackgerritSundar Nadathur proposed openstack/nova-specs master: Updated Nova-Cyborg interaction spec.  https://review.opendev.org/68415106:31
*** mkrai has quit IRC06:37
*** mkrai has joined #openstack-nova06:38
*** psachin has joined #openstack-nova06:45
*** dlbewley has quit IRC06:51
*** lpetrut has quit IRC06:55
*** Xuchu has quit IRC07:00
*** ociuhandu has joined #openstack-nova07:02
*** ociuhandu has quit IRC07:07
*** Xuchu has joined #openstack-nova07:07
*** jawad_axd has quit IRC07:13
*** brinzhang_ has joined #openstack-nova07:13
openstackgerritMerged openstack/nova master: Refactor rebuild_instance  https://review.opendev.org/68841907:16
*** brinzhang has quit IRC07:16
*** brinzhang has joined #openstack-nova07:19
*** brinzhang_ has quit IRC07:22
*** pcaruana has joined #openstack-nova07:27
*** jawad_axd has joined #openstack-nova07:27
*** yaawang has quit IRC07:31
*** jawad_ax_ has joined #openstack-nova07:36
*** nanzha has quit IRC07:36
*** nanzha has joined #openstack-nova07:38
*** jawad_axd has quit IRC07:38
*** ccamacho has quit IRC07:43
*** panda|pto has quit IRC07:46
*** panda has joined #openstack-nova07:46
*** Xuchu has quit IRC07:47
*** mkrai has quit IRC07:56
*** jamesden_ has quit IRC08:09
*** jamesdenton has joined #openstack-nova08:09
*** mkrai has joined #openstack-nova08:18
openstackgerritMerged openstack/nova stable/stein: libvirt: Ignore volume exceptions during post_live_migration  https://review.opendev.org/69128208:23
*** mkrai has quit IRC08:24
*** mkrai has joined #openstack-nova08:24
*** yaawang has joined #openstack-nova08:26
*** markvoelker has joined #openstack-nova08:28
openstackgerritSilvan Kaiser proposed openstack/nova master: Move Nova Quobyte driver to LibvirtMountedFileSystemVolumeDriver  https://review.opendev.org/68706608:31
*** ivve has joined #openstack-nova08:32
openstackgerritSilvan Kaiser proposed openstack/nova master: [WIP]Move Nova Quobyte driver to LibvirtMountedFileSystemVolumeDriver  https://review.opendev.org/68706608:32
*** markvoelker has quit IRC08:33
*** tkajinam has quit IRC08:53
*** mkrai has quit IRC08:57
*** mkrai has joined #openstack-nova09:01
*** ralonsoh has joined #openstack-nova09:07
*** Dinesh_Bhor has quit IRC09:07
*** trident has quit IRC09:27
*** links has quit IRC09:30
*** links has joined #openstack-nova09:33
*** trident has joined #openstack-nova09:34
*** ociuhandu has joined #openstack-nova09:40
*** Liang__ has quit IRC09:59
*** mkrai has quit IRC10:01
*** trident has quit IRC10:30
*** sapd1 has joined #openstack-nova10:31
*** ociuhandu has quit IRC10:34
*** trident has joined #openstack-nova10:36
*** mkrai has joined #openstack-nova10:39
*** jraju__ has joined #openstack-nova10:40
*** links has quit IRC10:41
*** tbachman has quit IRC10:42
*** brinzhang has quit IRC11:10
*** sapd1 has quit IRC11:12
*** liuyulong has joined #openstack-nova11:22
*** pcaruana has quit IRC11:26
*** dviroel has joined #openstack-nova11:37
*** pcaruana has joined #openstack-nova11:41
*** cdent has joined #openstack-nova11:44
*** sapd1_ has quit IRC11:52
*** sapd1 has joined #openstack-nova11:53
*** mkrai has quit IRC12:00
*** sapd1 has quit IRC12:18
*** sapd1 has joined #openstack-nova12:18
*** markvoelker has joined #openstack-nova12:24
*** udesale has quit IRC12:38
*** udesale has joined #openstack-nova12:38
*** psachin has quit IRC12:38
*** francoisp has quit IRC12:38
*** spatel has joined #openstack-nova12:40
*** Sundar has joined #openstack-nova12:47
*** udesale has quit IRC12:59
*** mmethot_ has joined #openstack-nova13:03
openstackgerritMerged openstack/nova master: Default AZ for instance if cross_az_attach=False and checking from API  https://review.opendev.org/46967513:05
*** mmethot has quit IRC13:05
*** mriedem has joined #openstack-nova13:08
*** trident has quit IRC13:09
openstackgerritMatt Riedemann proposed openstack/nova stable/rocky: libvirt: Ignore volume exceptions during post_live_migration  https://review.opendev.org/69128313:09
mriedemneed a couple of cores to review gibi's evacuate + qos ports changes, the top is trivial, the bottom is a little more work but not bad, lots of test coverage and i'm +2 on it https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports-ussuri+status:open13:12
mriedems/a couple of cores/one other core/13:14
*** trident has joined #openstack-nova13:15
efriedI tried to hit those on Wed but didn't have it in me. I'll try again today.13:23
efriedmriedem, dansmith: do you think the cyborg tempest job should ultimately be voting or nonvoting in nova? The ironic job is nonvoting but the neutron jobs are voting, so I'm not sure there's a clear precedent for "integration test with $service"13:24
sean-k-mooneyefried: well if neturon is broken we cant boot vms13:25
sean-k-mooneyefried: if ironic is broken we just can manage bematal but the other virt dirvers shoudl still work13:25
sean-k-mooneyefried: i would assume it would be non voting at least at first13:25
efriedseems like a reasonable line of thought. Thanks.13:26
*** tbachman has joined #openstack-nova13:26
*** READ10 has joined #openstack-nova13:27
mriedemthe ironic job is voting in other projects, just not nova, likely because no one cared to make it voting in nova13:33
mriedemand historically the ironic jobs were very inconsistent and failed / timed out frequently13:33
Sundarsean-k-mooney, efried: I agree it should be non-voting now. With the fake driver, we are putting up a VM without devices.13:33
sean-k-mooneyright its just testing the api workflow13:34
mriedemwe also have a barbican job in the experimental queue, but i'm not sure how stable that is either13:34
sean-k-mooneyeven so that is valuable to test13:34
sean-k-mooneywith zuulv3 and the fact we can manage this all in repo its not hard to change if we think its stable and worth it13:35
Sundarmriedem, sean-k-mooney, efried: What is the criterion to place a job in experimental, instead of check alone?13:36
efried"We don't want this to run with every patch set"13:36
efried"only on demand"13:36
efriedConsidering how integrated the cyborg callouts are in the general flow of nova, IMO this one needs to be in check13:37
*** damien_r has joined #openstack-nova13:37
*** damien_r has quit IRC13:38
efried...once the code is in place. Which is what's signified by the job-add patch being on top of the series.13:38
*** damien_r has joined #openstack-nova13:39
sean-k-mooneyefried: ya i think it should be in check too as non voting13:39
*** damien_r has quit IRC13:39
*** damien_r has joined #openstack-nova13:39
Sundarefried: Re. the criterion "We don't want this to run with every patch set" -- we may want to run the tests when anything changes in the compute manager or virt driver, at least.13:41
SundarWell, at least libvirt driver13:41
sean-k-mooneySundar: right erric was commenting about why it would be in experimental13:41
sean-k-mooneyif its in experimental one of the resaons for that is we dont want it to run on every patch13:41
sean-k-mooneyfor cyborg we want it to run on most patches13:42
sean-k-mooneyso it better to keep it in check13:42
Sundarsean-k-mooney: Sure, makes sense. How about gate?13:42
efriedSundar: If you wanted to try to come up with a reasonable setting for 'irrelevant-files' you could do that. But for something coupled into the compute manager and virt driver and scheduler and conductor like this, I think that would be very difficult to nail down effectively.13:42
efriedSundar: if it's nonvoting, it doesn't make sense for it to be in the gate queue.13:42
dansmithdefinitely13:42
dansmithmaybe nova/doc :)13:43
Sundarefried: So check only?13:43
sean-k-mooneythe default irrelevant-files we use for tempest jobs should be fine13:43
efriedSundar: yes. Check, nonvoting. I commented as such in the patch.13:43
sean-k-mooneySundar: i think so  we dont run most of the backend specific jobs in gate13:43
efriedwhich btw is here https://review.opendev.org/#/c/670999/ for those following along at home13:44
efriedSundar: if we made it voting at some point, we would probably want to add it to gate for the same reasons that motivated us to make it voting (whatever those are).13:44
Sundarefried: Good. Yumeng wasn't clear about that last night.13:45
*** damien_r has quit IRC13:45
Sundarefried, sean-k-mooney: On the PTG front,I'll schedule a cross-project Cyborg/Nova.13:46
*** artom has quit IRC13:46
Sundarfor whatever topics are remaining -- specs or patch13:47
mriedemsome non-voting jobs in check are more restrictive on when they run, see the nova-lvm job13:47
mriedemtalking about a gating cyborg job in nova is jumping the gun by a mile13:47
Sundarmriedem: Sure, was just asking13:47
mriedemdansmith: the state of the gate sub-thread about slowness and timing out of the POST to resize a server reminded me of something you said in my cross-cell series at some point, the api currently calls conductor which calls the scheduler and it's all blocking until we pick a dest compute and cast to it,13:48
mriedemyou said something about adding long_rpc_timeout to that,13:49
sean-k-mooneySundar: i can try to follow some of the etherpad remotely but i wont be at the ptg13:49
dansmithmriedem: oh for resize...13:49
mriedemi don't think that would help the particular case found in the email (tempest would have timed out after 3 minutes i think) but in general it's maybe something we should do anyway since that could be a lengthy blocking resize call on a big deployment like cern13:49
dansmithmriedem: yeah13:49
mriedemi was surprised when i found that was a blocking call in the first place13:50
mriedemsince usually that's a no-no from the api13:50
dansmithmriedem: when looking into the cells scheduler fail, I found one of those page_cleaner errors too, but it was only like 8s over the expected 1s, yours is 161s, which is a big(ger) deal13:50
mriedemdansmith: so one question, and maybe the answer is "meh, doesn't matter", but would we want to only use the long_rpc_timeout in the call from the api but *not* the reschedule call from the compute where we won't hit the scheduler again and should be much faster?13:51
sean-k-mooneymriedem: so a resize blocks all the way untill the scduler select a destitaiton ? y a i would have expected that to be asyc13:51
mriedemor just keep it simple and use long_rpc_timeout for both13:51
mriedemsean-k-mooney: more than that, until it selects a dest and conductor casts to it's prep_resize method13:51
dansmithmriedem: well, long_rpc for the first call for sure, because that's the cascade.. on the second one, yeah I dunno.. could do both and if they're really running, then let 'em run13:52
mriedemthe reschedule still has to re-swap allocations i guess...13:52
mriedemyeah seems simple to just do both13:52
dansmithso,13:53
dansmithcrazy thought just entered my head13:53
*** kaisers_ has joined #openstack-nova13:53
* sean-k-mooney braces for what is comming13:53
dansmithwe *could* add configs for each service that lets you provide rpc method names that we do as long-call,13:53
*** nanzha has quit IRC13:53
dansmithwhich would make it easier to experiment with that on a running system when there's an issue like this13:54
*** nanzha has joined #openstack-nova13:54
dansmithwe could hard-code the ones we know are important, but others could be converted by virtue of being mentioned13:54
dansmithI know, it's a little crazy, but..13:54
mriedemseems a bit fickle, meaning typos would be easy to screw that up, or someone renaming a method, though i doubt that ever happens13:55
sean-k-mooneyhow would we do that. add a decorator that inspected the config on invocation and convert them to a long rpc if found or something like that13:55
mriedemyou'd just check if the method name is in the configured list13:55
mriedemif 'migrate_server' in CONF.conductor.long_rpc_timeout_calls13:56
mriedemis one way13:56
dansmithmriedem: yep all true13:56
sean-k-mooneyright i was just wonderign would htis be spread in the code or could we centralise it in one mentod that they all uses13:56
mriedemi think that would be more useful before we had long_rpc_timeout where you otherwise just had to configure the global rpc_response_timeout for everything if you had one slow thing, like select_destinations13:57
mriedemi'm sure vio deals with that13:57
mriedemnow having said that,13:57
mriedemi think we know it's an issue for some object indirection api db queries on big lists13:57
mriedeme.g. https://review.opendev.org/#/c/633042/13:58
mriedembut that would also be harder to configure properly i think13:58
dansmithwe should make the indirection methods use it always anyway probably13:59
mriedemlong_rpc_timeout? i thought about it when investigating that bug13:59
dansmithyes13:59
sean-k-mooneywhat would be the cost of makeing long_rpc the default out of interest. i assume there is a reason for being selectiv beyond the fact many rpc methods dont strictly need it14:00
dansmithit increases rabbit load a little bit14:01
sean-k-mooneyok im just wondering how much of a wack a mole problem this is14:02
mriedemfor object methods, could we distinguish between an object list query and just a single object query?14:02
dansmithit's exactly a whack-a-mole problem, which is why making it config-driven so people can whack said moles without code changes might be reasonable :)14:02
dansmithmriedem: yes14:02
dansmithmriedem: if you're requesting so many objects that you can't get it done in under the default limit, there are probably other issues, and if it's a load thing, then small queries can be delayed behind big ones, so.. I dunno how meaningful that distinction is14:04
mriedemyeah. at what point do we let someone configure themselves to wait too long rather than solve their bigger load problems by scaling out/up their control plane.14:05
mriedemi guess that answer probably comes down to their SLAs14:05
mriedemor whatever their performance testing thresholds are14:05
dansmithyeah, I'm not really suggesting we let them configure long rpc calls externally to solve problems,14:06
dansmithmore to experiment and report (or even just for our own experimentation in devstack)14:06
dansmithnot that it's really easier for us I guess14:06
dansmithanyway, it was just a thought14:06
*** eharney has joined #openstack-nova14:11
*** jaosorior has joined #openstack-nova14:13
Sundarmriedem, sean-k-mooney, dansmith: What would you like to see, either in the PTG or in terms of spec/patches, to move Cyborg forward?14:15
dansmithSundar: I'm not going to be at the ptg14:17
openstackgerritMatt Riedemann proposed openstack/nova master: Use long_rpc_timeout in conductor migrate_server RPC API call  https://review.opendev.org/69255014:20
Sundardansmith: I see. Would you like to see any changes in https://review.opendev.org/#/q/status:open+project:openstack/nova+bp/nova-cyborg-interaction ? Just checking to see what it takes to merge this patch series.14:23
dansmithSundar: reviews. doesn't look like it's had much of any, and the little it has had seems pretty un-diverse14:23
*** jawad_ax_ has quit IRC14:23
dansmithwhich I know is why you're asking, just saying14:23
dansmithI popped up one to see and it looks like it's in merge conflict and still has stuff like random whitespace damage in it (I'll comment in a sec), but that kind of stuff doesn't make it look like it's ready to go, fwiw14:24
*** jawad_axd has joined #openstack-nova14:24
*** maciejjozefczyk has joined #openstack-nova14:25
SundarThe conflicts happened recently. The series has passed Zuul checks for the most par and has been ready to go since Train merged.14:26
SundarPlease ignore the conflicts for now and focus on the content14:26
* dansmith wanders off to some patches not in merge conflict14:28
mriedemis there functional testing anywhere in the series?14:28
mriedemi'm not even talking tempest,14:28
mriedembut like creating a cyborg fixture in nova like we have for cinder and neutron,14:28
*** mtreinish has joined #openstack-nova14:28
*** jawad_axd has quit IRC14:28
mriedemso that we can start writing functional in-tree tests that use that fixture and at least exercise the runtime code in nova, despite using a fixture for the cyborg apis14:28
mriedemi didn't ask/expect anyone to review my cross-cell resize series until i had some solid functional testing of at least the happy paths,14:29
mriedemi don't think i even started asking for reviews until i had a multi-cell gate job working14:29
mriedemright now this all looks like unit tests14:29
dansmithmriedem: please ignore the conflicts and lack of testing and focus on the content14:30
mriedemthe next closest thing to the cyborg stuff is probably gibi's qos ports series14:30
mriedemand he did a lot of functional test for those b/c of the complicated nature of dealing with nested provider allocations14:30
mriedemso maybe you could build on some of that while also working on a beginning of a cyborg fixture for functional testing - at least the basic happy path,14:31
mriedemcreate a server with arqs and then delete the server and verify the arqs are deleted/unbound whatever happens to those14:31
mriedemthe fixture would obviously have to track the state of the arqs associated with a server14:31
mriedemwhich we do in the NeutronFixture for ports and volumes in the CinderFixture14:31
*** ociuhandu has joined #openstack-nova14:31
Sundarmriedem: Looking at the discussion we had at the Cyborg/Nova cross-project at Denver PTG: https://etherpad.openstack.org/p/ptg-train-xproj-nova-cyborg14:32
SundarThe criterion for integration was said to be upstream tempest CI with a fake driver. We got the fake driver done, and tempest can create/delete VMs wtth that fake driver now14:33
dansmithSundar: yeah, that's necessary too,14:34
dansmithSundar: but just about anything with tentacles running from API to the deepest depths of the compute service has serious functional testing14:34
Sundarmriedem, dansmith: Just to be clear, you are talking about extending today's Nova functional testing, right? Not gabbi?14:35
mriedemnova doesn't use gabbi14:35
mriedemplacement does14:35
*** ociuhandu has quit IRC14:36
mriedemSundar: some solid examples are what gibi did with qos port functional testing, for which this is the parent class https://github.com/openstack/nova/blob/master/nova/tests/functional/test_servers.py#L556514:36
mriedemso here is an example https://github.com/openstack/nova/blob/master/nova/tests/functional/test_servers.py#L599614:37
mriedem"Tests creating a server with a pre-existing port that has a resource                           request for a QoS minimum bandwidth policy."14:37
mriedemwe have other tests that interact with cinder https://github.com/openstack/nova/blob/master/nova/tests/functional/test_boot_from_volume.py14:37
mriedemwell, usinga CinderFixture14:38
mriedemhttps://github.com/openstack/nova/blob/master/nova/tests/functional/test_multiattach.py14:38
Sundarmriedem: Thanks, will look at it and follow up.14:40
mriedemi traced the instance create/delete through the conductor and compute logs for the test_server_basic_ops tempest test and it looked fine, i saw the creating/waiting/deleting arqs stuff, no weird errors in the logs, so that's all good to see14:44
mriedemonce the series is rebased, if tests are passing and it's ready for review, put it into a runway slot14:44
mriedemSundar: ^14:45
Sundarmriedem: Thanks, will do.14:47
mriedemSundar: this is going to require a microversion bump https://review.opendev.org/#/c/631245/36/nova/api/openstack/compute/schemas/server_external_events.py@3614:48
mriedemyou can't change schema on a versioned api without a new microversion14:48
dansmiththis does not look "ready to go since train" to me14:49
dansmiththere are things used in early patches that aren't added until later patches14:49
dansmithand presumably very large test (even unit) gaps to allow that to happen14:50
*** jangutter has quit IRC14:50
Sundardansmith: "used in early patches that aren't added until later patches" -- What are you referring to?14:53
dansmithSundar: the event change. I left comments14:53
mriedemSundar: https://review.opendev.org/#/c/631244/43/nova/compute/manager.py@260914:54
mriedemSundar: note that nova still generally tries to subscribe to a philosophy of anything we merge today can be in production today, and people can CD nova14:54
mriedemeven if that's not the reality of how the vast majority of openstack consumers consume openstack, it's a development philosophy to try and avoid merging broken code14:55
*** bnemec is now known as beekneemech14:56
*** artom has joined #openstack-nova14:57
mriedemhaving said that i can't easily find anything about that in https://docs.openstack.org/nova/latest/contributor/ but i know it comes up every so often, and not all projects in openstack subscribe to the same philosophy, e.g. if it's all working by RC1 whatever, let it ride14:57
*** jraju__ has quit IRC14:58
dansmithprobably because it's core openstack philosophy than not everyone has adhered to14:59
mriedemclosest is, https://docs.openstack.org/nova/latest/contributor/process.html#smooth-upgrades "upgrade from any commit, to any future commit, within the same major release"14:59
dansmithis, or was14:59
mriedemdansmith: i'm just not sure it's clearly documented anywhere14:59
mriedemand probably should be14:59
mriedemlike something simple in https://docs.openstack.org/nova/latest/contributor/policies.html15:00
dansmithsounds like a good thing to do tome15:01
SundarI have generally thought of the patch series as a unified whole. Merging individual parts will not give you any useful functionality. The base patch has a procedural -2, so that the whole series will merge together.15:01
dansmithit's fine to not have incremental functionality,15:01
SundarThat said, I don't believe in merging 'broken patches' either15:01
dansmithit's not fine to merge things that use something before the thing is available15:01
dansmithand we can't arrange for the whole series to merge as a whole,15:02
dansmithwhich means the -2 holds things until all the pieces are ready, but they still have to be mergeable in isolation, because they will15:02
mriedemespecially since it takes about 3 days of rechecks to merge anything in nova right now15:02
dansmiththree days? please15:03
mriedemok ok 5 days15:03
dansmithmy notifications thing has been going all week.. maybe since last week15:03
mriedemfor big series like this it's always best to work from the bottom up where everything is noop until "turned on" in the API at the end if possible15:03
dansmithyeah, since monday15:03
mriedemso there is less risk in merging the bottom noop stuff15:04
*** JamesBenson has joined #openstack-nova15:05
Sundarmriedem: The code flow starts from the API, such as getting device profile info based on the extra specs. If the API stuff came later, wouldn't the patch sequence be out of order? We can only do UT for the earlier patches then.15:07
cdentmriedem: I assume you saw my response on the gate+placement email? Sorry I've not been able to dig more deeply. I've been engaged elsewhere...To such an extent that I may come off the mailing list pretty soon to maintain sanity.15:08
efriedSundar: in this case the "master switch" could conceivably be the part that processes the device profile out of the flavor.15:09
mriedemcdent: yup, and replied, and thanks15:09
mriedemcdent: i think the likeliest culprit is noise neighbors on oversubscribed nodes as you said15:09
*** maciejjozefczyk has quit IRC15:13
*** kaisers_ is now known as kaisers_away15:14
*** maciejjozefczyk has joined #openstack-nova15:14
cdentmriedem: the gist of what I can find on that innodb error (now that I've managed to actually read your responses) is "you disk i/o is way constrained"15:17
*** cdent has left #openstack-nova15:18
*** cdent has joined #openstack-nova15:18
*** kaisers_away is now known as kaisers_15:18
*** kaisers_ is now known as kaisers_away15:19
mriedemcdent: heh, yeah looking at this dstat graph, i/o is just 0 from 16:52 to 16:5515:19
cdentoh dear15:19
cdentthere should be some kind of "how am I supposed to work in these conditions" meme we can interject here15:20
*** gyee has joined #openstack-nova15:20
efriedhttps://makeameme.org/meme/i-cant-work-2sl5sn15:21
mriedemhttps://toddroblog.files.wordpress.com/2017/02/drunk-hobo-flips-double-bird-1.jpg?w=809 ?15:21
mriedemoh blurred out?!15:21
*** kaisers_away is now known as kaisers_15:22
mriedemi wish i knew about https://lamada.eu/dstat-graph/# years ago15:23
cdentyeah, why don't things like that get passed around  more easily?15:26
*** dlbewley has joined #openstack-nova15:26
mriedemprobably because i was too shy to say "i'm dumb, how can i see this dstat output in a graph?"15:27
mriedemi assume all of my co-workers to write tools to generate graphs on the fly for everything they do15:27
cdenthttps://www.youtube.com/watch?v=MKIaS0lh-uo15:29
*** macz has joined #openstack-nova15:29
mriedemheh, it's rare to get a repo man reference in here15:30
cdentwhat provider is that 0 I/O on?15:33
mriedeminap15:33
openstackgerritMatt Riedemann proposed openstack/nova master: doc: link to nova code review guide from dev policies  https://review.opendev.org/69256915:35
*** igordc has joined #openstack-nova15:36
*** TxGirlGeek has joined #openstack-nova15:38
*** kaisers_ is now known as kaisers_away15:39
*** igordc has quit IRC15:47
*** artom has quit IRC15:47
*** TxGirlGeek has quit IRC15:47
* dansmith puts on "Round and Round" as he goes round with Zuul again15:49
efriedI already rechecked a few dansmith15:54
openstackgerritMatt Riedemann proposed openstack/nova master: Document CD mentality policy for nova contributors  https://review.opendev.org/69257215:54
mriedemdansmith: i took a crack at documenting that ^15:54
dansmithefried: in that case, https://www.youtube.com/watch?v=0u8teXR8VE415:54
mriedemdansmith: thoughts on the no graceful shutdown thing i mentioned here? http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010495.html - wondering if we should go back to like a 10 second shutdown timeout in case that is screwing up the guest during things that transfer the root disk or snapshot it (shelve/unshelve) leading to ssh failures later15:56
mriedemi had made that change in devstack b/c of a real bug where we see the 60 second shutdown timeout in nova not doing anything, potentially leading to overall job timeouts if we're waiting a full minute to power off guests during a tempest run15:57
dansmithmriedem: I doubt that cirros is responding to the graceful shutdown request at all15:57
dansmithmriedem: so I'm not sure it matters15:58
dansmithif you have a devstack up you should be able to test15:58
dansmitheven still, unless it's writing to the disk when the destroy happens, it wouldn't likely do any damage (and shouldn't really anyway)15:58
dansmithdon't we get the console of the guest we fail to ssh into if we do? unless it's sitting at a panic or some unhappy state, I'd not be concerned15:59
mriedemyes we do15:59
*** mmethot_ has quit IRC15:59
*** kaisers_away is now known as kaisers_16:00
*** kaisers_ is now known as kaisers_away16:00
sean-k-mooneymriedem: are you suspecting disk curruption?16:00
mriedemthere are several guest ssh failure bugs in the gate right now, many different reasons (dhcp lease fails, out of space on the guest disk [growroot fails], kernel panic)16:00
sean-k-mooneydue to an agressive shutdown?16:00
mriedemi'm grasping at straws16:00
mriedemin one case that i caught in the gate where the 60 second shutdown timed out, i got the guest console log http://paste.openstack.org/show/752002/16:01
mriedemcdent wondered (in the bug) if the metadata api retry loop in the guest was making it ignore the shutdown request16:01
mriedemhttps://bugs.launchpad.net/nova/+bug/182989616:01
openstackLaunchpad bug 1829896 in OpenStack Compute (nova) "libvirt: "Instance failed to shutdown in 60 seconds." in the gate" [Undecided,New]16:01
sean-k-mooneywell in that case it got a dhcp lease but then could not hit the metadata service16:02
mriedemi'm also wondering if there are ideas on figuring out what is eating up all of the disk in these guests when growroot fails16:02
sean-k-mooneyso that looks like a neutron issue with the metadata proxy16:02
sean-k-mooneyif glean/cloud init is waithing for the metadata then16:03
sean-k-mooneythe ssh service may not have started16:03
openstackgerritLee Yarwood proposed openstack/nova stable/queens: Add 'path' query parameter to console access url  https://review.opendev.org/69257316:03
*** ivve has quit IRC16:03
lyarwoodmriedem: yup, working on the other change now thanks16:06
*** jmlowe has quit IRC16:07
*** maciejjozefczyk has quit IRC16:07
sean-k-mooneymriedem: i do know we have had issue with hot plug if the early init in the guest takes too long16:07
*** maciejjozefczyk has joined #openstack-nova16:07
mriedemlyarwood: i'm sort of inclined to say you guys (RH) should keep that one downstream16:08
sean-k-mooneybut we also see  Starting acpid: OK16:08
mriedemsince i don't see people clamoring for using the bleeding edge novnc stuff back on queens16:08
mriedemi'm not like -2, but those are my "feelings"16:09
mriedemwhat you humans would call feelings anyway16:09
lyarwoodhaha yeah no issues, thought I'd at least post it upstream first to see if anyone wanted it16:09
sean-k-mooneyah ha proof mriedem is actully an irc bot !!!16:10
sean-k-mooneymriedem: on other random straw is there seams to be very littly entropy in the guest16:10
sean-k-mooneyrandom: dd urandom read with 31 bits of entropy available16:10
sean-k-mooneythe topic of enabling the virtio random number generator by default has come up before. i wonder if it could be related to that.16:12
*** markvoelker has quit IRC16:12
mriedemi seem to remember a recentish patch of kashyap's related to this16:14
*** artom has joined #openstack-nova16:14
mriedemi'm just thinking of this https://review.opendev.org/#/c/577385/16:15
*** artom has joined #openstack-nova16:15
sean-k-mooneyya there is that one but we also talked a little about adding the random number gereate to the libvirt xml by defualt rather then needing to set the image property16:16
mriedemwe don't use that in our devstack guests though since we don't have hw_rng_model=virtio in the image16:16
mriedemoh16:16
mriedemwell, i have to go to what you humans would call lunch now16:16
*** mriedem is now known as mriedem_feeds16:16
*** tbachman has quit IRC16:23
*** markvoelker has joined #openstack-nova16:23
*** jmlowe has joined #openstack-nova16:23
*** tbachman has joined #openstack-nova16:24
openstackgerritLee Yarwood proposed openstack/nova stable/queens: Reduce scope of 'path' query parameter to noVNC consoles  https://review.opendev.org/69258116:28
*** jmlowe has quit IRC16:32
*** antonym has quit IRC16:48
*** nanzha has quit IRC16:48
*** kaisers_away is now known as kaisers_16:50
*** kaisers_ is now known as kaisers_away16:50
*** ociuhandu has joined #openstack-nova16:52
*** cdent has quit IRC16:53
*** Sundar has quit IRC16:57
*** tbachman has quit IRC16:58
*** kaisers_away is now known as kaisers_17:00
*** kaisers_ is now known as kaisers_away17:00
*** tbachman has joined #openstack-nova17:00
*** jaosorior has quit IRC17:01
*** tbachman has quit IRC17:04
*** ivve has joined #openstack-nova17:05
*** jmlowe has joined #openstack-nova17:07
*** tbachman has joined #openstack-nova17:10
*** igordc has joined #openstack-nova17:13
dansmithcripes17:19
dansmithcache notifications patch is gonna fail again17:19
openstackgerritJohn Garbutt proposed openstack/nova-specs master: Add Unified Limits Spec  https://review.opendev.org/60220117:21
*** antonym has joined #openstack-nova17:22
*** ociuhandu has quit IRC17:27
*** ociuhandu has joined #openstack-nova17:29
*** ociuhandu has quit IRC17:33
*** maciejjozefczyk has quit IRC17:33
*** jawad_axd has joined #openstack-nova17:33
*** mriedem_feeds is now known as mriedem17:34
*** ganso has quit IRC17:34
*** jawad_axd has quit IRC17:34
mriedemdansmith: it's because of your sin17:34
* dansmith goes to repent17:34
efrieddansmith: I'm +A on mriedem's CD doc patch, but the predecessor (link shuffle) needs to be sent as well17:35
efriedhttps://review.opendev.org/#/c/692569/117:36
dansmithefried: oh sorry I missed that earlier I guess17:36
dansmithI dun getified it17:36
*** ganso has joined #openstack-nova17:38
mriedemdansmith: i looked at a dstat from one of the 'timed out waiting for cell' failed jobs and the only thing around that time is a spike in paging and networking which doesn't tell me much17:40
dansmithmriedem: yeah I looked pretty in depth at one dstat from a cell fail job17:40
dansmithand it looked overly normal to me17:40
dansmithmysql wasn't spiking that I could tell, it was the big memory user, but only like 400MB at the time17:40
dansmithIO wasn't crazy, etc17:41
mriedemyeah nothing obvious like the 0 i/ops in the other thing earlier today17:41
dansmithI didn't catch the zero iops thing17:42
mriedemand there is no rabbit involved b/c we're going straight to the db yeah?17:42
dansmithyou were assuming zero iops meant bad?17:42
mriedemthe 0 iops thing was related to another bug in the ML where we get long timeouts17:42
mriedemwhich triggered the resize long_rpc_timeout discussion17:42
mriedemand a very obvious message in the mysql error log17:43
dansmithyou mean zero iops for a long period of time or something I assume/17:43
mriedem3 minutes yeah17:43
mriedemwhile placement was processing a POST /allocations request17:43
dansmithokay, yeah, one or two samples with zero iops wouldn't be alarming to me, but minutes of it, for sure17:43
*** READ10 has quit IRC17:44
*** eharney has quit IRC17:44
mriedemand https://bugs.launchpad.net/nova/+bug/1825584 shouldn't be related to the scheduler timing out waiting for a response from cell1 b/c there is no uwsgi involved there17:46
openstackLaunchpad bug 1825584 in OpenStack Compute (nova) "eventlet monkey-patching breaks AMQP heartbeat on uWSGI" [Undecided,Confirmed]17:46
mriedemjust scheduler running with eventlet17:46
*** henriqueof has joined #openstack-nova17:53
*** kaisers_away is now known as kaisers_17:55
*** kaisers_ is now known as kaisers_away17:55
*** ralonsoh has quit IRC18:06
*** tbachman has quit IRC18:12
*** ociuhandu has joined #openstack-nova18:20
*** jaosorior has joined #openstack-nova18:24
*** ociuhandu has quit IRC18:28
openstackgerritMerged openstack/nova stable/rocky: libvirt: Ignore volume exceptions during post_live_migration  https://review.opendev.org/69128318:30
openstackgerritMatt Riedemann proposed openstack/nova stable/queens: libvirt: Ignore volume exceptions during post_live_migration  https://review.opendev.org/69128418:36
*** kaisers_away is now known as kaisers_18:38
*** kaisers_ is now known as kaisers_away18:38
*** kaisers_away is now known as kaisers_18:40
*** kaisers_ is now known as kaisers_away18:40
mriedeminteresting that we don't have a task_state change while confirming a resized server18:48
mriedemto prevent other actions from happening while still confirming and cleaning up the source host18:48
mriedembut we do have task_states.RESIZE_REVERTING for reverting a resize18:57
*** kaisers_away is now known as kaisers_18:58
*** kaisers_ is now known as kaisers_away18:58
*** luksky has joined #openstack-nova19:10
*** jmlowe has quit IRC19:12
*** maciejjozefczyk has joined #openstack-nova19:14
*** jaosorior has quit IRC19:25
*** sapd1 has quit IRC19:27
*** sapd1 has joined #openstack-nova19:28
*** maciejjozefczyk has quit IRC19:32
*** eharney has joined #openstack-nova19:35
openstackgerritMerged openstack/nova master: doc: link to nova code review guide from dev policies  https://review.opendev.org/69256919:36
*** markvoelker has quit IRC19:36
openstackgerritMerged openstack/nova master: Document CD mentality policy for nova contributors  https://review.opendev.org/69257219:36
*** jmlowe has joined #openstack-nova19:39
*** dklyle has quit IRC19:44
mriedemdansmith: MessagingTimeouts in nova-next in the latest failure in https://review.opendev.org/#/c/691390/19:46
mriedemlooks like just a slow node?19:47
mriedemdstat shows ram usage around 6.5gb for most of the run19:47
mriedemwell, at least around the time it was timing out19:47
mriedemload average really spikes around the time of the timeouts19:48
*** dklyle has joined #openstack-nova19:51
*** markvoelker has joined #openstack-nova19:53
*** jamesdenton has quit IRC19:57
*** kaisers_away is now known as kaisers_19:58
*** kaisers_ is now known as kaisers_away19:58
*** kaisers_away is now known as kaisers_19:59
*** kaisers_ is now known as kaisers_away19:59
sean-k-mooneymriedem: you ruled out memory right19:59
sean-k-mooneyi allocated a little too many hugpage when playing with dpdk today and when i almost exausted memory in the vm i was getting big cpu spikes20:00
*** dklyle has quit IRC20:00
*** dklyle has joined #openstack-nova20:00
sean-k-mooneylike all core 100% for 3 secons then normal then spike then normal20:00
mriedempeakmemtracker shows mysql as the top memory user but it doesn't look like it's running out20:08
mriedemshould be 8gb of ram on the node20:08
sean-k-mooneyya you are proably fine20:08
sean-k-mooneyi had no swap and allcoated 4G of hugepages instead of 2 by mistake so i  was running out20:09
*** JamesBenson has quit IRC20:15
*** JamesBenson has joined #openstack-nova20:15
*** kaisers_away is now known as kaisers_20:18
*** kaisers_ is now known as kaisers_away20:18
efriedjroll: You available to brainstorm some of this vTPM stuff?20:21
jrollefried: maybe?20:21
jrollI'm not really an expert on the TPM bits, but I have a few minutes to chat, yes20:22
efriedjroll: I'm just trying to understand how the nova flow needs to work.20:22
jrollsure, I can try to help.20:22
efriedIf each boot needs to be able to specify an independent value, that's a bigger change than other things.20:23
efriedBasically what I'm trying to figure out is where the value of `secret` comes from in20:24
efried      <tpm model='tpm-tis'>20:24
efried        <backend type='emulator' version='2.0'>20:24
efried          <encryption secret='6dd3e4a5-1d76-44ce-961f-f119f5aad935'/>20:24
efried        </backend>20:24
efried      </tpm>20:24
jrollright20:24
jrollso take the fact that's it's a tpm out of your head20:24
efriedhah, that's easy enough20:24
efriedI can't even spell tpm yet.20:24
jrollbasically, there's a file for the instance on the hypervisor's disk, which needs to be encrypted20:24
jrollto encrypt a file, you need a secret20:25
efriedto your understanding, is the value of `secret` the actual secret, or is it an identifier for a secret that needs to be looked up?20:25
jrollthe 6dd3e4a5-1d76-44ce-961f-f119f5aad935 in your paste is a reference to a secret object stored in libvirt (i.e. virsh secret-set 6dd3e4a5-1d76-44ce-961f-f119f5aad935 myawesomesecret)20:25
efried"a reference" <= ✔20:25
jrollsee https://libvirt.org/formatsecret.html#vTPMUsageType20:26
efried"stored in libvirt" -- from the host or from within the VM20:26
efried...okay...20:26
jrollfrom the host, yeah20:26
jrollthe VM knows nothing about this file20:26
efriedoh, look, a vTPM usage man page, that's convenient.20:26
jroll:D20:26
jrollI found it from a libvirt page you linked in that spec :P20:26
efriedbutbutbut...20:26
efriedI thought the whole point of this was that someone who walks away with the disk can't get at the TPM20:27
jrollcorrect, because the file representing the TPM is encrypted20:27
efriedwith a key that's available on the host tho?20:27
jrollI *assume*, but have not verified, that this secret is stored in memory, not on disk20:27
jrollwhich is why I mentioned we may want to call virSecretSetValue on every instance start()20:28
jrollas if I'm correct about it being in-memory, a host reboot or instance migration would mean we need to set that secret again20:28
efriedI thought I read somewhere that it's in memory20:28
efriedhttps://github.com/qemu/qemu/blob/master/docs/specs/tpm.txt20:28
efriedL1420:29
efriedand 3120:29
jrollThe TIS interface makes a memory mapped IO region in the area 0xfed40000 -20:29
jroll0xfed44fff available to the guest operating system.20:29
jrollI read that as it maps the decrypted version of the file on disk into memory, but I'm not good with terms at this level20:30
*** ociuhandu has joined #openstack-nova20:30
jrollI thought I read somewhere that swtpm keeps the state on disk, but I don't see it now20:31
efried"...reboot or ... migration ... set that secret again"20:31
jrollso if swtpm keeps it in memory, and encrypted, that's even better20:31
efriedIf the host has possession of the master secret, everything sounds easy. But it also sounds like it doesn't satisfy your use case, which is "must not be compromised if someone walks off with the disk".20:32
jrollif the host keeps the master secret in memory only, that means it is not compromised if someone walks off with the disk20:32
efriedokay, how does the master secret get into memory?20:32
jrollvia virSecretSetValue20:33
jrollper https://libvirt.org/formatsecret.html#vTPMUsageType20:33
efried...with a value or file that comes from where?20:33
jrollthat's the TODO in this spec that I don't have an answer to20:33
jroll:)20:33
efriedBecause I don't expect a flow like20:34
efried- Host boots20:34
efried- Operator intervention is required to type in a password or load up a file20:34
efried- now your VMs can start20:34
efriedto be acceptable in general20:34
jrollI agree20:34
jrollso we'd have to have something in the instance database record, which is either a secret, or a reference to a secret20:35
jrollideally the secret would be in barbican or equivalent20:35
jrolland that secret could be fetched by nova-compute as needed20:35
*** ociuhandu has quit IRC20:36
efriedand you're allowed to fetch it by virtue of... being an authed user?20:36
jrollI would assume by being the nova user20:37
jrollor having access to the rabbit bus, if we had nova-compute ask nova-conductor for it20:37
efried<red flag> upcall </red flag>20:38
jrollheh20:38
efriedI was looking over some stuff about trusted certs for image signatures20:38
jrolldo computes not ever ask the conductor for info? what about all the database reads?20:38
efriedhttps://docs.openstack.org/nova/latest/user/certificate-validation.html20:39
*** ociuhandu has joined #openstack-nova20:39
efriedjroll: at least in theory, traffic only ever flows one way, from the conductor down to the compute20:39
efriedThere may be one upcall left over, I can't remember, but we've been working hard to purge them and not allow any new ones.20:39
jrollerm, how does a compute read from the database?20:40
efriedI think there are two databases20:40
jrollit isn't an RPC upcall, I get that, but it's still a call to the conductor, just buried in o.vo20:40
efriedI'm not real well versed on this stuff I'm afraid20:40
jrollok20:41
umbSublimeI'm not sure that would apply, but say the secret were stored in barbican couldn't the compute host query barbican by means of the service_token ?20:41
jrolljust sayin': https://github.com/openstack/nova/blob/master/nova/cmd/compute.py#L5320:41
jrollumbSublime: probably, yeah20:41
* jroll reads this cert validation thing20:41
efriedjroll: I don't know how relevant the certificate validation thing is; I was just looking at how it tries to bootstrap secrets.20:42
*** ociuhandu has quit IRC20:43
efriedunfortunately it seems to have added new CLI/API fields -- that would be the "specify a secret to nova boot" route. I would hate to have to do that because it blows up the feature scope significantly.20:43
jrollright20:43
efriedwithout that, we like add a new extra spec and parlay it into xml and we're done.20:44
efriedwith it, we've got microversions and client/CLI enhancements and on and on.20:44
jrollit looks similar to how TPM secret storage could work, in that nova-compute has to fetch a secret from barbican to boot the instance20:44
jrollin the image validation case, the user provides the secret reference, because the user signs the image20:44
jrollin the vTPM case, only nova needs to know the secret, not the user, so it can handle everything internally20:45
efriedBut how does nova know the secret?20:45
efriedIn a way that doesn't allow you to get at it if you steal the disk?20:45
jrollnova could generate the secret for a TPM at instance create time, put it in barbican, and then pull it later as needed (migration, etc)20:45
*** kaisers_away is now known as kaisers_20:46
*** kaisers_ is now known as kaisers_away20:46
*** slaweq has joined #openstack-nova20:46
efriedOkay. I guess I should go figure out how barbican works. Hopefully VMG's homegrown key manager equivalent responds the same way as barbican/vault does?20:46
jrollclose enough20:47
efriedbecause penick said something about using that in place of barbican20:47
jrollif we do this all in castellan, we can make a castellan backend for our thing20:47
efried...yeah20:47
jrollseems like the right thing to do anyway20:47
*** ociuhandu has joined #openstack-nova20:47
efriedright, iiuc nova requires whatever key manager to be brokered/brokerable through castellan.20:47
jrollmhm20:48
*** kaisers_away is now known as kaisers_20:49
*** kaisers_ is now known as kaisers_away20:49
efriedokay, so20:50
efried- nova creates a new secret in $backend via castellan20:50
efried- nova uses secret to virSecretSetValue to get $secret_uuid20:50
efried- nova boots instance with <encryption secret='$secret_uuid'>20:50
efriedNow does the tpm show up to the VM user as virginal?20:50
jrollefried: anything else I can help with?20:50
jrollI assume just like a new TPM20:50
jrollor hope?20:50
efriedAnd the VM user bootstraps the tpm with his own secret, which only he knows20:50
jrollthat is a question for a qemu or swtpm person, I guess20:50
jrollright20:50
*** igordc has quit IRC20:50
efriedso nova ("the nova user") has to be the root of trust20:51
efriedin this scenario20:51
*** kaisers_away is now known as kaisers_20:51
*** kaisers_ is now known as kaisers_away20:51
*** abaindur has joined #openstack-nova20:51
jrollI guess by some definition, yes20:52
jrollit always comes down to the "hardware" supplier :)20:52
efriedbecause at any time "the nova user" could go grab that secret, decrypt the vtpm, and get at the VM user's secret20:52
*** kaisers_away is now known as kaisers_20:52
*** abaindur has quit IRC20:52
*** kaisers_ is now known as kaisers_away20:52
jrollright20:52
efriedokay, so "walk away with disk" is a requirement, but "compromised root" is not.20:52
jrollassuming the nova user has the encrypted vtpm state20:52
efriedyeah, ability to read memory or whatever.20:52
*** abaindur has joined #openstack-nova20:53
jrollyeah, I think that's something we just have to live with, in an emulated world20:53
jrollif you've root on the hypervisor you can just go ahead and read the VM's memory anyway, right20:53
efriedI guess?20:53
efriedOkay, thanks for the talk jroll. If our assumptions are correct, I think I can work with this.20:54
*** abaindur has quit IRC20:54
jrollefried: you're welcome, I hope they are correct :)20:54
efriedpenick said he was going to get "security people" to vet things, but it would be a while before they were available.20:54
*** abaindur has joined #openstack-nova20:54
jroll¯\_(ツ)_/¯20:54
efriedI'll at least write up spec content under these assumptions, so that they can have something to point at and say it's wrong, rather than just a void.20:55
jrollI forgot about this thing for the last couple months, until today20:55
openstackgerritMerged openstack/nova master: Switch to devstack-plugin-ceph-tempest-py3 for ceph  https://review.opendev.org/69176520:56
*** ociuhandu has quit IRC20:57
*** ociuhandu has joined #openstack-nova21:01
*** slaweq has quit IRC21:05
*** ociuhandu has quit IRC21:06
*** dklyle has quit IRC21:08
*** dklyle has joined #openstack-nova21:14
*** spatel has quit IRC21:21
*** dklyle has quit IRC21:22
*** dklyle has joined #openstack-nova21:23
*** luksky has quit IRC21:25
*** JamesBen_ has joined #openstack-nova21:26
*** JamesBen_ has quit IRC21:27
*** JamesBenson has quit IRC21:29
openstackgerritMerged openstack/nova master: Avoid error 500 on shelve task_state race  https://review.opendev.org/69220621:34
openstackgerritMatt Riedemann proposed openstack/nova master: Convert legacy nova-live-migration and nova-multinode-grenade to py3  https://review.opendev.org/69237421:38
*** dklyle has quit IRC21:38
*** mriedem has quit IRC21:39
*** dklyle has joined #openstack-nova21:44
*** dklyle has quit IRC21:51
*** dklyle has joined #openstack-nova21:52
*** dklyle has quit IRC21:53
*** dklyle has joined #openstack-nova21:53
*** ociuhandu has joined #openstack-nova21:53
*** ociuhandu has quit IRC22:04
*** ociuhandu has joined #openstack-nova22:11
*** ociuhandu has quit IRC22:19
*** dklyle has quit IRC22:22
*** jamesdenton has joined #openstack-nova22:35
*** dklyle has joined #openstack-nova22:37
*** gyee has quit IRC22:39
*** pcaruana has quit IRC22:44
openstackgerritMerged openstack/nova stable/train: Add regression test for bug 1824435  https://review.opendev.org/69140222:50
openstackbug 1824435 in OpenStack Compute (nova) train "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt)22:50
openstackgerritMerged openstack/nova stable/queens: libvirt: Ignore volume exceptions during post_live_migration  https://review.opendev.org/69128422:50
*** macz has quit IRC23:02
*** ociuhandu has joined #openstack-nova23:11
*** dannins has quit IRC23:12
*** ociuhandu has quit IRC23:16
*** mdbooth has quit IRC23:22
*** mdbooth has joined #openstack-nova23:23
*** ociuhandu has joined #openstack-nova23:38
*** ociuhandu has quit IRC23:46
*** ociuhandu has joined #openstack-nova23:47
openstackgerritArtom Lifshitz proposed openstack/nova stable/train: Avoid error 500 on shelve task_state race  https://review.opendev.org/69262823:50
*** ociuhandu has quit IRC23:54
« Thursday, 2019-10-31 Index Saturday, 2019-11-02 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!