| *** ociuhandu has joined #openstack-nova | 00:00 | |
| *** ociuhandu has quit IRC | 00:05 | |
| *** Liang__ has quit IRC | 00:06 | |
| *** mriedem has joined #openstack-nova | 00:38 | |
| *** ociuhandu has joined #openstack-nova | 00:56 | |
| *** ociuhandu has quit IRC | 01:04 | |
| *** mriedem has quit IRC | 01:10 | |
| *** slaweq__ has joined #openstack-nova | 01:19 | |
| *** slaweq__ is now known as slaweq | 01:19 | |
| gibi_ptg | ~> | 01:25 |
|---|---|---|
| *** nanzha has joined #openstack-nova | 01:27 | |
| gibi_ptg | stephenfin: hi! it would be nice to sync up about the nova project update today after 17:30 | 01:32 |
| *** ociuhandu has joined #openstack-nova | 01:37 | |
| *** ociuhandu has quit IRC | 01:45 | |
| *** bnemec has quit IRC | 01:45 | |
| *** sapd1 has quit IRC | 01:48 | |
| *** sapd1 has joined #openstack-nova | 01:48 | |
| *** markvoelker has joined #openstack-nova | 01:56 | |
| *** markvoelker has quit IRC | 02:01 | |
| *** sapd1 has quit IRC | 02:01 | |
| *** ociuhandu has joined #openstack-nova | 02:29 | |
| *** ociuhandu has quit IRC | 02:33 | |
| *** nanzha has quit IRC | 02:36 | |
| *** nanzha has joined #openstack-nova | 02:37 | |
| *** artom has quit IRC | 02:41 | |
| *** yaawang has quit IRC | 02:44 | |
| *** slaweq has quit IRC | 02:52 | |
| *** mkrai has joined #openstack-nova | 02:52 | |
| *** bnemec has joined #openstack-nova | 02:59 | |
| *** mkrai has quit IRC | 03:01 | |
| *** mkrai_ has joined #openstack-nova | 03:01 | |
| *** mkrai_ has quit IRC | 03:14 | |
| *** tkajinam has joined #openstack-nova | 03:18 | |
| *** mkrai has joined #openstack-nova | 03:19 | |
| *** psachin has joined #openstack-nova | 03:21 | |
| *** slaweq has joined #openstack-nova | 03:32 | |
| *** ricolin has joined #openstack-nova | 03:33 | |
| *** bnemec has quit IRC | 03:34 | |
| *** xek_ has joined #openstack-nova | 03:36 | |
| *** xek_ has quit IRC | 03:38 | |
| *** tbachman has joined #openstack-nova | 03:38 | |
| *** xek_ has joined #openstack-nova | 03:39 | |
| *** tbachman_ has joined #openstack-nova | 03:42 | |
| *** tbachman has quit IRC | 03:43 | |
| *** tbachman_ is now known as tbachman | 03:43 | |
| *** adrianc_ has joined #openstack-nova | 03:44 | |
| *** bnemec has joined #openstack-nova | 03:46 | |
| *** xek_ has quit IRC | 03:47 | |
| *** slaweq has quit IRC | 03:48 | |
| *** slaweq has joined #openstack-nova | 03:50 | |
| *** ricolin_ has joined #openstack-nova | 03:55 | |
| *** markvoelker has joined #openstack-nova | 03:57 | |
| *** ricolin has quit IRC | 03:57 | |
| *** ricolin_ has quit IRC | 03:57 | |
| *** ricolin has joined #openstack-nova | 03:58 | |
| *** markvoelker has quit IRC | 04:02 | |
| *** davee__ has joined #openstack-nova | 04:03 | |
| *** davee_ has quit IRC | 04:05 | |
| *** tkajinam has quit IRC | 04:19 | |
| *** ricolin has quit IRC | 04:23 | |
| *** slaweq has quit IRC | 04:24 | |
| *** bnemec has quit IRC | 04:32 | |
| *** adrianc_ has quit IRC | 04:32 | |
| *** yaawang has joined #openstack-nova | 04:58 | |
| *** psachin has quit IRC | 04:59 | |
| *** mkrai has quit IRC | 05:00 | |
| *** ricolin has joined #openstack-nova | 05:08 | |
| *** ricolin has quit IRC | 05:09 | |
| *** slaweq_ has joined #openstack-nova | 05:12 | |
| *** slaweq_ has quit IRC | 05:22 | |
| *** ratailor has joined #openstack-nova | 05:22 | |
| *** slaweq_ has joined #openstack-nova | 05:23 | |
| *** bnemec has joined #openstack-nova | 05:27 | |
| *** slaweq_ has quit IRC | 05:35 | |
| *** gibi_cn has joined #openstack-nova | 05:36 | |
| gibi_cn | alex_xu: talked to stephenfin we will meet up at 14:00 in front of the Marketplace entrance | 05:38 |
| *** gibi_cn has quit IRC | 05:48 | |
| *** bnemec has quit IRC | 05:50 | |
| *** ociuhandu has joined #openstack-nova | 05:52 | |
| *** links has joined #openstack-nova | 05:53 | |
| *** ileixe has quit IRC | 05:53 | |
| *** slaweq_ has joined #openstack-nova | 05:54 | |
| *** ileixe has joined #openstack-nova | 05:56 | |
| *** ociuhandu has quit IRC | 05:57 | |
| *** markvoelker has joined #openstack-nova | 05:58 | |
| *** adrianc_ has joined #openstack-nova | 05:59 | |
| *** ratailor_ has joined #openstack-nova | 06:00 | |
| *** ratailor has quit IRC | 06:02 | |
| *** slaweq_ has quit IRC | 06:03 | |
| *** markvoelker has quit IRC | 06:03 | |
| *** adrianc__ has joined #openstack-nova | 06:19 | |
| *** adrianc_ has quit IRC | 06:22 | |
| *** adrianc__ has quit IRC | 06:27 | |
| *** adrianc_ has joined #openstack-nova | 06:28 | |
| *** bnemec has joined #openstack-nova | 06:31 | |
| *** jawad_axd has joined #openstack-nova | 06:31 | |
| *** sapd1 has joined #openstack-nova | 06:34 | |
| *** markvoelker has joined #openstack-nova | 06:36 | |
| *** ratailor__ has joined #openstack-nova | 06:41 | |
| *** ratailor_ has quit IRC | 06:43 | |
| *** adrianc_ has quit IRC | 06:52 | |
| *** markvoelker has quit IRC | 06:53 | |
| *** markvoelker has joined #openstack-nova | 06:53 | |
| *** bnemec has quit IRC | 06:56 | |
| *** markvoelker has quit IRC | 06:58 | |
| *** nanzha has quit IRC | 07:00 | |
| *** dpawlik has joined #openstack-nova | 07:05 | |
| *** ileixe has quit IRC | 07:09 | |
| *** ileixe has joined #openstack-nova | 07:09 | |
| *** nanzha has joined #openstack-nova | 07:10 | |
| *** jawad_axd has quit IRC | 07:14 | |
| *** slaweq has joined #openstack-nova | 07:30 | |
| *** jawad_axd has joined #openstack-nova | 07:30 | |
| *** dpawlik has quit IRC | 07:32 | |
| *** ratailor__ is now known as ratailor|lunch | 07:33 | |
| *** dpawlik has joined #openstack-nova | 07:36 | |
| *** slaweq has quit IRC | 07:38 | |
| *** luksky has joined #openstack-nova | 07:49 | |
| *** ociuhandu has joined #openstack-nova | 07:53 | |
| *** ociuhandu has quit IRC | 07:53 | |
| *** ociuhandu has joined #openstack-nova | 07:55 | |
| *** markvoelker has joined #openstack-nova | 07:55 | |
| *** damien_r has joined #openstack-nova | 07:58 | |
| *** damien_r has quit IRC | 07:58 | |
| *** slaweq has joined #openstack-nova | 07:59 | |
| *** damien_r has joined #openstack-nova | 07:59 | |
| *** ociuhandu has quit IRC | 08:01 | |
| *** damien_r has quit IRC | 08:03 | |
| *** damien_r has joined #openstack-nova | 08:03 | |
| *** markvoelker has quit IRC | 08:06 | |
| *** markvoelker has joined #openstack-nova | 08:07 | |
| *** ygk_12345 has joined #openstack-nova | 08:19 | |
| *** ygk_12345 has quit IRC | 08:19 | |
| *** tesseract has joined #openstack-nova | 08:24 | |
| *** pcaruana has joined #openstack-nova | 08:30 | |
| *** ratailor|lunch is now known as ratailor | 08:34 | |
| *** links has quit IRC | 08:39 | |
| *** links has joined #openstack-nova | 08:47 | |
| *** slaweq has quit IRC | 08:50 | |
| *** ralonsoh has joined #openstack-nova | 08:52 | |
| *** slaweq has joined #openstack-nova | 08:52 | |
| *** mrch_ has joined #openstack-nova | 09:00 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Add flavor group https://review.opendev.org/663563 | 09:06 |
| *** luksky has quit IRC | 09:21 | |
| *** cdent has joined #openstack-nova | 09:25 | |
| *** slaweq has quit IRC | 09:27 | |
| *** HagunKim has joined #openstack-nova | 09:29 | |
| *** derekh has joined #openstack-nova | 09:35 | |
| *** markvoelker has quit IRC | 09:42 | |
| *** dpawlik has quit IRC | 09:48 | |
| *** luksky has joined #openstack-nova | 09:49 | |
| *** markvoelker has joined #openstack-nova | 09:50 | |
| *** dpawlik has joined #openstack-nova | 09:52 | |
| *** markvoelker has quit IRC | 09:54 | |
| *** chason has joined #openstack-nova | 09:54 | |
| *** markvoelker has joined #openstack-nova | 09:55 | |
| *** markvoelker has quit IRC | 09:56 | |
| *** markvoelker has joined #openstack-nova | 09:56 | |
| *** markvoelker has quit IRC | 09:58 | |
| *** markvoelker has joined #openstack-nova | 09:59 | |
| *** chason has quit IRC | 09:59 | |
| *** jraju__ has joined #openstack-nova | 10:01 | |
| *** links has quit IRC | 10:02 | |
| *** lpetrut has joined #openstack-nova | 10:02 | |
| *** dtantsur|afk is now known as dtantsur | 10:15 | |
| *** markvoelker has quit IRC | 10:24 | |
| *** ratailor has quit IRC | 10:28 | |
| *** brault has joined #openstack-nova | 10:36 | |
| *** xek_ has joined #openstack-nova | 10:45 | |
| *** xek_ has quit IRC | 10:46 | |
| *** xek has joined #openstack-nova | 10:48 | |
| *** ociuhandu has joined #openstack-nova | 10:55 | |
| *** ociuhandu has quit IRC | 10:56 | |
| *** ociuhandu has joined #openstack-nova | 10:56 | |
| *** links has joined #openstack-nova | 11:04 | |
| *** jraju__ has quit IRC | 11:05 | |
| *** ociuhandu has quit IRC | 11:10 | |
| *** markvoelker has joined #openstack-nova | 11:10 | |
| *** tbachman has quit IRC | 11:29 | |
| *** CeeMac has joined #openstack-nova | 11:35 | |
| *** xek has quit IRC | 11:35 | |
| *** xek has joined #openstack-nova | 11:36 | |
| *** HagunKim has quit IRC | 11:38 | |
| *** tetsuro has quit IRC | 11:42 | |
| *** markvoelker has quit IRC | 11:48 | |
| *** mrch_ has quit IRC | 11:51 | |
| *** mrch_ has joined #openstack-nova | 11:52 | |
| openstackgerrit | Merged openstack/nova stable/train: Avoid error 500 on shelve task_state race https://review.opendev.org/692628 | 11:52 |
| *** arxcruz is now known as arxcruz|lunch | 11:55 | |
| *** brault has quit IRC | 11:58 | |
| *** nanzha has quit IRC | 12:02 | |
| *** brault has joined #openstack-nova | 12:03 | |
| *** nanzha has joined #openstack-nova | 12:10 | |
| *** xek has quit IRC | 12:22 | |
| *** ociuhandu has joined #openstack-nova | 12:22 | |
| *** jraju__ has joined #openstack-nova | 12:22 | |
| *** links has quit IRC | 12:23 | |
| *** xek has joined #openstack-nova | 12:25 | |
| *** markvoelker has joined #openstack-nova | 12:27 | |
| *** markvoelker has quit IRC | 12:28 | |
| *** xek has quit IRC | 12:28 | |
| *** xek has joined #openstack-nova | 12:29 | |
| *** jawad_axd has quit IRC | 12:31 | |
| *** slaweq_ has joined #openstack-nova | 12:31 | |
| *** jawad_axd has joined #openstack-nova | 12:31 | |
| *** FlorianFa has joined #openstack-nova | 12:36 | |
| *** ygk_12345 has joined #openstack-nova | 12:42 | |
| *** ociuhandu has quit IRC | 12:44 | |
| *** ociuhandu has joined #openstack-nova | 12:45 | |
| *** slaweq_ has quit IRC | 12:45 | |
| *** dviroel has joined #openstack-nova | 12:46 | |
| *** zbr is now known as zbr|ooo | 12:48 | |
| *** ociuhandu has quit IRC | 12:51 | |
| *** dpawlik has quit IRC | 12:55 | |
| *** arxcruz|lunch is now known as arxcruz | 12:55 | |
| *** mrch has joined #openstack-nova | 12:56 | |
| *** mrch has quit IRC | 12:56 | |
| *** mrch has joined #openstack-nova | 12:57 | |
| *** mrch_ has quit IRC | 12:59 | |
| *** mrch_ has joined #openstack-nova | 13:00 | |
| *** markvoelker has joined #openstack-nova | 13:00 | |
| *** markvoelker has quit IRC | 13:01 | |
| *** dpawlik has joined #openstack-nova | 13:01 | |
| *** nanzha has quit IRC | 13:02 | |
| *** mrch has quit IRC | 13:02 | |
| *** nanzha has joined #openstack-nova | 13:02 | |
| *** markvoelker has joined #openstack-nova | 13:04 | |
| *** tbachman has joined #openstack-nova | 13:04 | |
| *** yoctozepto has quit IRC | 13:17 | |
| *** yoctozepto has joined #openstack-nova | 13:21 | |
| *** ociuhandu has joined #openstack-nova | 13:25 | |
| *** mriedem has joined #openstack-nova | 13:25 | |
| *** henriqueof has joined #openstack-nova | 13:26 | |
| *** markvoelker has quit IRC | 13:36 | |
| *** markvoelker has joined #openstack-nova | 13:36 | |
| *** ociuhandu has quit IRC | 13:42 | |
| *** ociuhandu has joined #openstack-nova | 13:43 | |
| *** ociuhandu has quit IRC | 13:49 | |
| *** markvoelker has quit IRC | 13:51 | |
| mriedem | dansmith: so back in rocky the aggregate mirroring stuff was added with a todo to remove some placement failures in the api if you didn't have the api configured for placement (to ease upgrades), | 13:54 |
| mriedem | i'm removing that handling now but there is one thing that is bugging me on the remove case, | 13:54 |
| mriedem | https://review.opendev.org/#/c/660852/1/nova/compute/api.py@5615 | 13:54 |
| mriedem | which is the sync_aggregates command can't be used to sync and remove hosts from aggregates, only add | 13:54 |
| mriedem | so i'm trying to decide if placement errors in the remove case should just be caught and logged or actually result in the compute API operation failing | 13:55 |
| mriedem | i'm thinking the latter since otherwise it means we're just silently failing and your scheduling results could be screwed up, and it would be confusing to figure that out later | 13:55 |
| *** mmethot has joined #openstack-nova | 14:02 | |
| *** ygk_12345 has quit IRC | 14:13 | |
| mriedem | nevermind, going with the latter. it's a bigger diff but it's more correct i think | 14:16 |
| dansmith | mriedem: okay can read that stuff in a sec | 14:26 |
| dansmith | also, zomg my aggregates patch merged | 14:26 |
| dansmith | er, notificatons I mean | 14:26 |
| *** victor286 has joined #openstack-nova | 14:29 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove PlacementAPIConnectFailure handling from AggregateAPI https://review.opendev.org/660852 | 14:29 |
| mriedem | yeah a few things did over the weekend | 14:29 |
| mriedem | it's a miracle | 14:30 |
| cdent | praise be | 14:30 |
| *** artom has joined #openstack-nova | 14:31 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove PlacementAPIConnectFailure handling from AggregateAPI https://review.opendev.org/660852 | 14:31 |
| mriedem | cdent: i finally got that cleaned up ^ what i had sitting didn't ... sit well with me | 14:31 |
| mriedem | because it was like, "oh something failed, wah, nice try retrying it sucker" | 14:31 |
| cdent | a good lesson for entitled whippersnappers, I say | 14:33 |
| cdent | learn em up about life | 14:34 |
| mriedem | back in my day we didn't have idempotent apis | 14:34 |
| mriedem | we had to fight to delete things | 14:34 |
| cdent | exactly so | 14:34 |
| cdent | What has idempotency got us? Social media addiction! | 14:35 |
| *** eharney has joined #openstack-nova | 14:35 | |
| mriedem | that's a bit of a stretch, even for me | 14:36 |
| *** jraju__ has quit IRC | 14:41 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Log reason for remove_host action failing https://review.opendev.org/692833 | 14:41 |
| *** artom has quit IRC | 14:42 | |
| *** artom has joined #openstack-nova | 14:42 | |
| mriedem | dansmith: i know it's a bit big and not very fun to review, but the bottom few patches of the cross-cell-resize series are ready for review and +2ed by gibi if you get some time to sift through one or two of those | 14:43 |
| dansmith | sure, I need to get back to that, sorry | 14:44 |
| mriedem | np | 14:44 |
| mriedem | huh, maybe another case for long_rpc_timeout - for reserve_block_device_name call to compute when attaching a volume https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4 | 14:46 |
| mriedem | we must have some really slow nodes in the gate | 14:46 |
| *** nanzha has quit IRC | 14:48 | |
| *** jaosorior has joined #openstack-nova | 14:51 | |
| *** jawad_axd has quit IRC | 14:54 | |
| *** bnemec has joined #openstack-nova | 14:54 | |
| *** jawad_axd has joined #openstack-nova | 14:54 | |
| mriedem | inap and ovh again | 14:54 |
| mriedem | just like in the 'state of the gate' email | 14:54 |
| *** nanzha has joined #openstack-nova | 14:55 | |
| *** dpawlik has quit IRC | 14:55 | |
| *** xek has quit IRC | 14:56 | |
| *** dpawlik has joined #openstack-nova | 14:56 | |
| *** ociuhandu has joined #openstack-nova | 14:56 | |
| *** jawad_ax_ has joined #openstack-nova | 14:57 | |
| efried | mriedem: is https://review.opendev.org/#/c/692550/ (long_rpc_timeout) going to manifest in the gate anywhere in a way it can be seen easily? I would think not; we would have to get lucky and see an operation that takes longer than the old timeout. | 14:57 |
| efried | just wondering if we're waiting to see anything in particular before approving ^ | 14:57 |
| mriedem | not likely, the timeout is 30 minutes by default | 14:58 |
| mriedem | tempest would timeout way before that | 14:58 |
| *** xek has joined #openstack-nova | 14:58 | |
| efried | I don't mean hitting the long one, I mean seeing an operation that takes somewhere *between* the old timeout and the new one (i.e. would have timed out before this change but doesn't now) | 14:58 |
| *** jawad_axd has quit IRC | 14:59 | |
| efried | anyway, I'm getting ready to +2 that one, unless you or dansmith tells me there's a reason not to. | 15:00 |
| mriedem | i noticed it because of a gate failure http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010494.html | 15:00 |
| mriedem | if that's what you're asking | 15:00 |
| dansmith | efried: I can look after I'm done with a review | 15:01 |
| mriedem | api response timed out waiting for POST /allocations to finish which took 3 minutes b/c we had no i/o on the node | 15:01 |
| mriedem | ^ is not realistic though, and i'm not sure why that would happen in the gate | 15:01 |
| efried | right, my point is, in order to see it "working", we would have to find that same situation again, but where it *didn't* timeout, but took longer than the original timeout. | 15:01 |
| mriedem | as cdent said in the ML, what do we do if this happens? | 15:01 |
| *** jawad_ax_ has quit IRC | 15:01 | |
| mriedem | efried: that is probably not very possible without adding some logging that logs if an operation took over a given threshold, | 15:02 |
| efried | yeah, seems like we've ascertained the I/O thing is just overloading. | 15:02 |
| *** mugsie has quit IRC | 15:02 | |
| mriedem | i'm not sure if there is logging in oslo.messaging that goes off if an operation is over the given 60 second heartbeat, but there probably is | 15:02 |
| efried | It's not worth it, I was just asking whether there would be any other way to tell. Doesn't sound like it. But that's not a reason not to merge the thing. | 15:02 |
| efried | The harm would be, if something is wrong in such a way that it's never going to come back, you've just wasted time. | 15:03 |
| efried | but such is the eternal dilemma of timeouts. | 15:03 |
| efried | Re the I/O thing, perhaps it would benefit us to have fewer zuul nodes if it means a lower rate of spurious failure. | 15:04 |
| mriedem | i see this in oslo.messaging but it's debug level which we don't index https://github.com/openstack/oslo.messaging/blob/6bca848f5b272149bb32353a62f9d37108fcbe15/oslo_messaging/_drivers/amqpdriver.py#L533 | 15:04 |
| mriedem | what do you mean by fewer zuul nodes? | 15:04 |
| efried | or, if what cdent suggested would work, if making the CI nodes less CPU-powerful would reduce their ability to burden the I/O. Same result, really: overall fewer runs occur in a given amount of time, but if the success rate increases, it's a win. | 15:05 |
| *** mugsie has joined #openstack-nova | 15:05 | |
| efried | mriedem: I mean, if the I/O is overloaded, try to make it less overloaded by reducing the number of things hitting it at a time. | 15:06 |
| mriedem | reducing cpu is likely going to cause failures in other ways i'd think | 15:06 |
| *** markvoelker has joined #openstack-nova | 15:06 | |
| mriedem | historically cpu/ram has only gotten higher in devstack nodes in the gate over time | 15:06 |
| mriedem | when i started in openstack a full dsvm-tempest job took 45 minutes and could use 4vcpu/gb ram | 15:06 |
| mriedem | now we're basically topped out at 8cpu/ram with 2 api workers per control service | 15:07 |
| dansmith | efried: oh yeah that migrate_server one, mriedem and I discussed last week, so probably good for a +2 | 15:07 |
| *** dpawlik has quit IRC | 15:07 | |
| *** spatel has joined #openstack-nova | 15:07 | |
| efried | dansmith: that's done, waiting for your +A | 15:07 |
| mriedem | we could try dropping API_WORKERS=1 in devstack if that would help with load | 15:07 |
| dansmith | yup yup | 15:07 |
| mriedem | but we might hit more timeouts with that, idk | 15:07 |
| efried | mriedem: except that would slow down everywhere | 15:08 |
| efried | It seems like we're hitting this problem on specific providers, yes? | 15:08 |
| dansmith | mriedem: might have to lower tempest parallelism too | 15:08 |
| mriedem | efried: yes, inap and ovh | 15:08 |
| mriedem | dansmith: i thought about that last week as well, | 15:08 |
| efried | So I'm looking for an answer that would affect those providers without impacting others. | 15:08 |
| mriedem | currently we're 4 tests at a time | 15:08 |
| dansmith | mriedem: I just mean if you reduce the api workers | 15:08 |
| mriedem | yeah...so if node_provider in inap/ovh, API_WORKERS=1, tempest concurrency=2, build_timeout * 2 | 15:09 |
| mriedem | or just get them to stop throttling disk io so much | 15:09 |
| dansmith | I dunno how large those pools are, but could we just run unit, functional, doc, etc jobs on those providers? | 15:09 |
| dansmith | I think we've done that kind of job-provider affinity before | 15:09 |
| mriedem | that's not a bad idea, | 15:09 |
| mriedem | and yeah i think provider job affinity has been done for baremetal jobs | 15:10 |
| dansmith | yeah | 15:10 |
| dansmith | if they ran *all* those kinds of jobs, the others could focus on the devstack jobs | 15:10 |
| *** dosaboy has quit IRC | 15:10 | |
| efried | I want to say the infra guys have poo-poohed that idea in the past | 15:11 |
| mriedem | i don't know how easy it would be | 15:11 |
| mriedem | it's likely not easy | 15:12 |
| *** ociuhandu has quit IRC | 15:12 | |
| efried | it was this thread I was thinking of most recently, but I don't see the response http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009595.html | 15:12 |
| *** ociuhandu has joined #openstack-nova | 15:12 | |
| mriedem | i'm assuming if we had a kind of node provider label for that kind of thing, then projects like nova could opt into which pool of providers each job runs in | 15:12 |
| dansmith | well, if we have some affinity today, I would think it'd be doable | 15:12 |
| *** eharney has quit IRC | 15:12 | |
| mriedem | because some functional jobs do use devstack in some projects | 15:13 |
| *** slaweq_ has joined #openstack-nova | 15:13 | |
| *** lpetrut has quit IRC | 15:13 | |
| dansmith | ah | 15:13 |
| mriedem | so i'd think this kind of thing has to be up to each project since they know the jobs they run | 15:13 |
| efried | http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009592.html | 15:14 |
| efried | also no response, hm | 15:14 |
| dansmith | ah yeah, that's exactly it | 15:14 |
| *** luyao has quit IRC | 15:17 | |
| *** mrch has joined #openstack-nova | 15:17 | |
| *** slaweq_ has quit IRC | 15:18 | |
| *** ociuhandu has quit IRC | 15:18 | |
| *** eharney has joined #openstack-nova | 15:18 | |
| mriedem | likely need to tag with infra or ask -infra people directly to comment | 15:19 |
| mriedem | and they are probably all in shanghai this week | 15:19 |
| *** tbachman has quit IRC | 15:19 | |
| efried | yeah | 15:19 |
| efried | fwiw it looks like ovn has ~200 nodes | 15:19 |
| dansmith | ovh? | 15:20 |
| efried | isn't that the provider that's been choking? | 15:20 |
| efried | http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1 | 15:20 |
| dansmith | you said ovn | 15:20 |
| efried | oh, sorry | 15:20 |
| efried | yes, ovh | 15:20 |
| dansmith | which is a network technology, so just making sure :) | 15:21 |
| mriedem | ez bake ovn | 15:21 |
| efried | taking mriedem back to his childhood | 15:21 |
| mriedem | dude i'm making some muffins in the thing right now | 15:21 |
| *** mrch_ has quit IRC | 15:21 | |
| efried | Takes *way* longer with these stupid CFLs | 15:22 |
| dansmith | pretty sure mriedem never left childhood | 15:22 |
| *** mrch_ has joined #openstack-nova | 15:22 | |
| * mriedem puts his legos away and pretends he never heard that | 15:22 | |
| *** mrch has quit IRC | 15:23 | |
| mriedem | i will say, since maya likes legos i do find myself tempted around xmas to splurge on a $150 lego set just so i can help build it | 15:23 |
| dansmith | nice | 15:23 |
| efried | donnyd: can you think of an easy way $nodepool_provider could switch up configuration to reduce the chance of choking I/O? | 15:24 |
| donnyd | On FN? | 15:24 |
| *** tbachman has joined #openstack-nova | 15:24 | |
| mriedem | "but dad i only like the friends and disney princess sets" "you're getting medieval castle gdi" | 15:24 |
| efried | specifically ovh seems to be the problem at the moment. | 15:25 |
| dansmith | mriedem: lol | 15:25 |
| *** JamesBenson has joined #openstack-nova | 15:25 | |
| donnyd | Oh, well I know I avoid that issue by using local storage on FN. Not sure what ovh has on the backend of their instances | 15:25 |
| efried | not asking for action on your part donnyd, just advice. If a patch has 10 zuul jobs an any one of them lands on ovh and chokes on I/O and times out, the whole patch has to be retried, which is a royal PITA. Been happening a really lot over the last week or two. | 15:26 |
| efried | Yeah, I guess they would need to look into where the actual bottleneck is. | 15:26 |
| efried | not sure if there's any way to tell from here via grafana... | 15:26 |
| donnyd | The reality is we should try and label jobs by what they are bound by | 15:26 |
| efried | heh, we were just talking about your email from Sept. | 15:27 |
| donnyd | If there is a cpu bound job, scheduling on FN would be less optimal | 15:27 |
| efried | I couldn't see anywhere infra had responded to that idea, but I thought I remembered them shutting it down hard for some reason. | 15:27 |
| cdent | efried, mriedem : what I meant by less cpu, was less cpu for all nodes, not per node | 15:27 |
| donnyd | But IO bound jobs will go like stink on FN | 15:27 |
| cdent | so that we can run fewr jobs | 15:27 |
| cdent | because we run too many at once | 15:28 |
| donnyd | I shut down FN when I put in my gen set | 15:28 |
| cdent | because oversubscribers are liars | 15:28 |
| efried | cdent: okay, that's what I was suggesting too. | 15:28 |
| donnyd | I oversub FN by a small margin, but not much | 15:28 |
| donnyd | Usually 100% of the memory is utilized | 15:29 |
| efried | so like a white lie? | 15:29 |
| cdent | donnyd in my experience what you're doing seems to be working much better than some of the other providers, so kudos to you | 15:29 |
| donnyd | But CPU can be oversub like 1.125 | 15:29 |
| donnyd | cdent: I need faster CPUs for sure | 15:29 |
| donnyd | Mine are so slow compared to others | 15:30 |
| mriedem | looking dstat when this messaging timeout happened https://zuul.opendev.org/t/openstack/build/ef0196fe84804b44ac106d011c8c29ea/log/controller/logs/screen-n-api.txt.gz?severity=4 around the time of the timeout cpu usage is low, io is basically 0 and load is spiked | 15:30 |
| donnyd | The biggest difference for FN is local NVME storage for all instances | 15:30 |
| cdent | question: isn't this something infra will already have a plan and solution for? | 15:30 |
| cdent | this is what cloud mgt/provisioning is all about, presumably? | 15:30 |
| mriedem | i wonder if these providers live migrate the ci vms around frequently? | 15:31 |
| donnyd | If there was a way to benchmark providers and then prefer them for jobs that are bound by something I think we could optimize the CI | 15:31 |
| donnyd | Butttt.... if that provider breaks or goes away the issue becomes jobs failing because they were dependent on that super fast thing provider_x does | 15:32 |
| mriedem | cdent: infra is just tenants so i'm not sure how much control they have | 15:32 |
| mriedem | besides may requesting minimums in a flavor or something | 15:32 |
| mriedem | ? | 15:32 |
| cdent | mriedem: but presumably they can tweak their consumption? | 15:32 |
| mriedem | *maybe | 15:32 |
| donnyd | mriedem: well a simple benchmark could expose what jobs run optimal | 15:32 |
| donnyd | That was my idea and it got quickly struck down because of the issue listed above efried | 15:34 |
| donnyd | I do all kinds of custom things and so does mnaser at vexxhost to support the CI... not sure how much usage it gets though. I think limestone is on board with it as well | 15:35 |
| efried | ah, thanks donnyd, I remember now | 15:35 |
| donnyd | NP | 15:35 |
| donnyd | Hopefully that helps | 15:36 |
| efried | So really $provider needs to be able to run $job. In this case ovh needs to go figure out what the bottleneck is and fix it. | 15:36 |
| donnyd | Bingo | 15:36 |
| *** ociuhandu has joined #openstack-nova | 15:36 | |
| efried | even if it's by reducing the number of nodes they provide | 15:36 |
| donnyd | But they probably don't even know its an issue till someone complains | 15:36 |
| efried | not sure where else to complain | 15:36 |
| donnyd | probably infra | 15:37 |
| efried | openstack-discuss@ has a thread | 15:37 |
| donnyd | You can always schedule the job using maybe the numa label made for sean-k-mooney | 15:37 |
| efried | I guess we wait until after the summit then, infra folks are going to be pretty sparse for the next week I imagine. | 15:38 |
| donnyd | That would ensure the job runs on FN vexxhost or limestone if I am not mistaken | 15:38 |
| efried | donnyd: yeah, but you have to put that in the job def itself, right? | 15:38 |
| efried | You can't do it one off to make a particular patch merge | 15:38 |
| donnyd | Yea, you have to assign it a label | 15:38 |
| donnyd | No | 15:38 |
| efried | that's our problem here, getting approved patches through the gate. | 15:38 |
| donnyd | Sry, that is correct efried | 15:38 |
| donnyd | Well patch the job with that label... then it will run anyways wont it | 15:39 |
| efried | heh, we should just change all our devstack-based job defs to run on the fast providers... | 15:39 |
| donnyd | Well I wouldn't call FN fast | 15:39 |
| efried | wait until somebody notices | 15:39 |
| efried | okay, s/fast/reliable/ | 15:39 |
| donnyd | Just better at IO bound jobs | 15:39 |
| efried | Yeah, I would rather my patch take 2h and succeed than 1.5h and fail. | 15:40 |
| *** david-lyle has quit IRC | 15:40 | |
| donnyd | I'm sure ironic and tripleo hate FN because my CPUs are old | 15:40 |
| *** david-lyle has joined #openstack-nova | 15:40 | |
| donnyd | And they are mostly CPU bound | 15:40 |
| donnyd | Give it a swing with the label | 15:40 |
| donnyd | And if it works, you have a bandaid | 15:41 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999 | 15:42 |
| *** macz has joined #openstack-nova | 15:46 | |
| *** ociuhandu has quit IRC | 15:48 | |
| *** ociuhandu has joined #openstack-nova | 15:49 | |
| *** dosaboy has joined #openstack-nova | 15:50 | |
| *** dosaboy has quit IRC | 15:50 | |
| *** dosaboy has joined #openstack-nova | 15:50 | |
| *** dosaboy has quit IRC | 15:51 | |
| *** dosaboy has joined #openstack-nova | 15:52 | |
| *** dosaboy has quit IRC | 15:53 | |
| *** markvoelker has quit IRC | 15:54 | |
| *** ociuhandu has quit IRC | 15:54 | |
| *** dosaboy has joined #openstack-nova | 15:55 | |
| *** david-lyle is now known as dklyle | 15:57 | |
| *** dtantsur is now known as dtantsur|afk | 15:57 | |
| *** jawad_axd has joined #openstack-nova | 15:59 | |
| *** tbachman has quit IRC | 15:59 | |
| *** mrch_ has quit IRC | 16:00 | |
| *** tbachman has joined #openstack-nova | 16:01 | |
| *** TxGirlGeek has joined #openstack-nova | 16:01 | |
| *** jawad_axd has quit IRC | 16:03 | |
| *** gyee has joined #openstack-nova | 16:03 | |
| *** tbachman has quit IRC | 16:05 | |
| *** tbachman has joined #openstack-nova | 16:11 | |
| efried | donnyd: I wasn't serious about the label. That would be pretty publicly greedy of us. | 16:14 |
| *** xek has quit IRC | 16:14 | |
| *** xek has joined #openstack-nova | 16:15 | |
| *** victor286 has quit IRC | 16:16 | |
| dansmith | mriedem: question for you in here while I look at the test: https://review.opendev.org/#/c/635646/48/nova/conductor/tasks/cross_cell_migrate.py | 16:19 |
| *** jawad_axd has joined #openstack-nova | 16:19 | |
| dansmith | holy hell, everything in the check/gate are running tests and only minutes old | 16:19 |
| dansmith | we should send the whole community to china more often | 16:19 |
| *** ociuhandu has joined #openstack-nova | 16:22 | |
| *** jawad_axd has quit IRC | 16:24 | |
| *** luksky has quit IRC | 16:24 | |
| donnyd | Well not really efried | 16:24 |
| donnyd | If a job has requirements to run and a specific label meets the requirements then I don't see the issue | 16:24 |
| donnyd | But it's your call... just pointing out the options | 16:25 |
| *** spatel has quit IRC | 16:25 | |
| mriedem | dansmith: thanks, replied | 16:26 |
| mriedem | dansmith: is my sarcasm detector picking this up correctly? https://review.opendev.org/#/c/635080/48/nova/tests/unit/compute/test_compute_mgr.py@10545 | 16:27 |
| dansmith | mriedem: yes. | 16:27 |
| *** ociuhandu has quit IRC | 16:27 | |
| mriedem | heh, verbose commentage is how i keep from feeling lonely | 16:28 |
| *** ociuhandu has joined #openstack-nova | 16:31 | |
| *** nanzha has quit IRC | 16:33 | |
| dansmith | mriedem: oops, my "...yes I see" was supposed to go into the fault_clone complaint nit | 16:34 |
| dansmith | went back to note that and picked the wrong comment to edit | 16:35 |
| dansmith | mriedem: one more question | 16:35 |
| *** jawad_axd has joined #openstack-nova | 16:40 | |
| openstackgerrit | Merged openstack/nova master: Use long_rpc_timeout in conductor migrate_server RPC API call https://review.opendev.org/692550 | 16:40 |
| mriedem | replied | 16:41 |
| *** dustinc_pto is now known as dustinc | 16:41 | |
| mriedem | agree about the mapping == add a unit test wrinkle for that? | 16:43 |
| mriedem | will do - and the variable name nits | 16:43 |
| *** jawad_axd has quit IRC | 16:45 | |
| *** jaosorior has quit IRC | 16:45 | |
| dansmith | cool | 16:45 |
| efried | dansmith: I just put a bunch more words in the 'flavor groups' spec. Would you still be -1 if the answer to the backward compat question were "you can ignore flavor groups and keep doing extra specs forever"? IMO there's still too much complexity (most of which still isn't addressed) to make it worthwhile. | 16:48 |
| *** mvkr has quit IRC | 16:48 | |
| efried | whoah, that timeout patch merged already?? Didn't we approve it like an hour ago?? | 16:49 |
| dansmith | efried: you mean would I still be effectively -2 I assume | 16:49 |
| efried | yeah | 16:49 |
| efried | pretty sure we're in agreement on it, I'm just wondering how aggressively I/we should say "despite feeding you paths to address all the holes, don't bother doing that, because this is going to die anyway" | 16:50 |
| dansmith | efried: and without reading, you mean keeping extra_specs for the hard flavor case and only allowing the groups for compose-ability? I still think it's practically useless without some way to define which things can be composed, and agree it's far too much change to too many fundamental things to be worth it or viable | 16:50 |
| efried | yeah, that's more or less what I said. | 16:50 |
| efried | in the review | 16:50 |
| dansmith | I think I'm still effectively -2 even with keeping extra_specs for those reasons, yeah | 16:51 |
| dansmith | I think that it's legit to -2 something like this on "this isn't the direction we want to go" and if you and I both -2 it for that reason, I think that is a fairly tight case | 16:51 |
| dansmith | it's not like we haven't considered it or reviewed it in detail | 16:51 |
| dansmith | certainly don't want to just flip him the bird, but you wanted to be more upfront with people (as do I) so... that'd be pretty upfront | 16:52 |
| efried | yeah, right now we're both -1.9 I think, but I don't want the author to take our -1s as a message that he should go try to fill it in. | 16:52 |
| dansmith | yeah, fair point | 16:52 |
| dansmith | I tried to call out my effectively -2ness for that reason, but maybe -2 would be better | 16:52 |
| efried | so if you wouldn't mind reading my latest comments (when you get a chance) and if you agree with what I'm saying, go ahead and -2 and I'll follow suit. | 16:53 |
| dansmith | sure | 16:53 |
| efried | thx | 16:53 |
| mriedem | i have run into "I don't want the author to take our -1s as a message that he should go try to fill it in." on a couple of specs for ussuri even though i've said a few times, "i don't think we need to do this, the existing alternatives or sufficient or this doesn't fit with the project IMO" but the specs continue to be updated | 16:53 |
| mriedem | *are sufficient | 16:54 |
| mriedem | i probably haven't -2ed because they are mostly trivial things to do, just plumbing, | 16:54 |
| mriedem | not anywhere near the complexity of this flavor groups tihng | 16:54 |
| mriedem | *thing | 16:54 |
| cdent | Explicit words might be the way to go. | 16:56 |
| dansmith | cdent: my words were explicit, but I hedged on the vote | 16:57 |
| dansmith | because people sometimes accuse me of closing the door to further reviews with a -2 or even a -1 | 16:57 |
| cdent | well closing the door is what's wanted here, yeah? | 16:58 |
| * cdent is all for early door closes | 16:58 | |
| dansmith | well, | 16:58 |
| dansmith | yes, but I want to do that in concert with other reviewers, like efried and I just did | 16:58 |
| dansmith | so which do I need to kill? there is public, private, and shared | 16:59 |
| dansmith | oops, wrong window | 16:59 |
| cdent | all three /me is all for early kills | 16:59 |
| cdent | ! | 16:59 |
| mriedem | https://review.opendev.org/#/c/682302/ and https://review.opendev.org/#/c/580336/ are the ones i'm struggling with fwiw | 17:02 |
| mriedem | the latter has been coming up since berlin | 17:02 |
| mriedem | and we have decent alternatives already available, though like i said the change itself would probably be pretty minimal | 17:03 |
| *** ociuhandu has quit IRC | 17:04 | |
| *** ociuhandu has joined #openstack-nova | 17:05 | |
| *** ociuhandu has quit IRC | 17:05 | |
| dansmith | efried: done | 17:06 |
| efried | thanks dansmith, following | 17:06 |
| *** ociuhandu has joined #openstack-nova | 17:06 | |
| *** damien_r has quit IRC | 17:07 | |
| dansmith | huh, devstack defaults to configuring us for unversioned notifications apparently? | 17:09 |
| mriedem | yeah, because downstream projects don't all use versioned | 17:10 |
| mriedem | i seem to remember this coming up recently(ish) too when debugging something for watcher | 17:10 |
| mriedem | i think b/c we (nova) changed our default | 17:11 |
| mriedem | yup https://review.opendev.org/#/q/Ied9d50b07c368d5c2be658c744f340a8d1ee41e0 | 17:11 |
| *** ociuhandu has quit IRC | 17:12 | |
| *** markvoelker has joined #openstack-nova | 17:15 | |
| *** mgoddard has quit IRC | 17:19 | |
| *** jawad_axd has joined #openstack-nova | 17:21 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Follow up to I3e28c0163dc14dacf847c5a69730ba2e29650370 https://review.opendev.org/692856 | 17:21 |
| *** mgoddard has joined #openstack-nova | 17:22 | |
| *** derekh has quit IRC | 17:23 | |
| mriedem | heh that long_rpc_timeout change just merge conflicted most of the cross-cell-resize series, yay | 17:23 |
| dansmith | red pill or blue pill... | 17:25 |
| *** jawad_axd has quit IRC | 17:25 | |
| mriedem | oh well, needed to work on adding unit tests to https://review.opendev.org/#/c/637630/47 anyway so might as well rebsae | 17:26 |
| mriedem | *rebase | 17:26 |
| * cdent awaits the power brownout | 17:27 | |
| mriedem | the gate is empty due to the summit so it's actually the best time | 17:27 |
| *** artom has quit IRC | 17:28 | |
| mriedem | that reminds me, | 17:31 |
| mriedem | efried: https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports-ussuri+status:open if you have some time - i can probably answer questions | 17:31 |
| mriedem | the bottom one is mostly only big b/c of tests | 17:32 |
| *** artom has joined #openstack-nova | 17:32 | |
| efried | mriedem: oh, yeah, lost on my list... | 17:33 |
| *** gregwork has joined #openstack-nova | 17:38 | |
| *** cdent has quit IRC | 17:41 | |
| *** ociuhandu has joined #openstack-nova | 17:53 | |
| *** ociuhandu has quit IRC | 17:58 | |
| *** jmlowe has quit IRC | 17:59 | |
| *** pcaruana has quit IRC | 18:08 | |
| *** pcaruana has joined #openstack-nova | 18:08 | |
| openstackgerrit | Merged openstack/nova stable/stein: Avoid error 500 on shelve task_state race https://review.opendev.org/692630 | 18:10 |
| openstackgerrit | Merged openstack/nova master: Add finish_snapshot_based_resize_at_dest compute method https://review.opendev.org/635080 | 18:10 |
| *** ociuhandu has joined #openstack-nova | 18:12 | |
| *** ralonsoh has quit IRC | 18:19 | |
| *** jmlowe has joined #openstack-nova | 18:22 | |
| dansmith | teehee | 18:35 |
| dansmith | starting up 100 fake compute services on one machine... load of over 40 as they all slam the ever-loving crap out of conductor trying to create their db entries | 18:35 |
| openstackgerrit | Merged openstack/nova master: Add FinishResizeAtDestTask https://review.opendev.org/635646 | 18:36 |
| *** amodi has joined #openstack-nova | 18:42 | |
| mriedem | dansmith: how big is the host? | 18:49 |
| dansmith | how big? | 18:49 |
| dansmith | about yay big | 18:50 |
| mriedem | i do the fake compute thing in a 8vcpu/8gb ram vm sometimes and can't really go over 30 computes | 18:50 |
| dansmith | about 19" wide | 18:50 |
| mriedem | cpu/ram | 18:50 |
| dansmith | oh, 32G memory, 8 cores if that's what you mean | 18:50 |
| mriedem | yeah | 18:50 |
| mriedem | did you adjust API_WORKERS or let it do ncpu/2? | 18:51 |
| dansmith | just defaults | 18:51 |
| mriedem | ah. fake computes huh. so all that time trying to figure out which network to delete and you found out you can't create a fake driver vm with networking anyway right? | 18:52 |
| mriedem | i always forget that when i do a devstack with the fake driver | 18:52 |
| *** tesseract has quit IRC | 18:53 | |
| mriedem | i had written up some of that here at one point https://docs.openstack.org/devstack/latest/guides/nova.html#fake-virt-driver | 18:53 |
| dansmith | one real compute and then a bunch of extra fakes on the same machine | 18:53 |
| mriedem | ah ok | 18:53 |
| mriedem | smells like someone is playing with aggregates | 18:53 |
| *** xek has quit IRC | 18:53 | |
| *** ociuhandu has quit IRC | 19:02 | |
| *** markvoelker has quit IRC | 19:06 | |
| *** efried has quit IRC | 19:06 | |
| *** efried has joined #openstack-nova | 19:07 | |
| mriedem | wonder if we can nuke this yet, nothing in or out of tree returns True for it that i can find https://opendev.org/openstack/nova/src/branch/master/nova/virt/driver.py#L1740 | 19:17 |
| mriedem | oh nvm i guess it defaults to True in the ComputeDriver parent class and zvm doesn't override it... | 19:18 |
| mriedem | or xenapi | 19:18 |
| *** mmethot has quit IRC | 19:29 | |
| *** ociuhandu has joined #openstack-nova | 19:33 | |
| *** ociuhandu has quit IRC | 19:38 | |
| *** ociuhandu has joined #openstack-nova | 19:38 | |
| *** mmethot has joined #openstack-nova | 19:38 | |
| *** mmethot has quit IRC | 19:46 | |
| *** ociuhandu has quit IRC | 19:48 | |
| *** ociuhandu has joined #openstack-nova | 19:51 | |
| *** spatel has joined #openstack-nova | 19:57 | |
| *** mmethot has joined #openstack-nova | 20:00 | |
| mriedem | artom: https://review.opendev.org/#/c/594139/ - am i misunderstanding your comment? | 20:07 |
| artom | mriedem, lemme reload context | 20:08 |
| artom | o_O | 20:09 |
| artom | I am apparently retarded. | 20:10 |
| mriedem | did your brain fart on how if/elif works? | 20:10 |
| artom | Clearly | 20:10 |
| artom | I remember double-checking myself as well | 20:10 |
| artom | Thinking "no way Matt did that" | 20:10 |
| mriedem | s'ok | 20:10 |
| mriedem | that rollback code is all gorpy | 20:11 |
| *** ociuhandu has quit IRC | 20:11 | |
| artom | mriedem, fixed | 20:12 |
| artom | mriedem, lemme look it over once again, then +1 | 20:12 |
| mriedem | sure, thanks | 20:13 |
| artom | I was busy shitting all over https://review.opendev.org/#/c/512815/8 | 20:13 |
| artom | Politely | 20:13 |
| openstackgerrit | Merged openstack/nova stable/rocky: Avoid error 500 on shelve task_state race https://review.opendev.org/692631 | 20:18 |
| eandersson | btw anyone got any experience with the keep alive issue and openstack services? I noticed a bug or two reported for nova and failures | 20:21 |
| eandersson | > 'Connection aborted.', BadStatusLine("''",) | 20:21 |
| *** abaindur has joined #openstack-nova | 20:22 | |
| eandersson | A few of the bug reports indicates that disabling keepalive would resolve some of these. | 20:22 |
| efried | mriedem: One question on the qos patches (I'm on the bottom one rn): It looks like some of the code is being hit for both evacuate and rebuild. It wouldn't do any harm for the latter, since it's just populating the rg/rp mappings. But just wanted to confirm that's what I'm seeing? | 20:26 |
| mriedem | artom: oh wow that's fun, smells like starlingx | 20:26 |
| artom | mriedem, yeah eh? | 20:26 |
| mriedem | starlingx did a lot of hacks with libvirt channels | 20:26 |
| efried | mriedem: specifically these changes? https://review.opendev.org/#/c/688387/8/nova/conductor/manager.py | 20:26 |
| mriedem | artom: good comments on that patch btw | 20:27 |
| mriedem | all valid poitns | 20:27 |
| mriedem | *points | 20:27 |
| artom | I'm useful! | 20:27 |
| mriedem | makes up for your lack of if/elif knowledge :) | 20:28 |
| artom | Hahaha | 20:28 |
| mriedem | efried: hmm, weird | 20:28 |
| mriedem | s/weird/where/? | 20:28 |
| mriedem | https://review.opendev.org/#/c/688387/8/nova/compute/manager.py only does the port mapping stuff if evacuate | 20:28 |
| mriedem | same in https://review.opendev.org/#/c/688387/8/nova/conductor/manager.py | 20:29 |
| mriedem | or are you asking, why do we only do that for evacuate? | 20:29 |
| efried | No | 20:29 |
| efried | in one file the bool is called 'evacuate', so I can buy that it's only happening for evacuate. In the other it's called 'recreate' and the comments imply we're hitting the path for both evacuate and rebuild. | 20:29 |
| mriedem | in compute manager that evacuate variable used to be called recreate as well, | 20:30 |
| efried | so either it's the same and the var name and comments are confusing, or... | 20:30 |
| mriedem | i changed that awhile back, | 20:30 |
| mriedem | it's still recreate in conductor b/c no one has renamed the variable for clarity yet | 20:30 |
| *** ociuhandu has joined #openstack-nova | 20:30 | |
| efried | okay. So recreate is evacuate. | 20:30 |
| efried | and... `not recreate` is rebuild, which is a kind of recreate, but whatever. | 20:31 |
| mriedem | right | 20:31 |
| mriedem | hence the confusion | 20:31 |
| efried | thanks for clarifying. | 20:31 |
| efried | someone should rename that variable. | 20:31 |
| mriedem | https://review.opendev.org/#/c/508190/ | 20:31 |
| mriedem | that's me doing it in compute | 20:31 |
| efried | ah, so you didn't actually try to rename it all the way through the stack. That would indeed be hairy. | 20:33 |
| mriedem | right, i wasn't going to touch rpc stuff for that | 20:33 |
| mriedem | and the driver interface is likely nbd but it's at least an email | 20:34 |
| *** abaindur has quit IRC | 20:34 | |
| mriedem | there is no in-tree driver that implements that anymore anyway, it was around for the old baremetal driver that eventually became ironic | 20:34 |
| *** ociuhandu has quit IRC | 20:36 | |
| *** luksky has joined #openstack-nova | 20:37 | |
| *** ociuhandu has joined #openstack-nova | 20:37 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: cond: rename 'recreate' var to 'evacuate' https://review.opendev.org/692900 | 20:39 |
| efried | mriedem: ^ | 20:39 |
| *** jmlowe has quit IRC | 20:40 | |
| *** ociuhandu has quit IRC | 20:42 | |
| mriedem | comments inline | 20:51 |
| *** eharney has quit IRC | 20:57 | |
| *** jmlowe has joined #openstack-nova | 21:00 | |
| *** tbachman has quit IRC | 21:02 | |
| *** jawad_axd has joined #openstack-nova | 21:03 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: cond: rename 'recreate' var to 'evacuate' https://review.opendev.org/692900 | 21:03 |
| efried | done and done | 21:04 |
| efried | and +A on gibi_ptg's patches. | 21:06 |
| mriedem | sweet | 21:07 |
| *** jawad_axd has quit IRC | 21:08 | |
| efried | Gonna go do some more damage on osc, then come back to vtpm | 21:09 |
| *** tbachman has joined #openstack-nova | 21:14 | |
| *** artom has quit IRC | 21:15 | |
| *** jawad_axd has joined #openstack-nova | 21:24 | |
| mriedem | heave ho | 21:24 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Follow up to I3e28c0163dc14dacf847c5a69730ba2e29650370 https://review.opendev.org/692856 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass exception through TaskBase.rollback https://review.opendev.org/692689 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute CrossCellMigrationTask from MigrationTask https://review.opendev.org/635668 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Refresh instance in MigrationTask.execute Exception handler https://review.opendev.org/669012 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Plumb allow_cross_cell_resize into compute API resize() https://review.opendev.org/635684 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted() https://review.opendev.org/636224 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize https://review.opendev.org/636253 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor https://review.opendev.org/642591 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing https://review.opendev.org/642592 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate https://review.opendev.org/643450 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing https://review.opendev.org/643451 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing https://review.opendev.org/669013 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize_at_source compute method https://review.opendev.org/637058 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add ConfirmResizeTask https://review.opendev.org/637070 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method https://review.opendev.org/637075 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API https://review.opendev.org/637316 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize_at_dest compute method https://review.opendev.org/637630 | 21:25 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations https://review.opendev.org/639453 | 21:25 |
| *** jawad_axd has quit IRC | 21:28 | |
| *** pcaruana has quit IRC | 21:30 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906 | 21:30 |
| openstack | bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,Triaged] https://launchpad.net/bugs/1824435 | 21:30 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907 | 21:30 |
| *** ociuhandu has joined #openstack-nova | 21:33 | |
| *** ociuhandu has quit IRC | 21:38 | |
| *** jawad_axd has joined #openstack-nova | 21:45 | |
| mriedem | melwitt: i saw this today https://i.imgur.com/beq9YYf.jpg | 21:48 |
| *** jawad_axd has quit IRC | 21:49 | |
| melwitt | lol yesss | 21:50 |
| *** efried has quit IRC | 21:55 | |
| *** mvkr has joined #openstack-nova | 21:56 | |
| *** efried has joined #openstack-nova | 21:57 | |
| *** eharney has joined #openstack-nova | 21:58 | |
| *** TxGirlGeek has quit IRC | 22:00 | |
| *** TxGirlGeek has joined #openstack-nova | 22:04 | |
| mriedem | dansmith: another spec which we've basically said no to in the past but the pushback is "we have some customers that want nova to do this" https://review.opendev.org/#/c/672400/ | 22:06 |
| mriedem | because pre-creating a port is hard | 22:06 |
| mriedem | if enough diverse users/vendors came forward asking for this like people did for passing through volume type on boot from volume then maybe it's a different discussion | 22:09 |
| dansmith | well, I haven't read that other than the commit message, but.. | 22:09 |
| dansmith | I'm pretty conflicted on these | 22:09 |
| dansmith | since we're split, it really seems like an impossible-to-win scenario chasing every attribute of the other service, especially when they version their api so differently from us | 22:10 |
| mriedem | "we're split" meaning we (nova) from neutron? | 22:10 |
| dansmith | but I also really sympathize with simple things being hard because of how we decided in 2013 to segregate the project | 22:10 |
| dansmith | yes | 22:10 |
| *** spatel has quit IRC | 22:11 | |
| mriedem | i'm pretty sure vnic_type is going to be in any neutron deployment because we rely on that extension pretty heavily https://docs.openstack.org/api-ref/network/v2/index.html#port-binding-extended-attributes | 22:11 |
| mriedem | alright, i won't block it | 22:12 |
| mriedem | i'm fairly certain starlingx had that in their nova fork already | 22:12 |
| mriedem | maybe sean-k-mooney would know if red hat wants that as well | 22:13 |
| dansmith | I'm not saying we shouldn't block it | 22:13 |
| dansmith | I'm saying we're screwed either way | 22:13 |
| *** tbachman has quit IRC | 22:20 | |
| *** jawad_axd has joined #openstack-nova | 22:21 | |
| *** jawad_axd has quit IRC | 22:25 | |
| *** ociuhandu has joined #openstack-nova | 22:30 | |
| *** dviroel has quit IRC | 22:34 | |
| *** ociuhandu has quit IRC | 22:39 | |
| *** jawad_axd has joined #openstack-nova | 22:41 | |
| *** luksky has quit IRC | 22:45 | |
| *** jawad_axd has quit IRC | 22:46 | |
| *** mriedem has quit IRC | 22:49 | |
| *** JamesBen_ has joined #openstack-nova | 22:57 | |
| *** JamesBenson has quit IRC | 23:00 | |
| openstackgerrit | Merged openstack/nova master: Allow evacuating server with port resource request https://review.opendev.org/688387 | 23:00 |
| openstackgerrit | Merged openstack/nova master: Enable evacuation with qos ports https://review.opendev.org/688688 | 23:00 |
| *** JamesBen_ has quit IRC | 23:01 | |
| *** jawad_axd has joined #openstack-nova | 23:02 | |
| *** jawad_axd has quit IRC | 23:07 | |
| *** ociuhandu has joined #openstack-nova | 23:10 | |
| efried | dansmith: help me here, I swear somewhere we had a functional test to validate the redirects in our .htaccess file?? | 23:11 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add regression test for bug 1824435 https://review.opendev.org/692906 | 23:14 |
| openstack | bug 1824435 in OpenStack Compute (nova) stein "fill_virtual_interface_list migration fails on second attempt" [Medium,In progress] https://launchpad.net/bugs/1824435 - Assigned to melanie witt (melwitt) | 23:14 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Remove redundant call to get/create default security group https://review.opendev.org/692907 | 23:14 |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Add integration testing for heal_allocations https://review.opendev.org/692923 | 23:14 |
| efried | found it | 23:14 |
| *** ociuhandu has quit IRC | 23:18 | |
| *** jawad_axd has joined #openstack-nova | 23:23 | |
| *** jawad_axd has quit IRC | 23:28 | |
| *** ociuhandu has joined #openstack-nova | 23:31 | |
| *** ociuhandu has quit IRC | 23:40 | |
| openstackgerrit | Merged openstack/nova stable/queens: Avoid error 500 on shelve task_state race https://review.opendev.org/692632 | 23:43 |
| openstackgerrit | Merged openstack/nova stable/stein: Imported Translations from Zanata https://review.opendev.org/692701 | 23:43 |
| *** jawad_axd has joined #openstack-nova | 23:44 | |
| *** macz has quit IRC | 23:46 | |
| *** macz has joined #openstack-nova | 23:47 | |
| *** jawad_axd has quit IRC | 23:48 | |
| *** Liang__ has joined #openstack-nova | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
