| opendevreview | melanie witt proposed openstack/nova master: Support create with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870932 | 03:45 |
|---|---|---|
| opendevreview | melanie witt proposed openstack/nova master: Validate key manager create access in API https://review.opendev.org/c/openstack/nova/+/919989 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Clean up unused ephemeral encryption libvirt secrets https://review.opendev.org/c/openstack/nova/+/919990 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: libvirt: "Auto heal" ephemeral encryption secrets on guest launch https://review.opendev.org/c/openstack/nova/+/919991 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Support (resize|cold migration) with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870933 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Support live migration with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/905512 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Support rebuild with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870939 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Add Glance standardized encryption image properties https://review.opendev.org/c/openstack/nova/+/921345 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Support encrypted source images for qcow2 https://review.opendev.org/c/openstack/nova/+/919992 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Add encryption support to qemu-img rebase https://review.opendev.org/c/openstack/nova/+/870936 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Add backing_encryption_secret_uuid to BlockDeviceMapping https://review.opendev.org/c/openstack/nova/+/907960 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Support encrypted backing files for qcow2 https://review.opendev.org/c/openstack/nova/+/907961 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Support rescue with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/873675 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Support snapshot with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/870937 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: Support cross cell resize with ephemeral encryption for qcow2 https://review.opendev.org/c/openstack/nova/+/909595 | 03:45 |
| opendevreview | melanie witt proposed openstack/nova master: libvirt: Introduce support for raw with LUKS https://review.opendev.org/c/openstack/nova/+/884313 | 03:45 |
| *** ykarel_ is now known as ykarel | 07:01 | |
| sahid | 13:09 < sahid> o/ | 11:11 |
| sahid | 13:10 < sahid> regarding placement, is there a way to make the in_tree query using POST instead of GET? | 11:11 |
| sahid | 13:10 < sahid> we are using routed provider network, which create many aggregates per hosts | 11:11 |
| sahid | 13:11 < sahid> we are in that situation where the URI is too long when calling an placement API with request that use in_tree | 11:12 |
| sahid | bauzas: perhaps you have some idea | 11:12 |
| sahid | hum in same time using POST does not look right in an API's point of view | 11:50 |
| sahid | we can perhaps improve nova to batch requests | 11:50 |
| sean-k-mooney | sahid: in general no | 12:49 |
| sean-k-mooney | we had discussed moving the query form the uri to a post boday a long time ago but there was resitance to doing that | 12:50 |
| bauzas | sahid: sorry, have a flapping network for now | 13:00 |
| sean-k-mooney | sahid: unless your hitting the RFC spec limit on a uri, which i think is 64k you are likely hitting a limit in your loadbalancer or webserver | 13:01 |
| sean-k-mooney | so that is what i woudl try adjusting first | 13:01 |
| sean-k-mooney | sahid: api changes are not backporatbale and allowing an alternitive way to query palcyement would be an api change | 13:01 |
| sean-k-mooney | so for your current deployment that is really the only option aviable (adjust the limit in your loadbalabcer/webserver) or reduce the number of network segments | 13:02 |
| sahid | sean-k-mooney: yes thank you it's basically on what we are currently working | 13:08 |
| sahid | in same time i'm going to impl fix for our current release, I'm not sure that is going to change the API | 13:09 |
| sean-k-mooney | what kind of fix | 13:09 |
| sean-k-mooney | modifying the qurey size limit is valid but modifing the call to placement would be ill advised | 13:10 |
| sahid | yes rights, give me a moment I'm going to share it with you | 13:11 |
| sean-k-mooney | batching request is also proably hnot someting we woudl do in nova | 13:11 |
| sean-k-mooney | that has performance and correctness issues | 13:11 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | 13:12 |
| sahid | sean-k-mooney: i guess but you also want to handle enviroment that scale | 13:13 |
| sean-k-mooney | sure but in general we would not expec there to be more hten a hand ful of network segments per neutorn providree network and more then a hand full of ports per vm | 13:15 |
| sean-k-mooney | i.e. 5 segments per network and 5 port per vm for 25 ish uuids max for a large vm/deployment | 13:16 |
| sean-k-mooney | you can have more but i would think that is very atypical | 13:16 |
| sahid | sean-k-mooney: with rpn you can't have more than one physnet attached to a segment for a given network, that means if you want to have small broadcast domain you have to have lot of networks | 13:17 |
| sahid | I have fixed that missing in neutron, but it's only available start to zed | 13:18 |
| sean-k-mooney | yes having lots of network is intened apprch | 13:19 |
| sahid | for each host, you have an agregate related that is created by neutron | 13:19 |
| sean-k-mooney | not one large network with lots of segments | 13:19 |
| sahid | to ensure the vm is schedule on a host that handle the requested physnet | 13:19 |
| sean-k-mooney | you do not have 1 aggreate per host | 13:19 |
| sean-k-mooney | you have one aggrate per network segment and all hosts on the relevent phsynet related to the segment are added to that aggreate | 13:20 |
| sahid | you have a aggregate for each segments, (sorry) | 13:20 |
| sean-k-mooney | this is a know design limiattion with segment by the way | 13:20 |
| sahid | if you have 100 networks and two physnet per networks, you hqve 200 aggregates | 13:20 |
| sean-k-mooney | i orgially tried to have neutron not sue physnest for that mapping | 13:20 |
| sahid | sean-k-mooney: yes fixed for zed | 13:20 |
| sean-k-mooney | so in zed you can create segments and map the to specific hosts? | 13:21 |
| sean-k-mooney | without using phsynets | 13:21 |
| sean-k-mooney | adding multiple phsynets to a network is not reallly the correct way to model this | 13:21 |
| sahid | https://github.com/openstack/neutron-specs/blob/master/specs/zed/multiple-routed-provider-network-per-compute.rst | 13:21 |
| sean-k-mooney | you should have muliptle phsynets but you should not have network span physnets | 13:22 |
| sahid | sean-k-mooney: you want ability to extend you using subnet to take advantage of the scheduling, basically the user request a network, if one subnet is out of ips, then the process switch to the next subnet | 13:23 |
| sahid | if we don't have this mech, the vm is in error | 13:23 |
| sahid | which is not really "cloud" | 13:23 |
| sean-k-mooney | sure but you can have multipel subnets mapped to the same segment | 13:23 |
| sean-k-mooney | segmetns were ment to model l2 broadcast domains | 13:24 |
| sean-k-mooney | and be mappable to indeivgual hosts | 13:24 |
| sean-k-mooney | there was an api for that in the orginal propaosl | 13:24 |
| sean-k-mooney | and along the way it got replaced with using the phsynet for segment affintiy | 13:24 |
| sahid | you that means you remove the 1:1 mapping between l2 and l3 | 13:24 |
| sean-k-mooney | which was the architureal mistake that should have been fixed | 13:24 |
| sean-k-mooney | i mena neutron should have an api to allow you to mapp computes hots to network segments | 13:25 |
| sean-k-mooney | and allow you to have multiple subnets per segment | 13:25 |
| sahid | sean-k-mooney: I think we can do that in neutron | 13:25 |
| sahid | currently there is a real desir of having a 1:1 mapping between a vlan and a subnet | 13:27 |
| sahid | you don't want to have two subnet sharing the same vlan | 13:28 |
| sean-k-mooney | how this was ment to scale orgianly is 1 segment per broadcast doamin (TOR switch), each segment should have multipel l3 routed subnets, and each segment was intended to mapped to the specific host that are attach to that broadcast domain | 13:28 |
| sahid | i see | 13:29 |
| sean-k-mooney | if you want a 1:1 mapping between vlan and subnet that requireds 1 network per subnet | 13:29 |
| sahid | no anymore :-) | 13:29 |
| sahid | but yes previously it was like that | 13:29 |
| sean-k-mooney | right but im saying i dont think the feature added in zed is good for scaling | 13:30 |
| sean-k-mooney | or at least it not implemted how i would do that | 13:30 |
| sahid | it is good from an user point of view | 13:30 |
| sean-k-mooney | not if it causes schduign issue by an expoltion of aggreates meaning they cant use it | 13:31 |
| sahid | interesting point :-) perhaps the design used by placement is not so ideal ... ;D | 13:32 |
| sean-k-mooney | or neutron is not useing it properly | 13:33 |
| sean-k-mooney | sahid: im really not inclidned to extend nova ot supprot this neutron feature | 13:33 |
| sean-k-mooney | to me its breakign the desing of how l3 routed networks was intended to work | 13:33 |
| *** hemna- is now known as hemna | 13:36 | |
| sahid | sean-k-mooney: not really sure, the datastructure in neutron was originaly exposed to handle multiple-segments | 13:38 |
| sahid | but anyway, back in the fix proposed, it is not going to change nova behavior for small deployments and helps large deployments | 13:39 |
| sean-k-mooney | neutorn often add extentsion without addign the nova support. | 13:39 |
| sean-k-mooney | for example noava still does nto supprot the multi physnet extention i belive | 13:39 |
| sahid | sean-k-mooney: yes in some situation it always take the first subnet | 13:40 |
| sahid | I this the issue is when you specify a fixed ip when creating the vm | 13:40 |
| sean-k-mooney | sahid: it is a change in behavior as today we only make 1 palcmenet query per schduilng request (ignoriang a deprected config option) | 13:40 |
| sahid | sean-k-mooney: it will alqys be the same one request | 13:41 |
| sahid | the limit of 200 is basically the limit we have observed | 13:42 |
| sahid | (about 220) | 13:42 |
| sean-k-mooney | the fix is raisign an exaction based on a hard coded max query size | 13:42 |
| sean-k-mooney | that is deployment sepciric | 13:42 |
| sean-k-mooney | we have a custoemr with a similar probelem for a diffent reason | 13:42 |
| sean-k-mooney | they had ~600 aggrates per host | 13:42 |
| sean-k-mooney | before they hit the limit in our installer | 13:42 |
| sean-k-mooney | so add ign a hardcoded check like this would 1 break them and 2 break the workaround they want to use | 13:43 |
| sean-k-mooney | which is to increase the query string limit | 13:43 |
| sean-k-mooney | which they tested and works for them | 13:43 |
| sahid | not sure to understand | 13:44 |
| sahid | the fix split the number of uuid passed to the uri | 13:44 |
| sahid | if it is more than 200 | 13:44 |
| sean-k-mooney | which intoduces multipel quereis | 13:44 |
| sahid | only for env with more than 200 | 13:45 |
| sean-k-mooney | gibi: you were looking into this on our side | 13:45 |
| sean-k-mooney | gibi: didnt you mention we doen seem to need to call _get_sharing_providers in teh comptue manager today ? | 13:46 |
| sahid | so for most of the deployment we don't change the behavior | 13:46 |
| sahid | and for the other, we fix their issue | 13:46 |
| sean-k-mooney | sahid: we cant use a constant fo 200 | 13:46 |
| sean-k-mooney | if we were to do this we need to have it be configurable | 13:47 |
| sean-k-mooney | but the reason im pinign gibi into the conversation is i think they determined this was not curestly used in compute manager where its called | 13:47 |
| sean-k-mooney | sahid: https://bugzilla.redhat.com/show_bug.cgi?id=2283075 | 13:48 |
| sean-k-mooney | sahid: we were considering if we coudl remove this funciton entirely | 13:48 |
| sean-k-mooney | the pagingation fo this query is perhaps somethign we coudl do but i think we would need to make the size configurabel if we did | 13:49 |
| sahid | hum... interesting yes | 13:50 |
| gibi | sean-k-mooney: what I remember it is used in the resource tracker to cache the sharing realtionship into the internal provider_tree. | 13:50 |
| sean-k-mooney | right and today im not sure that cache is actully used in the resouce tracker for anything | 13:51 |
| sean-k-mooney | sahid: if the bug you reported is just with the compute manager/resouce tracker and not the sechduler | 13:52 |
| sean-k-mooney | i guess your propsoed patch is not a behaivor chage when scheduling | 13:52 |
| sean-k-mooney | so the bug you reported is proably valid | 13:53 |
| sean-k-mooney | what im not sure would be valid is making muliple request ot the allcoation candiate api in a similar way | 13:53 |
| sean-k-mooney | that is what i tought you were suggesting but looking at your patch its not | 13:54 |
| *** bauzas_ is now known as bauzas | 14:00 | |
| gibi | I belive what is proposed is heavy behavior change. Today the returned RP needs to be in all the aggregates passed, in the new code it only needs to be in a subset of those aggregates. | 14:00 |
| gibi | potentially in the last loop the RP only need to be in the last aggregate | 14:01 |
| gibi | we cannot paginat the query itself | 14:01 |
| sean-k-mooney | what is proposed is not affecting the allocation candiate query | 14:02 |
| sean-k-mooney | at least that is what i understood sahid was asserting | 14:02 |
| gibi | it affects the meaning of _get_sharing_providers | 14:03 |
| gibi | we cannot just do that bindly | 14:03 |
| sean-k-mooney | yes i know | 14:03 |
| sean-k-mooney | but but what im saying is we shoudl not be calling that when generating the allcoation candiate request | 14:03 |
| sean-k-mooney | but your right that we have to be carful which is why i was orginally pushing back on shaids approch. but if it only affecting the resouce-tracker cache that less fo a concern | 14:04 |
| gibi | even if it is only used in the tracker we cannot allow to have wrong data in the tracker, maybe tomorrow a new feature will start using that data. | 14:06 |
| sean-k-mooney | sure | 14:06 |
| sean-k-mooney | but i never said we shoudl | 14:07 |
| gibi | so I'm hard -1 in the current version of the patch | 14:07 |
| gibi | I guess I'm not arguing against you but against the patch itself :) | 14:08 |
| sean-k-mooney | so the current verison of the patch should not chagne the behaivor | 14:08 |
| sean-k-mooney | either we would loop over all the aggreates and return the comined set | 14:08 |
| sean-k-mooney | or raise the same excption we raised previosusly | 14:08 |
| sean-k-mooney | the main issues is that there coudl be a chagne in the assocation during the iterations | 14:09 |
| gibi | no, that is my point, we cannot just combine the results | 14:09 |
| sean-k-mooney | this is the set of sharing resouce providers that share with this host | 14:09 |
| sean-k-mooney | why can we not just combine? | 14:09 |
| gibi | no this is all the RPs that in an aggregate | 14:09 |
| gibi | this is not just a single RP | 14:10 |
| sean-k-mooney | no | 14:10 |
| sean-k-mooney | "/resource_providers?member_of=in:%s&required=%s" % ( | 14:10 |
| sean-k-mooney | aggs, os_traits.MISC_SHARES_VIA_AGGREGATE) | 14:10 |
| sean-k-mooney | this is only the resouce providers in teh aggrate with the os_traits.MISC_SHARES_VIA_AGGREGATE triat | 14:10 |
| sean-k-mooney | so only the sharing providers | 14:10 |
| gibi | but not for a single host | 14:10 |
| gibi | there is no 'this host' filter | 14:11 |
| gibi | in the queru | 14:11 |
| gibi | in the query | 14:11 |
| sean-k-mooney | there does not need to be | 14:11 |
| sean-k-mooney | we are asking for all the sharing provider in the aggrate our host is a member of | 14:11 |
| sean-k-mooney | that what the in: query is doing | 14:11 |
| sean-k-mooney | the agg_uuids var is the list of aggreates the current host is a member of | 14:12 |
| sean-k-mooney | we tyically expect only one RP in an aggrate to have MISC_SHARES_VIA_AGGREGATE althogh that is not strictly required | 14:13 |
| gibi | ahh I misremembered the meaning of 'in:' the query | 14:13 |
| gibi | that is not AND, that is OR | 14:13 |
| sean-k-mooney | well member_of=in:% | 14:13 |
| sean-k-mooney | so member_of is "filter by aggartge" and in is the same meaing as container or in in python | 14:14 |
| sean-k-mooney | i.e. filter by aggrage where the aggreate uuid is in "list of uuids" | 14:14 |
| sean-k-mooney | and we prefilter the aggreate list to only the ones that this host is in | 14:15 |
| sean-k-mooney | im still -1 on sahid patch as is because i woudl like to see a repoducer test and then test coverage for this | 14:16 |
| sean-k-mooney | but i think exteranlly the beahivor is the same but internally there is now pagination | 14:16 |
| sean-k-mooney | i dont think hardcodeing the pagination to 200 uuids however is a good thing | 14:17 |
| sean-k-mooney | sicne that depend on yoru loadbalnacer config | 14:17 |
| sahid | sorry guys I had to switch to a meeting. | 14:21 |
| sahid | fair enough I will add a configurable | 14:21 |
| gibi | yeah I was wrong as I thought we query for PRs that are in _all_ the aggregates but we query for RPs that are in _any_ of the aggregates. I will drop my -1 | 14:24 |
| sean-k-mooney | oh you tought the in was an and not an or just read that | 14:25 |
| sean-k-mooney | ya its an or of all the uuids in the list | 14:25 |
| sean-k-mooney | in is basically AnyOf rather then AllOf | 14:26 |
| gibi | yeah | 14:27 |
| gibi | and as it is an OR I agree that we can do multiple queries and combine the results | 14:27 |
| sean-k-mooney | im kind of ok in the preodics | 14:28 |
| sean-k-mooney | im not sure that is ok for the schduler but the schduler scales the query based on teh guest request | 14:28 |
| sean-k-mooney | not the number of aggrates that exist | 14:28 |
| sean-k-mooney | so i think tis less of a problem there | 14:28 |
| *** haleyb|out is now known as haleyb | 14:51 | |
| opendevreview | Will Szumski proposed openstack/nova stable/2023.2: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921686 | 15:19 |
| opendevreview | Will Szumski proposed openstack/nova stable/2023.1: Use configuration for single enabled mdev type https://review.opendev.org/c/openstack/nova/+/921687 | 15:19 |
| opendevreview | Sahid Orentino Ferdjaoui proposed openstack/nova master: scheduler: fix _get_sharing_providers to support unlimited aggr https://review.opendev.org/c/openstack/nova/+/921665 | 15:50 |
| melwitt | dansmith: here's one of the kernel panics I got recently https://fbf109e10bffd8a657b2-ace28e47b1213bdab56305960a7f8056.ssl.cf5.rackcdn.com/919986/2/check/tempest-integrated-compute/4b51a9c/testr_results.html | 16:07 |
| melwitt | and this https://86989508e8a17de7a050-16e5a6d00ae68a8470fe17c7f38ebfca.ssl.cf2.rackcdn.com/921586/1/check/tempest-integrated-compute-rbac-old-defaults/8980965/testr_results.html | 16:07 |
| dansmith | okay yeah that first one looks like what we had seen before | 16:08 |
| dansmith | and the second | 16:08 |
| melwitt | this one also looks like https://5edd60feb751137d1041-26db3adf591a82ec37c96a7d3180086f.ssl.cf5.rackcdn.com/870932/41/check/nova-next/be498dc/testr_results.html | 16:10 |
| melwitt | this one is different https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/testr_results.html | 16:11 |
| melwitt | (soft lock up ^) | 16:11 |
| melwitt | this one is also different https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d1c/919991/4/check/tempest-integrated-compute-enforce-scope-new-defaults/d1c7ab9/testr_results.html | 16:12 |
| sean-k-mooney | the soft lockup seams ot be related to a memroy allcotion | 16:13 |
| sean-k-mooney | i wonder if that is related to the amount of ram we are usign for the vms but its got 128mb | 16:14 |
| sean-k-mooney | so it shoudl not be a proablem | 16:14 |
| sean-k-mooney | 18.269292] Freeing unused decrypted memory: 2036K | 16:15 |
| sean-k-mooney | [ 18.395056] Freeing unused kernel image (initmem) memory: 3244K | 16:15 |
| sean-k-mooney | [ 18.397470] Write protecting the kernel read-only data: 30720k | 16:15 |
| sean-k-mooney | [ 18.419370] Freeing unused kernel image (text/rodata gap) memory: 2036K | 16:15 |
| sean-k-mooney | [ 18.433426] Freeing unused kernel image (rodata/data gap) memory: 1448K | 16:15 |
| sean-k-mooney | it has already freed the ram disk and early allcoations | 16:15 |
| sean-k-mooney | cirros-0.6.2-x86_64-disk.img so that pretty new | 16:19 |
| sean-k-mooney | or at least its using a ubuntu 22.04 kernel | 16:19 |
| sean-k-mooney | it look like eitehr a real out of memory issue or a kernel bug to me | 16:20 |
| sean-k-mooney | 0.6.2 was released in may so we coudl perhaps ask for a 0 | 16:21 |
| sean-k-mooney | 0.6.3 release with the latest hardwer enabeling kernel package? | 16:21 |
| melwitt | it seems like all of the swift put_object failures I see occur in the nova-lvm job on tests that have nothing to do with volumes. I don't get it Jun 10 04:45:27.576181 np0037692500 devstack@g-api.service[86519]: ERROR glance.common.wsgi Got error from Swift: put_object('glance', 'b1cfd7cf-ff23-4b05-be12-48ce3e14d593-00001', ...) failure and no ability to reset contents for reupload.. | 17:00 |
| sean-k-mooney | hum... we dont need swift in that job | 17:01 |
| sean-k-mooney | the nova-lvm jobs is ment to test nova with the lvm storage backend | 17:01 |
| sean-k-mooney | and nova never talks to swift | 17:01 |
| melwitt | that's where glance is uploading images though | 17:01 |
| sean-k-mooney | right but it could and proably should be using the file backend | 17:02 |
| sean-k-mooney | not the swift backend | 17:02 |
| melwitt | oh, yeah | 17:02 |
| dansmith | why "should" ? | 17:04 |
| sean-k-mooney | as in why it should not be using swift? | 17:07 |
| dansmith | yeah | 17:07 |
| sean-k-mooney | just to keep the job smaller | 17:07 |
| sean-k-mooney | its not really imporant to testing the lvm storag backend | 17:07 |
| dansmith | the default config is to use swift though right? | 17:07 |
| sean-k-mooney | no file | 17:07 |
| sean-k-mooney | although maybe that depend on if swift is deployed | 17:08 |
| dansmith | I think if swift is deployed (which it is in the integrated job, AFAIK) then we'll use it | 17:08 |
| sean-k-mooney | swift is deployed by default in devstack although i always turn it off so im used to glance using file | 17:08 |
| dansmith | I don't see that the LVM job is configuring it specifically.. and point being just that the fewer deltas from the base job the easier it is to reason about failures and why they're different | 17:08 |
| sean-k-mooney | or cinder | 17:09 |
| sean-k-mooney | well i would not expect the base job to use swift for glance | 17:09 |
| dansmith | I would unless we don't deploy swift | 17:09 |
| dansmith | I don't see an "enable_service" for the nova-lvm job that is opting into swift, is my point | 17:10 |
| dansmith | okay but maybe integrated-compute does disable it? | 17:12 |
| sean-k-mooney | i dont know i might have actully bee looking at the multi store job not lvm | 17:12 |
| sean-k-mooney | the multi store job does turn it off http://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_29d/919989/4/check/nova-ceph-multistore/29db296/controller/logs/_.localrc_auto.txt | 17:13 |
| sean-k-mooney | maybe i misread the job name but i dont have a nova-lvm tab open so im kind of confuged why i tought it was nova-lvm | 17:14 |
| dansmith | okay, tbh I thought more of our jobs were running with swift-backed glance since it's the default, but perhaps "most" is not quite right | 17:14 |
| dansmith | https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L151 | 17:15 |
| sean-k-mooney | so i would have personaly tough glance is rarely backed by swift and file or cinder backend woudl be more common | 17:15 |
| dansmith | not me :) | 17:15 |
| dansmith | backing it by file requires shared storage and I think doing so by cinder is rather expensive, IIRC | 17:16 |
| sean-k-mooney | i guess im just used to swift not being deployed in the enviornment i look at | 17:16 |
| sean-k-mooney | actully we were talking about timeout an dthe integreated compute job and its deritives | 17:17 |
| sean-k-mooney | https://github.com/openstack/tempest/blob/master/zuul.d/integrated-gate.yaml#L141 | 17:17 |
| sean-k-mooney | looks like its already reduced to 4 | 17:17 |
| sean-k-mooney | i guess we clampped it to 4 because of failure when we rased the default for the rest to 6 | 17:18 |
| sean-k-mooney | dansmith: looking at a clople of the tiemout they are all on rax but spread across 3 datacenters rax-iad, rax-dfw and rax-ord | 17:22 |
| sean-k-mooney | dansmith: https://tinyurl.com/3bf6nhkh | 17:30 |
| sean-k-mooney | actull i think i need to grop by somethign | 17:31 |
| sean-k-mooney | so that i only get on ehit per job | 17:31 |
| dansmith | yeah, it wouldn't surprise me at all if rax is the oldest/slowest of the providers | 17:32 |
| dansmith | maybe we could group only smaller jobs to run on rax | 17:32 |
| sean-k-mooney | they also run our vms on xen not kvm i belive | 17:32 |
| *** bauzas_ is now known as bauzas | 17:36 | |
| sean-k-mooney | ok this is better https://tinyurl.com/35dfcfcy | 17:38 |
| sean-k-mooney | only 72 hits mostly rax | 17:38 |
| sean-k-mooney | what would be useful is if we could break down the average job runtime by provider | 17:39 |
| *** bauzas_ is now known as bauzas | 17:56 | |
| *** bauzas_ is now known as bauzas | 19:01 | |
| *** bauzas_ is now known as bauzas | 19:46 | |
| *** bauzas_ is now known as bauzas | 19:54 | |
| *** bauzas_ is now known as bauzas | 20:48 | |
| *** bauzas_ is now known as bauzas | 21:54 | |
| *** bauzas_ is now known as bauzas | 22:06 | |
| *** bauzas_ is now known as bauzas | 22:18 | |
| *** bauzas_ is now known as bauzas | 23:18 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!