Friday, 2022-01-28

fricklerslaweq: I did a bit of local testing myself. IMO the amount of changes that would be needed in devstack show what changes operators would have to go through and to me this shows what a bad idea that PTG decision might be07:10
opendevreviewJakob Meng proposed openstack/devstack master: (WIP) Hack to show bug on clean CentOS 8 Stream  https://review.opendev.org/c/openstack/devstack/+/82681307:41
slaweqfrickler: I don't think it's that bad. I have locally devstack deployed with those new settings already. Main changes which I had to do is basically "revert" of changes made earlier in patch https://review.opendev.org/c/openstack/devstack/+/797450 as now (again) project admin needs to be used instead of system admin for most of those things07:51
slaweqthere are some other issues there, but not that huge07:51
slaweqToday I should send few patches to try it together in upstream ci07:51
*** bhagyashris_ is now known as bhagyashris08:01
opendevreviewMartin Kopec proposed openstack/tempest master: Fix test_rebuild_server test by waiting for floating ip  https://review.opendev.org/c/openstack/tempest/+/81408508:02
*** jpena|off is now known as jpena08:13
opendevreviewSlawek Kaplonski proposed openstack/devstack master: Fix deployment of Neutron with enforced scopes  https://review.opendev.org/c/openstack/devstack/+/82685108:53
opendevreviewSlawek Kaplonski proposed openstack/devstack master: Revert "Disable enforcing scopes in Neutron temporary"  https://review.opendev.org/c/openstack/devstack/+/82685208:53
*** bhagyashris_ is now known as bhagyashris08:53
slaweqfrickler: ^^ lets wait for ci results there but I hope that will be enough to make it working again with scope enforcement in Neutron08:54
fricklerslaweq: ah, you kind of cheated and added a new cloud definition. that's exactly the point, doing that as a provider for thousands of customers is really a big thing. but for fixing devstack I think it is good enough for now09:52
fricklerkopecmartin: you were asking about newcomer tasks. one might to clean up the use of "openstack ... | grep | get_field " and replace with "openstack -f value -c COL", like seen e.g. here https://review.opendev.org/c/openstack/devstack/+/826851/1/lib/neutron_plugins/services/l310:21
kopecmartinfrickler: great, thanks, i'm gonna make a note of it in the priority etherpad10:23
kopecmartingmann: when you have a sec, can you please check this bug fix? https://review.opendev.org/c/openstack/tempest/+/81408513:01
opendevreviewMerged openstack/tempest master: Revert "Check VM's console log before trying to SSH to it."  https://review.opendev.org/c/openstack/tempest/+/81763413:20
opendevreviewJames Parker proposed openstack/whitebox-tempest-plugin master: Test resize with mem_page_size in flavor  https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/82477214:14
opendevreviewDan Smith proposed openstack/grenade master: Add grenade-skip-level job  https://review.opendev.org/c/openstack/grenade/+/82610114:37
vkmco/15:00
enriquetasoo/15:00
enriquetaso\o15:00
vkmcnot sure how many people will join us today 15:00
tbarron\o/15:00
carlosso/15:00
vkmcwe can try to start a meeting with the bot15:00
vkmc!startmeeting devstack-cephadm15:00
opendevmeetvkmc: Error: "startmeeting" is not a valid command.15:00
enriquetasooops15:00
vkmc!start devstack-cephadm15:00
opendevmeetvkmc: Error: "start" is not a valid command.15:00
vkmcoh noes15:00
vkmc#startmeeting devstack-cephadm15:01
opendevmeetMeeting started Fri Jan 28 15:01:23 2022 UTC and is due to finish in 60 minutes.  The chair is vkmc. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
opendevmeetThe meeting name has been set to 'devstack_cephadm'15:01
fmounto/15:01
toskyo/15:01
vkmco/ 15:02
vkmclet's give people a couple of minutes to join 15:02
enriquetaso\o15:03
rosmaitao/15:03
carlosso/15:04
vkmc#link https://etherpad.opendev.org/p/devstack-plugin-ceph15:04
vkmcjust set an etherpad to take notes, if needed 15:04
vkmclet's get started 15:05
vkmc#topic motivations15:05
vkmcso, as mentioned in the mailing list thread 15:05
*** fmount is now known as fpantano15:06
vkmcwe (as in manila team) are aiming to implement a plugin for devstack to deploy a ceph cluster using the cephadm tool 15:07
vkmcthis tool is developed and maintained by the ceph community 15:07
vkmcand it allows users to get specific ceph versions very easily and enforces good practices for ceph clusters15:08
fpantanovkmc: yeah and it's a good idea as ceph-ansible is not used anymore (cephadm is the official deploy tool)15:08
sean-k-mooneyi would suggest privoting the curernt plugin instead of developing a new one15:08
vkmcagree :) 15:08
fpantanoand there's also the orchestrator managing the ceph containers, so it's not just a matter of deploy15:09
sean-k-mooneyso tha twe only have to test with one way of deploying ceph in the gate15:09
toskybecause everyone else needs it, as this is the new way for deploying ceph15:09
toskyand we don't need to change all the jobs 15:09
vkmcall good reasons15:09
vkmcseems we don't need to get more into detail on why we want to do this 15:09
fpantano+115:09
vkmcso, I guess, we are already covering this 15:09
vkmcbut15:09
vkmc#topic design approach 15:10
vkmcas sean-k-mooney mentions, instead of implementing two different plugins15:10
sean-k-mooneyi think for most porject how ceph is deployed is less relevent then ceph is deployed. we do care about version to a degree but not nessialy is it in a contiaenr or disto pacakage or upstream ceph package15:10
vkmcwe should work on devstack-ceph-plugin as we know it, allowing users to toggle between the two15:10
sean-k-mooney+115:10
vkmcand, as tosky mentions, it will be easier as well from a ci standpoint 15:11
vkmcso we don't need to change the jobs definitions 15:11
toskyso, do we need the switch?15:11
sean-k-mooneyi think until its stable yes15:11
vkmcmaybe, with time, we will need to come with a strategy to turn on the option to deploy with cephadm, start testing with that 15:12
sean-k-mooneyi would sugess that for this cycle we likely dont want to enable cephadm by default untill we are happy it work for the majorty of the jobs15:12
toskyright15:12
vkmcand slowly deprecate the current way of deploying things15:12
fpantanoagree15:12
sean-k-mooneyya so similar to the ovn swtich15:12
vkmcwhich leads me to15:12
toskyfpantano: so just to clarify: is ceph-ansible still useful for pacific? I'd say it is, as we use it15:12
vkmc#topic initial patch15:12
tbarronand devstack-plugin-ceph is branched now so old stable branches can continue as they do now15:12
fpantanotosky: afaik it's useful only for upgrades purposes15:13
vkmc#link https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/82648415:13
toskyis there a specific mapping of what is compatible with what from the ceph, ceph-ansible and cephadm point of you?15:13
tbarrontosky: we aren't really using ceph-ansible either with the old branches15:13
tbarronthe approach was even older15:13
fpantanotbarron: ah, so ceph-ansible here is out of the picture already15:13
toskyoh15:13
tbarroni think so, someone correct me if I'm wrong15:13
toskytbarron: by "old branches" do you also include the current master, right? Now that I think about it, the code installs the packages and does some magic15:14
tbarronleseb authored a lot of early ceph-ansible and early devstack-plugin-ceph but the latter was not really adapted to use ceph-ansible15:14
tbarroneven on master15:14
vkmcyeah, we don't use ceph-ansible in the plugin15:14
vkmcceph-ansible is being used for other deployment tools, such as a tripleo15:15
vkmcbut in the devstack-plugin-ceph we have been installing packages and configuring things manually 15:15
toskyok, so is it correct to say that the current method used by devstack-plugin-ceph just works because it happened not to use ceph-ansible, but it's an handcrafted set of basic steps?15:15
sean-k-mooneydoes cephadm still have a dep on docker/podman on the host. i have not used it in a while15:15
vkmcsean-k-mooney, it does15:16
tosky(sorry for the questions, just to have the full current status on the logs)15:16
tbarrontosky: +1 (but check me folks)15:16
fpantanosean-k-mooney: yes, it has podman dep15:16
sean-k-mooneyack15:16
vkmctosky++15:16
sean-k-mooneywe shoudl have that avaible in the relevent distros i guess15:16
sean-k-mooneybut devstack wont set it up by default15:16
vkmcright 15:16
sean-k-mooneyso the plugin will have to check and enable it although will cephadm do that for us15:16
sean-k-mooneythe less we have to maintain the better15:17
vkmcsean-k-mooney, https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484/8/devstack/lib/cephadm#10015:17
vkmcagree15:17
toskyit should really be "install and run", at least it was on Debian when I switched it - you may just need to enable some parameters depending on the kernel version used, but that could be figured out I guess15:17
sean-k-mooneyack15:17
sean-k-mooneyvkmc: im not sure how may other have this useacse but will you support multi node ceph deployment as part of this work15:18
sean-k-mooneywe at least will need to copy the keys/ceph config in the zuul jobs to multiple nodes15:18
vkmcsean-k-mooney, if we are currently testing that in ci, we will need to see how to do such a deployment with cephadm in our ci, yes15:19
sean-k-mooneybut do you plan to supprot adding addtion nodes with osds via running devstack and the plugin on multiple nodes15:19
fpantanosean-k-mooney: ack, the current review is standalone, but in theory, having 1 node cluster and ssh access to other nodes means you can enroll them as part of the cluster 15:19
tbarronsean-k-mooney: cephadm can do scale-out well15:19
sean-k-mooneyi think nova really only uses one ceph node(the contoler) and copies the keys/config and install the ceph client on the compute15:19
tbarronsean-k-mooney: oh, you are talking multiple ceph clients rather than multi-node ceph cluster, right?15:20
sean-k-mooneywell potentally both15:20
sean-k-mooneythe former is used today15:20
tbarronI don't see any issue either way.15:21
sean-k-mooneythe later might be nice for some glance/cinder/genade testing15:21
sean-k-mooneysorry that is a bit off topic please continue15:21
fpantanotbarron: me neither, let's have a plan (and maybe multiple reviews) to reach that status15:21
tbarronfpantano: +115:21
vkmcfpantano++15:22
toskymultiple ceph clients is probably the first step as it may be used in multinode jobs15:22
sean-k-mooneys /may/will :)15:22
fpantano+115:22
sean-k-mooneyalthough i think we do that via roles in zull today15:22
sean-k-mooneyso it might not be fully in scope fo the plug but rather the job defitions/roles15:23
tbarronand probably that part won't change, but as fpantano says, can look in future reviews15:23
sean-k-mooneyi belive how it work today it the first node complete then we just copy the data to the others15:23
vkmcso I take this is an scenario needed by nova, right?15:23
sean-k-mooneythen stack on the remainder so that should be pretty independet15:23
sean-k-mooneythe ceph multinode job does live/colde migration testing15:24
sean-k-mooneyso both computes need to be able to connect to the singel ceph instance on the contoler15:24
sean-k-mooneythat could be step 2 howewver since we can use the legacy install mode for multinode for now15:25
fpantanoyeah, I'd say let's have a single node working, then multinode is the goal and the CI can be tested against this scenario15:25
vkmcfpantano++15:25
vkmcok, so... in theory, cephadm is a tool that can be used to deploy a single node cluster working, and then we can scale as needed 15:27
vkmcand test different scenarios15:27
vkmcdepending on what we need 15:27
vkmcI'm not familiar with all the different scenarios we need to test, I guess this varies between services 15:28
vkmcso that leads me to 15:28
vkmcshall we establish some SMEs for each of the services to work on this? 15:28
sean-k-mooneywell one way to test is a DNM patch to the differnet service that flips the flag15:29
sean-k-mooneyif the jobs explode then we know there is work to do15:29
sean-k-mooneyzuul speculitive execution is nice that way15:29
tbarrons/if/when/15:29
sean-k-mooney:)15:29
vkmcall right :) 15:30
toskythe jobs currently defind in devstack-plugin-ceph already sets up various services, I'd say having a cephadm version of them passing is step 115:30
tbarron+115:31
fpantano++ /me not familiar with the current jobs but I agree, we can enable cephadm first and see how it goes and what we missed on the first place ^15:31
sean-k-mooneynova has 2 cpeh jobs but both inherit form  devstack-plugin-ceph-tempest-py3 and devstack-plugin-ceph-multinode-tempest-py315:32
toskythe WIP patch can switch all of them, and when it's stable we can simply duplicate them15:32
toskysean-k-mooney: so if you depend on the DNM patch which use the new defaults you should see the results, and same for the jobs in cinder which inherits from those devstack-plugin-ceph jobs15:33
sean-k-mooneyyes15:33
sean-k-mooneywe can have one patch to nova that does that and we can recheck it and tweak until they are happy15:33
tbarronso I see nova, cinder, manila folks here, but not the other consumer, glance.15:34
* tbarron exempts rosmaita since he's cinder ptl15:34
sean-k-mooneywell also maybe ironic?15:34
tbarronwould be if I forgot them15:35
sean-k-mooneydo they supprot boot form ceph via the iscis gateway15:35
tbarronsean-k-mooney: i dunno.  they talk about it in ptg.15:35
sean-k-mooneyi know they have cidner boot supprot with iscsi but no idea if they test that with ceph and the iscis gateway15:35
sean-k-mooneyproably not15:35
sean-k-mooneyat east with devstack15:35
sean-k-mooneynova's  nova-ceph-multistore job15:36
sean-k-mooneywill test glance15:36
* tbarron wonders if dansmith can be enticed to help if glance jobs blow up15:36
sean-k-mooneyso maybe that is ok for now15:36
toskysean-k-mooney: a quick search with codesearch.openstack.org doesn't show any ironic repository15:36
sean-k-mooneyack15:36
sean-k-mooneyhttps://github.com/openstack/nova/blob/master/.zuul.yaml#L46715:37
* rosmaita is not paying any attention15:37
tbarronhe's bragging again15:37
vkmcxD15:37
vkmcok so15:38
sean-k-mooneyso for those not familar with that job we confirue glacne with both the file and ceph backend and test direct usage of images backed by rbd ensuring we dont download them15:38
sean-k-mooneyso that will fail if glance does not work 15:38
sean-k-mooneywith cephadm15:39
fpantano+1 thanks for clarifying ^15:39
opendevreviewJames Parker proposed openstack/whitebox-tempest-plugin master: [WIP] Add multiple hugepage size job  https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/82501115:41
vkmcwhich jobs do we want to see passing before getting the cephadm script merged? as a initial step I mean 15:42
toskymy suggestion are the jobs in the repository itself15:43
vkmcvery well 15:43
vkmcso let's start from there 15:43
toskyin fact there are:15:43
tosky(moment)15:44
dansmithtbarron: sorry, just saw this, reading back15:44
tbarrondansmith: ty15:44
tosky- devstack-plugin-ceph-tempest-py3 is the base baseline15:44
tosky- devstack-plugin-ceph-multinode-tempest-py3 its multinode version15:44
tosky - devstack-plugin-ceph-cephfs-native and devstack-plugin-ceph-cephfs-nfs are non voting, so up to the manila team whether they should be considered as blocking15:45
tosky- there is also devstack-plugin-ceph-tempest-fedora-latest which is voting15:45
sean-k-mooneyyep devstack-plugin-ceph-tempest-py3 is the MVP i think but that is really up to the plugin core team so just my 2 cents15:45
tosky- non voting: devstack-plugin-ceph-master-tempest (but that should be probably not too different from the basic tempest job at this point)15:46
opendevreviewJames Parker proposed openstack/whitebox-tempest-plugin master: [WIP] Add multiple hugepage size job  https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/82501115:46
tosky- experimental: devstack-plugin-ceph-compute-local-ephemeral (nova? What is the status of that?)15:46
dansmithtbarron: oh, I don't know much about ceph really, I just worked on that job because we really needed test coverage there15:46
toskythat's the current status of https://opendev.org/openstack/devstack-plugin-ceph/src/branch/master/.zuul.yaml as far as I can see15:46
dansmithtbarron: so yeah I can help if that job blows up in general, but I'm still not quite sure what is being discussed.. some change coming to ceph?15:47
abhishekkdansmith, tbarron I will have a look if there is something wrong 15:47
vkmcabhishekk++15:47
toskydansmith: tl;dr right now devtack-plugin-ceph deploys ceph manually; it has never used the more blessed way (ceph-ansible), and there is a new official way (cephadm)15:47
tbarronabhishekk: dansmith excellent.  The idea is to change devstack-plugin-ceph to use cephadm to deploy the target ceph cluster15:48
toskydansmith: we are discussing implementing the support for cephadm for the deployment15:48
dansmithah15:48
tbarronin theory client side (e.g. glance) everything would "just work"15:48
tbarronheh15:48
dansmithwell, I don't think that job does anything special during setup, it just wires nova and glance together to do the good stuff15:48
fpantanodansmith: and a first review has been submitted: https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/82648415:49
dansmithso if you don't a-break-a the setup, I doubt it will a-break-a the testing :)15:49
abhishekk++15:49
dansmith"special during *ceph* setup I meant"15:49
tbarronyup, but it will be good to know whom to consult when the tests break anyways and it's not clear why15:51
abhishekktbarron, you can ping me (I will be around 1800 UTC)15:52
dansmithI can deflect with the best of them, so yeah.. count me in :)15:52
vkmcdansmith++ 15:52
vkmcgreat :) 15:52
vkmcso abhishekk and dansmith for glance, tosky, rosmaita, eharney and enriquetaso for cinder, sean-k-mooney for nova, tbarron, gouthamr and vkmc for manila15:53
sean-k-mooneyi may or may not join this meeting every week but feel fee to ping me if needed15:54
vkmconce we toggle jobs and if we see issues on specific jobs15:54
toskyyep15:54
*** ykarel_ is now known as ykarel15:54
vkmcsean-k-mooney, good thing you mention this, we won't probably meet every week... maybe our next sync would be in one month from now, depending on how fast we get initial things merged 15:55
vkmcI'll continue to communicate on the mailing list 15:55
toskyand once the first results are promising, we can trigger a few other component-specific jobs in the new mode15:55
vkmctosky++ sounds good 15:55
fpantano+1 we have a plan :D15:55
vkmc+1 we do 15:55
toskyvkmc: technically we want also to hear from QA people, as they are also core on that repository iirc15:56
vkmcsmall bird told me carloss is joining us in the manila side as well :D15:56
toskykopecmartin, gmann  ^^15:56
carloss:D15:56
vkmccarloss++ 15:56
vkmcok, 3 more minutes til top of the hour, so let's wrap here15:57
toskyand also because the change impacts jobs which votes on tempest.git itself15:57
vkmcnext action item is to get review for that initial patch and start enabling it for the different jobs we have in that repo 15:57
vkmcand we can sync up again in a few weeks15:57
toskyenable for testing, and when they are stable, start flipping them in a stable way?15:58
vkmcanything else? 15:58
vkmctosky, yes15:58
toskyperfect, thanks15:59
vkmcgreat!15:59
vkmc#endmeeting15:59
opendevmeetMeeting ended Fri Jan 28 15:59:42 2022 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:59
opendevmeetMinutes:        https://meetings.opendev.org/meetings/devstack_cephadm/2022/devstack_cephadm.2022-01-28-15.01.html15:59
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/devstack_cephadm/2022/devstack_cephadm.2022-01-28-15.01.txt15:59
opendevmeetLog:            https://meetings.opendev.org/meetings/devstack_cephadm/2022/devstack_cephadm.2022-01-28-15.01.log.html15:59
vkmcthanks everyone for joining and for your feedback! 15:59
fpantanovkmc++ thanks for starting this!15:59
toskythank you!16:00
enriquetasothanks16:00
enriquetaso!!16:00
opendevmeetenriquetaso: Error: "!" is not a valid command.16:00
fpantanothanks everyone16:00
enriquetasook16:00
opendevreviewBenny Kopilov proposed openstack/tempest master: Update volume schema for microversion  https://review.opendev.org/c/openstack/tempest/+/82636416:05
gmannkopecmartin: +A16:33
gmanntosky: vkmc have not read the log fully but i think agreement is to o in existing plugin. +1 from me. feel free to ping me for help/review on the job modification 16:39
toskygmann: yep, existing repo, keep it working; thanks!16:44
sean-k-mooneyfrickler: do you know if there is a cirros uefi boot capable image 16:50
fricklersean-k-mooney: the readme says it should be possible, but I don't know how to do it. if you need further help, maybe ask directly in #cirros on liberachat https://github.com/cirros-dev/cirros/blob/master/README.md17:00
*** jpena is now known as jpena|off17:06
sean-k-mooneyfrickler: no worries ill read up on it17:15
sean-k-mooneyfrickler: all that is requried is for the image to have the efi partion 17:16
sean-k-mooneythen in glace we set an image property17:16
abhishekkgmann, any difference in 'devstack_localrc' and 'devstack_local_conf' or they both writes to local.conf file?17:19
sean-k-mooneyyes they are different17:22
sean-k-mooneydevstack_localrc does nto override local.conf17:22
sean-k-mooneyabhishekk: the local.conf support extra seciontion that localrc does not 17:23
toskyhttps://opendev.org/openstack/devstack/raw/branch/master/roles/write-devstack-local-conf/README.rst17:24
sean-k-mooneythe devstack_localrc mapps to the [[local|localrc]] section of the local.conf17:24
abhishekksean-k-mooney, ack, so I can move the contents under 'devstack_localrc' to 'devstack_local_conf' and remove the former17:24
sean-k-mooney yes although you can use both at the same time17:24
abhishekksean-k-mooney, tosky ack, thank you17:25
sean-k-mooneygenerhttps://github.com/openstack/nova/blob/master/.zuul.yaml#L174-L19317:26
sean-k-mooney* generally https://github.com/openstack/nova/blob/master/.zuul.yaml#L174-L193 we use local.conf for the config overrides17:26
sean-k-mooneyand localrc for the devstack macro constants17:26
sean-k-mooneyboth can be done in local_conf but we normally partion it this way17:27
abhishekkyeah, same thing we do in glance17:27
abhishekkjust wanted to check the difference and now I got it :D 17:27
gmannabhishekk: yeah devstack_local_conf is one you can move/use17:32
abhishekkgmann, ack, thanks 17:33
opendevreviewMerged openstack/devstack master: Use distro pip on Ubuntu  https://review.opendev.org/c/openstack/devstack/+/82592018:49
opendevreviewmelanie witt proposed openstack/devstack master: Configure nova unified limits quotas  https://review.opendev.org/c/openstack/devstack/+/78996220:00
opendevreviewmelanie witt proposed openstack/tempest master: Add configuration for compute unified limits feature  https://review.opendev.org/c/openstack/tempest/+/79018620:01
opendevreviewmelanie witt proposed openstack/tempest master: Tests for nova unified quotas  https://review.opendev.org/c/openstack/tempest/+/80431120:09
opendevreviewMerged openstack/tempest master: Fix test_rebuild_server test by waiting for floating ip  https://review.opendev.org/c/openstack/tempest/+/81408520:23

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!