Thursday, 2022-04-21

*** ysandeep|out is now known as ysandeep03:09
*** ysandeep is now known as ysandeep|afk03:58
*** ysandeep|afk is now known as ysandeep04:38
opendevreviewJonathan Rosser proposed openstack/ansible-role-python_venv_build master: Improve python version detection when symlinking libraries  https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/83882606:54
opendevreviewJonathan Rosser proposed openstack/ansible-role-uwsgi master: Refactor installation of libpython for debian/ubuntu  https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/83883007:05
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Do not install UCA repository for ubuntu 22.04  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/83883207:11
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Unify debian and ubuntu cache prep scripts  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83883607:46
*** ysandeep is now known as ysandeep|lunch07:52
*** ysandeep|lunch is now known as ysandeep08:36
noonedeadpunkandrewbonney: mgariepy can you kindly review bumps to unblock gates https://review.opendev.org/q/topic:bump_osa+status:open - in order for X to pass, W should be merged first.09:21
jrossernoonedeadpunk: 22.04 works pretty well, it is the usual things like no upstream packages for mariadb, and there is some python brokenness in horizon which I didn't yet look at09:38
jrosserwhat is nice is that the few changes that i have made seem to be making things more general to also work with 22.04 rather than adding specific vars or settings only for 22.0409:38
noonedeadpunkthankfully ubuntu doesn't break everything around with each release (unlike redhat)09:46
* noonedeadpunk can't wait for 22.04.109:46
noonedeadpunkbtw regarding nobody cares about centos 8 distro path - seems it's fixed https://zuul.opendev.org/t/openstack/build/e227187b7d1546148d8b0c97a07d5a7d09:47
noonedeadpunkuh, what awful wording I used but I hope you got idea :)09:48
noonedeadpunkwill try to play with centos 9.... or with Octavia.... Not sure what is less pleasent task to do :p09:49
*** ysandeep is now known as ysandeep|afk10:50
jrosserdamiandabrowski[m]: i think you are right with your comment on https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/83865711:02
jrosserwe are OK for focal, as the version of ceph there is 15.x11:02
jrosserbut unfortunately i just start test 22.04 and there i find ceph 17.x already in the repo11:03
jrossernoonedeadpunk: ^ more trouble with ceph pins :(11:03
jrosserperhaps we have to be more forceful about it, set priority -1 on pacakges: src:ceph from both UCA and the ubuntu repo11:04
jrosserso it will fail hard unless any part of ceph comes from ceph.com11:04
jrosserand also configure no-recommends somewhere in openstack-hosts11:06
noonedeadpunkI think I missed the output of issue when ceph_client tries to downgrade librbd... Does it lead to some package removal?11:09
noonedeadpunkAs at least for focal, in package requirements it says smth like librbd1>=11.smth11:10
damiandabrowski[m]noonedeadpunk: https://paste.openstack.org/show/bRMsII6P2s1YyjugFuWk/11:11
noonedeadpunkso technically, if ceph.com repo has prio >1000, it should jsut downgrade it...11:11
damiandabrowski[m]something like this11:11
damiandabrowski[m]another thing is to allow downgrades but it sounds nasty11:11
noonedeadpunkmaybe we should just allow downgrades for ceph?:)11:11
damiandabrowski[m]another option*11:11
noonedeadpunkdamiandabrowski[m]: and what prio you have for repo in /etc/apt/preferences.d?11:12
damiandabrowski[m]i no longer have this test environment, but it was before jrosser's patch 11:12
noonedeadpunkhis patch doesn't change that...11:13
noonedeadpunkshould be 100111:13
damiandabrowski[m]ouh, but still - i will have to build AIO from scratch to see this error again11:14
noonedeadpunkok-ok11:15
jrosserunless we allow it, downgrade is an error11:16
jrosserand it would also uninstall ${who-knows-what} as dependancies11:16
noonedeadpunkmaybe indeed we can jsut add ` allow_downgrade: true` here https://opendev.org/openstack/openstack-ansible-ceph_client/src/branch/master/tasks/ceph_install.yml ?11:16
jrosserwhich wouldnt then get re-installed11:16
jrosserdowngrade is bad :/11:16
damiandabrowski[m]btw. i still stick to the idea to just add ceph repo on all baremetal hosts with high prio :D 11:17
jrosserbecause a previous play could have installed ABC which requires librbd11:17
jrosserdowngrading librbd will uninstall ABC?11:17
noonedeadpunkno, unless ABC requires librbd version higher then would be installed after downgrade afaik11:17
jrosserthis needs to be tested11:18
noonedeadpunkdoes ceph repos already have smth for 22.04?11:18
jrosserbecasue it is kind of the horrible situation with OFED all over again11:18
noonedeadpunkAs I heard stories about redhat dropping ubuntu packaging for quite a while now11:19
noonedeadpunkoh, well, OFED lol11:19
noonedeadpunkyes, that would be interesting11:20
jrosserceph.com only goes up to focal11:20
jrosserbut tbh 22.04 is a bit of an outlier here as it is soooo early11:20
jrosserbut still they ship 17.x in the distro repo which is going to cause another different problem11:21
noonedeadpunkjust in case there's nothing for 22.04 present in ceph repos11:21
noonedeadpunkand I kind of unsure if they will add 16.x for it11:22
noonedeadpunkat all11:22
* damiandabrowski[m] need to leave for some time, will try to follow the conversation on mobile11:22
noonedeadpunkif we forget about 22.04 - https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657 sounds like proper solution11:22
noonedeadpunkbut it might happen that with 22.04 we will have to bump ceph release to 17.x for everything11:23
damiandabrowski[m]noonedeadpunk: i think it depends on `ceph_stable_release` :D 11:23
noonedeadpunkit does11:24
noonedeadpunkbut it will also affect ceph-ansible which might not be ready for quincy11:24
jrosserit does feel like we might have experimental support for 22.04 in Y but certainly not ceph11:25
noonedeadpunkyup...11:25
jrosserso making a plan for ceph 17.x transition in Z would be helpful11:25
jrosserthen we can tidy up the mess11:25
noonedeadpunkand on top of that to get mariadb packages we would likely need to wait for their new release11:26
noonedeadpunkas they don't build packages for older ones11:26
jrosseryes - we can make a hack like for centos-9 to use the distro package in the meantime11:26
jrosserbut another reason for only experimental support11:26
noonedeadpunkexactly...11:26
noonedeadpunkThough I'm quite afraid about libvirt version that focal would have 11:27
noonedeadpunkwhich might be already not supported in Z11:27
noonedeadpunkbut anyway11:27
noonedeadpunkwe can backprot things if needed11:27
damiandabrowski[m]when i mentioned `ceph_stable_release` i was thinking about the situation, when someone want's to install octopus on 20.0411:28
jrosseri think in that case we would say "external ceph cluster"11:28
noonedeadpunkI guess what Damian means is that ceph_client would try to downgrade things as well11:29
jrosseryes it would11:29
noonedeadpunktbh I like disabling installation of recommended packages more....11:30
jrosserwe should probably do this anyway11:30
damiandabrowski[m]i know i'm pain in the ass right now, but what's the disadvantage of setting ceph repo with high prio on all baremetal nodes? we'll have it unnecessarily defined in a few places, that's right11:30
damiandabrowski[m]but in overall, it should make our lives much easier :D 11:31
damiandabrowski[m]and for ex. we still be able to support ubuntu 20.04 + octopus scenario11:31
noonedeadpunkThere's no issue with that approach, except there should be no librbd installed when ceph is not used at all:)11:31
damiandabrowski[m]we will still be able*11:31
noonedeadpunkIt's just with what we fight - with matter or with result11:33
damiandabrowski[m]you may be right..11:33
noonedeadpunkenabling ceph repo early is fixing result of the thing that likely should not happen....11:33
noonedeadpunkit's easy fix indeed though11:34
opendevreviewMerged openstack/openstack-ansible stable/victoria: Bump SHAa for Victoria  https://review.opendev.org/c/openstack/openstack-ansible/+/83875411:39
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Add default package manager config  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/83885511:55
opendevreviewMerged openstack/openstack-ansible stable/wallaby: Bump SHAs for Wallaby  https://review.opendev.org/c/openstack/openstack-ansible/+/83875812:10
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: DNM  https://review.opendev.org/c/openstack/openstack-ansible/+/83885812:16
damiandabrowski[m]hmm, what does DNS mean? do not merge?12:18
damiandabrowski[m]what's the difference between dnm and wip?12:18
damiandabrowski[m]DNM*12:18
mgariepydo not merge yes12:18
mgariepyusually dnm patch can be only to tests something in the gate, wip means it's work in progress and not finished yet12:19
damiandabrowski[m]ahh, thanks!12:20
noonedeadpunkor Deep And Meaningfull :D12:20
mgariepyho then i was wrong all those years 12:21
noonedeadpunkor Does not Matter 12:21
noonedeadpunkBut all represent kind of DO NOT MERGE anyway lol12:21
mgariepylol12:22
*** ysandeep|afk is now known as ysandeep12:22
damiandabrowski[m]:D12:27
*** ysandeep is now known as ysandeep|afk12:43
opendevreviewJonathan Rosser proposed openstack/ansible-role-uwsgi master: Refactor installation of libpython for debian/ubuntu  https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/83883012:54
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Prevent ceph packages installing from ubuntu-cloud-archive  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/83865712:54
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Unify debian and ubuntu cache prep scripts  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83883612:55
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Unify debian and ubuntu cache prep scripts  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83883612:55
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Do not install UCA repository for ubuntu 22.04  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/83883212:56
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-openstack_hosts master: Do not install recomended packages with apt  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/83889013:13
*** ysandeep|afk is now known as ysandeep13:24
*** ysandeep is now known as ysandeep|afk13:41
NeilHanlonlol noonedeadpunk, i love all those definitions :) 13:50
noonedeadpunkjrosser: btw are we sure that what we do regarding ceph related to missing ceph-volume?13:55
jrosseractually no13:55
jrosseri was just assuming that was some side effect of accidentally installing 17.x13:55
jrosserbut i have not checked that13:55
noonedeadpunkok, I just realized what the issue is:)13:55
mgariepythe ceph-volume is onluy in a sperate pkg13:56
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Add default package manager config  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/83885513:57
noonedeadpunkyeah, we still need to lower prio for some reason13:57
noonedeadpunkor dunno... How that happens actually... https://zuul.opendev.org/t/openstack/build/009a78dbd38b4194a048c51e7f8cdce1/log/logs/host/apt/history.log.txt13:59
noonedeadpunk(l56)13:59
noonedeadpunkhm... https://zuul.opendev.org/t/openstack/build/009a78dbd38b4194a048c51e7f8cdce1/log/logs/etc/host/apt/preferences.d/ceph_community_pin.pref.txt14:00
noonedeadpunklike either our pin does not work for https://zuul.opendev.org/t/openstack/build/009a78dbd38b4194a048c51e7f8cdce1/log/logs/etc/host/apt/sources.list.d/mirror_mtl01_iweb_opendev_org_ceph_deb_octopus.list.txt14:01
noonedeadpunkor we add wrong repo....14:01
jrosseroh well14:01
noonedeadpunkrelease file looks good https://mirror.mtl01.iweb.opendev.org/ceph-deb-octopus/lists/debian-ceph-octopus_focal_Release14:02
NeilHanlonoh good it's not a rocky issue this time :P 14:02
jrosserbut this wont work in CI https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/838657/6/defaults/main.yml14:02
noonedeadpunkyeah, once upon a time ubuntu brings interesting stuff as well lol14:02
mgariepyisnt recommends one that needs to be turned off?14:03
noonedeadpunkWhy not? we use upstream uca - not infra one from what I see in https://zuul.opendev.org/t/openstack/build/009a78dbd38b4194a048c51e7f8cdce1/log/logs/etc/host/apt/sources.list.d/uca.list.txt14:04
noonedeadpunkmgariepy: it's there? https://zuul.opendev.org/t/openstack/build/009a78dbd38b4194a048c51e7f8cdce1/log/logs/etc/host/apt/apt.conf.d/99openstack-ansible.txt14:05
noonedeadpunkalso I don't see this time that lxc pulls in librbd114:06
noonedeadpunkbut we still get ceph 1714:06
mgariepyhttps://paste.openstack.org/show/bq0yJcOaQddlr6vUxrPQ/14:06
jrossernot suggests14:06
jrosserrecommends14:06
noonedeadpunkpfffff14:06
mgariepyno-install-recommends 14:06
noonedeadpunkdamn it14:07
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Add default package manager config  https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/83885514:08
noonedeadpunkok, that explains14:08
noonedeadpunkthanks)14:08
NeilHanlonjrosser: do you need/want me to poke at hacking to make rocky work with respect to that centos-release- stuff? 14:09
jrosserNeilHanlon: i put a hack in here https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/837582/20/roles/glusterfs/tasks/main.yml#4414:10
NeilHanlon👍 cool. hopefully will have those fixed upstream soon14:11
NeilHanlonputting it in my notes to update that after14:11
noonedeadpunkwhile reading octavia certs code my eyes are bleeding....14:26
mgariepyis it that bad ?14:27
noonedeadpunkwell.... these 2 facts are exact same things in the end https://opendev.org/openstack/openstack-ansible-os_octavia/src/branch/master/tasks/octavia_certs.yml#L101-L10214:27
noonedeadpunkI had to make some debug to realize that: "(hostvars[octavia_cert_setup_host]['octavia_ca_certificate_fact'] == hostvars[octavia_cert_setup_host]['octavia_server_ca_fact'])": true14:28
mgariepyshouldn't this use the pki stuff ?14:29
noonedeadpunkTHat's what I'm trying to do but got mind broken14:35
noonedeadpunkwhen tried to map things14:35
noonedeadpunkas some migration should be made as well...14:36
spatelI want to apply thirdparty certificate to haproxy. can i bundle my crt/key/chain-ca and give it to this variable? haproxy_user_ssl_cert: ? 14:36
spatelhmm - https://docs.openstack.org/project-deploy-guide/openstack-ansible/draft/app-advanced-config-sslcertificates.html14:38
noonedeadpunkyes, I think this is correct way of doing that14:39
spatelassuming this is going to be chain certificate - haproxy_user_ssl_ca_cert: /etc/openstack_deploy/ssl/ExampleCA.crt 14:39
johnsomnoonedeadpunk If you haven't seen it, there is a guide to how it should be set up: https://docs.openstack.org/octavia/latest/admin/guides/certificates.html14:41
noonedeadpunkjohnsom: I pushed some fix for it today ;) https://review.opendev.org/c/openstack/octavia/+/83885114:42
noonedeadpunknow it's jsut our codebase that's hard to read :)14:42
johnsomnoonedeadpunk Hmmm, will review. 14:43
noonedeadpunkas last thing you want is to loose certs :)14:43
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Unify debian and ubuntu cache prep scripts  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83883614:47
jrosserneed to merge this to unblock things https://review.opendev.org/c/openstack/openstack-ansible/+/83876214:55
noonedeadpunkjrosser: do you recall where we create combined .pem for haproxy?14:59
*** ysandeep|afk is now known as ysandeep15:00
jrosserthe pki role makes a variety of different chains depending on what wierdness we need :)15:00
noonedeadpunkthen just need to figure out hwo we trigger that lol15:00
jrosserwell certainly here actually https://opendev.org/openstack/ansible-role-pki/src/branch/master/tasks/standalone/create_cert.yml#L78-L8115:02
noonedeadpunkjrosser: it creates only chain + cert. But haproxy has that + key15:02
noonedeadpunkso I'm looking where last merging with key happens15:03
jrosserin my AIO they are separate files?15:03
noonedeadpunkunless you check `/etc/haproxy/ssl/` ?15:04
jrosserhmm15:04
jrosserhere? https://github.com/openstack/openstack-ansible-haproxy_server/blob/800254b3549ed9c772798637bca7695a294dd869/tasks/haproxy_ssl_letsencrypt.yml#L106-L11415:05
jrosseroh well thats in the LE tasks.....15:05
noonedeadpunkmaybe we run that anyway....15:06
jrosserhere we go https://github.com/openstack/openstack-ansible-haproxy_server/blob/1dc4fa621c153f1503933f1bb185d9fef2789f79/handlers/main.yml#L16-L2215:06
jrosserhandler15:06
noonedeadpunkI liked that assemble more....15:06
jrosseri was thinking about spatel question, and if a user supplied cert should only apply to external15:07
noonedeadpunkI bet there's way to make both15:07
noonedeadpunkbut then cert must be wildcard15:07
spateljrosser i want to apply that to external VIP15:07
noonedeadpunkor sun15:07
jrosserimho they should be different>15:08
jrosser?15:08
jrosserinternal from PKI role and external from <wherever>15:08
noonedeadpunkbut not sure if you can have own ssl covering external endpoint and self-signed for internal15:08
jrosserright - i think we may have a functionality gap there15:08
noonedeadpunkright now I guess you can either go with all yours or all self signed. Or leave internal without ssl15:09
noonedeadpunkyup, I'd say it would be nice to use different ones 15:09
jrosserso this code iterates over all the vip https://github.com/openstack/openstack-ansible-haproxy_server/blob/master/vars/main.yml#L41-L7715:09
noonedeadpunkas well as controll lets encrypt to run for external only15:10
jrossertbh i think we have two sets of half finished code15:10
jrosseroriginally i patched for LE only on external15:11
jrosserthe haproxy config file references two different sets of cert files, one for each vip15:15
jrosserso thats OK15:15
jrosserits that this doesnt distinguish between the VIP when applying the user provided certificate15:17
jrosserthis will be an issue for internal https use cases bacause the external cert on the internal vip will not be valid15:18
jrossernoonedeadpunk: maybe it needs a list of VIP to apply to?15:23
jrosser`'src': ((haproxy_user_ssl_cert is defined) and (haproxy_user_ssl_vips | intersect(_haproxy_tls_vip_binds))) | ternary(haproxy_user_ssl_cert, haproxy_pki_certs_path ~ _cert_basename ~ '.crt')`15:23
jrosserspatel: would be interested to get this certs stuff right......15:25
noonedeadpunk` external cert on the internal vip will not be valid` unless it's wildcard or sun :)15:25
noonedeadpunkbut yes15:25
noonedeadpunkwell, we would need to change whole logic heavily I guess15:25
jrosserhow about my attempt ^^15:26
spatelcurrently i did by hand override my haproxy.pem with my cert15:26
spatelI am not using SSL on internal vip 15:27
noonedeadpunkjrosser: that's too complex to think about haproxy for me today, sorry15:27
jrosserno problem! :)15:27
noonedeadpunkI mean - we also have extra vips there or smth like that...15:28
noonedeadpunkand basically we kind of lack variables at the moment15:28
jrosseri was thinking to have the user supply a list of the VIP that the user cert should apply to15:28
jrosserthen it's not so complicated logic15:28
noonedeadpunkbut that doesn't cover usecase of having 2 different certs on different vips?15:29
jrosserit would default to using the PKI generated cert15:29
jrosserso anywhere you didnt say to put the user supplied one would get that15:29
noonedeadpunkI was just thinking if I don't want to issue wildcard (as I do now), but jsut 2 certs (as it's cheaper?) to cover internal and external separately...15:30
noonedeadpunkBut then you'd need to have list of certs kind of15:30
jrosseryes you would15:31
noonedeadpunkand things go quite complicated....15:31
jrosserthere is also an argument that 'proper' cert on the inside is less secure15:31
noonedeadpunkbut yeah, I agree, your solution improves things15:31
noonedeadpunkand easy to achieve15:31
jrossermutual TLS with a private CA cannot be talked to from something with a public trusted certificate15:32
noonedeadpunkI felt myself more comfortable regarding encryption and certs at ages of SSLv2 lol15:38
noonedeadpunkNow I feel like I don't understand a shit there15:38
*** ysandeep is now known as ysandeep|dinner16:07
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Unify debian and ubuntu cache prep scripts  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83883616:37
opendevreviewMerged openstack/openstack-ansible stable/xena: Bump SHAs for Xena  https://review.opendev.org/c/openstack/openstack-ansible/+/83876217:24
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Unify debian and ubuntu cache prep scripts  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83883617:27
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Unify debian and ubuntu cache prep scripts  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83883617:50
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Unify debian and ubuntu cache prep scripts  https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/83883618:13
*** ysandeep|dinner is now known as ysandeep18:14
*** ysandeep is now known as ysandeep|out18:20
opendevreviewMerged openstack/ansible-role-pki master: Explicitly use community.crypto collection  https://review.opendev.org/c/openstack/ansible-role-pki/+/83871418:24
opendevreviewMerged openstack/ansible-role-python_venv_build master: Improve python version detection when symlinking libraries  https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/83882619:16

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!