Tuesday, 2021-04-27

*** sshnaidm_ has joined #openstack-ansible00:00
*** sshnaidm|afk has quit IRC00:01
*** sshnaidm_ has quit IRC00:07
*** sshnaidm_ has joined #openstack-ansible00:08
*** luksky has quit IRC00:08
*** recyclehero has quit IRC00:18
*** recyclehero has joined #openstack-ansible00:38
*** gyee has quit IRC01:22
*** mrda has quit IRC02:14
*** recyclehero has quit IRC02:14
*** evrardjp has quit IRC02:33
*** mrda has joined #openstack-ansible02:33
*** evrardjp has joined #openstack-ansible02:33
*** rh-jelabarre has quit IRC02:34
*** spatel_ has joined #openstack-ansible02:45
*** spatel_ is now known as spatel02:45
*** spatel has quit IRC03:36
*** miloa has joined #openstack-ansible04:58
*** recyclehero has joined #openstack-ansible05:05
*** miloa has quit IRC05:07
*** recyclehero has quit IRC05:26
openstackgerritMerged openstack/openstack-ansible-specs master: Add Root CA spec  https://review.opendev.org/c/openstack/openstack-ansible-specs/+/75880505:31
*** SiavashSardari has joined #openstack-ansible05:35
*** SiavashSardari has quit IRC05:50
*** recyclehero has joined #openstack-ansible05:57
*** luksky has joined #openstack-ansible05:59
*** pto has joined #openstack-ansible06:11
*** pto has quit IRC06:19
jrosseri do wonder how that config-options-in-vault protects the token used to access vault06:26
jrosserunless i miss something it just moves the problem to protecting that instead06:26
jrosserhttps://opendev.org/openstack/castellan/src/branch/master/castellan/_config_driver.py06:30
*** tsturm has quit IRC06:35
*** pto has joined #openstack-ansible06:35
noonedeadpunkyeah, it's not06:41
noonedeadpunkbut I haven't found it indeed (expected to see in oslo though)06:42
*** shyamb has joined #openstack-ansible07:00
jrossermaybe its runtime pluggable07:02
jrosserhttps://opendev.org/openstack/castellan/src/branch/master/setup.cfg#L33-L3407:03
noonedeadpunkyeah, already found. I meant about how this helps to protect vault is not really clear :)07:04
jrossermaybe the reasoning is that a compromised service could read it's own config file07:04
jrosseror anything that it has read permission to07:04
jrosserand its a step further to then gain root to read the file with the vault token in07:05
noonedeadpunkbut it will have read permissions to the included files anyway...07:05
noonedeadpunkbut yeah, it's step futher. Anyway, I think it might be useful07:05
jrosseryes, though i think that the use case is not totally well explained07:05
noonedeadpunkas what I'm dreming about is to store user_secrets in vault07:05
jrosserah well, i was thinking about that07:06
jrosserwe already store the whole blob in vault in case we need to re-deploy our deploy host07:06
noonedeadpunkand having vault deployed is a huge help there07:06
jrosserbut it still just sits there plaintext after that, which is not so great07:07
noonedeadpunkbut do you use vault-ansible? or just script to store suer_secrets?07:07
noonedeadpunkyeah...07:07
jrosserhashi vault07:07
jrosserthen in playbook pre_tasks we do a bunch of these http://paste.openstack.org/show/804787/07:09
jrosserone of the challenges with using hashi-vault heavily with ansible is you need to only read the secrets once and cache them for the lifetime of the playbook07:11
jrosserotherwise if you template variables directly with a lookup reading from vault then it does that hundreds and hundreds of times every time the var is evaluated, and the deployment speed is really bad07:12
noonedeadpunkdoh, gotcha07:15
noonedeadpunkI was just thinking about using ansible vault keyring script07:15
*** andrewbonney has joined #openstack-ansible07:15
jrosserah ok07:16
noonedeadpunkBut I guess there it will be really geting secrets each time07:16
jrosseri never really looked at vars plugins07:16
jrossermaybe thats a way to have something equivalent to user_secrets but injected at runtime from some secure source07:17
jrosseranyway..... i was going to take another look at rabbitmq/pki07:18
noonedeadpunkyeah. And I will have a look at ansible-core 2.11...07:19
jrosserright now i re-used rabbitmq_user_ssl_cert vars to wire everything together07:19
jrosserif i move the pki role include into the rabbitmq role, any ideas how that would work07:20
jrosseri.e do we leave the old vars for user supplied certs07:20
noonedeadpunkwell, the way I wath thinking about it, that we provide pki role bunch of variables, and put all logic into pki role07:22
jrosseri'm not totally sure right now how we operate this in two different modes, where the pki role generates everything, or where the user supplies some07:22
noonedeadpunkIe if user provided certificate is not empty -we take it instead of the generating self-signed one07:22
jrosserok, it's just that the logic right now is spread everywhere, if you define the haproxy/rabbitmq user_certs vars then the existing roles fetch them from the path you give07:23
jrosserand each role may need to do different things to install / concatenate / whatever the certs to install them for the particular application07:23
noonedeadpunkok, so we might want to limit pki role to jsut generating stuf if required?07:24
jrosserperhaps, i'm not really sure07:24
jrosserdepends how much we want to keep the old vars, maybe07:25
noonedeadpunkbut I think there super limited amount of options. I mean there can be either pkcs#12 or pem or combined pem as output?07:25
jrosseryeah actually - perhaps i look also at moving the install part into the pki role07:26
noonedeadpunkyeah, I guess we can try that out, and just provide path and format we expect to see as a result?07:27
noonedeadpunkLet me know if I can help or pick up some work regarding it07:29
*** shyamb has quit IRC07:29
jrosserahhh well i can make some pki role vars like pki_install_certs: "{{ user_supplied_cert_path ~ default(pki_role_made_cert_path) }}"07:30
jrosser| default...07:30
*** shyamb has joined #openstack-ansible07:33
*** shyamb has quit IRC07:36
*** shyamb has joined #openstack-ansible07:36
*** shyamb has quit IRC07:38
*** shyamb has joined #openstack-ansible07:39
*** tosky has joined #openstack-ansible07:41
*** rpittau|afk is now known as rpittau07:45
* noonedeadpunk struggling with openstack upgrade atm :(07:46
noonedeadpunkI found nasty thing - `lxc.mount.entry = /openstack/control01_galera_container-c96842be var/lib/mysql none bind,create=dir 0 0`07:46
noonedeadpunkand in `/openstack/control01_galera_container-c96842be` we put all kind of different stuff by default....07:46
noonedeadpunkwhich I think acts weird when you try to re-join empty node to the cluster07:47
noonedeadpunkthis all is super weurd https://codesearch.opendev.org/?q=%2Fopenstack%2F%7B%7B%20inventory_hostname%20%7D%7D&i=nope&files=&excludeFiles=&repos=07:55
noonedeadpunkand I have no idea how we can change that in terms of upgrade07:55
noonedeadpunkand that's super strange https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/glance_all.yml#L29-L3307:58
noonedeadpunkjrosser: ^07:59
noonedeadpunkI really no idea how to properly fix that.... especially for glance08:05
jrosserso i'm guessing that it's trying to put persistent data outside the container?08:05
jrosserwell persistent / large08:06
jrosserin that the LXC might be on the system disk and /openstack be some other larger device08:06
noonedeadpunkyes, but we bind mount same directory into 2 different places in container08:07
noonedeadpunkand on top of that we put mac_generation script there08:07
jrosseri'm just looking on my lab setup08:09
jrosserthose mounts don't seem present on the glance container08:09
noonedeadpunkyeah, they're just for file08:11
noonedeadpunkbut, for galera I tried to re-add member to cluster and it was failing08:11
*** shyamb has quit IRC08:11
noonedeadpunkuntil I removed https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/master/tasks/lxc_container_config.yml#L95-L101 as it's in the exact same directory as /var/lib/mysql08:13
noonedeadpunkand thus SST transfer was failing08:13
noonedeadpunkand IST is not appropriate as start position is 00000000-0000-0000-0000-000000000000:-108:13
*** shyamb has joined #openstack-ansible08:48
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: DNM - Test PKI role  https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/78803208:55
openstackgerritJonathan Rosser proposed openstack/ansible-role-pki master: WIP - Create server certificates  https://review.opendev.org/c/openstack/ansible-role-pki/+/78802108:57
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_container_create master: Do not create extras in /openstack/{{ inventory_hostname }}  https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/78822209:01
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: WIP - Test PKI role  https://review.opendev.org/c/openstack/openstack-ansible/+/78803109:04
jrossernoonedeadpunk: i've made it much more like python_venv_build for rabbitmq now https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/788032/2/tasks/main.yml09:04
jrossergot a few hours of meetings now :(09:05
* noonedeadpunk has disaster upgrade with broken galera and rabbit cluster :(09:06
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-rabbitmq_server master: DNM - Test PKI role  https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/78803209:06
*** snapdeal has joined #openstack-ansible09:14
*** SiavashSardari has joined #openstack-ansible09:34
*** pto_ has joined #openstack-ansible09:56
*** pto__ has joined #openstack-ansible09:57
*** pto_ has quit IRC09:57
noonedeadpunkdoh, centos does not have libselinux bindings for python38 and not ging to have :(09:59
*** pto has quit IRC09:59
*** sshnaidm_ is now known as sshnaidm10:00
noonedeadpunkand ansible-core has soft requirement of py38....10:01
jrosserit's almost like one half of RH doesnt know what the other is doing10:09
*** shyamb has quit IRC10:22
*** shyamb has joined #openstack-ansible10:27
*** shyamb has quit IRC10:32
*** pto__ has quit IRC10:36
*** pto has joined #openstack-ansible10:37
*** pto has quit IRC10:41
*** pto has joined #openstack-ansible10:48
*** pto has quit IRC10:52
*** shyamb has joined #openstack-ansible11:02
*** pto has joined #openstack-ansible11:19
*** macz_ has joined #openstack-ansible11:24
*** macz_ has quit IRC11:28
*** rh-jelabarre has joined #openstack-ansible11:30
*** SiavashSardari has quit IRC11:33
noonedeadpunkI think that's why they are going to have ansible running in container...11:37
*** pto has quit IRC11:46
jrossermaybe we should look at pyenv a bit more11:50
jrosserthat would let us have a guaranteed python version on all OS11:51
noonedeadpunkbut we would still need to build libselinux...11:58
noonedeadpunkalong with dnf or apt modules I guess?11:59
noonedeadpunkor lxc and etc...11:59
*** pto has joined #openstack-ansible12:01
*** shyamb has quit IRC12:02
noonedeadpunkoh, but we don't need lxc on deploy host - only on dest12:04
noonedeadpunkso just libselinux kind of...12:04
*** pto has quit IRC12:23
*** pto has joined #openstack-ansible12:24
*** pto has quit IRC12:28
*** shyamb has joined #openstack-ansible12:35
*** macz_ has joined #openstack-ansible12:37
*** macz_ has quit IRC12:42
*** snapdeal has quit IRC12:43
*** spatel_ has joined #openstack-ansible12:46
*** spatel_ is now known as spatel12:46
*** macz_ has joined #openstack-ansible12:58
*** pto has joined #openstack-ansible13:00
*** macz_ has quit IRC13:02
*** pto has quit IRC13:06
*** pto has joined #openstack-ansible13:06
*** fridtjof[m] has joined #openstack-ansible13:09
*** pto has quit IRC13:16
*** pto has joined #openstack-ansible13:17
*** shyamb has quit IRC13:20
*** pto has quit IRC13:21
*** pto has joined #openstack-ansible13:42
*** pto has quit IRC13:47
fridtjof[m]looking at the production network example (https://docs.openstack.org/openstack-ansible/victoria/user/prod/example.html), do I still need a veth pair for neutron on compute hosts if my external network is just a simple flat (no vlans) bridge?14:03
fridtjof[m]Reason i'm asking is because that's the one thing preventing me from using netplan or networkd for network configuration, and I'd like to avoid setting up ifupdown on ubuntu 20.04...14:04
admin0fridtjof[m], you can use netplan just fine14:08
admin0on ubuntu 2014:08
admin0you need to create 4 bridges and that is all about it :)14:08
admin0in terms of requirement14:08
admin0if you are using linuxbridge, you do not need to do anything further .. if ovs, then you need diff setup in controllers vs computes14:09
admin0didn't understood "do I still need a veth pair for neutron on compute hosts"  ??14:09
fridtjof[m]iirc, how i understood it a year ago or two (stein, i think), that was necessary because neutron agents will run in both containers (infra hosts) and on metal (compute hosts), but still expect the same kind of access to tenant networks14:11
noonedeadpunkI believe you might need veth pair for octavia or trove dpeloyment, but not on computes but on controllers, when they are shared with network nodes14:11
noonedeadpunkneutron agents always run on metal by default. only neutron api runs in container14:12
fridtjof[m]oh i see, then that was a misunderstanding on my part. I'll try without a veth pair, and report back14:14
fridtjof[m]that would greatly simplify my deployment preparation14:14
*** gshippey has joined #openstack-ansible14:22
jrosserthere was a veth pair used to create eth12/13/14 on hosts which was necessary for the example configs when using flat networks14:22
jrosserimho thats really just a suggestion and generalisation and not how you'd do it in a production setup14:22
*** sshnaidm has quit IRC14:38
*** sshnaidm has joined #openstack-ansible14:40
*** d34dh0r53 has quit IRC14:47
*** d34dh0r53 has joined #openstack-ansible14:47
*** pto has joined #openstack-ansible14:53
noonedeadpunk#startmeeting openstack_ansible_meeting15:00
openstackMeeting started Tue Apr 27 15:00:44 2021 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"15:00
openstackThe meeting name has been set to 'openstack_ansible_meeting'15:00
noonedeadpunk#topic rollcall15:00
noonedeadpunko/15:00
*** openstack changes topic to "rollcall (Meeting topic: openstack_ansible_meeting)"15:00
mgariepyhello !15:01
noonedeadpunkhey mgariepy! how're you doing?:)15:01
mgariepynot too bad.15:01
mgariepyhow are you doing ?15:02
noonedeadpunkhaving bad upgrade :(15:02
mgariepyreally ? how comes?15:02
noonedeadpunkbut that's fine)15:02
* noonedeadpunk no idea15:03
mgariepywhat's failing ?15:03
noonedeadpunkended up with both broken rabbit, so had to wipe it and re-create from scratch, including vhosts and permissions15:03
mgariepyouch15:03
noonedeadpunkand now galera falled apart as well15:03
openstackgerritwu.chunyang proposed openstack/openstack-ansible master: setup.cfg: Replace dashes with underscores  https://review.opendev.org/c/openstack/openstack-ansible/+/78831215:03
noonedeadpunkok, so...15:05
noonedeadpunk#topic office hours15:05
*** openstack changes topic to "office hours (Meeting topic: openstack_ansible_meeting)"15:05
noonedeadpunkI still haven't sent anything from PTG, will do this right after the meeting15:05
*** ajg20 has joined #openstack-ansible15:06
noonedeadpunkAlso, there's a question regarding what we should do with ansible goes ahead15:06
noonedeadpunkToday ansible-core 2.11 has been released which has soft requirement of py3815:07
noonedeadpunkI think it mean that no CI has been done for py36, but not sure15:07
noonedeadpunkI think it's smth we can already use for W, but probably worth postponing for X?15:07
jrossero/ sorry i'm late15:08
noonedeadpunkAs there we will drop bionic with it's 3.6 and already bullseye will be stable with 3.9 on board15:09
noonedeadpunkthe only issue is centos atm. But might be stream get some libselinux python binding till then?15:09
ajg20Hello, I have setup OpenStack-Ansible on a server and I am getting the following error "Error: Failed to perform requested operation on instance "cirros", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance15:10
ajg20d376970d-19a0-4bc5-a47e-43ef6ed2d63c.]." More details http://paste.openstack.org/show/804801/ . Can someone help me out?15:10
noonedeadpunkwhile we can build pyenv ofc, I really dunno about building libselinux bindings...15:10
noonedeadpunkajg20: having meeting now, will be able to help in an hour or so15:11
ajg20Thank you, Let me know when you have free.15:12
*** pto has quit IRC15:13
noonedeadpunkSo I'd say let's use ansible-base 2.10 for W and see how things will go during next cycle?15:14
jrosseryeah, i think so15:14
jrosseri wonder if spotz might know who to ask how we're supposed to use ansible-core on centos8 w.r.t python selinux bindings15:15
noonedeadpunkI think that with 2.12 ansible-core they will implement some kind of containers for ansible-core15:16
noonedeadpunkso that they won't need to worry about py in centos15:17
noonedeadpunkAnother thing we've briefly discussed during the day is https://review.opendev.org/c/openstack/openstack-ansible-specs/+/78805715:17
jrosserthere is this https://ansible-runner.readthedocs.io/en/latest/15:18
noonedeadpunk`Python 2.7+ and 3.6+ are supported and installable via pip` lol15:19
jrosseroh :/15:19
noonedeadpunkpretty deserted I guess? https://ansible-runner.readthedocs.io/en/latest/install.html#changelog15:19
noonedeadpunkdunno though15:19
*** macz_ has joined #openstack-ansible15:21
jrosserconfig in vault is one thing15:21
jrosserbut all deployment secrets in vault in the general sense, i.e OSA wide, is kind of something else again15:22
noonedeadpunkyeah, agree15:22
jrosserand as a deployment tool i kind of figure we should care about both of those15:22
noonedeadpunkand this blueprint is regarding config in vault only15:22
jrosseryes15:22
jrosserdeployment environment secrets in vault poses some interesting chicken/egg challenges too15:23
noonedeadpunkprobably as soon as we will have vault deployment, we can figure out how we want to integrate it with secrets as well15:23
noonedeadpunkAs I'm not sure about time during the upcoming cycle for this thing.15:23
jrosserno me neither15:24
noonedeadpunkWhile blueprint is assigned to the patch owner :)15:24
noonedeadpunkAnd I have vault role in exact OSA format, where used even galera_role15:25
jrosserah interesting, we've got one too using the internal data store and raft HA15:25
noonedeadpunkI used haproxy in octavia....15:26
noonedeadpunkbut I mean it should fit perfectly I guess to what we have in terms of containers and stuff15:26
noonedeadpunkbut yeah, now I got chicken/egg situation...15:27
jrosseri did another pass on the pki role setup for rabbitmq15:29
noonedeadpunkI think as long as they're going to lead implementation I'm pretty much fine with it15:29
jrosserthats looking a lot more like python_venv_build approach now15:29
noonedeadpunkoh, I saw pki has passed, haven;t checked rabbit15:29
jrosserhalf way through doing the same for haproxy though theres more complexity there with needing to keep the original functionality + certbot and stuff15:29
noonedeadpunkyeah, I was thinking about smth like that, but haven't digged into details yet15:29
jrosseri still need to look at the variable names again, make them rabbitmq specific in defaults/main.yml with the option for a deployment wide openstack_<var> global setting15:30
noonedeadpunkyeah, you read my thoughts!15:31
noonedeadpunkI'm wondering if there's good usecase to use letsencrypt for rabbit/galera as well15:31
noonedeadpunkin case of dns-01 auth15:31
jrosseras a positive though it's really cleaned up the code for ssl in the rabbit role15:31
noonedeadpunknot sure how to implement it though15:31
noonedeadpunkyep, a lot of dropped stuff. And I think overall it will be the way cleaner15:32
jrosserright, so theres a var in the pki role, pki_method15:32
jrosseri'd intended to allow that to be used as some kind of extension where we could add certbot support in the future to the role15:33
jrosserand allow the caller to specify the backend used to issue the cert15:33
noonedeadpunkoh, nice idea, yes15:34
jrosserif we do it right then you'd be able to specify that per certificate15:34
noonedeadpunkand eventuyally that might be even some third part tooling (forgot what exact thing you talked about previously)15:36
jrosseryes indeed, we've got step-ca running here as an internal CA15:36
jrosserso i'd like to leave the door open for adding more opinionated backends like that to the pki role15:37
noonedeadpunkyep, agree15:37
jrosseris there anything else we need to prioritise to get merged for a release15:41
noonedeadpunkoh, centos stream15:41
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible master: Add centos-8 stream jobs  https://review.opendev.org/c/openstack/openstack-ansible/+/77622615:42
jrosseroh urgh distro_metal_centos8 fails, of course!15:44
noonedeadpunkon cinder :(15:50
*** spatel has quit IRC15:50
jrosseri also looked at the tempestconf fix we had, though i think we can't test that with a depends-on15:51
jrosseroh i think it is possible, if we make a DNM patch which sets tempest_tempestconf_pip_packages to the url of the tempestconf change in gerrit15:53
noonedeadpunkum....15:59
noonedeadpunkI think the issue with gerrit change is in refs?15:59
noonedeadpunkanyway..16:03
noonedeadpunk#endmeeting16:03
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/osa-review-board-v3"16:03
openstackMeeting ended Tue Apr 27 16:03:18 2021 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:03
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-04-27-15.00.html16:03
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-04-27-15.00.txt16:03
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-04-27-15.00.log.html16:03
*** jamesden_ has joined #openstack-ansible16:21
*** jamesdenton has quit IRC16:22
spotzjrosser: You still need me to check on ansible-core?16:30
*** gyee has joined #openstack-ansible16:30
*** rpittau is now known as rpittau|afk16:33
noonedeadpunkI'm actually wondering what ooo is going to do with ansible version and centos that does not have libselinux binding for py38...16:36
*** spatel_ has joined #openstack-ansible17:02
*** spatel_ is now known as spatel17:02
*** andrewbonney has quit IRC17:05
*** pto has joined #openstack-ansible17:18
*** pto has quit IRC17:18
openstackgerritDmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: DNM - test patch for senlin tempest testing  https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/75404518:01
*** Adri2000 has quit IRC18:28
*** openstackstatus has quit IRC18:31
*** openstackstatus has joined #openstack-ansible18:32
*** ChanServ sets mode: +v openstackstatus18:32
spatelHow are you folks doing :) look like lots of activity going on19:18
spateldealing with some massive DDoS issue :( and finally implemented protection plan19:19
spatelhope no more trouble so i get more time to play with my OVN deployment.19:19
*** spatel has quit IRC20:05
*** gshippey has quit IRC21:14
*** cloudnull has quit IRC21:24
*** cloudnull has joined #openstack-ansible21:25
*** macz_ has quit IRC22:03
*** macz_ has joined #openstack-ansible22:28
*** macz_ has quit IRC22:33
*** luksky has quit IRC22:34
*** tosky has quit IRC23:08

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!