15:03:06 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:03:06 <opendevmeet> Meeting started Tue Nov 29 15:03:06 2022 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:03:06 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:03:06 <opendevmeet> The meeting name has been set to 'openstack_ansible_meeting'
15:03:20 <noonedeadpunk> #topic rollcall
15:03:45 <noonedeadpunk> sorry for being late - was on a short walk and realized that it's already time for meeting :)
15:03:47 <noonedeadpunk> o/
15:03:48 <damiandabrowski> hi!
15:04:18 <jamesdenton> o/
15:04:53 <NeilHanlon> o/
15:05:14 <noonedeadpunk> #topic office hours
15:05:24 <noonedeadpunk> I don't think we have new and not addressed bugs
15:06:06 <jamesdenton> i have 1 but have not submitted yet. not a big one
15:06:14 <noonedeadpunk> Ok, good.
15:06:38 <noonedeadpunk> Also there was one unsubmitted from ML but it was said that https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/865701 worked nicely
15:07:02 <noonedeadpunk> I haven't checked failures though so dunno if they're related or not
15:07:35 <jamesdenton> my designate deploy was pre-wallaby so did not run into that.
15:11:20 <noonedeadpunk> failures seem unrelated at first glance
15:11:23 <noonedeadpunk> will try to re-check
15:11:51 <noonedeadpunk> so, we have 3 huge topics
15:12:08 <noonedeadpunk> 1. osa/zookeeper 2. osa/ovn 3. osa-ironic-tidy
15:12:43 <noonedeadpunk> regarding osa/zookeeper I think it's ready for review. I will add a follow-up patch to trigger zookeeper deployment for integrated repo as well (for validate job)
15:13:09 <jrosser> o/ hello
15:13:39 <jrosser> zookeeper looks ok i just found one tiny typo
15:13:43 <jamesdenton> noonedeadpunk for later: https://bugs.launchpad.net/openstack-ansible/+bug/1998223
15:15:46 <noonedeadpunk> I tried to work on ovn, I have quite limited knowledge overall but I think it should work generaly
15:16:13 <noonedeadpunk> I posted updates to jamesdenton patch to set default at the end (but it will be defined in neutron role only) and some upgrade path
15:16:34 <jamesdenton> i installed the patches over a working install and it seemed to land OK, but i was really pushing a non->ssl upgrade and didn't get it. I'm doing a fresh deploy now and will kick the tires
15:16:46 <jamesdenton> this is a 6-node environment so i'll flesh it out
15:16:51 <opendevreview> Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper master: Add SSL support for zookeeper  https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/865449
15:17:37 <noonedeadpunk> yeah, that is smth worth testing
15:18:29 <spatel> love people started using OVN, can't wait to see it default on AIO
15:18:44 <noonedeadpunk> and I still wasn't able to look into ironic - has this in my todo list but simply ENOTIME. Despite jrosser explained very good what the issue is still need to think what can be done not in rush
15:18:49 <jamesdenton> i will try to revisit the osa/ovn default patches today, too.
15:19:12 <jrosser> there are many things addressed in ironic patches
15:19:20 <jrosser> mostly small but the consoles is the bog one
15:19:23 <jrosser> *big
15:19:30 <jrosser> though this is blocking a lot https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/865566
15:19:53 <jamesdenton> is the console code still maintained? i can't recall
15:20:09 <jrosser> for ironic or osa?
15:20:15 <jamesdenton> ironic
15:20:36 <jrosser> ipmi-sol seems to work fine, with also talk of adding some graphical stuff too
15:20:48 <jamesdenton> cool. i guess i was thinking of something else
15:21:16 <jamesdenton> shellinabox.
15:21:23 <jrosser> ah yes
15:21:55 <noonedeadpunk> Actually for 865566 I think there was some fix for centos8....
15:22:05 <noonedeadpunk> But I can't recall what it was
15:23:11 <noonedeadpunk> oh, wow, it's failed on rally...
15:23:26 <jrosser> oh rally - i thought it was tempest /o\
15:23:50 <jrosser> also ENOTIME here making keeping on top of all this pretty tricky
15:24:06 <jamesdenton> we should ask santa nicely for more
15:24:15 <noonedeadpunk> The tricky thing with all that that we kind of need to branch roles asap
15:24:30 <noonedeadpunk> or well next week latest
15:25:03 <damiandabrowski> i'll spend some time tomorrow on reviews
15:25:04 <noonedeadpunk> So we need to review things that already passing CI and decide what we want to land for sure
15:25:24 <noonedeadpunk> I think that internal TLS is out of scope at this point
15:25:50 <noonedeadpunk> I really want to land ironic fixes as well
15:26:18 <noonedeadpunk> the question is likely about ovn and if we feel comfortable to land that big change so late
15:26:46 <jamesdenton> ovn default or ovn ssl or both?
15:26:51 <noonedeadpunk> ovn default
15:26:56 <mgariepy> ovn ssl please
15:26:58 <noonedeadpunk> or both :D
15:26:59 <mgariepy> also.
15:27:08 <mgariepy> so we don't have to swap it live !
15:27:41 <noonedeadpunk> though I feel quite fine about migration path for ovn default
15:27:58 <noonedeadpunk> I won't expect too much issues if ppl use our upgrade script or follow upgrade guide
15:28:02 <jamesdenton> Well, i think that a "preflight" check for neutron could help mitigate operators forgetting to specify ml2.lxb. Just a check at the beginning of neutron playbook that looks for the var and fails if it isn't there. my biggest worry is that LXB folks would accidentally deploy ovn on top and break everything
15:28:14 <damiandabrowski> i received a question from my company about OVS/OVN TLS. Do we plan to encrypt only management traffic or everything?
15:28:20 <noonedeadpunk> The only issue I see is that we don't have proper ovn docs and all diagrams would refference only ovs/lxb
15:28:45 <jamesdenton> :(    docs are.. slow. lemme check where i'm at on that
15:29:07 <noonedeadpunk> I think it covers only management traffic
15:29:19 <noonedeadpunk> From what I understood
15:29:20 <jrosser> it has never been about data plane?
15:29:46 <damiandabrowski> ack, thanks
15:29:56 <noonedeadpunk> well, we can land docs a bit later as well
15:30:51 <jamesdenton> also, i just saw your 'define-neutron-plugin' playbook. sorry
15:31:42 <spatel> we should put some check where people don't accidentally apply ovn playbook on lxb infra
15:33:03 <noonedeadpunk> I checked that upgrade playbook both on ovs (on our sandbox) and lxb (on aio) and it looked quite fair
15:34:17 <noonedeadpunk> I don't think we can/should do more rather then write release note and update upgrade docs to include step for defining variables if they're not yet present
15:35:21 <spatel> Do we have successful path / dock of converting LXB to OVN?
15:35:36 <jamesdenton> define... successful
15:35:46 <noonedeadpunk> jamesdenton has a blog describing it - was shared in ML quite recently as well
15:35:49 <noonedeadpunk> but yeah :D
15:36:08 <spatel> Like playbook to convert running production to ovn :)
15:36:21 <noonedeadpunk> oh, wow, quite ambitious :D
15:36:41 <spatel> jamesdenton send me that link i would like to give it a try in my deployment env
15:36:43 <jamesdenton> playbook? naw. but you could make one from the steps
15:36:54 <jamesdenton> https://www.jimmdenton.com/migrating-lxb-to-ovn/
15:37:05 <spatel> +1
15:37:31 <spatel> jamesdenton no kidding :) - https://www.jimmdenton.com/assets/images/2022-08-31-migrating-lxb-to-ovn/walk-away.gif
15:37:40 <noonedeadpunk> I can only imagine what playbook that would be
15:37:45 <jamesdenton> the gifs are the best part
15:38:01 <damiandabrowski> there was also a talk about it in Berlin: https://www.youtube.com/watch?v=O68Fzry50ic
15:38:02 <jamesdenton> not a playbook for faint of heart
15:38:16 <noonedeadpunk> it was from ovs though iirc
15:38:28 <noonedeadpunk> migration from ovs is simple (comparing)
15:38:30 <jamesdenton> yes, i think there may even be scripts for that
15:38:31 <spatel> ovs to ovn is easy but lxb to ovn tricky
15:39:02 <damiandabrowski> ahh sorry, i missed that we're talking about LXB
15:39:25 <jamesdenton> spatel has volunteered to write the playbook
15:39:38 <noonedeadpunk> Now it's in the meeting logs ;)
15:39:53 <spatel> I wish.. trust me..
15:39:57 <damiandabrowski> \o/
15:40:29 <spatel> I am underwater last few month.. building new datacenter so migrating all my openstack cloud to new DC
15:40:40 <jamesdenton> busy is good
15:40:42 <spatel> not fun when you move your DC
15:41:02 <noonedeadpunk> oh ,well. we have one more thing that is named sahara
15:41:11 <jamesdenton> oh?
15:41:18 <noonedeadpunk> so sahara is simply broken on Zed as upstream service
15:41:38 <noonedeadpunk> And eventually it's not passing our tempest either (mostly becuase of jsonschema version)
15:42:31 <noonedeadpunk> So kind of 2 things we can do - disable tempest tests for sahara or use Yoga u-c for it
15:43:33 <noonedeadpunk> I even proposed some patch to fix it https://review.opendev.org/c/openstack/sahara/+/864728 but meh...
15:43:48 <noonedeadpunk> it seems a bit more tricky then expected.
15:44:05 <noonedeadpunk> (or better say complex)
15:44:41 <jamesdenton> Hmm, well maybe just a release note to say.. don't upgrade to Zed until Sahara is fixed and move on?
15:45:28 <jrosser> i wonder if anyone uses it
15:45:30 <noonedeadpunk> yeah. but we won't be able to merge any patches to our repos
15:45:40 <noonedeadpunk> or well, to os_sahara
15:46:05 <noonedeadpunk> jrosser: that's good quetsion, but sahara has osa integration test in their pipeline as well
15:46:27 <noonedeadpunk> so it's quite a rabbit hole tbh
15:46:29 <damiandabrowski> (i have one thing to discuss regarding internal tls, so please let me know if we're done discussing other things)
15:46:53 <jrosser> i would say tc maybe declare project dead if a working release is not exsting for Zed?
15:47:14 <noonedeadpunk> I think I will go on and propose to comment out https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables_sahara.yml.j2#L13-L14
15:47:29 <noonedeadpunk> jrosser: yup, that's smth we're on tbh
15:47:30 <jrosser> good idea
15:47:39 <noonedeadpunk> but it's not that easy either
15:47:51 <noonedeadpunk> (and sahara not the only project)
15:49:06 <noonedeadpunk> I think one of main issues that there was a PTL volunteer for sahara but we don't see much activity but want to give some chance...
15:49:09 <noonedeadpunk> so ugh
15:49:19 <noonedeadpunk> damiandabrowski: go on I guess :)
15:49:51 <damiandabrowski> thanks!
15:49:59 <damiandabrowski> As Dmitriy suggested, we are not aiming to implement internal TLS in Zed due to the lack of time. We'll implement it in Antelope instead. (but I'm not stopping working on it)
15:50:01 <damiandabrowski> It gives us plenty of time so I came up with a slightly different idea of implementing it.
15:50:04 <damiandabrowski> As you may know, the biggest challange is to provide a way for a smooth transition from unencrypted to encrypted traffic.
15:50:07 <damiandabrowski> It's tricky because we configure all haproxy endpoints at once during haproxy-install.yml.
15:50:13 <damiandabrowski> At the beginning we planned to implement a temporary feature to handle both HTTP & HTTPS traffic by haproxy.
15:50:15 <damiandabrowski> Unfortunately it's quite complex and do not solve all our current issues(for ex. haproxy uses variables from different roles which may be not accessible in haproxy hostgroup).
15:50:16 <damiandabrowski> So I was thinking about avoiding haproxy endpoint configuration directly when running haproxy-install.yml playbook and move it to the service roles.
15:50:30 <damiandabrowski> In this case, for ex. os_glance would contain import_role pointing to the subset of tasks from haproxy-install.yml to configure endpoints for glance(the same as we do for pki role).
15:50:31 <damiandabrowski> It may give us 2 main benefits:
15:50:31 <damiandabrowski> What do you think? do you see any blockers? For sure we'll need to work on playbooks/common-tasks/haproxy-endpoint-manage.yml to avoid managing non-existent haproxy endpoint, but except this I don't see any real issues.
15:50:32 <damiandabrowski> It would look pretty similar to kolla-ansible, except the fact I don't think we need a separate role for this. https://github.com/openstack/kolla-ansible/tree/master/ansible/roles/haproxy-config
15:50:37 <damiandabrowski> 2. No more problems with trying to access unavailable variables when running haproxy-install.yml
15:50:39 <damiandabrowski> 1. No need to worry about http->https transition for backends.
15:52:13 <jrosser> wherever we have variables that must be accessed across roles those need to be in group_vars
15:52:19 <jrosser> haproxy or not haproxy
15:52:25 <noonedeadpunk> The one problem I see is that os_glance role is not running against haproxy_all hosts. While we do delegating for pki/python_venv_build we delegate only to 1 host
15:52:41 <noonedeadpunk> And I don't think ansible can delegate to groups (but I never tried)
15:53:22 <damiandabrowski> https://paste.openstack.org/show/bvi2iSjNMbJrCyM51bw1/
15:53:23 <damiandabrowski> you can use loop, i tested it yesterday
15:54:10 <jrosser> :( we should use proper var scopes when we need
15:54:36 <damiandabrowski> but it actually solves our issue, isn't it?
15:54:51 <damiandabrowski> so according to my test, if os_glance role includes haproxy role - it has glance vars available
15:54:55 <noonedeadpunk> another hassle I see is certs/let's encrypt. But maybe we can issue them during haproxy-install playbook and just somehow utilize later
15:55:17 <noonedeadpunk> yeah, you can delegate_facts I guess or smth
15:55:56 <noonedeadpunk> I kind of like the idea as it would solve some of our issues. I'm not sure it won't bring another ones though :D
15:55:56 <jrosser> looping over delegate_to just feels like re-inveting the host loop of a playbook too
15:56:02 <noonedeadpunk> But we wil lnever know until we try
15:56:42 <noonedeadpunk> Well, even thinking about moving haproxy service configuration to the service playbook sounds like improvement, isn't it?
15:56:45 <damiandabrowski> so to summarize: this idea may require some tweaks but you don't have anything against that and I can prepare some PoC?
15:57:23 <damiandabrowski> for me it definitely sounds like an improvement :D
15:58:04 <noonedeadpunk> My opinion is that I'm not sure if we will gain any profit and will be able not to make role even more complex, but it might be worth trying to see if it's ok
15:58:25 <jrosser> there will be also some corner cases
15:58:27 <noonedeadpunk> As it might result in quite a simplification as well
15:58:35 <jrosser> LE is somehow tied to horizon
15:58:46 <jrosser> and consoles are not tied to one service
15:59:15 <jrosser> so there are some plusses, but it will also have tricky parts which are similar to the things that are tricky today
15:59:16 <noonedeadpunk> I guess for LE we did some trick for keystone as well? Not sure though....
15:59:42 <jrosser> port 80 and 443 are there for the benefit of horizon
16:00:00 <noonedeadpunk> ah, yes, true
16:00:07 <jrosser> but it is not making much sense to deploy LE as part of horizon
16:00:15 <jrosser> thats like step zero before everything else
16:01:00 <damiandabrowski> okok thanks for your input! I made some notes and will look into that soon
16:02:55 <noonedeadpunk> #endmeeting