15:00:41 <noonedeadpunk> #startmeeting openstack_ansible_meeting
15:00:42 <openstack> Meeting started Tue May 18 15:00:41 2021 UTC and is due to finish in 60 minutes.  The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:43 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:45 <openstack> The meeting name has been set to 'openstack_ansible_meeting'
15:00:51 <noonedeadpunk> #topic rollcall
15:00:54 <noonedeadpunk> o/
15:02:06 <jnamdar> I'm getting a 404 on /v1 or /v2
15:06:48 <noonedeadpunk> #topic office hours
15:07:12 <noonedeadpunk> I can't actually recall new bugs that were submitted recentl;y, so skipping that topic
15:08:05 <noonedeadpunk> I want to return to the topic of dropping branches. For some reason I thought that ocata will be just dropped, but seems like we need to mark it eol
15:08:56 <noonedeadpunk> So I wonder if we should limit to ocata? or take pike with it? as at the end it feels up to us to pick stuff to drop
15:09:56 <noonedeadpunk> we kind of discussed that later and stopped only on Ocata
15:10:28 <noonedeadpunk> so I will proceed with marking it as eol
15:10:48 <noonedeadpunk> and abandon all patches that were submitted for it
15:10:56 <admin0> i am in the process of upgrading one env from ocata -> latest , so please dont delete files :D
15:11:17 <noonedeadpunk> oh...
15:11:38 <noonedeadpunk> I was going to drop branch and tags related to ocata
15:11:52 <admin0> how can i request more time on this
15:12:05 * admin0 will send pizzas :)
15:12:16 <noonedeadpunk> about what time period are we talking about?
15:12:29 <admin0> can i pm ?
15:12:51 <noonedeadpunk> um, yes, but descision has to be taken in channel :)
15:13:31 <admin0> there is a very important cloud .. but in ocata that needs to be upgraded .. i am setting up a lab to replicate that this week .. within 15-20 days i will be able to tell ..
15:14:26 <noonedeadpunk> I can imagine how challenging it would be to spawn ocata deployment in 2021
15:14:39 <admin0> it is
15:15:00 <admin0> if its deleted, if there can be way how I can clone an earlier branch/tag or preserve it , that would be great
15:15:27 <admin0> is asking till end of the year a big deal ?
15:15:27 <noonedeadpunk> I think you will still be able to checkout to sha from master....
15:15:35 <jrosser> the tags arent deleted are they?
15:15:55 <admin0> can i bribe you with a pizza to keep it till end of year :)
15:16:14 <admin0> and beer
15:16:15 <spatel> curious what is the rush to delete old tags?
15:16:20 <noonedeadpunk> jrosser: but tags are on branch? Actually along with sha's...
15:16:50 <spatel> why don't we keep them if they don't cost $$
15:17:02 <admin0> i have one critial infrastructure handling some government infrastrucutre to be upgraded from ocata -> latest till end of year
15:17:19 <admin0> so if it can be kept till end of year, i would be very grateful
15:17:32 <admin0> "an undisclosed" government will be very grateful
15:17:41 <jrosser> i was trying to follow the discussion in other channels about this
15:18:29 <noonedeadpunk> admin0: I think that would be super challenging also because really other projects are dropping their branches super actively nowadays
15:18:50 <noonedeadpunk> I wasn't following discussions, but was following ML
15:19:24 <jrosser> when l look in the nova repo (on github) there are still tags for <branch>-em on really ancient branches
15:19:34 <jrosser> but the branches themselves are removed
15:19:41 <jrosser> afaik it's just a pointer to a sha
15:20:10 <noonedeadpunk> yeah, it's a pointer to sha.... but sha in specific tree?
15:20:34 <noonedeadpunk> I mean if you merge old branch with master, then SHA will be valid
15:20:56 <jrosser> though the point about the other repos is totally valid for admin0 case
15:21:12 <jrosser> it's just generally happening that the old branches are being removed
15:21:46 <admin0> if its have to be dropped, i cannot stop it .. but i would need some help to figure out what i can do now to clone or do stuff so that i will be able to deploy ocata
15:22:10 <noonedeadpunk> actually now I wonder if I did right by checkout to stable/train for EM... I think I should have checkout to train-em instead
15:22:28 <noonedeadpunk> admin0: well, there's no rush for us specifically
15:22:47 <jrosser> but look i think this isnt necessarly a problem
15:22:48 <jrosser> https://github.com/openstack/nova/tree/kilo-eol
15:22:59 <jrosser> the code is there, the branch is deleted
15:23:10 <jrosser> but the tags remain as references
15:23:16 <noonedeadpunk> then I've missed about hob git works...
15:24:31 <noonedeadpunk> well, ok, then it solves everybody problems I think
15:24:39 <jrosser> yes, i think it's fine
15:24:59 <noonedeadpunk> admin0: you will be able to checkout to ocata-em instead of stable/ocata
15:25:08 <admin0> that is fine
15:25:28 <admin0> can i checkout tags also ? or are tags also gone
15:25:30 <admin0> i usually checkout tags
15:25:32 <admin0> like x.y.z
15:25:49 <admin0> i don't checkout the branches like stable/X
15:25:52 <noonedeadpunk> now I don't know:) if em is working then tags should be also fine
15:26:14 <noonedeadpunk> I think I should actually do more research then
15:26:30 <jrosser> theres em and eol
15:26:41 <jrosser> some projects already em and eol stein
15:26:42 <noonedeadpunk> ah!
15:26:52 <noonedeadpunk> ok, now I got how this works:)
15:27:04 <admin0> as long as tags remain, i am not too worried about branch-names
15:27:13 <noonedeadpunk> I mean new tag is created on top of the branch that is dropped
15:27:21 <jrosser> yeah
15:28:01 <admin0> so tags will work ?
15:28:03 <noonedeadpunk> nova has all tags in place
15:28:43 <noonedeadpunk> well, anyway we can probably postpone dropping ocata as well?
15:29:32 <noonedeadpunk> I think nothing would happen if we drop branch somewhere autumn with X release? Especially it's so important
15:29:56 <jrosser> this sounds fine
15:30:15 <admin0> thank you
15:31:13 <noonedeadpunk> ok, cool.
15:32:11 <noonedeadpunk> next thing I want to discuss is how we see root CA generation. I mean - should we disable SSL or don't enable CA when variables not set, or jsut create dummy default CA?
15:32:50 <noonedeadpunk> As I think we should then say about these variables as like required ones?
15:33:17 <noonedeadpunk> and underline how important is to set them both during upgrade to W and for new setup?
15:34:18 <noonedeadpunk> fwiw W deadline is July 2
15:34:38 <jrosser> i guess that always creating the internal self-signed CA is good
15:34:52 <jrosser> even when there is a proper certificate for the public endpoint
15:35:29 <jrosser> it would still be needed for rabbitmq regardless, unless the deployer also provided their own certificates for each rabbit node
15:35:46 <noonedeadpunk> let's probably then include reference to the sample with these variables _everywhere_ - in deploy guide, in aio guide, in release notes...
15:36:16 <jrosser> the problem we have is that there is so many possible ways that this could be done
15:36:26 <jrosser> and i think we have to choose a "sensible default", as usual
15:37:33 <jrosser> if we want to move on and do further work for ssl on galera and internal endpoint then the certs become a key component
15:37:55 <noonedeadpunk> yeah. but the main concern here is that rotate CA is not so easy I guess. And in case it's missed during upgrade, you will get your production deployment with dummy info in CA
15:38:18 <jrosser> i think that rotating the intermediate is actually quite easy
15:38:44 <jrosser> the root CA itself is another matter
15:38:46 <noonedeadpunk> and info is not in root but in intermediate?
15:39:16 <noonedeadpunk> I just thought that all these org stuff and country and etc are in root?
15:39:24 <jrosser> root signs intermediate, intermediate signs server cert
15:39:59 <jrosser> so if there is a requirement to replace the trust chain for whatever reason, then thats easy up to the point of wanting to rotate the actual root CA certificate
15:40:43 <noonedeadpunk> Ah, ok, openstack_pki_authorities contain both root and intermediate
15:40:53 <jrosser> yes, as many of either as you want
15:40:55 <jrosser> it's a list
15:41:14 <noonedeadpunk> and I think intermediate can be even skipped for $reason ?
15:41:17 <jrosser> so i had in mind that we could have intermediates for services, SSH certs, <whatever>
15:41:34 <noonedeadpunk> yeah, that make sense
15:41:45 <jrosser> then you can split the risk/cost of replacing an intermediate easily without having to change everything
15:42:09 <noonedeadpunk> well, that's in case we have intermediate per service?
15:42:23 <jrosser> and as it stands now, you could extend the list of intermediates trivially
15:42:46 <jrosser> and have new certs made off the new intermediate, and their trust chain will validate against the original root CA
15:43:10 <jrosser> so keeping the root CA super-safe is perhaps key
15:43:44 <noonedeadpunk> sounds like it is
15:43:49 <jrosser> and thats something i've not tested here, supplying the root CA, plus intermediate and it's key out-of-band to the PKI role
15:43:58 <jrosser> as that would be the totally best way
15:45:16 <jrosser> i think thats it should be a case of copying the files to the right dirs in /etc/openstack_deploy/pki/roots/ and then not defining the variable that creates CA in the PKI role
15:46:57 <jrosser> tbh this is why i kind of paused at the documentation part
15:47:07 <noonedeadpunk> ok, thinking about reasonable defaults, probably you was right about placing https://review.opendev.org/c/openstack/openstack-ansible/+/788031/7/tests/roles/bootstrap-host/files/user_variables_pki.yml to group_vars instead
15:47:09 <jrosser> there is something self-contained we need for CI
15:47:27 <jrosser> and then there is whatever points we need for overriding these variables for production deployments
15:47:49 <jrosser> that might want a self-contained CA, or derived from a company CA, or whatever else
15:49:07 <noonedeadpunk> right, agred
15:49:09 <noonedeadpunk> *agreed
15:49:15 <jrosser> yes, so setting openstack_pki_authorities: [] in user_variables would stop any new CA being made
15:50:39 <jrosser> this potential complexity is a reason to keep this simple just for rabbit/haproxy for W and try to shake out some of the production use-cases during X
15:52:28 <noonedeadpunk> yeah, the point with docs, is to make ppl aware that they will get `Example Corporation` CA by default, and they really need to change that
15:52:54 <noonedeadpunk> as for upgrades I bet this will be confusing
15:53:02 <noonedeadpunk> also
15:53:16 <noonedeadpunk> should we enable ssl for all endpoints by default?
15:53:38 <jrosser> perhaps during X cycle?
15:53:52 <noonedeadpunk> yeah, might be.
15:54:16 <jrosser> i'm not sure how that should be done at haproxy
15:54:17 <noonedeadpunk> As I guess we need to write upgrade path for that
15:54:34 <jrosser> i mean it can be done today of course with setting all endpoints to SSL
15:54:50 <jrosser> be if the same certificate is the right thing to do on the inside and outside
15:55:05 <jrosser> *but if
15:55:24 <noonedeadpunk> which is only the case with wildcards  Iguess...
15:55:40 <noonedeadpunk> and with lets encrypt that would be tough
15:55:43 <jrosser> if we want to eventually also do mutual-tls on the internal endpoint then we must use the internal CA i think
15:56:01 <jrosser> and seperatly have LE or whatever on the external
15:56:40 <noonedeadpunk> yeah, let's leave it for now, agree
15:57:13 <noonedeadpunk> I just thought that we can just use internal ca for internal/admin endpoint by default
15:57:27 <jrosser> yes, i think that would be a good thing
15:57:29 <noonedeadpunk> and do mutual-tls later on
15:57:44 <jrosser> we can put an IP: SAN in there and it should just work with what we have already
16:00:54 <noonedeadpunk> #endmeeting