| *** dtantsur_ is now known as dtantsur | 10:22 | |
| *** dtantsur_ is now known as dtantsur | 10:51 | |
| *** ykarel is now known as ykarel|away | 13:47 | |
| gmann | tc-members: in case you have not seen, I think there was some confusion or misunderstanding on centos stream 9 readiness. It seem centos team wanted to keep centos stream 8 http://lists.openstack.org/pipermail/openstack-discuss/2021-November/026001.html | 16:04 |
|---|---|---|
| gmann | this can leads to Yoga testing runtime re-updates if we are going back to support centos stream 8 and so does py3.6 | 16:05 |
| gmann | spotz: what your thought on this, I think you checked it with centos team when we updated centos stream 8 to centos stream 9? | 16:06 |
| TheJulia | Honestly, my perception is that it is a good target strive for, but it is still in beta of a being a perpetual beta stream of sorts. It has a higher burden to reach due to breaking changes for some projects. Stating 8 is supported is solid foundation, for the most part. | 16:07 |
| gmann | yeah that was what we discussed in TC PTG and thought of keeping 8 until 9 is ready | 16:08 |
| spotz | gmann Yeah as amoralej said in that email are plan for RDO is for Yoga to be the transitionall release with 8 and 9 stream supported and then moving to 9. I know we've done similiar transitional OS changes in OSA | 16:09 |
| spotz | Doing transitional alllows operators to hold off on the OS upgrade if they need to and for others it allows them to go ahead and do it now | 16:10 |
| fungi | spotz: i guess the question is, if we can only test on one, do we test on centos stream 8 or 9? | 16:17 |
| gmann | fungi: yeah, I just replied on same question in ML. this is same for every distro since starting. | 16:18 |
| gmann | we do not test the distro upgrade in upstream | 16:19 |
| gmann | that is why we do not support two version of same distro in single release as it need greande or any other tool to set up it and more than that maintain it | 16:19 |
| gmann | testing in single version 'latest' one is all good from upstream perspective as minimum | 16:20 |
| spotz | hrm | 16:21 |
| spotz | let me see if amoralej is about and if he can join | 16:21 |
| amoralej | hi | 16:22 |
| spotz | Thanks amoralej. So the question is if we can only test on one OS version which one should we be oon? I wasn't sure if we do more testing within RDO | 16:23 |
| amoralej | mmm | 16:24 |
| gmann | yeah, should we move back to cs 8 or keep cs9 in our testing runtime? | 16:24 |
| amoralej | no, if only one, then cs9 | 16:24 |
| amoralej | i'd say | 16:24 |
| gmann | amoralej: keeping both require us to test both in CI (if we leave the py version things which we do lower and higher only) which we have not done for any distro yet | 16:25 |
| gmann | or any other extra work which we need to check | 16:25 |
| amoralej | as explained, the point is that on every new centos major version, we (RDO) are required by users to provide one "pivot" release supporting both | 16:25 |
| amoralej | in the past i think, we didn't hit issues because of... maybe luck | 16:25 |
| amoralej | my concern is if we find issues with py36 and cs8, i guess projects may be reluctant to fix as it's not supported | 16:26 |
| spotz | And I know OSA has done similar but can't remember the details. Most folkls aren't online today as it's a US holiday so I can't doublecheck | 16:26 |
| spotz | Y'all caught me before heading to the barn:) | 16:27 |
| gmann | amoralej: yeah agree from upgrade point of view , I tried to mention that in http://lists.openstack.org/pipermail/openstack-discuss/2021-November/026016.html | 16:27 |
| gmann | amoralej: yes that is true, I do not think we can force project to accept the py36 related fixes as it is not tested | 16:28 |
| spotz | I'm assuming big differences in Python between the 2 versions, or are we worried abouot the unknown? | 16:28 |
| amoralej | mostly worried about the unknown, I'd say, python provides deprecation policies | 16:29 |
| gmann | amoralej: but I think confusion is that is cs9 is completely ready or not? not just you are more concern on c8->c9 for users | 16:29 |
| amoralej | gmann, since today we have cs9 jobs running in puppet-openstack-integration gates | 16:30 |
| amoralej | so, i'd say it's ready | 16:30 |
| gmann | ok | 16:30 |
| amoralej | and i think it should be the "main" distro for yoga | 16:30 |
| amoralej | i mean, in centos terms | 16:30 |
| gmann | ok, that is what we have in current testing runtime. | 16:30 |
| amoralej | yes, that's why i said, if there is only one, then cs9 | 16:31 |
| gmann | amoralej: not sure how to solve your concern on py36 related fix if any | 16:31 |
| gmann | or we can see if it comes and project want to be litlle flexible or not? | 16:31 |
| amoralej | there may be a chance to have reduced coverage for py3.6 and cs8 ? | 16:31 |
| amoralej | i.e. only unit tests, but no functional ones | 16:32 |
| amoralej | or something like that | 16:32 |
| spotz | fungi^? | 16:32 |
| gmann | amoralej: yes, with cs9 update, we are removing the py36 testing completly from project side | 16:32 |
| gmann | so py36 will not be tested anymore - example - https://review.opendev.org/c/openstack/placement/+/819206 | 16:33 |
| gmann | amoralej: that is what we do when we update the testing runtime but any project can keep it tested as per their opinion/bandwidth. | 16:34 |
| amoralej | so, iiuc, the policy establish the minimal versions of python tested | 16:35 |
| amoralej | but projects can keep others if they have the bandwidth? | 16:35 |
| gmann | amoralej: yeah, they can test more py version or even more distro version say test both cs8 and cs9 based on bandwidth, we do not put bar on max limit of testing | 16:36 |
| gmann | its is just minimum they need to test is this(defined testing runtime) | 16:36 |
| amoralej | ok, then it may become a negotiation with projects in case we hit issues, i guess | 16:36 |
| amoralej | thinking in previous similar situations | 16:37 |
| gmann | yeah | 16:37 |
| amoralej | when going from c6->c7 and c7->c8, we applied similar approach | 16:37 |
| amoralej | of supporting two versions, although in the case of c7->c8, it was py27 and py3.6, and i think both were supported | 16:37 |
| amoralej | we should ask python to stop delivering new versions :) | 16:38 |
| amoralej | that'd make our lives easier | 16:38 |
| gmann | :), or keep moving with them on their latest | 16:38 |
| gmann | amoralej: in devstack we can keep cs8 support for project to tests if they want | 16:39 |
| amoralej | ok, that'd be a good compromise solution probably | 16:39 |
| gmann | unit test jobs for py36 definition are not removed so adding it in project gate is also just add the job to run | 16:39 |
| amoralej | i was thinking in unit tests because of resource usage | 16:40 |
| amoralej | but yeah, keeping support for cs8 in devstack during yoga would be good | 16:41 |
| gmann | sure | 16:41 |
| spotz_ | So that should help alleviate concerns? | 16:41 |
| amoralej | well, it will depends on projects good will up to some point | 16:43 |
| amoralej | but i understand that running both can be too much | 16:43 |
| fungi | should we be providing "pivot releases" for all distros, or is centos "special" in that regard? | 16:43 |
| amoralej | fungi, tbh, i'm not sure how others distros have been managing os upgrades | 16:44 |
| fungi | i thought centos didn't support in-place upgrading between major releases anyway, though i wouldn't be surprised i'm operating on outdated information since most of my experience is with debian derivatives | 16:44 |
| amoralej | actually, what some operators are doing is 1st to upgrade openstack to the pivot release in-place | 16:45 |
| fungi | in past, the theory was that you coupled the distro upgrade and openstack upgrade, rolling out new servers with the new distro and new openstack versions and migrating workloads across | 16:45 |
| amoralej | then, start reinstaling hosts with the new OS release and same openstack release | 16:46 |
| amoralej | that's the usefulness of pivor releases | 16:46 |
| amoralej | so they can have single openstack release with mixed OS release | 16:46 |
| fungi | for distro packaged release upgrades, in debian derivatives they're tied to the distro version which provides them, so you upgrade your openstack along with the rest of your other distro packages at the same time if you're performing an in-place upgrade | 16:47 |
| fungi | (server by server of course, doesn't have to be all of them at once since we support running components from adjacent releases on different servers in the same deployment during upgrades) | 16:48 |
| amoralej | in the past we tried to follow that approach, but some big operators requested us to provide this pivot release | 16:49 |
| amoralej | that's where it comes from | 16:49 |
| amoralej | and i think it's a good approach in terms of making upgrades easier | 16:50 |
| gmann | so in summary: 1. no change in Yoga testing runtime and we move to cs9 and drop py36. 2. we will not put hard stop on cs8 support and we can a) have devstack keep supporting cs8 in Yoga b) it can be negotiated with project to add py36 job if any py36 breaking changes are observed by RDO(or any distro interested in py36) but it depends on project decision because TC testing runtime is minimum things to test not max limit. | 16:50 |
| gmann | next: let's explore the possibility of such pivot release testing support in future. in PTG or so | 16:50 |
| spotz__ | +1 | 16:51 |
| gmann | If you agree, this is something I will put on ML and as most of folks are on leave today, we can wait until next week to conclude it as final decision where more TC memebrs will be there is they have any objection ? | 16:51 |
| amoralej | wfm | 16:51 |
| gmann | cool | 16:52 |
| gmann | amoralej: I will add it in TC agenda for next things to explore if we cna do and make upgrade easy. "next: let's explore the possibility of such pivot release testing support in future. in PTG or so" | 16:52 |
| gmann | may be for next cycle . | 16:52 |
| amoralej | ok, sounds good | 16:53 |
| spotz__ | Ok going to ride:) ping and I’ll respond when I get home | 16:53 |
| gmann | thanks amoralej for joining and discussion. I hope we did not disturb you in your holiday or so | 16:54 |
| gmann | spotz: thanks | 16:54 |
| amoralej | no, no problem | 16:54 |
| fungi | as dtantsur pointed out on the ml, we do test "pivot releases" for ubuntu already, that's exactly how grenade works. but we don't list the platform used by grenade in the pti | 17:10 |
| gmann | fungi: is it? how. grenade test the old node distro version and just upgrade the code on that. it does not install the new distro version on new node and then update code. | 17:12 |
| fungi | grenade installs release n-1 on the "old" distro which n-1 targeted, then we upgrade to release n on the old platform and test the result | 17:12 |
| fungi | victoria, for example, is when we switched to ubuntu-focal nodes, but normal grenade jobs for stable/victoria use ubuntu-bionic (what ussuri needs) | 17:14 |
| fungi | so we test victoria on ubuntu-bionic in grenade | 17:14 |
| fungi | even though the victoria pti doesn't mention bionic | 17:14 |
| gmann | yeah but it does not test release n-1 to release n from old platform to new platform | 17:15 |
| fungi | right, that was my point | 17:16 |
| fungi | so victoria was a "pivot release" for ubuntu since it was technically tested on both bionic and focal | 17:16 |
| fungi | however it's worth noting that we also explicitly claimed the default python3 from bionic (3.6) as supported because it was the default python3 on the most recent centos at the time (8) | 17:18 |
| gmann | humm, if that is enough then we can add centos job also there but I feel 'test release n-1 to release n from old platform to new platform' is something is more reliable to say. | 17:18 |
| gmann | as minimum and leaving all other combination of possible upgrade cases | 17:19 |
| fungi | testing distro upgrades would likely be intractable for at least two reasons i can think of: 1. upgrading the distro release takes a lot of time, and 2. centos doesn't even support in-place upgrades as far as i'm aware | 17:19 |
| gmann | yeah, let's keep exploring on these type of upgrade testing as next and something we can do in upstream | 17:25 |
| fungi | getting yoga grenade going on centos-8-stream nodes seems like the most straightforward way to replicate how we're testing "pivot releases" exist for ubuntu | 17:33 |
| fungi | but separately, this also raises the question of whether the pti should cover grenade | 17:34 |
| gmann | yeah, that is something we need to work on 'as upstream and supported testing in pti can we add some upgrade testing' | 17:34 |
| gmann | which bring an important question on 'need of more people in grenade team'. | 17:35 |
| gmann | or any another upgrade tooling | 17:35 |
| *** amoralej is now known as amoralej|off | 18:20 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!