| *** mhen_ is now known as mhen | 01:45 | |
| kopecmartin | gmann: do we have a plan for grenade's jobs to use ubuntu 24.04? the suggested goal for watcher is to move jobs to that ubuntu version - https://etherpad.opendev.org/p/oct2024-ptg-watcher | 06:09 |
|---|---|---|
| frickler | kopecmartin: I think you would have to opt into that by overriding nodesets. 24.04 was only added as "official" runtime for 2025.1, so you cannot expect older stable branches to support it in general | 07:00 |
| kopecmartin | frickler: of course, i think that the question is more about having a job that tests upgrade from stable 22.04 to master 24.04 | 07:13 |
| frickler | kopecmartin: I don't think we ever did a job that upgrades the base OS? pretty certainly not within grenade, that sounds out of scope to me | 08:02 |
| kopecmartin | right, i misunderstood and we don't update os version for stable releases, right? | 08:06 |
| frickler | yes, even kolla doesn't do that. so in general grenade could only test upgrades on 24.04 from 2025.1 on to newer branches | 08:10 |
| kopecmartin | makes sense, thanks! | 08:14 |
| frickler | clarkb: fungi: I vaguely remember there were some reasons some time ago to stick to the original cirros download site, but I don't remember the details, please add a note if you do https://review.opendev.org/c/openstack/devstack/+/930647 | 10:56 |
| opendevreview | Takashi Kajinami proposed openstack/bashate master: Drop unnecessary 'x' bit from doc config file https://review.opendev.org/c/openstack/bashate/+/932868 | 11:21 |
| fungi | i don't remember ever hearing a reason for that | 11:54 |
| frickler | ok, change is approved now anyway, so let's just watch whether the caching in the images will adapt properly | 12:01 |
| opendevreview | Merged openstack/tempest master: Test python 3.12 https://review.opendev.org/c/openstack/tempest/+/931104 | 13:01 |
| gmann | dansmith: kopecmartin: can you review this https://review.opendev.org/c/openstack/grenade/+/932767 | 14:00 |
| dansmith | gmann: got it | 14:00 |
| kopecmartin | done | 14:01 |
| gmann | thanks | 14:01 |
| gmann | kopecmartin: for greande job, it will still run/test the stable/2024.2 -> master on ubuntu 22.04 because that is base version for stable/2024.2 | 14:02 |
| gmann | kopecmartin: devstack/tempet jobs will be running on 24.04 | 14:02 |
| tkajinam | gmann, the problem in GLOBAL_EVNE is seen in source version. because the switch in base job enabled it in both source and target we have to fix the script executed in source version so that it works with GLOBAL_VENV=True | 14:12 |
| tkajinam | If you say that is unexpected switch then we have to fix grenade somehow to upgrade source with GLOBAL_VENV=False to dest with GLOBAL_VENV=True | 14:12 |
| gmann | tkajinam: do you have failure link for stable branch greande? | 14:13 |
| tkajinam | gmann, the issue is not seen in stable but in master | 14:14 |
| gmann | tkajinam: if any script running as part of old node setup or new node upgrade script, those are the only one impacted | 14:14 |
| tkajinam | because ugrande in master upgrade 2024.2 to master we have to make sure both branches work with GLOBAL_VENV=True | 14:14 |
| gmann | tkajinam: during old node setup? | 14:14 |
| gmann | tkajinam: let me check but it should be required in stable/2024.1 branch | 14:14 |
| tkajinam | https://zuul.opendev.org/t/openstack/build/d1bdf9480d35462b98ff5153671450df See this one for example | 14:15 |
| tkajinam | https://3cca4824cda4efd66a4a-d97d31900addfa62359a210d03244371.ssl.cf1.rackcdn.com/930999/5/check/manila-grenade-skip-level-always/d1bdf94/controller/logs/grenade.sh_log.txt | 14:15 |
| tkajinam | it failed at the stage to create resources in the source version, before executing upgrade | 14:15 |
| gmann | tkajinam: "/opt/stack/new/manila/devstack/upgrade/resources.sh: line 90: manila: command not found" | 14:16 |
| gmann | tkajinam: error is in new node so fixing master upgrade/resources.sh should make grenade green with GLOBAL_VENV | 14:16 |
| gmann | if we are at new node setup then it means old node setup is all done | 14:17 |
| tkajinam | are you saying the script is executed AFTER the deployment is upgraded to master ? | 14:17 |
| tkajinam | https://review.opendev.org/c/openstack/manila/+/932910 indicates only after we fix 2024.1/2 the grenade pass | 14:18 |
| gmann | tkajinam: flow for grenade is 1. install old devstack on old node 2. upgrade it to new node (new code which run upgrade + resource.sh + upgrade.sh) | 14:19 |
| tkajinam | doesn't it run resource.sh before upgrade to check whether the resources are valid after upgrade ? | 14:20 |
| tkajinam | I though that's what should be/is tested in grenade | 14:20 |
| gmann | tkajinam: did you try it without stable branch change and we can see where it fail for old node setup | 14:20 |
| tkajinam | gmann, I initially try https://review.opendev.org/c/openstack/manila/+/932665/ directly on master but it failed so I determined that we need that fix in stable branches | 14:20 |
| tkajinam | https://zuul.opendev.org/t/openstack/build/7be3c6fb19094c52ae19abacd0ed67b8 | 14:21 |
| gmann | tkajinam: yes but from the master branch (new node) not from the stable branch (old node) | 14:21 |
| tkajinam | this is what I get when I proposed that fix patch directly to master | 14:21 |
| gmann | tkajinam: let me check, but it is confusing with lot os changes, my suggestion is to fix it i single change and w can see the whole pic. anyways | 14:22 |
| opendevreview | Merged openstack/devstack master: Bump cirros version to 0.6.3 https://review.opendev.org/c/openstack/devstack/+/930647 | 14:22 |
| tkajinam | I don't know if I can do it in single change... unless I break workflow and do port-foward | 14:22 |
| tkajinam | technically we can do it but I've seen many cases where port forward didn't work and people didn't merge master changes after they merge fixes for stable branches | 14:23 |
| gmann | tkajinam: is this enbale the GLOBAL_VENV ? https://review.opendev.org/c/openstack/manila/+/932665/3 | 14:23 |
| tkajinam | and mess up whole backport chain as decleared by cherry-picked from commit | 14:23 |
| gmann | that is why I am all confused with this disable/enable/fix changes as they are split into all different one | 14:23 |
| tkajinam | gmann, this one is; https://review.opendev.org/c/openstack/manila/+/932910 | 14:23 |
| clarkb | I don't undersatnd the forward port concern | 14:24 |
| tkajinam | if people merge things timely and get no problems in the middle everything works | 14:24 |
| clarkb | we've used it many times in the past successfully and it allows you to confirm things are working as you go without extra changes to enable/disable stuff | 14:24 |
| gmann | tkajinam: I am asking where you saw the failure when GLOBAL_VENV is enabled and master change did not fix it | 14:24 |
| tkajinam | gmann, https://zuul.opendev.org/t/openstack/build/7be3c6fb19094c52ae19abacd0ed67b8 | 14:25 |
| gmann | tkajinam: if fix is up we do not need to disable it. it take same time/thing to merge disable it or merging the fix | 14:25 |
| gmann | if fix is taking time and not known then I will say disable it to unblock the things | 14:25 |
| tkajinam | https://review.opendev.org/c/openstack/manila/+/932665/1 is directly proposed on master without GLOBAL_VENV=false | 14:25 |
| tkajinam | which I tried first but failed | 14:25 |
| tkajinam | maybe I'm too much opinionated the tons of mess we saw in TripleO | 14:26 |
| tkajinam | *by* the tons of ... | 14:26 |
| clarkb | I think those were tripleo problems not gerrit zuul openstack problems | 14:27 |
| gmann | tkajinam: if you just fix directly on master without disable it, it can give clear pic when and where all we need fix. with disable flag I am finding hard if things running in venv or not | 14:27 |
| tkajinam | maybe it won't be a problem with active reviewers and enough care to patches | 14:28 |
| gmann | tkajinam: designate, manila core are active and if you have fix up then it is same thing | 14:28 |
| johnsom | o/ | 14:29 |
| gmann | tkajinam: is this still with GLOBAL_VENV enable or disable? https://review.opendev.org/c/openstack/manila/+/932665/1 | 14:29 |
| tkajinam | gmann, See https://review.opendev.org/c/openstack/manila/+/932913 | 14:29 |
| tkajinam | gmann, the first one has NO GLOBAL_VENV OVERRIDE | 14:30 |
| tkajinam | so it should run with GLOBAL_VENV=True | 14:30 |
| gmann | tkajinam: I am referring the failure with GLOBAL_VENV enable and it does not pass with master only fix, the link you gave me https://zuul.opendev.org/t/openstack/build/7be3c6fb19094c52ae19abacd0ed67b8 | 14:30 |
| tkajinam | gmann, I don't know how I can explain it better, but the result is from the pathc set 1 | 14:31 |
| gmann | tkajinam: ok, let's wait for this and see https://review.opendev.org/c/openstack/manila/+/932913 | 14:31 |
| tkajinam | and patch set 1 is directly submitted on master, without 932671 which I added later since patch set 2 | 14:32 |
| gmann | tkajinam: yeah PS1 still depends on the GLOBAL_VENV disabled right | 14:32 |
| tkajinam | no | 14:32 |
| gmann | tkajinam: ok | 14:32 |
| tkajinam | PS1 does not | 14:32 |
| tkajinam | PS2 or PS3 does | 14:32 |
| tkajinam | gerrit is sometimes too tricky | 14:32 |
| tkajinam | designate already merged the first patch to disable GLOBAL_VENV in master so I think we can start merging https://review.opendev.org/q/I4d9c26e25851be0a955f4172a530eff266a89331 | 14:34 |
| tkajinam | and then https://review.opendev.org/c/openstack/designate/+/932907 | 14:34 |
| tkajinam | I also created the change to demonstrate the designate fix in master does not fix grenade https://review.opendev.org/c/openstack/designate/+/932914 | 14:36 |
| gmann | tkajinam: that is the issue I am saying, your fix still disable the GLOBAL_VNEV and not sure if that is complete fix or not https://review.opendev.org/c/openstack/designate/+/932688 | 14:38 |
| tkajinam | gmann, there is a separate change to enable GLOBAL_VENV which can be merged once the fix lands stable branches https://review.opendev.org/c/openstack/designate/+/932907 | 14:39 |
| gmann | tkajinam: either you should have enable it in the fix to see where we are at and if fix fixes the issue or not | 14:39 |
| tkajinam | 932907 enables GLOBAL_VENV and depends on the backport to 2024.2 it should prove the fix fixes the problem. | 14:41 |
| tkajinam | it you think what I'm doing does not make sense at all then please feel free to abandon these and take them over | 14:42 |
| johnsom | Catching up on what happened overnight, it would be great if these patches were in a chain to sequence them | 14:42 |
| gmann | or just a fix up without disable/enable to see where all we need the fix and test fix is actually making grenade green or not | 14:43 |
| gmann | *if actually | 14:43 |
| gmann | tkajinam: anyways, I will check 932914 and 932913 but I am 100% sure we should not require any change in stable/2024.1 at least which has nothing to do with GLOBAL_VENV enabled from current master onwards | 14:44 |
| tkajinam | I've been saying that https://review.opendev.org/c/openstack/designate/+/932907 and https://review.opendev.org/c/openstack/manila/+/932910 already confirm we have the enough and valid fixes | 14:44 |
| tkajinam | but probably seeing the actual failures from these DNM patches may give better view, if my understanding is correct. | 14:45 |
| gmann | tkajinam: I am saying stable/2024.1 backport should not be required and might not stable/2024.2 also | 14:45 |
| tkajinam | 2024.1 backport is required to fix SLURP job | 14:45 |
| tkajinam | we need to fix the issue in both 2024.1 and 2024.2 | 14:46 |
| gmann | humm, let me check on those DNM changes once they finish the gate | 14:46 |
| tkajinam | assuming we need to fix the source versions (which are still subject to discussions) | 14:46 |
| opendevreview | Merged openstack/bashate master: Drop unnecessary 'x' bit from doc config file https://review.opendev.org/c/openstack/bashate/+/932868 | 14:46 |
| opendevreview | Jason Grosso proposed openstack/tempest master: [WIP]test_boot_nova_vm_unadressed_port https://review.opendev.org/c/openstack/tempest/+/932919 | 15:41 |
| opendevreview | Merged openstack/grenade master: Run nova-status upgrade always https://review.opendev.org/c/openstack/grenade/+/932767 | 15:43 |
| gmann | tkajinam: for manila, I saw no. as issue is in installing the manilaclient we need the fix on stable branhces also as manilaclient installation happen on old node also. I changed my vote to +1 on your fixes. | 17:04 |
| gmann | same thing for designate which is configure_designatedashboard() | 17:04 |
| gmann | basically here issue was on devstack scripts side not on grenade scripts but error came from greande scripts | 17:07 |
| gmann | that was the confusing part | 17:07 |
| gmann | devstack fixes are needed to be backported and merge in reverse direction (oldest stable branch fix first) to make grenade job green | 17:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!