| tkajinam | hi. seems skip level grenade job is broken because of the recent tooz release which contains https://review.opendev.org/c/openstack/tooz/+/891355 | 08:18 |
|---|---|---|
| tkajinam | the problem seems to be that etcd3 in stable/2023.1 is 3.3 which does not provide /v3 endpoint and it's not updated to 3.4 or later during upgrade process. | 08:19 |
| tkajinam | wondering if we can(or should) update etcd during the process | 08:19 |
| tkajinam | https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-always&project=openstack/nova | 08:19 |
| tkajinam | examples ^^^ | 08:19 |
| opendevreview | Takashi Kajinami proposed openstack/devstack stable/2023.1: Stable-only: Force v1beta etcd api https://review.opendev.org/c/openstack/devstack/+/901221 | 08:27 |
| frickler | tkajinam: IMO upgrading etcd in grenade would seem sensible. I'm not sure about possible operator impact though | 08:29 |
| opendevreview | Takashi Kajinami proposed openstack/devstack stable/2023.1: Stable-only: Force v1beta etcd api https://review.opendev.org/c/openstack/devstack/+/901223 | 08:29 |
| tkajinam | probably this may be an alternative solution ^^^ | 08:30 |
| frickler | that may fix the testing, but doesn't seem feasible for existing deployments, at least I wouldn't know how to communicate this properly to affected parties | 08:33 |
| tkajinam | frickler, we have an interface to override api version so operators need to set the one according to the etcd version they have. When they have etcd 3.3 then they may start with v1alpha, then upgrade etcd to 3.4 and then switch to v1beta. then upgrade etcd to 3.5 and switch to v1 | 08:33 |
| tkajinam | we can consider detecting the available versions automatically. though. we discussed it a while ago but no progress was made. I can take time to look into it later. | 08:41 |
| opendevreview | Takashi Kajinami proposed openstack/devstack stable/2023.1: Stable-only: Force v1beta etcd api https://review.opendev.org/c/openstack/devstack/+/901221 | 09:10 |
| -opendevstatus- NOTICE: Gerrit will be unavailable for a short time starting at 15:30 UTC as it is upgraded to the 3.8 release. https://lists.opendev.org/archives/list/service-announce@lists.opendev.org/thread/XT26HFG2FOZL3UHZVLXCCANDZ3TJZM7Q/ | 14:07 | |
| dansmith | tkajinam: you think that's only stable related? I see it on master too: https://zuul.opendev.org/t/openstack/build/f3154ea9c84143259e4d325cb96f384a | 15:07 |
| dansmith | I see I'm late to the party from reading -discuss | 15:20 |
| dansmith | damn timezones :) | 15:20 |
| tkajinam | dansmith, the problem is caused by the config file generated at the source version so we need it only in stable/2023.1 | 15:29 |
| dansmith | tkajinam: okay perhaps I was conflating that and the etcd thing, but the latter is definitely seen on master | 15:30 |
| dansmith | (i.e. non-grenade things) | 15:30 |
| tkajinam | https://bugs.launchpad.net/python-tooz/+bug/2043810 | 15:30 |
| tkajinam | ah, wait | 15:30 |
| tkajinam | hmm. that specific failure seems to be caused by | 15:31 |
| tkajinam | 2023-11-17 14:56:36.287910 | controller | The conflict is caused by: | 15:31 |
| tkajinam | 2023-11-17 14:56:36.287947 | controller | etcd3 0.12.0 depends on grpcio>=1.27.1 | 15:31 |
| tkajinam | 2023-11-17 14:56:36.287980 | controller | The user requested (constraint) grpcio===1.58.0 | 15:31 |
| tkajinam | but that's not related to our recent discussion. maybe it's a new thing, but the error looks quite strange | 15:32 |
| tkajinam | grpcio===1.58.0 does not conflict with grpcio>=1.27.1 | 15:32 |
| dansmith | wonderful | 15:33 |
| tkajinam | maybe we have to drop etcd3 driver. we deprecated it a while ago because python-etcd3 is unmaintained. if etcd3 is causing strange thing then it's the time. | 15:34 |
| tkajinam | though I may attempt to find what that strange error is | 15:34 |
| dansmith | I guess I'm surprised everyone isn't hitting this ^ because it looks fairly deterministic.. is it worth a recheck or no? | 15:36 |
| tkajinam | I'm trying to check whether the error consistently appears. though I can't check zuul/gerrit now because of server errors, likely caused by upgrade going on now | 15:38 |
| tkajinam | I'd suggest recheck once | 15:38 |
| dansmith | yup, pokay | 15:39 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Add plugin specific sriov vnic port parameter https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/900239 | 16:33 |
| -opendevstatus- NOTICE: The Gerrit upgrade is complete, however we have Zuul offline in parallel for a schema migration, so any events occurring during this time will be lost (requiring a recheck or similar to trigger jobs once it returns to service); we'll update again once this is complete. | 16:34 | |
| -opendevstatus- NOTICE: Zuul is fully back in service now, but any events occurring prior to 17:05 UTC may need a recheck to trigger jobs. | 17:13 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!