Saturday, 2022-04-30

mnaser286 periodic and 464 periodic buildsets.. that seems.. excessive lol02:20
mnaserah, it runs at 2:01 utc and that was a few minutes ago02:22
fungiyeah, that's mostly the stable branch jobs for every openstack project. it's grown the more branches we leave open since the extended maintenance model went into effect11:16
corvusi'm going to perform a rolling restart+reboot of all zuul servers17:11
corvusthis will include the relaxed ansible restrictions17:11
fungithanks! i'm semi-around now (was in the midst of gardening and away from computers)19:26
fungii assume this is mostly the long wait for graceful stop on executors, ze01-06 seem to have stopped19:28
fungieverything else seems to be on a newer revision19:28
fungi(besides the remaining executors, i mean)19:29
corvusslower than that i'm afraid -- i've only stopped the mergers and 1/2 the executors, so everything still running is on the old version19:38
corvusi'm about to bring the mergers up now19:38
fungiahh, okay, i was going by the versions reported by the components page19:38
corvusmergers are up now, so it looks like dev32 is our target version19:39
fungioh, got it, when i looked everything was on dev3 except the running executors which were still on dev219:39
corvusrebooting executors now19:42
corvuser (01-06) that is19:42
corvus01-06 are up; stopping 07-12 now19:54
corvuswaiting on 09 and 10 to stop19:56
fungilooks like it's down to just 09 and 10 now20:05
fungiand they're down now too21:13
corvusi'll reboot them now21:17
corvusall the executors are running now... i'll cycle zuul01 now21:53
fungilgtm, everything except zuul02 seems to be updated22:29
corvusi'll restart zuul02 now23:10
corvusat this point we should be running zuul 623:11
corvusand having said that, it sure looks like there's a problem that warrants a rollback to 5.x23:14
corvusany of the builds i look at on openstack's tenant seem to have errors relating to ansible on the controller23:15
fungiooh, er yeah, i see all the retry_limit results23:16
corvushere's a sample:
fungido we need to rerun our ansible-manage?23:16
corvusthat should all be in the zuul-executor container23:16
fungi"The conditional check 'hostvars[inventory_hostname]['nodepool']['interface_ip'] |ipv6' failed. The error was: The ipv6 filter requires python's netaddr be installed on the ansible controller"23:16
fungii guess the executor images are incomplete?23:17
corvushrm, i'm a little confused... i'm going to poke around in what's running23:18
fungipip list in the container says netaddr 0.8.0 is present (on ze01 nayway)23:19
corvushrm.  but `/usr/local/lib/zuul/ansible/2.9/bin/pip list|grep netaddr` is empty23:20
corvus(inside the container)23:20
fungiohh, right there's a separate venv for ansible23:21
fungiand i agree netaddr isn't in it23:22
corvuslemme try running that on 5.2.523:22
corvusyes it's in there23:23
fungilooks like /usr/local/lib/zuul/ansible/2.9 is part of the container image, i don't see it mapped into the container from the host fs anyway23:23
corvushere's the requirements difference between 5.2.5 and master:
fungiclearly netaddr was a transitive dep of something there23:25
fungilooks like we dropped boto3, Jinja2, ara, added openshift23:26
corvusopenshift was there already23:26
corvusi'm eyeing ara....23:26
funginevermind, openshift was also there before23:26
fungias was boto323:27
fungiyeah, it was probably dragged in by ara (i can't imagine Jinja2 depending on it anyway)23:27
corvusyeah, a pip install of that version of ara drags in netaddr23:28
corvusokay, i think we should roll back to 5.2.5, then add netaddr, then try again23:29
fungimystery solved i guess, we need netaddr for ipv6 support in ansible, but ansible doesn't declare a dep on it23:29
fungipossible there are other libs in a similar situation, but hopefully not23:29
corvusyeah... one of those "technically not required unless you want to use it" sort of things :/23:29
fungiwe can just roll back the executors, right?23:30
corvusbest to do the whole thing i think23:30
fungiah, okay23:31
corvusi have pulled 5.2.5 on all the hosts and retagged that as "latest"23:35
corvusi'm going to hard-stop the executors now23:36
corvusand restart them and the mergers23:36
corvusbringing up zuul02 now23:38
corvusfungi: hah, i just saw your commit message in the scheduler log :)23:39
fungiand yeah, hopefully not duplicating effort23:39
corvusnope, i had my hands full, thanks :)23:40
funginp, i feel like i'm waving my pompoms from the sidelines here anyway23:41
corvusit'll take a bit for zuul02 to come up; once that's done we can restart zuul01 and the rollback should be done.  i don't think we need to delete state for this rollback, but it'll be good to run a change through its paces23:41
fungiyeah, i was wondering if there were any zk schema changes we needed to worry about, but figured it's not been that long since we last updated23:42

Generated by 2.17.3 by Marius Gedminas - find it at!