Sunday, 2022-01-30

gmannfungi: frickler:  boxiang has given the access and these patches are passing and ready to review:
opendevreviewMerged openstack/project-config master: Add openstack-skyline irc channel in access an gerrit bot
fungipip 22.0 was just released. looks like it adds ansi color and a non-ascii progress bar to its stdout15:05
fungihopefully it's smart enough to avoid that when not run under a tty15:05
fungiat least they added a --no-color option to the cli15:06
fungihuh, suddenly ara isn't installed in the system-config-run-eavesdrop job, causing the post-run playbook to fail when it attempts to generate an ara report15:21
fungii don't see anything we merged recently which would have caused this to start happening15:22
fungioh! secondary failure15:24
fungiit's actually failing in the run phase, thanks to new pip dropping python 3.6 support from get-pip.py15:24
funginow to figure out how we've worked around this for earlier instances15:25
Clark[m]I think we've always relief on respecting python requires. But I believe it takes arguments like pip and you can cap it manually15:26
fungithis is what we did for 3.5:
fungiand yeah, the latest version just hard bails with an error on too-old python:
fungier, for the exact task15:28
Clark[m]Oh right because the script itself doesn't run on older python.15:34
opendevreviewJeremy Stanley proposed opendev/system-config master: Use versioned URL for Ubuntu Bionic
fungiClark[m]: yep ^15:38
fungithat's not going to be easy to get merged16:13
fungilooks like new pip is also breaking on downloading virtualenv?
fungiHTML doctype missing or incorrect. Expected <!DOCTYPE html>16:14
fungirecommends --use-deprecated=html5lib or complaining on
fungiyeah, that issue is blowing up16:17
fungiapparently pip now wants any implementation of the simple api to consist exclusively of valid html5 documents16:19
Clark[m]Apache's auto listings aren't valid html5 I guess? That would only affect our wheel caches though?16:32
fungiaha, i think it's our wheel index causing this. it doesn't start with <!DOCTYPE html>16:32
fungiyep, we can prepend it, i'm sure16:32
Clark[m]This shouldn't be fatal for us is what I'm getting at and if an entire pip install breaks because one of two valid indexes breaks that would be a pip bug16:32
Clark[m]fungi: is the entire install failing or does it fallback and ignore the wheel cache?16:33
Clark[m]But also what a silly change16:33
fungiClark[m]: it's not mod_autoindex at fault in this case...
Clark[m]Aha an easy fix at least, but we should file a pip bug if this is a fatal error16:34
opendevreviewJeremy Stanley proposed openstack/project-config master: Make wheel indices valid HTML 5
fungiClark[m]: ^ yep16:39
fungihopefully that one will pass without needing 826968 so we don't need to bypass testing16:41
fungiworst case i'll hand-edit the indices in afs temporarily to get the other merged16:42
Clark[m]We should be able to land a change that uses pip 21 then roll forward from there?16:43
fungialso possible, yeah16:44
Clark[m]I can't get over how silly a change like that is though. It provides no benefit when you just need a list of entities. The bug even notes the built in python parser is sufficient16:44
fungiwell, except that if we try to use to do it, we need 82696816:45
Clark[m]Ya might have to squash two things together I guess16:45
fungialso pressure is mounting in that pip issue to have the "feature" reverted or 22.0 yanked16:46
fungiClark[m]: good news! we only run the linters job when that file is the only one changed16:50
fungiso we should be able to merge it normally, wait for the wheel mirrors to rebuild (or trigger that sooner), and then recheck the python 3.6 fix16:50
Clark[m]Sounds good I can approve shortly. Still bootstrapping my day16:51
fungino rush, and thanks!16:51
fungii need to switch to laundry and cooking anyway16:52
clarkbI have approved the wheel doctype fix17:27
clarkbI need to go eat breakfast and do weekend things though. Not sure how much more help I'll be today17:27
fungino worries, i'll get it merged once the indices have regenerated17:27
opendevreviewMerged openstack/project-config master: Make wheel indices valid HTML 5
fungiokay, probably the most expedient way to update those is to reenqueue the periodic builds for openstack/requirements17:37
fungii don't think i've ever done zuul-client enqueue for a timer-triggered ref before17:38
fungii did it like `sudo zuul-client enqueue-ref --tenant=openstack --pipeline=periodic --project=openstack/requirements --ref=refs/heads/master` though the ref shows up in the status page as 000000017:41
fungiall the builds are error17:41
Clark[m]I think periodic enqueue and dequeue has been very iffy18:04
Clark[m]Does merging a requirements change trigger rebuilds? It did in the past iirc18:04
Clark[m]But now need to afk18:04
fungithe publish-wheel-cache-* jobs only run in periodic, from what i can see18:06
fungiyeah, nothing in post or promote pipelines for openstack/requirements18:08
fungianyway, it looks like the pip maintainers have decided an automatic fallback to the prior library on error is called for, so maybe we'll see a new pip release with that in short order18:08
fungiif we want to override mod_autoindex, it looks like the solution is probably to set SuppressHTMLPreamble in our IndexOptions for those vhosts, and then provide a HeaderName directive pointing to a file which contains <!DOCTYPE html>18:16
opendevreviewJeremy Stanley proposed opendev/system-config master: Override DOCTYPE in wheel cache autoindex
fungifrickler: that ^ should address the problem you observed19:01
fungii manually tested it on mirror.sjc1.vexxhost because we currently have max-servers=0 there19:01
fungii can consider editing the wheel indices in afs for 826969 rather than waiting for the periodic jobs to kick off (which also may fail for related reasons)19:14
fungii guess it's a question of whether we want to wait for pip 22.0.1 to (hopefully) temporarily make this go away so that we can merge these normally, or assume that it may not happen soon enough to ignore for now19:16
fungiif we don't want to take our chances and wait, then i think it's probably going to be updating the indices in afs to make 826969 take effect sooner, bypassing zuul to merge 826974, waiting for that to deploy, and then rechecking 826968 which should hopefully be able to merge on its own after that and unblock us (unless we find yet still more pip 22.0 has broken which these failures are masking19:18
fungifor now)19:18
fungii'll give folks who might be around an hour or so to object to that plan, and then assuming no concerns raised i'll start working through it in hopes we'll have things going again for monday19:19
fungipip 22.0.1 didn't back out the default, only fixed some other bugs, so probably still going to be an issue for us until the above is done. to add a graceful fallback is still awaiting further review19:56
fungiinfra-root: seeing no objections yet, starting in a few minutes i'll bypass zuul to merge 826974 and manually apply 826969 to the files in afs while the first is deploying20:26
opendevreviewMerged opendev/system-config master: Override DOCTYPE in wheel cache autoindex
fungi#status log Bypassed gating to merge and manually applied the results of to files in AFS in order to avoid waiting for daily periodic jobs to trigger.21:10
opendevstatusfungi: finished logging21:10
fungionce 826974 hopefully deploys, i'll recheck 826968 and see if it passes21:11
fungihuh... 826974 does not seem to have resulted in a deployment to the mirror servers21:14
funginevermind, it ran faster than i expected21:16
fungichecking now to see if that's sorted it21:17
fungisystem-config-run jobs aren't bailing straight away this time, which is a good sign21:30
fungiugh, i think the mirrors need a harder apache restart, i'm still seeing the old doctype from autoindex even though the configuration's in place21:36
fungiyeah, confirmed. i'll go through and restart apache on all of them as soon as i'm done with dinner21:38
fungirunning this in a root screen session on bridge: ansible-playbook -f 20 /home/zuul/src/
fungiokay, that wasn't what i needed. it's the same thing the deploy job does, so didn't actually have any effect21:58
fungii just looped over all of them with ssh and did `sudo systemctl restart apache2` which seems to have done the trick22:04
ianwsorry had to run around, back now22:38
ianwwhat a weird mess22:44
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Allow the bootloader element with the default block device
fungiyay! looks like the index workarounds are successfully working around23:06
ianwfungi: thanks for chasing it down!23:08
fungiianw: my pleasure! care to approve 826968 now that it's finally passing?23:14
fungithat should get us able to merge other system-config changes again23:14
ianw++, thanks23:15
fungionce it lands, i'll recheck the change which originally clued me into today's problems (825882). seeing it land should provide some much-needed closure23:22

Generated by 2.17.3 by Marius Gedminas - find it at!