Wednesday, 2021-08-04

Clark[m]ianw ^ you might be interested in that00:00
Clark[m]Might be interesting to see if repasting the same data produces the same result00:05
Clark[m]A trivial paste of "Test" does not have this problem00:05
Clark[m]I need to get back to dinner though00:05
ianwinteresting, i wonder if it's a diff thing and it's trying to format it?00:07
ianwAug  4 00:10:40 paste01 docker-lodgeit[647]: [pid: 13|app: 0|req: 325241/325241] () {42 vars in 957 bytes} [Wed Aug  4 00:10:40 2021] GET /show/807865/ => generated 11016 bytes in 11 msecs (HTTP/1.1 200) 2 headers in 82 bytes (1 switches on core 0)00:11
ianwit does not show an error00:11
ianwi dunno, i just pasted the raw content and it seems to show00:15
fungishould be able to query both pastes from the db and compare, maybe there's something "extra" in the one that's breaking and doesn't show in the raw view for some reason but causes the prettified version to choke00:35
fungilike the paste type or something00:36
opendevreviewMerged opendev/system-config master: Update the python docker images again
ianwselect language from pastes where paste_id=807865;00:55
ianw| language |00:55
ianw| diff     |00:55
ianwwhich is interesting because "diff" doesn't appear to be a language type in the list00:56
ianwit does, however, seem to remember your last type.  i wonder if timburke has an old cookie that remembers a language type it now doesn't know about00:57
ianwalthough diff highlighting seems useful, maybe it's a missing package00:57
ianwindeed, putting it in a "unified diff" creates this issue01:01
ianwmust trigger
ianw ... i guess that must not be failing, but not returning anything either01:02
ianwfungi: heh, you touched it last; under the official rules of open source maintainership i believe that means you own it :)
fungiand yeah, i've used the diff mode in lodgeit many times. i wonder what broke. does it rely on pygments for the diff colorization?01:16
fungiseems so01:17
ianwsort of, but it has it's own parsing01:17
ianwi think is saying01:18
ianw"if our parser can't handle it, run it through basic pygments"01:19
ianwwith # TODO: we do not yet support diffs made by GNU Diff!01:19
ianwi presume that means context diff format01:19
ianwAug  4 00:55:58 paste01 docker-lodgeit[647]:   File "/usr/local/lib/python3.7/site-packages/lodgeit/lib/", line 131, in _parse_udiff01:21
ianwAug  4 00:55:58 paste01 docker-lodgeit[647]:     line =
ianwAug  4 00:55:58 paste01 docker-lodgeit[647]: AttributeError: 'list_iterator' object has no attribute 'next'01:21
ianwthat actually *does* seem like the bit you touched :)01:21
ianwthat probably needs to be next() now01:23
fungibecause of newer python i guess?01:24
opendevreviewIan Wienand proposed opendev/lodgeit master: diff parser: update calls to next()
Clark[m]There was an error promoting python-base:3.9 this time around. I'm less concerned about that due to the previous promotes and not much if anything uses 3.9. also it may have failed in a way that actually promoted anyway? I'm not sure about that01:27
fungiyeah, i guess we were running it with 2.7 previously01:27
Clark[m] looks new to me01:28
Clark[m]I suspect we're good with base images01:29
Clark[m]ianw: that fix to lodgeit looks about right to me. I think next() works on python2 as well01:30
fungiianw: i have a feeling that may need deeper changes, "next" is also a function parameter01:31
ianwheh of course it is01:32
Clark[m]fungi: oh are we overriding a built in?01:32
fungiDiffRenderer._highlight_line has a "next" parameter01:32
fungiClark[m]: shadowing it with a variable, yeah01:32
ianwi think we lucked out that it doesn't use .next() in there, but i can update it01:34
opendevreviewIan Wienand proposed opendev/lodgeit master: diff parser : update next variable
fungithough only in the _highlight_line() method so it may not need more, yeah01:35
fungii just finished looking for other cases and that was the only place, which didn't need the next() builtin treatment01:36
fungiand yeah, it uses nextline for the same thing in other places, so that's a good cleanup anyway, thanks!01:37
opendevreviewIan Wienand proposed openstack/project-config master: Add CentOS 8 Stream wheel publish jobs
opendevreviewIan Wienand proposed openstack/project-config master: Add CentOS 8 Stream wheel publish jobs
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
opendevreviewxinliang proposed openstack/diskimage-builder master: Introduce openEuler distro
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
opendevreviewIan Wienand proposed openstack/project-config master: Add CentOS 8 Stream wheel publish jobs
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
opendevreviewMerged opendev/lodgeit master: diff parser: update calls to next()
opendevreviewIan Wienand proposed opendev/ master: Remove review-test
opendevreviewMerged opendev/lodgeit master: diff parser : update next variable
opendevreviewIan Wienand proposed opendev/ master: Remove records
opendevreviewIan Wienand proposed opendev/system-config master: borg-backup-server: log prune output to file
ianwi'm running the backup prune now04:48
ianw#status cleaned up review-test and review01 servers and related volumes, databases, etc.04:48
opendevstatusianw: unknown command04:48
ianw#status log cleaned up review-test and review01 servers and related volumes, databases, etc.04:48
opendevstatusianw: finished logging04:49
ianwtimburke_ fungi clarkb: i pulled those fixes and restarted paste and now shows up as a diff.  it lives to fight another day ...04:55
*** ykarel|away is now known as ykarel05:08
opendevreviewIan Wienand proposed opendev/system-config master: lodgeit: disable getRecent API endpoint
*** marios is now known as marios|ruck06:06
*** jpena|off is now known as jpena07:01
xinliangianw: please help to review these patches when you have time.
xinliangplus this one
*** rpittau|afk is now known as rpittau07:13
*** ksambor|away is now known as ksambor08:14
*** ykarel is now known as ykarel|lunch08:43
*** ykarel|lunch is now known as ykarel09:23
*** diablo_rojo is now known as Guest328109:49
*** Guest3281 is now known as diablo_rojo10:09
*** dviroel|out is now known as dviroel11:16
*** jpena is now known as jpena|lunch11:26
*** marios is now known as marios|ruck11:27
*** jpena|lunch is now known as jpena12:32
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule
opendevreviewSorin Sbârnea proposed openstack/project-config master: Allow elastic-recheck cores to create branches
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container
opendevreviewJeremy Stanley proposed openstack/project-config master: Reintroduce x/tap-as-a-service shared ACL
fungii need to pop out to some errands, but should be back by ~16:00 utc14:54
clarkbfungi: when you get back should be a safe one to approve if it passes your review14:56
clarkbI'm going to take a look at modifying the known hosts fixup to automatically add in the gitea ssh host keys14:57
clarkbI think that won't work because ansible only knows about the port 22 host keys14:58
clarkbBut I'll look at the facts cache and see if I can say that for certain14:58
opendevreviewMerged openstack/project-config master: Reintroduce x/tap-as-a-service shared ACL
*** jpena is now known as jpena|off15:05
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule
*** ykarel is now known as ykarel|away15:33
yoctozeptoconfig-core: morning! please consider merging :-)15:55
*** marios|ruck is now known as marios|out16:00
opendevreviewSorin Sbârnea proposed zuul/zuul-jobs master: Include podman installation with molecule
*** rpittau is now known as rpittau|afk16:14
opendevreviewMerged openstack/project-config master: Allow kolla cores to edit kolla hashtags
fungiclarkb: rackspace has opened a ticket saying they need to offline migrate elasticsearch07, and that they're planning to reboot it on saturday "at or after 21:52 utc"16:29
fungithough we can reboot it ourselves any time before then to start the migration if we prefer16:29
clarkbfungi: ok, in theory ES will handle that gracefully16:29
fungii assume we want to initiate the reboot ourselves, like todayish?16:30
clarkbwe can also do that. You want to check the cluster status and if it is green then we can reboot16:30
fungido we need to double-check all the shards are sane first?16:30
clarkbif it isn't green we want to figure that out and get it green then we can reboot16:30
fungiyeah, that16:30
clarkbI'm deep into zuul sos reviews right now, but I can check that later today if we want. Or happy to help someone else check it16:30
clarkbthere is a base cluster status rest api request you can make and it will tell you red, yellow, green. If not green then probably one of the cluster members has died, you just ask system for which isn't running and restart it16:31
clarkbis the usual thing16:31
fungi has a lot of detail16:31
fungibut says all 70 shards are successful and none are failed16:31
fungii expect that's the same as "green"16:32
clarkbfungi: shows green so ya16:33
clarkbin that case it should be completely safe to reboot es07. Just double check that elasticsearch has started again post reboot and start it if not (I can't remember if we allow it to auto start)16:33
fungiahh, perfect16:33
fungiso i can just reboot the server, no need to gracefully stop the service on it16:33
clarkbyou'll see the cluster degrade to yellow while 07 is out of the cluster and then when it rejoins 07 and rebalances16:33
clarkbfungi: shouldn't need to. systemd will ask elasticsearch to stop16:34
fungias far as it doesn't take so long that systemd gets impatient anyway16:34
opendevreviewMerged opendev/system-config master: Use the gitea api in the gitea renaming playbook
clarkbI think there are a few more changes that have hit the node request orphaning issue. I'll dequeue and enqueue them now17:04
clarkbbecause I'm paranoid I'm going to rereview my proposed user cleanups list now. But then if I don't find anything scary in it will proceed with retiring this batch. Final cleanup can happen in a few weeks as usual17:09
fungisounds good17:11
fungii did a test reboot of elasticsearch07 and the server's been up for half an hour now, but the cluster status is still "yellow" (12 unassigned shards). it looks like the service did not start on boot, and systemd reports elasticsearch.service is disabled, inactive and dead17:17
fungido we normally manually start these after reboot?17:17
clarkbfungi: ya I wasn't sure how we had those set up which is why I emntioned we needed to check17:17
clarkbit wouldn't surprise me if that was intentional in order to have control over the order of startups and cluster bring up17:18
clarkbnote it will still be yellow after starting 07's service again as the cluster has to rebalance itself17:18
fungithat's what i thought was happening at first and after 30 minutes of yellow i decided to look closer17:20
fungii manually started the service unit and now it claims to be running17:20
fungiand now there's no unassigned shards and 4 being initialized17:21
fungiand the count is falling17:23
fungii assume once that reaches 0 it'll be back to green17:23
fungithen i'll do another outage for 07 to actually perform the migration step17:23
clarkbyup once initializing is done it should be green17:24
clarkbyou may see migrating shards after that but they don't effect cluster health that is just an optimization to balance disk use17:24
fungiokay, cluster is green again, now i'm taking 07 down for the offline host migration17:58
clarkbsounds good thanks18:02
*** ricolin_ is now known as ricolin18:02
clarkbI've reviewed my list of account retirements and decided to update one of them which I corss checked with fungi and we agree both are fine but I think one is more likely to be correct if this user shows up 8 years later18:03
clarkbI'm going to take a break here and will run the retirment script against this list after lunch (I'm worried lunch will be late if I run the script first :) )18:03
clarkb803039 maybe hit by the zuul node request issue too. But I suspect we're now getting close enough to potentially restarting zuul taht just waiting might be quicker for that one?18:07
fungiyeah, 803407,2 still has a couple of jobs queued18:14
clarkbAll of the previous changes for the improvements to gitea project management have landed is definitely the biggest in scope and probably has the most potential for breakage but it is tested18:20
clarkbcorvus: mordred: ^ you may be interested in that one in particular since you were involved in the original authorship there18:20
fungiclarkb: is the queued state for the two remaining jobs on 803407,2 due to the problem it's intended to solve? irony if so...18:30
clarkbfungi: could be18:31
clarkbfungi: zuul doesn't require clean check so maybe if the unitest jobs pass we approve it and let it try again tehre?18:31
fungiassuming the unit test jobs complete i'll go ahead and approve it, which will supersede in the gate anyway, yeah18:31
fungii finished reviewing it a few minutes ago and was just waiting to approve it since the check jobs looked close to completing, but now that it still hasn't gotten nodes assigned to a couple of them i expect that's not going to happen without some disruption anyway18:33
fungioh, though the py36 job just failed for it18:33
*** diablo_rojo is now known as Guest331618:34
fungiand now the py38 job has as well18:34
*** Guest3316 is now known as diablo_rojo18:35
clarkbI'm going to sort out lunch now but back in a bit to do user retirements and help wit hzuul if I can18:48
johnsomHere is an odd one. undercloud/var/log/containers/octavia. The worker.log file. The raw link pulls up the log fine, but the "pretty" (and linkable) link throws a "This logfile could not be found" error.18:52
fungialso very slow to browse... large log somewhere in that manifest maybe?19:01
johnsomProbably, these tripleo jobs seem to have a ton of stuff captured19:01
fungiyeah, might just be slow from sheer file count19:01
fungiit's certainly reminding me how much i need a new workstation19:02
johnsomEven on mine the fans spin up19:03
fungijohnsom: *is* loading for me at least, albeit... ...very... ...slowly...19:03
johnsomIf I don't use the "raw" link I get: Network Error (Unable to fetch URL, check your network connectivity, browser plugins, ad-blockers, or try to refresh this page)
johnsomBut clicking that link also loads it fine.19:04
fungicould be your browser (or a plugin/extension) blocking the fetch?19:04
fungior maybe you need bigger fans19:04
johnsomMaybe I need a bigger fan19:05
fungimy younger brother is all about evaporative liquid cooling these days19:05
johnsomThe octavia.log right above it opens fine.19:05
johnsomI have not gone that far (yet). So far I have stuck to air cooling with massive (quiet) heatsinks19:06
fungianother possibility is that ovh's swift endpoint is randomly failing/refusing requests19:06
fungibut since i'm able to load the htmlified version of the worker.log i don't think it's failing for everyone19:07
fungielasticsearch cluster is back to "green" status following the elasticsearch07 offline migration, so i'm going to close that ticket19:09
fungi#status log Offline migrated in order to mitigate an unplanned service outage19:10
opendevstatusfungi: finished logging19:10
johnsomDeveloper tools give more info: Access to XMLHttpRequest at '' from origin '' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.19:11
fungiinteresting. we upload all those files to swift with the cors headers set19:12
fungimy browser would also refuse to pull the file if it wasn't included19:13
fungiso seems like it may be an ovh problem19:13
johnsomHmm, now it is loading for me.19:13
fungi"eventually consistent"19:13
fungilet me introduce you to openstack...19:14
johnsomOh, we go way back....19:14
johnsomIt's probably DNS or the load balancer. GRIN19:14
fungiyeah, who do i complain to about that again?19:14
johnsomNot sure, let me get back to you.19:15
* johnsom thinks it's time for lunch19:15
timburke_there may also be a bug in Swift's handling of non-2xx CORS responses. it wouldn't surprise me if we didn't gracefully handle CORS in 503 or ratelimited responses, for example19:15
johnsomYeah, so playing around a bit, I got a similar error on job-output.json.gz with dev tools loaded up. The response was a 404 without the CORS header19:21
fungiyeah my reading of these tea leaves is that ovh's swift is maybe-eventually-but-not-presently consistent19:24
timburke_interesting -- we've actually got a test for that ( but maybe the LB is dropping the Access-Control-Allow-Origin header for non-2xx responses. i'd check in with my old OVH contacts, but they've moved on to other things :-(19:29
fungiyeah, swift ops moving on to other things also possibly related to running outdated swift ;)19:35
fungiwhich is cause and which is effect could be left open to interpretation19:36
clarkbuser retirements are running now20:19
clarkbI'll get the log posted when done20:19
*** dviroel is now known as dviroel|out20:38
clarkbok one failed becausae I had network blip and need to rerun for that. Otherwise two things I noticed as I paid attention to it going by. One of the accounts is frickler's I think and the other was for Mellanox Cinder CI20:42
clarkbin the case of the mellanox cinder ci account it doesn't seem to have left any comments in years so I'm fairly certain that one is fine but heads up if mellanox or cinder wonder about that20:42
clarkbthe one that failed has been rerun and the log is in the typical place. Now to pm frickler with details in case we need to revert that one20:45
clarkbok thats all done. I think we are done for now and can followup with the second pass of cleanups on these accounts if no one complains20:49
*** timburke_ is now known as timburke20:57
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup unused puppet modules from modules.env
clarkbwe're making progress ^ :)21:10
clarkbianw: responded to the two different sshds does indeed confuse things21:23
ianw1007G  299G  709G  30% /opt/backups-20201022:18
ianwprune worked well22:18
opendevreviewIan Wienand proposed openstack/project-config master: Add CentOS 8 Stream wheel publish jobs
ianwahh indeed i forgot about the 222 thing22:24
ianwspeaking of ssh, the last thing I have on the cleanup list now is "decide on sshfp records"22:35
ianwi'm mostly of the mind to just keep it as is, with no sshfp records for review.  bridge / admins want the sshfp records for the port 22 ssh.  normal people want it for gerrit ssh.  i don't want to access it via a separate name22:37
opendevreviewMerged opendev/ master: Remove review-test
opendevreviewMerged opendev/ master: Remove records
ianwZuul encountered a syntax error while parsing its configuration in the22:40
ianwrepo openstack/project-config on branch master. : line 8293 : Unknown projects: x/tap-as-a-service22:40
ianwdo we know about that?22:40
fungiianw: that got renamed over the weekend, i bet that requires a full-reconfigure22:42
fungibut we're angling to restart zuul soon anyway for the reissued node request bug22:42
clarkbfungi: ianw that issue won't be fixed by the restart, its in the config and we need to remove it22:48
clarkber update it to say openstack/tap-as-a-service I guess22:48
clarkbinfra-root I've just noticed that mysql dump for zanata doesn't seem to be doing anything useful right now on translate0122:52
clarkbwe get a small file with only comments on it22:52
clarkbNoticed this when looking into an error reported to oepnstack-discuss22:52
clarkboh except we do the borg stream thing now so the local backups being sad may not be an issue22:53
clarkbyup that has contents that look like successful mysql dumps, nevermind22:56
clarkbIn the thread I suggest that maybe the solution here is to delete the user and start over again and I didn't want to do that if we didn't have working backups22:56
opendevreviewClark Boylan proposed openstack/project-config master: Rename x/tap-as-a-service to openstack/tap-as-a-service
clarkbfungi: ianw ^ that should fix the project-config error23:01
ianwyeah weirdly i've pushed several changes and that's the first time it's come up23:02
fungiwe missed that too when renaming23:02
clarkbianw: zuul will only check it if you modify the zuul config iirc23:02
clarkbotherwise it knows that its ok to keep running with the broken config23:03
clarkbfungi: yes I think we should go to split changes for renames again23:03
ianwyeah it didn't make this comment on either of the two previous revisions @
clarkbhuh that is curious23:04
clarkbianw: thanks for the +2 on the known_hsots chagne. At this point in the day my priority is helpign with that zuul fix and restart. I'll try to update hostvars tomorrow and land that chagne though23:11
opendevreviewClark Boylan proposed opendev/system-config master: Cleanup unused puppet modules from modules.env
clarkbapparently the httpd module uses the firwall module I did not expect that23:14
opendevreviewMerged openstack/project-config master: Rename x/tap-as-a-service to openstack/tap-as-a-service

Generated by 2.17.2 by Marius Gedminas - find it at!