Thursday, 2021-08-05

corvusclarkb, fungi: i'd like to restart zuul now00:01
clarkbcorvus: ok I'm around00:02
corvuslooks like image promotion is good00:02
corvusi don't see any release activity00:02
clarkbI don't see any openstack release activity00:02
corvusrunning the pull playbook now00:03
corvusseems to actually be doing something :)00:04
clarkbya the pull runs much longer when it has to download images :)00:04
corvusokay, ready.  i'm restarting now00:04
corvus#status log restarted all of zuul on commit d87a9a8b8f00887c37bb382271181cb5d703ba3f00:07
opendevstatuscorvus: finished logging00:07
fungithat was quick!00:09
fungii'm around to help keep an eye on things post-restart at least00:09
clarkbI don't think it has loaded configs yet00:09
corvusyeah, was just filling the dead time with status update :)00:09
corvusit's doing cat jobs for openstack now.00:11
corvusis openstack keeping branches open for longer?00:12
clarkbcorvus: sort of. The extended mainteance of branches has allowed branches to stick around00:13
corvuswas just thinking this is taking longer than it used to, and i saw the word "ocata" a lot.00:13
clarkbbut more recently we gave openstack release managers permission to delete those branches as they get really bitrotted and they have done that for some projects00:13
fungimore than intended. elodilles has been working on eol'ing and deleting branches which have fallen out of extended maintenance00:13
fungiand yeah, now that it's accessible via a specific access-controlled api call, we've granted that to the openstack release managers over just the openstack/ namespace projects00:14
fungiand removed their previous global extra access to some stuff too00:15
fungiso now they're no longer special in our global config00:15
TheJuliais zuul down?00:16
clarkbTheJulia: yes see above00:16
corvuswell, technically no00:16
clarkbit is being restarted to fix a bug where jobs could lose their node assignments causing them to hang out forever00:16
corvusbut it's not running jobs right now :)00:16
clarkbah fair enough it is working through loading its config so not fully operational00:16
clarkbI see the openstack tenant in the list now00:17
corvusit's queing; total perceived event outage time should be like 10 seconds00:18
corvusand re-enqueue is running now00:18
clarkbat least one job has started00:19
corvusre-enqueue done00:22
corvusTheJulia: should be gtg now00:22
TheJuliacorvus: thanks00:40
Clark[m]ianw: did you have an opinion on planning/scheduling for the Gerrit restart and mariadb jdbc connector flip?01:32
Clark[m]I think we want to land the first one and make sure the images promote. Then we can land the second. Then we do a pull and restart? I can work on getting changes landed tomorrow then we do the restart towards the end of my day maybe?01:33
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
ianwClark[m]: yeah nothing in particular.  I guess we know what to look for and can switch back quickly if issues03:21
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
*** ykarel|away is now known as ykarel04:47
*** marios is now known as marios|ruck06:31
*** jpena|off is now known as jpena07:08
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
*** rpittau|afk is now known as rpittau07:47
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
*** diablo_rojo is now known as Guest337108:27
opendevreviewMerged opendev/yaml2ical master: Update hacking version
opendevreviewMerged opendev/yaml2ical master: Report which week a meeting occurs.
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
opendevreviewxinliang proposed zuul/zuul-jobs master: Fix install podman error on Ubuntu aarch64 Bionic
*** dviroel|out is now known as dviroel11:25
*** jpena is now known as jpena|lunch11:37
iskullhi all,11:46
iskullI am new to openstack and need to setup 6 server infrastructure that can be scaled out11:47
iskullany one who can share some good documentation or guide11:47
iskullalso want to use SAN storage for the Block storage and VM root volume11:47
iskullhave checked with Charmed OpenStack team they require 12 servers and I have to go with 6 due to limitation of hardware11:49
*** Guest3371 is now known as diablo_rojo11:56
*** jpena|lunch is now known as jpena12:39
clarkbfungi: did you want to review before I approve it? That will rebuild our gerrit images which ensures we have the openid fix and the mariadb fix allowing us to switch over to mariadb jdbc connector (the child change does that)14:38
clarkbfungi: I'll plan to approve that change after some breakfast probably with intention of restarting gerrit later today or tomorrow depending on how timing goes14:38
clarkblooks like the zanata user with trouble logging in has requested that we delete their account. I'll give that a go after breakfast too (I've never done it before so who knows what will happne :) )14:42
fungii'll take a look thanks14:44
fungiclarkb: you refreshed some plugin versions for 3.3, i guess there were no plugin updates for 3.2?14:47
fungithe commit message doesn't really mention them either way14:47
clarkbfungi: we do that periodically beacuse we track the stable-3.3 branch for gerrit to build our 3.3 images. If plugins get new versions we want to update to match14:49
clarkbI don't know if they got new features or bug fixes though it is technically possibly14:49
clarkbbasically we don't want to build a new 3.3 image on new gerrit with old plugins14:50
mordredI have recently discovered the Storm Weather Shanty Choir. In case anyone is in need of a good Shanty14:51
fungii'll look them up. right now i only have the port isaac's fisherman's friends collection14:52
fungiwhich are great, but feel over-produced at times14:52
clarkbfungi: if every plugin maintained a stable branch or did reliable release tags this issue would go away. As is its a mix of behaviors and we try to keep up with periodic updates14:52
fungiclarkb: yeah, i get why we update the plugins when rebuilding, just wondering at the lack of 3.2 plugin updates... but maybe that's to be expected14:52
clarkbI think mordred  and corvus worked out some sort of system for doing upstream CI testing that we could try and incorporate ourselves14:53
*** ykarel is now known as ykarel|away14:53
clarkbfungi: 3.2 hasn't had a release since the last time we updated its list14:53
clarkbfungi: we are already running the most up to date versions of plugins there14:53
fungialso is there a script for finding those latest versions, or are you reviewing them all and adding them manually?14:53
clarkbfungi: I scan the repos manually as sometimes new stable branches show up or plugin stable releases don't get new tags14:54
fungiahh, okay. thanks14:54
mordredYeah, it would be easier if they always branched plugins14:56
mordredBut they don't. Yeah though, i do think there might be opportunity to port in the upstream stuff. I can't remember what's where tbh14:56
corvusrelated, they want to start building the actual releases with zuul, so we need to set up a post pipeline for them.  as part of that we can make a json file with build/branch/version info.  that might help us downstream?14:57
clarkbcorvus: oh ya I bet we could reconsume that to produce our images using the same refs14:58
clarkbthen branches and tags don't really matter as much14:58
corvusthat's a volunteer activity i haven't had time to start on yet, in case any one wants to pitch in on it with me14:58
* mordred can try to help on that some next week14:59
corvuskk thx15:00
clarkbI think the stable-3.4 merge up change is going to land soon if it hasn't landed already which is good15:01
corvusit is raining here.  oO15:01
clarkbwe aren't using 3.3 or 3.4 but it feels weird to deploy on 3.2 with fixes that may not be on the later branches then you regress when you upgrade15:01
clarkb#status Log A user account has been deleted from Zanata in an effort to fix the issue described in
opendevstatusclarkb: finished logging15:02
clarkbLooks like we will get a gitea 1.14.6 before 1.15.015:08
clarkbthe changelog update for 1.14.6 was approved half an hour ago.15:08
clarkbI don't expect any issues doing that update though15:09
clarkbyay for getting the non 1.15.0 fixes landed separately15:09
*** jpena is now known as jpena|off15:15
clarkb is the 3.4 upstream merge15:19
opendevreviewMerged opendev/system-config master: Rebuild gerrit images to include fixes
clarkbnow if the deploy jobs run successfully for that I can land the next chagne15:53
clarkb is the next change if you want to review that one too fungi15:54
clarkbits WIP because I don't want it landing until we're sure images have the bug fix15:54
clarkb did succeed so I think we're good there15:56
clarkbI;m about to pop out for some exercise but can work on landing the mariadb config chagne when I get back. I was thinking that it would be good to restart with ianw around in case we have to debug that further, but I guess we can always ninja revert to the mysql jdbc connector and try again later if we have to15:57
clarkbmordred: ^ not sure if you followed that mariadb thing at all but gerrit orm code has conditions to catch duplicate entry errors from the database and ignore them rather than doing an insert .. on duplicate key update type deal16:00
clarkbdebugging that was fun16:00
*** marios|ruck is now known as marios|out16:08
*** sshnaidm is now known as sshnaidm|afk16:29
*** rpittau is now known as rpittau|afk16:41
clarkbI have approved to switch us back to the mariadb connector17:47
clarkbthe mysql connector is still on the images if we need to roll back again17:48
clarkband now I promised to do more zuul reviews /me finds a comfy spot to do that17:48
fungiSubject: Element Matrix Services - Your trial is about to expire18:39
fungijust received that today at our infra-root address18:39
mordredclarkb: gross18:44
opendevreviewMerged opendev/system-config master: Use the mariadb jdbc connector in gerrit
clarkbfungi: I think that is expected they gave us a month free but there should be billing info in there. Was there any indication we'll be cancelled or shut down?18:45
fungii'll re-read it18:46
clarkbfungi: probably what we want to do is confirm with jbryce after billing happens that all looked well on his side18:46
clarkbfungi: looks like its just a heads up and no indication we'll be turned off18:46
fungiyeah, it's just a request for feedback, looks like18:46
fungii'm going to try to get a bit of yardwork done over the next hour or so, but will try to keep an eye on the computer when i stop to catch my breath18:49
fungi803373 has deployed, what was the plan for restarting gerrit?19:13
Clark[m]We need to pull the new image and then up -d. I'm eating lunch but can start that after. In say an hour19:15
Clark[m]Then we test toggling the reviewed flag functions19:16
fungiwfm, i'm caught up on yardwork now and just enjoying the breeze on the patio, so free to help whenever19:50
fungithough in an hour-ish i'll likely start to be distracted by dinner prep19:50
fungii guess i can docker-compose pull on review.o.o while the service is up, right? so we only need to down/up when the time comes19:51
clarkbfungi: did you want to do the docker-compose pull? I just need a few more minutes19:57
fungiyep, on it now19:59
fungiall three images have been successfully pulled20:00
fungiopendevorg/gerrit   3.2       f991f86f3054   5 hours ago    811MB20:00
clarkbok I'm at the keybaord properly now after lunch. I'm ready if you are20:00
clarkbthe openstack gate queues are empty and I see no release jobs that might be sad if talking to gerrit while it is down20:01
clarkb9d0dcdc652a2 is the current image20:01
fungistatus notice The Gerrit service on is going down for a quick restart to adjust its database connection configuration, and should return to service momentarily20:01
clarkbfungi: lgtm20:01
fungidoes that wording look okay?20:01
fungi#status notice The Gerrit service on is going down for a quick restart to adjust its database connection configuration, and should return to service momentarily20:01
opendevstatusfungi: sending notice20:01
-opendevstatus- NOTICE: The Gerrit service on is going down for a quick restart to adjust its database connection configuration, and should return to service momentarily20:02
fungionce that's hit most channels i'll down and up -d20:02
clarkband ya if you do a docker-compose up -d it should udpate the gerrit container because it has a new image20:02
fungiokay, i think the status notice is done, it doesn't seem to let you know when it finishes other than logging, can't recall if it ever did20:05
clarkbI'm ready20:05
fungidown it goes20:05
fungiand starting again20:05
fungiwebui is up again20:07
clarkbI am able to pull up a file I have previously marked reviewed and it doesn't throw an error at me20:07
clarkbIf I mark it unreviewed and then open it again it successfully transitions to reviewed20:07
clarkbThat was what threw errors previously with the mariadb connector so that looks good20:08
fungii can't remember how to toggle or view the reviewed state for a file20:08
clarkbfungi: its on the top level page of the change it shows "Reviewed" on the lines for each file. If you move your mouse to the right of that you get a mark unrevewied button20:08
clarkbI'm going to log out and log back in again just to be sure openid is working as expected as well since that got some updates too (I tested that on the test machien but can't be too careful)20:09
clarkbI will not be testing my ability to delete my openid's email address because if that doesn't stop me like it should I'll have a bad afternoon20:09
fungiinteresting, it doesn't show that state on the file list for me20:09
fungii'm not logged in ;)20:09
clarkbthat will do it :)20:10
clarkbfwiw logigng out and back in again worked great for me20:10
fungiyeah, seems to be nonbroken20:11
clarkbthis all looks good to me20:12
clarkbI guess early next week we can land the change to remove the mysql connector from the image if nothing pops up20:12
clarkb that chagne20:13
fungiyup, i'll go ahead and review that now while it's fresh in my mind20:13
clarkbThere is a gitea 1.14.6 I'll get a change up for that shortly (need to double check the diffs)20:14
fungiclarkb: on 803374 the code comment above it still says mysql-connector20:15
opendevreviewClark Boylan proposed opendev/system-config master: Update to gitea v1.14.6
clarkbfungi: ya our role supported both for review01 and review0220:19
clarkbfungi: there is further cleanup we can probably do20:19
clarkbinfra-root I think we should probably try and upgrade to 803650 as the changelog indicates a very minimal delta20:19
clarkbfungi: note I don't think it is a comment? the comment was removed as part of teh switch back to mariadb. But there is the ability to select mysql in the current role because it had to run against the old and new server20:20
fungiclarkb: the comment i marked says "Download mysql-connector so that gerrit doens't download it during init." and we're no longer downloading that after your change merges20:21
opendevreviewClark Boylan proposed opendev/system-config master: Update gitea project creation to only use the REST API
opendevreviewClark Boylan proposed opendev/system-config master: WIP Prepare gitea 1.15.0 upgrade
clarkbfungi: oh sorry I was looking at the chagne that merged not the one that is still open for review20:22
clarkbfungi: ya I can update that. We might also want to start removing the mysql support from the role in the same change. Why don't I check with ianw to see what he thinks about that and I can push a new patchset one way or another20:23
fungiwfm, this is entirely non-urgent cleanup after all20:23
clarkbI'd like to approve 803650 now to upgrade gitea. Any objections?21:43
clarkbI've approved it. You have however long it takes to gate to stop it :)21:57
ianwclarkb: thanks for running upgrade, looks like things are working22:10
clarkbianw: yup everything seems happy. Though I did not intentionally try to break my account and see if gerrit stops me22:10
clarkbfungi: ^ you might be willing to test that with your testing account though?22:10
clarkbianw: on do you want me to maybe try and pulling out the mariadb vs mysql support in the ansible role there too?22:11
clarkbianw: and we can land a cleanup for all that stuff in one go?22:11
clarkbnote I haven't done the ssh known hosts change yet because I wanted to get the restart done, but at this point I think I'll have to do that one tomorrow (but it is on my todo list022:12
ianwclarkb: i feel like i might have removed the mysql bits when review01 was removed?22:13
clarkbianw: they are still in the secure.conf file at least. See the diff in that chagne (at the top of the file)22:13
clarkbya gerrit_reviewdb_database_type is still in the role and everything.22:14
ianwbut yeah, i'd support any cleanup there; as elon says, the best part is no part :)22:14
clarkbalright I'll do that update in that chagne and we can land it next week after we're happy mysql connector bits won't be necessary22:14
clarkber I guess it is the diff of the parent change22:15
ianwi've got a few quick ones hanging around if people have some time22:17
ianw was from the other day when hound wasn't indexing zuul22:17
ianw tests openafs setup on centos-8-stream22:17
ianw(for wheels)22:17
ianw blocks a paste endpoint that is giving errors22:18
ianw was changes i used to run the prune on backup02, which just makes it automatically log the output22:18
clarkbianw: do we use h2 in testing or mariadb?22:19
clarkbwe must use mariadb because it is set as a host var?22:19
ianwwe should be using mariadb22:19
ianwfungi: other ones of interest are and 803405 which removes debian-stable from openstack-ansible.  so far i haven't been able to get reviews on them, and additionally stable/train is broken for other reasons22:21
ianwnot sure what to do here.  force merge might be an option as we already merged
opendevreviewClark Boylan proposed opendev/system-config master: Remove the mysql support from our gerrit role and image
clarkbbah I forgot to make the update fuingi suggested22:24
opendevreviewClark Boylan proposed opendev/system-config master: Remove the mysql support from our gerrit role and image
clarkbianw: ^ can you check my note in there about tesing (the dockerfile comment says we predownload so that gerrit.war init doesn't do it for us but now I wonder if gerrit assumes mysql or if it will properly see mariadb is there)22:26
ianwi actually think they removed the auto-downloading .jar bits ...22:29
ianwi have a vague memory of finding that change when setting this up, let me dig22:30
clarkbianw: I have approved the paste change22:31
ianwclarkb: is what i was thinking of22:37
clarkbfrom the commit message I'm not entirely sure what the effect of that is. but it does remove library management from init things22:38
ianwoh another fairly easy one is which adds the publish jobs for centos-8-strem wheels.  i've already created /mounted the volumes22:41
fungiclarkb: sorry, post-dinner fog... test what with my testing account?22:48
clarkbfungi: attempting to delete the email address associated with that accounts openid22:55
clarkbfungi: previously gerrit allowed you to od this and the account would be orphaned. But now it should return an error and not do it. However, there are a cuple of other gotchas including: you cannot delete your preferred email address so you need to add a second email address and make it preferred then try the delete22:56
clarkbfungi: I'm happy to walk you through it tomorrow if interested via a screenshare22:56
fungiyeah, i'm happy to test that tomorrow. for tonight i'm no longer in immediate vicinity of a machine which can easily authenticate to that account22:57
clarkbno worries23:00
clarkbthe gitea upgrade change has about 13 minutes left according to zuul and then it has to wait behind the hourly deploy jobs23:03
opendevreviewMerged opendev/system-config master: Update to gitea v1.14.6
opendevreviewMerged opendev/system-config master: lodgeit: disable getRecent API endpoint
clarkbI'm going to need to move some furniture around (new couch tomorrow) but I'll keep an eye on the gitea upgrade23:14
fungicongrats on achieving new seating23:26
clarkbthere is quite the production and shipping backlog on things like that right now23:31
clarkbwe bought it months ago23:31

Generated by 2.17.2 by Marius Gedminas - find it at!