Thursday, 2021-07-22

opendevreviewIan Wienand proposed opendev/system-config master: Add a playbook to grab collated info from hosts  https://review.opendev.org/c/opendev/system-config/+/80167101:42
opendevreviewIan Wienand proposed opendev/system-config master: Add a playbook to grab collated info from hosts  https://review.opendev.org/c/opendev/system-config/+/80167101:45
*** marios is now known as marios|ruck05:51
*** ykarel|away is now known as ykarel06:12
*** amoralej|off is now known as amoralej07:00
opendevreviewSorin Sb├órnea proposed opendev/git-review master: Make default output less verbose  https://review.opendev.org/c/opendev/git-review/+/77264607:32
opendevreviewMerged openstack/diskimage-builder master: Replace deprecated import of ABCs from collections  https://review.opendev.org/c/openstack/diskimage-builder/+/80109207:38
*** rpittau|afk is now known as rpittau07:49
opendevreviewMerged openstack/diskimage-builder master: Auto find greatest Fedora cloud image sub-release  https://review.opendev.org/c/openstack/diskimage-builder/+/75599207:53
*** bhagyashris_ is now known as bhagyashris08:05
*** bhagyashris_ is now known as bhagyashris09:35
opendevreviewTakashi Kajinami proposed openstack/diskimage-builder master: Update IRC networks  https://review.opendev.org/c/openstack/diskimage-builder/+/80175810:22
*** ykarel is now known as ykarel|lunch10:25
opendevreviewMerged openstack/diskimage-builder master: Add a keylime-agent element and a tpm-emulator element  https://review.opendev.org/c/openstack/diskimage-builder/+/78960110:34
*** ykarel|lunch is now known as ykarel11:52
opendevreviewKartikeya Jain proposed openstack/diskimage-builder master: Add new element oracle-linux  https://review.opendev.org/c/openstack/diskimage-builder/+/80177412:13
*** amoralej is now known as amoralej|lunch12:28
*** ykarel_ is now known as ykarel12:38
*** elodilles is now known as elodilles_afk12:48
*** amoralej|lunch is now known as amoralej13:29
*** elodilles_afk is now known as elodilles14:34
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container  https://review.opendev.org/c/opendev/elastic-recheck/+/72962314:43
*** ykarel_ is now known as ykarel14:56
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container  https://review.opendev.org/c/opendev/elastic-recheck/+/72962315:02
*** ykarel is now known as ykarel|away15:08
clarkbslow start this morning but looking to approve the gitea upgrade shortly. Any reason to not do that at this point?15:45
fungii know of none15:46
clarkbI have approved it. And while i wait for it to gate I should go water my garden15:47
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container  https://review.opendev.org/c/opendev/elastic-recheck/+/72962315:52
clarkbfungi: ianw: if you can review the matrix spec today that would be great.15:56
fungii'm in the middle of it actually, but so far no concerns15:56
clarkbgreat15:56
*** rpittau is now known as rpittau|afk16:02
opendevreviewClark Boylan proposed opendev/system-config master: Upgrade gitea to 1.14.5  https://review.opendev.org/c/opendev/system-config/+/80190616:19
clarkbI just noticed that there is another release we can update too. That is pushed as a followon since it should be straightforward to approve that if 1.14.4 goes well16:19
clarkbfungi: if 1.14.4 rolls out smoothly I think we shoukld try and land ^ quickly behind it just to avoid any unnecessary lag there16:23
fungiyeah, i concur16:23
*** marios|ruck is now known as marios|out16:24
*** amoralej is now known as amoralej|off16:32
opendevreviewJames E. Blair proposed opendev/system-config master: Add DNSSEC configuration for gating.dev  https://review.opendev.org/c/opendev/system-config/+/80190716:36
opendevreviewJames E. Blair proposed opendev/system-config master: Serve gating.dev static website  https://review.opendev.org/c/opendev/system-config/+/80190816:37
opendevreviewMerged opendev/system-config master: Update gitea to 1.14.4  https://review.opendev.org/c/opendev/system-config/+/80027417:01
clarkbthe job to deploy that should start momentarily17:03
clarkbI'm watching it17:03
fungii'm still around17:03
clarkbinteresting it did the next job in the hourly buildset instead of the gitea deploy job in the deploy buildset17:04
clarkbI wonder if we have to wait for that whole buildset now? change in behavior with semaphores in zk maybe?17:05
corvusi'm not sure there's a strict ordering of semaphores across pipelines; it may be semi-random?17:07
clarkbah that could be. In any case the job won't run as soon as I expected as other jobs are getting the semaphore17:08
fungithat would make some sense, yeah17:10
opendevreviewJames E. Blair proposed opendev/system-config master: Serve gating.dev static website  https://review.opendev.org/c/opendev/system-config/+/80190817:24
corvusclarkb, fungi, mordred: if you could approve https://review.opendev.org/801907 i can go ahead and add the dns key to the registrar17:25
clarkbcorvus: actual host/group vars have the secret info ready?17:27
corvusyep17:27
fungiapproved it now17:31
clarkbalright the job is running now17:56
opendevreviewMerged opendev/system-config master: Add DNSSEC configuration for gating.dev  https://review.opendev.org/c/opendev/system-config/+/80190717:56
clarkbhttps://gitea01.opendev.org:3081/opendev/system-config seems to be up18:01
clarkbit wasn't the fastest update, but after a couple of minutes the service finishes it upgrade stuff and the ssh daemon was started18:01
fungiyeah, lgtm18:08
fungiseems like they make minor changes to fonts on every update18:08
clarkbThe first 3 are done now. 4th is in progress18:10
clarkband now 4 is done. We are halfway through the set.18:12
clarkbfungi: I guess if all 8 checkout I should go ahead and approve the 1.14.5 update?18:12
fungiyep18:15
clarkbThen I need to remember to clean up my held node once that is all done18:16
clarkball 8 look reachable and render the repo as expected. I'm approving 1.14.5 update now18:27
fungithanks18:32
clarkbIt feels good to catch back up to them again. Time to start looking at gerrit 3.3 upgrades soon as well I think18:36
clarkbwe could go to 3.4 but luca mentioend there are a ton of bugfixes in the queue for 3.4 so might be better to step through 3.3 first? I also kno that you can downgrade from 3.3 to 3.2 with a manual notedb edit if necessary which is a nice option18:37
fungiinteresting18:38
clarkbI added a release note to gerrit for the oepnid change I'm making and that release note goes against 3.4 I guess. The list of changes is quite long there.18:38
fungigood to know18:40
clarkbfungi: side note I didn't break the gerrit opendev theme with the gitea update :)18:44
clarkbat least it currently loads for me in what I would consider the correct manner18:45
fungiyeah, seems to still be intact18:45
fungifuture improvement would be to centralize/generalize our logo storage in the repo and then bake copies into images without having them depend on one another18:46
clarkbI think right now its fetching it via https in the browser, but ya if we can stick that in a location on the gerrit image and the gitea image from a central source that would be best18:47
clarkbThen we don't have to remember a bunch of update locations etc18:47
fungiand also a gitea outage doesn't degrade gerrit performance that way18:48
clarkbmordred and others may find https://engineering.fb.com/2021/07/22/data-infrastructure/mysql/ to be interesting19:05
clarkbianw: related to ^ do we have an explicit record old db state step before shutting down review01 and then also a shutdown the db step?19:07
opendevreviewMerged opendev/system-config master: Upgrade gitea to 1.14.5  https://review.opendev.org/c/opendev/system-config/+/80190619:41
Clark[m]I'm eating lunch but gitea01 has updated and looks good on a phone19:51
*** sshnaidm is now known as sshnaidm|afk20:10
clarkb#status log Upgraded gitea to version 1.14.520:59
opendevstatusclarkb: finished logging20:59
clarkbForgot to do that earlier20:59
fungiwell now you don't have to redo it21:00
fungiit does look like they may have adjusted some icons as well21:01
fungialso this looks kinda weird, do we have an incomplete sync or something? https://opendev.org/opendev/bindep/commit/64205044f52b08d55a4c4f994aecede6ce51428d21:04
clarkbthat belongs to changes/20/742320/meta ?21:05
clarkbI think that may be complete. Its just the notedb content?21:05
clarkbI'm not sure how well that all renders in gitea21:06
fungioh! that's a notes ref not the commit itself21:06
fungistrange21:06
clarkbyes its the metadata about the change if I'm reading it correctly21:07
fungiyep, the commit is https://opendev.org/opendev/bindep/commit/3bee5129d72497070de7ded454ae88c2ad7e263421:07
fungiwhich unfortunately doesn't display the note (i think there was a todo to change the displayed notes ref base or something in the config)21:08
fungiand because it treats the notes refs as independent commits it makes the commit graph fairly useless: https://opendev.org/opendev/bindep/graph21:09
clarkboh wow ya that is pretty useless21:10
fungiaha, also it's including unmerged refs in that: https://opendev.org/opendev/bindep/commit/4ebb650b203c42e6e0d64aaee063470e01f9d41021:10
fungiso yeah, how gerrit stores patchsets and git notes seems to be incompatible with the design of that commit graph feature21:12
ianwclarkb: we don't have a record old db step; are you interested in keeping the <3 gerrit state?21:49
clarkbianw: ya I was thinking it might be a good idea to take a final snapshot of that entire db (the pre notedb stuff and accountpatchreviewdb) before we clean up that server?21:52
clarkbfungi: for the matrix spec I guess your rollcall +1 indicates you're ok with it landing despite your feedback? I'm not super clear on that since you didn't leave a code review vote21:55
clarkbI think the debian afs mirror may be at quota22:01
clarkbat least the grafana graph shows that it ran out of space22:01
clarkbwe freed up 140GB of the yum-puppetlabs repo so we should be able to bump debian up. Maybe from 600GB to 640?22:02
clarkbfungi: ^ You might have the most context for that repo and how it might grow over time22:02
ianwclarkb: i can; the Marie Kondo in me says that nothing about it "sparks joy" so perhaps it is better left as a memory22:09
clarkbianw: in my head it would mostly be used to help correct any potential issues we discover from the migration. Granted we haven't seen any of those that I know of other than the account problems. But having the ability to look back at the info as it was pre migration could be helpful if we run into similar issueswith changes?22:10
ianwclarkb / fungi : i feel like there was no real objection to removing stretch, so we have a way forward to reduce the overhead, so bumping it for now seems ok while we do the work22:11
clarkbianw: ya I don't expect the quota bump to be necessary long term. Wondering what an appropriate bump might be? Is 40GB too little or is that sufficient?22:15
clarkblooks like there may be a bit of room there but I'll go ahead and bump it by 40GB to avoid failed releases and updates22:18
ianwclarkb: ok, i've made an "attic" directory on review02 for things of interest to archive.  i'll run a dump against the trove db and save it in there22:18
clarkbianw: thanks22:18
ianwi think that should be plenty22:20
ianw(40gb)22:20
clarkbok its bumped on the RW volume and the next regular update should push the quota bump to the RO volume22:23
clarkbmy biggest concern right now is that we might break the mirror because we stop updating it. This should avoid that problem and then we can trim back down again via stretch removal22:25
clarkbwe might also consider dropping the fedora mirror volume quota down since it is set quite large22:26
clarkbbtu that is less urgent and might be better to do that for all the volumes once we've trimmed a bit more22:26
clarkbGerrit screenshots making it into the tech press for all the wrong reasons https://arstechnica.com/gadgets/2021/07/google-pushed-a-one-character-typo-to-production-bricking-chrome-os-devices/22:52
clarkbalso check out the change number.22:54
fungiclarkb: i'm in favor of merging the matrix spec, yes23:01
fungii left feedback as it occurred to me23:01
clarkbfungi: thanks for confirming. ianw  did you want to review that matrix spec before I approve it? I can approve it a bit later today if so23:02
fungii figured code review for something with a rollcall was not especially relevant23:02
fungibut happy to add a cr +1 as well (there is no core reviewer permission for +2 on the specs repo i don't think)23:02
clarkbya its a +1. I guess in my head I associate CR as this is ok to merge as is and Roll Call as I am comfortable with this plan23:03
clarkbthey are slightly different things23:03
fungias for the old gerrit db, that's in trove anyway right?23:04
fungiso deleting the server instance isn't going to impact it23:04
clarkbfungi: correct, but the next step is to delete the trove instance :)23:04
fungiawesome23:08
ianwok, backups in ~gerrit2/attic/old-db-2021-07 along with a readme for future admins who wonder what the heck we were doing :)23:09
fungior s/doing/smoking/23:10
clarkbI'm trying to push an update to my upstream gerrit change and it is hanging. It is unfortunate we can't really compare our srever performance against the upstream one23:16
clarkbthough I suppose it could be local network problems too. I've got some weird rtt aross the local network23:19
clarkbstrace says it is stuck on a read23:20
clarkbif in doubt try again. went through on second attempt23:21
fungias one does23:27
opendevreviewClark Boylan proposed opendev/system-config master: Push a patch to try and prevent gerrit openid deletion  https://review.opendev.org/c/opendev/system-config/+/80083223:43
clarkbI got my first +2 upstream so I've set up a new hold and pushed the current state of the diff into ^23:44
clarkbI'll redo my functional testing tomorrow on the held node23:44

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!