Thursday, 2021-07-01

clarkbcorvus: mordred: I'm noticing that I may have done a derp with timezones and the EMS discussion. I see now there may have been a desire to do 17:00 UK time instead of UTC. I'm able to do both (UK time would be 16:00 UTC). I'm usually awake well in advance of that and will try to clarify in the morning. Sorry about that00:27
clarkbalso yay timezones00:27
corvusclarkb: i think that should be okay for me00:36
*** ysandeep|away is now known as ysandeep01:09
mordredclarkb: no worries - should be fine for me too01:18
*** ykarel|away is now known as ykarel04:39
opendevreviewIan Wienand proposed zuul/zuul-jobs master: Add introduction to the container jobs documentation
opendevreviewIan Wienand proposed opendev/lodgeit master: Add mariadb requirements
opendevreviewMerged opendev/base-jobs master: Add opendev-intermediate-registry-base
*** bhagyashris is now known as bhagyashris|ruck05:22
*** chandankumar is now known as chkumar|rover05:31
*** gthiemon1e is now known as gthiemonge06:31
*** amoralej|off is now known as amoralej06:45
opendevreviewIan Wienand proposed opendev/lodgeit master: Add mariadb connector to container
opendevreviewIan Wienand proposed opendev/lodgeit master: Drop Python 2.7 support
opendevreviewIan Wienand proposed opendev/system-config master: Add paste service
ianwclarkb/fungi: ^ ready for review.  i don't care about paste that much, what really interests me is getting  a bit of experience with the mariadb container, which is the same as rolled out to review0207:16
*** gthiemon1e is now known as gthiemonge07:18
*** jpena|off is now known as jpena07:34
opendevreviewIan Wienand proposed opendev/base-jobs master: Revert "Add opendev-intermediate-registry-base"
*** ysandeep is now known as ysandeep|lunch07:48
opendevreviewMerged opendev/base-jobs master: Revert "Add opendev-intermediate-registry-base"
opendevreviewIan Wienand proposed opendev/lodgeit master: Add mariadb connector to container
opendevreviewIan Wienand proposed opendev/system-config master: Add paste service
opendevreviewIan Wienand proposed opendev/system-config master: lodgeit: use mariadb connector
opendevreviewIan Wienand proposed zuul/zuul-jobs master: Add introduction to the container jobs documentation
*** slaweq_ is now known as slaweq08:29
opendevreviewIan Wienand proposed zuul/zuul-jobs master: Add introduction to the container jobs documentation
*** ykarel is now known as ykarel|lunch08:44
*** ysandeep|lunch is now known as ysandeep08:45
opendevreviewIan Wienand proposed zuul/zuul-jobs master: Add introduction to the container jobs documentation
*** sshnaidm is now known as sshnaidm|afk09:24
opendevreviewMerged opendev/system-config master: refstack: trigger image upload
*** ykarel|lunch is now known as ykarel09:59
*** sshnaidm|afk is now known as sshnaidm10:47
*** dviroel|out is now known as dviroel11:32
*** jpena is now known as jpena|lunch11:47
funginoonedeadpunk: if you get time, can you take a look at ianw's two changes proposed changes for opendev/lodgeit? i know you've had some interest in it in the past12:26
fungiin particular, the first one is dropping python 2.7 support, and i don't know if that will impact your deployment12:26
noonedeadpunkyeah, that's fine as eventually it's dropping py27 only for CI I guess12:29
noonedeadpunkbut I'm not sure under which python it works atm? As  Iguess it's 2.7?12:29
funginoonedeadpunk: these are preparing for 798400 which runs it from the containers we're building, so effectively switching to 3.x12:33
fungithough you're right, it does mean we risk merging a change which breaks 2.7 support while working on switching over our deployment12:34
fungiianw: ^ we might want to call for a bit of a freeze on merging functional changes for lodgeit between when we merge 798990 and when we switch production over12:35
funginot that it gets that many changes12:35
noonedeadpunkand iirc there's issue with py 3.8 due to old Werkzeug and I never had time to re-work using script12:42
noonedeadpunkand was reasonably -CR12:42
*** jpena|lunch is now known as jpena12:45
*** amoralej is now known as amoralej|lunch13:02
*** corvus is now known as Guest117013:12
funginoonedeadpunk: that's probably worth mentioning on 798990 in that case13:15
*** marios is now known as marios|call13:26
*** amoralej|lunch is now known as amoralej13:54
clarkbGuest1170: mordred looks like 16:00UTC is preferable14:21
clarkbalso looks like the matrix oftc bridge had a hiccup?14:22
*** ykarel is now known as ykarel|away14:25
Guest1170clarkb: ack;  what'd the hiccup look like?14:52
fungiGuest1170: well, for starters you seem to be stuck with a guest nick14:53
fungi(or was that intentional?)14:53
Guest1170neat, that is unapparent to me ;)14:53
fungibut there was a mass exit and rejoin which had a bunch of ...[m] nicks, looked like a netsplit14:54
*** Guest1170 is now known as corvus14:54
fungireason was "Quit: Bridge terminating on SIGTERM)14:54
*** corvus is now known as Guest117514:55
Guest1175ok, i had a chat with oftc-irc, and it said i was called Guest1170, but i told it i wanted to be corvus and it said ok.14:55
fungiit already switched you back to a guest nick again14:55
fungilike your session isn't identified to the nickserv14:55
fungi(and with enforce set on your nick)14:55
Guest1175usually oftc-irc handles that, but i guess whatever happened disrupted that process; i'll chat with nickserv14:56
*** Guest1175 is now known as corvus14:57
*** corvus is now known as jeblair14:58
*** jeblair is now known as corvus14:58
corvushow my doing now?14:58
fungiseems to be working now14:59
fungiit hasn't forced you back since 14:5714:59
fungiwhereas previously it was near-instant14:59
corvusseems like oftc-irc got out of sync with nickserv; so i had to manually identify to nickserv, then change my nick with oftc-irc twice (because it still thought i was corvus)15:00
clarkbI see a corvus now15:02
fungithat's fun15:04
corvusit's interesting how invisible that can be to folks on the matrix side15:05
fungiyeah, i guess because they're seeing just the matrix end so if it's out of sync, then that won't be visible15:06
*** marios|call is now known as marios15:26
clarkb is interesting. Good to see them be so proactive about this though15:30
*** sshnaidm is now known as sshnaidm|afk15:46
corvusneat; i'll upgrade my synapse later today15:52
corvusmordred: can you tell what synapse you're running, and what the upgrade process is like?15:53
corvus(just wondering if that would give us clues to what life with ems hosting would be like -- do they just do that automatically in the background, etc...)15:53
mordredlemme see15:54
mordredso - it's not showing me a version that I see at the moment - it does say: "(last update: 10:58:58)"15:59
mordredoh - found it15:59
* mordred uploaded an image: (173KiB) < >16:00
*** dviroel is now known as dviroel|lunch16:00
clarkbmordred: corvus: its time now if you want to join16:01
corvusmordred: neat; so seems like they're keeping stuff up-to-date automatically without you noticing at all16:01
*** ysandeep is now known as ysandeep|dinner16:01
fungimayhaps that was the reason for the bridge dropping and reconnecting earlier16:02
*** amoralej is now known as amoralej|off16:07
*** marios is now known as marios|afk16:12
*** marios|afk is now known as marios|bosu16:22
*** marios|bosu is now known as marios|out16:23
*** dviroel|lunch is now known as dviroel16:28
clarkbthat went well. They don't seem opposed to our idea of how we host users separate from the homeserver. It sounds like the next steps will be for me to tell jbryce what I have learned since a credit card is necessary to take the next step (I'm meeting with him later today) then sign up at a trial level and start interacting with EMS directly16:30
clarkbmordred: corvus: (and the rest of infra-root) I'll keep you updated on that process but I can probably boostrap things to the point where other admins can start interacting with the small setup?16:30
mordredclarkb: yah. notable when you're setting up the homeserver, there is a field on the wizard called "custom DNS" - which you definitely want to expand/enable16:32
clarkbmordred: ok. I'm sure I'll ask lots of questions when I get to that points16:32
mordredthat's what you need to do for us to be able to set the dns records and whatnot16:32
mordredit's mostly very straightforward16:32
mordredbut I messed that up on my first stab at setting up the inaugust homeserver16:32
clarkbcorvus: also I guess we should update the spec at this point since EMS didn't object immediately :) ?16:33
mordredclarkb: we'll also need to publish two well-known files on opendev.org16:33
clarkbya that should be easy we just add it to the docker image (we already od that for the logo file iirc)16:34
mordredthose are mine, fwiw16:34
fungipopping out to the hardware store for a bit, but should return shortly16:35
*** jpena is now known as jpena|off16:36
*** ysandeep|dinner is now known as ysandeep16:44
corvusclarkb, mordred: yep, i'll update the spec to say "ems hosted"17:07
*** ysandeep is now known as ysandeep|out17:24
fungiokay, hardware has been obtained. also dirt. growing up in the mountains i would have never thought that buying dirt and rocks is a thing. now that i live on a sandbar where those resources are scarce, i understand it, but still feel inherently weird doing it17:51
clarkbfungi: you can also get giant dump trucks to dump it off in front of your drive way around here17:53
clarkbbut then if I want to get rid of it again that is far more problematic as we have an invasive beetle around here and dirt is quarantined17:53
fungiyeah, here too, but they bring it from not this island, so it's not cheap17:53
fungidump trucks of dirt being a thing here too i mean17:53
fungihere you usually just buy sand by the truckload if you need fill. or crushed shells17:54
fungithey're constantly dredging all that out of the inlets to keep them from shoaling over, so there's a surplus17:54
fungihere we put dirt in containers, it's way to precious to just dump on the ground ;)17:55
fungiotherwise it could just be carried away on the next tide17:56
clarkbdiablo_rojo_phone: looks like the ptgbot container change is really close (it hit an issue in the gate after ianw approved it) there is a commetn on the change for how to fix the latest thing18:01
diablo_rojoclarkb, I will get the change updated today18:02
corvusWe have clay on the ground so we have to buy dirt if we want things to grow, but we started making our own soil, so hopefully we won't need to buy any more.18:38
clarkbwe got email from rackspace indicating afsdb03 had trouble. I see it has been up fro a couple of hours now and bos status against it seems fine18:44
fungimay be worth checking that vos release is still coming back clean for stuff in our logs, but sounds like it's okay18:50
*** dviroel is now known as dviroel|sfk19:00
*** dviroel|sfk is now known as dviroel|afk19:01
*** gthiemon1e is now known as gthiemonge19:05
clarkbinfra-root I'm finally in a spot where I think I can keep an eye on fallout so I'll approve that now20:18
clarkbplease say something if you don't get puppet jobs running when you expect them to20:19
clarkbit only affects testing and is an easy revert if necessary20:19
fungithanks for the heads up!20:21
*** dviroel|afk is now known as dviroel20:21
opendevreviewMerged opendev/system-config master: Run the puppet apply job less often
clarkbfungi: our arm64 queue looks weird. We are still running jobs not not as many as I would expect based on the grafana graphs21:42
clarkbalso linaro seems completely unused?21:42
clarkbour max servers value for linaro is 40 though21:42
clarkb[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired21:43
clarkbconfirmed the cert expired on the 27th of june21:44
clarkbkevinz: ^ fyi the linaro cloud's ssl cert has expired and that is preventing us frmo scheduling workload to it21:44
clarkbI suspect the weirdness is that we need linaro to fail enough times that osuosl picks it up and osuosl has less overal headroom so it just takes longer?21:45
clarkbkevinz: if you use for that LE provisioning they changed their default cert provider to zerossl. There are command line flags you can use to force letsencypt again (we ran into this somewhat recently)21:46
*** dviroel is now known as dviroel|out21:52
fungiclarkb: yeah, it seems like the cert for the api there is not auto-updating. we get into this same situation every few months and then he replaces the cert21:59
clarkbmaybe we should add that host to certcheck?22:01
funginot a terrible idea, at least then we could give kevinz a heads up one month in advance22:17
fungido we need to keep running track-upstream for our gerrit? the replacement gerrit server e-mails us about a failure ~hourly from that cronjob, and i'm not certain we even still rely on it for anything of note22:22
fungiwere we still relying on it in production until the very recent move to using upstream gerrit source for our images?22:23
clarkbfungi: we were relying on it on gerrit 21322:26
clarkbbut when we upgraded to 3.2 we applied our patches using patch and diffs in the container dir. I think we can go that route for any potential future patching and just stop the upstream tracking22:27
clarkb(also we are unforked now and should keep it that way if at all possible)22:27
ianwclarkb / corvus: so after merging that intermediate registry pull yesterday i went to apply it an what started starting me in the face but ...
fungiclarkb: okay, so we could probably retire our gerrit fork repo, and also disable/remove our track-upstream cronjob?22:34
clarkbfungi: pretty sure we can yes. is the git repo we check out in our jobs22:36
clarkbnot our fork22:36
clarkbianw: ah we have it in system-config not base-josb?22:37
ianwbasically we already had it with "parent: opendev-buildset-registry-consumer" @
ianwi had totally missed it and didn't notice it.  i think some of our jobs are incorrectly parented now, need to audit them22:38
ianwkopecmartin: do you want me to pull the refstack container?  i wasn't quite clear on the ordering required22:43
opendevreviewJeremy Stanley proposed openstack/project-config master: Drop use of track-upstream
opendevreviewJeremy Stanley proposed opendev/system-config master: Good riddance to track-upstream and its cronjob
opendevreviewJeremy Stanley proposed openstack/project-config master: Drop use of track-upstream
opendevreviewIan Wienand proposed openstack/diskimage-builder master: [wip] test centos8-stream with hwrng map

Generated by 2.17.2 by Marius Gedminas - find it at!