Friday, 2021-05-28

fungipaladox: i assume you've been following our progress there, shouldn't be much longer00:02
paladoxOh, nope00:03
paladoxI’ve been busy watching all the drama in fsf00:03
paladoxAnd also the freenode policy discussion forum00:03
fungifun times, yeah00:03
paladoxOh moving to oftc00:03
fungishh, not so loud, don't let the new regime hear you ;)00:03
paladoxOnce openstack moves I can delete my account :P00:05
fungiyeah, we'll do them all at once00:11
ianwi feel like i should move to weechat00:19
ianwbut i grep ~/.config/hexchat/logs/ to find random things quite a lot00:20
corvusi feel quite certain that weechat must have a logging facility :)00:20
paladoxI migrated to irccloud00:22
paladoxBetter history management00:22
paladoxPaid the £35 for the year00:22
corvuspaladox: i migrated to matrix and am loving it; here's my thinking:
corvus(all of my irc interaction is now through matrix)00:24
ianwis it worth the effort of running a matrix homeserver, other than to say you are doing it?00:24
corvusianw: nope00:24
ianw:) that was the vibe i got from reading things00:24
corvusianw: if you want to host rooms, it's a bit more compelling.  and having element run one for you if you do feel it's worthwhile seems very reasonable.00:26
paladoxI guess matrix is similar in terms of irccloud (modern and also better history management)00:26
corvuspaladox: yeah, i think you get all those features, plus other networks, interactions, interfaces00:26
corvusi've actually restarted the weechat client i'm using to talk with you right now several times during this conversation (as i'm trying out startup uptions.  :)00:27
paladoxCan you search in it? That’s a missing feature00:27
paladoxI didn’t have that under my previously client but still would be useful00:27
corvuspaladox: the element client seems to have some search capability; i haven't used it yet00:29
corvuslooks like there's search in room, and search all rooms00:30
corvusi also think i remember seeing something in synapse config about search; so that might be something you could tune if you run your own homeserver00:30
corvusbut honestly, i know little about that; only enough to say "it's worth looking into"00:30
corvuspaladox: matrix is getting a bridge to that new network ;) any day now (it's currently in testing)00:32
paladoxLibera? Or oftc?00:32
paladoxI herd that the bridge is now active for libera00:33
corvuslibera; oftc, freenode, and several others exist already00:33
paladoxYeh libera is now active00:33
corvusyeah, it's active, but in testing, they don't really want people using it :)00:33
corvusbut i guess they're having trouble getting that message out :)00:33
paladoxI wonder why it takes so longer to open that.00:34
corvusit takes some cooperation with the irc server admins, and they're busy.  but they're also setting it up in a really nice new way that will be awesome for matrix users00:35
*** tkajinam has quit IRC00:35
corvusthey're setting up a dedicated homeserver for libera, so the matrix usernames and room addresses will all be foo:libera.chat00:35
corvuswhich makes complete sense and is totally obvious in retrospect and all the other network users are getting jealous :)00:36
*** hamalq has quit IRC00:36
*** hamalq has joined #opendev00:37
openstackgerritIan Wienand proposed opendev/system-config master: WIP : update ARA
fungiianw: i grep .weechat/logs/* all the time01:00
ianwfungi: good to know, i had pessimistically imagined it would be some sort of Gaussian-Markov-Chain-Tensorflow-ML-NoSQL type on-disk concoction, but clearly .txt files have some life left :)01:02
fungiyeah, just plain old text copies of every channel i'm in, kept forever unless i delete them01:04
fungiwhich... i don't01:04
prometheanfirethat's about what I'm doing, except on the bouncer side, not client side01:15
openstackgerritIan Wienand proposed opendev/system-config master: WIP : update ARA
*** hamalq has quit IRC01:16
openstackgerritMerged openstack/diskimage-builder master: Remove octvia-v1-dsvm-* jobs
openstackgerritMerged ttygroup/gertty master: Highlight WIP state in change view
*** jhesketh has quit IRC01:50
openstackgerritIan Wienand proposed opendev/system-config master: WIP : update ARA
openstackgerritJeremy Stanley proposed opendev/system-config master: Retool accessbot for OFTC
fungiclarkb: corvus: ^ addressed your points, mostly by adding lots of clarifying code comments and/or todos02:02
openstackgerritJeremy Stanley proposed opendev/system-config master: Retool accessbot for OFTC
fungiand added yet one more code comment02:09
openstackgerritIan Wienand proposed openstack/project-config master: Add
openstackgerritIan Wienand proposed opendev/system-config master: Update ARA
openstackgerritIan Wienand proposed opendev/system-config master: bridge: upgrade to Ansible 4.0.0
openstackgerritIan Wienand proposed opendev/system-config master: Remove documentation
openstackgerritMerged opendev/system-config master: More puppetry and inventory cleanups
openstackgerritIan Wienand proposed zuul/zuul-jobs master: ensure-zookeeper: better match return code
openstackgerritMerged opendev/system-config master: python-builder: don't force siblings install
*** d34dh0r53 has quit IRC04:48
openstackgerritMerged openstack/diskimage-builder master: bootloader: remove extlinux/syslinux path
*** zbr4 has joined #opendev05:04
*** auristor has quit IRC05:04
*** zbr has quit IRC05:06
*** zbr4 is now known as zbr05:06
*** marios has joined #opendev05:11
*** timburke has quit IRC05:55
*** hashar has joined #opendev07:13
*** marios has quit IRC07:15
*** andrewbonney has joined #opendev07:21
*** marios has joined #opendev07:27
*** marios has quit IRC07:30
*** lourot has quit IRC07:45
*** lourot has joined #opendev07:46
*** dtantsur has quit IRC07:49
*** sshnaidm is now known as sshnaidm|off08:04
*** hashar has quit IRC08:14
*** marios has joined #opendev08:27
openstackgerritMerged openstack/project-config master: Retire x/gearman-plugin
*** tosky has joined #opendev09:19
openstackgerritRodion Gyrbu proposed zuul/zuul-jobs master: Bumb golang version Add new task with check installed go version
*** CeeMac has quit IRC10:21
openstackgerritRodion Gyrbu proposed zuul/zuul-jobs master: Bumb golang version Add new task with check installed go version
openstackgerritRodion Gyrbu proposed zuul/zuul-jobs master: Bumb golang version Add new task with check installed go version
*** auristor has joined #opendev12:24
fungiinfra-root: my plan for today is to get all the current changes under topic:oftc merged. the one which changes the accessbot script in system-config and the one which changes the channels.yaml for it in project-config need deployment temporarily halted first, for safety, or at least to have eavesdrop in the emergency disable list so it doesn't try to do new things on the old network12:42
fungii'll take care of that once more people are around, just in case of explosions12:43
fungialso i have weekend visitors arriving in a few hours so my availability will be spotty, but i hope to get channel topics replicated at some point later today so everything's ready for reconfiguring other bots tomorrow12:44
openstackgerritMerged openstack/project-config master: Accessbot OFTC channel list stopgap
openstackgerritMerged openstack/project-config master: Clean up accessbot channels without services
fricklerfungi: what time do you plan to do the switch tomorrow? I can try to be around to help with possible issues13:57
*** openstackgerrit has quit IRC14:06
fungifrickler: i didn't have a specific time in mind since i'm also entertaining guests, but can coordinate a time which is convenient for others14:12
yoctozeptomorning infra14:17
yoctozeptois the gerrit bot gone permanently?14:17
fungino, probably some outage knocked it down again, but it was reporting changes merged in here 20 minutes ago14:19
fungioh, actually it was just restarting after 792842 merged14:19
fungiit will likely rejoin channels when it has something to report14:20
fungii'm not planning to switch its network until tomorrow14:20
fungiyoctozepto: did it fail to report a change in the past few minutes?14:20
corvusit reports to more channels than it can join, so it treats the channels as an lru cache and joins and leaves as needed14:21
yoctozeptofungi: no idea, thinking, just noticed it gone14:21
yoctozeptothinking none*14:21
fungithe only channels it got configuration removed for were #openstack-sahara, #puppet-openstack and #rdo14:21
fungiin preparation for tomorrow since we still need to get up with the current owners of those channels (as well as #edeploy) to grant us access14:22
*** openstackgerrit has joined #opendev14:22
openstackgerritRodion Gyrbu proposed zuul/zuul-jobs master: Bumb golang version
fungiyoctozepto: ^ it's still working14:24
yoctozeptofungi: yay14:24
cenn~offtopic: can I ask what the bot uses underneath? As in what's it built on.14:24
openstackgerritRodion Gyrbu proposed zuul/zuul-jobs master: Bump golang version
cennthanks corvus14:26
corvuscenn: based on python 'irc' library and
cenngotcha. thanks for the tip. re: irc library14:28
openstackgerritRodion Gyrbu proposed zuul/zuul-jobs master: Bump golang version
clarkbfungi: I'm around and able to help review, run commands on servers, etc just let me know what i can do to help14:59
*** mlavalle has joined #opendev15:11
clarkblooks like there are some new patchsets on some of the changes. /me tries to review those15:18
clarkbfungi: the most recent updates to are really helpful, thanks15:21
clarkblooks like that was the only one I needed to review too15:22
*** sboyron has joined #opendev15:36
fricklerfungi: o.k., so anything not too late, like earlier than 18:00 UTC would work for me, it would just be good to know when it will be a couple of hours before the event15:38
fungifrickler: i could probably do as early as others are available too15:42
clarkbfrickler: fungi: I'm happy for early too as the openinfra live event has shifted my sleep schedule and I'm up early the last couple of days15:42
clarkbI'll have to be on the laptop instead of at my desk so others can sleep but that isn't a big deal15:43
clarkbmaybe as early as 1300UTC?15:43
fricklerclarkb: that would be pretty great for me15:44
fungii could do 14:00 or even 15:00 so as not to rush you on a saturday15:44
clarkb1400 would be better for sure and can do that without much trouble15:44
fricklero.k., then 14 it is15:45
clarkbsee you then :)15:45
fungiinfra-root: i've run disable-ansible on bridge15:45
fungii'll give it a few and then go ahead with the wip changes for accessbot/channels15:45
fricklero.k., see you tomorrow, then15:46
fungithanks frickler! have a great evening15:47
fungiokay, approving the remaining accessbot changes15:49
fungiand changing the creds for accessbot on bridge next15:50
corvusi'm going to be afk for a while today, but will be around tomorrow15:51
*** marios has quit IRC15:52
fungiand that's done15:52
fungithanks corvus! i don't expect problems today, accessbot should be generally non-user-impacting15:52
fungionce those merge and we have an updated accessbot image, i'll pull it on eavesdrop and if necessary manually update the copy of the channel list and config there, then run it and monitor the log for signs of trouble15:55
fungior i guess i could just unlock ansible at that point and watch it run once it fires15:55
fungiwhich might be easier15:55
clarkbfungi: could you get away with manually running some playbooks instead? I think service-eavesdrop.yaml ?15:55
clarkbfungi: you don't need to unlock ansible, you can run it yourself instead15:56
fungigood point15:56
clarkbthat is what I did with some recent chagnes and is nice because it gives you the control and logging there without zuul running away with it15:56
fungiinlaws are supposed to be arriving any minute so i'll probably have to step away for a bit shortly15:56
clarkbok, I should go eat a bit more breakfast then15:57
openstackgerritMerged openstack/project-config master: Switch the IRC access check to OFTC
openstackgerritMerged openstack/project-config master: Add channel-specific options example for accessbot
openstackgerritMerged opendev/system-config master: Temporarily drop non-admins from statusbot
*** fressi has joined #opendev16:27
*** fressi has quit IRC16:30
openstackgerritGhanshyam proposed zuul/zuul-jobs master: DNM: testing 791085
openstackgerritMerged opendev/system-config master: Retool accessbot for OFTC
fungiokay, now to make sure we get a new container image16:46
fungioh, right, we don't promote tags for that so just whatever the gate job uploaded is what we'll use16:54
fungiare we missing a promote job?
*** andrewbonney has quit IRC17:10
clarkbfungi: seems it is only in system-config17:11
fungithat was a system-config change17:12
clarkbit should run when docker/accessbot/ changes which it did17:13
clarkbmight need to check the zuul logs17:14
clarkbI'm not seeing anything obvious for why that job didn't run17:14
clarkbit should run in the deploy pipeline17:15
clarkbok I think I get it17:15
clarkbfungi: it isn't running because there are changes enqueued ahead of it in deploy and they are all just doing not much because of the ansible lock file :/17:16
clarkbfungi: we can maybe put eavesdrop in the emergency file, then undo the ansible disable lock file and let things flush, then remove eavesdrop from the emergency file, put the lock back then run things manually17:16
fungiyeah, good call, i'll do that now, thanks!17:23
fungiokay, that's done, will wait for them to run (i'm in no hurry)17:24
fungikinda strange the image tagging would have been blocked though, opendev-promote-docs ran for it already shortly after it merged17:28
fungi(says it was in the promote pipeline)17:28
fungithat change wouldn't have been enqueued twice in promote, i don't think?17:29
clarkbthey are different pipelines17:30
clarkbthe image is promoted in the deploy pipeline which is affected by the ansible lock17:30
clarkbthe promote pipeline is not17:30
fungigot it17:45
fungielsewhere we tag images from the promote pipeline, i guess17:46
clarkbI think it is mostly in the deploy pipelien to ensure we update the docker image first then update the deployments17:46
clarkbit updated the latest tag btw17:46
clarkbso now I think you can hit disable ansible again, remove eavesdrop from the emergency file then run the service-eavesdrop.yaml playbook?17:46
*** openstackstatus has quit IRC17:47
*** openstackstatus has joined #opendev17:47
*** ChanServ sets mode: +v openstackstatus17:47
fungiyep, on it17:48
fungiokay, swapped from emergency list to disable-ansible17:51
fungihave something coming out of the oven in a few minutes and then i can try running the eavesdrop playbook17:54
clarkbsounds good17:54
clarkbI'm going to grab some lunch now too18:05
*** slittle1 has left #opendev18:12
fungiso command line would be something like this i guess:18:17
fungisudo ansible-playbook --limit -v /home/zuul/src/
fungior should i do base first?18:17
clarkbin the past I have done base first because I was adding new servers18:21
clarkbin this case because it is an existing server I don't think base is necessary18:21
clarkbalso I think you can leave off the --limit because that playbook sets up puppet on bridge too18:21
clarkbbut probably fine either way18:22
clarkbfungi: ^18:22
fungiahh, yeah18:24
fungimy main concern is making sure project-config is updated before it tries to run accessbot18:25
clarkbservice-eavesdrop runs sync-project-config18:26
fungioh, so it does18:29
fungiokay, so i'll run that and then check the log18:29
fungisudo ansible-playbook -v /home/zuul/src/
fungiis what i'm running18:29
clarkbthen I'm not sure if it runs accessbot on demand or in a cron or what18:30
fungiit gets run by the playbook i think, there is no cron18:31
fungino fails or unreachables18:34
fungiit didn't actually run accessbot though18:34
fungitrying to work out where the compose file for it ends up18:34
fungioh, it didn't update the server line in accessbot.config, guess i need to solve that first18:36
clarkbfungi: system-config/playbooks/roles/accessbot/templates/accessbot.config.j218:37
clarkbI'm not seeing what triggers accessbot to run yet but trying to figure that out18:37
clarkbfungi: it is the infra-prod-run-accessbot job18:38
openstackgerritJeremy Stanley proposed opendev/system-config master: Update accessbot config to use OFTC
fungiclarkb: ^18:38
fungiwe'll need that first anyway18:39
clarkbso when you are happy with things you can run `sudo ansible-playbook -v /home/zuul/src/` I think18:39
fungiawesome, thanks18:39
clarkball that does is run `/usr/local/bin/accessbot` if you want to do it more by hand18:39
fungiwhich explains why there's no compose file18:40
fungii guess that pulls the image automatically when run as well?18:40
fungiopendevorg/accessbot   latest              6d8d7d0b9760        2 hours ago         125MB18:41
fungiaccording to `sudo docker image list` i guess it's updated18:41
clarkbfungi: I don't think so, it just runs docker run?18:41
clarkbya so we must update the image with the playbook you already ran18:42
clarkbthen the playbook to run the tool just runs whatever that is18:42
clarkbfungi: is there a change for gerritbot to not sasl yet?18:42
fungiyeah, i wasn't sure if docker had extra magic to pull images since it references them like
fungino, haven't edited any of the bots yet, for gerritbot i think we should be able to find the sasl implementation in its git history and unwind it to go back to using identify18:43
clarkbya I'm looking at that now. I think we may want a aprtial revert because forcing ssl seems like a good idea18:43
fungiworth checking meetbot and statusbot as well to see if we need to adjust anything other than configuration18:44
clarkbI think meetbot is fine, just configuration18:48
*** hamalq has joined #opendev18:50
clarkbgit revert handles this very oddly. I might be better off just doing it by hand18:52
fungistatusbot should be similar to meetbot18:52
fungiif we can easily make gerritbot support both modes, that could also allow us to easily switch back to sasl if the oftc admins get around to finishing the sasl implementation there (faq claims it's planned)18:53
fungishould be possible to run a second gerritbot temporarily to test, with the creds in the password file18:53
clarkbI'll take a look at it now18:54
fungii can also hack on it over the weekend if necessary18:54
*** slaweq has joined #opendev18:57
*** hamalq has quit IRC19:25
*** hamalq has joined #opendev19:26
openstackgerritClark Boylan proposed opendev/gerritbot master: Support normal auth in gerritbot
clarkbfungi: ^ I have not tested that at all19:40
clarkbfungi: but I think the general shape of that is correct. I want to look at statusbot now. Sort of constructing a todo list via these changes19:41
paladoxi've joined oftc!19:41
fungipaladox: welcome to the light universe19:42
fungiclarkb: thanks! i'll take a look shortly19:42
openstackgerritMerged opendev/system-config master: Update accessbot config to use OFTC
clarkbfungi: ^ now I think you can rerun service-eavesdrop.yaml, check that the configs updated then run the run-accessbot.yaml playbook19:48
fungiyep, will do in a bit19:50
fungireapplying service-eavesdrop now19:56
clarkbfungi: is it still going?20:03
openstackgerritClark Boylan proposed opendev/statusbot master: Add non SASL auth back to statusbot
clarkbfungi: ^ I've hit a snag on that one. Not sure how to deal with the caps issue. YOu probably have etter thoughts than I do though20:05
clarkblooking at eavesdrop I don't see the updated config file yet20:07
clarkbfungi: did the playbook run?20:08
*** jdwidari has joined #opendev20:12
clarkbfungi: I think the problem is taht system-config hasn't updated yet due to disabling ansible. You can work around this via your own checkout iirc20:12
*** jdwidari has quit IRC20:15
openstackgerritClark Boylan proposed opendev/gerritbot master: Support normal auth in gerritbot
openstackgerritClark Boylan proposed opendev/gerritbot master: Update to python3.8
fungiahh, yeah20:30
fungisorry, back and forth between computer and stove20:30
fungiclarkb: there's no identify-msg cap on oftc, so i ripped it out for accessbot, just identifying after the privmsg from nickserv saying to authenticate20:31
clarkbfungi: right but statusbot tries to confirm the auth status of those talking to it in order to determien if they have perms to do the status updates20:32
clarkbI'm not sure how to keep that behavior20:32
fungiahh, one way is to parse the response from nickserv20:32
fungiclarkb: accessbot looks for msg.startswith('You are successfully identified')20:35
fungifrom nickserv20:35
*** sboyron has quit IRC20:36
clarkbfungi: not of the bot but of the people talking to the bot. I think you can use /msg nickserv status but then you need to redo the state machine in the bot to do lookups after getting requests20:37
openstackgerritClark Boylan proposed opendev/system-config master: Assort IRC TODOs
fungioh, got it20:44
fungiyeah, that's a conundrum20:45
fungiwe could actively check, or we could just punt for now20:48
clarkbya actively checking should be doable just requires a bit more understanding of the bot framework than I've got now (to understand how to set up a callback for the actual update action should verification succeed)20:49
fungii'd be okay for now with not checking and assuming the nicks listed there are registered with enforce on20:50
clarkb"with enforce on" ?20:50
fungiit's not ideal, but the risk is low20:50
fungi /msg nickserv set enforce on20:50
fungii think that's the syntax anyway20:50
fungibasically asking nickserv to kick anyone who tries to use your nick while you're not around if they don't identify20:51
clarkbah, it appears that isn't a default but /me toggles it now20:51
fungithere's still a brief window where they could join a channel and ask statusbot to do something, but that does involve finding a time when that nick is not already in use20:51
fungior exploiting a netsplit where the attacker and statusbot are on the other side of the split from nickserv20:52
fungiand from the normal nick user20:52
clarkbmakes sense. will need to be updated to support that but I've got to pop out for a bit now20:53
fungiupdating the system-config repo on bridge is not straightforward, since zuul normally pushes to it and there's no remote20:56
fungibut at this point i'm inclined to just unleash ansible on it, the credentials are updated so if accessbot gets rerun before the hostname update is applied it will just do nothing20:57
fungii've removed the disable file again, will wait for it to run and check back in a bit20:58
clarkbfungi: ok. fwiw ywhat I do is I update a checkout in my homedir20:58
clarkbthen you do ansible-playbook -v /home/foo/system-config/playbooks/service-eavesdrop.yaml. The downside to this is that host and group vars are not used from your local checkout that way since they are dfined in specific locations but the roles and playbooks are relative to each other so that should work20:59
*** hamalq has quit IRC21:01
*** hamalq has joined #opendev21:02
clarkbfungi: looks like the file udpated and the play is about over. The next job should run it for real21:03
*** hamalq has quit IRC21:23
*** hamalq has joined #opendev21:23
*** hamalq has quit IRC21:38
fungiyeah looks like it worked? i see the right access list and mlock for #opendev now, where it was incomplete before21:42
fungiwill check things more thoroughly here in a moment21:43
fungiokay, that seems to have worked as desired21:56
fungii'm going to shift gears to replicating channel topics21:56
*** lourot has quit IRC22:00
*** lourot has joined #opendev22:01
*** dpawlik has quit IRC22:15
*** dpawlik has joined #opendev22:15
*** dmellado_ has joined #opendev23:06
clarkbfungi: anything I can do to help with topics?23:06
fungiconveniently i can use /list #channame to get topics on channels whether i'm in the or not23:06
fungiso just doing it the lazy way23:07
*** dmellado has quit IRC23:07
fungipasting the /list commands, then scrape the results out of my client log23:07
*** dmellado_ is now known as dmellado23:07
clarkbif you'd like I can do a subset manually too. Not sure if that makes it easier23:08
fungiwe'll see if they thwap me23:09
fungii'm about to just try the full batch in one go23:09
openstackgerritClark Boylan proposed opendev/statusbot master: Add non SASL auth back to statusbot
clarkbfungi: ok. Also ^ I went ahead and updated that to rip out the cap stuff23:14
fungioh, cool23:14
clarkbI still haven't tried connecting either of those bots to an irc network without sasl though23:14
clarkbnot sure I'll get to that today23:15
clarkbif someone else wants to give it a go feel free (thinking maybe frickler may want to given the accessbot fixups frickler did)23:15
fungithe /list calls seem to not be getting blocked so far23:20
fungiokay, i have a text list of channels with their topics23:23
fungithat went fairly smoothly23:23
fungijust need to rework it into a list of messages to chanserv now23:23
clarkbwith the power of sed!23:24
*** tosky has quit IRC23:24
clarkbI'm going to need to help (consume) dinner shortly. Anything I can look at in the next little bit or are we good for now and see you tomorrow?23:41
*** mlavalle has quit IRC23:46
*** irclogbot_1 has quit IRC23:53
*** irclogbot_1 has joined #opendev23:56
funginah, i think i've got the topic work under control anyway23:57
fungithanks for the help, and enjoy dinner!23:57
fungii'm mainly just poking at this off and on during breaks from entertaining guests here23:58

Generated by 2.17.2 by Marius Gedminas - find it at!