Saturday, 2021-05-29

fungihave to filter out the subset which had no topics, as /list generates an error in some of those cases (when nobody is joined to the channel)00:00
paladoxDinner was nice for me :P (had papa john’s)00:01
fungithat exists in the uk?00:01
paladoxWe have dominos and Pizza Hut too00:01
paladoxPizza Hut is a big eat in place00:01
fungii remember my first wife insisting on eating at pizza hut at one point on a trip there. i really didn't understand00:02
paladoxThough they have delivery shops too00:02
paladoxPizza Hut here is really nice. Buffet is cheap00:02
fungii'm much more about steeping in the local fare wherever i travel. eating american chain food outside the usa (or even in the usa) doesn't appeal to me at all00:02
paladoxI like the Toby carvery (eat there weekly, nice roast dinner)00:03
fungibut i suppose if i lived in the uk, eating american chain food would be a different story00:03
paladoxIt’s a chain but it’s U.K. based00:03
paladoxThey do the pizzas over here differently. Tastes different I think.00:04
paladoxAlso we have extra toppings like sweetcorn00:04
fungiinteresting, i would eat corn on pizza. i should make that sometime. i eat more homemade pizza than restaurant anyway00:05
paladoxIt tastes nice00:05
paladoxSubway also has sweetcorn here00:06
paladoxI like the cheese at subway. Had the us one and it didn’t taste as nice.00:06
fungiwell, cheese is done much better on that side of the atlantic00:08
fungiokay, so we have 94 channels with topics00:08
fungiworking on scripting the replication of those now00:08
paladoxWendy is also apparently coming to the U.K.00:10
paladoxAnd also Taco Bell is here (I wouldn’t eat there tho)00:10
paladoxThere’s dunkin donuts but that doesn’t seem as widely available yet00:12
fungiin the future, all restaurants will be tack bell. at least if you believe repo man00:12
fungier, taco bell00:12
*** cloudnull has quit IRC02:37
*** prometheanfire has quit IRC02:45
*** prometheanfire has joined #opendev02:45
*** icey has quit IRC06:02
*** iurygregory has quit IRC07:16
*** tosky has joined #opendev08:04
openstackgerritRadosław Piliszek proposed openstack/project-config master: Add yoctozepto as IRC op
*** tbarron has left #opendev11:44
*** iurygregory has joined #opendev12:41
openstackgerritAndreas Jaeger proposed openstack/project-config master: Remove AJaeger from accessbot
openstackgerritMerged opendev/gerritbot master: Update to python3.8
openstackgerritAndreas Jaeger proposed openstack/project-config master: Remove AJaeger from accessbot
*** openstackgerrit has quit IRC13:47
fungii should be around in a few minutes to start merging stuff13:51
* frickler is here, too13:53
*** openstackgerrit has joined #opendev13:55
openstackgerritMerged openstack/project-config master: Remove AJaeger from accessbot
clarkbI'm around as well13:59
clarkbthough trying to get gears spinning13:59
fricklerI'm going through 793648, just one nit so far13:59
clarkbfrickler: fungi  re ^ I'm not sure what will happen with gerritbot and accessbot when that lands. I suspect meetbot will be fine thoug hand that is a good start14:01
fungiokay, sorry about that, i'm back now14:04
fungidoes anyone recall whether the ssl+ipv6 bug in twisted-python only impacts meetbot, or statusbot as well?14:05
fungitrying to figure out where we should substitute oftc's v4-only round-robin hostname14:05
clarkbfungi: its supybot specific because it uses twisted14:07
clarkbonly meetbot is supybot iirc14:07
fungiahh, okay yes14:07
fungistatusbot just inherited the /etc/hosts override by virtue of running there and not in a container14:07
clarkbfwiw I think meetbot is the one we should worry about first since it is expected to work14:08
clarkbstatusbot and gerritbot will likely be more drawn out14:08
fungiyeah, i concur, and then we can switch our coordination discussion to the new place14:08
clarkbfungi: feel free to take relevant bits out of my assortment change to make specific changes that are useful at points in time14:09
openstackgerritJeremy Stanley proposed opendev/system-config master: Assort IRC TODOs
fungiclarkb: oh, okay yeah we can split it up i guess14:14
fungior we could block ansible for eavesdrop and then make local edits one bot at a time and restart14:14
clarkbfungi: both wfm14:15
fungishould we try manually copying 793643 into the deployed container image, or merge it and let an updated image get built then troubleshoot that directly?14:17
clarkbI really don't have much confidence in those bot changes as I wasn't able to run either of them14:18
clarkbit might be better to try manually applying it for easy rollback?14:18
fungiokay, can give that a shot, so i guess i'll put eavesdrop in the emergency file first14:19
*** tosky has quit IRC14:25
clarkbya I gues sput eavesdrop in emergency, update meetbot config and restart it, then land a change to reflect that if it is happy?14:30
clarkbthen look at probably statusbot then gerritbot?14:31
fungiyeah, i added eavesdrop to emergency list a few minutes ago14:31
fungigetting ready to update the meetbot nick, identify password and hostname in its config14:31
fungiwas just double-checking some weirdness frickler mentioned over in oftc14:32
clarkbfungi: you also need to substitute the server value which it uses to key other values14:33 = bar14:33
clarkbthe servername14:33
fungilooks like /var/lib/meetbot/openstack.conf is the config on disk14:34
clarkbfungi: FreeNode is the existing servername14:37
fungiyeah, i'm adjusting that14:38
*** openstack has joined #opendev14:55
*** ChanServ sets mode: +o openstack14:55
clarkbfungi: if the ansible run has started before your edits of that file then the edits are not seen14:56
fungiyeah, that may be what transpired14:56
fungii'll redo them14:56
clarkbfungi: no I see why that happened14:56
fricklerbut this may also hint to what is's waiting for: Received "Password accepted" from NickServ on FreeNode.14:56
clarkbyou need to use the inventory name not the cname14:56
clarkbI'll fix the emergency file14:56
clarkbemergency.yaml update to eavesdrop01.openstack.org14:57
fungiyep, i forgot we rebuilt that server at one point14:57
fungithanks :/14:58
fungibut yeah, this was at least a good confirmation that the same errors for users.conf permissions and so on are there with the old config14:58
*** opendevmeet has joined #opendev15:14
fricklerjust mving the dir should be safe, though?15:14
fungii'm backing it up to my homedir separately just in case, but yeah15:14
clarkbya should be safe I'm not super concerned15:15
clarkblooks like opendevmeet is back after the mv ?15:15
clarkbnow we just need to update the symlink in /srv/meetbot-openstack?15:16
clarkbI'm taking notes at
*** slaweq has quit IRC15:18
clarkbwhy don't we split the exising change up that way we can get thorugh this in better ite sized pieces15:19
clarkbshould I start writing an update for meetbot?15:19
fungiyep, that sounds good. please do15:19
fungifrickler: can you propose the code change, i guess that will have to be separate15:20
clarkbwe use supybot out of packages so that may be "weird"15:20
clarkbmaybe we just vendor that entire file in the puppet module and have it write it in place15:20
fungioh, this is a mismatch in packaged supybot?15:21
fricklerI was just checking, latest upstream release contains my fix
fricklerbut also it has been archived in 201815:21
clarkblatest upstream is not really active though. But perhaps we could install using pip or something15:21
fungiyeah, limnoria is almost certainly also updated15:21
fungiwe could also just shadow that bit in meetbot maybe?15:22
fungion a positive note, nothing is likely to undo the local edit to that file since odds of a new supybot package entering xenial are slim, so it's not urgent we have it solved this weekend15:23
fungiit's more of a concern when we replace/upgrade eavesdrop15:23
clarkbfungi: hiera('openstack_meetbot_password') needs to be updated15:24
frickleroh, that from a deb pkg15:24
fungifrickler: yep15:24
fungiclarkb: yes, working on it, thanks15:24
frickleryeah, I think we can leave it as is for now, then, just need to remember when we upgrade to bionic or newer. though hopefully that would have th updated upstream version, then. that file was last changed 9 years ago according to github15:25
fungiclarkb: done and committed15:25
fungifrickler: oh, great point, the version in bionic/focal is almost certainly fixed for this15:26
fungiso maybe we don't worry about patching separately, but i'll double-check what's in bionic to confirm15:26
clarkbremote: Swap meetbot networks15:26
fricklerthat fix even is from 2011
clarkbI think that may be all that is needed to meetbot. That should ensure the link target too15:27
*** slaweq has joined #opendev15:27
clarkbwe do need to followup on the irc2logs cron which I made a note about in the etherpad too15:27
*** Dragnslcr has joined #opendev15:29
fungiclarkb: do we need any config management updates to create the symlink correctly or is that also templated based on the site name?15:29
clarkbfungi: its already templated one sec I will get a link15:29
*** Dragnslcr has quit IRC15:29
* frickler will step away for a couple of minutes15:30
clarkbya I need a break as well15:30
fungiclarkb: want me to get the puppet-meetbot change for the cron or do you have it in progress?15:30
fungioh, the cron actually uses ${varlib}/logs/ChannelLogger/${network} in puppet so should just work?15:32
fungii'll fix it up manually on the server for now until we get 793657 merged15:33
fungiit's /var/lib/meetbot/openstack/logs/ChannelLogger/OFTC in the cronjob now15:34
clarkbyup the cron should be handled by the change15:39
clarkbI think that change may cover everything but the supybot update15:39
clarkbfungi: are you running the xz?15:41
fungithat's tar making a compressed backup of the channel logs15:42
clarkbnext up statusbot and/or gerritbot?15:42
fungiyeah, i'm just checking the supybot version in bionic first15:43
fungiokay, bad news... the supybot package in bionic has the same strings as xenial's, and supybot was removed from ubuntu before focal, so we'd probably need to make a patched supybot package as a stopgap until the limnoria rewrite15:47
clarkbfungi: or install it with pip?15:47
fungiyeah, that could be an option15:47
clarkbalso need python2.715:48
fungiokay, so i think we can consider meetbot complete for now with the approval of 79365715:49
fungistatusbot should be the easiest one to tackle next, i guess?15:49
clarkbI think so15:50
clarkbgerritbot deals with a lot more state15:50
clarkbbut the updates to both are very similar as far as enabling ssl and adding normal nickserv auth back in15:50
*** Guest92 has joined #opendev15:50
*** Guest92 has quit IRC15:50
fricklerdo we have some plan about the meetbot rewrite yet? I just added a note about it to the xenial etherpad15:54
fungifrickler: there's an infra spec15:54
*** jentoio has joined #opendev15:55
fungiokay, so as tested in #opendev-meeting, meetbot isn't changing the channel topic. i believe that's because supybot relies on +o instead of talking to chanserv like statusbot does, so need to figure out a good workaround for that15:55
clarkbwas meetbot just chanop before on freenode?15:55
fungiwe could call this a low-priority feature gap and save it for later15:55
clarkband ya I think that is a relatively minor thing in the overall swap15:56
fungiyes, because we had different granular mode masks for acls on freenode we could set +O on meetbot's acl and it would be auto-opped when joining that channel15:56
fungiokay, i'll make myself a reminder to include that note in wrap-up communication15:57
*** jonathon13 has joined #opendev15:58
*** jonathon13 has quit IRC15:58
fungithat seems to have just started today15:59
frickleriiuc we could only set autoop to on for the whole channel, that would get admins opped, too.15:59
fungii've been in here since at least april with no sign of spam16:00
clarkblooks like +Mz is the option you were talking about fungi?16:00
fungisimilar spam hit #openstack for the first time a few minutes ago16:01
corvusi like avoiding having admins opped under normal circumstances and banning through chanserv instead16:01
fungisad being on oftc doesn't sufficiently insulate us from the war between freenode and libera16:01
clarkbI think the issue is they are all single use connections?16:01
clarkbbut ya we can start with that and see if we can make it work16:02
fungiright, they seem to part the channel before we even have time to type up the akick entry16:02
fungibut maybe it won't persist16:02
fungianyway, looking at statusbot next16:02
fungii've directly edited the statusbot config on eavesdrop to change the nick, password and hostname16:06
fungiand restarted the service16:06
clarkbfungi: did you update the code too?16:06
fungino, right, we need the version without sasl auth set16:06
fungigrabbing that now16:07
*** shadowdaemon has joined #opendev16:08
frickler"docker ps -a" lists a lot of 6 month old stopped containers, can those be cleaned up?16:09
frickleralso there's a second accessbot running with an untagged image version?16:10
clarkbfrickler: looking at them with --no-trunc I think so16:10
clarkbI think that may have been related to the gerritbot dockerization work and debugging why it struggled with that16:10
fungioky, i've overwritten /usr/local/lib/python2.7/dist-packages/statusbot/ with the version from 79364516:11
fungialso added use_sasl=False to the config16:12
clarkbit should be the default but ++ to being explicit16:12
fungiwell, copied it from the config in that change16:13
fungihowever, looks like the process is dying right away when i try to start it, will likely need to try again in foreground outside systemd16:14
clarkbfungi: it should be in docker?16:14
clarkbI don't think systemd will do anything?16:14
*** yoctozepto is now known as Guest9616:16
fungioh, did we move it to docker? oops16:16
*** yoctozepto has joined #opendev16:16
*** xfactor[m]1 has joined #opendev16:16
*** Guest96 has quit IRC16:16
*** xfactor[m]1 has quit IRC16:16
clarkbmaybe it didn't I thought it had because gerritbot did16:17
clarkbtrying tosort that out now16:17
fungino, i don't see any compose file for it at least16:17
clarkbya looking at service-eavesdrop accessbot and gerritbot have been converted but not the others16:17
clarkbsorry about that16:17
corvusthere are 2 unnamed containers running on eavesdrop16:18
clarkbcorvus: Isee one? looks like an old accessbot16:19
corvusclarkb: jolly_wescoff has been running 1ad27e44cc81 for 7 weeks16:19
fungithat's the statusbot traceback16:19
clarkbcorvus: ya that is an accessbot container according to the command16:20
fungiis jolly_wescoff a randomly-generated container name? is that something docker does when you don't give a name?16:20
clarkbI suspect that one can be cleaned up?16:20
clarkbfungi: ya16:20
corvusfrickler: yes16:20
corvussorry tabfail again16:21
corvusso there are 2 old access bots running?16:21
clarkbcorvus: I see a current accessbot and an old accessbot16:21
corvusclarkb: the accessbot that started 23 hours ago is current?16:22
clarkbcorvus: aiui yes16:22
fungithat one may have been a stuck accessbot run during credential change16:22
corvusdoesn't accessbot terminate when it's done its work?16:22
clarkboh I see what you are saying16:22
clarkbsorry I was confused because you said there are two unnamed ones16:22
corvuslike, i would think any accessbot older than 45 minutes is probably dead16:23
clarkbthe newre one that is stuck is named16:23
corvusyeah, that one is unnamed too16:23
corvusi eman okay it has a name16:23
corvusbut "happy_cori" doesn't say "accessbot" to me16:23
clarkbI see and ya maybe that is stuck trying t oconnect with the bad credentials. I think both can be stopped16:23
corvusokay i will clean those up16:24
clarkbfungi: that traceback is weird becuase channels was a previously provided init arg16:24
fungiyeah, i'm still trying to untangle that relative to the source16:24
clarkbfungi: ok channels was not supplid when we didn't do sasl16:25
clarkbwhen we added sasl we started providing it. Maybe the sasl mixin is what handles that channels list?16:25
*** jentoio has quit IRC16:27
*** jentoio has joined #opendev16:27
fungiclarkb: oh, yes i think that was related... looking back at the old sasl addition change now16:28
*** jentoio has quit IRC16:28
fungilooks like i cribbed the sasl implementation from for ptgbot16:32
fungiand indeed it similarly added the channels list16:33
clarkbremote: Add non SASL auth back to statusbot16:33
clarkbthat is a bit of a brute force but maybe that will be happier?16:33
clarkbas a heads up the kids are now old enough to understand what a 3 day weekend is and I'm going to have to pop out soon for some activities in the sun16:34
fungiokay, so previously we iterated over self.channel_list looks like, after connecting. the sasl mix-in allows you to pass a channels list instead in the constructor for auto-joining channels, so the join loop was cleaned up16:34
clarkbgerritbot is likely to suffer a similar problem since it is very similar to accessbot here16:36
clarkber statusbot16:36
clarkb(too many bots)16:36
fungiclarkb: did you mean to remove the on_welcome method entirely, or just partially?16:37
clarkbfungi: its only purpose was to handle identity-msg cap which we don't support anymore here16:38
fungistray added paste maybe?16:38
*** jentoio has joined #opendev16:38
clarkboh wait I see what you mean16:38
fungiokay, i'll remove the def line for it16:38
clarkbyes that is broken16:38
clarkbI can update the change16:38
fungiident_password seems to be an unexpected keyword argument as well16:39
fungii'll try taking it out16:39
clarkbfungi: yup that isn't in the old change iether I'll do another update :)16:39
clarkbok latest ps should hvae those two fixes for extra on_welcome and the ident_passwd16:40
*** opendevstatus has joined #opendev16:40
fungiwelcome, opendevstatus!16:41
corvus#status foo16:42
fungii'll double-check your latest patchset matches the edits i'm running with there16:42
fungidebug log says it's still joining channels16:42
fungibut hopefully it'll respond as soon as it's done16:42
opendevstatuscorvus: unknown command16:43
corvusok.  that should either emit an error to this channel, or log a message that it's ignoring me16:43
corvusgood :)16:43
fungiand clarkb's latest patchset does indeed match what is running there16:43
clarkbI'm working on a gerritbot update for similar fixes16:44
fungii'll get the private hiera updated16:45
fungier, hostvars16:45
clarkbremote: Support normal auth in gerritbot16:45
fungiokay, private hostvars have been updated for statusbot and gerritbot16:48
fungii'm going to restart statusbot so it's running under systemd instead of non-daemonized16:50
*** opendevstatus has quit IRC16:50
*** opendevstatus has joined #opendev16:51
fungilooks like i should be able to try patching your altered in over top of /var/lib/docker/aufs/mnt/8a4b37c735b5d08cd2e8794c8250cb758c0e8e4d0f27752cc3303e205678de12/usr/local/lib/python3.8/site-packages/gerritbot/ and then restarting the container16:52
clarkbfungi: that sounds about right though I haven't confirmed any of the specific paths there16:53
fricklerok I'm out for now, will try to check later or otherwise tomorrow16:53
*** mobomelter has joined #opendev16:55
*** mobomelter has quit IRC16:55
fungithanks for all the help frickler!16:56
fungigerritbot's been restarted with updated config and the code from 79364316:59
clarkbfungi: doesn't look like it started16:59
fungiyeah, i'm trying to find where the startup would have been logged16:59
fungiclaims it's logging to syslog, but maybe not if it never starts17:00
clarkbI see a traceback there around my config loading change I think looking at it17:00
clarkbsyslog parts it out to /var/log/containers17:01
*** slaweq_ has joined #opendev17:01
fungiaha, yeah there's a traceback17:01
clarkbI think this may be a python3 change17:01
clarkband we sasled on python2 and now my configparser revert is bad?17:02
fungiraise TypeError("option values must be strings")17:02
clarkbconfig = configparser.ConfigParser({'server_password': None}) on that line17:02
fungi'' instead of None maybe?17:02
*** slaweq has quit IRC17:03
clarkbI think python3 configparser takes a defaults parameter now instead17:03
clarkbconfig = ConfigParser.ConfigParser({'force_ssl': 'false', 'server_password': None}) is the python2 pre sasl code17:04
clarkbhrm the python3 example does what I'm doing but without None17:04
corvusmaybe we need an empty string now?17:05
clarkbmy concern with using empty string is that the irc lib may try to supply that as the server password17:05
clarkbinstead I think I can just check if the section exists similar to how use_sasl is handled17:05
fungiit does at least run with that17:06
fungiMay 29 17:05:56 eavesdrop01 docker-gerritbot[1386]: TypeError: connect() got an unexpected keyword argument 'channels'17:06
fungitrying it with the channels=channels, line removed from the nosasl class constructor17:07
*** opendevreview has joined #opendev17:07
opendevreviewClark Boylan proposed opendev/gerritbot master: Support normal auth in gerritbot
fungiokay, so aside from merging some changes and unleashing ansible back onto eavesdrop i think we're in a fairly good place now?17:08
opendevreviewClark Boylan proposed opendev/gerritbot master: Support normal auth in gerritbot
fungiand sending out an update to the ml, which i can do in a little while if things seem stable17:09
clarkbfungi: ^ maybe try that version?17:09
clarkb++ to fairly good plcae now17:09
fungisure, will do17:09
fungijust a sec17:09
*** opendevreview has quit IRC17:09
fungistarting up now on patchset #517:10
fungiMay 29 17:11:01 eavesdrop01 docker-gerritbot[1386]: 2021-05-29 17:11:01,918 INFO gerritbot: Identified with IRC server.17:11
fungilooks like that one's all set, will review again17:11
yoctozeptomorning infra17:12
clarkbfungi: the gerritbot logs show that it may be double logging things which is annoying but not the end of the world17:12
yoctozeptoI wonder how spambots harvested our channel names17:12
clarkbI don't understand why17:12
clarkbyoctozepto: they are listable on the server17:13
clarkbI doubt it is anything sophisticated, but I wouldn't know as I didn't do it17:13
yoctozeptoas they hit many #openstack- and #zuul yet not non-opendev-related ones17:13
clarkbthey may sort by nick count too17:13
*** opendevreview has joined #opendev17:15
opendevreviewClark Boylan proposed opendev/system-config master: DNM testing gerritbot
clarkbfungi: ^ps5 seems to work17:15
fungithanks for the help!17:16
clarkbok kids are asking when they can enjoy the weekend now.17:17
*** supastupid has joined #opendev17:17
yoctozeptothose other channels have +s mode set17:17
yoctozeptohmm, I wonder if it's targetting whois17:17
*** supastupid has quit IRC17:17
yoctozeptowell, I can't argue with that folk's nick!17:18
clarkbyoctozepto: yes I doubt its anything sophisticated. ask the server for listings then go17:18
clarkband ya we could try +s17:18
*** jentoio has quit IRC17:18
clarkbor +Mz etc. corvus indicated a desire to see before we jump at any options though17:18
opendevreviewJeremy Stanley proposed opendev/system-config master: Assort IRC TODOs
yoctozepto+s is what debian folks seem to use17:19
fungithat's rebased now on top of the meetbot changes17:19
yoctozepto+nst precisely17:19
yoctozeptowe have +ntc17:19
fungiyoctozepto: good point, we can add +s to the default mask in accessbot/channels.yaml rather easily17:19
yoctozeptoyeah, it should not bother legitimate users17:20
yoctozeptoperhaps will just shut down this crappy spam17:20
fungithe +s is for spam! ;)17:20
clarkbfungi: before I pop out is there anything else I can do to help? or are we satisified for now?17:20
yoctozeptofungi: hopefully not asking for more :D17:20
fungiclarkb: i think we're good. i'd like to go ahead and merge the outstanding changes and then try to reenable ansible for eavesdrop but i can handle that17:21
clarkbfungi: sounds like a good plan. I would double check the docker images get updated in docker hub too as that was a good check last time17:22
clarkbI'll try to check back in later today17:22
yoctozeptowhich bot logs channels now?17:23
corvusclarkb: i indicated a desire to what now?17:23
yoctozeptoopendevmeet: are you the one?17:23
yoctozeptoopendevstatus: or is it you?17:23
yoctozeptoopendevreview: because you are for gerrit stuff17:23
clarkbcorvus: to not set +Mz and try to use normal kickban procedures17:23
clarkbyoctozepto: meet17:23
*** jentoio has joined #opendev17:23
yoctozeptoclarkb: thanks!17:23
* corvus looks up Mz to see what i said17:24
yoctozeptoM - client may speak only when registered and identified to NickServ17:24
yoctozeptoz - messages that would otherwise be blocked (banned, quieted, channel moderation) go to channel operators (shown as <nick:@#channel> msg)17:24
corvusyoctozepto: i'm looking at the docs, thanks :)17:24
yoctozeptoleaving here for the sake of discussion :-)17:24
yoctozeptoalso mine s17:25
yoctozeptos - secret, channel will not show up in /whois unless clients share the channel17:25
yoctozeptoI dare you to try it first17:25
clarkbya I think +s will be less disruptive17:25
clarkband a good first step17:25
corvusclarkb: yeah, i mean, i think we were hoping oftc would have less spam, but if it doesn't, we'll have to set those17:25
yoctozeptoso, can we set +s now and check how it goes?17:26
clarkbcorvus: ++ and setting the least invasive (+s) first makes sense and we can monitor how it goes and update accordinly17:26
clarkbyoctozepto: you can propose that chnage if you'd like. It would probably help17:26
yoctozeptoclarkb: ok, looking where17:26
clarkb there17:27
clarkbsorry had to find it myself17:27
corvusi spent quite some time making #zuul show up in the matrix room directory, +s will undo that, but even so it's still the least disruptive and a good place to start17:27
corvus(just wanted to indicate that it's not zero-disruption; having it unset has a legit purpose)17:27
clarkbalright I better afk now beore the kids figure out how to disable the internets at home17:28
clarkbI'll hcekc back in again later17:28
opendevreviewRadosław Piliszek proposed openstack/project-config master: [OFTC] +s channel mode
opendevreviewMerged opendev/statusbot master: Add non SASL auth back to statusbot
yoctozeptocorvus: matrix works on whois?17:30
opendevreviewMerged opendev/gerritbot master: Support normal auth in gerritbot
yoctozeptoI mean, for this functionality17:30
yoctozeptofwiw, this spam is unevenly distributed17:31
yoctozeptoI wonder if we took all the channels that get hit by it17:31
fungi~fungi/meetbot-logs-backup_2021-05-29.tar.xz on eavesdrop is 696M according to du17:31
yoctozeptoand intersect users in them17:32
yoctozeptowho could be patient 017:32
yoctozepto* by unevenly I mean the clarkb's proposed theory of user count sorting does not seem to apply17:33
*** Carnal has joined #opendev17:33
yoctozeptoas -sdks get hit, yet not -qa17:33
fungicorvus: fwiw, the channels i set up here on oftc didn't start getting spam until a few hours ago, in particular this channel had none dating back at least a month17:33
yoctozeptosame for the others I have joined17:34
fungibut yeah, if the user count increased a few hours ago that would make sense17:34
yoctozeptothough I have a shorter timeframe17:34
yoctozeptoperhaps one of our users is the target of the evil whois17:34
corvusfungi: i /joined #oftc but no one has said anything since i joined a while ago; i'm not sure what the reporting procedure is, or if they're aware of it17:34
corvusyoctozepto: if an oftc bridged channel does not have +s set, then it will appear in the matrix room directory17:35
corvusthat's a nice thing for people using matrix to join a bridged channel; it's an extra bit of confirmation that they found the right room before they join17:36
yoctozeptocorvus: ack, thanks for confirming17:36
fungiall the spam seems to be backscatter from freenode vs libera, so maybe the spammers will get bored of telling people on oftc about it17:36
yoctozepto#openstack-tc got my opinion on that spam campaign already17:37
*** docontherocks has joined #opendev17:38
yoctozeptooh, it has now spammed in #oftc17:40
yoctozeptoI am on #oftc17:44
yoctozeptoand got their attention17:44
opendevreviewMerged openstack/project-config master: Add yoctozepto as IRC op
*** Unit193 has joined #opendev17:53
fungichecking to see if we got an updated latest label for the gerritbot container image on dockerhub now17:54
fungi says published 25 minutes ago17:55
yoctozeptohmm, I seem unable to op myself17:56
yoctozeptodon't tell me I made a typo17:56
fungiyoctozepto: it hasn't deployed yet17:56
fungii blocked ansible from running anything on the server were we do our accessbot runs, so i'm only just now checking to make sure it's safe to resume17:57
yoctozeptofungi: oh, I thought the 'deploy' pipeline was for deploying :-)17:57
fungiit is, but it didn't because we didn't let it17:57
yoctozeptooh, that makes sense now17:57
fungii suppose i could have saved that change for after i unleashed ansible on the server again17:57
fungianyway, i think we should be okay to let it run there again17:58
fungioh, nope not yet, 793648 still needs to merge17:58
fungithen i can undo it17:58
yoctozeptono rush, I was only testing17:59
paladoxHi :)18:00
*** otoburb has joined #opendev18:01
*** otoburb is now known as Guest9918:01
*** jmorgan has joined #opendev18:01
*** jmorgan has quit IRC18:04
fungihowdy! welcome to our new place, make yourself at home. don't mind the spammers, they ought to get bored soon enough18:05
*** jmorgan has joined #opendev18:05
fungithe bots are pretty much all moved over, we're basically just merging configuration management changes now to reflect what we needed for that, and then i'll be sending an update to the service-discuss mailing list with some info on where we're at with things18:06
*** jmorgan has quit IRC18:09
opendevreviewMerged opendev/system-config master: Assort IRC TODOs
fungiokay, taking eavesdrop01 back out of the emergency disable list now that's merged18:12
*** jmorgan has joined #opendev18:13
opendevreviewJeremy Stanley proposed openstack/project-config master: Revert "Accessbot OFTC channel list stopgap"
fungiupdated the commit message on that to reflect the commit id of the actual commit which ended up merging18:14
*** jmorgan has quit IRC18:20
*** jmorgan has joined #opendev18:20
*** opendevreview has quit IRC18:26
*** opendevstatus has quit IRC18:28
*** opendevmeet has joined #opendev18:29
*** adbraun[m|gr] has joined #opendev19:01
*** adbraun[m|gr] has quit IRC19:02
corvusyoctozepto, fungi: is there a decision/timetable for adding +s?19:14
yoctozeptocorvus: I proposed the change, it's now in your hands ;-)19:15
corvusjust wondering if that's something we should go ahead and try asap19:15
yoctozeptoI would19:15
corvusyoctozepto: oh, missed that; link?19:15
corvusah i see it 65919:15
yoctozeptoI'm now discussing what could be done from oftc side on #oftc19:16
yoctozeptoit would help if more of you infra folks joined there19:18
corvusthanks (and i'm following that)19:18
corvusi'm just in there with a semi-anonymous lurker account :)19:18
yoctozeptooh you, creeping on me like that ;p19:19
corvusyoctozepto: i joined before you ;)19:19
yoctozeptocorvus: well, I can't really tell :D19:19
yoctozeptobut I'm glad you have the copy of the conversations19:19
fungii've joined as well19:20
fungi(only just now)19:20
corvusbut yeah, if they have the capability to join/monitor all our channels to aid in the effort, that would be great.  and we have a convenient list of channels :)19:21
fungiindeed we do19:21
yoctozeptoI sent dwfreed earlier an example list of channels that we got spam on19:21
corvusi'll try joining with my legit account; i just wasn't sure what the traffic would be like, and i'm taking baby steps with my own homeserver (i'm currently joined with a account)19:21
fungii'd like to figure out if we can whittle them down further, but we don't log all of them so there are at least some i can't say for sure are actually used by anyone other than bots19:21
yoctozeptoand Unit193 joined there19:21
yoctozeptocorvus: ah, that's understandable19:22
yoctozeptofungi: yeah, we probably have too many channels still19:22
yoctozeptooddly, #openstack-qa is still  spam-free19:23
yoctozeptoa miracle19:23
yoctozeptoI think, +s would really help19:23
yoctozeptoit's as if someone's channel list is being targetted19:23
fungimaybe not enough people in that channel to draw attention yet?19:23
corvus +2 from me19:24
fungibut yeah, if there are channels which explicitly don't want +s we can override the mlock on a channel-specific basis quite easily19:24
yoctozeptofungi: smaller channels get hit19:24
fungii just was wary of blanketing it given it causes the channels not to appear in the matrix bridge list19:24
yoctozeptoI guess we are better off showcasing available channels elsewhere19:25
yoctozeptoand let's not forget +s might actually not help at all and we can just revert it then19:25
fungii've reset the topic on 793659 to oftc so it will appear in aggregate with our other oftc migration related changes19:25
corvusfungi: fwiw from me as a matrix champion -- right now, i'd happily trade that to deal with the spam19:25
fungisure, this should also exercise accessbot updates now that i've taken eavesdrop back out of the emergency disable list. approving19:26
yoctozeptomatrix champion, nice19:26
corvusheh, champion like "advocate" not like "victor" :)19:27
*** jpnurmi has joined #opendev19:27
fungicongratulations neo, you've conquered the matrix19:27
yoctozeptooh, don't be so shy19:28
*** jpnurmi has quit IRC19:28
*** mtreinish has joined #opendev19:30
*** fauxrensix has joined #opendev19:31
*** fauxrensix has quit IRC19:32
*** opendevreview has joined #opendev19:34
opendevreviewMerged openstack/project-config master: [OFTC] +s channel mode
fungiyoctozepto: you're a chanop in all the channels now19:38
yoctozeptofungi: thanks, let me try it out19:38
fungii think that mush have been a separate accessbot run from the +s addition19:38
fungier, must have19:39
yoctozepto"You do not have access to the OP command"19:39
yoctozeptoI must be unaware how chanop works here19:39
fungiwhich channel did you try in? in only spot checked a handful19:39
yoctozeptothis and #openstack-kolla19:40
*** benl90 has joined #opendev19:40
fungiif you `/msg chanserv access #openstack-kolla list` it does show you as having chanop there19:40
*** benl90 has quit IRC19:41
*** ChanServ sets mode: +o fungi19:41
*** ChanServ sets mode: -o fungi19:41
fungiyoctozepto: try `/msg chanserv op #opendev`19:41
yoctozeptofungi: that's precisely what i'm trying19:42
yoctozeptobut you are right about that list19:43
yoctozeptoperhaps with chanop I cannot op?19:43
yoctozeptolet me try part and join19:43
*** yoctozepto has left #opendev19:43
*** yoctozepto has joined #opendev19:43
*** ChanServ sets mode: +o yoctozepto19:43
yoctozeptoyeah, I need to part and join19:44
yoctozeptounless you changed something else...19:44
yoctozeptono, seemingly it just works everywhere now19:44
*** ChanServ sets mode: -o yoctozepto19:44
yoctozeptothen I guess it required more wait for the info to propagate in the network19:45
fungiunfortunately the mlock doesn't seem to be updated even after 793659 ran infra-prod-run-accessbot in deploy19:46
fungii know i tested mlock updates with manual runs of the new accessbot19:47
yoctozeptook, I will test +s on #openstack-kolla for now19:51
yoctozeptook, let's see if spam stops there19:52
*** jmorgan has quit IRC19:53
*** jmorgan has joined #opendev19:54
fungiwell, i think something caused the channels.yaml to not be updated19:54
fungithe /etc/accessbot/channels.yaml file on eavesdrop still has mode: '+cnt'19:55
fungiso the accessbot script is likely fine19:55
fungistrangely, the filesystem says /etc/accessbot/channels.yaml was modified at 19:43, same time as the deploy finished19:59
fungidid it copy the wrong state of that file from the git repo?19:59
yoctozeptosorry, wrong tab19:59
yoctozeptoor rather, wrong focus while typing blindly :-)20:00
fungiand ~zuul/src/ is checked out to Merge "[OFTC] +s channel mode"20:00
fungimaybe we really fail to update the accessbot channels file with that job20:00
yoctozeptowell, I somehow got my perms20:00
*** int0x1C8 has joined #opendev20:01
yoctozeptoanyhow, going to bed now20:01
yoctozeptonighty night20:01
*** int0x1C8 has quit IRC20:01
*** Ganneff has joined #opendev20:02
corvusfungi: afaict the run-accessbot playbook literally only runs accessbot20:03
corvusi don't see anything that would cause any files to update on eavesdrop20:03
corvusfungi: perhaps it should include the accessbot role before run-accessbot20:05
corvusyeah, that looks sensible to me20:05
opendevreviewJames E. Blair proposed opendev/system-config master: Run accessbot role before running accessbot
corvusfungi: ^ maybe?20:07
*** deeprave has joined #opendev20:12
*** deeprave has quit IRC20:12
fungicorvus: oh, yep, thanks!20:14
fungijuggling this with preparing a 3 lb salmon fillet20:14
*** Erimitis-t has joined #opendev20:15
fungithat completely explains what i'm seeing20:16
*** Erimitis-t has quit IRC20:16
*** agrajag` has joined #opendev20:17
*** agrajag` has quit IRC20:18
*** jmorgan has quit IRC20:24
*** jmorgan has joined #opendev20:24
*** jmorgan has quit IRC20:27
*** jmorgan has joined #opendev20:27
*** liszt24 has joined #opendev20:29
*** liszt24 has quit IRC20:29
*** slaweq_ has quit IRC20:50
*** string has joined #opendev20:53
*** string has quit IRC20:54
clarkbcorvus: fungi: I think the service-eavesdrop.yaml playbook does update project-config20:55
fungiyeah, but we don't run that in deploy for accessbot channel changes, just infra-prod-run-accessbot20:58
fungiwhich runs run-accessbot.yaml20:59
fungiso reuses the old channels.yaml already present on the server previously20:59
clarkblooks like things are mostly under control? I'm going to resume family time if so21:00
clarkband we got all the earlier changes landed looks like? so we're running with CD again? thats good21:00
opendevreviewMerged opendev/system-config master: Run accessbot role before running accessbot
fungiclarkb: yep, nothing too exciting. we'll hopefully see +s on all the channels here in a few more minutes once the waiting infra-prod-run-accessbot build in deploy finishes21:07
fungilooks like it's starting now21:11
fungilooks like it's in progress adding +s to the mlock across all the channels now21:15
*** ChanServ sets mode: +ntsc 21:19
fungiokay, that's done21:20
fungiso if we see more spam on any of the channels in the list after 21:20 utc, those are ones which either used an old channel list or are finding channels to spam in some other way21:21
fungiassuming no new developments, i'll start drafting an update to the ml after dinner21:25
Unit193+R and +M are also options, fwiw.22:43
fungiyep, but without +f sending them to someone who can walk them through registration, we have a lot of users get confused by +R22:52
fungifor many, "i need to talk to a developer on $project" is their first experience with irc22:53
fungiand lots of them seem to have trouble reading detailed instructions22:53
fungibut +Mz is certainly something we're considering if +s doesn't take care of things22:54
fungiwe'd prefer not to have to leave people opped in channels though, so that would be a bit of an adjustment as well22:55
fungianyway, i haven't spotted any more spam activity in our channels since the mass +s23:09
fungiupdate sent to service-discuss23:58

Generated by 2.17.2 by Marius Gedminas - find it at!