Wednesday, 2021-05-19

*** mlavalle has quit IRC00:05
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] Use containers for functional testing
*** cloudnull has joined #opendev03:47
*** ykarel has joined #opendev04:24
*** ysandeep|away is now known as ysandeep05:16
*** ralonsoh has joined #opendev05:36
*** sboyron has joined #opendev05:47
*** marios has joined #opendev05:59
*** slaweq has joined #opendev06:33
*** vishalmanchanda has joined #opendev06:35
*** hashar has joined #opendev06:38
openstackgerritMatthias Runge proposed openstack/project-config master: Retire panko, python-pankoclient and puppet-panko
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] Use containers for functional testing
*** fressi has joined #opendev07:07
*** lucasagomes has joined #opendev07:16
*** andrewbonney has joined #opendev07:17
*** amoralej|off is now known as amoralej07:20
*** ralonsoh has quit IRC07:24
*** ralonsoh has joined #opendev07:24
*** rpittau|afk is now known as rpittau07:33
*** SWAT has quit IRC07:36
*** tosky has joined #opendev07:38
*** ysandeep is now known as ysandeep|lunch07:40
*** SWAT has joined #opendev07:43
*** ChanServ has quit IRC07:54
*** ChanServ has joined #opendev07:54
*** services. sets mode: +o ChanServ07:54
*** jpena|off is now known as jpena07:56
*** DSpider has joined #opendev08:30
*** DSpider has quit IRC08:31
*** ysandeep|lunch is now known as ysandeep09:13
*** vishalmanchanda has quit IRC09:14
*** seongsoocho has quit IRC09:16
*** mnasiadka has quit IRC09:17
*** seongsoocho has joined #opendev09:21
*** auristor has quit IRC09:27
*** auristor has joined #opendev09:28
*** seongsoocho has quit IRC09:32
*** whoami-rajat has joined #opendev09:33
*** seongsoocho has joined #opendev09:35
*** vishalmanchanda has joined #opendev09:38
*** mnasiadka has joined #opendev09:40
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] Use containers for functional testing
*** dtantsur|afk is now known as dtantsur10:16
*** lpetrut has joined #opendev11:29
*** jpena is now known as jpena|lunch11:31
*** klonn has joined #opendev12:01
fricklerso #freenode is kinda exploding right now, will be interesting to see where this ends. maybe at least it will solve the spam issue ;) semi-serious question: should we consider running our own server?12:01
*** amoralej is now known as amoralej|lunch12:09
dmsimardfrickler: there was a thread about threat to freenode on openstack-discuss12:21
fungii already do, it's not hard (i've run irc servers, including the usual suite of service bots) for years, but previously we'd figured that oftc was a fine alternative12:21
fungii already squat a bunch of our channel equivalents on oftc, since i'm always connected there anyway for other projects who communicate there12:22
*** jpena|lunch is now known as jpena12:22
*** auristor has quit IRC12:24
*** auristor has joined #opendev12:25
yoctozeptooftc is fine by me as well, already lurking and seems slightly stabler from my location here12:25
yoctozepto(compared to freenode)12:25
*** ysandeep is now known as ysandeep|afk12:26
fricklerso my current take from watching #freenode is that the hostile takeover has already happened and we should consider taking some action soonish, see also . likely oftc seems the best way forward, though I'd also be willing to give a chance12:31
fungisince nothing seems to be on fire here at the moment, i'm popping out for a morning walk. i'll check back in before i need to do some errands, but plan to be around for good before we get started on merging the ansible changes for the listservs12:37
fungiback in an hour-ish12:38
*** ysandeep|afk is now known as ysandeep13:01
*** amoralej|lunch is now known as amoralej13:04
*** ricolin has joined #opendev13:08
*** ricolin has quit IRC13:29
*** klonn has quit IRC13:30
*** klonn has joined #opendev13:30
*** ysandeep is now known as ysandeep|mtg13:38
*** corvus has quit IRC13:53
*** tosky has quit IRC13:56
*** tosky has joined #opendev13:59
*** corvus has joined #opendev14:09
*** fressi has left #opendev14:10
*** corvus has quit IRC14:11
mordreda large part of the cost to moving is going to be moving. if we're going to move away from freenode I'd strongly recommend we consider matrix instead of just a different irc network. we could run our own homeserver, pay element to run one for us, or just use one of the public ones14:17
mordredI'm currently paying element to run a homeserver for me and have been using it as my IRC bouncer for over a year now. i've been pleased with it overall14:18
yoctozeptooh, good to know14:19
fungioftc is also bridged to matrix, apparently14:19
yoctozeptoI have never used matrix yet14:19
yoctozeptomight be time to learn it14:19
fungii need to head out to run errands before we start on the mailman config management switch, back soon hopefully14:20
mordredfungi: yeah - you can use the various irc networks from matrix bridges - but I'd suggest, fwiw, that if we're going to move we *move* and not attempt to maintain a footprint in 2 places14:21
fungiso, let's say freenode falls over next week, who's rewriting all our irc bots to be matrix bots?14:23
*** earne1 has joined #opendev14:24
*** ysandeep|mtg is now known as ysandeep14:25
mordredfair point14:29
mordrednot saying it's a slam dunk - just that it's worth considering as plans are considered. it's entirely possible that "rewrite the bots" is the thing that makes it a no go :)14:30
JayFI don't understand why folks want to migrate off IRC :( (admittedly I'm not familiar with matrix; but I can tell it's not IRC from the name :P )14:30
clarkbfungi: fwiw I want to double check a few things between the held test server and production (mostly configs lining up) but then I guess I'll approve the ansible listserv change14:31
clarkbJayF: matrix maintains a lot of the properties that people enjoy about irc while adding in functionality that make people enjoy newer chat systems14:31
JayFthe thing I enjoy about IRC is that I've used the same client and semantics in it for ... 20-ish years?14:32
*** corvus has joined #opendev14:34
openstackgerritMonty Taylor proposed opendev/system-config master: Start building gerrit 3.4
clarkbJayF: ya I tend to use weechat plugins to make other chat systems like IRC :)14:41
openstackgerritRiccardo Pittau proposed opendev/glean master: [WIP] test dib on ubuntu focal and do not test xenial anymore
JayFI mean, I happily use a Discord client (for personal uses only) and Slack (because my company requires it)... but I always prefered the simplicity of IRC14:42
corvusi'd be happy with oftc or matrix (i would probably switch from irssi to weechat to use matrix).  i do run my own matrix homeserver, and have for years; or at least, i have a vm where synapse is running and every few years i debug why it's not connected.  :)14:43
clarkbJayF: weeslack works pretty well for slack. discord actively punishes (and bans etc) users of third party clients so there aren't great options for it14:43
JayFclarkb: well... most third party slack clients fail at enough of slack that I need to use a slack client (including weeslack, which falls over when connected to large slack servers)14:44
JayFclarkb: losing things like embedded files and threading is just not an option for how we use it internally14:44
JayFand that is basically my fear about a higher level protocol being used instead of IRC: most "oh it works with IRC with a bridge" comes with MAJOR tradeoffs for the IRC users, and IME that support gets worse and worse over time as the newer protocol adds another 10 ways to post a dancing cat in chat14:45
clarkbJayF: the best thing about weeslack is the dancing cats don't render and you can opt into them :P14:45
clarkbbut ya if you're conencted to very large servers I have heard others say that can be problematic14:46
JayFI mean, Rackspace had a few thousand folks on Slack, and it broke my client14:46
JayFno way I'd even consider hooking up to Verizon slack, even if the security folks would allow it14:46
clarkbfwiw I've also heard people say matrix doesn't work well in large setups. However, I suspect we're below those thresholds and don't need to worry about it14:46
corvusclarkb: i suspect the homeserver matters there -- if you run a homeserver on a slow dsl link, do not join #matrix (ask me how i know).  but not likely an issue if using a well-hosted homeserver.14:50
corvusmaybe we could ask mozilla folks if they've had scaling issues14:51
clarkbcorvus: not a bad idea14:53
*** hashar is now known as hasharAway14:54
clarkbinfra-root and cc fungi specifically since fungi indicated interest I've double checked that files, apache configs, and /etc/mailman/sites all hash to the same value (in the case of sites I did a sort first)14:54
clarkbI also re double checked that running newlist on a list that exists is an error14:54
clarkbThe onyl differences I've noticed are that files and apache config files have slighyl different ownership/permissions14:55
clarkbhowever I don't think any of those matter for functionality14:55
clarkbanything else you think we should check or should we approve ?14:55
clarkbfungi: ^ I added and to the emergency file. Maybe you want to go ahead and +A the change if you are ready?15:01
hasharAwayJayF: hi, about migrating off of irc, I guess it is due to usability concern. It is super complicated for non tech folk, clients can be a madness to setup and you get to know some commands to register your nickname.  There is also no backscroll available to catchup conversations that happened while offline15:01
clarkbthen once it lands we can work through manual applications of it15:01
*** lpetrut has quit IRC15:01
hasharAwayJayF: compared to Slack or Discord.  Also where are the GIF of cats on irc? :D15:01
JayFhasharAway: folks who have those problems; I usually suggest using irccloud.15:02
hasharAwayJayF: Wikimedia constantly has the discussion we end up having communities on various platform (Slack definitely, some on Discord for sure).  WMF internally does use Slack here and there depending on the team.  The technology folks are most here on freenode though15:02
hasharAwayyeah irc cloud is nice solution, I personally highly recommend it. + They are involved in improving the irc protocol so that alone is worth spending a few buck for their service15:03
*** rpittau is now known as rpittau|bbl15:03
hasharAwayI definitely love IRC, that was my 2nd exposure to the internet after Usenet ..15:03
hasharAwayanyway, I came around to state that wikimedia volunteers (and staff I guess) are looking at the freenode thing.  There is a discussion that started with the wide community at
hasharAwaywill surely take a few days to settle on something (stay with freenode or/and try to fix it,  move to fork by freenode staff,  something else)15:05
avassclarkb: do discord ban third party clients? #nim uses a bot to bridge slack+discord+irc15:07
clarkbavass: they do15:08
*** mlavalle has joined #opendev15:08
clarkbavass: recent case of it too15:08
clarkbfungi: let me know when you are ready to do listserv things so that we can sync up15:14
fungii'm back from errands, going to eat real quick while i finish catching up on irc drama15:19
clarkbfungi: cool, maybe double check my double checking assertions above to make sure you're comfortable with the file perms and ownership deltas15:20 is root:root instead of root:list and 444 vs 644 (that is test vs prod in that order)15:20
fungiyeah, the permissions on those files are fine, read-only for a root owner is slightly silly imo, but it's irrelevant so long as those particular files are readable by the users which need them we're good (and they're world-readable, so that ticks the box)15:26
fungiclarkb: i've approved 789622 now15:27
clarkbcool, I'll keep an eye on it though the merge should be a noop until we remove a host from the emergency file and manually run the playbook against that host15:28
clarkbfungi: I'll set up a root screen on bridge for that and we can both hang out on the listservs and run the playbook when ready15:29
*** ykarel has quit IRC15:29
clarkbI guess I should eat breakfast while I wait for it to merge15:29
clarkboh! I also need to set one var in host/group vars. Will do that now15:29
dtantsurJayF: re "why against IRC". I *love* IRC, but I suspect it may be a serious show-stopped for newcomers, especially those without a strong FOSS background.15:31
JayFdtantsur: I've had success pointing those folks at irccloud.15:32
dtantsurI don't see how this is related15:32
dtantsurSetting up a semi-decent client is not an issue; coping with how IRC does things is15:32
dtantsurNot realizing that logging off makes it impossible to respond to you, for one15:32
JayFdtantsur: irccloud is a web-based IRC client that maintains a persistent presence15:33
dtantsurtrying to send files through some non-standard means (maybe this infamous irccloud)15:33
JayFdtantsur: it's what I typically point newcomers to use to connect to open source channels15:33
clarkbfungi: ok host vars are updated if you want to check the git log there15:33
dtantsur"Stay connected for 2 hours while inactive"15:33
dtantsurand file sharing feature that people assume to work outside of irccloud15:34
earne1On the topic of IRC, any plans to move this onto
dtantsurearne1: likely OFTC or even another platform15:35
clarkbearne1: not yet, we already squat channels on oftc so more likely there15:35
dtantsurOutreachy uses Zulip and tbh I really enjoyed having threads15:35
clarkbdtantsur: I thought irccloud gave you a link to the files shared there15:36
clarkbdtantsur: I recently tried zulip and enjoyed the threading too, but the client situation is really painful15:36
clarkbI basically ended up just using the browser client and while it worked is nowhere near as functional as something like weechat15:36
dtantsurclarkb: dunno, weechat tries to display these file transfers somehow, but this *somehow* never worked to me15:36
fungiyeah, the console client for zulip has some *very* rough edges at this point15:36
dtantsuryeah, I'd be sad without weechat15:36
dtantsurI know there is a matrix plugin and a so-so slack plugin, maybe zulip too?15:37
fungiright, and no weechat plugin for zulip that i was able to find15:37
dtantsurle sigh15:37
fungibut yeah zulup-term is still in a painful state15:37
dtantsurto be clear, I've never tries matrix, but mozilla uses it, so it cannot be too bad :)15:37
fungimany of these newer chat protocols are built on top of web technologies and assume all users are going to be connecting through a web browser or a phone15:38
fungimatrix does seem to have a fairly broad client ecosystem at this point though, yeah15:38
dtantsurhaving phone support would be vary handy during face-to-face (*sob* *sob*) events btw15:39
clarkbdtantsur: I used connectbot + screen + weechat for many years15:39
clarkbthe onyl downside is it drains your battery if you keep a connection open long term, but for polling its fine15:39
clarkbI think there are android clients using the weechat protocol too15:39
clarkbbut I have never used them15:39
mordredthe matrix phone client is very good15:40
mordredI've used the android weechat clients and they're ok-ish - but questionable security15:40
clarkbmordred: I thought they used client cert auth? It should be fairly robust15:40
clarkb(you have to keep your ca safe though)15:41
mordredbut the nice thing matrix-wise is that these features are easily available to our entire community not just those of us in a position to run our own bouncer servers and whatnot15:41
clarkbmordred: definitely15:41
clarkbbut its also a giant effort that no one has time for right now (or at least I don't have time for it)15:41
mordredthat's the downside15:41
mordredit's a shame - my thoughts are mostly that if the community is going to have to eat the cost of a move to somewhere, such a time is the best time to make a tech shift if we were ever going to do so15:42
dtantsuron the other hand, currently our huge community pretty much relies on volunteers15:42
dtantsurwhich is not exactly fair15:42
fungiright now our focus is split between keeping things we absolutely need running, and shutting down anything we don't absolutely need so we have enough people for the first thing15:42
dtantsurgiven the Big Names that participate in OpenStack, getting some money for a hosted server should be doable.15:42
* dtantsur wishful thinking15:43
mordredif we did want to go matrix, just paying element to host one for us would likely be more tractible than deciding to start running one ourselves15:43
JayFI suspect the resources needed to run our own server are more human resources than server resources15:43
dtantsurI mean paying for a server15:44
dtantsurI definitely don't suggest creating much more work for our infra team15:44
corvusi do like matrix, but honestly, the cost of switching to oftc is relatively small in terms of both technology and training.  if we need to vacate freenode in short order, oftc seems like a tractable solution.  matrix would be a nice thing to actually plan for.15:44
dtantsuryeah, I don't disagree15:45
clarkbcorvus: ++15:45
dtantsur(assuming OFTC is well funded)15:45
JayFOFTC has been around a long time, and is run by a foundation15:45
JayFIt's the obvious choice if we don't wanna go to libera15:45
*** gothicserpent has joined #opendev15:46
dtantsurdo we know if OFTC is okay hosting us?15:46
dtantsurand yes, IRC has a benefit of preventing me from spamming #openstack-ironic with cat gifs, that's for sure :)15:47
JayFIt's been stated on the mailing list that we've been camping the openstack- channels on OFTC as a backup for a while15:47
corvusthey seemed quite happy when i asked about it years ago15:47
corvusquite supportive in getting all those initial channels set up15:48
dtantsursounds like a no-brainer to me15:48
fungigranted i wouldn't be surprised if a freenode mass exodus puts some new strain on their network15:48
dtantsurI would expect libra to receive the most migrations given that it's the advertised destination15:50
openstackgerritMerged opendev/system-config master: Ansible mailman configs
*** marios has quit IRC15:56
corvusmy 2 cents: i feel like we need to establish a timeline -- 1) are we moving?  2) how long can we give ourselves to move?  if it's sufficiently long to plan for updating our docs for matrix, then let's seriously consider it.  if it isn't, then let's move to oftc and then seriously consider moving to matrix.15:57
corvusmordred: does has a bridge to oftc?15:58
fungisupposedly yes15:58
corvus says yes15:58
fungii haven't double-checked it, but saw others say it did15:58
*** lucasagomes has quit IRC15:59
fungiand yes, i agree we're currently stuck at #1 are we moving? signs point to yes, but it may make sense to give it a few days for more information to help determine that16:02
*** ysandeep is now known as ysandeep|away16:03
clarkbcorvus: judging from conversations on the openstack discuss list and in the openstack tc channel it seems that our users would like to move. I would put the answer to 1) as most likely16:03
fungiright now the information which has been circulated seems a bit one-sided, but since there doesn't seem to be much effort yet at a rebuttal it's worth assuming the worst conspiracy theories may be true16:03
fungiwhere "worst" are in the vein of the overlord unceremoniously turning off everything in an effort to drive users to some new platform in which they have a financial stake16:05
dtantsurwithout judging who is right and who is wrong, even all current freenode staff members have left, it's no longer a stable platform for us16:05
dmsimardIf it has an impact, I am told that OFTC doesn't support SASL while freenode/ do16:05
fungidid all of the staff really leave? or is someone just saying all the staff left? which reports are credible? making quick decisions based on incomplete and/or unconfirmed information may be rash16:06
toskyfungi: if it's not all, at least a good part of the staff has left16:07
dtantsurwe're not going to move this afternoon anyway. reaching out to OFTC to probe the ground may be a good early move anyway.16:07
toskyfungi: we received this from a trusted member of the KDE community, who is also one of the former freenode administrators:
JayFLots of other projects have already moved this afternoon; I don't think we should assume waiting is the best path. Right now, every OpenStack contributor still has to maintain a working nickname on this network. These are literally the last set of channels I'm in on freenode (all the others just moved)16:09
dtantsur(or start looking into what it takes to move to matrix. or making up our mind at least)16:09
clarkbJayF: I mean "maintain a working nickname" is basically free if you've previously done the setup16:09
JayFexcept now that email/password are in the hands of $unknown_entities16:09
JayFand I can't drop the nick as long as we continue to use this service16:09
JayFat least two of the resigning staff strongly suggested dropping your nicks or resetting email/password to keep yours private16:10
JayFhow do we know that services DB isn't going to be sold now? Or used for $nefarious_purposes?16:10
clarkbre email those are already largely public (if not because of the billions of exposed accounts indexed by haveibeenpwned then due to work on open source with git). And passwords should be unique already?16:11
mordredwell - my email is a lost cause because of git16:11
mordredmy password is specific to freenode, so losing it to someone only risks them having access to my freenode info - which admins of freenode already have16:11
mordredclarkb: jinx16:11
JayFI guess that's a decent point. Doesn't make me feel less itchy to get an account off a system that appears to have been taken over with hostility.16:12
avassI hope people use password managers :)16:13
fungidepends on what you mean by password managers, but i don't reuse passwords between different systems16:14
avassfungi: well yeah, pretty much just that16:14
corvusfuchs is a well-known friendly op who has helped us many times16:14
corvusor i should say Fuchs16:15
clarkbcorvus: and is one of the staff that quit right?16:15
fungiyep, network effect indicates their account is posted at
fungiaccount of what's transpired i mean16:15
fungii don't know for sure that is really under their control16:15
dtantsurI think if the news were false, we would already have received a global message through freenode itself16:16
dtantsurso far the only global message confirmed the news16:16
cenn#freenode is chaos rn, so.. at the least staff is not in control / has abandoned control16:17
JayFI've seen no evidence that indicates the story we've heard is not true, or close enough to true, to believe. In any event, a network can't survive without staff, and freenode just had the bulk of their staff resign. Which means the 'whodunit' aspect of this whole thing doesn't matter; freenode is no longer a suitable host.16:18
fungiand i suppose we can assume that if they unresign (freenode staff drama seems to be a semi-regular event after all) then we should still see it as a sign that their operations are unstable16:20
toskyfungi: I don't think they are going to unresign after that email; I don't think Fuchs would have sent that email to kde-community16:21
toskyfungi: again: Fuchs is a trusted member (also) of the KDE community, and I trust what's written there16:22
fungii have to admit that from the perspective of the sorts of drama which permeate efnet and undernet this seems mild, but i've always felt oftc would have been a better place for the channels we've got here anyway16:22
fungii just always get suspicious when people try to rush decisions16:24
*** hamalq has joined #opendev16:25
clarkbfungi: I'm almost ready to do I have a root screen up if you want to join and check the command that is queued up. We'll run base first because it was skipped by the automatic run due to being in the emergency file16:28
fungii've attached16:31
fungialso logged into both list servers to double-check stuff16:31
clarkbI've removed from the emergency file. Will run the base playbook now16:32
*** cgoncalves has quit IRC16:32
*** cgoncalves has joined #opendev16:33
clarkbfungi: that lgtm very noopy16:33
clarkbfungi: ready for me to run service-lists.yaml?16:33
clarkbok proceeding16:34
clarkbfungi: the ansible side lgtm. It reloaded apache2 because the file permissions chagned on that fiel I think16:36
fungiyep, i can still browse the website16:37
clarkbfungi: I am able to browse the web archives for the kata-dev list16:37
fungiand archives as well16:37
clarkbfungi: it also skipped creating the lists because they arleady exist which is what we wanted16:37
clarkbshould we proceed with it will be significantly more verbose as there are a lot more lists, but it shoudl skip them all too16:38
fungino services were restarted, so if configs were changed we wouldn't know right away16:38
clarkbfungi: do we want to restart the mailman service manually?16:38
clarkbapache was reloaded so those config checks should be done, its just mailman to worry about I think16:39
fungiyeah, and i guess exim was already being configured by ansible so no risk there16:40
fungii'll restart the mailman services on it just to make sure they start16:40
clarkb++ thanks16:41
fungi#status log Restarted mailman services on in order to verify configuration after config management changes16:41
openstackstatusfungi: finished logging16:42
clarkbfungi: should I remove lists.o.o from emergency now and run base against it?16:42
fungiyeah, i count 9 mailman processes on the server, so should be working16:42
clarkbfungi: oh wait we should double check the crontab on first16:42
fungii doubt the list traffic there is constant enough that we can reasonably wait to watch it process some posts16:43
fungithere are no crontabs in /var/spool/cron/crontabs/16:43
clarkbthe cron is multisite only16:43
clarkbso ya nothing to check on for cron16:43
clarkbwe'll want to check and clean that up on lists.o.o though16:43
clarkbI'm remove it from emergency.yaml and run base.yaml against lists.openstack.org16:44
fungiyep, ready and watching16:44
fungiafter this is done, we can send something to service-discuss about it, which doubles as confirmation the plumbing is intact16:45
*** rpittau|bbl is now known as rpittau16:45
fungithat looked smooth16:47
clarkbfungi: base.yaml lgtm are you ready for me to proceed to service-lists.yaml? Also maybe we rerun service-lists.yaml one mroe time without the --limit once we've done the two servers independently16:47
fungiyeah, great idea16:47
fungiand i'm ready when you are16:47
clarkbcool proceeding then16:47
*** cgoncalves has quit IRC16:52
clarkbfungi: the crontab looks fine actually.16:52
clarkbthere is an extra puppet comment to remove but otherwise its good as is I think16:52
*** cgoncalves has joined #opendev16:53
fungiagreed, the autogenerated by puppet comment block can be cleaned up16:54
*** hasharAway is now known as hashar16:55
clarkbfungi: its done16:56
clarkbapache was reloaded16:56
clarkbI can still get http://lists.opendev.org16:56
clarkbfungi: do you want to restart the mailman services on lists.o.o too?16:56
clarkb(I think we should asking if you want to run the commands or if I should, note they are vhost specific units/init scripts)16:57
fungiclarkb: luckily they're partitioned by site, so yeah we can restart the site daemons and then test16:57
clarkbfungi: ++16:57
fungii'll do that after i check stuff on the fs16:57
fungipipermail hasn't suddenly started listing hidden lists, so that's good16:59
fungilists with non-public archives are also still correctly flagged17:00
fungiand trying to browse those private archives anonymously still returns a 404 not found17:01
fungilooking right so far17:01
fungiready for me to restart mailman-opendev daemons?17:01
fungistopping mailman-opendev the process count for mailman dropped from 45 to 36 as expected (9 processes per site)17:03
fungistarting it again brought the total back to 4517:03
clarkbI see the new processes now too17:03
clarkbshould I send an email to service-discuss now?17:03
fungiyes please17:03
fungiif i receive that, then we're probably safe to proceed with the rerunning of the playbooks without --limit17:04
*** ricolin_ has joined #opendev17:04
clarkbreally quickly before sending I think may not be using the right template?17:05
clarkbthat isn't fatal so I think we can proceed and try to figure that out17:05
clarkbemail sent17:06
clarkbfungi: I think I got the email sent back to me from lists17:07
clarkbfungi: have you recieved it yet?17:08
fungiyep, received17:09
clarkbfungi: I guess we restart the others now?17:09
*** jpena is now known as jpena|off17:10
fungias for the template, the individual list pages do use the custom openstack one but the main list index seems not to. i don't recall if it ever did though17:10
*** ralonsoh has quit IRC17:11
fungii'll go ahead and restart all 5 of the sites (including just to make sure things are sane after that last seemingly no-op playbook run17:11
clarkbya I don't recall either. I can see we have a listinfo.html template in that dir17:11
fungithere are a bunch of defunct runners left over from my attempts to stop services17:12
fungigiving them a minute to maybe wake up17:12
fungiokay, they did eventually terminate17:12
fungiand started again, we're back at 45 daemons (9 kinds of runners for 5 sites)17:13
clarkbI think the listinfo.html we have is for list info on a single list17:13
clarkband that is why we get the branded output for single lists17:13
fungi#status log Restarted mailman services on in order to verify configuration after config management changes17:13
openstackstatusfungi: finished logging17:13
clarkbI suspect that we may just not template the thing for the listing (is it fully cgi without a template?)17:13
clarkbfungi: should I rerun the service-lists.yaml playbook now without the limit specified?17:14
clarkbya if I grep for strings on the listing page in the templates dir I don't get any hits17:15
clarkbfairly certain that the "issue" there is the cgi for listing the lists doesn't use a template the way we brand the other things, but I could be wrong about that and someone else should probably double check me17:16
clarkbfungi: I've got the no limit service-lists.yaml playbook queued up. Ready to run that when you are17:18
fungieach of the pages uses a separate template, there is a default but we can supply our own as an override17:18
clarkbfungi: right looking in the defaults dir I don't find what the template is for that page17:18
fungiahh, because there's probably no need since it doesn't need to be copied per-list17:19
fungiyeah, ready for the unlimited run now17:20
clarkbthat was based on grepping for 'Below is a listing' in the default templates dir17:20
clarkbok running unlimited run now17:20
clarkbusing synchronize to sync the files says it changed things when it doesn't change then :/ oh well17:23
*** frigo has joined #opendev17:24
*** dtantsur is now known as dtantsur|afk17:25
clarkbfungi: another weirdness is that using snychronize to copy the templates is preserving the owner uid and group gid from bridge17:27
clarkbI'm not sure how to address that :/17:27
clarkbmaybe synchronize is the wrong tool here, but file and copy don't really do what I want either which is to say take the entirety of a dir and put it over here and set attributes17:28
clarkbfungi: cool service-lists.yaml has finished without limit and it looked ok to me17:28
clarkb should be ready for review if we are happy with these results17:29
clarkbI'll work on a change to clean up the old puppetry as well as see if I can sort out how to copy those files in a better way for permissions17:29
clarkbs/ready for review/ready for approval/17:30
clarkbI have cleaned up the puppet comment in the crontab17:33
*** rpittau is now known as rpittau|afk17:37
fungiwe've seen that behavior with zuul jobs too. basically have to set ownership/perms before sync17:37
*** frigo has quit IRC17:40
openstackgerritClark Boylan proposed opendev/system-config master: Cleanup mailman puppetry
clarkbinfra-root I think and should be ready to go now17:44
*** mkowalski has quit IRC17:49
openstackgerritClark Boylan proposed opendev/system-config master: Fixup openstack mailman template perms
clarkbinfra-root ^ and that one should fixup the perms thing with template files17:54
clarkbI need to go make lunch for the kids now, but I'll still be around today if we want to keep pushing on this stuff17:55
clarkbfungi: did you put mk-archives-index together for indexing retired lists? can you maybe double check on that too?18:01
*** amoralej is now known as amoralej|off18:02
*** andrewbonney has quit IRC18:05
fungiyeah, i'll take a look18:08
fungiit's mainly for keeping track of what lists we have archives for whether or not they're still active vs historic archives18:09
*** vishalmanchanda has quit IRC18:22
openstackgerritClark Boylan proposed opendev/system-config master: Fixup openstack mailman template perms
clarkbtesting is like the best thing ever18:31
clarkbfungi: do you think we should approve now? also reviews on the other two chagnes I just pushed up arem uch appreciated18:51
fungii'll take a look18:51
fungiapproved it now, and looking at the other two as well18:53
*** mkowalski has joined #opendev18:54
*** mtreinish has quit IRC18:55
*** hashar has quit IRC18:55
*** mkowalski has quit IRC18:55
*** mkowalski has joined #opendev18:56
*** mtreinish has joined #opendev19:01
*** DSpider has joined #opendev19:06
openstackgerritClark Boylan proposed opendev/system-config master: Remove system-config-legacy-logstash-filters job
clarkbinfra-root ^ that is sort of a temp taking change19:08
clarkbwe said we wouldn't be turning anything off (at least not yet) service wise in the ongoing discusson on openstack-discuss, but that job runs against all our changes to system-config and isn't quick19:08
clarkbI'm thinking that since the ELK stuff is fairly fixed in time now we can safely stop running that job19:08
corvusclarkb: i left comments on puppet removal and mm templates changes19:11
openstackgerritClark Boylan proposed opendev/system-config master: Cleanup mailman puppetry
clarkbcorvus: ^ thanks those were definitely in need of fixing19:14
clarkbcorvus: for the rsync opts one do you have a preference? I feel like the copy is easier to understand but far more verbose19:14
clarkbwhich makes me waffle on which I prefer19:14
corvusclarkb: did you see my inline comment?19:15
corvusmaybe i should have used more words, sorry; am eating lunch at same time :)19:15
corvusmy idea is that if we can use those extra rsync opts, then we get best of both worlds -- ownership is correct, without additional verbosity19:16
openstackgerritMerged opendev/system-config master: Add infra-prod-service-lists job
clarkbya it wasn't clear if you prefered that since I don't think you voted either way. I'll update it19:16
clarkblooks like didn't cause the deploy job to run. It will run during our periodic run instead19:19
clarkbor if we land an update like 79222819:19
clarkbI'm just reading up on rsync really quickly then can push that19:19
*** whoami-rajat has quit IRC19:19
openstackgerritClark Boylan proposed opendev/system-config master: Fixup openstack mailman template perms
clarkbthanks for the reviews19:22
openstackgerritAnanya proposed opendev/elastic-recheck master: Fixing: TypeError: '<' not supported between instances of 'list' and 'str'
openstackgerritAnanya proposed opendev/elastic-recheck master: Fixing '<' not supported between instances of 'list' and 'str'
*** auristor has quit IRC20:10
*** auristor has joined #opendev20:10

Generated by 2.17.2 by Marius Gedminas - find it at!