Tuesday, 2021-05-25

* jeblairtest lurks using matrix00:06
clarkbjeblairtest: it seems to work00:06
jeblairtestindeed.  and since the bridge already had this channel, i got history when i joined too.  pretty neat.00:07
corvusin case anyone else is curious (i was): the bridge has a service bot, and you can give it your nickserv password and it will connect as you and authn.00:09
corvusscreenshot of the help text: https://i.imgur.com/5yF3KcK.png00:11
corvusi'm running a self-hosted synapse + element via docker-compose00:12
clarkbcorvus: and so you bridge the main irc bridge onto your server?00:14
corvusclarkb: yeah, though i think the terminology is that i joined a room hosted on matrix.org via federation, and that room is bridged to irc00:17
clarkbgotcha00:18
corvus(so the room is "#freenode_#opendev:matrix.org", and my homeserver joined that room, as it would any room not hosted on my own homeserver)00:18
clarkband you could just point an element client directly at the matrix.org servers and join via them?00:18
clarkbthe synapse server in this case isn't strictly necessary?00:19
corvusclarkb: correct, i'm doing complex stuff for fun and learning00:19
*** calcmandan has joined #opendev00:19
corvusclarkb: the easiest way to join this room via matrix without any effort is to register on the public matrix.org homeserver, log in with the matrix.org element client, and join the channel from there.00:19
clarkbcool, just want to make sure I understand enough of it when we say "people can use matrix's bridge" that doesn't entail running a server too :)00:20
corvusyep; i mean, i haven't tried it, but that's certainly the idea, and should be trivial for us to validate :)00:20
corvusi mostly wanted to refresh my knowledge of matrix operations, and also maintain a homeserver for my own privacy00:21
corvusincidentally, you can also use the matrix.org hosted element to connect to your own homeserver, or vice-versa.  it's basically a stateless js app.00:23
clarkbthere is a desktop app too iirc00:24
clarkbwhich may just be a js app wrapped in a browser presented as a natiev thing00:24
corvusand https://github.com/poljar/weechat-matrix00:24
corvuswhich is next on my list :)00:25
jeblairtest...and done00:33
jeblairtestclarkb: the weechat plugin is quite pleasant and easy to use.  verifying e2e encryption keys for private DMs is a little awkward, but can be done in other clients.00:53
jeblairtesti think i'm going to become a weechat user; and will seriously consider using the bridge for irc instead of a direct connection.00:54
*** ysandeep|away has quit IRC01:05
*** ChanServ has quit IRC01:05
*** corvus has quit IRC01:10
jeblairtestping01:15
*** ysandeep|away has joined #opendev01:15
*** ChanServ has joined #opendev01:15
*** weber.freenode.net sets mode: +o ChanServ01:15
jeblairtesti'm currently unable to connect to freenode via my normal irc client :(01:16
*** jeblair has joined #opendev01:16
*** jeblair is now known as corvus01:16
corvusokay, back; i think freenode may be having issues01:17
openstackgerritIan Wienand proposed opendev/system-config master: ask.openstack.org static site  https://review.opendev.org/c/opendev/system-config/+/79278901:32
openstackgerritIan Wienand proposed opendev/system-config master: ask.openstack.org static site  https://review.opendev.org/c/opendev/system-config/+/79278902:24
openstackgerritIan Wienand proposed opendev/system-config master: bridge: upgrade to Ansible 4.0.0  https://review.opendev.org/c/opendev/system-config/+/79286603:12
ianwdmsimard: it was a long time ago, but was there a reason we didn't put https://github.com/ansible-community/ara/commit/0cad657e0e8246ed6cdb3588fd4b43cf91c314f4 into a release?03:48
ianwclarkb: i really thought 3.11.0 dib should be not leaving profiledir directories behind03:54
ianwhttps://review.opendev.org/c/openstack/diskimage-builder/+/78730303:55
openstackgerritIan Wienand proposed opendev/system-config master: ask.openstack.org static site  https://review.opendev.org/c/opendev/system-config/+/79278904:15
*** ykarel|away has joined #opendev04:17
*** ricolin has joined #opendev04:18
*** rishabhhpe has joined #opendev04:28
*** redrobot8 has joined #opendev04:36
*** redrobot has quit IRC04:39
*** redrobot8 is now known as redrobot04:39
openstackgerritIan Wienand proposed opendev/system-config master: ask.openstack.org static site  https://review.opendev.org/c/opendev/system-config/+/79278904:41
*** ysandeep|away is now known as ysandeep|ruck04:48
openstackgerritIan Wienand proposed opendev/system-config master: ask.openstack.org static site  https://review.opendev.org/c/opendev/system-config/+/79278905:14
*** marios has joined #opendev05:25
openstackgerritIan Wienand proposed opendev/system-config master: ask.openstack.org static site  https://review.opendev.org/c/opendev/system-config/+/79278905:37
*** ykarel|away is now known as ykarel05:48
*** slaweq has joined #opendev06:03
*** rishabhhpe has quit IRC06:05
*** rishabhhpe has joined #opendev06:06
openstackgerritMerged opendev/system-config master: Symlink python3 to python for gerrit image build  https://review.opendev.org/c/opendev/system-config/+/79204106:09
openstackgerritIan Wienand proposed opendev/system-config master: ask.openstack.org static site  https://review.opendev.org/c/opendev/system-config/+/79278906:10
*** sboyron has joined #opendev06:17
*** ralonsoh has joined #opendev06:19
*** rishabhhpe has quit IRC06:31
*** rishabhhpe has joined #opendev06:32
dpawlikthanks for the investigation fungi and mnaser!06:39
*** rishabhhpe has quit IRC06:52
*** rishabhhpe has joined #opendev06:53
*** vishalmanchanda has joined #opendev06:54
ianwclarkb: ^^ ready for review now06:54
ianw#status log cleared leaked files and rebooted nb01/nb0206:55
openstackstatusianw: finished logging06:55
ianwforgot that from before06:56
*** sshnaidm|afk is now known as sshnaidm06:59
*** amoralej|off is now known as amoralej07:02
*** fressi has joined #opendev07:10
*** andrewbonney has joined #opendev07:17
*** rpittau|afk is now known as rpittau07:18
*** rishabhhpe has quit IRC07:19
*** rishabhhpe has joined #opendev07:20
*** tosky has joined #opendev07:22
*** rishabhhpe has quit IRC07:30
*** rishabhhpe has joined #opendev07:33
*** Tengu has quit IRC07:42
*** priteau has joined #opendev07:44
*** Tengu has joined #opendev07:49
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: Fix buildset-registry test on focal  https://review.opendev.org/c/zuul/zuul-jobs/+/79234907:52
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: Add ensure-skopeo role  https://review.opendev.org/c/zuul/zuul-jobs/+/79290007:52
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: Fix ensure-kubernetes role for focal  https://review.opendev.org/c/zuul/zuul-jobs/+/79290107:52
*** jpena|off is now known as jpena08:00
*** lucasagomes has joined #opendev08:04
*** ykarel is now known as ykarel|lunch08:05
noonedeadpunkHey there! Have a question regarding https://review.opendev.org/c/opendev/system-config/+/792651 - I'm not sure, but I guess that afs volume should already exist at this point? And I'm not sure how it should be created... Feels a bit like it's a manual action?08:06
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: Fix buildset-registry test on focal  https://review.opendev.org/c/zuul/zuul-jobs/+/79234908:10
*** vishalmanchanda has quit IRC08:11
*** seongsoocho has quit IRC08:12
*** rishabhhpe has quit IRC08:15
*** rishabhhpe has joined #opendev08:16
*** mnasiadka has quit IRC08:17
*** whoami-rajat_ has joined #opendev08:19
*** vishalmanchanda has joined #opendev08:21
*** mnasiadka has joined #opendev08:26
*** seongsoocho has joined #opendev08:30
*** dtantsur has quit IRC08:34
*** dtantsur has joined #opendev08:40
*** whoami-rajat_ is now known as whoami-rajat08:42
*** stephenfin has joined #opendev08:46
*** hashar has joined #opendev08:47
*** dtantsur has quit IRC08:57
*** dtantsur has joined #opendev08:58
dtantsuris it only me or freenode started having many more disconnects?08:58
*** ykarel|lunch has quit IRC09:35
*** ykarel|lunch has joined #opendev09:36
*** ykarel|lunch has quit IRC09:37
*** ykarel|lunch has joined #opendev09:37
*** ykarel|lunch is now known as ykarel09:54
*** fressi has quit IRC09:54
*** tinwood has quit IRC10:08
*** mattmceuen has quit IRC10:08
*** ttx has quit IRC10:08
*** wendar has quit IRC10:08
*** rm_work has quit IRC10:08
*** tinwood has joined #opendev10:09
*** mattmceuen has joined #opendev10:09
*** wendar has joined #opendev10:09
*** ttx has joined #opendev10:09
*** rm_work has joined #opendev10:09
*** jeblairtest has quit IRC10:12
*** reed has quit IRC10:12
*** mordred has quit IRC10:12
*** gtema has quit IRC10:12
*** dirk has quit IRC10:12
*** fressi has joined #opendev10:13
*** fungi has quit IRC10:13
*** mugsie has quit IRC10:13
*** fungi has joined #opendev10:14
*** mugsie has joined #opendev10:14
*** Tengu has quit IRC10:14
*** ralonsoh has quit IRC10:14
*** slaweq has quit IRC10:14
*** ricolin has quit IRC10:14
*** corvus has quit IRC10:14
*** openstackgerrit has quit IRC10:14
*** mkowalski has quit IRC10:14
*** kopecmartin has quit IRC10:14
*** jonher has quit IRC10:14
*** bhagyashris has quit IRC10:14
*** akahat has quit IRC10:14
*** cenn has quit IRC10:14
*** Tengu has joined #opendev10:16
*** ralonsoh has joined #opendev10:16
*** slaweq has joined #opendev10:16
*** ricolin has joined #opendev10:16
*** corvus has joined #opendev10:16
*** openstackgerrit has joined #opendev10:16
*** mkowalski has joined #opendev10:16
*** kopecmartin has joined #opendev10:16
*** jonher has joined #opendev10:16
*** bhagyashris has joined #opendev10:16
*** akahat has joined #opendev10:16
*** cenn has joined #opendev10:16
*** mnaser has quit IRC10:16
*** fdegir has quit IRC10:16
*** mhu has quit IRC10:16
*** felixedel has quit IRC10:16
*** icey has quit IRC10:16
*** amotoki has quit IRC10:16
*** CeeMac has quit IRC10:18
*** parallax has quit IRC10:18
*** jrosser has quit IRC10:18
*** paladox has quit IRC10:18
*** StevenK has quit IRC10:18
*** JayF has quit IRC10:18
*** timburke has quit IRC10:18
*** sshnaidm has quit IRC10:18
*** calcmandan has quit IRC10:18
*** gothicserpent has quit IRC10:18
*** guillaumec has quit IRC10:18
*** Dmitrii-Sh has quit IRC10:18
*** lourot has quit IRC10:18
*** swest has quit IRC10:18
*** amoralej has quit IRC10:18
*** smcginnis has quit IRC10:18
*** tristanC has quit IRC10:18
*** otherwiseguy has quit IRC10:18
*** raukadah has quit IRC10:18
*** dtantsur has quit IRC10:18
*** iurygregory has quit IRC10:18
*** dmellado has quit IRC10:18
*** slittle1 has quit IRC10:18
*** tobiash has quit IRC10:18
*** toomer has quit IRC10:18
*** amorin has quit IRC10:18
*** vishalmanchanda has quit IRC10:18
*** sboyron has quit IRC10:18
*** owalsh has quit IRC10:18
*** dmsimard has quit IRC10:18
*** jpena has quit IRC10:18
*** Jeffrey4l has quit IRC10:18
*** SpamapS has quit IRC10:18
*** dtantsur has joined #opendev10:20
*** vishalmanchanda has joined #opendev10:20
*** sboyron has joined #opendev10:20
*** calcmandan has joined #opendev10:20
*** timburke has joined #opendev10:20
*** slittle1 has joined #opendev10:20
*** owalsh has joined #opendev10:20
*** sshnaidm has joined #opendev10:20
*** dmsimard has joined #opendev10:20
*** JayF has joined #opendev10:20
*** StevenK has joined #opendev10:20
*** jrosser has joined #opendev10:20
*** paladox has joined #opendev10:20
*** parallax has joined #opendev10:20
*** CeeMac has joined #opendev10:20
*** jpena has joined #opendev10:20
*** gothicserpent has joined #opendev10:20
*** guillaumec has joined #opendev10:20
*** SpamapS has joined #opendev10:20
*** iurygregory has joined #opendev10:20
*** dmellado has joined #opendev10:20
*** Dmitrii-Sh has joined #opendev10:20
*** Jeffrey4l has joined #opendev10:20
*** lourot has joined #opendev10:20
*** swest has joined #opendev10:20
*** tobiash has joined #opendev10:20
*** raukadah has joined #opendev10:20
*** otherwiseguy has joined #opendev10:20
*** tristanC has joined #opendev10:20
*** smcginnis has joined #opendev10:20
*** amoralej has joined #opendev10:20
*** toomer has joined #opendev10:20
*** amorin has joined #opendev10:20
*** mnaser has joined #opendev10:21
*** fdegir has joined #opendev10:21
*** mhu has joined #opendev10:21
*** felixedel has joined #opendev10:21
*** icey has joined #opendev10:21
*** amotoki has joined #opendev10:21
*** mnasiadka has quit IRC10:21
*** rishabhhpe has quit IRC10:21
*** redrobot has quit IRC10:21
*** priteau has quit IRC10:21
*** ysandeep|ruck has quit IRC10:21
*** ChanServ has quit IRC10:21
*** mnasiadka has joined #opendev10:24
*** rishabhhpe has joined #opendev10:24
*** priteau has joined #opendev10:24
*** redrobot has joined #opendev10:24
*** ysandeep|ruck has joined #opendev10:24
*** ChanServ has joined #opendev10:24
*** weber.freenode.net sets mode: +o ChanServ10:24
*** ianw has quit IRC10:27
*** ianw has joined #opendev10:27
fricklerdtantsur: there was a global notice now about moving servers around. but I would also support moving soonish10:32
dtantsuryeah, it came after I complained :)10:32
*** gtema has joined #opendev10:34
*** mordred has joined #opendev10:36
*** gtema has quit IRC10:40
*** mordred has quit IRC10:41
*** fressi has joined #opendev10:45
*** whoami-rajat has quit IRC10:49
*** jeblairtest has joined #opendev10:49
*** reed has joined #opendev10:52
*** smcginni1 has joined #opendev10:57
*** gothicserpent has quit IRC10:57
*** smcginnis has quit IRC10:57
*** swest has quit IRC10:57
*** swest has joined #opendev10:58
*** timburke has quit IRC10:59
*** guillaumec has quit IRC11:00
*** Dmitrii-Sh has quit IRC11:00
*** Dmitrii-Sh8 has joined #opendev11:00
*** fressi has quit IRC11:00
*** guillaumec has joined #opendev11:00
*** gothicserpent has joined #opendev11:01
*** sshnaidm has quit IRC11:06
*** sshnaidm has joined #opendev11:07
*** dtantsur is now known as dtantsur|brb11:12
*** jpena is now known as jpena|lunch11:30
*** whoami-rajat has joined #opendev11:35
*** dirk has joined #opendev11:40
*** gtema has joined #opendev11:48
*** fressi has joined #opendev11:48
*** mordred has joined #opendev12:01
*** tosky has quit IRC12:08
openstackgerritTristan Cacqueray proposed zuul/zuul-jobs master: Add ansible-lint job  https://review.opendev.org/c/zuul/zuul-jobs/+/53208312:12
*** ysandeep|ruck is now known as ysandeep|lunch12:13
*** fressi has quit IRC12:22
*** jpena|lunch is now known as jpena12:28
*** fressi has joined #opendev12:28
*** ykarel has quit IRC12:34
*** ykarel has joined #opendev12:34
*** fressi has quit IRC12:38
*** rishabhhpe has quit IRC12:38
*** rishabhhpe has joined #opendev12:39
*** amoralej is now known as amoralej|lunch12:40
*** fressi has joined #opendev12:42
*** fressi has quit IRC12:57
*** dtantsur|brb is now known as dtantsur12:58
*** fressi has joined #opendev13:00
*** ysandeep|lunch is now known as ysandeep|ruck13:06
*** rishabhhpe has quit IRC13:17
*** rishabhhpe has joined #opendev13:18
funginoonedeadpunk: yeah, we'll need to create a mirror.erlang volume in afs, i should be able to get to that today hopefully: https://docs.opendev.org/opendev/system-config/latest/afs.html#mirrors13:27
*** ricolin has quit IRC13:28
*** amoralej|lunch is now known as amoralej13:29
noonedeadpunkI saw that but I guess that requires infra root?:)13:30
*** artom has joined #opendev13:36
funginoonedeadpunk: correct. er, well, more precisely, it requires someone with admin access to our openafs servers in our kerberos13:39
*** rishabhhpe has quit IRC13:45
*** rishabhhpe has joined #opendev13:46
*** akurbatov has joined #opendev13:50
*** fressi has quit IRC13:56
*** fressi has joined #opendev13:58
*** mlavalle has joined #opendev13:59
yoctozeptonoonedeadpunk: btw, why are not you using the rabbitmq-provided erlang? are you installing other erlang software?14:01
noonedeadpunkwhat do you mean under rabbitmq-provided erlang? It requires erlang to match used rabbitmq version. And if you're not using external rabbit version that differs from what provided by distro, you don't have erlang?14:03
noonedeadpunkOr I'm missing smth?14:03
noonedeadpunk*internal rabbit version14:03
yoctozeptonoonedeadpunk: I'm not following you now14:03
yoctozeptoI will try to explain14:04
yoctozeptoI understand you would either14:04
yoctozepto1) use distro-packaged erlang and rabbitmq (or, well, close-to-distro packaged one like with centos sig related to rdo)14:04
yoctozepto2) use rabbitmq-provided erlang and rabbitmq14:04
yoctozeptoin Kolla we use (2)14:05
yoctozeptoit has smaller footprint and is presumably tested by rabbitmq team14:05
noonedeadpunkwell, we use 2 as well. But rabbit does not provide erlang?14:05
yoctozeptoyes, they do?14:05
noonedeadpunkI mean if we look at https://packagecloud.io/rabbitmq/rabbitmq-server/packages/ubuntu/groovy/rabbitmq-server_3.8.16-1_all.deb they just require it?14:05
yoctozeptohttps://www.rabbitmq.com/which-erlang.html#erlang-repositories14:06
yoctozeptoDebian Erlang packages from Team RabbitMQ on Cloudsmith.io and Launchpad.14:06
yoctozeptoZero dependency Erlang RPM from Team RabbitMQ, also available from Cloudsmith.io14:06
noonedeadpunkI wonder how erlang rpm matches with erlang repo for debian?14:07
noonedeadpunkah14:07
noonedeadpunkYeah, I see. We just sticked to https://packagecloud.io/ for a while...14:08
yoctozeptowell, they basically package the same versions with their recipes from deb and rpm14:08
noonedeadpunkBut might be worth reconsdidering this...14:08
yoctozeptoit works for Kolla14:08
yoctozeptoif OSA also adopts, it means more end users14:08
yoctozeptoprofit14:08
yoctozepto:D14:08
noonedeadpunkI mean we used that path https://www.rabbitmq.com/install-debian.html#apt-packagecloud14:09
noonedeadpunkexcept we get erlang from different place...14:09
noonedeadpunkyoctozepto: and how you get available versions from https://dl.cloudsmith.io/ as it seems they don't allow listing?14:13
noonedeadpunkI think that was the reason why we avoided it....14:14
yoctozeptonoonedeadpunk: we get the repo contents like good citizens? ;d14:14
noonedeadpunkhaha14:14
yoctozeptono, seriously; the rpm and deb repos need to follow a specific structure14:15
yoctozeptothat's why they work in the first place ;p14:15
noonedeadpunkwell, yes... But it's a bit annoying when you want to know if they already have bullseye or not14:16
priteauHi. I've seen two post failures in the last hour, is there an issue with one of the providers?14:17
priteauhttps://zuul.opendev.org/t/openstack/build/2aff25cf7cf947bf8f92086262ed116014:17
priteauhttps://zuul.opendev.org/t/openstack/build/e9e51b05a0a0488982f0d5b5065c9d0014:17
noonedeadpunkyoctozepto: but I think  you're right14:18
noonedeadpunkfungi: hold on a bit with afs creation :)14:18
yoctozeptonoonedeadpunk: thanks14:19
yoctozeptonoonedeadpunk: bullseye works for us with that repo14:19
yoctozeptothe stuff you want to look at lives in openstack/kolla14:19
yoctozeptoshould be easy to grep the right parts14:19
yoctozeptoin wallaby and master we have bullseye14:19
fungipriteau: i hadn't heard anyone mention post_failure results yet, i'll look at those and see if i can identify a cause14:22
noonedeadpunkyoctozepto: um, they for real have just 3 rabbitmq versions in repo????14:27
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: Add ensure-skopeo role  https://review.opendev.org/c/zuul/zuul-jobs/+/79290014:27
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: Fix buildset-registry test on focal  https://review.opendev.org/c/zuul/zuul-jobs/+/79234914:28
noonedeadpunkAnd, um, how you control that they won't change for users between releases and complete rabbit upgrades?14:28
noonedeadpunkI mean - you deployed train with 3.7. Run apt upgrade and got 3.8 ?14:28
yoctozeptonoonedeadpunk: we pin erlang version to minor; they seem to have been upgrading fine on these lines14:29
noonedeadpunkah, so upgrade jsut does not made...14:29
yoctozeptoprecisely on train we pin to 3.714:29
noonedeadpunkbut then it won't be installed either?14:30
noonedeadpunkin case of re-installation?14:30
noonedeadpunkI'm just kind of confused with https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/deb/ubuntu/dists/bionic/main/binary-amd64/Packages.gz14:30
noonedeadpunkas there's no 3.714:30
yoctozeptonoonedeadpunk: ah, weird; we are using the PPA on deb and it is well-equipped14:34
yoctozeptoperhaps this cloudsmith mirror has limits14:34
yoctozeptoor whatever14:34
yoctozeptogood catch though14:34
*** fressi has quit IRC14:36
noonedeadpunkeventually for us it's super easy to switch and eventually users can pick up whatever repo they want https://opendev.org/openstack/openstack-ansible-rabbitmq_server/src/branch/master/defaults/main.yml#L67-L8014:37
noonedeadpunkBut I guess we will leave with current repos by default, as cloudsmith feels keeping bare minimum versions in loop14:37
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: Add ensure-skopeo role  https://review.opendev.org/c/zuul/zuul-jobs/+/79290014:38
openstackgerritGuillaume Chauvel proposed zuul/zuul-jobs master: Fix buildset-registry test on focal  https://review.opendev.org/c/zuul/zuul-jobs/+/79234914:38
noonedeadpunkprobably if you don't care about revisions and minor version that's fine....14:39
noonedeadpunkbut deployments won't be really consistent after deployment14:39
yoctozeptothey are with pins; but yeah, we are using the PPA, not cloudsmith there14:40
yoctozeptobut PPA from rabbitmq team nonetheless14:40
noonedeadpunkwell, erlang cloudsmith is also having only major versions14:40
noonedeadpunkor you use ppa there as well?14:40
noonedeadpunklike among all 22 erlang there's only 22.3.4.16-114:43
*** ykarel is now known as ykarel|away14:43
noonedeadpunkand whole set of 23.3.*14:43
fungipriteau: build 2aff25cf7cf947bf8f92086262ed1160 ran from our ze11 executor, the error seems to be a 401 unauthorized response from storage101.iad3.clouddrive.com/v1/MossoCloudFS_28a933f9-900d-40b8-9de3-deb24d213aa2/zuul_opendev_logs_2af14:45
funginow to see if the other example is the same14:45
clarkbinfra-root I'd like to approve https://review.opendev.org/c/opendev/system-config/+/792708 today and start making progress on that. I have a meeting in a few minutes and after that I'll probably +A it with ianw's +2 if anyone else wants to look at cert updates can you let me know and/or review it before then?14:53
yoctozeptonoonedeadpunk: ppa for erlang, packagecloud for rabbitmq14:54
yoctozeptomultitasking so I was not precise, sorry14:55
fungipriteau: build e9e51b05a0a0488982f0d5b5065c9d00 ran from our ze10 executor, but the error seems to be the same (same swift api endpoint even, just different container14:55
priteauThanks fungi14:56
fungii'm checking logs across our executors, looks like it may have been a narrow window of time14:56
*** rishabhhpe has quit IRC14:58
*** ykarel|away has quit IRC14:59
noonedeadpunkah, ppa for erlang... it's also weird though https://launchpad.net/~rabbitmq/+archive/ubuntu/rabbitmq-erlang15:00
yoctozeptoit's what we use and rely on rabbitmq to do it right15:01
yoctozeptoso far so good15:01
noonedeadpunkI'd say we should have puting more truth into official erlang repos... But they're just not super reliable lately15:01
yoctozeptowhat can I say? :D15:01
noonedeadpunkin terms of apt-cacher-ng15:02
yoctozeptopsst, I'd say we work collectively to make openstack rabbitmq-less15:02
fungipriteau: i find logs for 22 occurrences between 13:14:35-14:17:06 UTC, so it was happening to some small percentage of builds over roughly an hour span, and we haven't seen any recurrence for approximately the past 45 minutes now15:02
priteaufungi: Great, thanks for checking :)15:02
noonedeadpunkreplace rabbit with... qrouterd?:)15:02
noonedeadpunkI mean messaging in general is great, isn't it?15:03
fungipriteau: so anyway, safe to recheck those, and probably a small enough impact we needn't bother spamming a notice to all the irc channels, but i'll log it15:03
yoctozeptonoonedeadpunk: well, we have two things to tackle: messaging for rpc and messaging for notifications15:05
fungi#status log A unidentified incident in one region of one of our storage donors caused a small percentage of Zuul job builds to report a POST_FAILURE result with no uploaded logs between 13:14 and 14:18 UTC, these can be safely rechecked15:05
openstackstatusfungi: finished logging15:05
yoctozeptoqdrouterd could work for rpc but it does not seem popular15:06
yoctozeptoand for notifications perhaps kafka is a better choice anyway15:06
*** ysandeep|ruck is now known as ysandeep|away15:06
yoctozeptoI really feel rabbitmq is here just by chance ;d15:06
yoctozeptoironic has json rpc support15:06
yoctozeptoso it can work without rabbitmq/qdrouterd15:07
yoctozeptonot sure about limitations of that15:07
noonedeadpunkwell, originally rabbit is the way more powerfull than we use15:08
yoctozeptoyeah, we are introducing too much complexity, especially for the majority scenarios with just rpc15:10
yoctozeptobut we are getting really offtopic here on #opendev15:10
*** sshnaidm is now known as sshnaidm|afk15:11
*** sshnaidm|afk is now known as sshnaidm15:35
*** lucasagomes has quit IRC15:58
clarkbI have approved the ethercalc LE cert change16:01
*** marios is now known as marios|out16:02
clarkbassuming that goes well I'll try to work through the other two services today as well16:04
fungithanks! i'm fairly heads down working on accessbot for now16:07
fungias part of the wip changes, i'm reworking the channels config syntax to reflect that these are access levels rather than a flag mask, because that's going to actually need reflecting in how accessbot works there, and continuing to refer to masks instead of levels would likely lead to confusion16:09
*** ykarel|away has joined #opendev16:10
clarkbfungi: and that is due to the different chanserv dialect on oftc?16:12
clarkbI think the accessbot configuration sort of already mapped logical levels to masks then applied those logical levels to users. But ya if the implementation deals with masks that would be confusing if the chanserv doesn't16:12
*** tosky has joined #opendev16:14
fungiright, in particular the full_mask stuff where accessbot would use it as a limiting filter on access flags for existing unlisted ops16:16
fungiinstead there are basically four levels: master, chanop, member, and unset16:16
fungiso if there are existing users in a channel's access list, the closest analog would be to make sure their level is <= the allowed level16:18
clarkbianw: the ask change lgtm. Looks like DNS hasn't been updated to include an acme challenge record yet, but ya I think we can go ahead with that probably? Then when we're happy with the static setup update the A/AAAA records to a CNAME16:19
openstackgerritJeremy Stanley proposed openstack/project-config master: Switch the IRC access check to OFTC  https://review.opendev.org/c/openstack/project-config/+/79284316:19
openstackgerritJeremy Stanley proposed openstack/project-config master: Revert "Accessbot OFTC channel list stopgap"  https://review.opendev.org/c/openstack/project-config/+/79285716:19
fungianyway, the config in 792843 reflects the new model now and i've cleaned up all the remaining todo entries there16:20
fungiwill have the corresponding accessbot change pushed up later today hopefully, but i need to perform some extensive testing on it first16:20
clarkbok, and then we also need to be careful that we don't land any of that until we're intending to switch? because the changes won't support both freenode and oftc right?16:21
clarkb(you had been marking things WIP previously, just want to make sure I'm keeping up)16:22
*** ykarel|away has quit IRC16:22
fungiyep, still wip, and yes the system-config and project-config changes will probably have to get some level of atomicity around deployment16:24
fungithough in reality i think if we merge the config change first then accessbot will just break until we land the accessbot change16:24
clarkbmakes sense16:25
fungibut regardless i expect we'll probably put deployment jobs on hold globally before approving the first of that topic until all of them have merged16:27
*** mlavalle has quit IRC16:32
*** amoralej is now known as amoralej|off16:35
*** marios|out has quit IRC16:38
*** rpittau is now known as rpittau|afk16:41
clarkbianw: thinking about the dib leaks, what if they are all associated with image updates doing a forceful restart of the container? I think we can tell docker-compose to do a more graceful stop with a timeout which may give dib time to cleanup after receiving the sigint?16:42
clarkbhrm we already set a stop grace period so nevermind16:47
clarkbservice-nodepool is still failing after nb01 and nb02 got cleaned up. Looks like docker-compose up -d is failing on nb03 for some reason. I'm rerunning with --verbose there and looking into it16:51
clarkbI am able to reproduce it but --verbose doesn't really tell me much more info. I think it is trying to stop the container to start it on a newer image and that is afiling16:53
clarkbI'll see what docker-compose down says16:53
*** dtantsur is now known as dtantsur|afk16:54
*** hashar has quit IRC16:54
clarkbyup docker-compose down fails with the same error16:57
*** jpena is now known as jpena|off16:58
clarkbI'll try with an even longer timeout. Though it seems the processes aren't running but the docker container is still "up"16:58
clarkbethercalc change has a job that timed out :( will recheck it once it reports back16:59
clarkbI am attempt docker kill against the container directly now17:07
*** timburke has joined #opendev17:07
clarkbno luck via that either. I notice there is an old stale cp from a dib build a a handful of sync processes. The dumb-init process is tehre too but no dib or nodepool-builder from what I can see17:07
clarkbthis might just be a reboot and force the process table to be happy that way situation17:08
clarkbthough I may let ianw look at it before we take that drastic action17:08
fungiaccessbot overhaul is actually not too bad, the nickserv identifying "just works" as-is after i rip out the identify-msg cap and adjust how service nicks are parsed from the responses. i think the bulk of the work is just going to come down to replacing the mask handling with canned access levels17:08
clarkband even that may just be changing some var names and it would work? maybe changing order of command args to  chanserv?17:09
clarkbI wonder if we've tripped over a docker arm64 bug :/17:10
clarkbre ethercalc I realized I could be impatient and promote teh change to restart its jobs. I have done this17:13
clarkbon nb03 I have gone as far as trying to restart the docker service. This too appears to not be functioning :/17:14
fungino, not just changing some var names unfortunately. it's very granular iterating over flags and composing flag diffs to set/unset17:18
fungihowever i can rip most of that out and simplify17:18
clarkbah17:21
clarkbsystemctl restart docker.service did eventually succeed but docker compose down is still failing. I don't think this is anything to do with dib or nodepool itself and instead docker. For this reason I think I'll just go ahead with a reboot now17:27
clarkbreboots theer are slow but it has rebooted17:34
clarkband docker-compose up -d succeeded17:35
*** andrewbonney has quit IRC17:36
*** fressi has joined #opendev17:37
clarkbanyway service-nodepool should hopefully succeed now17:37
yoctozeptoanyone of you aware whether our Chinese users are able to use matrix.org ?17:45
yoctozepto(or join the matrix in any applicable way)17:45
fungiyoctozepto: i do not know, it's certainly come up as something worth finding out17:46
clarkbgoogle indicates it is problematic17:47
fungihttps://news.ycombinator.com/item?id=23393935 from a year ago says it's blocked there17:47
fungialso apparently google briefly kicked the element matrix client out of the android app store in february, because users reported it could be used to reach channels with bad content17:51
yoctozeptoeh, users17:59
yoctozeptook, so we know oftc is China-friendly17:59
yoctozeptoif we lift the registration need17:59
yoctozeptogoing from matrix (or just any irc webclient) to our channels will be easy-peasy18:00
*** stephenfin has quit IRC18:00
yoctozeptoso I guess we could go win-win by just going oftc now18:00
yoctozeptoand gradually migrate to matrix if we confirm a way for China to join those spaces18:01
clarkbI'm not sure I'd do a gradual migration to anything. Doing so is likely to fracture communication channels18:01
*** mlavalle has joined #opendev18:03
yoctozeptowe can't really stop people from doing it and there seems to be push to use the newer media18:03
yoctozeptothough the portal rooms seems to be a sweet spot18:03
yoctozeptosee me posting on oftc from matrix18:03
yoctozeptoah, sorry, only fungi in there18:03
yoctozeptoI thought I saw you too18:04
fungiin #openstack anyway18:04
clarkbI stopped hanging out in #openstack years ago when I realized that I was the only one answering questions18:04
clarkbyoctozepto: my comment is more along the lines of what opendev can do. I don't think we should support matrix and irc bots for example18:05
clarkbif there is a shift to matrix then our bots should follow and leave behind irc18:05
yoctozeptooic18:05
yoctozeptomakes sense18:05
yoctozeptoperhaps we will find some middle ground18:05
yoctozeptothe bridging is nice18:05
yoctozeptoirc could supply messages in matrix18:06
openstackgerritMerged opendev/system-config master: Provision ethercalc LE cert  https://review.opendev.org/c/opendev/system-config/+/79270818:07
yoctozeptook, time for some rest18:07
yoctozeptoyou enjoy your day18:07
clarkbyou too18:07
*** ralonsoh has quit IRC18:25
*** mnaser has quit IRC18:30
*** fdegir has quit IRC18:30
*** mhu has quit IRC18:30
*** felixedel has quit IRC18:30
*** icey has quit IRC18:30
*** amotoki has quit IRC18:30
*** icey has joined #opendev18:30
*** felixedel has joined #opendev18:30
*** mhu has joined #opendev18:30
*** mnaser has joined #opendev18:31
*** fdegir has joined #opendev18:31
*** amotoki has joined #opendev18:31
clarkbethercalc LE provisioning lgtm so I have approved the next change to swap its vhost config over to the LE cert18:43
clarkbI'm also going to approve the change to provision a storyboard LE cert now18:44
fungiawesome, thanks18:46
openstackgerritJeremy Stanley proposed openstack/project-config master: Switch the IRC access check to OFTC  https://review.opendev.org/c/openstack/project-config/+/79284318:52
openstackgerritJeremy Stanley proposed openstack/project-config master: Revert "Accessbot OFTC channel list stopgap"  https://review.opendev.org/c/openstack/project-config/+/79285718:52
clarkbI'm seeing a few post failures in zuul status, the one I spot checked briefly is https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_138/792827/1/gate/system-config-puppet-apply-4-ubuntu-xenial/13839ed/job-output.txt and seems to be limestone problems?19:02
clarkbanyway heads up /me goes back to the meeting19:02
*** sboyron has quit IRC19:07
openstackgerritMerged opendev/system-config master: Switch ethercalc to the new LE cert  https://review.opendev.org/c/opendev/system-config/+/79282719:14
openstackgerritMerged opendev/system-config master: Provision LE cert for storyboard.openstack.org  https://review.opendev.org/c/opendev/system-config/+/79285219:43
*** hashar has joined #opendev19:51
openstackgerritJeremy Stanley proposed openstack/project-config master: Don't use standard labels in vexxhost-ca-ymq-1  https://review.opendev.org/c/openstack/project-config/+/78576919:51
ianwmemory unplug might not be totally crazy, have to look into it19:59
ianwi bet though differences between kvm/qemu and xen would be annoying in a general sense.  but i guess it only needs to work in a subset20:00
*** lourot has quit IRC20:03
*** lourot has joined #opendev20:06
*** lourot has quit IRC20:07
*** lourot has joined #opendev20:07
*** lourot has quit IRC20:10
*** lourot has joined #opendev20:12
ianwclarkb: is nb03 still in a sad state?20:17
clarkbianw: I don't think so. I ended up rebooting it and taht seemed to make the process table happy again20:17
*** lourot has quit IRC20:17
*** lourot has joined #opendev20:18
ianwcool, thanks.  and yeah, i think even with the timeout, shutdown at random times is probably part of the issue20:18
clarkbfwiw docker-compose down, docker kill $containerid, systemctl restart docker.service as well as `kill $pids` all failed to stop the processes20:19
clarkbthe reboot was not fast but once it completed everything seemed happy again20:19
fungiyay! i got past the identify hurdle, and have it adding admins to channels with the intended access levels now, though it looks like i might have an idempotency problem to work through (luckily i'm just testing with a couple of channels for now)20:40
*** slaweq has quit IRC20:43
*** hashar has quit IRC20:46
openstackgerritJeremy Stanley proposed opendev/system-config master: Retool accessbot for OFTC  https://review.opendev.org/c/opendev/system-config/+/79306221:11
fungiinfra-root: that's ^ locally tested against a subset of channels and seems to be working for me21:12
*** elod has quit IRC21:16
*** elod has joined #opendev21:18
*** akurbatov has quit IRC21:48
*** corvus has quit IRC21:51
*** jeblairtest is now known as corvus21:51
*** corvus has joined #opendev21:52
*** vishalmanchanda has quit IRC22:13
clarkbethercalc's cert lgtm22:17
clarkbstoryboard's provisioned cert also lgtm so I have approved the chagne to swap it in apache22:20
*** tobiash has quit IRC22:24
corvusclarkb: is there a way to re-order buffers in weechat?22:25
clarkbcorvus: yes one sec22:26
clarkbcorvus: `/buffer move $number` will move the current buffer to that location22:26
clarkband then shifting happens to the others as necessary22:26
corvus(i totally did google that and failed, btw)  thanks!22:26
clarkbthere is also /buffer swap22:26
openstackgerritMerged opendev/system-config master: Switch storyboard to LE cert  https://review.opendev.org/c/opendev/system-config/+/79285322:35
clarkbfungi: https://review.opendev.org/c/opendev/system-config/+/793062 failed the eavesdrop job. Not suer why yet as I was going to review it first22:45
fungithe depends-on to project-config may not be working22:46
clarkbfungi: also re your question on https://review.opendev.org/c/openstack/project-config/+/792843/4/accessbot/channels.yaml does oftc chanserv not have any channel modes/flags ? thinking things like +r or similar22:46
fungithere are channel modes, but they're not part of access list entries22:47
fungithe modes handling is left intact22:47
fungioh, i see what you're saying, yes i seem to have removed modes which weren't related to the #openstack-unregistered channel forwarding22:48
fungii'll add some of those back after checking they're the same (assuming i can find the mlock info for that network)22:49
clarkbfungi: ya I think I'm getting confused andsaw that stuff seemed to be removed and confused it with the allowed: full stuff22:51
clarkbits not clear to me what the old mask: full_mask was intending to achieve22:51
clarkbis that how we promoted other channel specific ops?22:51
fungikinda22:52
fungiit supposedly worked like this:22:52
clarkbmaybe allowed: ops makes more sense for those chanells then?22:52
fungiyou set a default mask for channels, if there was a channel-specific operator in the access list, then they got any flags which weren't part of that default mask removed22:52
fungiif you overrode the default mask with a channel specific one (like full_mask) then only flags not present in that mask were removed from entries22:53
clarkbgotcha22:55
clarkbthen ya i think your rewrite to say "this is what they get" rather than a mask makes sense in the other dialect22:55
fungii'm thinking instead (at least for now), we just ignore any access list entries which are present if those users aren't in our config, but instead of encouraging a proliferation of manual chanops additions we encourage people to add channel-specific entries to the channel list in git22:55
fungibasically treat all channels as if (in olden terms) they had mask: full_mask set22:56
fungibecause we never really seem to have had a policy for when we'd allow channels to add that anyway22:56
*** tosky has quit IRC23:03
openstackgerritMerged opendev/system-config master: Provision LE cert for translate.openstack.org  https://review.opendev.org/c/opendev/system-config/+/79285423:09

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!