Thursday, 2020-07-16

rlandy|ruckysandeep: the problem is here ...00:02
rlandy|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/bcb2416/logs/buildah-builds/container-builds/cfa29691-bea2-4e35-8e1c-be6e3db88495/base/os/octavia-base/octavia-api/octavia-api-build.log00:02
rlandy|ruckError: Unable to find a match: python3-ovn-octavia-provider00:02
rlandy|ruckthat package was built00:02
rlandy|ruckand added to octavia current00:02
rlandy|ruckwhich never got promoted00:02
rlandy|ruckbecause of scenario01000:03
rlandy|ruckI may need to remove that from criteria to promote octavia00:03
rlandy|ruckand clear the error00:03
ysandeepack, i think you are taking about https://review.opendev.org/#/c/740982/ which solves the issue00:03
rlandy|ruckysandeep: that kind of caused the problem ...00:04
rlandy|ruckchicken and egg situation00:04
ysandeepohhh00:05
rlandy|ruckthe package is stuck in the component pipeline00:05
rlandy|ruckwhich can't pass00:05
rlandy|ruckbecause current-triple is old00:05
rlandy|ruckussuri is promoting00:06
ysandeep>> ussuri is promoting -- :) good news00:06
rlandy|ruckhopefully00:06
rlandy|ruck2020-07-16 00:30:59,365 27170 INFO     promoter sftp> -mkdir /var/www/html/images/centos8/00:32
rlandy|ruck2020-07-16 00:30:59,365 27170 INFO     promoter Couldn't create directory: Failure00:32
rlandy|ruck2020-07-16 00:30:59,365 27170 INFO     promoter sftp> -mkdir /var/www/html/images/centos8/ussuri/00:32
rlandy|ruck2020-07-16 00:30:59,366 27170 INFO     promoter Couldn't create directory: Failure00:32
rlandy|ruck2020-07-16 00:30:59,366 27170 INFO     promoter sftp> -mkdir /var/www/html/images/centos8/ussuri/rdo_trunk/00:32
rlandy|ruck2020-07-16 00:30:59,366 27170 INFO     promoter Couldn't create directory: Failure00:32
rlandy|ruckdo we have ussuri images?00:32
rlandy|ruck[ ]overcloud-full.tar.md5 2020-07-15 21:33 5300:33
rlandy|ruckit's there00:33
rlandy|ruckysandeep: ok - stein is promoting now00:33
ysandeeprlandy|ruck, ack00:34
ysandeeprlandy|ruck, did we add any check job in downstream.. on rebasing https://code.engineering.redhat.com/gerrit/#/c/203233/ is see tripleo-ci-rhel-8-standalone-rhos-17  triggering for the first time.00:45
ysandeepnvm found it - https://code.engineering.redhat.com/gerrit/#/c/205807/00:48
rlandy|ruckysandeep: the ussuri bm failure is diff - that's image build00:51
ysandeeprlandy|ruck, okay00:52
*** Tengu has quit IRC00:55
*** Tengu has joined #oooq00:56
ysandeepbm failing because nova endpoint missing from undercloud01:01
ysandeep(undercloud) [zuul@undercloud ~]$ openstack flavor create --id auto --ram 6144 --disk 49 --vcpus 1 oooq_compute01:01
ysandeeppublic endpoint for compute service not found01:01
*** Tengu has quit IRC01:08
*** Tengu has joined #oooq01:10
ysandeeprlandy|ruck, I will try to deploy master bits on envA today to make sure we can reproduce this on master bits too.01:20
rlandy|ruckok01:21
ysandeepi think its related to nova less undercloud, something might have changed01:21
*** Goneri has quit IRC01:23
*** ysandeep is now known as ysandeep|afk01:23
rlandy|ruckysandeep|afk: ok - logging off ... will promote downstream tomorrow01:24
*** rlandy|ruck has quit IRC01:24
*** holser has quit IRC01:28
*** ysandeep|afk is now known as ysandeep01:59
*** apetrich has quit IRC02:14
*** rfolco has quit IRC02:15
*** ysandeep is now known as ysandeep|afk02:24
*** skramaja has joined #oooq03:19
*** ykarel|away has joined #oooq04:15
*** ykarel|away is now known as ykarel04:16
*** soniya29 has joined #oooq04:52
*** udesale has joined #oooq04:57
*** ysandeep|afk is now known as ysandeep|rover05:00
*** ratailor has joined #oooq05:19
*** saneax has joined #oooq05:47
*** marios has joined #oooq05:52
*** amoralej|off is now known as amoralej06:33
bhagyashrismarios, add into your review list https://review.rdoproject.org/r/#/c/28510/ whenever you will get time thank you :)06:50
ysandeep|roverfolks, looking for reviews on https://review.rdoproject.org/r/#/c/28344/ new c8 train job , test are green06:53
ysandeep|rovertests*06:53
mariosack bhagyashris06:56
*** zbr has quit IRC07:16
*** bogdando has joined #oooq07:23
*** udesale has quit IRC07:26
*** udesale has joined #oooq07:31
ysandeep|roverzbr|ruck, Hey o/ I will away for a while.07:38
*** ysandeep|rover is now known as ysandeep|lunch07:38
*** apetrich has joined #oooq07:59
*** zbr has joined #oooq08:02
*** tosky has joined #oooq08:02
zbr|ruckmarios: does https://hackmd.io/@rh-openstack-ci/mol answers your questions?08:02
*** saneax has quit IRC08:02
*** saneax has joined #oooq08:04
marioszbr|ruck: which question are you referring to08:08
zbrbasically related to https://review.rdoproject.org/r/#/c/28455/08:11
zbri cannot remove the job because it need to remove it from pipeline first08:11
zbrthat is one of the reasons why i said, defining custom pipelines in rdo-jobs, may have not being the best idea08:12
zbrespecially for this type of jobs08:12
zbrmol-promoter was added to periodic and already run as expected over the night.08:12
*** tosky has quit IRC08:16
*** holser has joined #oooq08:30
*** tosky has joined #oooq08:33
arxcruzzbr: around?08:48
arxcruzzbr: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/tempest.yml#L81 is this logic correct? should it be '=' or '==' also works?08:48
zbr|ruckyes but busy with rucking, around usuri08:48
zbr|rucki do not remember, better to google it :p08:49
zbr|rucklikely it does not need double equal08:49
zbr|ruckbecause it cannot be an asignement, but best to check the doc08:49
*** zbr has quit IRC08:50
chandankumarzbr|ruck: still on promoter?08:51
*** derekh has joined #oooq08:53
arxcruzzbr|ruck: kinda hard to find exactly what i want o google when the keyword is 'version' :/08:57
zbr|ruckchandankumar: connected on tmate but not doing anything09:02
*** ykarel is now known as ykarel|lunch09:12
*** ysandeep|lunch is now known as ysandeep|rover09:12
*** dtantsur|afk is now known as dtantsur09:15
ysandeep|roverchandankumar, zbr|ruck fyi.. i wrote a bz for ussuri - https://bugs.launchpad.net/tripleo/+bug/188778309:36
openstackLaunchpad bug 1887783 in tripleo "Ussuri check/Gate jobs are failing because of missing ovn containers, Jobs failing with Error - Not found image: docker://docker.io/tripleou/centos-binary-neutron-metadata-agent-ovn"" [Critical,Triaged]09:36
chandankumarzbr|ruck: ykarel|lunch marios https://review.rdoproject.org/r/28562 fix for ovn09:36
chandankumarget this merged, asap09:36
marioscherrypick needs votes please when folks have time https://review.opendev.org/#/c/741383/109:38
marioschandankumar: checking09:38
ykarel|lunchack09:38
*** ykarel|lunch is now known as ykarel09:38
zbr|ruckwhere is this variable needed in production?09:38
zbr|ruckchandankumar: i can understand the code and its test, but i do not get it where is needed in production.09:39
zbr|ruckwe have a j2 file that needs this to be expanded?09:39
chandankumarzbr|ruck: pm09:40
mariossshnaidm: are you willing to revisit your -1 @ https://review.opendev.org/#/c/738844/ please09:46
akahatchandankumar, ysandeep|rover what is the decision about this job: periodic-tripleo-ci-centos-8-containers-undercloud-minion are we going to add it in the train or not?09:47
sshnaidmmarios, yeah, I'm not sure it's worth to run non-voting job until at least a first success, but maybe upgrades is an exception really09:49
ysandeep|roverakahat, i think that will work.. you can try with a test patch, but this job is currently broken even in master.. we have a bz for that09:49
mariossshnaidm: so i don't know ... the request is reasonable like 'i want to see upgrades run so i'll leave the comment to run it'09:49
mariossshnaidm: but at same time i think it will mean it never runs09:49
akahatysandeep|rover, ack.09:49
mariossshnaidm: and we want it to09:49
mariossshnaidm: fwiw, the goal is to merge the jobs now and then go in again and continue to fix bugs/voting etc09:49
mariossshnaidm: this can be discussed some more and perhaps we move all of them to experimental for master... personally i lean towards keeping in check09:50
ysandeep|roverakahat, https://bugs.launchpad.net/tripleo/+bug/1886914 fyi..09:50
openstackLaunchpad bug 1886914 in tripleo "Undercloud minion failing validation ERROR: Heat Engine host count is 1 or less." [Critical,In progress] - Assigned to Alex Schultz (alex-schultz)09:50
akahatysandeep|rover, thanks :)09:51
sshnaidmmarios, fair enough09:51
akahatysandeep|rover, it's merged. i'll create testpatch for it.09:53
ysandeep|roverakahat, yes just saw- it was fixed last night after master promotion09:54
*** dmellado has quit IRC09:59
*** dmellado has joined #oooq10:07
chandankumarmarios: commented on the patch10:20
mariosack chandankumar10:26
chandankumartwo more patches are coming10:26
ysandeep|roverakahat, arxcruz Please let me know if have inputs regarding https://bugs.launchpad.net/tripleo/+bug/1887790 , acting as gate blocker10:39
openstackLaunchpad bug 1887790 in tripleo "tripleo-ci-centos-8-scenario010-standalone check/gate/periodic jobs failing on tempest tests with error - tempest.lib.exceptions.Forbidden: Forbidden , Details: {'faultcode': 'Client', 'faultstring': 'Policy does not allow this request to be performed.', 'debuginfo': None" [Critical,Triaged]10:39
chandankumarysandeep|rover: check ovn patch in tht it will fix it10:39
chandankumarysandeep|rover: https://review.opendev.org/71463910:39
ysandeep|roverchandankumar, no its a different error, https://review.opendev.org/#/c/714639/ also hit that issue10:40
chandankumarysandeep|rover: checking10:41
arxcruzysandeep|rover: so, need to ping some guy from octavia, not sure which configuration iw that10:41
ysandeep|roverI think it started last night may be after promotion10:41
chandankumarysandeep|rover: https://review.opendev.org/731501 also landed10:42
ysandeep|roverchandankumar, hmm timing corelates to above patch10:44
ysandeep|roverbut tripleo-ci-centos-8-scenario010-standalone in the patch10:45
ysandeep|roverpassed* in the patch10:45
ysandeep|roverchandankumar, is it okay if I mark it non voting for the meantine to unblock gate?10:47
chandankumarysandeep|rover: can you check temepstconf version in the failed job10:48
chandankumar?10:48
ysandeep|roverpython-tempestconf (2.5.1.dev31) is this is the right place to look - https://6aee315f5a36c0c3ea7c-7637ce74125a0f2a5404346b147f7a3a.ssl.cf2.rackcdn.com/714639/5/check/tripleo-ci-centos-8-scenario010-standalone/23cdb0b/logs/undercloud/var/log/extra/pip.txt10:49
ysandeep|roverif* this10:49
chandankumarysandeep|rover: check rpm list10:50
ysandeep|roverpython3-tempestconf-2.5.1-0.20200710191435.e93f9ff.el8.noarch10:52
ysandeep|roverchecking good jobs to compare10:52
chandankumarysandeep|rover: https://opendev.org/osf/python-tempestconf/commit/7ee63b1517b7412c8e25f2842b207339a70f62c6 is the last commit10:53
ysandeep|roverEarlier good job have same rpm package version - python3-tempestconf-2.5.1-0.20200710191435.e93f9ff.el8.noarch - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b80/741290/5/check/tripleo-ci-centos-8-scenario010-standalone/b80afff/logs/undercloud/var/log/extra/rpm-list.txt10:55
toskyysandeep|rover: the fix for octavia tempest plugin is not there yet; it was added in the next commit10:57
ysandeep|rovertosky, Hey o/ thank you, do you have that commit handy?10:58
toskyysandeep|rover: https://opendev.org/osf/python-tempestconf/commit/7ee63b1517b7412c8e25f2842b207339a70f62c611:00
ysandeep|rovertosky, thanks!11:03
marioszbr|ruck: chandankumar: how did it go on ussuri are we waiting on promotion now?11:26
marioszbr|ruck: chandankumar: i mean re the missing containers and chandankumar patch to fix11:26
chandankumarmarios: we patched the promoter11:26
marioschandankumar: ack so i guess we are waiting on a new promotion11:26
marioszbr|ruck: ? ^^11:26
chandankumarmarios: http://promoter.rdoproject.org/centos8_ussuri.log11:26
marioschandankumar: ack :/11:27
chandankumarpromotion criteria not fulfilled so not promoted11:27
ykarelchandankumar, marios good to push missing containers atlest to unblock11:27
ykarelpull those 5 containers from trunk registry and push to docker.io11:28
ykarelno new run since last promoted hash https://trunk.rdoproject.org/api-centos8-ussuri/api/civotes_agg_detail.html?ref_hash=85fc616e61a5150ac9dd9104c77f65c011:30
ykarelnew promotion will take time11:30
ykarelso good to fix existing one11:30
mariosagree ykarel11:30
marioszbr|ruck: ysandeep|rover: 14:27 < chandankumar> promotion criteria not fulfilled so not promoted11:30
marios14:27 < ykarel> chandankumar, marios good to push missing containers atlest to unblock11:30
marios14:28 < ykarel> pull those 5 containers from trunk registry and push to docker.io11:31
marioszbr|ruck: are you folks on that? to unblock ussuri gates11:31
mariosthere is gonna be a bunch of angry americans in an hour or so ...11:31
ykareland master patches too in ci repos11:31
ykarelthey are also blocked11:31
marioszbr|ruck: chandankumar: who has access to the rdo promoter... seems my key is not there11:34
mariosPermission denied11:34
ykarelmine too removed11:34
marioswe used to all have access it was restricted at some point but i think we need to add the team11:34
chandankumarmarios: ysandeep|rover can you do that11:35
ysandeep|roveri don't have access too11:35
chandankumarysandeep|rover: use tmate11:35
ysandeep|roverreading ^^11:41
marioschandankumar: is the tmate the vexx one?11:45
marioschandankumar: i see ah... no11:45
mariostraincentos811:45
marioscentos811:45
*** pojadhav is now known as pojadhav|brb11:48
chandankumarmarios: list of missing containers {'ovn-sb-db-server', 'neutron-metadata-agent-ovn', 'ovn-nb-db-server', 'ovn-controller', 'ovn-northd'}11:52
*** rlandy has joined #oooq11:52
marioschandankumar: yes11:52
*** rlandy is now known as rlandy|ruck11:52
ysandeep|rovermarios, chandankumar, our savior is here :)11:52
ysandeep|roverrlandy++11:52
rlandy|ruckysandeep|rover: now I'm worried :)11:53
rlandy|ruckwhat's up?11:53
ysandeep|roverrlandy|ruck, please take your time to settle down, I will brief you in the meantime11:54
rlandy|ruckysandeep|rover: ok - 5 mins11:54
ysandeep|roverrlandy|ruck, We hit an issue after last night ussuri promotion, Ussuri check/gate blocked because of some missing containers. We talking about bz https://bugs.launchpad.net/tripleo/+bug/188778311:55
openstackLaunchpad bug 1887783 in tripleo "Ussuri check/Gate jobs are failing because of missing ovn containers, Jobs failing with Error - Not found image: docker://docker.io/tripleou/centos-binary-neutron-metadata-agent-ovn"" [Critical,Triaged]11:55
ysandeep|roverbasically ovn related containers are missing - list of missing containers {'ovn-sb-db-server', 'neutron-metadata-agent-ovn', 'ovn-nb-db-server', 'ovn-controller', 'ovn-northd'}11:56
mariosrlandy|ruck:11:56
rlandy|ruckmarios: ysandeep|rover: let's chat11:57
rlandy|ruckysandeep|rover: https://meet.google.com/ayw-fgke-mbg?pli=1&authuser=011:58
ysandeep|roverrlandy|ruck, chandankumar helped with https://review.rdoproject.org/r/#/c/28562/ to solve the issue .. but to unblock gate we can pull missing container from rdo and push to docker.io11:58
rlandy|ruckysandeep|rover: yeah ...11:59
rlandy|ruckjoin me on google meet11:59
ysandeep|roverrlandy|ruck, joining11:59
zbr|ruckmarios: check you google chat for tmate11:59
zbr|rucki am still waiting for a votes on mol changes: https://review.rdoproject.org/r/#/c/28455/ and https://review.rdoproject.org/r/#/c/28454/12:00
zbr|rucki also included link to hackmd12:00
*** amoralej is now known as amoralej|lunch12:04
rlandy|ruckzbr|ruck: join us on  https://meet.google.com/ayw-fgke-mbg?pli=1&authuser=012:05
rlandy|ruckzbr|ruck: we want to patch the promoter and then rerun12:05
rlandy|ruckare you doing something on that server?12:05
chandankumarzbr|ruck: any help needed on promoter server?12:16
*** rfolco has joined #oooq12:20
*** ykarel_ has joined #oooq12:23
rlandy|ruckpanda: need your help if you are around12:24
*** ykarel has quit IRC12:25
rlandy|ruckmarios: actually we need you for container pull/push question12:25
rlandy|ruck https://meet.google.com/ayw-fgke-mbg?pli=1&authuser=012:26
rlandy|ruckmarios: ^^12:26
rfolcozbr|ruck, hi, I see you merged the first patch. Thanks. Should I continue from there ? or you did some other change ?12:29
zbr|ruckrfolco: wait a bit, busy and i also need to upload other fixes to your change.12:29
rfolcozbr|ruck, ok12:30
rlandy|ruckykarel_: can you join us on  https://meet.google.com/ayw-fgke-mbg?pli=1&authuser=012:30
*** ratailor has quit IRC12:30
mariosrlandy|ruck: o/12:32
mariosrlandy|ruck: hey12:32
mariosrlandy|ruck: joining12:32
ykarel_rlandy|ruck, ack12:33
mariosrlandy|ruck: sec plugging in12:33
rlandy|ruckykarel_: no worries12:33
rlandy|ruckpanad joined12:33
rlandy|ruckpanda joined12:33
pandarlandy|ruck: https://hackmd.io/KyIJPdghQEqX1MFV974fnA12:34
*** fuzzball81 has quit IRC12:39
*** pojadhav|brb is now known as pojadhav12:40
*** fuzzball81 has joined #oooq12:41
mariosrfolco: o/13:01
rfolcogoing13:01
akahatarxcruz, rfolco, zbr|ruck , sshnaidm, rlandy, marios, ysandeep|rover , bhagyashris, soniya29, pojadhav, akahat, weshay, chandankumar meeting13:02
*** ykarel_ is now known as ykarel|away13:08
ysandeep|roverrlandy|ruck, https://review.opendev.org/#/c/741435/13:15
ysandeep|roverrlandy|ruck, https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=1fcd094313791317563b22f5dcf54d3b13:16
ysandeep|roverrlandy|ruck, https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push/c8945ab/logs/build.log13:19
*** TrevorV has joined #oooq13:32
chandankumarysandeep|rover: rlandy|ruck ykarel|away  https://review.rdoproject.org/r/#/c/28568/13:48
*** rlandy|ruck is now known as rlandy|ruck|mtg14:02
*** Goneri has joined #oooq14:02
*** pojadhav is now known as pojadhav|brb14:03
*** TrevorV has quit IRC14:15
ysandeep|roverrlandy|ruck|mtg, leaving for the day o/ , please review https://code.engineering.redhat.com/gerrit/#/c/206448/ whenever you have time.. test run is green14:29
rlandy|ruck|mtgzbr|ruck: ysandeep|rover: marios: panda: chandankumar and I are leaving the promoter inactive atm14:30
rlandy|ruck|mtgwe want to get a new hash on master to try the cherry-picked code14:30
rlandy|ruck|mtgpls leave the promoter inactive unless you need it14:31
mariosrlandy|ruck|mtg: ack14:31
ysandeep|roverack14:31
rlandy|ruck|mtgysandeep|rover: ykarel|away: marios: ussuri containers should be updated now14:32
ykarel|away\o/14:33
*** rlandy|ruck|mtg is now known as rlandy|ruck14:34
*** pojadhav|brb is now known as pojadhav14:37
mariosthanks rlandy|ruck14:44
*** TrevorV has joined #oooq14:46
*** Goneri has quit IRC14:55
*** derekh has quit IRC14:55
*** zbr|ruck has quit IRC14:55
*** irclogbot_0 has quit IRC14:55
*** Goneri has joined #oooq14:57
*** derekh has joined #oooq14:57
*** zbr|ruck has joined #oooq14:57
*** irclogbot_0 has joined #oooq14:57
*** ysandeep|rover is now known as ysandeep|away15:02
*** skramaja has quit IRC15:19
*** matbu has quit IRC15:27
*** marios has quit IRC15:31
*** udesale_ has joined #oooq15:32
*** matbu has joined #oooq15:33
*** udesale has quit IRC15:34
*** amoralej|lunch is now known as amoralej|off15:45
*** ykarel|away has quit IRC15:50
*** saneax has quit IRC15:59
*** bogdando has quit IRC16:02
*** udesale_ has quit IRC16:25
*** sshnaidm is now known as sshnaidm|afk16:28
zbr|ruckrfolco: who wrote get_hash role?16:38
rfolcozbr|ruck, I started it16:39
zbr|ruckrfolco: in that case probably you can help me with https://review.rdoproject.org/r/#/c/28571/ -- it never run on centos-716:39
zbr|rucki made one fix around use of release var, but got on another issue16:40
rfolcozbr|ruck, looking16:40
zbr|ruckfeel free to update it later, after the promoter one.16:40
*** dtantsur is now known as dtantsur|afk16:40
zbr|ruckwe have no component pipelines on c7, so no wonder it fails. i guess it needs few kicks to make it run.16:41
rfolcozbr|ruck, there is no component commit.yaml for centos7 afaik16:41
zbr|ruckthat is if we ever need to use get_hash role on centos-7 hosts, if not, we can remove the test.16:42
rfolcozbr|ruck, and there won't be16:42
rfolcozbr|ruck, centos7 use the aggregated commit yaml, not the component one.16:42
zbr|rucki am sure you know much better than me how to avoid that error.16:43
rfolcozbr|ruck, perhaps it is worth keeping the job as a negative test. get_hash needs to retrieve this instead16:43
rfolcohttps://trunk.rdoproject.org/centos7-train/consistent/commit.yaml16:43
rfolcozbr|ruck, ok let me think about it and will propose a change there16:44
rfolcozbr|ruck, keep the job as nv for now16:44
zbr|ruckperfect.16:44
*** derekh has quit IRC17:00
*** jmasud has quit IRC17:57
*** jmasud has joined #oooq17:57
rfolcozbr|ruck, good news, now we are getting much far on the mol-promoter job (until the end of verify)... easy fix.... and not breaking legacy18:14
rfolcohttps://review.rdoproject.org/r/#/c/28553/18:14
rfolcoplease review and I'll fix verify.yml (selinux issue) in a separate patch18:14
*** soniya29 has quit IRC18:43
*** Goneri has quit IRC19:14
*** Goneri has joined #oooq19:24
*** TrevorV has quit IRC20:27
*** jbadiapa has quit IRC21:20
*** rfolco has quit IRC21:45
*** rlandy|ruck is now known as rlandy|ruck|afk21:52
*** rfolco has joined #oooq22:53
*** tosky has quit IRC23:04
*** saneax has joined #oooq23:10
*** jmasud has quit IRC23:16
*** matbu has quit IRC23:16
*** zbr|ruck has quit IRC23:16
*** irclogbot_0 has quit IRC23:16
*** jmasud has joined #oooq23:22
*** matbu has joined #oooq23:22
*** zbr|ruck has joined #oooq23:22
*** irclogbot_0 has joined #oooq23:22

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!