Friday, 2020-09-11

*** sanjayu__ has quit IRC00:19
*** udesale has joined #oooq03:12
*** jmasud has quit IRC03:21
*** jmasud has joined #oooq03:23
*** skramaja has joined #oooq03:42
*** ykarel|away has joined #oooq03:54
*** udesale has quit IRC04:13
*** ratailor has joined #oooq04:18
*** ysandeep|away is now known as ysandeep04:24
*** ykarel|away is now known as ykarel04:38
*** skramaja has quit IRC04:38
*** skramaja has joined #oooq04:39
*** ratailor has quit IRC05:06
*** ratailor has joined #oooq05:07
*** marios has joined #oooq05:08
*** ratailor has quit IRC05:11
*** udesale has joined #oooq06:24
*** apetrich has joined #oooq06:53
*** apetrich has quit IRC06:55
*** apetrich has joined #oooq06:56
*** jtomasek has joined #oooq07:02
*** amoralej|off is now known as amoralej07:07
*** saneax has joined #oooq07:24
*** bogdando has joined #oooq07:28
*** ysandeep is now known as ysandeep|lunch07:34
*** saneax has quit IRC07:44
*** jpena|off is now known as jpena07:44
matbujpena: o/ morning can we merged this cherry pick ? https://review.rdoproject.org/r/#/c/29131/07:52
matbumarios: o/ hey dude, another CI expertise question... do we build master branches somewhere with delorean ? I wanted to check something07:53
matbujpena: thank you07:53
mariosmatbu: yeah maybe you are referring to 'current' ?07:55
mariossec will fetch link07:56
mariosmatbu: so e.g. https://trunk.rdoproject.org/centos8-master/ ... 'current' is the latest commits from the projects07:56
mariosmatbu: https://trunk.rdoproject.org/centos8-master/current/delorean.repo07:56
matbumarios: from all of us ? i'm surprised because the delorean.repo contains per component path07:57
mariosmatbu: yeah for centos8 it is all 'component' based... you want the tripleo component then ?07:57
marioshttps://trunk.rdoproject.org/centos8/component/tripleo/05/04/0504cf2dbff1003ecd27744ce2555cda257effc2_2dfb8295/07:58
mariosmatbu: that ^07:58
matbumarios: actually i wanted validation component (i'm looking for validation package)07:58
mariosopenstack-tripleo-validations-12.4.1-0.20200910204932.928fcb0.el8.noarch.rpm07:58
mariosmatbu: ^^?07:58
matbumarios: https://trunk.rdoproject.org/centos8/component/validation/94/6e/946e3a858686564c9c2c2e5cf5b479655a9adfc4_d9a985ba/07:58
mariosmatbu: ack i was going to ask... if you want openstack-tripleo-validations then tripleo component07:59
matbumarios: but it pin the tag07:59
mariosmatbu: but if you want python3-validations then the one you pointed07:59
mariosah also validations-common in there ... k07:59
matbumarios: yep and not sure if it's the latest master commit08:00
matbui will check that08:00
matbuha nop it's pinning the latest version08:01
matbumarios: do you know where this component pipeline is defined and where i can play with it ? (modify thing, adding test and so on)08:02
matbusorry for bugging you about that on a friday morning :D08:02
mariosmatbu: np but what do you want? e.g. to run a job?08:02
matbumarios: well i want to see why it's not building last master commit and see how we can add job to test things08:03
matbueventually run a job yes08:03
matbufew years ago all of that was in centos ci but i guess it has changed08:03
mariosmatbu: yeah let me fetch you something for validation component08:03
mariosmatbu: https://review.rdoproject.org/r/gitweb?p=config.git;a=blob;f=zuul.d/upstream.yaml#l493 https://review.rdoproject.org/r/gitweb?p=rdo-jobs.git;a=blob;f=zuul.d/project-templates-components.yaml#l425 https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/component-jobs.yaml#L113408:05
matbumarios: hmm thank you08:08
matbumarios: i will look into that, for example i see validation-ussuri which is not relevant since we are branchless in VF08:09
matbujust the branch ussuri remain on the repos, but we need to drop it08:09
*** ysandeep|lunch is now known as ysandeep08:36
*** ssbarnea has joined #oooq08:38
*** derekh has joined #oooq08:42
*** tosky has joined #oooq08:47
*** pojadhav is now known as pojadhav|afk08:55
akahatmarios, bhagyashri|rover i'm updating panda's patch (Promoter new configuration engine: stage environment).. hoping that it will fix the log file issue.08:56
mariosakahat: which log file issue? I am working on the py36 test specifically test_promoter_functional.py::test_promote_qcows[qcow_legacyconf_single] (I know what the problem is now and trying to fix it)08:57
mariosakahat: are you working on tripleo-ci-promotion-staging-single-pipeline-centos-8 ?08:58
akahatmarios, yes. that's right.09:00
mariosakahat: ack ok cool well, lets see if we can work around each other, i am focusing on the py36. i will check with you before updating again working locally for now09:01
akahatmarios, okay.09:01
*** ssbarnea has quit IRC09:06
mariosakahat: please be careful otherwise we can't work at the same time https://review.rdoproject.org/r/#/c/28081/61..62/ci-scripts/dlrnapi_promoter/test_ini_files_validation.py09:10
mariosakahat: looks like you didn't have the latest version before pushing your updates09:11
mariosakahat: https://review.rdoproject.org/r/#/c/28081/61..62/ci-scripts/dlrnapi_promoter/test_ini_files_validation.py09:11
akahatmarios, oops..09:11
akahatlet me fix it..09:11
mariosakahat: ack09:12
akahatmarios, fixed it.. thanks.. i'll take care next time :)09:15
mariosakahat: ack09:15
*** ratailor has joined #oooq09:16
mariosakahat: i'll wait for the run and i will update something minor for the py36 tests09:16
mariosakahat: k?09:16
akahatmarios, yes. sure09:16
*** chem has joined #oooq09:18
*** jtomasek has quit IRC09:28
mariosakahat: going to update09:31
akahatmarios, okay.. i think i need to add expanduser.. i'll update in next run.09:31
mariosakahat: ack gimme couple mins grabbing v63 to update09:31
akahatsure.09:32
mariosakahat: k done please wait for the run before updating again09:33
mariosakahat: oops nit updating again sec09:34
mariosakahat: k now done ;)09:34
*** saneax has joined #oooq09:35
akahatmarios, i like promoter jobs.. it's not taking too much time :)09:36
*** ykarel is now known as ykarel|lunch09:36
mariosakahat: ;)09:39
*** sanjayu_ has joined #oooq09:41
*** saneax has quit IRC09:42
akahatmarios, you and me are on same page :)09:42
bhagyashri|roverakahat, ack09:47
mariosakahat: looks like it breaks the py36 test I mean the update you made ... ERROR    promoter:common.py:164 Invalid Log file: /home/zuul/web/promoter_logs/centos8_master.log https://logserver.rdoproject.org/81/28081/65/check/tox-py36-ci-config/9d7161d/tox/reports.html09:48
mariosakahat: does it fix what you were chasing?09:48
akahatmarios, yes..09:50
akahatmarios, sending review..09:53
akahatmarios, i messed up.. fixing.09:54
mariosakahat: ack09:56
mariosakahat: no akahat please be careful you are overwriting my changes again09:57
marioshttps://review.rdoproject.org/r/#/c/28081/65..66/ci-scripts/dlrnapi_promoter/qcow_client.py09:57
mariosakahat: T^09:57
marioshttps://review.rdoproject.org/r/#/c/28081/65..66/ci-scripts/dlrnapi_promoter/test_promoter_functional.py09:57
marios!09:57
akahatmarios, yes.. i fetched your patch first.. then updated the changes.. don't know how it got override.09:58
akahat:(09:58
mariosakahat: k please restore09:58
mariosakahat: thanks10:04
*** ratailor_ has joined #oooq10:09
akahatmarios, quick question: why there are two os.stat https://review.rdoproject.org/r/#/c/28081/67/ci-scripts/dlrnapi_promoter/promoter_integration_checks.py@257 and 25110:10
*** ratailor has quit IRC10:11
mariosakahat: i don't know i didn't touch that code... looks like there are chained exceptions though maybe panda can tell us more later10:14
mariosakahat: well... the extra one looks like an addition 'not from panda' https://review.rdoproject.org/r/#/c/28081/10..67/ci-scripts/dlrnapi_promoter/promoter_integration_checks.py10:14
mariosakahat: added note for later10:15
akahatmarios, okay.. sending review patch with some minor linting issues fixed.10:16
mariosakahat: but still breaks the py36 tests https://logserver.rdoproject.org/81/28081/67/check/tox-py36-ci-config/ebc59aa/tox/reports.html ERROR    promoter:common.py:164 Invalid Log file: /home/zuul/web/promoter_logs/centos8_master.log10:16
mariosakahat: and we had the same discussion with bhagyashri|rover and panda but lets see on the call today10:16
*** ykarel|lunch is now known as ykarel10:16
mariosakahat: imo we should leave the linting last but sure10:17
mariosakahat: are you done with the single-pipeline stuff still failing10:17
akahatmarios, let me test this one patch..10:18
bhagyashri|rovermarios, yes that day my concern was where exactly that file has been created i know the setup_logging() method. but afaik that method logs the content in the logfile which we passed to this method but before passing log_filr to setup_logging() method the file should be created10:19
bhagyashri|roverakahat, ^10:19
*** jmasud has quit IRC10:20
akahatbhagyashri|rover, i think file is created by setup_logging() method.. we need to crate directory.10:22
*** jmasud has joined #oooq10:22
bhagyashri|roverakahat, we dont need to create directory that is been created here10:24
bhagyashri|rover2020-09-11 10:04:50.228359 | LOOP [promoter : Create a directory for the promoter logs]10:24
bhagyashri|rover2020-09-11 10:04:50.516537 | rdo-centos-8 | changed: "~/web/promoter_logs"10:24
bhagyashri|rover2020-09-11 10:04:50.516972 | rdo-centos-8 | changed: All items complete10:24
bhagyashri|rover2020-09-11 10:04:50.517058 |10:24
akahatbhagyashri|rover, oh.. okay..10:32
akahatmarios, you need to remove "\n" from ssh key otherwise it will fail.10:35
akahathttps://logserver.rdoproject.org/81/28081/68/check/tox-py36-ci-config/3093def/job-output.txt10:35
mariosakahat: thanks was switching to that one next, it should be the only failing test left (apart from the logfile stuffs but we can discuss those)10:38
mariosakahat: well for py36 anyway10:38
*** chem has quit IRC10:50
*** chem has joined #oooq10:53
akahatmarios, good news.. 1 job is remaining to fix. :)10:57
mariosakahat: thanks i saw the update10:58
mariosdid it report already?10:58
*** sanjayu__ has joined #oooq11:03
mariosakahat: looks like it still hit it11:04
marioshttps://logserver.rdoproject.org/81/28081/69/check/tox-py36-ci-config/b7c3929/tox/reports.html ERROR    promoter:common.py:164 Invalid Log file: /home/zuul/web/promoter_logs/centos8_master.log11:04
*** sanjayu_ has quit IRC11:05
marios        * Passed ci-scripts/dlrnapi_promoter/test_qcow_client_unit.py::TestQcowConnectionClient::test_instance_sftp_connect_close11:07
mariosakahat: i am going to update now? is that OK or are you updating again for the logfile11:07
akahatmarios, give me few mins.. i'm updating remaining places.11:08
mariosakahat: ack let me know then np11:09
mariosakahat: take yr time11:09
akahatmarios, thanks :)11:09
* chandankumar is stepiing out for an hour11:11
* zbr anyone up for a virtual coffee?11:14
marioszbr: you stood me up 2x @ the watercooler so i'm gonna ghost you for a while11:15
marioszbr: instead i'm gonna do some cardio11:15
zbrgood idea with cardio, true, we need to move it a little bit later.11:15
marioszbr: ;)11:15
zbrarx also requested that, and i observed that I do not always start the day at 8am11:16
zbri am not sure how we can find good times for these11:17
bhagyashri|roverakahat, ahh i never thought on os.path.expanduser() great!11:17
zbrthere is also an expandvars11:18
*** jtomasek has joined #oooq11:28
*** sanjayu__ has quit IRC11:28
*** chem has quit IRC11:29
*** chem has joined #oooq11:31
* zbr out for lunch11:40
*** jpena is now known as jpena|lunch11:42
weshay|ruckmatbu, 0/11:47
weshay|ruckneed to check tags for validation rpms?11:48
weshay|ruckbhagyashri|rover, rfolco need some eyes on http://dashboard-ci.tripleo.org/d/wb8HBhrWk/cockpit?orgId=1 "upstream periodic jobs"11:58
akahatmarios, hey.. i'm done with the patch... you can push your code.11:58
akahatmarios, one more thing before you push.. when last time you tested that ssh key???11:59
mariosakahat: ack yes i changed the key i did not add it it was addd in v1311:59
mariosakahat: when i changed it locally it worked OK11:59
akahatmarios, i mean this ssh key is not valid..  https://review.rdoproject.org/r/#/c/28081/70/ci-scripts/dlrnapi_promoter/test_unit_fixtures.py@44 one character is missing.12:00
akahatso it might not work.. you need to change the key.12:00
mariosakahat: yes12:00
mariosakahat: exactly i did and it works12:00
mariosakahat: it should make py36 green, checking latest run12:00
mariosakahat: i will update in a sec12:00
mariosakahat: still one more on the log file issue? https://logserver.rdoproject.org/81/28081/70/check/tox-py36-ci-config/cf128ab/tox/reports.html12:01
mariosERROR    promoter:common.py:164 Invalid Log file: /home/zuul/web/promoter_logs/centos8_master.log12:01
*** rfolco|ruck has joined #oooq12:01
mariosakahat: are you done updating or will you do that one too ?12:01
bhagyashri|roverweshay|ruck, ack looking12:01
*** rlandy has joined #oooq12:01
mariosakahat: holding on updating for now can do it after the call go ahead ...12:03
* marios brb12:03
weshay|ruckbhagyashri|rover, mark this.. "promotion-blocker" https://bugs.launchpad.net/tripleo/+bug/189528812:06
openstackLaunchpad bug 1895288 in tripleo "[train] MemoryError: [tripleotraincentos8/centos-binary-nova-libvirt] Memory Error failing on few jobs" [High,Triaged]12:06
weshay|ruckhttps://bugs.launchpad.net/tripleo/+bug/1895248 same here12:06
openstackLaunchpad bug 1895248 in tripleo "[stein] (LoadBalancerScenarioTest:test_load_balancer_ipv4_CRUD) show_loadbalancer provisioning_status updated to an invalid state of ERROR is failing on fs038" [High,Triaged]12:06
bhagyashri|roverweshay|ruck, done12:06
bhagyashri|roversorry i forgot to add that12:07
bhagyashri|roverthere is one more issue on train but waiting for testproject result12:07
weshay|ruckk12:08
weshay|ruckthanks12:08
matbuweshay|ruck: o/12:09
weshay|ruckmatbu, hey12:09
matbuweshay|ruck: i reverted one patch from tripleo-validations12:09
matbuif you are talking about the promotion blocker12:10
weshay|rucksee it12:10
weshay|ruckmatbu, https://review.opendev.org/#/c/751244/12:10
weshay|ruckright?12:10
matbuwe are pinning validations-libs in rdoinfo, i think we need to track master instead12:10
weshay|ruckhow does that impact packaging?12:10
* weshay|ruck looks12:11
weshay|ruckat rdoinfo12:11
matbuweshay|ruck: it override path between validation-common and tripleo-validations12:12
matbuweshay|ruck: it's a long story, but we split roles between v-common and tripleo-v12:12
weshay|ruckk.12:13
matbuweshay|ruck: but the review that was doing the clean in tripleo-v wasn't merged, so rpm is not happy12:13
weshay|ruckwant me to test it in periodic?12:13
matbuthe revert ?12:14
matbuif it's possible it would be nice12:15
*** Goneri has joined #oooq12:16
*** amoralej is now known as amoralej|lunch12:17
bhagyashri|roverweshay|ruck, matbu fyi i have tested the master job https://review.rdoproject.org/r/#/c/29224/ depends on: https://review.opendev.org/#/c/751244/ but it's still failing12:19
*** pojadhav|afk is now known as pojadhav12:34
rfolco|ruckbhagyashri|rover, pls see last comment in https://review.opendev.org/#/c/751244/12:40
bhagyashri|roverrfolco|ruck, ok thanks12:41
*** ratailor_ has quit IRC12:42
*** skramaja has quit IRC12:43
*** jpena|lunch is now known as jpena12:44
rfolco|ruckbhagyashri|rover, too many issue looks like, where do you want me to help with12:45
bhagyashri|roverrfolco|ruck, actually i just reported train issue after 2nd run result12:48
bhagyashri|roverso probably train12:48
bhagyashri|roverwill need more focus12:48
rfolco|ruckbhagyashri|rover, ignore the comment in https://review.opendev.org/#/c/751244/, see ykarel response. Maybe he has some other ideas on what is missing...12:50
ykarelrfolco|ruck, missing in what?12:50
rfolco|ruckykarel, bhagyashri|rover is applying that patch to see if the validation bug is gone. But still failing. Right, bhagyashri|rover ?12:51
bhagyashri|roverrfolco|ruck, yes it's still failing with conflict error12:51
bhagyashri|roverhttps://review.opendev.org/#/c/751244/12:52
ykarelokk ^, there patch is not being tested12:52
ykarelit fails early then patch actually get's applied12:52
bhagyashri|roverykarel, how? i see you put recheck on the patch after that also it failed12:58
ykarelbhagyashri|rover, repo-setup <- install packages <- dlrn builds <- install build packages12:59
ykarelit failed at instal packages13:00
bhagyashri|roverok13:03
mariosakahat: ok... going to update now?13:06
mariosakahat: (i am going to update) then its yours gimme couple mins just finished phone13:07
akahatmarios, yes. please update.. it will take some time to debug for me.13:07
mariosakahat: thanks13:07
*** matbu has quit IRC13:09
*** matbu has joined #oooq13:10
mariosakahat: k done at v71 https://review.rdoproject.org/r/#/c/28081/71/ci-scripts/dlrnapi_promoter/test_unit_fixtures.py13:10
mariosakahat: all yours13:10
akahatmarios, cool.. thanks :)13:11
rlandyysandeep: hey ... we didn't promote 16.2 or 17 yesterday as I wanted to go through  the process with you13:13
ysandeeprlandy, sure, please let me know whenever you are free13:14
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-periodic-rhos-1713:14
rlandyysandeep: ^^ 17 looked better yesterday :(13:14
ysandeep:(13:15
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-rhos-17/e189bcf/logs/undercloud/var/log/tempest/stestr_results.html13:15
rlandyysandeep: ^^ tempest failures13:15
rlandywonder if it's at all related to the upgraded dep from yesterday?13:16
rlandyperiodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2openstack/tripleo-cimasteropenstack-periodic-integration-rhos-16.2master4 hrs 8 mins 25 secs2020-09-10 13:01:09FAILURE13:16
rlandyysandeep: ovb has been failing in 16.213:16
rlandyand it's running now13:16
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2/c33181e/logs/undercloud/var/log/tempest/stestr_results.html.gz13:17
rlandyysandeep: ^^ tempest issues there as well13:17
rlandyysandeep: possibly because we increased the node numbers???13:17
ysandeeprlandy, need to check..13:18
weshay|ruckysandeep, https://review.opendev.org/#/c/751273/13:19
rlandyysandeep: ok - let me know what you find - I need to work on getting the container builds modified for rel del import13:19
*** amoralej|lunch is now known as amoralej13:19
ysandeeprlandy, i am able to proceed with envC - using Ironic root disk hints - https://code.engineering.redhat.com/gerrit/#/c/211472/13:19
rlandygreat13:20
weshay|ruckysandeep, was it chrony?13:20
ysandeepwe can add root disk hint  in instack.json to tell which root disk to use.. i will close the pnt ticket13:21
ysandeepno need to pull pendrive as we have a workaround13:21
*** TrevorV has joined #oooq13:27
weshay|ruckbhagyashri|rover, rfolco|ruck need anything?13:32
rfolco|ruckweshay|ruck, just unsure what to do with validation bug13:32
rfolco|ruckweshay|ruck, https://review.opendev.org/#/c/751244/ isn't enough looks like13:33
weshay|ruckrfolco|ruck, get w/ matbu13:33
weshay|ruckmatbu, can you debrief rfolco|ruck please13:34
rfolco|ruckmatbu, anything we can help with ? maybe some testproject test to ensure fixes are in place ?13:34
matburfolco|ruck: hey, why do you think it's not enough ?13:37
*** udesale_ has joined #oooq13:37
matburfolco|ruck: we want to revert this one in order to be able to merge .. (let me find the link)13:37
matburfolco|ruck: this https://review.opendev.org/#/c/713204/13:37
rfolco|ruckmatbu, ok, so revert https://review.opendev.org/#/c/751244/ and merge https://review.opendev.org/#/c/713204/13:38
weshay|ruckzbr, 0/13:38
zbro/13:38
matburfolco|ruck: right and then re-revert :D13:38
*** udesale has quit IRC13:39
rfolco|ruckbhagyashri|rover, can I borrow your testproject to test what matbu just said?13:39
rfolco|ruckmatbu, thx for clarifying13:39
bhagyashri|roverrfolco|ruck, sure13:40
rfolco|ruckbhagyashri|rover, could you paste the link pls13:40
*** saneax has joined #oooq13:40
bhagyashri|roverrfolco|ruck, https://review.rdoproject.org/r/#/c/29224/13:41
rfolco|ruckbhagyashri|rover, thx13:41
weshay|ruckzbr, let's chat early so I can get you out of here13:41
zbrsure, joining now.13:42
weshay|ruckzbr, https://github.com/weshayutin/gri13:44
ysandeeprlandy, do we need to add list all of internal projects here - https://code.engineering.redhat.com/gerrit/#/c/210870/7/zuul.d/component-jobs.yaml@1288 , or only these few (for now i copied your ovb job required-projects list).13:50
*** matbu has quit IRC13:51
*** matbu has joined #oooq13:53
weshay|ruckrfolco|ruck, zbr we need zbr's work properly in jira13:58
rfolco|ruckzbr, talk to me boy13:58
rfolco|ruckzbr, if you want, send me a list of tasks and i create them for you ok13:59
rlandyysandeep: you can skip name14:07
rlandyjust add the project14:07
rlandyas far as  repos, walk the parenting scheme14:08
rlandyas long as you are not getting any upstream repos, it's fine14:08
ysandeeprlandy, I am just wondering why we added only these.. and not all internal projects.. like openstack-tripleo-heat-templates14:09
akahatready to merge: https://review.rdoproject.org/r/#/c/28081/14:11
akahatmarios, bhagyashri|rover weshay|ruck ^^14:11
bhagyashri|roverakahat, ack14:12
mariosakahat: bhagyashri|rover: rfolco|ruck: arxcruz|pto: \o/14:13
mariosnice work all14:13
mariosakahat: nicely done just in time for friday ;)14:13
marioslinters too!14:13
bhagyashri|rover\0/14:14
rfolco|ruckmarios, akahat good job, will review it by the end of the day14:15
mariosrfolco|ruck: no rush14:15
mariosrfolco|ruck: i don't think we should merge this before monday14:15
rfolco|ruckmarios, yes, I'll kill who does that14:15
rfolco|ruckmarios, patches like that should disable merge button on fridays automatically14:16
weshay|ruckakahat, sec14:16
rfolco|rucktopic: new_config_engine-patch7114:17
rfolco|rucklol14:17
akahat:)14:17
akahatgit automated naming system '14:18
mariosakahat: rfolco|ruck: i rebased the two patches on top of that let's see if their jobs are green too14:18
marioshttps://review.rdoproject.org/r/28019 https://review.rdoproject.org/r/2802014:18
marioshmm 'functional tests' i expect those will fail14:18
rfolco|rucknice14:20
weshay|ruckzbr, https://gitlab.cee.redhat.com/whayutin/gerrit-tools14:21
akahatmarios, okay... looking in to it.14:23
mariosakahat: wait it didn't report yet14:23
mariosakahat: monday :)14:23
mariosakahat: surely it must be end of day at 1954 ?14:24
marios:)14:24
mariosakahat: or do you shift your hours?14:24
rlandyysandeep: probably should be14:25
rlandyjust working on containers - will review in a bit14:25
akahatmarios, yes.. it's end of day.. my shift my hours.. :P14:26
ysandeeprlandy, thanks!, nw please take your time14:26
rlandysorry - just trying to spring rel del free with dockerfile imports14:26
mariosakahat: "the bugs *never* end" ... that's the problem ;)14:26
ysandeeprlandy, I have some bad news too.. we now need podman >= 1.6.4-15 , so have to unwind the stuff we did earlier :(14:28
ysandeepafter https://review.opendev.org/#/c/739481/ i hit some issue with bm baremetal.. tengu said we need >= 1.6.4-1514:28
ysandeeppodman* >= 1.6.4-1514:29
ysandeepi am checking what we are getting from rhosp-rhel-8.3-appstream currently, earlier we had issues with 1.6.4-1314:29
Tenguysandeep: can you check what streams you have for container-tools ?14:30
Tengufor instance, on osp-16.1, we must switch to 2.0 (default is rhel8) - maybe that's the same trick on osp-16.2?14:30
ysandeepTengu, we actually pinned last to last week as earlier we had some issues with podman 1.6.4-13..14:30
Tengumeh...14:30
Tengupodman, that never-ending story.14:31
rlandyysandeep: Tengu: so we did two things ...14:33
ysandeeptengu - 1.6.4-14 is what we are getting from rhosp-rhel-8.3-appstream currently, so its still not  >= 1.6.4-15 :(14:34
Tengudamned...14:34
ysandeep podman                   x86_64                   1.6.4-14.module+el8.3.0+7660+b7198318                     rhosp-rhel-8.3-appstream14:34
rlandywe disabled the  module_hotfixes=1 option so we would not get the podman version from http://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp16-2/osptrunk-deps.repo14:34
rlandyand then we locked podman to the beta version of rhel 8.314:34
rlandywe are pull beta on podman14:35
rlandybut rhos-release is on a diff version14:35
rlandyso we could take podman from there14:35
rlandygetting ...14:35
zbrrfolco|ruck: are you switching accounts or something like that? i tried to reply you on privated and got an error14:36
zbri wonder if this has anything to do with ruck rovering and nick change.14:36
rlandytoplink/vol/rhel-8/packages/podman/1.9.3/2.module+el8.2.1+6867+366c07d6/x86_64/podman-tests-1.9.3-2.module+el8.2.1+6867+366c07d6.x86_64.rpm14:38
rlandy^^ is in osp-trunk-deps14:38
ysandeeprlandy yes http://download.eng.bos.redhat.com/brewroot/repos/rhos-16.2-rhel-8-trunk-build/latest/x86_64/pkglist have podman-1.9.3 , switching to that probably we good idea..14:38
rlandy1.914:38
rlandyysandeep: we need to check the next cut of appstream14:38
ysandeeprlandy, some chatter ongoing on rel-del regarding this.. reading14:39
rlandy podman-1.6.4-20.module+el8.3.0+7842+fbbcd85c.x86_64.rpm                                     27-Aug-2020 14:00   12M14:40
rlandyysandeep: Tengu: if we get snapshot 3 of rhel 8.3 you will get ^^14:41
rlandyso request to update rhos-release to the new snapshot14:41
rlandyor add that as an additional repo and pull podman from there14:41
Tenguah. that would be better. Probably not without any error, but.....14:41
Tengurlandy, ysandeep in //: https://bugzilla.redhat.com/show_bug.cgi?id=1878187  (and I'll create the same for 16.2 now, with just the lower-constraint on 1.6.4-15)14:42
openstackbugzilla.redhat.com bug 1878187 in rhosp-release "Add upper and lower contraint for podman version" [High,New] - Assigned to jjoyce14:42
ysandeepTengu, thanks!14:42
Tengunp14:43
zbrwhat field are we supposed to use for qe on jira? "QA Contacts" or "QE Contacts",... not sure which jira admin allowed this to happen.14:43
Tenguysandeep: https://bugzilla.redhat.com/show_bug.cgi?id=1878189  for 16.214:46
openstackbugzilla.redhat.com bug 1878189 in rhosp-release "Add lower contraint for podman version" [High,New] - Assigned to jjoyce14:46
Tenguguess that should help a lot.14:46
ysandeepTengu, ack o/14:47
Tenguysandeep: and Alex is adding that new package to tripleo-ansible/bootstrap now.14:48
Tengumeaning we should be clear soon :).14:48
*** jtomasek has quit IRC14:49
rfolco|ruckzbr, I received and replied your msg14:55
rfolco|ruckzbr, I see your story on jira, are you ok with jira ?14:55
zbrrfolco|ruck: no idea where to add someone as QE14:57
rfolco|ruckzbr, there isn't a field like we had in ++ taiga ++ ... but I just added in your task as QA contacts .... edit the task, show all fields...15:00
zbrthere are two fields "QA .." and "QE ..." hidden by default, but you can enable them in edit.15:00
zbri was not sure about QA vs QE, ... like spaces vs tabs kind of subject.15:01
rfolco|ruckzbr, qe contact15:02
rfolco|ruckzbr, look again in the board, now I show it in the card15:02
ykarelrfolco|ruck, bhagyashri|rover is the issue with stein promotions known?15:02
ykareli see last promotion was 10 days ago15:02
rfolco|ruckykarel, yes :( let me get the bug15:02
rfolco|ruckykarel, octavia plugin issue: https://bugs.launchpad.net/tripleo/+bug/189524815:03
openstackLaunchpad bug 1895248 in tripleo "[stein] (LoadBalancerScenarioTest:test_load_balancer_ipv4_CRUD) show_loadbalancer provisioning_status updated to an invalid state of ERROR is failing on fs038" [High,Triaged]15:03
ykarelrfolco|ruck, but this is not in promotion criteria15:03
ykarelso promotion is not blocked atleast due to this15:04
rfolco|ruckwill double check last promotion log15:04
bhagyashri|roverykarel, this is http://promoter.rdoproject.org/config/CentOS-7/stein.ini15:04
rfolco|ruckbhagyashri|rover, weshay|ruck if you know stein blocker at the top of your head please speak up15:04
weshay|ruckrfolco|ruck, ya.. went through it bhagyashri|rover yesterday.. we accounted for all the errors and she's driving stein15:05
weshay|ruckbhagyashri|rover, you should probably add a few notes re: stein to the hackmd when you have time15:06
bhagyashri|roverweshay|ruck, added for fs03815:07
ykarelbhagyashri|rover, stein promotion likely not running ^15:07
weshay|ruckykarel, why?15:07
weshay|ruckykarel, V15:08
weshay|ruckhttp://38.102.83.109/promoter_logs/centos7_stein.log15:08
ykarelweshay|ruck, yes  here it's running but not in what bhagyashri|rover shared15:08
ysandeeprlandy, i am out now, see you on monday, when you get time.. please review below changes, these are ready to be merged:-15:08
ysandeephttps://code.engineering.redhat.com/gerrit/#/c/210870/  - envb had green run already(will deal with podman isssue sepately)15:08
ysandeephttps://code.engineering.redhat.com/gerrit/#/c/211472/ - envc root disk hint15:08
rlandysure15:09
weshay|ruckysandeep, btw.. you have my 100% approval if you want to pull baremetal from 17 to augment anything in 16.215:09
bhagyashri|roverykarel, weshay|ruck periodic-tripleo-ci-centos-7-standalone-upgrade-stein is failing earlier because of some tempest test failure but recently on testproject tempest test are passing here https://review.rdoproject.org/r/#/c/29231/ but failing with some different issue15:10
weshay|ruckysandeep, not that you have thought about that yet.. but totally fine by me as you move forward15:10
rlandyugh linters ....15:10
ykareli also see https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein# failing since 1st15:11
ykareland is in criteria15:11
bhagyashri|roverykarel, ohh i was referring wrong criteria15:12
rlandyzbr: hey - can you use yaml anchors in in var1.var2 variables like in https://code.engineering.redhat.com/gerrit/#/c/211525/1/zuul.d/tripleo-build-containers.yaml15:12
rlandyline 37215:12
ysandeepweshay|ruck, thanks.. i was able to pull envc from last error with a workaround patch - https://code.engineering.redhat.com/gerrit/#/c/211472/  but tripleo component failing to start mysql container(will continue debug on monday)..15:13
rfolco|ruckbhagyashri|rover, stein promotion log is huge, still loading for me15:13
weshay|ruckysandeep, k.. thank you .. have a good weekend15:13
rlandy2020-09-11 15:01:13.521765 | upstream-centos-7 | zuul.d/tripleo-build-containers.yaml:372:20: [error] too many spaces after colon (colons)15:13
bhagyashri|roverrfolco|ruck, ykarel this is the criteria list http://38.102.83.109/config/CentOS-7/stein.ini right?15:13
weshay|ruckrlandy, did you folks promote 16/1715:14
ykarelbhagyashri|rover, yes15:14
rlandyweshay|ruck: no - 16.2 was still running - so tripleo-ci-testing was not tests15:14
rlandytested15:14
rlandyand 17 failed in the last run15:14
rlandytempest errors15:14
rlandywhich I have not investigated yet15:14
rlandybusy helping get dockerfiles in shape for rel del imports15:15
*** ysandeep is now known as ysandeep|away15:15
weshay|ruckk.. rlandy bhagyashri|rover arx has a little patch to add to be able to skip tests on specific jobs15:15
weshay|ruckping him monday15:15
rlandyweshay|ruck; ack15:16
rlandybut we nmay have a real problem15:16
bhagyashri|roverykarel, currently upgrade-stein and featureset037-updates-stein is blocking15:17
rlandy I can promote without passing15:17
bhagyashri|roverweshay|ruck, ack15:17
rlandyidk if we want to15:17
ykarelbhagyashri|rover, ok is there any bug for those?15:18
rlandyzbr: nvm - fixed15:18
rfolco|ruckykarel, bhagyashri|rover I can file a bug for stein...15:20
rfolco|ruckCommandError: Error: The following files were not found: /home/zuul/containers-default-parameters.yaml15:20
rfolco|ruckhttps://logserver.rdoproject.org/openstack-periodic-integration-stable3/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein/9b5a301/logs/undercloud/home/zuul/overcloud_update_prepare.log.txt.gz15:20
bhagyashri|roverykarel, no. tbh the failures are inconsistent for upgrade-stein15:20
ykarelinconsistent also need to be root caused/solved :)15:21
bhagyashri|roverykarel, here tempest test are failing https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable3&job_name=periodic-tripleo-ci-centos-7-standalone-upgrade-stein15:22
bhagyashri|roverand on testproject patch https://review.rdoproject.org/r/#/c/29231/ tempest test are passing and failing with some different error15:23
bhagyashri|roverykarel, will check in detail15:23
bhagyashri|roverrfolco|ruck, ack, i think we should re run the upgrade-stein job on testproject https://review.rdoproject.org/r/#/c/29231/ wdyt?15:26
weshay|ruckykarel, bhagyashri|rover we're talking about the tempest failures?15:27
rfolco|ruckbhagyashri|rover, ykarel weshay|ruck inconsistent or not, the bug is created https://bugs.launchpad.net/tripleo/+bug/189531415:27
openstackLaunchpad bug 1895314 in tripleo "[stein] overcloud_update missing files" [Critical,Triaged]15:27
weshay|ruckre: inconsistent15:27
weshay|ruckbhagyashri|rover, did you testproject a change to cp container-preppare-params to containers-default-para?15:28
weshay|ruckdon't see one15:29
bhagyashri|roverweshay|ruck, nope i dont have patch for that yesterday we try to search but havent found that15:29
weshay|ruckrfolco|ruck, ^15:29
rfolco|ruckweshay|ruck, ok let me find where to rename this15:29
bhagyashri|roverand today also i tried to search but havent found that15:29
weshay|ruckbhagyashri|rover, in tqe.. after we finish writing the containers-prepare-param file15:30
weshay|ruckcp it to /home/zuul/containers-default-parameters.yaml15:30
rfolco|ruckbhagyashri|rover, you want to have the honor ?15:30
ykarelbhagyashri|rover, rfolco|ruck, ack15:31
rfolco|ruckbhagyashri|rover, working on it, speak up if you are too15:32
weshay|ruckbhagyashri|rover, http://codesearch.openstack.org/?q=containers-prepare-parameter.yaml&i=nope&files=&repos=15:32
weshay|rucknot sure why tripleo-upgrade is not respecting https://opendev.org/openstack/tripleo-upgrade/src/branch/master/defaults/main.yml#L10115:32
weshay|ruckah.. now I see15:33
weshay|ruckhttp://codesearch.openstack.org/?q=containers-default-parameters.yaml&i=nope&files=&repos=15:33
weshay|ruckbhagyashri|rover, rfolco|ruck see the issue?15:34
rfolco|ruckyep, just don't see why stein15:34
weshay|ruckcode search is your friend15:34
rfolco|ruckaaah15:34
rfolco|ruckfs37 does not have the condition15:35
rfolco|rucklike others15:35
rfolco|ruckhttps://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset037.yml#L1815:35
bhagyashri|roverrfolco|ruck, weshay|ruck hey sorry i was little way from keyboard. reading ...15:36
weshay|ruckimho... it should all be switch to a single file name15:36
rfolco|ruckweshay|ruck, but others have this https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset026.yml#L22-L2615:36
weshay|ruckbhagyashri|rover, please ask for help if something is not making sense.15:36
bhagyashri|roverweshay|ruck, ack15:37
weshay|ruckrfolco|ruck, so.. the only reason default-params15:37
weshay|ruckis used was because upgrades were not using the option to specify a file15:37
weshay|ruckwhich is wrong15:37
weshay|ruckso upgrades would just use the default from tc15:38
weshay|ruckhttps://opendev.org/openstack/tripleo-common/src/branch/master/tripleo_common/constants.py#L18415:38
weshay|ruckjose luis is the guy to sync w/15:38
weshay|ruckI tried to find his patch yesterday15:38
weshay|ruckbut didn't see it15:38
weshay|ruckmeh.. he's not here15:38
rfolco|ruckcan't we just patch this https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/general_config/featureset037.yml#L1815:39
weshay|ruckrfolco|ruck, I think so yes.. specify the file that exists :)15:39
weshay|ruckwould like to see tripleo-upgrades make a backup of the file before it's updated for update/upgrade15:40
rfolco|ruckcontainer_registry_file is used pre-rocky, after rocky15:40
rfolco|ruck# is uc_containers_prepare_file the one used to identify15:40
rfolco|ruck# the containers file (defaults to: containers-prepare-parameter.yaml)15:40
weshay|ruckrfolco|ruck, or https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/overcloud-deploy/tasks/pre-deploy.yml#L21615:41
*** ykarel is now known as ykarel|away15:42
chandankumarbhagyashri|rover: rfolco|ruck https://review.opendev.org/#/c/751162/ regarding this , please check the job definition or fs related to amphora image15:42
*** udesale_ has quit IRC15:45
*** marios is now known as marios|out15:50
bhagyashri|roverrfolco|ruck, weshay|ruck https://review.opendev.org/#/c/748664/2/config/general_config/featureset037.yml15:53
*** bogdando has quit IRC15:58
*** marios|out has quit IRC16:00
*** ykarel|away has quit IRC16:06
weshay|ruckbhagyashri|rover, ah.. there it is16:09
weshay|ruckbhagyashri|rover, right.. so he fixed it exactly backwards16:10
weshay|ruckand I +2'd it16:10
bhagyashri|roveri see16:11
weshay|ruckbhagyashri|rover, rfolco|ruck to be 100% safe.. I don't see the issue with having two files with the same content... w/ two different names16:16
weshay|ruckjust to see what happens16:16
weshay|ruckand let's take it from there16:16
weshay|ruckbhagyashri|rover, rfolco|ruck revert in validations just merged..  will trigger the validations component now16:23
weshay|ruckthen master16:23
*** derekh has quit IRC16:23
bhagyashri|roverok16:24
weshay|ruckI'm not confident this will fix it16:25
weshay|ruckah maybe it will16:25
weshay|ruckit was caught here https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-validation-master/166ed8d/job-output.txt16:25
weshay|ruckrfolco|ruck, check this out16:27
weshay|ruckhttps://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-validation-master/166ed8d/job-output.txt16:27
weshay|ruck946e3a858686564c9c2c2e5cf5b479655a9adfc416:27
weshay|ruck2020-09-10 22:32:30.299817 | primary | + dlrnapi --url https://trunk.rdoproject.org/api-centos8-master-uc --username review_rdoproject_org repo-promote --commit-hash 946e3a858686564c9c2c2e5cf5b479655a9adfc4 --distro-hash d9a985baccfaa1c0d1112c5b39d0ae8f51ea97c4 --promote-name component-ci-testing16:27
weshay|ruckoh.. promote to component-ci-testing16:28
weshay|ruckk16:28
weshay|ruckhttps://trunk.rdoproject.org/centos8-master/component/validation/promoted-components/commit.yaml16:29
weshay|ruckso that build was promoted..16:29
weshay|ruckhttps://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-8/component/master.yaml#L7816:30
*** saneax has quit IRC16:31
weshay|ruckhttps://review.rdoproject.org/zuul/builds?pipeline=openstack-promote-component&job_name=periodic-tripleo-centos-8-train-component-validation-promote-to-promoted-components16:32
weshay|ruckI need to bring up monitoring on validations16:32
weshay|ruckrfolco|ruck, that looks like it validations was promoted w/ the job failing16:33
weshay|ruckhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-validation-master16:34
*** rfolco|ruck is now known as rfolco|ruck|brb16:39
bhagyashri|roverweshay|ruck, leaving for the day :)16:48
weshay|ruckbhagyashri|rover, have a nice weekend16:48
bhagyashri|roverBye have a nice weekend16:48
bhagyashri|roverrfolco|ruck|brb, ^16:48
*** jpena is now known as jpena|off17:34
*** apetrich has quit IRC17:44
*** rfolco|ruck|brb is now known as rfolco|ruck17:46
rfolco|ruckweshay|ruck, how the component promoted if job failed ? did you find out ?17:51
weshay|ruckrfolco|ruck, no not yet18:08
weshay|ruckrfolco|ruck, I started trolling through logs.. didn't find the job run where it promoted18:08
weshay|ruckbringing up the monitoring for validations now18:08
rfolco|ruckweshay|ruck, ok, I am trying a new fix for stein. Will help with that in a bi18:09
rfolco|ruckbit18:09
weshay|ruckk18:09
rfolco|ruckand did not forget scen12, will do it, be patient18:09
weshay|ruckno review.. just fyi https://review.rdoproject.org/r/2929818:09
weshay|ruckk18:09
rfolco|ruckweshay|ruck, fyi https://review.rdoproject.org/r/2929918:17
weshay|ruckperfect. +218:18
rfolco|ruckweshay|ruck, jira guy replied, I replied back, hopefully he'll add us to next list18:28
weshay|rucksaw18:28
weshay|ruck+118:28
weshay|ruckrfolco|ruck, ok.. so validatoins added18:31
weshay|ruckscroll to the bottom18:32
weshay|ruckhttp://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline?orgId=1&from=now-30d&to=now18:32
weshay|ruckit failed on the same error 2020-09-02 22:58:09masterNoneperiodic-tripleo-ci-centos-8-standalone-validation-master18:32
weshay|ruckand then passed18:32
rfolco|rucklast 2 runs failed18:33
weshay|ruckah wait18:33
weshay|ruckrfolco|ruck, ya.. the last two runs failed on what the integration is seeing18:36
*** chem has quit IRC18:36
weshay|ruckrfolco|ruck, the question is why didn18:37
weshay|ruckdidn't the component line fail first18:38
*** chem has joined #oooq18:38
weshay|ruckthey failed at the same time18:38
rfolco|ruckweshay|ruck, last fail is 09-1018:39
weshay|ruckrfolco|ruck, https://trunk.rdoproject.org/centos8/component/validation/94/6e/946e3a858686564c9c2c2e5cf5b479655a9adfc4_d9a985ba/18:42
weshay|ruckso.. validations-lib fd1bd18:42
rfolco|ruckyes this is the last one promoted ok18:42
weshay|ruckhttps://opendev.org/openstack/validations-libs/commits/branch/master18:42
weshay|ruckand https://opendev.org/openstack/validations-common/commits/branch/master  946e3 latest18:43
weshay|ruckrfolco|ruck, I think it's coming from the tripleo compnent18:44
weshay|ruckhttps://opendev.org/openstack/tripleo-validations/commits/branch/master https://trunk.rdoproject.org/centos8-master/component/tripleo/promoted-components/18:45
weshay|rucktripleo-validatoins18:45
weshay|ruckand tripleo never failed on it18:45
weshay|ruckit's some f.. up w/ how some validations code is in tripleo and some is in validations18:46
weshay|ruckI have a meeting about validations ci on monday18:46
weshay|ruckprobably need to get tripleo-validations removed from master18:47
rfolco|ruckthis is weird, component=validations... not tripleo... if the job reported failure, it should not pass on this task18:48
rfolco|ruckTASK [promote-hash : Check reports from DLRN]18:48
rfolco|ruckthis is reporting false positive18:48
rfolco|ruckhttps://logserver.rdoproject.org/openstack-promote-component/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-train-component-validation-promote-to-promoted-components/7d80dc8/job-output.txt18:48
rfolco|ruckwait weshay|ruck18:49
rfolco|rucktrain???18:49
rfolco|ruckexport RELEASE=train18:49
rfolco|ruckah yeah its train18:50
rfolco|ruckno it should be master, isn't ?18:50
rfolco|ruckit sounds like there is some branch confusion in the config for these component jobs18:52
rfolco|rucknah, just... nevermind. Still looking.18:54
rfolco|ruckclear18:54
rfolco|ruckdelete18:54
rfolco|ruckstart over18:54
rfolco|ruckweshay|ruck, actually it is correct. It promotes with latest good hash, which is 946e3a858686564c9c2c2e5cf5b479655a9adfc4_d9a985ba18:59
rfolco|ruckno matter last 2 failed, it just ignores it18:59
rfolco|ruckweshay|ruck, if you want to change this behavior, make it fail if already promoted that hash, its another story19:00
weshay|ruckrfolco|ruck, back19:01
weshay|ruckrfolco|ruck, aye... so component pipeline is fine19:01
weshay|ruckrfolco|ruck, it's some weird combination of the old validations and new19:02
* weshay|ruck sends an email19:02
rfolco|ruckk19:02
weshay|ruckrfolco|ruck, sent.. see if that email makes sense19:07
rfolco|ruckweshay|ruck, I don't know how these components are combined together, but yeah, it seems to be valid question19:09
weshay|ruckrfolco|ruck, they are defined in rdoinfo19:10
weshay|ruckyou know where that is?19:10
rfolco|ruckweshay|ruck, nope19:11
weshay|ruckrfolco|ruck, https://docs.google.com/presentation/d/1kmUsRwUzs02ZPAFfpq0uqpXCtXFDrhFeFDWmXooMvdY/edit#slide=id.g87db96c7e8_0_4219:12
weshay|ruckoops19:12
weshay|ruckrfolco|ruck, https://docs.google.com/presentation/d/11bmasGIldG6rGswhbZH5iGkpyJuLuF58gKpPUWBQM4w/edit#slide=id.g86140b98d3_0_519:13
rfolco|ruckI see, package-config defines ow they are grouped and built together19:14
weshay|ruckrfolco|ruck, aye19:16
*** TrevorV has quit IRC20:37
*** rlandy is now known as rlandy|afk21:09
*** rfolco|ruck has quit IRC21:20
*** rlandy|afk is now known as rlandy21:53
*** tosky has quit IRC23:20
*** chem has quit IRC23:39
*** chem has joined #oooq23:41

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!