*** dsneddon has joined #oooq | 00:03 | |
*** dsneddon has quit IRC | 00:09 | |
*** rfolco has quit IRC | 00:31 | |
*** dsneddon has joined #oooq | 00:40 | |
*** Goneri has joined #oooq | 00:46 | |
*** dsneddon has quit IRC | 00:47 | |
*** saneax has quit IRC | 00:52 | |
*** dsneddon has joined #oooq | 01:18 | |
*** Goneri has quit IRC | 01:24 | |
*** Goneri has joined #oooq | 01:28 | |
*** dsneddon has quit IRC | 01:31 | |
*** Goneri has quit IRC | 01:34 | |
*** dsneddon has joined #oooq | 02:06 | |
*** dsneddon has quit IRC | 02:11 | |
*** dsneddon has joined #oooq | 02:37 | |
*** dsneddon has quit IRC | 02:42 | |
*** dsneddon has joined #oooq | 02:58 | |
*** dsneddon has quit IRC | 03:02 | |
*** dsneddon has joined #oooq | 03:29 | |
*** dsneddon has quit IRC | 03:34 | |
*** tcw has joined #oooq | 03:36 | |
*** ykarel|away has joined #oooq | 03:53 | |
*** dsneddon has joined #oooq | 04:07 | |
*** rascasoft has quit IRC | 04:14 | |
*** rascasoft has joined #oooq | 04:16 | |
*** dsneddon has quit IRC | 04:20 | |
*** altlogbot_0 has quit IRC | 04:36 | |
*** altlogbot_0 has joined #oooq | 04:36 | |
*** raukadah is now known as chandankumar | 04:41 | |
*** dsneddon has joined #oooq | 04:49 | |
*** dsneddon has quit IRC | 04:53 | |
*** sshnaidm is now known as sshnaidm|off | 05:03 | |
*** quiquell has joined #oooq | 05:22 | |
*** dsneddon has joined #oooq | 05:24 | |
*** udesale has joined #oooq | 05:26 | |
*** dsneddon has quit IRC | 05:29 | |
*** jtomasek has joined #oooq | 05:31 | |
*** saneax has joined #oooq | 05:33 | |
*** sanjayu_ has joined #oooq | 05:34 | |
*** sanjayu_ has joined #oooq | 05:34 | |
*** marios has joined #oooq | 05:37 | |
*** dtantsur|afk is now known as dtantsur | 05:44 | |
*** ykarel|away is now known as ykarel | 05:44 | |
*** dsneddon has joined #oooq | 05:45 | |
*** saneax has quit IRC | 05:46 | |
*** skramaja has joined #oooq | 05:57 | |
*** ksambor has quit IRC | 05:59 | |
*** jbadiapa has joined #oooq | 06:02 | |
arxcruz | e7YPGgQhyY | 06:03 |
---|---|---|
ykarel | what password is this :) | 06:04 |
marios | arxcruz: delete ! | 06:04 |
marios | :D | 06:04 |
arxcruz | nah, not password | 06:05 |
arxcruz | u can try :D | 06:05 |
marios | arxcruz: what's the name of your first teacher (just curious) also any favourite pets? | 06:05 |
arxcruz | marios: lol | 06:05 |
arxcruz | marios: for that i only use one answer for everything | 06:06 |
arxcruz | what's the name of your first teacher: answerxyz | 06:06 |
arxcruz | your favorite pets: answerxyz | 06:06 |
marios | :D | 06:06 |
arxcruz | I was just testing a password generator for gnome, and this was in the clipboard :) | 06:06 |
arxcruz | need a replacement for 1password for linux :( the only thing i miss from mac | 06:07 |
*** ksambor has joined #oooq | 06:08 | |
arxcruz | need to go to brazilian embassy to renew my daughter passport and start my visa process, brb | 06:16 |
*** ykarel is now known as ykarel|afk | 06:20 | |
*** ccamacho has joined #oooq | 06:29 | |
*** jfrancoa has joined #oooq | 06:37 | |
*** holser_ has joined #oooq | 06:43 | |
*** d0ugal has joined #oooq | 06:44 | |
*** dsneddon has quit IRC | 06:49 | |
*** fmount has quit IRC | 06:54 | |
chandankumar | arxcruz: what about using pass and diceware | 06:58 |
*** tosky has joined #oooq | 06:58 | |
*** fmount has joined #oooq | 06:59 | |
*** kopecmartin|off is now known as kopecmartin | 06:59 | |
chandankumar | marios: https://code.engineering.redhat.com/gerrit/#/c/167633/11 | 07:14 |
chandankumar | marios: epel is not needed, we need to check with sf guys regarding rh-python35 | 07:15 |
*** jfrancoa has quit IRC | 07:16 | |
*** amoralej|off is now known as amoralej | 07:18 | |
marios | chandankumar: ack but that is blocked see latest comment from apevec | 07:19 |
*** dsneddon has joined #oooq | 07:24 | |
chandankumar | marios: ack! | 07:28 |
*** jfrancoa has joined #oooq | 07:29 | |
*** dsneddon has quit IRC | 07:32 | |
*** aakarsh has quit IRC | 07:41 | |
*** hamzy has quit IRC | 07:41 | |
*** irclogbot_1 has quit IRC | 07:44 | |
*** irclogbot_1 has joined #oooq | 07:44 | |
*** jaosorior has joined #oooq | 07:44 | |
arxcruz | chandankumar: what is diceware? | 07:46 |
chandankumar | arxcruz: random password generator which you can remember | 07:46 |
arxcruz | chandankumar: Right now I am using lesspass.com so, right now I have no idea what my password is | 07:46 |
arxcruz | for each site, I have a different password that is generated based on a master key in real time | 07:47 |
*** jjoyce has quit IRC | 07:47 | |
*** dtantsur has quit IRC | 07:47 | |
*** jjoyce has joined #oooq | 07:52 | |
*** dtantsur has joined #oooq | 07:52 | |
*** aakarsh has joined #oooq | 07:53 | |
*** hamzy has joined #oooq | 07:53 | |
*** jpena|off is now known as jpena | 07:55 | |
*** ykarel|afk is now known as ykarel | 07:55 | |
*** bogdando has joined #oooq | 08:00 | |
*** dsneddon has joined #oooq | 08:01 | |
zbr | i also use 1Password, quite good. I also have a free for open-source team-account if anyone is interested. I used LastPass in the past, good two. | 08:02 |
*** dsneddon has quit IRC | 08:06 | |
marios | periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/07633b8/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz#_2019-05-30_02_39_52 | 08:08 |
marios | sorry | 08:08 |
*** apetrich has joined #oooq | 08:10 | |
*** yolanda has joined #oooq | 08:11 | |
marios | chandankumar: check comment 4 pls https://bugs.launchpad.net/tripleo/+bug/1831223 | 08:16 |
openstack | Launchpad bug 1831223 in tripleo "[rocky][fs020] test_resize_server_revert_with_volume_attached and test_show_update_rebuild_list_server failing after moving to 2 compute nodes with Virtual Interface creation failed" [Critical,Triaged] | 08:16 |
*** holser_ has quit IRC | 08:16 | |
chandankumar | marios: on it | 08:18 |
chandankumar | marios: commented back, it is coming in 2 comp nodes jobs | 08:20 |
chandankumar | in 1 compute one got skipped and other were passing | 08:21 |
marios | chandankumar: ack /me afk for a bit | 08:21 |
marios | biab | 08:21 |
*** marios has quit IRC | 08:21 | |
*** ykarel is now known as ykarel|lunch | 08:37 | |
*** dsneddon has joined #oooq | 08:41 | |
dtantsur | hi folks! is there a nice way to tear down a quickstart environment (only tear down, not rebuild) | 08:43 |
*** dsneddon has quit IRC | 08:46 | |
*** chem has joined #oooq | 08:54 | |
dtantsur | arxcruz: ^^ | 09:05 |
arxcruz | there's a --teardown option in quickstart.s h | 09:07 |
dtantsur | arxcruz: will it skip creating the environment? | 09:09 |
arxcruz | dtantsur: I believe there's an option for that as well | 09:09 |
dtantsur | doesn't seem to be. maybe use --tags with some non-existing tag.. | 09:11 |
* dtantsur tries | 09:12 | |
*** marios has joined #oooq | 09:17 | |
dtantsur | hmm, seems to have worked | 09:17 |
*** marios_|ruck has joined #oooq | 09:17 | |
*** dsneddon has joined #oooq | 09:18 | |
*** dtantsur has quit IRC | 09:20 | |
*** holser_ has joined #oooq | 09:22 | |
*** dsneddon has quit IRC | 09:22 | |
*** dtantsur has joined #oooq | 09:26 | |
*** ykarel|lunch is now known as ykarel | 09:34 | |
zbr | marios: panda : please check https://review.opendev.org/#/c/661994/ | 09:37 |
zbr | chandankumar: ^ | 09:38 |
chandankumar | zbr: my query about rhel and rhel-8 files? | 09:39 |
zbr | chandankumar: my irc client is so annoying as instead of your nick it gives me "ChanServ" as autocomplete :p | 09:39 |
marios | chandankumar: please check comment #10 https://bugs.launchpad.net/tripleo/+bug/1830351 i see abishop commented there but yesterday has green? | 09:40 |
openstack | Launchpad bug 1830351 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens fails tempest run for tempest.api.volume.admin.test_volumes_backup test group" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 09:40 |
marios | zbr: ack will do | 09:41 |
ykarel | dtantsur, u could try running teardown playbooks like https://github.com/openstack/tripleo-quickstart/blob/master/playbooks/teardown.yml with -p quickstart option | 09:42 |
dtantsur | I guess I ended up doing something like that with "-T all -t none" | 09:43 |
ykarel | dtantsur, and everyting got cleaned up? like vms on virthost | 09:44 |
ykarel | which run under stack user | 09:44 |
dtantsur | yep, seems so | 09:44 |
marios | chandankumar: also that please comment #16 @ https://bugs.launchpad.net/tripleo/+bug/1792862 (sorry they are all promotion blockers and seem to be tempest related ) cc arxcruz and kopecmartin can you check - we merged the skip so removed the promotion blocker. but there is still ongoing work here? the bug is pretty old... | 09:49 |
openstack | Launchpad bug 1792862 in tripleo "[master] Telemetry Tempest integration tests failed giving Unable to complete operation on subnet b13c76ec-c851-48d0-91a9-245a2fdcad9b: One or more ports have an IP allocation from this subnet." [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 09:49 |
ykarel | dtantsur, ack | 09:50 |
ykarel | seems somehow teardown role is included | 09:50 |
ykarel | and tags teardown-all, none making the cleanup | 09:51 |
arxcruz | marios: so it start again? | 09:52 |
marios_|ruck | arxcruz: no it is skipped | 09:53 |
marios_|ruck | arxcruz: i'm standing in for panda today and so am catching up on all the blockers. | 09:53 |
marios_|ruck | arxcruz: i see the skip merged 6 days ago. wondering what else is left there. i removed the promotion-blocker tag, but is there more work | 09:54 |
*** dsneddon has joined #oooq | 09:55 | |
arxcruz | marios_|ruck: ok, so, it's on skip, i'll bring an env and check the root cause, and update the bug ok ? | 09:55 |
marios_|ruck | arxcruz: ack thanks sounds good cc chandankumar ^ | 09:56 |
chandankumar | marios_|ruck: yes, | 09:56 |
*** dsneddon has quit IRC | 10:00 | |
*** ykarel_ has joined #oooq | 10:05 | |
*** ykarel has quit IRC | 10:08 | |
marios_|ruck | chandankumar: arxcruz kopecmartin please see comment #10 when you get a chance did something merge over weekend for green run at https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens green periodic run this morning | 10:19 |
marios_|ruck | https://trello.com/c/dOpKEsnk/985-cixlp1830351tripleociproa-periodic-tripleo-ci-centos-7-ovb-1ctlr1comp-featureset020-queens-fails-tempest-run-for-tempestapivolum | 10:21 |
marios_|ruck | chandankumar: arxcruz its there so will likely come up on call later ^^ | 10:21 |
*** ykarel_ is now known as ykarel | 10:22 | |
chandankumar | marios_|ruck: we need to ingore check jobs, I was testing the queens by updating tempest tag to 19 there | 10:22 |
marios_|ruck | chandankumar: ack but i see green *periodic* this morning | 10:22 |
chandankumar | marios_|ruck: I think we can wait till tomorrow, as it sometimes it passes or fails | 10:22 |
marios_|ruck | chandankumar: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens https://review.rdoproject.org/zuul/build/8471a37d9127445b8eae4c3dc0e59a36 | 10:23 |
marios_|ruck | chandankumar: ah ok it isn't consistent? | 10:23 |
chandankumar | marios_|ruck: nope, not consistent | 10:23 |
marios_|ruck | chandankumar: can you please tell launchpad and/or trello about that | 10:23 |
chandankumar | marios_|ruck: will update the bz soon | 10:23 |
marios_|ruck | chandankumar: hold on i'll copy paste chate | 10:23 |
marios_|ruck | chandankumar: ack :D | 10:23 |
ykarel | marios_|ruck, chandankumar i think i posted reason of inconsistency in fs020 queens, may be u guys missed that | 10:25 |
ykarel | on Friday | 10:25 |
ykarel | or may be i was wrong so been ignored | 10:25 |
marios_|ruck | ykarel: ack yeah i wasn't racking bugs then panda was ruck/rover so ... was it email/irc? | 10:25 |
marios_|ruck | ykarel: s/racking/tracking :D | 10:25 |
ykarel | IRC | 10:25 |
chandankumar | ykarel: on tripleo or here? | 10:26 |
ykarel | :) | 10:27 |
ykarel | ok finding | 10:27 |
marios_|ruck | chandankumar: good question :D ykarel i/we can find but chan would help for logs search | 10:27 |
marios_|ruck | ykarel: i assume you were discussing with panda/ | 10:27 |
ykarel | i think it was chandan, but let me find | 10:27 |
*** dsneddon has joined #oooq | 10:27 | |
chandankumar | ykarel: I was discussing about other issue related to rocky failure | 10:28 |
marios_|ruck | ykarel: chandankumar k thanks gonna go look for some logs on a new f28 issue will check that in bit chandankumar maybe you can update trello i updated comment there i.e. we need to wait for few more runs | 10:28 |
ykarel | ykarel> chandankumar, i checked ur comment in fs020-queens, i think the issue is happening because of duplicated cinder service names(cinderv3) with different service type(volume,volumev3) | 10:28 |
ykarel | <ykarel> and that's why we see random SUCCESS in periodic pipeline | 10:28 |
ykarel | * amoralej is now known as amoralej|off | 10:28 |
ykarel | <ykarel> abishop and u can relate more on it as it's related to tempestconf and cinder | 10:28 |
ykarel | chandankumar, marios_|ruck ^^ | 10:28 |
chandankumar | ykarel: yes, it is updated in bz comments alos | 10:28 |
chandankumar | ykarel: abishop has replied to it | 10:29 |
marios_|ruck | ykarel: ack is that tempest or cinder issue? or tht issue. i know we were creating a 'fake' endpoint for v1 (which is removed/deprecated) but it wpointed to v3 | 10:29 |
marios_|ruck | ykarel: maybe thats why there is duplicate? | 10:29 |
ykarel | chandankumar, okk good, so it's related or not, let me check | 10:29 |
* ykarel reading bz | 10:29 | |
marios_|ruck | ykarel: chandankumar ack thanks | 10:29 |
ykarel | chandankumar, i don't see my message there in bz | 10:31 |
ykarel | i only see urs an alan's | 10:31 |
chandankumar | arxcruz: do we know why we need to check for both v2 and v3 https://opendev.org/openstack/python-tempestconf/src/branch/master/config_tempest/services/volume.py#L45 | 10:31 |
chandankumar | i think v2 is also deprectated | 10:31 |
ykarel | me seeing https://bugs.launchpad.net/tripleo/+bug/1830351, may be it's also tracked in different bz | 10:31 |
openstack | Launchpad bug 1830351 in tripleo "periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens fails tempest run for tempest.api.volume.admin.test_volumes_backup test group" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 10:31 |
arxcruz | chandankumar: nope | 10:32 |
*** dsneddon has quit IRC | 10:32 | |
ykarel | marios_|ruck, so issue from my point of view is:- two services are created for v3 with name: cinderv3 and type: volume and volumev3, and reason of inconsistent failures, when tempestconf detects volumev3 it pass, and when it detects volume it fails | 10:38 |
ykarel | but it's good if tempest guys look at it | 10:39 |
ykarel | as they can more quickly dig the code | 10:39 |
marios_|ruck | ykarel: thank you | 10:40 |
ykarel | let me know if needed, i can dig again to tempetconf, but issue that i looked on Friday was ^^ | 10:40 |
ykarel | marios_|ruck, can u review https://review.rdoproject.org/r/#/c/20989/ | 10:52 |
ykarel | marios_|ruck, it's mainly to get ansible-2.8 in RDO Train, which breaked last time, so we reverted, and now re-revert after handling the last issue | 10:52 |
ykarel | it should be good now, but as u are |ruck currently u should be aware | 10:53 |
ykarel | in case if the bump causes new issue | 10:53 |
marios_|ruck | ykarel: ack are we ready to try again /me looks | 10:59 |
ykarel | marios_|ruck, yes | 11:00 |
marios_|ruck | ykarel: ok lets see | 11:00 |
marios_|ruck | ykarel: thanks | 11:00 |
ykarel | i added link in commit message to the fix we trying | 11:00 |
marios_|ruck | ykarel: yeah i recall that thanks | 11:00 |
ykarel | i also tested container build job with the fix | 11:00 |
ykarel | so we should be good | 11:00 |
marios_|ruck | ykarel: anything in a playbook with package: ansible remove you tend to remember :D | 11:01 |
marios_|ruck | ykarel: ack | 11:01 |
ykarel | marios_|ruck, :) i also tried completely remove ansible usage, but for that we need to wait for upgrade of software factroy zuul | 11:02 |
ykarel | marios_|ruck, https://review.opendev.org/#/c/660247/ fyi | 11:03 |
marios_|ruck | ykarel: thanks | 11:03 |
*** dsneddon has joined #oooq | 11:08 | |
*** tosky has quit IRC | 11:11 | |
*** dsneddon has quit IRC | 11:13 | |
*** jpena is now known as jpena|lunch | 11:31 | |
weshay | 0/ | 11:33 |
weshay | chandankumar hey buddy... rfolco should be settting up a pre-sprint planning mtg | 11:37 |
weshay | ccccccdlvueffgehujglbbkkinneejflflvihiitduuj | 11:38 |
weshay | dang | 11:38 |
marios_|ruck | weshay: you have cats? | 11:38 |
weshay | heh.. just on a make shift desk atm | 11:38 |
weshay | panda how is the ruck | 11:39 |
weshay | marios_|ruck why are you ruck? | 11:39 |
*** udesale has quit IRC | 11:40 | |
*** udesale has joined #oooq | 11:40 | |
marios_|ruck | weshay: public holiday i believe | 11:40 |
weshay | ah k k | 11:41 |
weshay | marios_|ruck need anything? | 11:41 |
marios_|ruck | weshay: so mostly things are OK - few ongoing promotion blockers but should be clearing. i see some new ones that i will file but no big problems afaics @ http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=now-7d&to=now | 11:41 |
weshay | ya.. bank holiday in ireland | 11:41 |
marios_|ruck | weshay: panda isn't tracking in etherpad/etc so going on that ^^ updated a few trello cards in prod chain | 11:42 |
weshay | ya.. I saw we hit 90% pass in upstream.. that is unusually high | 11:42 |
marios_|ruck | weshay: and been harrassing tempest folks chandankumar++ arxcruz++ for ongoing/known issues tempest related | 11:42 |
weshay | marios_|ruck k great.. thanks :) | 11:42 |
weshay | zbr howdy sir | 11:42 |
marios_|ruck | weshay: introspection issue ongoing at https://bugs.launchpad.net/tripleo/+bug/1831053 comment #2 | 11:42 |
openstack | Launchpad bug 1831053 in tripleo "BMC node in periodic OVB jobs fails to retrieve metadata information" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 11:42 |
marios_|ruck | weshay: ack will let you know if things escalate and obv will hand off later before i go | 11:43 |
weshay | k.. marios_|ruck if it's introspection it would be worth a baremetal run or vexhost | 11:43 |
weshay | just to eliminate any questions about rdo-cloud | 11:44 |
*** dsneddon has joined #oooq | 11:45 | |
weshay | marios_|ruck rdo-cloud jobs are running at 86% pass also a little higher pass rate than normal :) | 11:45 |
*** saneax has joined #oooq | 11:46 | |
*** holser_ is now known as holser|lunch | 11:47 | |
chandankumar | marios_|ruck: /me is digging into the code | 11:48 |
marios_|ruck | chandankumar: ack well fyi theres a couple new tempest related ones on master like http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-master/fef4dda/logs/tempest.html.gz | 11:49 |
marios_|ruck | chandankumar: but waiting to see it more times before filing bug | 11:49 |
marios_|ruck | chandankumar: also f28 https://github.com/marios/tripleo_ruck_job_tool.git | 11:49 |
marios_|ruck | chandankumar: oops that tempest http://logs.openstack.org/61/662561/4/gate/tripleo-ci-fedora-28-standalone/b708bd0/logs/tempest.html.gz | 11:50 |
*** dsneddon has quit IRC | 11:50 | |
ykarel | marios_|ruck, re. fs030 and other ovb jobs they are failing/timeout since 31st | 11:51 |
marios_|ruck | ykarel: ack looks like bug was filed on 30th | 11:52 |
chandankumar | zbr: Hello | 11:52 |
chandankumar | zbr: I tested the os_tempest patch here https://code.engineering.redhat.com/gerrit/#/c/171117/ | 11:52 |
ykarel | marios_|ruck, /me not aware of bug, but marios_|ruck, https://review.rdoproject.org/r/#/c/20990/ should fix it and preventive action:- https://review.rdoproject.org/r/#/c/20991/ | 11:52 |
chandankumar | zbr: it is not working https://sf.hosted.upshift.rdu2.redhat.com/logs/17/171117/13/check/tripleo-ci-rhel-8-standalone-rhos-15/12b44bd/job-output.txt.gz#_2019-06-03_07_28_31_098700 | 11:52 |
marios_|ruck | ykarel: this one https://bugs.launchpad.net/tripleo/+bug/1831053 | 11:53 |
openstack | Launchpad bug 1831053 in tripleo "BMC node in periodic OVB jobs fails to retrieve metadata information" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 11:53 |
marios_|ruck | ykarel: looking thanks | 11:53 |
ykarel | marios_|ruck, ^^ is different | 11:53 |
marios_|ruck | ykarel: thanks don't think we even have/had a bug for that one then? | 11:55 |
ykarel | marios_|ruck, /me don't think so bug is there | 11:56 |
*** tesseract has joined #oooq | 11:57 | |
chandankumar | zbr: please have a look on the patch | 11:57 |
*** rfolco has joined #oooq | 11:58 | |
weshay | rfolco morning... bug triage? chandankumar you avail? | 11:59 |
chandankumar | weshay: Yes | 11:59 |
rfolco | weshay, give me 3 min please, switching rooms | 11:59 |
marios_|ruck | weshay: rfolco chandankumar do you need ruck there? | 11:59 |
weshay | marios_|ruck hrm.. if you have things under control otherwise it could help. If you are busy tracing errors no worries | 12:01 |
marios_|ruck | weshay: ac,] | 12:01 |
marios_|ruck | weshay: ack can join | 12:01 |
*** rlandy has joined #oooq | 12:06 | |
*** tosky has joined #oooq | 12:06 | |
marios_|ruck | chandankumar: rfolco weshay this https://bugs.launchpad.net/tripleo/+bugs?orderby=-datecreated&start=0 | 12:08 |
chandankumar | marios_|ruck: f28 bug https://bugs.launchpad.net/tripleo/+bug/1830718 | 12:09 |
openstack | Launchpad bug 1830718 in tripleo "[fedora28][check] tempest boot volume pattern and network basic ops tests are failing " [Critical,Triaged] | 12:09 |
rlandy | weshay; hello | 12:15 |
weshay | https://bugs.launchpad.net/tripleo/+bug/1829736 | 12:15 |
openstack | Launchpad bug 1829736 in tripleo "periodic-tripleo--master-containers-build is failing with "Failed to import docker-py - cannot import name __version__"" [Critical,In progress] | 12:15 |
weshay | https://bugs.launchpad.net/tripleo/+bug/1829468 | 12:16 |
openstack | Launchpad bug 1829468 in tripleo "OVB jobs failing in overcloud-prep-images with timeout on introspection" [Critical,Incomplete] | 12:16 |
marios_|ruck | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build-push | 12:16 |
*** amoralej is now known as amoralej|lunch | 12:16 | |
rlandy | weshay: welcome back ... I need to chat with you re: perf testing some time today | 12:17 |
weshay | https://bugs.launchpad.net/tripleo/+bug/1829382 | 12:17 |
openstack | Launchpad bug 1829382 in tripleo " ovb-3ctlr_1comp_1supp-featureset039 failing overcloud deploy "Failed to connect to NOVA_API database "" [High,Triaged] - Assigned to Martin Schuppert (mschuppert) | 12:17 |
*** dsneddon has joined #oooq | 12:21 | |
*** rfolco has quit IRC | 12:21 | |
weshay | marios_|ruck https://bugs.launchpad.net/tripleo/+bug/1827863 | 12:22 |
openstack | Launchpad bug 1827863 in tripleo "(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups" [High,Triaged] - Assigned to Marios Andreou (marios-b) | 12:22 |
weshay | rlandy hey :) | 12:22 |
weshay | few min | 12:23 |
weshay | chandankumar https://bugs.launchpad.net/tripleo/+bug/1827121 | 12:24 |
openstack | Launchpad bug 1827121 in tripleo "TestServerBasicOps failing on gate: SSHException: Error reading SSH protocol banner" [Critical,Triaged] - Assigned to Arx Cruz (arxcruz) | 12:24 |
rlandy | sshnaidm|off: you around by any chance ( despite the obvious |off)? | 12:24 |
*** rfolco has joined #oooq | 12:24 | |
weshay | marios_|ruck https://bugs.launchpad.net/tripleo/+bug/1826729 | 12:25 |
openstack | Launchpad bug 1826729 in tripleo "Requested MTU is too big, maximum is 1500" [Undecided,New] | 12:25 |
weshay | marios_|ruck can be closed https://bugs.launchpad.net/tripleo/+bug/1826179 ? | 12:25 |
openstack | Launchpad bug 1826179 in tripleo "standalone-upgrade fails during deploy steps (generate config) step1 for "Evaluation Error:... Could not find class ::tripleo::profile::base::placement::api" " [High,Triaged] - Assigned to Marios Andreou (marios-b) | 12:25 |
*** holser|lunch is now known as holser_ | 12:26 | |
*** apetrich has quit IRC | 12:26 | |
*** dsneddon has quit IRC | 12:27 | |
*** apetrich has joined #oooq | 12:27 | |
*** jpena|lunch is now known as jpena | 12:29 | |
*** vinaykns has joined #oooq | 12:29 | |
chandankumar | marios_|ruck: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/requirements.txt#L3 | 12:30 |
chandankumar | marios_|ruck: we need to update the ansible version here also | 12:31 |
rlandy | marios_|ruck: hey - do you know anything about placement? and the recent placement changes? | 12:33 |
zbr | weshay: hi! i am back. | 12:33 |
weshay | zbr hey there :) You are still my point guy for OSP 15, but atm it looks pretty calm.. Any work in progress on osp-15 that I need to be aware of? | 12:35 |
*** saneax has quit IRC | 12:35 | |
zbr | weshay: nope. the meeting were indeed calm, with no direct actions for us. | 12:36 |
weshay | k cool :) | 12:36 |
ykarel | marios_|ruck, re. https://bugs.launchpad.net/tripleo/+bug/1831053 the actual issue is what panda mentioned in description | 12:36 |
openstack | Launchpad bug 1831053 in tripleo "BMC node in periodic OVB jobs fails to retrieve metadata information" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 12:36 |
zbr | i managed to setup the VPN on Friday in order to be able to run RHEL stuff from my homelab. | 12:37 |
ykarel | we have seen this issue from time to time and related to rdo-cloud | 12:37 |
ykarel | mentioning here after i saw ur comment in CIX call | 12:37 |
zbr | weshay: i am trying to help chandan with a change to ansible os-tempest role (related to multi-os), should finish in soon. | 12:38 |
*** rascasoft has quit IRC | 12:39 | |
*** rascasoft has joined #oooq | 12:41 | |
*** jschlueter has quit IRC | 12:43 | |
*** jschlueter has joined #oooq | 12:43 | |
weshay | rlandy hey.. can chat for a few if needed | 12:47 |
rlandy | sure | 12:48 |
rlandy | joining your bj | 12:48 |
*** vinaykns has quit IRC | 12:48 | |
zbr | chandankumar: I do not understand the link you showed me about "not working", is not from my change: https://review.opendev.org/#/c/661994/ | 12:48 |
*** vkapalav has joined #oooq | 12:49 | |
chandankumar | zbr: on this review https://code.engineering.redhat.com/gerrit/#/c/171117/ we are testing the changes downstream | 12:49 |
chandankumar | check depends-on | 12:49 |
chandankumar | it runs RHOS-15 on RHEl-8 | 12:50 |
zbr | chandankumar: aha. ok I will add some debug for that string. somehow ansible output was useless in that case. | 12:51 |
zbr | chandankumar: i was confused because i did not see any reply from zuul on that change. | 12:51 |
chandankumar | zbr: rhel and rhel-8 file, can we merge it in a single one? | 12:51 |
chandankumar | zbr: it is not voting back | 12:51 |
zbr | chandankumar: not really, rhel is more of rhel-7. | 12:51 |
chandankumar | that is in the plan in another sprints | 12:51 |
zbr | we need to keep it compatible with old versions, which means that we need py2 packages for rhel/centos-7 | 12:52 |
zbr | that is why I added the new ones on -8 file. | 12:52 |
zbr | but is overriding those from the unversioned file | 12:53 |
*** vkapalav is now known as vinaykns | 12:57 | |
*** dsneddon has joined #oooq | 12:59 | |
rfolco | scrum time | 13:02 |
rfolco | marios, sshnaidm, weshay, panda, rlandy, arxcruz, rfolco, chkumar, zbr, kopecmartin | 13:02 |
*** dsneddon has quit IRC | 13:04 | |
*** Vorrtex has joined #oooq | 13:09 | |
*** aakarsh has quit IRC | 13:11 | |
*** jaosorior has quit IRC | 13:15 | |
*** amoralej|lunch is now known as amoralej | 13:26 | |
chandankumar | marios_|ruck: rlandy https://tree.taiga.io/project/tripleo-ci-board/issue/1096 gating tempest downstream patches | 13:30 |
*** dsneddon has joined #oooq | 13:31 | |
*** aakarsh has joined #oooq | 13:35 | |
*** dsneddon has quit IRC | 13:36 | |
*** panda has quit IRC | 13:39 | |
*** panda has joined #oooq | 13:42 | |
*** blueDuckRandom has joined #oooq | 13:59 | |
*** ykarel_ has joined #oooq | 14:00 | |
arxcruz | marios_|ruck: so, i do have an env up and I can reproduce the issue with telemetry | 14:00 |
arxcruz | i don't see what changed to make it start to fail | 14:01 |
*** ykarel has quit IRC | 14:03 | |
*** skramaja has quit IRC | 14:07 | |
*** dsneddon has joined #oooq | 14:07 | |
arxcruz | marios_|ruck: we can ask someone from telemetry to check | 14:09 |
marios_|ruck | arxcruz: ack thanks vry much can you hold the env for a while and maybe note on the bug so panda can pickup tomorrow | 14:11 |
*** dsneddon has quit IRC | 14:12 | |
arxcruz | marios_|ruck: sure | 14:16 |
rlandy | marios_|ruck: weshay: these are the two outstanding reviews for rhos-15 ... https://review.opendev.org/#/c/659794/ (marios_|ruck already voted) and https://code.engineering.redhat.com/gerrit/#/c/171119/ (where I have removed the depends-on to merge) | 14:22 |
*** quiquell has quit IRC | 14:22 | |
*** Goneri has joined #oooq | 14:24 | |
arxcruz | zbr: so https://review.opendev.org/#/c/661979/ or https://review.opendev.org/#/c/661994 ? | 14:24 |
marios_|ruck | rlandy: ack will check in bit | 14:26 |
*** ykarel_ is now known as ykarel | 14:26 | |
chandankumar | zbr: please have a look at version compair https://review.opendev.org/#/c/661979/11/vars/redhat.yml@25 | 14:37 |
rfolco | rlandy, https://code.engineering.redhat.com/gerrit/#/c/172246/ would you merge this one please? | 14:37 |
chandankumar | I am certainly doing something wrong there | 14:37 |
rlandy | rfolco: I can - this is a testable repo - is it possible to post test results? | 14:38 |
*** dsneddon has joined #oooq | 14:39 | |
rfolco | rlandy, I can run a dnm with depends on if you prefer... I tested it without chdir, and then realized the modules does not support it | 14:39 |
rfolco | the archive module I mean | 14:39 |
rlandy | rfolco: k - I'll merge this one - in future, where testable results are easier to review | 14:40 |
rfolco | rlandy, ok, thanks | 14:40 |
*** dsneddon has quit IRC | 14:43 | |
zbr | chandankumar: the code from downstream does not seem to respect the depends-on | 14:47 |
zbr | look at https://sf.hosted.upshift.rdu2.redhat.com/logs/17/171117/13/check/tripleo-ci-rhel-8-standalone-rhos-15/0af8ad6/job-output.txt.gz#_2019-06-03_09_36_59_211945 -- there is no debug task before it. | 14:48 |
chandankumar | zbr: did we changed depends-on? | 14:48 |
chandankumar | zbr: I think you updated the https://review.opendev.org/#/c/661979/ this patch, /me has not changed the depends-on | 14:49 |
zbr | chandankumar: no. please look at and tell me https://code.engineering.redhat.com/gerrit/#/c/171117/ | 14:49 |
zbr | after i added the debug statement, I did a recheck on the downstream change. | 14:49 |
chandankumar | zbr: I think you have updated my patch | 14:49 |
zbr | oops... | 14:50 |
zbr | chandankumar: btw, very sad that ansible gives such a cryptic error on "Install distro packages" task. | 14:51 |
chandankumar | zbr: yes, better we need to improve that error msg on ansible side which might come in new release | 14:52 |
zbr | chandankumar: file a bug, i will support it. | 14:56 |
chandankumar | zbr: sure, will do it | 14:56 |
chandankumar | weshay: rlandy https://review.rdoproject.org/r/#/c/20975/ to move 1 comp fs020 jobs | 14:59 |
*** ccamacho has quit IRC | 15:01 | |
*** ykarel_ has joined #oooq | 15:01 | |
rlandy | chandankumar: just checking those jobs are not referenced anywhere | 15:03 |
*** ccamacho has joined #oooq | 15:04 | |
*** ykarel has quit IRC | 15:04 | |
rlandy | chandankumar: k - w+ed | 15:06 |
zbr | marios_|ruck: failure to download packages from rax mirrors, is known? http://logs.openstack.org/65/649965/32/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/6f217e7/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz#_2019-06-03_10_13_31 | 15:07 |
*** ykarel_ is now known as ykarel | 15:07 | |
zbr | i wonder why I do not see any retries around these, afaik yum is supposed to retry 10 times by default. | 15:08 |
zbr | and clearly Error 14 (transfer closed) is a recoverable one. | 15:08 |
*** tosky has quit IRC | 15:08 | |
*** irclogbot_1 has quit IRC | 15:10 | |
*** irclogbot_0 has joined #oooq | 15:10 | |
*** altlogbot_0 has quit IRC | 15:11 | |
*** altlogbot_0 has joined #oooq | 15:11 | |
*** dsneddon has joined #oooq | 15:12 | |
*** ykarel is now known as ykarel|away | 15:12 | |
chandankumar | weshay: rfolco https://sf.hosted.upshift.rdu2.redhat.com/etherpad/p/tripleoci-rhel8 | 15:13 |
marios | rlandy: o/ i have an ai to ping you about that https://bugs.launchpad.net/tripleo/+bug/1831053 introspection issue we're trying to track down - raised in prod chain escalation call today and asked for ironic to check it out, but, do we have logs from a bm job we can check or can you perhaps kick a bm job to see if we can hit it? | 15:15 |
openstack | Launchpad bug 1831053 in tripleo "BMC node in periodic OVB jobs fails to retrieve metadata information" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 15:15 |
* rlandy looks | 15:16 | |
*** dsneddon has quit IRC | 15:17 | |
marios | rlandy: voted at https://code.engineering.redhat.com/gerrit/#/c/171119 already +2 in https://review.opendev.org/#/c/659794 | 15:20 |
rlandy | marios; thanks | 15:23 |
rlandy | sorry - git distracted with downstream - looking at ovb issue now | 15:23 |
rlandy | got | 15:23 |
rlandy | marios: as far as long from baremetal, yes | 15:24 |
rlandy | but baremetal does not use bmc | 15:24 |
rlandy | marios; you can look at bm logs here ... | 15:24 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envB-3ctlr_1comp-featureset001-stein/5e1a98c/ | 15:25 |
marios_|ruck | rlandy: ack thanks | 15:25 |
marios_|ruck | https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/ | 15:26 |
*** dsneddon has joined #oooq | 15:26 | |
rlandy | marios_|ruck: looking at the bug now | 15:26 |
rlandy | marios_|ruck: oh - you want master logs | 15:27 |
rlandy | getting | 15:27 |
marios_|ruck | rlandy: ack /me almost out but am thinking where best to note that link either on the bug or maybe better on the escalation card https://trello.com/c/hQ41vura/989-cixlp1831053tripleociproa-bmc-node-in-periodic-ovb-jobs-fails-to-retrieve-metadata-information | 15:27 |
rlandy | ha ... https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/0066e43/job-output.txt.gz | 15:28 |
rlandy | also timed out | 15:28 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/0066e43/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 15:28 |
rlandy | nothing in the log though | 15:28 |
marios_|ruck | rlandy: ack ok thanks for pointer anyway will check closer tomorrow | 15:29 |
rlandy | marios_|ruck: hmmm ... we also have a timeout in master bm | 15:29 |
rlandy | on introspection | 15:30 |
rlandy | you may have a real issue here | 15:30 |
marios_|ruck | rlandy: i can see all the jobs there https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/ | 15:30 |
marios_|ruck | i think thats the best link to share | 15:30 |
rlandy | I can get you/ironic devs on the hardware | 15:30 |
marios_|ruck | rlandy: ack cool well we escalated today to them dfg:hardprov so lets see i'll add pointer to logs for now | 15:30 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/periodic-hourly/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/periodic-tripleo-ci-centos-7-bm_envA-3ctlr_1comp-featureset001-master/0066e43/logs/undercloud/home/zuul/overcloud_prep_images.log.txt.gz | 15:30 |
rlandy | let me see if I can find an error | 15:31 |
marios_|ruck | rlandy: well usuasll we can find the errors in bmc log but we don't have it there | 15:31 |
rlandy | 2019-06-02 16:07:53.686193 | primary | TASK [overcloud-prep-images : Prepare the overcloud images for deploy] ********* | 15:32 |
rlandy | 2019-06-02 16:07:53.848610 | primary | Sunday 02 June 2019 16:07:53 +0000 (0:00:06.785) 0:00:51.912 *********** | 15:32 |
rlandy | 2019-06-02 18:32:09.629911 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master] | 15:32 |
rlandy | ^^ definitely a hang there | 15:32 |
rlandy | look at the time | 15:32 |
marios_|ruck | rlandy: ack thats the right symptom for the bug anyway | 15:32 |
* marios_|ruck shutdown sequence weshay need something before i go | 15:34 | |
marios_|ruck | ? | 15:34 |
*** tcw has quit IRC | 15:35 | |
*** tcw has joined #oooq | 15:35 | |
*** sanjayu_ has quit IRC | 15:36 | |
*** tcw has quit IRC | 15:36 | |
*** tcw has joined #oooq | 15:37 | |
*** tcw has quit IRC | 15:37 | |
*** tcw has joined #oooq | 15:38 | |
*** tcw has quit IRC | 15:38 | |
*** tcw has joined #oooq | 15:38 | |
*** tcw has quit IRC | 15:39 | |
chandankumar | rfolco: weshay I am logging out see ya tomorrow | 15:43 |
*** chandankumar is now known as raukadah | 15:43 | |
* marios_|ruck too weshay rfolco ttyl going in aminute | 15:43 | |
*** marios_|ruck has quit IRC | 15:44 | |
*** marios has quit IRC | 15:44 | |
*** zbr has quit IRC | 15:47 | |
*** bogdando has quit IRC | 15:49 | |
*** zbr has joined #oooq | 15:55 | |
*** ykarel|away has quit IRC | 15:56 | |
*** dtantsur is now known as dtantsur|afk | 16:15 | |
*** tesseract has quit IRC | 16:27 | |
zbr | @oooq: i think you may want to try ssh plugin on vscode: https://code.visualstudio.com/docs/remote/ssh --- i am surprised how nice it works, mainly only the editor is local. | 16:34 |
*** panda has quit IRC | 16:35 | |
*** panda has joined #oooq | 16:37 | |
sshnaidm|off | rlandy, hi | 16:51 |
rlandy | sshnaidm|off: hey - just carrying on with the ovb stuff | 16:51 |
rlandy | no worries - been in touch with kforde- | 16:51 |
rlandy | we can chat tomorrow | 16:51 |
*** holser_ has quit IRC | 16:52 | |
sshnaidm|off | rlandy, ok, did he find something? | 16:52 |
rlandy | sshnaidm|off: yeah - he said the hotfix for ovb was not implemented completely | 16:53 |
rlandy | I am running again now | 16:53 |
sshnaidm|off | rlandy, heh.. | 16:53 |
rlandy | but still seems very slow | 16:53 |
rlandy | sshnaidm|off: left notes on the card | 16:53 |
sshnaidm|off | rlandy, thanks, will read | 16:54 |
weshay | rlandy sshnaidm|off could it be the branch of ovb we're using? | 16:59 |
weshay | or pin? | 16:59 |
sshnaidm|off | weshay, it's not ovb fault, it's upshift | 16:59 |
rlandy | weshay: I think it's the same one working in rdocloud | 16:59 |
rlandy | the communication is extremely slow | 16:59 |
*** jpena is now known as jpena|off | 17:04 | |
*** jfrancoa has quit IRC | 17:18 | |
*** rlandy has quit IRC | 17:22 | |
*** amoralej is now known as amoralej|off | 17:22 | |
*** rlandy has joined #oooq | 17:22 | |
weshay | rlandy right.. but rdo-cloud I think is patched for ovb | 17:22 |
rlandy | weshay: yep so is upshift | 17:23 |
weshay | oh.. $new info for me | 17:23 |
weshay | k | 17:23 |
weshay | rlandy apparently I think the qe folks are running ovb on it | 17:23 |
weshay | rlandy Alex may have an example job | 17:23 |
rlandy | weshay: ack - we are comparing with the upgrade deployments as well | 17:24 |
weshay | k | 17:24 |
weshay | thanks | 17:24 |
rlandy | my vpn cut out | 17:25 |
rlandy | reconnecting | 17:25 |
rlandy | the nodes just shut down now | 17:25 |
*** Vorrtex has quit IRC | 17:38 | |
*** udesale has quit IRC | 17:41 | |
rfolco | weshay, I'm trying to collect more data to understand the root cause of connect reset by peer issue. Do you have any ideas on how to unblock me ? | 17:58 |
rfolco | https://code.engineering.redhat.com/gerrit/#/c/171832/12/playbooks/tripleo-buildcontainers/Dockerfile.j2 | 17:59 |
raukadah | rfolco: is it possible to build this dockerfile locally? | 18:06 |
raukadah | on the same env | 18:06 |
rfolco | raukadah, I did, cannot reproduce the same issue though | 18:14 |
*** altlogbot_0 has quit IRC | 18:43 | |
*** altlogbot_3 has joined #oooq | 18:43 | |
*** kopecmartin is now known as kopecmartin|off | 18:47 | |
rlandy | sshnaidm|off: this is where it leaves off ... afaict dhcp never completes ... https://sf.hosted.upshift.rdu2.redhat.com/logs/68/171268/16/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-internal/835f89e/logs/undercloud/var/log/containers/ironic-inspector/dnsmasq.log.txt.gz | 18:51 |
rlandy | weshay:is this log intentionally unreadable: overcloud_prep_images_introspection_debug.pcap.log.txt.gz? | 18:51 |
weshay | rlandy it's binary | 18:52 |
weshay | that has to be read w/ wireshark | 18:53 |
weshay | download and import | 18:53 |
*** dsneddon has quit IRC | 19:07 | |
*** altlogbot_3 has quit IRC | 19:21 | |
*** altlogbot_2 has joined #oooq | 19:25 | |
*** dsneddon has joined #oooq | 19:31 | |
*** dsneddon has quit IRC | 19:37 | |
rlandy | weshay: pls vote on https://review.opendev.org/#/c/659794/ to go forward with this change | 19:46 |
*** hamzy_ has joined #oooq | 20:06 | |
*** dsneddon has joined #oooq | 20:12 | |
*** hamzy has quit IRC | 20:15 | |
weshay | rlandy voted and wf | 20:16 |
*** jtomasek has quit IRC | 20:16 | |
rlandy | weshay: thanks | 20:17 |
*** Vorrtex has joined #oooq | 20:31 | |
*** altlogbot_2 has quit IRC | 20:45 | |
*** Goneri has quit IRC | 20:49 | |
*** jbadiapa has quit IRC | 20:52 | |
*** rlandy is now known as rlandy|brb | 21:04 | |
*** rlandy|brb is now known as rlandy | 21:18 | |
*** sshnaidm|off has quit IRC | 21:34 | |
*** sshnaidm has joined #oooq | 21:40 | |
*** sshnaidm is now known as sshnaidm|off | 21:41 | |
*** Goneri has joined #oooq | 21:47 | |
*** aakarsh has quit IRC | 22:24 | |
*** aakarsh has joined #oooq | 22:49 | |
*** zbr has quit IRC | 23:00 | |
*** sanjayu_ has joined #oooq | 23:02 | |
*** Vorrtex has quit IRC | 23:12 | |
*** dsneddon has quit IRC | 23:22 | |
*** dsneddon has joined #oooq | 23:26 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!