*** apetrich has quit IRC | 01:57 | |
*** ykarel|away has joined #oooq | 03:00 | |
*** udesale has joined #oooq | 03:03 | |
*** rlandy has quit IRC | 03:27 | |
*** ykarel|away has quit IRC | 03:40 | |
*** ykarel|away has joined #oooq | 03:57 | |
*** ykarel_ has joined #oooq | 04:00 | |
*** ykarel|away has quit IRC | 04:03 | |
*** ykarel_ is now known as ykarel | 04:14 | |
*** ratailor has joined #oooq | 04:24 | |
*** raukadah is now known as chandankumar | 04:28 | |
*** dsneddon has joined #oooq | 04:43 | |
*** skramaja has joined #oooq | 05:14 | |
*** ksambor has quit IRC | 05:36 | |
*** marios has joined #oooq | 05:42 | |
*** udesale has quit IRC | 05:44 | |
*** udesale has joined #oooq | 05:45 | |
*** quiquell|off is now known as quiquell | 05:49 | |
*** dsneddon has quit IRC | 05:49 | |
*** ksambor has joined #oooq | 05:51 | |
*** dsneddon has joined #oooq | 06:15 | |
*** kopecmartin|off is now known as kopecmartin | 06:26 | |
*** dsneddon has quit IRC | 06:28 | |
*** jfrancoa has joined #oooq | 06:58 | |
*** dsneddon has joined #oooq | 06:58 | |
*** jbadiapa has quit IRC | 06:59 | |
*** dsneddon has quit IRC | 07:12 | |
*** skramaja has quit IRC | 07:13 | |
*** brault has joined #oooq | 07:19 | |
*** tesseract has joined #oooq | 07:19 | |
*** skramaja has joined #oooq | 07:25 | |
*** skramaja_ has joined #oooq | 07:30 | |
*** skramaja has quit IRC | 07:30 | |
*** apetrich has joined #oooq | 07:30 | |
*** amoralej|off is now known as amoralej | 07:33 | |
*** marios has quit IRC | 07:37 | |
*** bogdando has joined #oooq | 07:45 | |
*** dsneddon has joined #oooq | 07:47 | |
*** jaosorior has joined #oooq | 07:49 | |
*** dsneddon has quit IRC | 07:52 | |
*** holser_ has joined #oooq | 07:54 | |
*** udesale has quit IRC | 07:57 | |
*** jpena|off is now known as jpena | 07:57 | |
*** udesale has joined #oooq | 07:58 | |
*** ykarel is now known as ykarel|lunch | 08:00 | |
*** udesale has quit IRC | 08:05 | |
*** udesale has joined #oooq | 08:08 | |
*** marios has joined #oooq | 08:14 | |
*** holser_ has quit IRC | 08:16 | |
*** holser_ has joined #oooq | 08:16 | |
*** udesale has quit IRC | 08:22 | |
*** dsneddon has joined #oooq | 08:25 | |
*** dsneddon has quit IRC | 08:30 | |
*** ykarel_ has joined #oooq | 08:40 | |
*** bogdando has quit IRC | 08:41 | |
*** ykarel|lunch has quit IRC | 08:43 | |
*** ykarel__ has joined #oooq | 08:45 | |
*** ykarel_ has quit IRC | 08:47 | |
*** ykarel__ has quit IRC | 08:49 | |
*** ykarel has joined #oooq | 08:52 | |
*** ykarel_ has joined #oooq | 08:58 | |
*** ykarel has quit IRC | 09:01 | |
*** ykarel_ is now known as ykarel | 09:02 | |
*** dmellado has quit IRC | 09:09 | |
*** dmellado has joined #oooq | 09:11 | |
*** dmellado has quit IRC | 09:11 | |
*** dmellado has joined #oooq | 09:15 | |
*** bogdando has joined #oooq | 09:15 | |
*** dsneddon has joined #oooq | 09:26 | |
*** udesale has joined #oooq | 09:27 | |
*** dsneddon has quit IRC | 09:31 | |
chandankumar | arxcruz: Hello | 09:36 |
---|---|---|
chandankumar | arxcruz: we cannot run os_tempest for rocky standalone as we need to port lots of patches | 09:36 |
chandankumar | like python-tempestconf distro workspace support and a lot more | 09:36 |
chandankumar | in toci if someway we tweak the script to not checkout branches for os_tempest | 09:37 |
chandankumar | then it will perfectly work | 09:37 |
*** dsneddon has joined #oooq | 10:00 | |
*** ykarel has quit IRC | 10:11 | |
*** sanjayu__ has quit IRC | 10:31 | |
chandankumar | marios: Hello | 10:41 |
marios | chandankumar: hi | 10:42 |
chandankumar | marios: Is it possible to restrict some project to clone master only in zuul workspace using toci script irrespective of branch override passed? | 10:42 |
chandankumar | https://review.opendev.org/#/c/658321/ does not works | 10:43 |
marios | chandankumar: hmm not something am aware of but do you mean on the job definition? | 10:45 |
marios | chandankumar: there is something like override-checkout | 10:46 |
marios | chandankumar: https://zuul-ci.org/docs/zuul/user/config.html#attr-job.override-checkout | 10:46 |
chandankumar | marios: yes branch_override -> it overrirde to OVERRIDE_ZUUL_BRANCH | 10:46 |
marios | maybe something panda knows more about though if its zuul related | 10:46 |
* chandankumar looks at panda | 10:46 | |
marios | chandankumar: going afk for a bit in a sec | 10:46 |
marios | chandankumar: you must only address him by "lord" or "sire" and never make eye contact | 10:47 |
chandankumar | aye aye | 10:47 |
panda | I know nothing | 10:48 |
panda | /nick jonsnow | 10:48 |
marios | biab | 10:49 |
*** marios has quit IRC | 10:49 | |
panda | chandankumar: what you mean ? | 10:49 |
panda | chandankumar: what you need ? | 10:49 |
panda | chandankumar: override_checkout forces zuul to get a different branch than the one that triggered the job\ | 10:49 |
chandankumar | panda: in stable rocky jobs, I want to clone os_tempest and dependencies from master | 10:50 |
*** holser_ is now known as holser|lunch | 10:50 | |
panda | chandankumar: is there a stable/rocky on os_tempest ? | 10:50 |
chandankumar | panda: yes | 10:50 |
panda | chandankumar: then why ? | 10:50 |
chandankumar | panda: https://review.opendev.org/#/c/658321/ | 10:50 |
chandankumar | panda: the stuff needed in os_tempest for running tempest in standalone is not there | 10:50 |
chandankumar | it requires a series of backport which is not feasable | 10:51 |
panda | chandankumar: ok, then override_checkout is really what you want to sue, but restricted to that project | 10:51 |
panda | use* | 10:52 |
*** ykarel has joined #oooq | 10:52 | |
chandankumar | basically for these projects https://opendev.org/openstack/tripleo-quickstart/src/branch/master/quickstart-extras-requirements.txt we need stuff from master | 10:53 |
panda | chandankumar: are all the requirements pulled by zuul | 10:54 |
panda | ? | 10:54 |
chandankumar | panda: http://logs.openstack.org/21/658321/1/check/tripleo-ci-centos-7-standalone-upgrade/a6bd768/job-output.txt.gz#_2019-05-10_10_31_25_189906 | 10:55 |
chandankumar | panda: yes | 10:55 |
panda | chandankumar: phew! | 10:56 |
panda | chandankumar: ok, then you might have chance | 10:56 |
chandankumar | panda: https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/run-test/templates/toci_gate_test.sh.j2#L167 | 10:57 |
chandankumar | what about adding a condition here to do git checkout master | 10:57 |
chandankumar | or is it possible to set branch in required projects for a specific project? | 10:58 |
chandankumar | panda: somethign like this https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/base.yaml#L34 | 10:59 |
panda | chandankumar: yes exactly | 11:02 |
panda | chandankumar: was looking exactly for this example | 11:02 |
*** dsneddon has quit IRC | 11:03 | |
*** panda is now known as panda|lunch | 11:07 | |
weshay | rfolco|ruck please fix this as you have time https://bugs.launchpad.net/tripleo/+bug/1828509 | 11:07 |
openstack | Launchpad bug 1828509 in tripleo "More deps needed on Team and repository tags in tripleo-quickstart" [Undecided,New] | 11:07 |
weshay | is marios here today? | 11:08 |
weshay | quiquell ping chat in a few? | 11:09 |
chandankumar | weshay: yes, He will be coming back soon | 11:09 |
weshay | k | 11:09 |
quiquell | weshay: yep, just ping me when you are free | 11:09 |
* chandankumar brb for an hour | 11:12 | |
weshay | quiquell /me joins | 11:12 |
quiquell | weshay: going to your blue | 11:13 |
*** dsneddon has joined #oooq | 11:15 | |
quiquell | | 6a6d23d7-65e7-43ea-9307-71b2e17d2ead | upstream-cloudinit-centos-7 | active | | 11:16 |
quiquell | | 88006cd1-d089-4bd3-b70a-8cbc7eb32f63 | upstream-cloudinit-fedora-28 | active | | 11:16 |
quiquell | | c1f9afa1-2f96-4264-8e06-5c111a9966ba | bmc-template | active | | 11:17 |
*** jpena is now known as jpena|lunch | 11:30 | |
weshay | quiquell check the images for me please | 11:33 |
quiquell | weshay: we need someone else tenant since I already have them | 11:35 |
weshay | matbu meet quiquell | 11:35 |
weshay | quiquell give matbu the commands cli | 11:35 |
quiquell | weshay: btw https://review.rdoproject.org/r/#/q/topic:quiquell-braindump | 11:36 |
quiquell | matbu: what's up | 11:36 |
quiquell | matbu: do a openstack image list and look for the images above | 11:37 |
*** marios has joined #oooq | 11:40 | |
matbu | quiquell: i dont have this one : upstream-cloudinit-fedora-28 | 11:43 |
quiquell | matbu: do you have the others ? | 11:46 |
matbu | quiquell: yep | 11:46 |
quiquell | weshay: ^ looks like fedora-28 is missing | 11:47 |
marios | https://review.opendev.org/#/c/655661/ weshay o/ good morning so lets merge it ? :) | 11:53 |
marios | jfrancoa: ^^ | 11:53 |
weshay | marios ya.. but we need a bug on the tempest tests | 11:53 |
weshay | I skipped | 11:53 |
marios | weshay: oh which ones? | 11:53 |
marios | weshay: damn didn't notice it | 11:53 |
marios | :/ | 11:53 |
marios | so we have a bug is this new /me checks | 11:53 |
weshay | matbu https://rdo-ci-fx2-01-s7-drac.mgmt.rdoci.lab.eng.rdu2.redhat.com/login.html | 11:54 |
weshay | marios look at my change | 11:54 |
marios | weshay: yah only just noticed the skips | 11:54 |
marios | weshay: we already have a bug for that we thought it was fixed cc chandankumar https://bugs.launchpad.net/tripleo/+bug/1827863 | 11:56 |
openstack | Launchpad bug 1827863 in tripleo "(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups" [Undecided,Triaged] | 11:56 |
*** Goneri has joined #oooq | 11:57 | |
marios | weshay: quiquell rfolco|ruck panda|lunch want to merge that? https://review.opendev.org/#/c/657779/ | 11:59 |
quiquell | marios: Can be my last merge :-) | 11:59 |
marios | quiquell: you mean... today ... | 12:00 |
marios | :/ | 12:00 |
marios | quiquell: what | 12:00 |
marios | quiquell: did | 12:00 |
marios | quiquell: we | 12:00 |
marios | quiquell: EEVER | 12:00 |
marios | quiquell: do | 12:00 |
marios | ? | 12:00 |
quiquell | marios: is not you is me | 12:00 |
*** panda|lunch is now known as panda | 12:01 | |
panda | hoe many times did I hear that ? | 12:01 |
* marios hugs panda | 12:01 | |
panda | we are too good for you, right | 12:01 |
marios | we gotta stick together man especially when SOME people are happy to leave us panda | 12:02 |
* marios deathgrip panda | 12:02 | |
weshay | ouch | 12:05 |
quiquell | marios: +w | 12:05 |
marios | thank you lord quiquell enrique jose emanuelle frekeriko parpinotti llorente pastora | 12:07 |
marios | (sorry if i missed any) | 12:07 |
panda | yazevedo | 12:09 |
weshay | marios https://bugs.launchpad.net/tripleo/+bug/1828550 | 12:09 |
openstack | Launchpad bug 1828550 in tripleo "Rocky standalone failing in tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" [High,Triaged] | 12:09 |
marios | 14:57 < marios> weshay: we already have a bug for that we thought it was fixed cc chandankumar https://bugs.launchpad.net/tripleo/+bug/1827863 | 12:11 |
openstack | Launchpad bug 1827863 in tripleo "(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups" [Undecided,Triaged] | 12:11 |
marios | weshay: we already have https://bugs.launchpad.net/tripleo/+bug/1827863 is that one different? | 12:11 |
marios | weshay: checking | 12:11 |
marios | weshay: ah rocky? | 12:11 |
weshay | panda is something up w/ your vi settings? https://code.engineering.redhat.com/gerrit/#/c/169073/4/rhos-release | 12:12 |
panda | weshay: no, there two ways to handle heredocs in bash, you either specify << EOF and then any indentation you provide will be ported to the file, or you can specify <<- EOF (with the minus) and all the tabs until EOF will be squashed. Unfortunately doesn't work with spaces, only with tabs. SO the two alternatives are: break script indentation, or add a bunch of tabs. That's what I'm askin in the comment. | 12:14 |
weshay | quiquell matbu the images should be fixed now | 12:14 |
weshay | matbu please confirm w/ quiquell | 12:14 |
weshay | panda why did you change the way the repos are defined and use EOF? | 12:15 |
weshay | panda why not just define url | 12:15 |
weshay | and continue the pattern as the other repos | 12:16 |
weshay | are we not able to wget those from centos? | 12:16 |
* weshay checks | 12:16 | |
panda | weshay: because the pattern before actually downloads a repo file taht is available | 12:16 |
panda | weshay: I didn't find any ready repo file for the opstools | 12:17 |
panda | I didn't check fo0r ceph | 12:17 |
*** dsneddon has quit IRC | 12:18 | |
weshay | panda ya.. neither is there | 12:18 |
weshay | panda ok.. go see if jshlueter will merge | 12:18 |
weshay | thanks for the explanation | 12:19 |
weshay | anyone interested in building bridges w/ rhos-delivery folks? | 12:20 |
*** rlandy has joined #oooq | 12:23 | |
marios | weshay: is that bug needed? I see you're pointing to stable/stein upgrade job in https://bugs.launchpad.net/tripleo/+bug/1828550 | 12:25 |
openstack | Launchpad bug 1828550 in tripleo "Rocky standalone failing in tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" [High,Triaged] | 12:25 |
marios | weshay: are you saying we have the same bug in stable/rocky? do we have that job in rocky don't think so | 12:25 |
chandankumar | weshay: marios rocky promotion depends on https://review.opendev.org/#/c/657964/3 | 12:26 |
marios | weshay: so should we delete that https://bugs.launchpad.net/tripleo/+bug/1828550 and keep https://bugs.launchpad.net/tripleo/+bug/1827863 which already has some info about the tempest conf info from tempest team | 12:26 |
openstack | Launchpad bug 1827863 in tripleo "(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups" [Undecided,Triaged] | 12:26 |
marios | weshay: ok so you want/need a separate bug against rocky | 12:27 |
marios | weshay: just making sure | 12:27 |
marios | weshay: it would be good to update https://bugs.launchpad.net/tripleo/+bug/1828550 to point to the rocky logs then | 12:27 |
openstack | Launchpad bug 1828550 in tripleo "Rocky standalone failing in tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" [High,Triaged] | 12:27 |
marios | weshay: err but there are no rocky jobs me confused | 12:28 |
marios | weshay: ah standalone vs standalone-upgrade | 12:28 |
marios | weshay: this is the difference between the bugs | 12:28 |
marios | weshay: :/ | 12:28 |
marios | weshay: ??? | 12:29 |
marios | weshay: no but you still have pointer to the standalone-upgrade stein in the description http://logs.openstack.org/61/655661/6/check/tripleo-ci-centos-7-standalone-upgrade-stein/2f69b10/logs/tempest.html.gz | 12:29 |
marios | weshay: want to jump on the phone? | 12:29 |
marios | perhaps chandankumar is around too ? | 12:29 |
* marios brb | 12:29 | |
chandankumar | marios: I am here, | 12:30 |
jfrancoa | marios: standalone-upgrade green! this is a wonderful Friday! :-D | 12:31 |
marios | jfrancoa: yeah but with skips https://review.opendev.org/#/c/655661/7/zuul.d/standalone-jobs.yaml | 12:31 |
marios | jfrancoa: so its a trick | 12:31 |
chandankumar | arxcruz: kopecmartin agenda here https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting lno: 27 | 12:31 |
*** jpena|lunch is now known as jpena | 12:31 | |
marios | chandankumar: any idea why it was still hitting the tempest issue yesterday at http://logs.openstack.org/61/655661/6/check/tripleo-ci-centos-7-standalone-upgrade-stein/2f69b10/logs/tempest.html.gz (the tempest issue being the one we discussed yesterday i.e. https://bugs.launchpad.net/tripleo/+bug/1827863 | 12:32 |
openstack | Launchpad bug 1827863 in tripleo "(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups" [Undecided,Triaged] | 12:32 |
marios | chandankumar: do you think it would be green today ? | 12:32 |
marios | chandankumar: we had promotion yesterday | 12:32 |
chandankumar | marios: for rocky na? | 12:33 |
marios | and today looks like | 12:33 |
chandankumar | for sten also? | 12:33 |
marios | chandankumar: no this is stein | 12:33 |
jfrancoa | marios: ohhh... I see..I didn't realize about that. shall we wait for the tempest package first before merging it? so we can merge it without tricks | 12:33 |
marios | chandankumar: its stable-stein standalone-upgrade. so it fails after upgrade with the stein things | 12:33 |
chandankumar | marios: I see there there new python-tempestconf got used | 12:34 |
weshay | jfrancoa meh.. imho better to get it gating.. I could have put that test in the rocky skip list, but I think it's only happening on upgrades | 12:34 |
marios | jfrancoa: yeah this is what we're currently negotiating :) it was merged actually we were/are waiting for promotion. that log ran yesterday afternoon our time so i'm not sure if it is because the fix didn't fix it, or because it was too early the fix wasn't there yet from promotion | 12:34 |
weshay | chandankumar we don't see that failure on other rocky jobs do we? | 12:35 |
chandankumar | weshay: on rocky periodic it is passing http://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/88d456a/logs/tempest.html.gz | 12:36 |
panda | arxcruz: thanks, I couldn't say it better than you. | 12:37 |
weshay | quiquell arxcruz I need you both on bluejeans for a minute | 12:38 |
arxcruz | panda: :) | 12:38 |
quiquell | weshay: going | 12:38 |
chandankumar | marios: standalone rocky job http://logs.openstack.org/67/657067/16/check/tripleo-ci-centos-7-standalone-rocky/07b10f8/logs/tempest.html.gz is passing in check | 12:39 |
marios | chandankumar: the bug in https://bugs.launchpad.net/tripleo/+bug/1827863 is specific to standalone-upgrade | 12:40 |
openstack | Launchpad bug 1827863 in tripleo "(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups" [Undecided,Triaged] | 12:40 |
marios | chandankumar: not standalone | 12:40 |
weshay | arxcruz you avail? | 12:40 |
marios | chandankumar: and the problem is in fact that before upgrade the flavors volumes created by tempestconf are incompatible with the stein defaults | 12:40 |
marios | chandankumar: as you wrote in comment 2 | 12:41 |
marios | chandankumar: and see kopecmartin comments. | 12:41 |
marios | chandankumar: but looks like it still hits these with https://review.rdoproject.org/r/#/c/20579/ 2.2.0? | 12:41 |
marios | chandankumar: i should check the version in the logs there | 12:41 |
chandankumar | marios: it is using the new version of tempestconf | 12:42 |
marios | chandankumar: but we had promotions for master/stein last 3 days in row afaics | 12:42 |
marios | chandankumar: yeah will check the versions there in a few mins | 12:42 |
marios | chandankumar: we might merge like that in the meantime weshay what do you think? merge gating with skips? | 12:42 |
marios | weshay: before we get another thing to break it | 12:43 |
marios | weshay: its only a matter of hours/days before it happens | 12:43 |
*** ratailor has quit IRC | 12:43 | |
marios | its inevitable at least this way we can have a converation about the breakage | 12:43 |
chandankumar | marios: donpt we think we need to perform cleanup | 12:43 |
*** holser|lunch is now known as holser_ | 12:44 | |
marios | May 09 18:59:56 Installed: python2-tempestconf-2.0.0-0.20180821043805.d7db90e.el7.noarch | 12:49 |
marios | May 09 19:16:28 Updated: python2-tempestconf-2.2.0-0.20190508044059.e0d5dd6.el7.noarch | 12:49 |
marios | chandankumar: ^^ | 12:49 |
marios | chandankumar: http://logs.openstack.org/61/655661/6/check/tripleo-ci-centos-7-standalone-upgrade-stein/2f69b10/logs/undercloud/var/log/yum.log.txt.gz | 12:49 |
chandankumar | marios: yes, seen that part, do we want add a word around to cleanup that flavor with zero disk? | 12:50 |
chandankumar | *work | 12:51 |
marios | chandankumar: so we need to delete the flavors created by tempest? | 12:51 |
marios | chandankumar: cant we get tempest to do that? | 12:51 |
marios | chandankumar: i would rather skip the tests until there is a proper fix in tempestconf rather than carry a hack. where would we put it in the stanndalone-upgrade role? | 12:52 |
chandankumar | marios: once thing we can do it is to run tempest clean up after tempest run finishes before upgrade | 12:52 |
marios | chandankumar: but i think we should go ahead and merge with skips weshay what say you https://review.opendev.org/#/c/655661/7 | 12:52 |
marios | chandankumar: ok that sounds good | 12:52 |
marios | weshay: we can have another patch on monday to unksip | 12:53 |
marios | weshay: monday is a long way away to leave a job like that | 12:53 |
weshay | ya.. removing the skip once it's fixed will be nice | 12:53 |
marios | sure pretty job you got there. shame if something happened to it over the weekend when its all alone. | 12:53 |
weshay | marios easy rather | 12:53 |
marios | weshay: i can even post it | 12:54 |
marios | weshay: unskip and we can merge as soon as it is green | 12:54 |
marios | weshay: well have no choice, it'll be gated | 12:54 |
marios | \o/ | 12:54 |
chandankumar | weshay: marios we can skip the test from fs override itself | 12:54 |
chandankumar | instead of putting in the master skip list | 12:55 |
*** vinaykns has joined #oooq | 12:55 | |
chandankumar | ah it is already there | 12:55 |
weshay | chandankumar ya.. we don't need to skip that for all the rocky jobs afaict | 12:55 |
chandankumar | marios: one question | 12:56 |
chandankumar | this job upgrades from rocky to stein | 12:56 |
chandankumar | in rocky we have updated tempestconf-2.2.0 and same in stein also | 12:57 |
chandankumar | so why lower version is getting installed there? | 12:57 |
marios | chandankumar: it deploys rocky and then updates with stein. maybe the version of rocky it deploys is very old | 12:57 |
*** amoralej is now known as amoralej|lunch | 12:57 | |
chandankumar | in rocky we donot have promotion na? | 12:58 |
chandankumar | rocky is blocked due to this https://review.opendev.org/#/c/657964/ | 12:58 |
chandankumar | once rocky got promoted | 12:58 |
chandankumar | we donot even need to put it in skip list | 12:58 |
marios | chandankumar: ah so the fix needs to go into the deploy tempestconf, not the updated one? | 12:59 |
marios | chandankumar: k good to know then thanks | 12:59 |
marios | chandankumar: maybe also tell launchpad about it if you get a chance | 12:59 |
weshay | chandankumar arxcruz kopecmartin tempest planning in 1min | 12:59 |
marios | weshay: once we're ready we can https://review.opendev.org/658353 | 13:00 |
chandankumar | weshay: https://etherpad.openstack.org/p/tripleo-tempest-squad-meeting | 13:01 |
weshay | arxcruz ok.. time to talk tempest | 13:02 |
chandankumar | kopecmartin: https://code.engineering.redhat.com/gerrit/159204 | 13:04 |
*** skramaja_ has quit IRC | 13:04 | |
weshay | kopecmartin Sai Sindhur Malleni <sai@redhat.com> | 13:06 |
*** quiquell is now known as quiquell|lunch | 13:09 | |
chandankumar | weshay: arxcruz https://review.opendev.org/#/c/658331/ | 13:11 |
marios | weshay: so we can move this done chandankumar rfolco|ruck https://tree.taiga.io/project/tripleo-ci-board/task/989? well it needs a qe | 13:16 |
rfolco|ruck | marios, yep would be great if you find a qe to verify this | 13:18 |
rlandy | marios: weshay: panda: pls review/vote ... https://review.opendev.org/#/c/658358/ - browbeat changes are not being included | 13:19 |
marios | panda: chandankumar rfolco|ruck rlandy weshay arxcruz kopecmartin i need a qe for this please https://tree.taiga.io/project/tripleo-ci-board/task/989 - it is about standalon-upgrade stein job and that it is green. qe should be pretty straightforward | 13:19 |
marios | kindly form orderly line for applications | 13:19 |
marios | rlandy: make me | 13:20 |
rlandy | marios: did you just fill in my application? | 13:20 |
marios | i did not. refreshing taiga | 13:21 |
rlandy | marios: k- you win - I'll qe your work | 13:21 |
rlandy | pls vote on my change | 13:21 |
rlandy | fair enough? | 13:21 |
marios | 16:20 < marios> rlandy: make me | 13:21 |
marios | (i already had) | 13:21 |
rlandy | marios: sorry - did not understand the 'make me' | 13:22 |
rlandy | ok - great | 13:22 |
rlandy | will still do your qe | 13:22 |
marios | rlandy: thank you | 13:22 |
rlandy | marios: can you run me through it quickly to speed things up? | 13:23 |
marios | rlandy: latest status on that is in comment https://bugs.launchpad.net/tripleo/+bug/1827863 comment #9 | 13:23 |
openstack | Launchpad bug 1827863 in tripleo "(stable/stein) standalone-upgrade job failing tempest post upgrade - test_server_security_groups" [Undecided,Triaged] | 13:23 |
rlandy | oh ok - I see | 13:23 |
rlandy | let me just kick off a new browbeat run with above review and I'll look | 13:23 |
marios | rlandy: adding you as qe in https://tree.taiga.io/project/tripleo-ci-board/task/989 cc lord rfolco|ruck | 13:24 |
rlandy | sure | 13:24 |
rfolco|ruck | wrong alias :) | 13:25 |
rlandy | arxcruz: you crack me up (response to quiquell|lunch's email) | 13:27 |
rlandy | marios: k - on it | 13:27 |
arxcruz | :) | 13:27 |
marios | rlandy: thanks i mean the job itself is still just merging now so it is not a firedrill don't change any other priorities you had for today. it can also be next week | 13:28 |
marios | rlandy: the really important thing is making that job vote ASAP | 13:28 |
marios | rlandy: and it is now in the hands of our lord and ruler zuul | 13:29 |
marios | praise be to zuul | 13:29 |
arxcruz | marios: zuul demands trial by combat | 13:29 |
rlandy | marios: priority for today - run browbeat job, wait for next failure (hope it's not something I can't fix)- so I have time in between | 13:29 |
marios | arxcruz: :D | 13:30 |
marios | rlandy: cool thanks | 13:30 |
chandankumar | arxcruz: weshay https://review.opendev.org/#/c/527305/ | 13:32 |
*** quiquell|lunch is now known as quiquell | 13:34 | |
*** aakarsh has quit IRC | 13:40 | |
rlandy | taiga, why do you keep hanging my browser?? | 13:41 |
rfolco|ruck | rlandy, attention deficit | 13:43 |
*** taiga has joined #oooq | 13:43 | |
rfolco|ruck | haha | 13:43 |
taiga | rlandy: because I hate you | 13:43 |
rlandy | rfolco|ruck: taiga: the problem is I am also running short on memory so when you hang my browser, I also have to remember what I was doing to re-create it | 13:44 |
rlandy | this is a problem | 13:44 |
marios | O M G | 13:44 |
marios | is this how skynet starts | 13:44 |
rlandy | because I am a peaces-loving person and I don't want taiga to hate me | 13:44 |
rlandy | peace-loving | 13:44 |
rfolco|ruck | now that email to pay for the service starts to link the points | 13:45 |
* marios notices red dot appeared on wall behind him | 13:45 | |
rfolco|ruck | if taiga.service != 'paid': | 13:46 |
rfolco|ruck | sleep(500) | 13:46 |
taiga | the delay in response on the pages is wanted. It's there to give you time to relax | 13:46 |
rlandy | or to boil my blood | 13:47 |
chandankumar | weshay: arxcruz kopecmartin https://github.com/redhat-openstack/infrared/tree/master/plugins/collect-logs | 13:48 |
rlandy | marios: question on https://review.opendev.org/#/c/655661/7/zuul.d/standalone-jobs.yaml ... line 94 | 13:54 |
marios | rlandy: yup | 13:54 |
rlandy | marios: hmmm ...I see this line used elsewhere ( in standalone scenarios) ... | 13:55 |
rlandy | what is the exact intent? | 13:56 |
rlandy | that this job should run on master plus and branch after rocky? | 13:56 |
rlandy | any | 13:56 |
*** taiga has quit IRC | 13:56 | |
marios | rlandy: we'd have standalone-upgrade-stein running on changes to master yah . but not rocky | 13:56 |
marios | rlandy: its master and stein only | 13:56 |
rlandy | marios: so going forward, this will resolve to master, stein, train ... | 13:57 |
marios | rlandy: ah i guess you meant "any branch after rocky" not and branch | 13:57 |
rlandy | ^^ correct? | 13:57 |
marios | rlandy: yeah i mean we can revisit by train | 13:57 |
rlandy | marios: well, let's get it right now | 13:57 |
rlandy | if we can | 13:57 |
rlandy | if this job is stein and master only, let's be specific | 13:58 |
rlandy | we miss stuff when we mass release change | 13:58 |
marios | rlandy: sure, but would you settle for a follow up patch . i mean there is no such thing as stable/train | 13:58 |
marios | rlandy: cos that is currently +2A and zuul is doing its thing | 13:58 |
rlandy | marios: ack - I'll comment and approve then | 13:58 |
* rlandy does not mess when lord zuul is happy | 13:59 | |
marios | rlandy: thanks, lets leave the task https://tree.taiga.io/project/tripleo-ci-board/task/989?kanban-status=1447276 in ready for review | 13:59 |
marios | rlandy: if you comment there i can followup patch what you need | 13:59 |
marios | rlandy: sound good? | 13:59 |
rlandy | finw | 13:59 |
marios | rlandy: thank you | 13:59 |
rlandy | fine | 13:59 |
*** zbr has joined #oooq | 13:59 | |
rlandy | ugh - bad typing day | 13:59 |
rfolco|ruck | weshay, chandankumar give me one min | 14:00 |
marios | rlandy: making that job vote gate is time sensitive. something will soon break it so we need to make it voting while it is green | 14:00 |
rlandy | ack | 14:00 |
* rlandy just doing QE job and checking | 14:00 | |
marios | weshay: rlandy: if https://review.opendev.org/655661 doesn't merge i'd be grateful for some recheck in US day today pretty please thank you | 14:00 |
rlandy | sure | 14:01 |
rfolco|ruck | weshay, chandankumar pre-planning ? https://bluejeans.com/5878458097 | 14:02 |
chandankumar | panda: ^^ | 14:03 |
rfolco|ruck | panda is with me | 14:03 |
rfolco|ruck | there | 14:03 |
*** aakarsh has joined #oooq | 14:04 | |
rlandy | marios: ok - when https://review.opendev.org/#/c/655661/ merges, will add a few links with passing and voting stein jobs to the card. Anything else required on the QE side? | 14:04 |
marios | rlandy: sounds good to me | 14:04 |
marios | rlandy: i think something like this branches: ^(?stable/stein|master) | 14:07 |
marios | rlandy: is what you mean? ^ | 14:07 |
rlandy | marios: no rush - I think we should discuss the right thing to do for all the jobs defined like this | 14:09 |
rlandy | makes an update headache | 14:09 |
rlandy | but yes, your definition looks correct if we don;t want it to run on train and up | 14:10 |
marios | rlandy: added here https://review.opendev.org/#/c/658353/2/zuul.d/standalone-jobs.yaml | 14:10 |
marios | rlandy: idea is we merge that once those tempest tests are green (waitin on rocky promote) so piggy backed that change into it | 14:11 |
marios | rlandy: oh zuul isn't happy with me | 14:11 |
marios | checking | 14:11 |
rlandy | marios: yep https://review.opendev.org/#/c/658353/ - revert to 1 | 14:11 |
rlandy | let's rethink this | 14:12 |
marios | rlandy: k | 14:12 |
rlandy | and not mess with what needs to get done here | 14:12 |
rlandy | marios: before you checkout, pls leave us a list of patches that need to merge. https://review.opendev.org/#/c/658353/ - https://review.opendev.org/#/c/655661/ - anything else? | 14:13 |
marios | rlandy: no just https://review.opendev.org/#/c/655661/ | 14:13 |
rlandy | fine | 14:13 |
marios | rlandy: the other one https://review.opendev.org/#/c/658353/ comes later or next week. whenever it can merge cos we'll be gating once 661 merges | 14:14 |
*** dsneddon has joined #oooq | 14:15 | |
*** amoralej|lunch is now known as amoralej | 14:17 | |
chandankumar | rfolco|ruck: weshay https://wiki.centos.org/About/Building_8 | 14:18 |
rlandy | panda: hmmm ... I think your suggestion on https://review.opendev.org/#/c/658358/2/roles/run-test/templates/toci_gate_test.sh.j2 is actually needed. $TRIPLEO_ROOT includes openstack apparently | 14:18 |
rlandy | still not picking up browbeat changes | 14:19 |
*** dsneddon has quit IRC | 14:20 | |
panda | rlandy: yes, I imagined there would be more exception to the names | 14:21 |
rlandy | panda: fixing | 14:21 |
rlandy | 2019-05-09 21:46:01.782603 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:152): [[ -d /home/zuul/src/opendev.org/openstack/browbeat ]] | 14:22 |
rlandy | ^^ never gets corrected | 14:22 |
*** quiquell is now known as quiquell|off | 14:37 | |
rlandy | panda: trying https://review.opendev.org/#/c/658358/3/roles/run-test/templates/toci_gate_test.sh.j2 | 14:39 |
panda | rlandy: good. | 14:40 |
rfolco|ruck | rlandy, is there any networking routing config to allow my rdo instance to reach my beaker box ? | 14:41 |
rlandy | rfolco|ruck: hmmm .. the rdo instance is public and your beaker box internal | 14:42 |
rlandy | you can try what we do internally - add a known key and ssh | 14:42 |
rlandy | my guess is - it won't resolve | 14:42 |
rlandy | you would need access to a non-routable 10. network | 14:43 |
rfolco|ruck | rlandy, it does not reach network 10 | 14:43 |
rlandy | non-addressable | 14:43 |
rlandy | correct | 14:43 |
rlandy | or dns - which would have to resolve internally | 14:44 |
rlandy | so unless you can install vpn-type config on that instance | 14:44 |
*** Vorrtex has joined #oooq | 14:44 | |
panda | rfolco|ruck: you can open a revere ssh tunnel with port forwarding. | 14:44 |
*** Goneri has quit IRC | 14:44 | |
rlandy | revere? reverse | 14:45 |
rfolco|ruck | severe | 14:45 |
rfolco|ruck | :) | 14:45 |
rfolco|ruck | short hostname is a host somewhere in akamai | 14:46 |
rfolco|ruck | $ ping rdo-ci-fx2-02-s5 | 14:46 |
rfolco|ruck | PING rdo-ci-fx2-02-s5.rdocloud (23.202.231.167) 56(84) bytes of data. | 14:46 |
rfolco|ruck | 64 bytes from a23-202-231-167.deploy.static.akamaitechnologies.com (23.202.231.167): icmp_seq=1 ttl=48 time=64.9 ms | 14:46 |
rlandy | rfolco|ruck: you'd do better to take an instance from the internal tenant on upshift | 14:47 |
weshay | rfolco|ruck what are you trying to do? | 14:48 |
rfolco|ruck | weshay, just using a rdo instance to reproduce the quickstart install-deps bug, and virthost is a beaker box | 14:48 |
rlandy | why do you need an rdo instance? | 14:49 |
rfolco|ruck | weshay, rdo - beaker --> public 38 - internal 10 nets | 14:49 |
*** jaosorior has quit IRC | 14:49 | |
weshay | rfolco|ruck you are communicating nothing there | 14:49 |
weshay | what are you trying to build, or setup | 14:49 |
weshay | ? | 14:49 |
rlandy | rfolco|ruck: why can't you ssh directly to beaker box from your laptop? | 14:50 |
rfolco|ruck | weshay, nm... will run from my laptop as rlandy says | 14:50 |
weshay | rfolco|ruck jump on my blue | 14:51 |
rlandy | rfolco|ruck: or of you need an instance, I promise internal upshift doesn't bite | 14:51 |
rfolco|ruck | weshay, the intent was to reproduce exactly as the bug reported | 14:51 |
rfolco|ruck | weshay, run deployer (quickstart) from a centos vm | 14:51 |
rfolco|ruck | with clean centos | 14:52 |
marios | rfolco|ruck: consider using the new script | 14:55 |
marios | rfolco|ruck: if you are using a fresh box | 14:55 |
marios | rfolco|ruck: ugh didn't merge yet https://review.opendev.org/#/c/657779/ | 14:56 |
marios | rfolco|ruck: but you can get it curl -o reproducer-fresh-box-setup.sh http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-containers-multinode/67d8eb6/logs/reproducer-quickstart/reproducer-fresh-box-setup.sh | 14:56 |
marios | rfolco|ruck: see readme http://logs.openstack.org/79/657779/3/check/tripleo-ci-centos-7-standalone/b81ab77/logs/README-reproducer.html | 14:56 |
rfolco|ruck | marios, ack | 15:04 |
*** ykarel is now known as ykarel|away | 15:11 | |
*** holser_ has quit IRC | 15:13 | |
*** holser_ has joined #oooq | 15:14 | |
* marios shutdown sequence | 15:15 | |
rlandy | marios: :( gate failure | 15:15 |
rlandy | marios: will recheck | 15:16 |
marios | rlandy: ack :( | 15:16 |
marios | rlandy: thank you | 15:16 |
rlandy | mario: np - have a good weekend | 15:16 |
marios | you too rlandy | 15:16 |
*** ykarel|away has quit IRC | 15:19 | |
rfolco|ruck | weshay, qs requirement is minimal install / basic functionality ? | 15:21 |
rfolco|ruck | should be*? | 15:22 |
rlandy | brb | 15:24 |
*** rlandy is now known as rlandy|brb | 15:24 | |
weshay | rfolco|ruck I think centos/rhel the install options are minimal, then base, then custom, then full | 15:29 |
weshay | it's just the set of rpms | 15:29 |
rfolco|ruck | nope | 15:29 |
weshay | you want something something that does not provide some of the missing rpms | 15:29 |
weshay | nope what? | 15:30 |
rfolco|ruck | default is minimal install | 15:30 |
weshay | rfolco|ruck blue | 15:30 |
rfolco|ruck | you can choose compute node, infra server, dev server, etc | 15:30 |
weshay | got to advanced | 15:30 |
*** dsneddon has joined #oooq | 15:47 | |
*** ykarel|away has joined #oooq | 15:49 | |
*** brault has quit IRC | 15:52 | |
*** rlandy|brb is now known as rlandy | 15:53 | |
*** dsneddon has quit IRC | 16:00 | |
*** dsneddon has joined #oooq | 16:01 | |
chandankumar | arxcruz: weshay rfolco|ruck kopecmartin I will drop an email related tempest issues tomorrow so that tempest team can take a look | 16:01 |
*** marios has quit IRC | 16:02 | |
chandankumar | time to logout, see ya next to next week | 16:02 |
chandankumar | Happy weekend team! | 16:02 |
*** chandankumar is now known as raukadah | 16:02 | |
*** udesale has quit IRC | 16:12 | |
*** udesale has joined #oooq | 16:12 | |
*** brault has joined #oooq | 16:15 | |
*** bogdando has quit IRC | 16:20 | |
*** brault has quit IRC | 16:20 | |
rlandy | lol - I just got an to a fire drill ... I don;t work in an office | 16:39 |
*** jfrancoa has quit IRC | 16:40 | |
weshay | lolz | 16:40 |
*** udesale has quit IRC | 16:49 | |
*** kopecmartin is now known as kopecmartin|off | 17:04 | |
*** jpena is now known as jpena|off | 17:06 | |
*** holser_ has quit IRC | 17:33 | |
*** Goneri has joined #oooq | 17:50 | |
rlandy | weshay: hello there ... what's your take on this ... the baremetal baseos - when the images are fetched ( not built) repo setup fails ( which is why the dir are not on the overcloud) ... | 18:15 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/46/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/0be0dc9/logs/undercloud/home/zuul/repo_setup.sh.1557439168.log.txt.gz#_2019-05-09_21_59_49 | 18:16 |
rlandy | rdo_dlrn= and tripleo_dlrn= | 18:16 |
rlandy | are both empty | 18:16 |
rlandy | but those urls do resolve | 18:16 |
rlandy | curl --silent https://trunk.rdoproject.org/centos7-master/ec/cd/eccdf5f81d211684cc9717c0c7c2bc527127a0ea_c6b6b114/delorean.repo -S 2>>~/dlrn_repo_curl_errors.log | grep baseurl | cut -d= -f2 | 18:16 |
rlandy | https://trunk.rdoproject.org/centos7/ec/cd/eccdf5f81d211684cc9717c0c7c2bc527127a0ea_c6b6b114 | 18:16 |
rlandy | ^^ what am I missing?? | 18:17 |
*** hamzy_ has quit IRC | 18:17 | |
rlandy | and it's here ... https://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/46/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/0be0dc9/logs/undercloud/etc/yum.repos.d/delorean-current.repo.txt.gz | 18:22 |
*** hamzy has joined #oooq | 18:24 | |
rfolco|ruck | weshay, https://review.opendev.org/#/c/658449/1 | 18:27 |
rfolco|ruck | weshay, made a little improvement | 18:27 |
rfolco|ruck | weshay, sudo -v check will warn user about sudoers.d :) | 18:28 |
rlandy | what, there was a puppy booth at RH summit?? weshay, I need to go next year, pls. | 18:40 |
*** amoralej is now known as amoralej|off | 19:00 | |
*** ykarel|away has quit IRC | 19:04 | |
*** dsneddon has quit IRC | 19:05 | |
*** tesseract has quit IRC | 19:12 | |
*** dsneddon has joined #oooq | 19:17 | |
weshay | hey | 19:19 |
rlandy | weshay: hello | 19:21 |
weshay | rfolco|ruck something is amiss http://logs.openstack.org/49/658449/1/check/tripleo-ci-centos-7-undercloud-containers/ed65f92/job-output.txt.gz#_2019-05-10_18_40_08_584962 | 19:22 |
rfolco|ruck | weshay, :( | 19:22 |
weshay | rlandy /me looks | 19:22 |
rfolco|ruck | weshay, will check | 19:22 |
weshay | sorry I was gone, I go out to lunch w/ my dad on fridays | 19:23 |
rlandy | np - dad is important | 19:23 |
rfolco|ruck | weshay, just got successful deploy on zuul-runner \o/ | 19:23 |
rfolco|ruck | arxcruz, fyi ^ | 19:23 |
weshay | nice | 19:23 |
weshay | rlandy I hate that curl | cut command | 19:24 |
rlandy | weshay: it works though | 19:24 |
rlandy | even on the box itself (overcloud) | 19:24 |
rlandy | I tested it | 19:24 |
rlandy | in case dns was not resolving | 19:24 |
weshay | rlandy ya.. I just did as well.. it's just flaky | 19:24 |
weshay | that only run w/ repo-setup role.. I wonder if we could do that w/ ansible to have a retry.. get the hash then pass off the variable | 19:25 |
weshay | because this happens more than we'd like | 19:25 |
weshay | rlandy so has that failed more than once in a row? | 19:25 |
rlandy | weshay: yep - that's a reproducible failure on this job | 19:25 |
rlandy | same error I showed you when we did the review | 19:26 |
rlandy | and you can see the hash resolve fail | 19:26 |
rlandy | yet | 19:26 |
rlandy | yet | 19:26 |
rlandy | it works great manually | 19:26 |
weshay | curl --silent https://trunk.rdoproject.org/centos7-master/58/f8/58f81b1cad691520ce6f1968f156f7720041c280_0129a5fa/delorean.repo -S | grep baseurl | cut -d= -f2 | 19:27 |
weshay | aye | 19:27 |
rlandy | weshay: my only thought is that dns is setup after the repos | 19:27 |
weshay | oh.. hrm | 19:27 |
rlandy | so they don't resolve at first | 19:27 |
weshay | rlandy it's silly not to setup dns first | 19:28 |
weshay | any reason why we can't change that | 19:28 |
rlandy | other than the fact that that setup is done in the overcloud deploy and we inject the repos to the image? | 19:28 |
rlandy | but I may be pulling at straws here | 19:29 |
weshay | I have to reframe back to baseos | 19:29 |
weshay | I wonder if it would help me if I tried a regular qs w/ baseos | 19:30 |
rlandy | and ... on the upside, marios's job is back in the gate | 19:30 |
weshay | ya | 19:30 |
rlandy | weshay: well, let's consider the use case ,... | 19:30 |
rlandy | currently, in theory, bm only runs periodic | 19:30 |
rlandy | back to this ... | 19:32 |
rlandy | so ***if *** baremetal ONLY runs in periodic ... | 19:33 |
rlandy | AND | 19:33 |
rlandy | periodic always build images , we are ok | 19:33 |
rlandy | the problem is only when we GET the baseos images | 19:33 |
rlandy | so .... | 19:33 |
rlandy | if the job is not periodic, then we don't go baseos | 19:34 |
rlandy | weshay: ^^ how about that? | 19:34 |
rlandy | baseos ONLY on periodic | 19:34 |
weshay | so.. blue w/ me for a sec.. it's difficult to get back into that context | 19:35 |
weshay | maybe because it's friday | 19:35 |
weshay | actually let me look at the logs for a bit | 19:35 |
rlandy | okie dokie | 19:36 |
weshay | rlandy https://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/46/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/0be0dc9/job-output.txt.gz#_2019-05-09_22_02_02_005641 | 19:36 |
weshay | so repo setup had to have worked there | 19:37 |
rlandy | on bj | 19:37 |
rlandy | I think it failed | 19:37 |
rlandy | but didn't fail the step | 19:38 |
rlandy | look at the log | 19:38 |
*** ksambor has quit IRC | 19:50 | |
weshay | rlandy https://hairycode.org/tag/ansible/ | 19:56 |
weshay | rsync_opts: -e 'ssh -F "{{ base_dir }}"/khaleesi/ssh.config.ansible' | 19:57 |
*** ccamacho has quit IRC | 20:32 | |
*** Vorrtex has quit IRC | 21:09 | |
*** rfolco|ruck has quit IRC | 21:11 | |
rlandy | finally!!!! | 21:20 |
rlandy | copied browbeat | 21:20 |
rlandy | weshay++ | 21:20 |
rlandy | thank you | 21:20 |
weshay | rlandy nice | 21:20 |
rlandy | weshay: marios patch is in ... https://review.opendev.org/#/c/655661/ | 22:05 |
*** rlandy has quit IRC | 23:06 | |
*** sanjayu__ has joined #oooq | 23:09 | |
*** vinaykns has quit IRC | 23:42 | |
*** rfolco|ruck has joined #oooq | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!