*** hamzy has joined #oooq | 00:10 | |
*** altlogbot_3 has quit IRC | 00:10 | |
*** Goneri has joined #oooq | 00:10 | |
*** altlogbot_0 has joined #oooq | 00:11 | |
*** saneax has joined #oooq | 01:30 | |
*** Goneri has quit IRC | 01:31 | |
*** Goneri has joined #oooq | 01:33 | |
*** apetrich has quit IRC | 01:57 | |
*** rlandy has joined #oooq | 02:06 | |
rlandy | weshay: hi - I'm send marios an email updated before tomorrow ... | 02:07 |
---|---|---|
rlandy | will tell him still work to be done install-deps | 02:07 |
*** rlandy has quit IRC | 02:15 | |
*** ykarel|away has joined #oooq | 02:24 | |
*** Goneri has quit IRC | 03:05 | |
*** saneax has quit IRC | 03:06 | |
*** ykarel|away is now known as ykarel | 03:58 | |
*** udesale has joined #oooq | 04:11 | |
*** ykarel has quit IRC | 04:53 | |
*** ykarel has joined #oooq | 05:11 | |
*** marios_|ruck has joined #oooq | 05:17 | |
*** marios has joined #oooq | 05:17 | |
*** ratailor has joined #oooq | 05:20 | |
*** ajo has quit IRC | 05:27 | |
*** ajo has joined #oooq | 05:33 | |
*** quiquell has joined #oooq | 05:37 | |
*** raukadah is now known as chandankumar | 05:44 | |
*** jfrancoa has joined #oooq | 05:55 | |
*** saneax has joined #oooq | 06:05 | |
*** kopecmartin|off is now known as kopecmartin | 06:06 | |
*** ksambor_ is now known as ksambor | 06:08 | |
marios_|ruck | ykarel: gitweb is back :) | 06:21 |
ykarel | marios_|ruck, yes it was back some days back | 06:22 |
marios_|ruck | ykarel: really? could have sworn it wasn't there yesterday | 06:22 |
ykarel | marios_|ruck, i didn't checked yesterday, but few days back i saw gitweb was working | 06:24 |
marios_|ruck | ykarel: ack. it *was* there on things that merged but not on reviews | 06:25 |
ykarel | i checked on unmerged changes only iirc | 06:26 |
ykarel | but may be u are right as u checked yesterday | 06:26 |
ykarel | anyway good it's working | 06:26 |
marios_|ruck | ykarel: yeah its a bit different but still :D | 06:27 |
marios_|ruck | small joys in life | 06:28 |
* marios_|ruck cries | 06:28 | |
*** dtantsur|afk is now known as dtantsur | 06:28 | |
chandankumar | ykarel: fs030 looks weired | 06:32 |
ykarel | chandankumar, how so, /me didn't checked | 06:33 |
marios_|ruck | ykarel: so seems the ansible revert fixed periodic containers build i see some green runs today | 06:37 |
ykarel | marios_|ruck, yes | 06:37 |
marios_|ruck | ykarel++ amoralej|off++ | 06:37 |
marios_|ruck | thanks | 06:37 |
ykarel | it got green yesteardy run | 06:37 |
marios_|ruck | ykarel: yah was not rucking in the afternoon/evening was on the 15 stuff | 06:37 |
ykarel | marios_|ruck, ack | 06:38 |
marios_|ruck | ykarel: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build-push | 06:38 |
*** skramaja has joined #oooq | 06:38 | |
ykarel | yes | 06:39 |
marios_|ruck | panda: around? do you have 5 mins for sync on https://bugs.launchpad.net/tripleo/+bug/1818060 i think its the main master promotion blocker left right now? | 07:04 |
openstack | Launchpad bug 1818060 in tripleo "Nodes periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master fail to get usable IPs though os-net-config with Error, some other host (BE:E5:4F:B9:21:B0) already uses address" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 07:04 |
marios_|ruck | panda: asking you cos you said you were going to check it and i see comments there from you | 07:05 |
marios_|ruck | chandankumar: looks like you're still working on https://bugs.launchpad.net/tripleo/+bug/1828859 thanks for looking into it afaics the only green runs are from your review but periodic still broken at https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens | 07:06 |
openstack | Launchpad bug 1828859 in tripleo "[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'" [Critical,Fix released] - Assigned to chandan kumar (chkumar246) | 07:06 |
*** amoralej|off is now known as amoralej | 07:15 | |
*** tesseract has joined #oooq | 07:18 | |
ykarel | marios_|ruck, there is new issue in master | 07:19 |
ykarel | both upstream and promotion in case u are not aware | 07:19 |
*** tosky has joined #oooq | 07:21 | |
marios_|ruck | ykarel: looking | 07:21 |
ykarel | more context can be found in #tripleo history of todays and yesterday:- http://eavesdrop.openstack.org/irclogs/%23tripleo/%23tripleo.2019-05-21.log.html#t2019-05-21T18:11:52 | 07:21 |
marios_|ruck | ykarel: oh i thought you meant just happened failing jobs | 07:22 |
ykarel | those jobs are failing since 21st morning | 07:23 |
ykarel | since puppet-nova patch merged, | 07:23 |
ykarel | i pushed revert after finding the issue | 07:23 |
*** holser_ has joined #oooq | 07:27 | |
*** holser_ has quit IRC | 07:27 | |
*** holser_ has joined #oooq | 07:28 | |
chandankumar | marios_|ruck: I am looking at tempest tests of rhos15 | 07:28 |
marios_|ruck | chandankumar: ack k | 07:28 |
chandankumar | verifying which one is fixed or not fixed | 07:28 |
marios_|ruck | chandankumar: sorry, whenever you next have time (for the queens thing) | 07:28 |
ykarel | yes please good to fix queens thing, eggs looking for downstream import this week | 07:29 |
chandankumar | np, I am on it | 07:29 |
marios_|ruck | ykarel: so sounds like instead of revert we will do alex patch(es)? | 07:31 |
ykarel | marios_|ruck, more work is needed, alex patch itself failing | 07:32 |
ykarel | so we disucssed today morning regarding the revert | 07:32 |
ykarel | see my comment in the revert, dhcp_domain is needed by nova metadata | 07:32 |
ykarel | so unless and until someone counters the substitue of dhcp_domain, revert should be good | 07:33 |
marios_|ruck | ykarel: ack. anyway we don't have bug right i think we need one for tracking this discussion probably | 07:34 |
ykarel | marios_|ruck, yes good to have bug | 07:34 |
ykarel | marios_|ruck, in the meanwhile to handle ovn agents mapping i pushed a tht patch to use fqdn_canonical, but there i need feedback from ovn guys if it passes | 07:34 |
marios_|ruck | ykarel: so its tempest fail? | 07:36 |
*** saneax has quit IRC | 07:36 | |
ykarel | marios_|ruck, neutron port binding fails | 07:36 |
ykarel | and hence tempest test fails | 07:36 |
marios_|ruck | test_network_basic_ops[compute,id-f323b3ba-82f8-4db7-8ea6-6a895869ec49,network,smoke] | 07:36 |
marios_|ruck | ykarel: ack yeah i was just expecting deploy fail in logs ack thanks | 07:37 |
*** apetrich has joined #oooq | 07:39 | |
*** jpena|off is now known as jpena | 07:44 | |
*** ykarel is now known as ykarel|lunch | 07:52 | |
chandankumar | marios_|ruck: all the 24 failed tests are known | 08:05 |
chandankumar | marios_|ruck: http://pastebin.test.redhat.com/765254 | 08:06 |
marios_|ruck | chandankumar: which bit? :D | 08:06 |
marios_|ruck | chandankumar: is that re the neutron port binding bug https://bugs.launchpad.net/tripleo/+bug/1829993 | 08:07 |
openstack | Launchpad bug 1829993 in tripleo "tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 & featureset001 fail tempest.scenario.test_network_basic_ops " [Critical,Triaged] | 08:07 |
chandankumar | marios_|ruck: it is related to rhos-15 tempest tests failure | 08:09 |
chandankumar | marios_|ruck: which one I look first, queens or above featureset001 faoilure? | 08:11 |
marios_|ruck | chandankumar: do whatever you're doing | 08:13 |
marios_|ruck | chandankumar: i was just looking at some other tempest issue thats all | 08:13 |
marios_|ruck | chandankumar: which isnt' a tempest issue tempest just catches it but anyway | 08:13 |
marios_|ruck | chandankumar: ok cool good on those being known | 08:13 |
chandankumar | marios_|ruck: does reproducer works for queens job? | 08:15 |
*** bogdando has joined #oooq | 08:15 | |
marios_|ruck | chandankumar: i don't see why not but haven't tried (branches) | 08:16 |
chandankumar | marios_|ruck: I need to reproduce fs030 | 08:16 |
*** ccamacho has joined #oooq | 08:23 | |
panda | the new last released pipeline runs every 12 hours ? | 08:36 |
*** ykarel|lunch is now known as ykarel | 08:38 | |
ykarel | it runs thrice a days afair, but need to check | 08:40 |
ykarel | yes it runs thrice:- '10 9,15,21 * * *' | 08:41 |
panda | ykarel: ok I see | 08:41 |
panda | so it's running, ok, it will probably fail ... | 08:42 |
ykarel | i updated ur testing patch | 08:42 |
ykarel | let me check if it finished | 08:42 |
panda | ykarel: it failed | 08:42 |
ykarel | panda, ok, so same will happen in periodic run | 08:43 |
panda | and also I should test stein | 08:43 |
ykarel | yes | 08:43 |
panda | it failed on neutron db upgrade | 08:45 |
panda | at least it's a new error | 08:45 |
panda | jfrancoa: we can't wait for chem to return, who's the next guy that can help me fix this ? | 08:45 |
jfrancoa | panda: hey, I'd say me...but that didn't work out very well :-D | 08:47 |
marios_|ruck | panda: updated the program call doc see if you want to add something | 08:47 |
marios_|ruck | panda: are you folks talking about the standalone-upgrade job container tag? | 08:47 |
panda | marios_|ruck: can i add "WE'RE ALL GONNA DIE!" | 08:47 |
panda | marios_|ruck: yes | 08:47 |
marios_|ruck | panda: k do you want to jump on a call jfrancoa are you available? | 08:47 |
marios_|ruck | panda: did you se my ping earlier about https://bugs.launchpad.net/tripleo/+bug/1818060 | 08:48 |
openstack | Launchpad bug 1818060 in tripleo "Nodes periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master fail to get usable IPs though os-net-config with Error, some other host (BE:E5:4F:B9:21:B0) already uses address" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 08:48 |
jfrancoa | panda: in the worst case, it's what I said, feel free to modify the playbook or the role, now that we have the stein job voting we'll know if the change breaks it | 08:48 |
panda | if I jump on a call I'll probably hurt it really bad | 08:48 |
marios_|ruck | panda: i wanted to talk to you about that since you were looking at it | 08:48 |
marios_|ruck | panda: ok so lets not blujeans if it isn't necessary | 08:48 |
marios_|ruck | panda: did i miss some discussion already? /me scrolling up | 08:49 |
jfrancoa | marios_|ruck: sure, I'm available | 08:49 |
ykarel | marios_|ruck, so my tht patch passed to clear ovb failures: https://review.opendev.org/#/c/660564/ | 08:49 |
ykarel | would ping ovn guys | 08:49 |
panda | marios_|ruck: let's chat 5 min it's simpler | 08:49 |
marios_|ruck | panda: ok bluejeans who is hosting | 08:49 |
panda | marios_|ruck: jfrancoa usually the ruck hosts | 08:50 |
marios_|ruck | https://redhat.bluejeans.com/7661925373/ | 08:50 |
*** panda is now known as panda|rover | 08:50 | |
marios_|ruck | jfrancoa: ^ | 08:50 |
*** derekh has joined #oooq | 08:51 | |
panda|rover | marios_|ruck: jfrancoa https://review.rdoproject.org/r/20867 | 08:57 |
*** saneax has joined #oooq | 09:03 | |
panda|rover | marios_|ruck: can you paste the docker.io bug here | 09:15 |
marios_|ruck | https://bugs.launchpad.net/tripleo/+bug/1829976 | 09:16 |
openstack | Launchpad bug 1829976 in tripleo " periodic-tripleo-standalone-upgrade-stein fails upgrade fetching new containers "Not found image: ... cinder-api:tripleo-ci-testing" " [Critical,Triaged] | 09:16 |
panda|rover | marios_|ruck: thanks | 09:16 |
marios_|ruck | panda|rover: maybe you want to reword that one or file new one for the tags? | 09:16 |
marios_|ruck | panda|rover: up to you | 09:16 |
marios_|ruck | panda|rover: we can track both docker.io and tags issue there | 09:16 |
marios_|ruck | panda|rover: maybe less confusing | 09:17 |
panda|rover | marios_|ruck: I'll close this, I'll open a new one once we have the latest periodic results in ~1h | 09:19 |
marios_|ruck | panda|rover: ack going afk few mins get some food then will check the retag stuff for a bit | 09:20 |
panda|rover | jfrancoa: container-prep is already run in standalone upgrade | 09:33 |
panda|rover | jfrancoa: I'll try to check what happens if we are using container_build_id | 09:33 |
jfrancoa | panda|rover: right, it's in the dependencies part of the role, isn't it? | 09:33 |
panda|rover | jfrancoa: yes | 09:33 |
panda|rover | jfrancoa: https://review.opendev.org/660615 testing here https://review.rdoproject.org/r/20867 | 09:42 |
jfrancoa | panda|rover: good, let's see if that works better than the dlrn_hash_tag | 09:44 |
jfrancoa | marios_|ruck: hey, is it safe to use the zuul-based reproducer-quickstart? last time I tried to use it wasn't very stable yet.. | 09:52 |
marios_|ruck | jfrancoa: mostly works but havent tried it in few days... hopefully if you hti issues they will be known ones | 09:53 |
jfrancoa | marios_|ruck: ack, I'll give it a try. thanks! | 09:55 |
*** holser_ is now known as holser|lunch | 09:57 | |
chandankumar | why we define names of stuff in caps and small in openstack, it causes too much issues | 10:33 |
*** skramaja has quit IRC | 10:34 | |
panda|rover | chandankumar: ? | 10:34 |
chandankumar | panda|rover: https://review.opendev.org/#/c/660649/ | 10:34 |
chandankumar | panda|rover: I scratched my head come with lots of workaround | 10:34 |
chandankumar | then I found that difference between Member and member | 10:35 |
chandankumar | ykarel: https://review.opendev.org/#/c/660649/ | 10:35 |
marios_|ruck | panda|rover: did you file a new bug for the standalone-upgrade ? | 10:37 |
marios_|ruck | panda|rover: i mean vs https://bugs.launchpad.net/tripleo/+bug/1829976 | 10:37 |
openstack | Launchpad bug 1829976 in tripleo " periodic-tripleo-standalone-upgrade-stein fails upgrade fetching new containers "Not found image: ... cinder-api:tripleo-ci-testing" " [Critical,Fix released] - Assigned to Gabriele Cerami (gcerami) | 10:37 |
panda|rover | marios_|ruck: the periodic job did not run yet | 10:37 |
marios_|ruck | panda|rover: well.. we can be pretty sure the 'tripleo-ci-testing bit will fail | 10:37 |
panda|rover | marios_|ruck: I'm running a test with the containers build id in parallel | 10:37 |
panda|rover | marios_|ruck: I can open a generic bug "standalone ugrade does not use the correct container tag" but it would be not based on a error, just on observation | 10:38 |
marios_|ruck | panda|rover: ack np | 10:40 |
marios_|ruck | panda|rover: i used that one for now can update | 10:40 |
marios_|ruck | panda|rover: posted the retag for now in case we want to explore it more https://review.opendev.org/660665 | 10:40 |
*** blueDuckRandom has joined #oooq | 10:56 | |
panda|rover | marios_|ruck: jfrancoa https://review.rdoproject.org/r/20867 succeeded ... periodic job failed. | 11:01 |
panda|rover | marios_|ruck: jfrancoa I mean the periodi job in the periodic pipeline instead of the one in check | 11:01 |
panda|rover | waiting for the logs before opening bug | 11:08 |
marios_|ruck | panda|rover: ack | 11:09 |
panda|rover | marios_|ruck: https://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-upgrade-stein/6336274/logs/undercloud/home/zuul/standalone_upgrade.log.txt.gz#_2019-05-22_10_55_41 | 11:09 |
*** ykarel is now known as ykarel|afk | 11:12 | |
panda|rover | marios_|ruck: https://bugs.launchpad.net/tripleo/+bug/1830025 | 11:16 |
openstack | Launchpad bug 1830025 in tripleo "periodic standalone upgrade job on stein fails to download images" [Critical,Fix committed] - Assigned to Gabriele Cerami (gcerami) | 11:16 |
*** panda|rover is now known as panda|rover|eat | 11:16 | |
panda|rover|eat | bbl | 11:16 |
arxcruz | when will sorin come back ? | 11:16 |
chandankumar | marios_|ruck: Here is the tempest queens issue https://review.opendev.org/#/c/660649/ | 11:17 |
marios_|ruck | chandankumar: thanks! checking | 11:17 |
chandankumar | marios_|ruck: sorry, https://storyboard.openstack.org/#!/story/2005755 | 11:17 |
marios_|ruck | arxcruz: the prophesies have spoken of his return | 11:17 |
marios_|ruck | prophecies? | 11:17 |
arxcruz | prophecy ? | 11:18 |
chandankumar | I might need to revert previous tqe changes also | 11:18 |
chandankumar | marios_|ruck: I need to spend some time on osa side centos jobs are broken there | 11:28 |
chandankumar | from a week or more | 11:28 |
marios_|ruck | chandankumar: ack thx | 11:28 |
*** udesale has quit IRC | 11:30 | |
*** udesale has joined #oooq | 11:31 | |
*** jpena is now known as jpena|lunch | 11:31 | |
*** sshnaidm|afk is now known as sshnaidm | 11:35 | |
weshay | chandankumar are you sure there patches out for all the tempest failures? | 11:35 |
weshay | or that they are all reported | 11:36 |
weshay | it could be the case, I just want to double check | 11:36 |
chandankumar | weshay: bugs are open for all the bugs, expect one | 11:36 |
weshay | I'll have to deploy a different setup | 11:36 |
chandankumar | weshay: one is fixed by kopecmartin already and merged | 11:36 |
chandankumar | I will cross check the bugs | 11:37 |
*** rfolco is now known as rfolco|dentist | 11:39 | |
rfolco|dentist | brb | 11:39 |
kopecmartin | chandankumar, https://bugs.launchpad.net/tripleo/+bug/1828859/comments/10 | 11:39 |
openstack | Launchpad bug 1828859 in tripleo "[queens] Unhandled error: NoOptionError: No option 'tempest_roles' in section: 'auth'" [Critical,Fix released] - Assigned to chandan kumar (chkumar246) | 11:39 |
*** vkapalav has joined #oooq | 11:40 | |
*** holser|lunch is now known as holser_ | 11:41 | |
weshay | kopecmartin++ | 11:45 |
weshay | marios_|ruck https://review.opendev.org/#/c/658745/ | 11:45 |
weshay | I think that fixes fs01/35 double check for me | 11:45 |
marios_|ruck | weshay: ack they look green ... you mean for https://bugs.launchpad.net/tripleo/+bug/1829993 | 11:47 |
openstack | Launchpad bug 1829993 in tripleo "tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 & featureset001 fail tempest.scenario.test_network_basic_ops " [Critical,Triaged] | 11:47 |
marios_|ruck | weshay: looking at logs | 11:47 |
marios_|ruck | weshay: program call ... updated status check | 11:47 |
weshay | marios_|ruck aye.. thanks... it's just stein .. not master :) | 11:48 |
* weshay updated | 11:48 | |
weshay | marios_|ruck I'll attend | 11:48 |
marios_|ruck | weshay: yeah i was confused by the branch i thought somehow it was a nuanced issue... | 11:48 |
weshay | unless you want to | 11:48 |
marios_|ruck | weshay: well i won't insist :D but i may join to listen in anyway for a bit | 11:49 |
*** skramaja has joined #oooq | 11:49 | |
marios_|ruck | weshay: you sure its just stein? | 11:50 |
marios_|ruck | weshay: i see the issue in http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/e91828a/logs/tempest.html.gz | 11:50 |
marios_|ruck | weshay: see bug links in description | 11:50 |
weshay | marios_|ruck the program call is stein only atm | 11:51 |
weshay | they don't want to hear about train :) | 11:51 |
marios_|ruck | weshay: ah ok | 11:51 |
marios_|ruck | weshay: kk | 11:51 |
marios_|ruck | weshay: i thought you were talking about https://bugs.launchpad.net/tripleo/+bug/1829993 | 11:52 |
openstack | Launchpad bug 1829993 in tripleo "tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035 & featureset001 fail tempest.scenario.test_network_basic_ops " [Critical,Triaged] | 11:52 |
marios_|ruck | weshay: cos the fix you pointing at it stein https://review.opendev.org/#/c/658745/ | 11:52 |
chandankumar | weshay: marios_|ruck more detailed report http://pastebin.test.redhat.com/765344 osp15 | 11:54 |
marios_|ruck | weshay: so it fixes it for stein but not master? | 11:54 |
marios_|ruck | a | 11:54 |
marios_|ruck | chandankumar: thanks | 11:54 |
weshay | chandankumar marios_|ruck if you guys are done looking at that one deployment | 11:54 |
weshay | care to try one that has open issues? | 11:55 |
weshay | or would you like help and for me to show you what I did? | 11:55 |
marios_|ruck | weshay: i wasn't looking there chandankumar was doing that while i was rucking in morning | 11:55 |
marios_|ruck | weshay: do you want me to stop looking at osp15 and switch | 11:55 |
marios_|ruck | weshay: we have to land 2 things there and decide where the job will run so not too much left waiting for rlandy to discuss some more | 11:56 |
weshay | marios_|ruck chandankumar I'll ping you guys in 20 and we'll chat and see what makes sense | 11:56 |
marios_|ruck | weshay: k | 11:56 |
chandankumar | weshay: we need to poke people on this https://trello.com/c/im5WRPw4/926-cixlp1821690tripleo-ci-tempest-testautoallocatenetwork-tempestapicomputeimagestestimages | 11:56 |
chandankumar | it is in rhos-15 and cix also | 11:56 |
*** rlandy has joined #oooq | 11:57 | |
chandankumar | kopecmartin: if we send a review on tempestconf, it is not going to come easily right now | 11:57 |
weshay | chandankumar yes.. that would be a good one | 11:57 |
chandankumar | we need to do a new release | 11:57 |
weshay | chandankumar marios_|ruck if you guys could put up a patch to make fs020 two compute | 11:58 |
weshay | that would also be excellent | 11:58 |
chandankumar | weshay: /me takes a look | 11:58 |
chandankumar | marios_|ruck: we need to recheck the https://review.opendev.org/#/c/643731/ rhel-8 dib changes | 11:59 |
rlandy | arxcruz: hi ... thanks for your email on tempest failure on bm | 11:59 |
chandankumar | nmargi updated hte parch | 11:59 |
rlandy | arxcruz: I only ran that to try simulate the fs001 failure on ovb | 11:59 |
chandankumar | I mean the downstream change | 11:59 |
arxcruz | rlandy: np | 11:59 |
* chandankumar does | 11:59 | |
rlandy | which think got fixed? | 12:00 |
arxcruz | i though it was some failure you needed my help | 12:00 |
chandankumar | sorry | 12:00 |
chandankumar | rlandy: marios_|ruck where we can test this change https://review.opendev.org/#/c/643731/ downstream on RHEL-8? | 12:00 |
*** apetrich has quit IRC | 12:01 | |
marios_|ruck | chandankumar: here https://code.engineering.redhat.com/gerrit/167633 | 12:01 |
marios_|ruck | chandankumar: but we're currently not using it in our job | 12:01 |
marios_|ruck | chandankumar: we are using the fake one | 12:01 |
rlandy | marios_|ruck: chandankumar: what would you need as the base to run on to the test? | 12:02 |
marios_|ruck | rlandy: on the job definition we'd have to switch the nodeset to use that | 12:02 |
marios_|ruck | rlandy: i.e. https://code.engineering.redhat.com/gerrit/#/c/167540/38/zuul.d/nodesets.yaml | 12:02 |
marios_|ruck | rlandy: that what you mean? | 12:03 |
rlandy | marios_|ruck: sshnaidm merged a nodeset for ovb | 12:04 |
marios_|ruck | rlandy: do you want to sync re the 15 job | 12:04 |
rlandy | marios_|ruck: sure | 12:04 |
marios_|ruck | rlandy: k sec, dropping from program call weshay is there | 12:04 |
rlandy | marios_|ruck: and if we added a build-images job to our job, it could run on the same node | 12:04 |
marios_|ruck | rlandy: https://redhat.bluejeans.com/7661925373/ | 12:04 |
rlandy | ovb build images | 12:04 |
rlandy | chandankumar: ^^ | 12:05 |
marios_|ruck | rlandy: ok | 12:05 |
marios_|ruck | rlandy: lets wait | 12:05 |
marios_|ruck | rlandy: till after program call | 12:05 |
rlandy | sshnaidm: are your fs001 jobs building images? | 12:05 |
marios_|ruck | rlandy: weshay chandankumar we all tal kthen | 12:05 |
sshnaidm | rlandy, yes | 12:05 |
rlandy | marios_|ruck: k - ping me when ready | 12:05 |
marios_|ruck | rlandy: me and weshay and chandankumar said we'd sync after the program call | 12:05 |
marios_|ruck | rlandy: k | 12:05 |
*** panda|rover|eat is now known as panda|rover | 12:06 | |
weshay | rlandy you were right about https://review.opendev.org/#/c/660508/ and the -y | 12:06 |
weshay | :) | 12:06 |
marios_|ruck | rlandy: abandon https://review.opendev.org/#/c/659794 ? | 12:06 |
marios_|ruck | weshay: abandon https://review.opendev.org/#/c/659879 ? | 12:07 |
weshay | marios_|ruck rlandy easy review hopefully.. but we need another job https://code.engineering.redhat.com/gerrit/#/c/170965/ | 12:07 |
weshay | marios_|ruck we need https://review.opendev.org/#/c/659794 for osp-14 | 12:08 |
marios_|ruck | sd] | 12:08 |
marios_|ruck | weshay: ah it wasn't in depends-on and i thought we didn't need it | 12:08 |
weshay | https://review.opendev.org/#/c/659879 abandoned | 12:09 |
lmiccini | o/ rlandy weshay good morning! can anybody give https://code.engineering.redhat.com/gerrit/#/c/171040/ the final kick please? | 12:11 |
rlandy | weshay: something is needed for rhos-14 - so keeping https://review.opendev.org/#/c/659794 for now | 12:11 |
rlandy | but don't merge | 12:11 |
rlandy | lmiccini: hi - I merged https://code.engineering.redhat.com/gerrit/#/c/171040/ | 12:12 |
rlandy | lmiccini: also - just fyi, we moved our periodic baremetal jobs to install on a base os and build images | 12:12 |
rlandy | lmiccini: iirc, you keep your own role there | 12:13 |
lmiccini | rlandy: thank you. guess I'll need to dig into that | 12:13 |
rlandy | lmiccini: if you'd like to consider ... https://review.opendev.org/#/c/648519/ | 12:14 |
rlandy | lmiccini: it also required some settings changes | 12:14 |
rlandy | which I can walk you through | 12:14 |
*** apetrich has joined #oooq | 12:14 | |
rlandy | but we have passing jobs for the first time in a long time now | 12:14 |
lmiccini | rlandy: I think I have that 'to-build-or-not-to-build' but I am bypassing it for now, I am building rpms/containers and injecting that into the overcloud image | 12:16 |
lmiccini | rlandy: as soon as I am done fixing the jenkins downstream pipelines let's have a chat if you have time :) (probably next week) | 12:16 |
rlandy | lmiccini: the difference is that the review above switches that from a playbook to a role so the decision can be made on the virthost | 12:16 |
rlandy | lmiccini: sure - as you have time | 12:16 |
*** ykarel|afk is now known as ykarel | 12:18 | |
*** apetrich has quit IRC | 12:19 | |
weshay | chandankumar marios_|ruck k.. join my blue please | 12:23 |
marios_|ruck | weshay:joining | 12:24 |
chandankumar | marios_|ruck: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/vars/tempest_skip_stein.yml | 12:31 |
*** jpena|lunch is now known as jpena | 12:33 | |
*** apetrich has joined #oooq | 12:35 | |
rlandy | woohoo - finally - passing periodics ... periodic-tripleo-ci-centos-7-bm_envE-3ctlr_1comp-featureset001-rockyopenstack/tripleo-ci-internal-jobsmasterperiodic-hourlymaster113632019-05-22T07:03:19SUCCESS | 12:43 |
rlandy | sshnaidm: what happened with ovb introspection? still stuck? | 12:44 |
sshnaidm | rlandy, yeah, need to look today | 12:45 |
rlandy | sshnaidm: can you hold the nodes? | 12:45 |
rlandy | sshnaidm: if yes, hold them and we can check IPMI | 12:46 |
sshnaidm | rlandy, yeah, checking.. | 12:46 |
jfrancoa | panda|rover: so the fix seems to work (https://review.opendev.org/#/c/660615/) , right? the standalone-upgrade-stein job passes and the test patch had both jobs passing | 12:47 |
*** amoralej is now known as amoralej|lunch | 12:51 | |
panda|rover | jfrancoa: that's when you come in and say: "this is passing, but it's not upgrading" or "it passes for real" | 12:52 |
panda|rover | jfrancoa: also stein passes, everywhere, master fails in upstream | 12:52 |
*** tcw has joined #oooq | 12:52 | |
panda|rover | jfrancoa: "Command '['systemctl', 'stop', u'tripleo_nova_compute.service']' returned non-zero exit status 1" | 12:53 |
jfrancoa | panda|rover: no, no...no negative comments...for what I checked it's configuring the right repos in deployment and upgrade, passing the right namespace (from tripleorocky to tripleostein), looks good man! kudos | 12:56 |
jfrancoa | panda|rover++ | 12:56 |
panda|rover | jfrancoa: mind voting there ? so the seal of approval from upgrades is official ? | 13:00 |
jfrancoa | panda|rover: sure, I was about to | 13:00 |
ykarel | panda|rover, so not fixing delorean_current part in standalone upgrade promotion jobs? | 13:05 |
ykarel | me disconnected about the plan, may be u posted somewhere | 13:05 |
lmiccini | rlandy: I have another one for you :( https://review.opendev.org/#/c/660555/ | 13:05 |
weshay | panda|rover let's chat | 13:06 |
panda|rover | ykarel: party pooper :) I knew I missed something ... | 13:07 |
ykarel | panda|rover, okk seems /me looking wrong or u missed something | 13:07 |
ykarel | looks like u ran without promotion release files | 13:08 |
panda|rover | ykarel: again ? | 13:08 |
ykarel | i diddn't see in Depends-on | 13:08 |
ykarel | atleast in last PS | 13:08 |
ykarel | panda|rover, /me seeing https://review.rdoproject.org/r/#/c/20867/2..4//COMMIT_MSG | 13:09 |
rlandy | lmiccini: https://review.opendev.org/#/c/660555/ looks ok to me ... marios_|ruck, panda|rover ... pls review | 13:10 |
ykarel | so ^^ reminds me of ocata breakages https://review.opendev.org/#/c/602171/ | 13:13 |
panda|rover | ykarel: what's missing ? | 13:14 |
ykarel | panda|rover, depends-on on https://review.opendev.org/#/c/638756/ | 13:14 |
ykarel | to test periodic jobs by promotion release files | 13:15 |
panda|rover | ykarel: dammit I thought it was there | 13:16 |
panda|rover | I was sure ... :) | 13:16 |
panda|rover | :( | 13:16 |
ykarel | panda|rover, it was there u removed it :) | 13:16 |
ykarel | may be while pushing update from local | 13:16 |
ykarel | panda|rover, people generally miss adding that depends-on and this results in merging wrong or incomplete patches | 13:18 |
ykarel | iirc sshnaidm pushed to make that change permanent, not sure where it was stuck | 13:18 |
ykarel | a recent example is queens fix | 13:19 |
weshay | marios_|ruck chandankumar ugh.. so results are back phase2-15_director-rhel-8.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph3.8 hr > 1.2 days3 hr 38 min9 of 1,323 failed (-132) | 13:21 |
weshay | panda|rover https://code.engineering.redhat.com/gerrit/#/c/167540/ | 13:29 |
*** rfolco|dentist is now known as rfolco | 13:29 | |
weshay | panda|rover https://review.opendev.org/#/c/660508/ | 13:29 |
*** Goneri has joined #oooq | 13:29 | |
weshay | panda|rover https://code.engineering.redhat.com/gerrit/#/c/167540/ | 13:29 |
*** dtantsur is now known as dtantsur|brb | 13:29 | |
weshay | panda|rover https://code.engineering.redhat.com/gerrit/#/c/170965/1 | 13:30 |
marios_|ruck | weshay: looking | 13:30 |
marios_|ruck | weshay: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP15/job/phase2-15_director-rhel-8.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/ that one? last failed is still 1 day ago? | 13:31 |
marios_|ruck | weshay: ah latest test results... so from one hundred something down to 9 fails now? | 13:32 |
panda|rover | ykarel: now I see what happened, I ran "git review -D 20867" to get your updated review | 13:32 |
panda|rover | ykarel: and didn't check if it succeded | 13:32 |
weshay | marios_|ruck ya.. we probably should switch targets, but we're only 30min away from the mtg, and we can ask there | 13:32 |
ykarel | panda|rover, ack, isn't it -d | 13:32 |
panda|rover | ykarel: yep, that's why it failed | 13:33 |
ykarel | panda|rover, ohkk | 13:33 |
*** ratailor has quit IRC | 13:41 | |
*** aakarsh has quit IRC | 13:42 | |
weshay | rlandy can you rereview https://review.opendev.org/#/c/660508/ | 13:44 |
weshay | last patch to merge | 13:44 |
weshay | rlandy arxcruz btw.. I think the ovb issues on fs01/35 were resolved by kamil yesterday | 13:46 |
arxcruz | weshay: what was the problem? is it related to the email i sent ? | 13:46 |
rlandy | weshay: was the bm issue the same as ovb? | 13:46 |
rlandy | ie: did the bm job reproduce the ovb error? | 13:46 |
marios_|ruck | weshay: chandankumar is there no 'full link to logs' i mean is that it https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP15/job/phase2-15_director-rhel-8.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/lastCompletedBuild/console | 13:47 |
marios_|ruck | weshay: chandankumar i.e. console output? we cant check logs of nodes from there? | 13:47 |
weshay | rlandy arxcruz https://review.opendev.org/#/c/658745/ | 13:48 |
chandankumar | marios_|ruck: yes | 13:48 |
chandankumar | marios_|ruck: go down | 13:48 |
chandankumar | marios_|ruck: you will get the log link http://cougar11.scl.lab.tlv.redhat.com/phase2-15_director-rhel-8.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/17/ | 13:48 |
rlandy | weshay: marios_|ruck: https://review.opendev.org/#/c/660508/ - now the -y is gone from everywhere? | 13:48 |
marios_|ruck | chandankumar: ack see it thanks | 13:48 |
marios_|ruck | rlandy: yeah i commented there i wasn't sure why that was necessary? | 13:48 |
marios_|ruck | rlandy: weshay did the -y cause issues ? | 13:49 |
rlandy | well initially it was in two places | 13:49 |
weshay | marios_|ruck no.. but it was dupe -y | 13:49 |
weshay | so we were passing yum -y -y foo | 13:49 |
rlandy | I thought we would keep it in one | 13:49 |
weshay | which works fine | 13:49 |
marios_|ruck | weshay: i see | 13:49 |
rlandy | oh it remained in package_manager | 13:49 |
rlandy | nvm | 13:50 |
rlandy | got it | 13:50 |
rlandy | weshay: k ... +2'ed https://review.opendev.org/#/c/660508 | 13:50 |
rlandy | let's try it when marios_|ruck is ready for the fallout | 13:50 |
* marios_|ruck hair and eyebrows on fire | 13:50 | |
marios_|ruck | rlandy: lets go for it! | 13:51 |
rlandy | marios_|ruck: k - here we go w+ | 13:51 |
weshay | rlandy marios_|ruck while ur in context https://code.engineering.redhat.com/gerrit/#/c/170965/1 | 13:51 |
*** amoralej|lunch is now known as amoralej | 13:52 | |
marios_|ruck | rlandy: no wait | 13:52 |
marios_|ruck | rlandy: is that centos vs "CentOS" OK | 13:52 |
marios_|ruck | rlandy: let me comment on the review | 13:52 |
weshay | yes | 13:52 |
weshay | marios_|ruck cat /etc/os-release | 13:52 |
* rlandy breathes again | 13:53 | |
marios_|ruck | weshay: ok sorry rlandy ... me reads more carefully (where were they getting NAME from ) | 13:53 |
weshay | [root@localhost ~]# cat /etc/os-release | 13:53 |
weshay | NAME="CentOS Linux" | 13:53 |
weshay | VERSION="7 (Core)" | 13:53 |
weshay | ID="centos" | 13:53 |
marios_|ruck | ah they were using Name you are using ID | 13:53 |
marios_|ruck | weshay: thanks | 13:53 |
weshay | ya.. ID seems to be there for scripting | 13:53 |
rlandy | weshay: so ... https://code.engineering.redhat.com/gerrit/#/c/170965/1 | 13:54 |
weshay | aye | 13:54 |
rlandy | firstly - you have extra space | 13:54 |
weshay | ah ya.. saw that /me fixes | 13:54 |
rlandy | secondly, remind me pls | 13:54 |
rlandy | what are we doing here | 13:54 |
rlandy | the needs_images throws me off | 13:54 |
weshay | rlandy need_images is the downstream equiv to tripleo-ci-testing | 13:54 |
rlandy | weshay: ok - so then we need to do some work on the job playbooks to get the right release file ... | 13:55 |
weshay | rlandy so we'll have two jobs... for now.. 1 that uses passes_phase_1 and one that is basically testing untested builds | 13:55 |
* rlandy gets | 13:55 | |
weshay | yes | 13:55 |
weshay | and zuul | 13:55 |
rlandy | http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/playbooks/configure-rhel.yaml#n20 | 13:56 |
rlandy | weshay: ^^ | 13:57 |
rlandy | so we'll need another case there | 13:57 |
rlandy | or we can rethink the naming | 13:57 |
panda|rover | jfrancoa: dang. we need to talk again. | 13:58 |
weshay | rlandy release file would be needs_images/rhos-15.yml | 13:58 |
weshay | rlandy that should work the same way as upstream | 13:58 |
weshay | rlandy we just need to pass a different release to the job | 13:58 |
jfrancoa | panda|rover: sure, bluejeans or irc? | 13:58 |
rlandy | }}/{{ 'promotion-testing-hash-' if 'periodic' in zuul.pipeline else '' }}{{ release }}.yml | 13:58 |
rlandy | weshay: ^^ | 13:59 |
* rlandy is thinking about this | 13:59 | |
weshay | rlandy look at what we do upstream | 13:59 |
weshay | match that | 13:59 |
weshay | it's the same construct | 13:59 |
rlandy | toci expects a pattern | 13:59 |
arxcruz | weshay: https://review.rdoproject.org/r/#/c/20662/ is ready for review btw, although it's failing in the linters, and i have no idea where's the error, locally it also fail on master, so it's not this patch that is making linters fail | 13:59 |
panda|rover | jfrancoa: mmhh, wait a secong | 13:59 |
marios_|ruck | weshay: chandankumar honey badger council | 14:00 |
panda|rover | ykarel: http://logs.rdoproject.org/67/20867/5/check/periodic-tripleo-ci-centos-7-standalone-upgrade-stein/86df487/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz fails to pull the containers, but they re there, I checked, maybe a transient problem ? | 14:00 |
ykarel | panda|rover, looking | 14:01 |
*** aakarsh has joined #oooq | 14:02 | |
marios_|ruck | weshay: this might be why the numbers are better now | 14:03 |
ykarel | panda|rover, hmm skopeo inspect is working, but strange both master and stein failed | 14:04 |
panda|rover | ykarel: they were probably pulling the images at the same time .... | 14:05 |
panda|rover | ykarel: trying a recheck | 14:06 |
chandankumar | weshay: marios_|ruck from geneve http://pastebin.test.redhat.com/765424 | 14:06 |
ykarel | panda|rover, ack | 14:07 |
weshay | chandankumar marios_|ruck sounds like we're off the hook for today... marios_|ruck continue to focus on the standalone osp-15 w/ rlandy | 14:11 |
marios_|ruck | weshay: ack | 14:11 |
*** quiquell has quit IRC | 14:12 | |
marios_|ruck | rlandy: still in this call but we can sync in a bit if necessary. really we should discuss about where/when to run that job? otherwise i think the last 2 things were mergable at least | 14:13 |
rlandy | marios_|ruck: ack when you are ready | 14:13 |
*** quiquell has joined #oooq | 14:21 | |
*** bogdando has quit IRC | 14:21 | |
*** bogdando has joined #oooq | 14:21 | |
weshay | arxcruz we can disable ansible-lint or try to find the fix for https://github.com/ansible/ansible/issues/9903 | 14:25 |
marios_|ruck | rlandy: ok just finished call... say in 4 mins? | 14:26 |
marios_|ruck | rlandy: or we can wait longer if you are middle sthing else | 14:26 |
weshay | arxcruz oof .. .that is old | 14:26 |
weshay | nvrmind | 14:26 |
rlandy | marios_|ruck: working with sshnaidm but it's fine let's chat before you go offline | 14:26 |
arxcruz | weshay: where's Sorin? | 14:27 |
chandankumar | marios_|ruck: rlandy this review is ok https://review.rdoproject.org/r/#/c/20884/ ? | 14:27 |
arxcruz | this is his area, even if i run it on master, I get failure | 14:27 |
weshay | arxcruz tomorrow | 14:28 |
rlandy | chandankumar: zuul fails there | 14:28 |
arxcruz | ok, i'll check with him tomorrow | 14:28 |
marios_|ruck | rlandy: is fine around for another hour | 14:28 |
weshay | arxcruz ya.. he returns from the dessert tomorrow | 14:28 |
marios_|ruck | rlandy: let me know when you finish | 14:28 |
arxcruz | the error doesn't shows where exactly is the problem | 14:28 |
rlandy | k | 14:28 |
arxcruz | weshay: anyway, besides the tox issue, the code is ready, my next step will be add the full job | 14:29 |
*** dtantsur|brb is now known as dtantsur | 14:29 | |
arxcruz | but i want to do it in another patch, this one is already too big, i don't want to add more complexity | 14:29 |
weshay | arxcruz chat w/ me for a sec | 14:29 |
arxcruz | weshay: sure | 14:30 |
*** ykarel is now known as ykarel|afk | 14:31 | |
marios_|ruck | chandankumar: checking | 14:36 |
* chandankumar headed home | 14:37 | |
weshay | sshnaidm rlandy https://review.rdoproject.org/r/#/c/20861/ | 14:38 |
rlandy | marios_|ruck: k - time to meet now? | 14:38 |
marios_|ruck | chandankumar: looks like zuul issue will comment | 14:38 |
marios_|ruck | rlandy: ack | 14:39 |
marios_|ruck | rlandy: https://redhat.bluejeans.com/7661925373/ | 14:39 |
rlandy | weshay: tox errors on that | 14:39 |
rlandy | marios_|ruck: k- joining | 14:39 |
arxcruz | weshay: rlandy found the issue upstream https://github.com/ansible/ansible-lint/issues/506 | 14:44 |
*** udesale has quit IRC | 14:44 | |
*** saneax has quit IRC | 14:44 | |
*** udesale has joined #oooq | 14:44 | |
*** saneax has joined #oooq | 14:44 | |
*** holser__ has joined #oooq | 14:44 | |
*** tosky has quit IRC | 14:45 | |
*** tosky has joined #oooq | 14:46 | |
*** holser_ has quit IRC | 14:46 | |
*** quiquell has quit IRC | 14:53 | |
rlandy | weshay: this afternoon, will need someone else to look at the container-prepare-parameters. - if you have time. there is some spacing voodoo there | 14:59 |
*** Vorrtex has joined #oooq | 15:01 | |
weshay | arxcruz going to schedule a mtg w/ you me and sshnaidm to discuss the layout of the code we have for the reproducer and zuul-runner | 15:02 |
weshay | rlandy sure.. happy to continue the streak | 15:02 |
*** skramaja has quit IRC | 15:03 | |
*** ykarel|afk is now known as ykarel | 15:06 | |
*** jtomasek has joined #oooq | 15:11 | |
arxcruz | weshay: okay, today? I'm heading to german class, i'll be back in 4 hours | 15:11 |
*** holser__ has quit IRC | 15:12 | |
weshay | arxcruz tomorrow | 15:12 |
*** holser_ has joined #oooq | 15:12 | |
* marios_|ruck shutdown sequence | 15:16 | |
marios_|ruck | weshay: so tomorrow ruck and phase 2 | 15:16 |
marios_|ruck | weshay: when they have new runs by then | 15:17 |
marios_|ruck | will sync with chandankumar | 15:17 |
weshay | marios_|ruck hold on phase 2 until the osp-15mtg | 15:17 |
marios_|ruck | weshay: ah ok | 15:17 |
weshay | marios_|ruck little bit of rucking, but mostly push on osp-15 standalone please | 15:17 |
marios_|ruck | weshay: ack synced with rlandy waiting on the install-deps to merge then we merge the job | 15:17 |
weshay | marios_|ruck I can give you more context if you want.. basically what I was talking about on the osp-15 call today | 15:17 |
marios_|ruck | weshay: rlandy noticed we don't have the stein equivalent like we do for OSP14/rocky | 15:18 |
weshay | marios_|ruck right.. so we need another job defined | 15:18 |
marios_|ruck | weshay: maybe thats next | 15:18 |
marios_|ruck | weshay: ah you mean standalone osp on their h/ware | 15:18 |
marios_|ruck | job | 15:18 |
weshay | no | 15:18 |
weshay | blue | 15:18 |
marios_|ruck | going | 15:19 |
*** saneax has quit IRC | 15:21 | |
panda|rover | jfrancoa: jobs passed again an this time with the right repo https://review.rdoproject.org/r/20867 | 15:28 |
panda|rover | jfrancoa: can you take a look again ? | 15:28 |
jfrancoa | panda|rover: sure, checking... | 15:28 |
sshnaidm | rlandy, found what's problem with ovb, it's incorrect nodepool provider configuration | 15:28 |
rlandy | sshnaidm: sorry - was in meeting | 15:30 |
rlandy | happy you found it | 15:30 |
sshnaidm | rlandy, can you join please #sf-ops | 15:30 |
sshnaidm | rlandy, need to check that I don't break baremetal with a fix | 15:30 |
weshay | rlandy marios_|ruck stein standalone on rhel8 is not a viable target because we do not have containers for it | 15:31 |
weshay | although we now have a viable rhel-8 image | 15:31 |
weshay | still need containers | 15:32 |
rlandy | weshay: we don't run rocky on rhel anything | 15:32 |
rlandy | we run on centos | 15:32 |
rlandy | even downstream | 15:32 |
rlandy | stein should be the same, no? | 15:32 |
jfrancoa | panda|rover: both jobs look fine, the stein and master one | 15:33 |
rlandy | am I missing the point here? | 15:33 |
panda|rover | jfrancoa: SHIP IT! | 15:33 |
jfrancoa | panda|rover: go go go! | 15:34 |
*** aakarsh has quit IRC | 15:35 | |
panda|rover | weshay: https://review.opendev.org/660615 has a chance to unblock promotion | 15:35 |
rlandy | sshnaidm: just got disconnected internal ... https://github.com/softwarefactory-project/sf-config | 15:35 |
*** dalvarez has quit IRC | 15:35 | |
*** amoralej has quit IRC | 15:35 | |
*** radez has quit IRC | 15:35 | |
sshnaidm | rlandy, https://softwarefactory-project.io/cgit/config/tree/nodepool/rdo-cloud.yaml#n362 | 15:36 |
sshnaidm | rlandy, yeah, today it's keeping disconnecting for me too | 15:36 |
sshnaidm | rlandy, that's how we define networks in rdo cloud - private with floating IPs | 15:36 |
*** jfrancoa has quit IRC | 15:36 | |
sshnaidm | rlandy, so let's try it | 15:37 |
rlandy | sshnaidm: yeah - we have no other way of knowing | 15:37 |
rlandy | it's not an obvious config | 15:37 |
rlandy | sshnaidm: all we need for bm is for zuul to be able to connect to the underclou | 15:37 |
rlandy | sshnaidm: ping me when you merge, I'll kick a test bm job | 15:38 |
sshnaidm | rlandy, can you merge it? vpn completely failed for me.. | 15:38 |
* rlandy tries | 15:38 | |
rlandy | sshnaidm: down for me as well | 15:39 |
rlandy | won't resolve | 15:39 |
sshnaidm | yeah, same thing | 15:39 |
rlandy | ok -let's give it a few minutes | 15:39 |
*** radez has joined #oooq | 15:40 | |
*** vkapalav has quit IRC | 15:42 | |
*** lmiccini has quit IRC | 15:43 | |
*** lmiccini has joined #oooq | 15:44 | |
*** holser_ has quit IRC | 15:47 | |
*** holser_ has joined #oooq | 15:48 | |
*** marios has quit IRC | 15:50 | |
*** marios_|ruck has quit IRC | 15:50 | |
*** holser__ has joined #oooq | 15:51 | |
sshnaidm | weshay, vpn fails for everyone | 15:51 |
*** fmount has quit IRC | 15:51 | |
*** holser_ has quit IRC | 15:53 | |
*** derekh has quit IRC | 15:59 | |
*** ykarel is now known as ykarel|away | 16:01 | |
*** weshay_ has joined #oooq | 16:02 | |
weshay_ | rlandy, hey | 16:02 |
weshay_ | the containers for stein would have to be rhel 8 based | 16:02 |
*** jpena is now known as jpena|off | 16:02 | |
rlandy | weshay: hi | 16:04 |
weshay_ | howdy | 16:04 |
rlandy | weshay: why though? | 16:04 |
*** bogdando has quit IRC | 16:04 | |
rlandy | not rdo on rhel | 16:04 |
weshay_ | rlandy, stein ships on rhel8 | 16:04 |
weshay_ | rlandy, yes | 16:05 |
rlandy | just regular stein | 16:05 |
weshay_ | it should be on -8 | 16:05 |
weshay_ | doesn't really make for a good comparison | 16:05 |
weshay_ | between the osp job | 16:05 |
*** jtomasek has quit IRC | 16:05 | |
weshay_ | centos-7 stein compared to rhel-8 osp-15 | 16:05 |
rlandy | weshay_: how about on fedora28? | 16:05 |
weshay_ | better to hold off | 16:05 |
weshay_ | rlandy, meh.. | 16:05 |
weshay_ | we have other work to do on osp-15 | 16:06 |
rlandy | weshay_: ack - ok | 16:06 |
weshay_ | rlandy, most importantly getting it running on need_images | 16:06 |
rlandy | we were just talking about where those jobs run | 16:06 |
weshay_ | ya.. getting one non-voting check job would be a good step forward as well | 16:06 |
rlandy | weshay_: I'm editing the projects file on the standalone job | 16:07 |
weshay_ | k | 16:07 |
rlandy | weshay_: once https://review.opendev.org/#/c/660508/ merges we want to merge https://code.engineering.redhat.com/gerrit/#/c/167540/ | 16:08 |
rlandy | so the projects file has to be accurate | 16:08 |
weshay_ | rlandy, we should figure out how to get that job going on need_images before the next big deployment failure in osp phase 1 | 16:08 |
rlandy | weshay_: network is out atm so I can't access it | 16:08 |
weshay_ | heh | 16:08 |
weshay_ | weee | 16:08 |
rlandy | weshay_: ack - as soon as I have access again | 16:09 |
rlandy | just spinning | 16:09 |
*** tesseract has quit IRC | 16:10 | |
*** jbadiapa has quit IRC | 16:10 | |
weshay_ | rfolco, any blockers on the images job? | 16:12 |
rlandy | weshay_: in the mean time, to calify, for needs_images, | 16:14 |
rlandy | clarify | 16:14 |
rlandy | we want a new release file | 16:14 |
rlandy | a new job | 16:14 |
rlandy | triggered how? | 16:14 |
weshay_ | rlandy, periodic trigger for now is fine | 16:14 |
*** tesseract has joined #oooq | 16:14 | |
*** tesseract has quit IRC | 16:14 | |
weshay_ | we'd want that to trigger often though.. at least through this osp-15 debacle | 16:15 |
*** ykarel|away has quit IRC | 16:15 | |
rlandy | weshay_: k - putting something together | 16:16 |
weshay_ | thanks | 16:16 |
*** tcw has quit IRC | 16:17 | |
*** tcw has joined #oooq | 16:17 | |
rfolco | weshay_, more or less. I am testing the playbooks in the rhel box. The build command is failing... | 16:17 |
weshay_ | rfolco, can you pastebin the error on a public pastebin? | 16:18 |
*** Vorrtex has quit IRC | 16:18 | |
sshnaidm | rlandy, vpn seems to work again | 16:20 |
*** tcw has quit IRC | 16:20 | |
*** tcw has joined #oooq | 16:20 | |
rlandy | trying | 16:20 |
rfolco | weshay_, I am re-executing now. rhos-release is creating delorean.repo (old way)... the rpm does not have the latest change looks like. | 16:21 |
rlandy | sshnaidm: still can't access the review - can you? | 16:21 |
weshay_ | rfolco, which rhose-release are you pulling? | 16:22 |
weshay_ | I can't get on the vpn either | 16:22 |
rlandy | internal irc just came back | 16:22 |
*** jpena|off is now known as jpena | 16:22 | |
rlandy | gone again | 16:22 |
weshay_ | oooh I'm on | 16:23 |
*** amoralej has joined #oooq | 16:23 | |
weshay_ | phx | 16:23 |
weshay | ping weshay_ get the f out of here | 16:23 |
sshnaidm | rlandy, yeah, merged | 16:24 |
rlandy | I'm on rdu2 | 16:24 |
*** weshay_ has quit IRC | 16:24 | |
rfolco | weshay_, should be http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm, but configure-rhel playbook has another one... I am also fixing this part | 16:24 |
*** jbadiapa has joined #oooq | 16:24 | |
rlandy | sshnaidm: ^^ still disconnected - https://code.engineering.redhat.com/gerrit/#/c/165815/ is a master bm test job | 16:25 |
*** kopecmartin is now known as kopecmartin|off | 16:25 | |
rlandy | ok - finally back | 16:26 |
rlandy | let's try this again | 16:26 |
weshay | rfolco variables here look incomplete to me https://code.engineering.redhat.com/gerrit/#/c/170815/20/playbooks/build-image/templates/build-image.sh.j2 | 16:27 |
rlandy | oh man - out again | 16:28 |
rfolco | weshay, yah I am tetsing it local now w/ DIB_YUM_REPO_CONF, just want to confirm what is really required there | 16:29 |
chandankumar | weshay: do we use the box to try infrared manually tomorrow? | 16:31 |
weshay | chandankumar not in your day... waiting on some infrared patches to land... will ask again in tomorrow's mtg what we should go after | 16:33 |
weshay | chandankumar we could try for fun and profit | 16:33 |
weshay | or just fun | 16:33 |
chandankumar | weshay: let's drive the fun tomorrow | 16:33 |
chandankumar | with marios | 16:33 |
rlandy | sshnaidm: periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master kicked - will see what shakes out | 16:33 |
weshay | sounds good to me | 16:33 |
chandankumar | anyway tomorrow is judgement day in India | 16:34 |
weshay | sshnaidm rfolco you guys are both building images in your respective jobs | 16:34 |
chandankumar | government election results will come | 16:34 |
weshay | sshnaidm rfolco by design | 16:34 |
weshay | can you two please share your work w/ each other | 16:34 |
chandankumar | rlandy: I didnot get time to look at rdo logs stuff will work on that tomorrow | 16:35 |
*** Vorrtex has joined #oooq | 16:35 | |
rfolco | sshnaidm, https://code.engineering.redhat.com/gerrit/#/q/topic:rhel-rdo-rocky+(status:open+OR+status:merged) | 16:35 |
sshnaidm | rlandy, now there is a problem, trying to solve on sf-ops | 16:35 |
rfolco | sshnaidm, weshay this is my build-image job on rhel... | 16:36 |
*** udesale has quit IRC | 16:38 | |
rlandy | chandankumar: no worries - I didn't get much time there either. I can also pick up tomorrow | 16:39 |
*** vinaykns has joined #oooq | 16:39 | |
chandankumar | secret stuff is new, I will learn that | 16:39 |
rlandy | sshnaidm: on the upside, bm looks ok so far | 16:39 |
rlandy | chandankumar: I can show you how to to get the secret | 16:40 |
rlandy | or sshnaidm also knows - if you want to start before I am on line | 16:40 |
chandankumar | rlandy: yes, will poke sshnaidm tomorrow! | 16:41 |
rfolco | weshay, http://rhos-release.virt.bos.redhat.com/repos/rhos-release/rhos-release-latest.noarch.rpm is not the latest built rpm, where do I get it ? | 16:42 |
*** udesale has joined #oooq | 16:43 | |
weshay | rfolco #rhos-delivery | 16:46 |
rfolco | weshay, k thx | 16:46 |
chandankumar | pune office people are celebrating for RHEL-8 | 16:46 |
chandankumar | for me nothing is there | 16:47 |
chandankumar | going together for dinner party | 16:47 |
chandankumar | sshnaidm: how nodes are getting populated if this review https://review.opendev.org/#/c/660710/ gets merged? | 16:50 |
chandankumar | there are some node files where there is no default_name: overcloud-novacompute-1 | 16:50 |
chandankumar | hostnamemap_override: overcloud-novacompute-bar-1 | 16:51 |
chandankumar | and what does topology does? | 16:51 |
sshnaidm | chandankumar, should be fine, like here: https://github.com/openstack/tripleo-quickstart/blob/master/config/nodes/1ctlr_2comp_1supp.yml#L1 | 16:53 |
chandankumar | so basically we have pre defined vms with that flavor named compute or controller | 16:54 |
chandankumar | what does scale means? | 16:54 |
chandankumar | rlandy: we have https://review.opendev.org/#/c/658262/ rhel8 support in progress for os_Temoest | 16:59 |
chandankumar | I think we can test in rhos-15 job | 16:59 |
rlandy | chandankumar: ok - just include that depends-on? or is there more to it than that? | 17:00 |
rlandy | we are using a mocked up rhel node | 17:01 |
rlandy | if that would could for your testing | 17:01 |
chandankumar | rlandy: yes, I will put a new patch from current one | 17:01 |
rlandy | chandankumar: https://code.engineering.redhat.com/gerrit/#/c/167540/ shoudl be ready to merge as soon as https://review.opendev.org/#/c/660508/ merges | 17:02 |
*** dsneddon has joined #oooq | 17:06 | |
*** udesale has quit IRC | 17:12 | |
*** chandankumar is now known as raukadah | 17:14 | |
*** jpena is now known as jpena|off | 17:17 | |
*** jtomasek has joined #oooq | 17:27 | |
rlandy | weshay: I am building the rhos-15 needs_images job now | 17:43 |
rlandy | should that be named periodic-* | 17:43 |
rlandy | report to dlrn? | 17:43 |
rlandy | or none of the above just use the correct release file | 17:44 |
weshay | rlandy it does not have to report to dlrn at this time, not sure if it ever will | 17:45 |
weshay | rlandy it certainly can be called periodic | 17:45 |
rlandy | weshay: ok - coming up | 17:45 |
weshay | rlandy rock on... | 17:45 |
weshay | I can give you more context of how I see that job would play a role for us.. if you want it | 17:46 |
weshay | rlandy can you retrigger fs001 on bm for me please | 17:54 |
weshay | I thought we were out of the woods, but we're not | 17:54 |
rlandy | weshay: sure - there is one running right now | 17:54 |
rlandy | oh wait - it's periodic | 17:55 |
rlandy | will retrigger | 17:55 |
weshay | rlandy /me is looking for validation of whether or not tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failures are real | 17:55 |
rlandy | weshay: just one moment for me to submit rhos-15 need-images stuff | 17:55 |
rlandy | weshay: pls review https://code.engineering.redhat.com/gerrit/171119 - not sure if this is what you intend | 17:58 |
rlandy | also note that the job will only get tripleo-ci-testing-hash if run in periodic pipeline | 17:59 |
rlandy | I will edit the playbook to test that next | 17:59 |
rlandy | as the spacing will be off | 17:59 |
rlandy | http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/playbooks/configure-rhel.yaml#n35 | 17:59 |
rlandy | }}/{{ 'promotion-testing-hash-' if 'periodic' in zuul.pipeline else '' }}{{ release }}.yml | 18:00 |
*** amoralej is now known as amoralej|off | 18:00 | |
weshay | rlandy tripleo-ci-testing hash? | 18:00 |
rlandy | ^^ as currently set, the would come out to | 18:00 |
rlandy | promotion-testing-hash-need-images/rhos-15.yaml | 18:00 |
weshay | that review looks good | 18:00 |
weshay | release_file: need_images/rhos-15.yml | 18:00 |
rlandy | weshay: ^^ see what I am describing above? | 18:00 |
rlandy | promotion-testing-hash-need-images/rhos-15.yaml | 18:00 |
rlandy | in a periodic pipeline | 18:01 |
rlandy | that is wrong I think | 18:01 |
rlandy | anyways, let me kick bm again | 18:01 |
weshay | rlandy that looks good to me :) | 18:01 |
weshay | +2 | 18:01 |
weshay | rlandy oh wait | 18:02 |
weshay | need_images/rhos-15 or need_images/rhos-15.yml | 18:02 |
weshay | I think w/o the yaml | 18:02 |
* weshay checks | 18:02 | |
rlandy | arxcruz: are you still connected to rdoci-hp-01.v100.rdoci.lab.eng.rdu2.redhat.com? | 18:03 |
rlandy | envD | 18:04 |
rlandy | weshay: bm kicked again .... but I see this virt-customize error in the previous job ... fatal: unable to access 'https://opendev.org/openstack/tripleo-ci/': Encountered end of file | 18:06 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/15/165815/53/check/periodic-tripleo-ci-centos-7-bm_envD-3ctlr_1comp-featureset001-master/dcd71e9/job-output.txt.gz#_2019-05-22_17_04_36_618651 | 18:07 |
rlandy | fixing rhos-15 | 18:07 |
weshay | rlandy ya.. 402fa89ab4460cc7fb62/work/src/code.engineering.redhat.com/tripleo-environments/config/release/need_images/rhos-15.yml.yml' | 18:08 |
weshay | rlandy I have no idea if that job will actually pass btw.. depends on the rpms there | 18:09 |
weshay | will show it to the rhos-delivery folks when we are done | 18:09 |
*** Goneri has quit IRC | 18:11 | |
rlandy | updated | 18:12 |
rlandy | weshay: running now ... https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/93e3d558ae04462b940b90d6818d3832?logfile=console.log | 18:20 |
rlandy | needs_images thing | 18:20 |
rlandy | and your bm job | 18:21 |
*** jtomasek has quit IRC | 18:23 | |
*** aakarsh has joined #oooq | 18:23 | |
*** jtomasek_ has joined #oooq | 18:24 | |
weshay | rlandy will be interesting to see what happens w/ containers here... as they are not built yet for needs_images | 18:27 |
weshay | uses older ones | 18:27 |
raukadah | weshay: rlandy any idea about this issue https://sf.hosted.upshift.rdu2.redhat.com/logs/17/171117/1/check/tripleo-ci-rhel-8-standalone-rhos-15/6329fa4/job-output.txt.gz#_2019-05-22_13_55_00_430185 | 18:27 |
*** tosky has quit IRC | 18:30 | |
*** tosky has joined #oooq | 18:30 | |
raukadah | is it possible to use this job as a third party job against os_tempest? | 18:31 |
raukadah | one blocker is rdo logs which I will get it fixed? | 18:31 |
raukadah | *. | 18:31 |
rfolco | weshay, do you have 5 min to chat about https://code.engineering.redhat.com/gerrit/#/c/167729? | 18:32 |
weshay | rfolco ya | 18:32 |
rfolco | weshay, your bj? | 18:32 |
rlandy | raukadah: looking | 18:40 |
rlandy | Failed to synchronize cache for repo 'rhos-release-extras', ignoring this repo | 18:42 |
raukadah | rlandy: will i do a recheck? | 18:42 |
rlandy | feel like we hot this one before | 18:42 |
rlandy | raukadah: you can type 'recheck' | 18:43 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/logs/17/171117/1/check/tripleo-ci-rhel-8-standalone-rhos-15/6329fa4/logs/undercloud/etc/yum.repos.d/rhos-release.repo.txt.gz | 18:44 |
rlandy | raukadah: ^^ so that is the repo | 18:44 |
raukadah | done recheck, let's see how it goes! | 18:44 |
rlandy | raukadah: we may need to inquire of #rhos-delivery people | 18:45 |
rlandy | to check what is available | 18:45 |
*** sshnaidm is now known as sshnaidm|afk | 18:45 | |
rlandy | weshay: well the needs-images run is deploying standalone | 18:46 |
rlandy | so it's using some containers | 18:46 |
weshay | rfolco https://images.rdoproject.org/master/ | 18:47 |
weshay | https://images.rdoproject.org/rocky/rdo_trunk/ | 18:47 |
weshay | https://images.rdoproject.org/rocky/rdo_trunk/centos7 | 18:48 |
rfolco | https://images.rdoproject.org/master/rdo_trunk/tripleo-ci-testing/ | 18:48 |
weshay | https://images.rdoproject.org/rocky/rdo_trunk/rhel7 | 18:48 |
*** holser__ is now known as holser-afk | 18:55 | |
weshay | rlandy need_images worked | 19:19 |
rlandy | weshay: not sure it really worked - it passed ... but are the repos etc. correct? | 19:21 |
weshay | rlandy I'm getting on w/ jjoyce to confirm that | 19:23 |
weshay | they are older than passed_phase_1 | 19:23 |
weshay | https://sf.hosted.upshift.rdu2.redhat.com/logs/19/171119/2/check/periodic-tripleo-ci-rhel-8-standalone-need-images-rhos-15/93e3d55/logs/undercloud/etc/yum.repos.d/rhos-release-15-trunk.repo.txt.gz | 19:23 |
*** jtomasek_ has quit IRC | 19:24 | |
*** jtomasek has joined #oooq | 19:24 | |
*** quiquell has joined #oooq | 19:26 | |
*** quiquell has quit IRC | 19:26 | |
rlandy | weshay: are the containers correct? - modify_append_tag: -updated-20190522142010 | 19:32 |
*** Goneri has joined #oooq | 19:37 | |
*** jtomasek has quit IRC | 19:42 | |
*** jtomasek_ has joined #oooq | 19:43 | |
weshay | rlandy ok.. let's chat for a sec | 19:43 |
rlandy | weshay: k - joining | 19:43 |
arxcruz | rlandy: i wasn't connected there... | 19:46 |
rlandy | arxcruz: np - I was just reprovisioning so wanted to tell anyone who might still need that info | 19:46 |
arxcruz | weshay: I found a workaround solution for the tox linters, i'll submit a patch tomorrow | 19:57 |
arxcruz | weshay: Ansible-lint.............................................................Passed | 19:57 |
weshay | arxcruz++ | 19:58 |
*** jtomasek_ has quit IRC | 20:02 | |
*** jtomasek has joined #oooq | 20:02 | |
weshay | rlandy http://www.mydailytutorials.com/ansible-replace-example/ | 20:07 |
*** Vorrtex has quit IRC | 20:24 | |
rfolco | weshay, success!! https://sf.hosted.upshift.rdu2.redhat.com/logs/16/170816/25/check/tripleo-buildimage-overcloud-rocky-rhel-7/9340e2d/logs/build-image.log.txt.gz | 20:42 |
weshay | nice! | 20:43 |
weshay | rfolco cool.. I think that logs everything we need too :) | 20:44 |
rfolco | :) | 20:46 |
rlandy | weshay: https://code.engineering.redhat.com/gerrit/#/c/167540 and https://code.engineering.redhat.com/gerrit/#/c/171119/ are rebased and should be mergeable when the upstream change is in | 20:47 |
rlandy | weshay: ping ... the rhos-14 jobs are non-voting | 21:08 |
rlandy | rhos-15 job should be non-voting as well? | 21:08 |
weshay | rlandy in periodic? | 21:16 |
weshay | doesn't matter | 21:16 |
rlandy | weshay: the rhos-15 job is also in check | 21:16 |
weshay | rlandy it can vote on ci changes | 21:16 |
rlandy | if we expect it to pass, it's ok | 21:16 |
weshay | just not tripleo changes yet | 21:16 |
rlandy | correct | 21:17 |
rlandy | just on this repo | 21:17 |
weshay | ya | 21:17 |
weshay | it can vote on any ci or zuul change | 21:17 |
*** aakarsh has quit IRC | 21:30 | |
weshay | rlandy you can see the containers get updated http://logs.openstack.org/94/659394/1/check/tripleo-ci-centos-7-standalone/cb090dc/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 21:32 |
weshay | via https://github.com/openstack/ansible-role-tripleo-modify-image | 21:32 |
weshay | I do not see that in the internal job | 21:32 |
rlandy | hmmm - maybe missing a setting | 21:33 |
weshay | http://logs.openstack.org/94/659394/1/check/tripleo-ci-centos-7-standalone/cb090dc/logs/undercloud/home/zuul/standalone_parameters.yaml.txt.gz | 21:33 |
weshay | don't se it yet | 21:33 |
weshay | modify_role: tripleo-modify-image | 21:34 |
weshay | modify_vars: | 21:34 |
weshay | tasks_from: yum_update.yml | 21:34 |
weshay | update_repo: gating-repo,delorean-current,quickstart-centos-ceph-nautilus | 21:34 |
weshay | that's it | 21:34 |
weshay | modify_role: tripleo-modify-image | 21:35 |
weshay | modify_vars: | 21:35 |
weshay | tasks_from: yum_update.yml | 21:35 |
weshay | update_repo: gating-repo,delorean-current,quickstart-centos-ceph-nautilus | 21:35 |
weshay | yum_repos_dir_path: /etc/yum.repos.d | 21:35 |
rlandy | lol | 21:35 |
weshay | so the osp job has rdo repos listed :) | 21:35 |
rlandy | the parameters file | 21:35 |
weshay | back to your file :) | 21:35 |
weshay | lolz | 21:35 |
rlandy | someone shoot me | 21:35 |
rlandy | but it's not modified | 21:35 |
rlandy | in rhos-15 | 21:35 |
rlandy | I have no idea where those settings should be | 21:36 |
rlandy | we add them to the file | 21:36 |
rlandy | so the placement is anyone's guess | 21:36 |
rlandy | this is my latest try ... | 21:36 |
rlandy | https://review.opendev.org/#/c/659794/5/roles/standalone/tasks/main.yml | 21:37 |
rlandy | weshay: I think we will be spending some more quality time with that file | 21:37 |
*** Goneri has quit IRC | 21:43 | |
weshay | rlandy just change the repos | 21:44 |
weshay | to | 21:44 |
weshay | rhelosp-15.0-trunk | 21:45 |
weshay | rhelosp-ceph-4.0-mon-pending | 21:46 |
weshay | would be the equiv | 21:46 |
weshay | rlandy probably can make release vars for that | 21:46 |
*** jtomasek has quit IRC | 21:47 | |
rlandy | weshay: where? | 21:47 |
weshay | rlandy want to chat? | 21:47 |
weshay | lolz | 21:48 |
rlandy | ok | 21:48 |
weshay | it's just quicker | 21:48 |
*** Goneri has joined #oooq | 21:53 | |
*** rlandy is now known as rlandy|bbl | 21:59 | |
*** Goneri has quit IRC | 22:32 | |
*** tosky has quit IRC | 23:11 | |
*** tosky has joined #oooq | 23:11 | |
*** aakarsh has joined #oooq | 23:20 | |
*** tosky has quit IRC | 23:34 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!