*** ysandeep|afk is now known as ysandeep | 00:10 | |
*** tosky has quit IRC | 00:50 | |
*** jmasud has quit IRC | 00:54 | |
*** jmasud has joined #oooq | 00:55 | |
*** rlandy|bbl is now known as rlandy | 02:22 | |
*** rlandy has quit IRC | 02:45 | |
*** apetrich has quit IRC | 03:09 | |
*** udesale has joined #oooq | 04:32 | |
*** dsneddon_ has quit IRC | 04:41 | |
*** skramaja has joined #oooq | 04:47 | |
*** ratailor has joined #oooq | 04:54 | |
*** dsneddon_ has joined #oooq | 06:21 | |
*** dsneddon_ has quit IRC | 06:26 | |
*** marios has joined #oooq | 06:40 | |
*** skramaja_ has joined #oooq | 06:51 | |
*** skramaja has quit IRC | 06:51 | |
*** dpawlik has joined #oooq | 06:52 | |
*** dpawlik has quit IRC | 06:53 | |
*** dpawlik has joined #oooq | 06:53 | |
*** jfrancoa has joined #oooq | 07:29 | |
*** amoralej|off is now known as amoralej | 07:32 | |
*** apetrich has joined #oooq | 07:36 | |
*** tesseract has joined #oooq | 07:51 | |
amoralej | chandankumar, wrt https://bugs.launchpad.net/tripleo/+bug/1867931 are you around? | 08:03 |
---|---|---|
openstack | Launchpad bug 1867931 in tripleo "periodic-tripleo-ci-centos-8-scenario010-standalone-network-master deploy fails with package conflict - "openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts"" [Critical,Triaged] | 08:03 |
amoralej | i'd like to create a reproducer locally, as i couldn't reproduced the issue out of the env | 08:04 |
*** bogdando has joined #oooq | 08:04 | |
chandankumar | amoralej, I think we need to waive the criteria to make it passing https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-network-master | 08:11 |
chandankumar | let me propose a patch | 08:12 |
amoralej | chandankumar, i'm trying to do minimal reproducer using tripleo-modify-image role | 08:17 |
chandankumar | ok | 08:17 |
chandankumar | amoralej, marios https://review.rdoproject.org/r/25996 | 08:19 |
*** tosky has joined #oooq | 08:21 | |
marios | chandankumar: k | 08:21 |
marios | reading bug | 08:21 |
marios | chandankumar: but that means we're promoting a bad neutron version? | 08:22 |
marios | chandankumar: how will it be fixed? | 08:23 |
chandankumar | marios, once we build the fresh container and then upgrade issues will be vanished | 08:23 |
chandankumar | it will work in integration pipeline | 08:23 |
chandankumar | any further issue can be caught in integration pipeline | 08:24 |
marios | chandankumar: ok good point on the integration pipeline , this is component promotion. that's a bit better then... | 08:24 |
chandankumar | marios, I will wait amoralej last reproducer work | 08:24 |
*** jbadiapa has joined #oooq | 08:33 | |
marios | fellow tripleo-ci folks please check https://review.opendev.org/#/c/710556/2 when you get a chance (I just added you) cgoncalves is adding a new FS but the only diff is 'standalone-upgrade' so IMO we should just add to allowed_overrides | 08:40 |
marios | i know panda is particularly fond of this approach and in his words 'lets just override everything why note' | 08:42 |
marios | (not) | 08:42 |
* marios twists needle in panda doll some more | 08:42 | |
amoralej | chandankumar, i've reproduced the issue | 08:44 |
amoralej | locally | 08:44 |
amoralej | with the role | 08:44 |
chandankumar | amoralej, awesome | 08:44 |
amoralej | there is some problem in yum_update.sh | 08:44 |
amoralej | when building the list of packages to update | 08:44 |
amoralej | that's wrong /usr/bin/dnf -y update openstack-neutron-common openstack-tripleo-common-container-base puppet-concat puppet-glance puppet-kafka puppet-keystone puppet-midonet puppet-vcsrepo python3-neutron" | 08:45 |
chandankumar | amoralej, sorry you mean it does not include python3-ovn-networking package? | 08:46 |
*** jpena|off is now known as jpena | 08:48 | |
amoralej | chandankumar, it does not include openstack-neutron which provides/obsoletes python3-networking-ovn | 08:52 |
amoralej | we could move provides/obsoletes to openstack-neutron-common | 08:52 |
amoralej | as workaround | 08:52 |
chandankumar | amoralej, next question is does openstack-neutron gets updated in the current repo? | 08:52 |
amoralej | chandankumar, from repo pov is fine | 08:53 |
amoralej | the problem is | 08:53 |
amoralej | as currently, openstack-neutron is not installed in the container | 08:53 |
amoralej | it's not detected to be updated | 08:53 |
chandankumar | amoralej, ah | 08:53 |
amoralej | it may happen with any package renaming | 08:53 |
amoralej | yum_update.sh only tries to update package which *are already* installed | 08:54 |
amoralej | if a package is updated by a different one using provides, it will fail | 08:54 |
*** jmasud has quit IRC | 08:54 | |
chandankumar | amoralej, ok got it | 08:56 |
chandankumar | amoralej, let's get the workaround in neutron-common and then we can re-run the job | 08:56 |
*** jmasud has joined #oooq | 08:57 | |
*** skramaja_ has quit IRC | 09:12 | |
*** skramaja has joined #oooq | 09:14 | |
amoralej | chandankumar, i explained what the problem is in https://bugs.launchpad.net/tripleo/+bug/1867931 | 09:17 |
openstack | Launchpad bug 1867931 in tripleo "periodic-tripleo-ci-centos-8-scenario010-standalone-network-master deploy fails with package conflict - "openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts"" [Critical,Triaged] | 09:17 |
amoralej | i'll propose workaround | 09:17 |
zbr | panda: https://review.rdoproject.org/r/#/c/25989/ | 09:29 |
panda | marios: this morning I woke up suddenly with a sharp pain in my back. | 09:37 |
arxcruz|rover | panda: are you okay man? | 09:37 |
marios | pokerface | 09:37 |
arxcruz|rover | I'm getting very stressed, every day, if my daughter cof, I got freaking nervous | 09:38 |
marios | arxcruz|rover: ( 10:42 * marios twists needle in panda doll some more | 09:38 |
arxcruz|rover | marios: ahhh hehehe, sorry | 09:38 |
marios | arxcruz|rover: yes agree this is very s***y situation :( and no one knows how long it will last | 09:38 |
arxcruz|rover | it's being hard, 2 weeks already without leave home | 09:38 |
marios | arxcruz|rover: so in germany there is measures for staying at home like italy? | 09:39 |
*** dtantsur|afk is now known as dtantsur | 09:39 | |
arxcruz|rover | marios: yes, even churches are closed | 09:40 |
arxcruz|rover | only essential | 09:40 |
marios | arxcruz|rover: here most things are (ordered to) close but there is no law about moving freely yet, even though like 70% people are just at home roads are empty | 09:41 |
arxcruz|rover | well, it started this monday, but I started not leaving home one week before, just in case | 09:41 |
marios | arxcruz|rover: it is suspected they will make it that we can't leave the house in the next few hours, but thre is no announcement yet | 09:41 |
arxcruz|rover | marios: you can move free around berlin, but you can't go to crowded places | 09:41 |
arxcruz|rover | in fact, crouded places are prohibited here | 09:42 |
marios | yah here too | 09:42 |
arxcruz|rover | but they say, if you can, stay at home | 09:42 |
marios | same | 09:42 |
marios | surreal | 09:42 |
arxcruz|rover | and in my building there is a couple that is positive to corona | 09:42 |
arxcruz|rover | yeah man, i thought I would be killing zombies in the apocalipse | 09:43 |
* marios twists needle in panda doll some more | 09:43 | |
marios | oh well we'll find ways to cope i guess | 09:43 |
*** derekh has joined #oooq | 09:43 | |
marios | :( | 09:43 |
*** rascasoft has joined #oooq | 09:47 | |
*** sshnaidm|afk is now known as sshnaidm | 09:49 | |
*** dsneddon_ has joined #oooq | 09:55 | |
*** dsneddon_ has quit IRC | 10:00 | |
zbr | marios or panda https://review.rdoproject.org/r/#/c/25987/ | 10:04 |
zbr | sshnaidm: should we make this voting? https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-zuul-runner-reproducer-fedora-30-host | 10:06 |
zbr | or is too soon? | 10:06 |
zbr | or maybe adding it to a weekly pipeline? | 10:07 |
sshnaidm | zbr, where do you see this job? | 10:08 |
zbr | https://review.rdoproject.org/r/#/c/25987/ | 10:08 |
zbr | i see non-voting there | 10:09 |
zbr | i have no knowledge about the project, but i fixed the skip_missing_interpreters, and i was surprised about the nv | 10:09 |
sshnaidm | zbr, yeah, ok to do it voting | 10:09 |
sshnaidm | zbr, it's not a developing project right now.. | 10:10 |
zbr | sshnaidm: you can also add it to weekend pipeline at the same time, it could prove practical. | 10:10 |
sshnaidm | zbr, this is stalled, no need | 10:10 |
zbr | at least we will know if it gets broken faster instead of two months later. ok, if you say so. | 10:10 |
marios | zbr: ack | 10:11 |
*** amoralej is now known as amoralej|afk | 10:13 | |
arxcruz|rover | chandankumar: https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-master/3e5f3ca/logs/undercloud/var/log/containers/stdouts/container-puppet-neutron.log.txt.gz | 10:16 |
arxcruz|rover | have you seen this error ? | 10:16 |
arxcruz|rover | 2020-03-18T17:40:18.540481168+00:00 stderr F <13>Mar 18 17:40:18 puppet-user: No such file or directory @ rb_sysopen - /etc/neutron/plugins/networking-ovn/networking-ovn-metadata-agent.ini | 10:16 |
arxcruz|rover | opening a promotion blocker bug | 10:16 |
marios | chandankumar: nice green baremeratl fs1 run(s) | 10:29 |
*** holser has joined #oooq | 10:36 | |
*** tesseract has quit IRC | 10:36 | |
*** tesseract has joined #oooq | 10:42 | |
*** tesseract has quit IRC | 10:46 | |
*** tesseract has joined #oooq | 10:48 | |
weshay|ruck | arxcruz|rover, https://review.opendev.org/#/c/713676/3/roles/build-containers/tasks/main.yaml | 10:50 |
arxcruz|rover | weshay|ruck: weird, i'm pretty sure i added the become :/ | 10:51 |
arxcruz|rover | weshay|ruck: doing right now | 10:51 |
weshay|ruck | :) | 10:51 |
weshay|ruck | arxcruz|rover, chandankumar let me grab a change for you re neutron ovn | 10:51 |
arxcruz|rover | weshay|ruck: https://bugs.launchpad.net/tripleo/+bug/1868079 | 10:53 |
openstack | Launchpad bug 1868079 in tripleo "puppet-neutron container failed" [Critical,Triaged] | 10:53 |
arxcruz|rover | the bug | 10:53 |
weshay|ruck | chandankumar, ugh | 10:54 |
weshay|ruck | arxcruz|rover, make chandankumar fix it :) | 10:55 |
weshay|ruck | chandankumar, I don't think we should have waived https://bugs.launchpad.net/tripleo/+bug/1867931 | 10:55 |
openstack | Launchpad bug 1867931 in tripleo "periodic-tripleo-ci-centos-8-scenario010-standalone-network-master deploy fails with package conflict - "openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts"" [Critical,Triaged] | 10:55 |
*** tesseract-RH has joined #oooq | 11:00 | |
weshay|ruck | chandankumar, amoralej|afk we have a held node for the neutron issue | 11:00 |
weshay|ruck | holding the waive | 11:00 |
weshay|ruck | arxcruz|rover, ignore centos-8 pipeline in integration for now | 11:01 |
*** yolanda has quit IRC | 11:01 | |
arxcruz|rover | ok | 11:01 |
panda | marios: not sure if you're interested at this point, added explanations on https://review.rdoproject.org/r/25938 | 11:02 |
*** tesseract-RH has left #oooq | 11:02 | |
*** tesseract has quit IRC | 11:03 | |
panda | marios: and for heaven's sake, put down that doll. | 11:04 |
marios | panda: thanks pandaman i will check again. what doll? | 11:06 |
marios | chandankumar: did you see the weird tempest error (and no logs) at https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-compute-master/d55ae94/job-output.txt | 11:07 |
marios | https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tempest-master/de4fa04/job-output.txt | 11:07 |
marios | * 2020-03-18 20:24:46.925453 | primary | Data could not be sent to remote host "127.0.0.2". Make sure this host can be reached over ssh: kex_exchange_identification: Connection closed by remote host | 11:07 |
*** holser has quit IRC | 11:08 | |
*** holser has joined #oooq | 11:10 | |
chandankumar | weshay|ruck, amoralej|afk fixed it | 11:11 |
weshay|ruck | chandankumar, patch? | 11:11 |
*** holser has quit IRC | 11:11 | |
chandankumar | weshay|ruck, https://review.rdoproject.org/r/#/c/25997/ | 11:11 |
*** holser has joined #oooq | 11:12 | |
weshay|ruck | :) | 11:12 |
weshay|ruck | not a ci bug :) | 11:12 |
chandankumar | weshay|ruck, rechecked here https://review.rdoproject.org/r/#/c/25325/ | 11:12 |
weshay|ruck | chandankumar++ | 11:12 |
* weshay|ruck looks | 11:12 | |
*** yolanda has joined #oooq | 11:13 | |
*** dtantsur is now known as dtantsur|afk | 11:20 | |
chandankumar | arxcruz|rover, weshay|ruck https://review.opendev.org/713809 | 11:21 |
*** ysandeep is now known as ysandeep|afk | 11:28 | |
chandankumar | marios, yes in the morning | 11:34 |
chandankumar | marios, I have seen this earlier also may due to network issue | 11:34 |
marios | chandankumar: ack hard to see whats up without logs though not sure how to proceed there | 11:36 |
*** ccamacho has quit IRC | 11:38 | |
*** ccamacho has joined #oooq | 11:44 | |
weshay|ruck | wow... two full promotions from baremetal :) | 11:54 |
*** rlandy has joined #oooq | 11:57 | |
chandankumar | amoralej|afk, Hello | 11:58 |
chandankumar | amoralej|afk, still not helping https://logserver.rdoproject.org/25/25325/17/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/42b1497/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 11:58 |
weshay|ruck | chandankumar, amoralej|afk we have a held node.. and I also set it up on my local env.. just rpm though | 11:59 |
weshay|ruck | chandankumar, amoralej|afk /me notes that internal osp packaging uses the same role to update containers | 12:00 |
* weshay|ruck wonders if the issue is that the config file has changed content? | 12:00 | |
weshay|ruck | not default from rpm | 12:00 |
* weshay|ruck checks w/ my env | 12:00 | |
sshnaidm | weshay|ruck, chandankumar hi, so wrt renaming, are you good with such an approach? https://review.rdoproject.org/r/#/q/topic:f027+(status:open+OR+status:merged) | 12:02 |
weshay|ruck | I am now that we're out of the weeds.. marios ^? | 12:03 |
chandankumar | weshay|ruck, need to include openstack-neutron-common package | 12:04 |
chandankumar | sorry python3-neutron | 12:04 |
weshay|ruck | amoralej|afk, chandankumar if it turns out the only time this package would upgrade from this version is right now.. I think we can waive the job | 12:04 |
weshay|ruck | chandankumar, that's in the neutron component no? | 12:05 |
* weshay|ruck looks | 12:05 | |
chandankumar | weshay|ruck, please merge waive patch | 12:05 |
rlandy | this could happen in other situations in general | 12:05 |
weshay|ruck | chandankumar, need more than that | 12:05 |
chandankumar | weshay|ruck, loike | 12:06 |
chandankumar | like | 12:06 |
chandankumar | sshnaidm, checking | 12:06 |
weshay|ruck | chandankumar, so note.. if this would not repeat in osp container builds | 12:07 |
rlandy | chandankumar: looks enough to me to continue | 12:07 |
rlandy | what else would we need to get through criteria? | 12:07 |
weshay|ruck | chandankumar, /me notes they ALSO use tripleo-modify-image to update their containers | 12:07 |
weshay|ruck | chandankumar, /me sets up locally.. give me a few | 12:08 |
rlandy | chandankumar: how often does a package get obsoleted by a different component? | 12:08 |
*** holser has quit IRC | 12:08 | |
chandankumar | rlandy, if obseletes keyword defined in spec file | 12:08 |
chandankumar | rlandy, https://review.rdoproject.org/r/#/c/25997/ | 12:09 |
rlandy | "but doing it at the python module level is more accurate as it's the package" | 12:10 |
*** ysandeep|afk is now known as ysandeep | 12:10 | |
rlandy | ^^ what are the chances we see that again? | 12:10 |
*** holser has joined #oooq | 12:11 | |
weshay|ruck | sshnaidm, need a patch to update criteria https://review.rdoproject.org/r/#/q/topic:f027+(status:open+OR+status:merged) | 12:11 |
chandankumar | rlandy, we obeselete a package when there is a rename of the package or some context exists in both project | 12:12 |
chandankumar | *same | 12:12 |
sshnaidm | weshay|ruck, this one? https://review.rdoproject.org/r/#/c/25835/ | 12:12 |
marios | weshay|ruck: reading back | 12:12 |
marios | weshay|ruck: k will add that to my reviews | 12:14 |
chandankumar | rlandy, let me check whether the fix landed in the consistent repo or not | 12:14 |
weshay|ruck | chandankumar++ | 12:15 |
chandankumar | we need to run network component consitent then network standalone jobs na? | 12:15 |
chandankumar | rlandy, ^^ | 12:15 |
chandankumar | rlandy, https://review.rdoproject.org/r/25325 here we donot promoted the fix so it is not there na? | 12:15 |
rlandy | chandankumar: we need to rerun the promote job | 12:16 |
chandankumar | rlandy, yes that's why fix is not there | 12:16 |
chandankumar | let's rerun it and see | 12:16 |
rlandy | chandankumar: it ran the first time | 12:16 |
rlandy | but let me rerun in testproject - sec | 12:16 |
*** udesale_ has joined #oooq | 12:19 | |
rlandy | chandankumar: weshay|ruck: https://review.rdoproject.org/r/#/c/25325/ | 12:19 |
rlandy | that will promote consistent-to-component-ci-testing and run scenario010 again | 12:20 |
*** udesale has quit IRC | 12:20 | |
chandankumar | rlandy, aye, hope to see it working | 12:20 |
rlandy | chandankumar: let's hope ... | 12:20 |
chandankumar | arxcruz|rover, weshay|ruck https://github.com/raukadah/openstack-tempest-skiplist | 12:20 |
rlandy | but we need to address this issue in general - in scrum | 12:21 |
chandankumar | arxcruz|rover, weshay|ruck I am keeping train and master skip list as rest will go away in sometime | 12:22 |
weshay|ruck | chandankumar, amoralej|afk so.. I'll rolled back.. updated the config file... /etc/neutron/plugins/networking-ovn/networking-ovn.ini then ran an update | 12:22 |
weshay|ruck | and it worked fine | 12:22 |
weshay|ruck | chandankumar, anything else to try? | 12:22 |
chandankumar | weshay|ruck, I am out of ideas | 12:23 |
weshay|ruck | chandankumar, k... /me removes -2 | 12:23 |
weshay|ruck | rlandy, ? | 12:23 |
chandankumar | weshay|ruck, please wait for rlandy patch result | 12:23 |
weshay|ruck | any further thoughts? | 12:23 |
weshay|ruck | k | 12:23 |
rlandy | weshay|ruck: in general ,lots | 12:23 |
rlandy | for this, let it go if the testproject passes | 12:24 |
weshay|ruck | k | 12:24 |
rlandy | this is our basic component pipeline mechanism | 12:24 |
rlandy | it needs to work for ALL update cases | 12:24 |
rlandy | how ever packages get changes/obsoleted/ | 12:25 |
*** amoralej|afk is now known as amoralej | 12:28 | |
*** jpena is now known as jpena|lunch | 12:30 | |
amoralej | weshay|ruck, chandankumar let me try with the new packages | 12:30 |
weshay|ruck | k.. thanks for your patience on this one amoralej | 12:30 |
amoralej | weshay|ruck, n p | 12:31 |
weshay|ruck | arxcruz|rover, you need anything? | 12:31 |
amoralej | weshay|ruck, what do you mean by "I'll rolled back.. updated the config file... /etc/neutron/plugins/networking-ovn/networking-ovn.ini then ran an update" ? | 12:31 |
arxcruz|rover | weshay|ruck: toilette paper | 12:32 |
weshay|ruck | LOLZ | 12:32 |
arxcruz|rover | lol just kidding, no, everything is fine | 12:32 |
chandankumar | Indian does not need to worry about toilet papers | 12:32 |
arxcruz|rover | I am scary to ask... | 12:32 |
chandankumar | arxcruz|rover, water is everywhere | 12:32 |
amoralej | chandankumar, weshay|ruck in my test, it works fine after https://review.rdoproject.org/r/#/c/25997/ | 12:33 |
amoralej | with the role, i mean | 12:33 |
chandankumar | amoralej, after rlandy https://review.rdoproject.org/r/#/c/25325/ review we can see real result that time we donot promoted consistent repo to component repo | 12:34 |
amoralej | chandankumar, i've tested again with the same reproducer playbook | 12:40 |
amoralej | that was reproducing the issue | 12:40 |
amoralej | and this time worked fine | 12:40 |
amoralej | after moving the provides/obsoletes | 12:40 |
rlandy | chandankumar: weshay|ruck: if the testproject works, you won't need the patch to remove criteria | 12:41 |
rlandy | it will promote on the next round | 12:41 |
rlandy | every two hours | 12:41 |
chandankumar | rlandy, aye | 12:43 |
rlandy | chandankumar: if this passes, I'll run the rest of the tests | 12:43 |
rlandy | and we will get a promotion to promoted-components | 12:43 |
rlandy | let's discuss more at scrum | 12:43 |
amoralej | rlandy, that patch has any "trick" applied? | 12:49 |
amoralej | or it's regular component promotion job? | 12:49 |
weshay|ruck | chandankumar, rlandy marios we ready to make podman default for master? https://review.opendev.org/#/c/636441/ | 12:51 |
weshay|ruck | the only thing I can think of is the upgrade job.. which should not block I think | 12:51 |
marios | weshay|ruck: looking (does updates work ok?) | 12:51 |
marios | yeah looks like | 12:52 |
marios | weshay|ruck: even standalone-upgrade green there | 12:52 |
marios | weshay|ruck: but at this point there shouldn't be any errors i mean master/nothing is using docker | 12:52 |
rlandy | amoralej: no tricks :) | 12:53 |
* marios recalls an issue for scenario4 and manila/no docker sthing sthing but that job is also green | 12:53 | |
rlandy | rerunning the other jobs so we can clean promote | 12:54 |
*** jmasud has quit IRC | 12:54 | |
amoralej | ack | 12:54 |
*** jmasud has joined #oooq | 12:56 | |
rlandy | https://review.rdoproject.org/r/#/c/25412/13/.zuul.yaml | 12:57 |
rlandy | chandankumar: amoralej: weshay|ruck ^^ that will run the rest of the network component pipeline jobs so we will get a promotion | 12:58 |
weshay|ruck | :) | 12:58 |
chandankumar | whY the virus color is always RED? | 13:00 |
rlandy | arxcruz|rover: zbr: chandankumar: scrum | 13:01 |
weshay|ruck | zbr, ur going to be sorry you missed | 13:03 |
zbr | joining... | 13:09 |
*** ysandeep is now known as ysandeep|away | 13:11 | |
bogdando | https://review.opendev.org/636441 https://sfw.so/uploads/posts/2009-07/1247041812_4.jpg | 13:12 |
weshay|ruck | https://docs.google.com/document/d/1jhHadIXwGDeR53TAdXTp3Ug8UkjeGhzXltpVFrBZmWY/edit# | 13:12 |
marios | rlandy: did she put you on the federal stop list? ;) | 13:14 |
rlandy | marios: no - but she did ban me from going to the grocery store | 13:14 |
rlandy | so I'll just starve to death | 13:15 |
rlandy | chandankumar: https://review.rdoproject.org/zuul/stream/ab13178e6c464533873d6facda00433a?logfile=console.log - no dice | 13:15 |
*** derekh has quit IRC | 13:26 | |
amoralej | rlandy, it seems it failed with different issue this time? | 13:27 |
rlandy | amoralej: yes - must have passed update | 13:27 |
rlandy | on meeting - will debug afterwards | 13:27 |
*** rascasoft_ has joined #oooq | 13:28 | |
*** rascasoft has quit IRC | 13:28 | |
*** rascasoft_ has quit IRC | 13:28 | |
amoralej | rlandy, ack, yes it passed TASK [tripleo_container_image_prepare : Run tripleo_container_image_prepare this time | 13:29 |
*** jpena|lunch is now known as jpena | 13:32 | |
*** rascasoft has joined #oooq | 13:32 | |
rascasoft | hey folks hi! Just wanted to know if you old fellas are all good, hope this moment passes by quickly, take care folks! | 13:39 |
weshay|ruck | everyone wave to rascasoft 0/ | 13:43 |
apetrich | arxcruz|rover, so I have a patch up https://review.opendev.org/#/c/713873/ for the https://bugs.launchpad.net/tripleo/+bug/1866621 but I don't think automation has caught up with that yet | 13:46 |
openstack | Launchpad bug 1866621 in tripleo "Can't run container mistral_db_sync" [High,Triaged] | 13:46 |
apetrich | rascasoft, o/ | 13:46 |
arxcruz|rover | apetrich: ack | 13:47 |
weshay|ruck | rlandy, chandankumar same error | 13:48 |
weshay|ruck | Error: Transaction check error:\n file /etc/neutron/plugins/networking-ovn/networking-ovn.ini from install of openstack-neutron-common-1:16.0.0-0.20200318094753.8c7ca77.el8.noarch conflicts with file from package python3-networking-ovn-7.0.0-0.20200109131747.eda5d7f.el8.noarch\n stderr_lines | 13:48 |
weshay|ruck | https://logserver.rdoproject.org/25/25325/17/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/42b1497/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz | 13:48 |
*** amoralej is now known as amoralej|lunch | 13:53 | |
marios | * https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-compute-master/d55ae94/job-output.txt | 13:56 |
marios | * 2020-03-18 11:07:29.190587 | primary | "cmd": "set -e\nif [ -d /openstack/venvs/tempest-untagged/bin ];\nthen\n. /openstack/venvs/tempest-untagged/bin/activate\nfi\ntempest run --concurrency 4 --blacklist-file /home/zuul/tempest/etc/tempest_blacklist.txt --whitelist-file /home/zuul | 13:56 |
marios | * https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tempest-master/de4fa04/job-output.txt | 13:56 |
marios | * 2020-03-18 20:24:46.925453 | primary | Data could not be sent to remote host "127.0.0.2". Make sure this host can be reached over ssh: kex_exchange_identification: Connection closed by remote host | 13:56 |
marios | * 2020-03-18 20:24:46.924876 | primary | fatal: [undercloud]: UNREACHABLE! => { | 13:57 |
chandankumar | marios, rlandy https://github.com/ansible/ansible/issues/13401 | 13:59 |
*** udesale_ has quit IRC | 14:00 | |
*** derekh has joined #oooq | 14:00 | |
rlandy | weshay|ruck: chandankumar: no - it's a different issue | 14:06 |
chandankumar | rlandy, sorry what issue? | 14:06 |
rlandy | chandankumar: why scenario010 standalone on network component failed | 14:09 |
chandankumar | rlandy, /me looks at logs | 14:10 |
chandankumar | rlandy, weshay|ruck https://logserver.rdoproject.org/25/25325/18/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/ab13178/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 14:11 |
chandankumar | it is a new issue all girbris lang | 14:12 |
chandankumar | cgoncalves, Helllo mate | 14:12 |
chandankumar | cgoncalves, please have a look we got some issue here https://logserver.rdoproject.org/25/25325/18/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/ab13178/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 14:12 |
chandankumar | cgoncalves, Configure octavia on overcloud | 14:12 |
rlandy | checking the other jobs on that hash | 14:13 |
chandankumar | rlandy, weshay|ruck since new team mates will be joining our next retro, can we put a card for short intro ? | 14:13 |
*** ysandeep|away is now known as ysandeep | 14:13 | |
rlandy | chandankumar: ack - ... | 14:14 |
rlandy | chandankumar: rfolco will be back next week also | 14:14 |
chandankumar | it will help them to know what everyone does in the team | 14:14 |
chandankumar | rlandy, rfoloco is on vacation | 14:14 |
rlandy | so he'll be back running | 14:14 |
chandankumar | rlandy, aye | 14:14 |
rlandy | chandankumar: ack - he has been the whole sprint :) | 14:14 |
chandankumar | rlandy, I will let him know since he is coming on monday :-) | 14:15 |
weshay|ruck | chandankumar, aye | 14:15 |
rlandy | cool | 14:16 |
chandankumar | rlandy, Thank you for doing TC work, It was awesome :-) | 14:16 |
weshay|ruck | rlandy++ | 14:17 |
chandankumar | rlandy++ | 14:17 |
*** amoralej|lunch is now known as amoralej | 14:22 | |
cgoncalves | chandankumar, hey. so, looking at the log, the error comes from Glance. | 14:24 |
*** jmasud has quit IRC | 14:24 | |
cgoncalves | stderr: 'Error finding address for http://192.168.24.1:9292/v2/images/d6f38bda-61a6-41c8-b509-53f0799ee1b2/file: Remote end closed connection without response' | 14:24 |
*** jmasud has joined #oooq | 14:25 | |
chandankumar | rlandy, cgoncalves can we get the bug up? | 14:27 |
*** ysandeep is now known as ysandeep|afk | 14:27 | |
cgoncalves | chandankumar, I can open one, sure | 14:28 |
*** TrevorV has joined #oooq | 14:30 | |
rlandy | chandankumar: wonder if that is legit/repeatable | 14:36 |
rlandy | other network jobs passed - fs010 is still running | 14:37 |
rlandy | chandankumar, marios: (assuming I am not on the do-not-fly list) I have a flight tomorrow morning so I will miss daily sync - will be on in US afternoon | 15:05 |
chandankumar | rlandy, stay safe | 15:06 |
marios | rlandy: ack | 15:07 |
*** skramaja has quit IRC | 15:08 | |
*** holser has quit IRC | 15:08 | |
*** holser has joined #oooq | 15:09 | |
rlandy | https://review.rdoproject.org/r/#/c/25412/ - all passed | 15:11 |
rlandy | chandankumar: rekicked https://review.rdoproject.org/r/#/c/25325/ to be sure | 15:11 |
rlandy | cgoncalves: hi - did you create a bug yet, if not I'll do that now, rekicked ^^ to ensure the error is repro | 15:12 |
marios | weshay|ruck: rlandy: chandankumar: when you get a chance please https://review.opendev.org/#/c/713659/ | 15:15 |
chandankumar | marios, one query in longer term, we are going to move to ansible operator, can we make it as a part of ansible operator? | 15:18 |
chandankumar | weshay|ruck, 66 | 15:18 |
chandankumar | weshay|ruck, ^^ | 15:18 |
rlandy | marios: pl see commnet | 15:18 |
rlandy | comment | 15:18 |
*** ratailor has quit IRC | 15:21 | |
marios | rlandy: ack thanks done | 15:21 |
marios | chandankumar: if you mean move the deployment of free-ipa to use an operator then yes that sounds like a fine idea | 15:22 |
chandankumar | marios, I am thinking forward to redo the work, since we have undercloud deployment working from operator working, we may want to take a look there | 15:23 |
chandankumar | *instead of redoing the work | 15:23 |
chandankumar | *from operator | 15:24 |
*** holser has quit IRC | 15:28 | |
*** rlandy is now known as rlandy|brb | 15:28 | |
*** holser has joined #oooq | 15:29 | |
*** Trevor_V has joined #oooq | 15:37 | |
cgoncalves | rlandy|brb, sorry, I have not yet. I've been in meetings for the past hour | 15:38 |
cgoncalves | go ahead. thank you | 15:39 |
*** TrevorV has quit IRC | 15:40 | |
*** rlandy|brb is now known as rlandy | 15:50 | |
*** bogdando has quit IRC | 15:50 | |
*** ysandeep|afk is now known as ysandeep | 15:52 | |
chandankumar | rlandy, weshay|ruck marios see ya tomorrow, stay safe | 16:02 |
*** chandankumar is now known as raukadah | 16:02 | |
rlandy | raukadah: you too - stay safe | 16:03 |
marios | o/ rlandy | 16:03 |
marios | sorry raukadah ^ | 16:03 |
marios | but also rlandy | 16:03 |
marios | o/ to all the people | 16:03 |
rlandy | marios: hey | 16:03 |
panda | marios: kthxbye | 16:05 |
rlandy | https://review.rdoproject.org/zuul/stream/466beed43e984cdc946cc4edc16da522?logfile=console.log | 16:06 |
rlandy | https://logserver.rdoproject.org/25/25325/19/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/466beed/job-output.txt | 16:12 |
rlandy | raukadah: weshay|ruck: amoralej: ^^ this next run passed deploy and failed in tempest | 16:12 |
rlandy | so we're getting somewhere now | 16:13 |
rlandy | invsetigating | 16:13 |
rlandy | investigating | 16:13 |
amoralej | 2020-03-19 16:05:23 | 2020-03-19 16:05:23.740 128585 CRITICAL tempest [-] Unhandled error: config_tempest.services.base.ServiceError: Request on service 'load-balancer' with url 'http://192.168.24.1:9876' failed with code 500 | 16:14 |
rlandy | amoralej: may be timing | 16:14 |
raukadah | amoralej, meed to check whether we have octavia support or not | 16:14 |
raukadah | in tempestconf | 16:14 |
weshay|ruck | rlandy, ah it worked the second time :) | 16:14 |
amoralej | 2020-03-19 16:07:19.022 26 ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: No 'octavia.api.drivers' driver found, looking for 'ovn': stevedore.exception.NoMatches: No 'octavia.api.drivers' driver found, looking for 'ovn' | 16:15 |
amoralej | not sure if that's an actual issue | 16:15 |
rlandy | what log? | 16:15 |
amoralej | https://logserver.rdoproject.org/25/25325/19/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/466beed/logs/undercloud/var/log/containers/octavia/octavia.log.txt.gz | 16:15 |
amoralej | that's probably expected until we have the package for the new ovn-octavia-provider | 16:16 |
amoralej | it's wip https://review.rdoproject.org/r/#/c/25972/ | 16:17 |
amoralej | but i'd expect we could live with amphora without ovn | 16:17 |
amoralej | cgoncalves, ^ | 16:17 |
amoralej | i mean, without ovn driver in octavia | 16:17 |
amoralej | api_settings.enabled_provider_drivers = {'amphora': 'The Octavia Amphora driver.', 'octavia': 'Deprecated alias of the Octavia Amphora driver.', 'ovn': 'Octavia OVN driver.'} | 16:18 |
*** johnsom has joined #oooq | 16:18 | |
cgoncalves | the OVN provider driver is auto-enabled if deployment is ML2/OVN | 16:18 |
amoralej | mmmm | 16:18 |
amoralej | is there any option in tripleo to disable it? | 16:19 |
cgoncalves | some folks wanted OVN driver enabled out of the box... | 16:19 |
amoralej | yeah, make sense to me | 16:19 |
amoralej | it's just timing issue | 16:19 |
cgoncalves | I wish there was an option | 16:19 |
amoralej | i think it's fine, but *right now* there is no package for ovn-octavia-provider yet | 16:19 |
cgoncalves | is there an ETA for having it? | 16:20 |
amoralej | cgoncalves, https://review.rdoproject.org/r/#/c/25972/ maciej is working on it | 16:20 |
cgoncalves | I see. thanks! | 16:22 |
cgoncalves | I guess/hope it can be merged still this week | 16:22 |
amoralej | cgoncalves, would it work if we install the old networking-ovn there? | 16:23 |
amoralej | mmm | 16:23 |
cgoncalves | amoralej, it would (assuming the driver is still there in the master branch) | 16:23 |
cgoncalves | https://github.com/openstack/networking-ovn/tree/master/networking_ovn/octavia | 16:23 |
cgoncalves | looks like so | 16:24 |
amoralej | https://github.com/openstack/networking-ovn/blob/master/setup.cfg#L61-L62 | 16:24 |
amoralej | it's this, right? | 16:24 |
amoralej | rlandy, raukadah would it make sense to remove this job from promotion criteria until we have the new package? | 16:25 |
amoralej | do we have coverage for non-octavia services in other jobs for network component? | 16:26 |
raukadah | amoralej, I am totally ok with that | 16:26 |
raukadah | weshay|ruck, rlandy ^^ | 16:26 |
rlandy | amoralej: we could ... | 16:26 |
rlandy | it will fail integration though | 16:26 |
rlandy | but we might as well let this content hit integration | 16:26 |
rlandy | we still have https://review.rdoproject.org/r/#/c/25996/ | 16:27 |
weshay|ruck | raukadah, if the rpm passed this time.. yes | 16:27 |
rlandy | weshay|ruck: raukadah: let's do this | 16:27 |
weshay|ruck | was there another change? | 16:27 |
rlandy | https://review.rdoproject.org/r/#/c/25996/ | 16:28 |
rlandy | weshay|ruck: ^^ pls remove your -2 | 16:28 |
rlandy | I will revert as soon as well merge | 16:28 |
rlandy | weshay|ruck: no other chang | 16:28 |
rlandy | previous error was a 500 | 16:28 |
amoralej | weshay|ruck, missing new package in octavia image https://review.rdoproject.org/r/#/c/25972/ | 16:28 |
amoralej | collateral damage of ovn moved to neutron | 16:29 |
rlandy | and the ETA on that is this week?? | 16:29 |
amoralej | will also need patch in kolla btw | 16:29 |
amoralej | i push quite much to ovn team | 16:29 |
rlandy | we're failing everything now in integration - my vote is to let it go | 16:29 |
rlandy | and deal with OVN | 16:29 |
rlandy | much smaller set of failures | 16:29 |
amoralej | it's in their shoulders, i think they need an upstream patch too | 16:30 |
rlandy | I +2'ed https://review.rdoproject.org/r/#/c/25996/ | 16:30 |
weshay|ruck | amoralej, ah. that is something update won't handle :) | 16:30 |
amoralej | https://review.opendev.org/#/c/713685/ | 16:30 |
amoralej | weshay|ruck, no, not update case :) | 16:30 |
rlandy | I assume we don't need another bug? | 16:30 |
amoralej | net new package | 16:30 |
rlandy | this is expected | 16:30 |
amoralej | workaround would be to disable ovn provider | 16:30 |
amoralej | which i'm not sure if it's possible | 16:31 |
amoralej | in octavia | 16:31 |
amoralej | apparently not | 16:31 |
amoralej | beagles, ^ | 16:32 |
johnsom | You can in the Octavia.conf (upstream it is not enabled until that driver is installed). Maybe this is a limitation of the configuration tooling? | 16:32 |
amoralej | johnsom, what option in octavia.conf? | 16:32 |
johnsom | amoralej https://github.com/openstack/octavia/blob/master/etc/octavia.conf#L47 | 16:33 |
*** dsneddon_ has joined #oooq | 16:38 | |
amoralej | there is no option to disable it in puppet-tripleo | 16:42 |
*** dsneddon_ has quit IRC | 16:44 | |
* marios out see you tomorrow folks | 16:47 | |
marios | enjoy day off those that chose friday ;) | 16:47 |
beagles | correct we auto-enable the driver agent and setup the config when deploying ovn | 16:50 |
beagles | amoralej, johnsom ^ | 16:50 |
weshay|ruck | amoralej, going to try w/ just changing ansible first https://review.rdoproject.org/r/26009 | 16:51 |
weshay|ruck | amoralej, feel free to update that | 16:51 |
amoralej | weshay|ruck, ok | 16:51 |
beagles | cgoncalves, ^ you should probs be aware of this as well | 16:51 |
*** marios is now known as marios|out | 16:51 | |
cgoncalves | beagles, your msg of ovn being auto-enabled or weshay|ruck msg? | 16:52 |
amoralej | weshay|ruck, i think https://bugs.launchpad.net/tripleo/+bug/1867608 is unrelated to ceph-ansible but good to test | 16:52 |
openstack | Launchpad bug 1867608 in tripleo "Error in keystone resources in tripleo-ansible with ansible 2.9" [Undecided,New] | 16:52 |
beagles | cgoncalves, do we need a heat parameter to allow us to enable/disable this so instead of removing scenario010 from voting or knocking it out of periodic/promotion when the ovn provider breaks stuff | 16:53 |
beagles | cgoncalves, just that its currently messed really | 16:53 |
beagles | cgoncalves, I assume you already know we automagically enable the driver agent if ovn | 16:53 |
beagles | ;) | 16:53 |
cgoncalves | beagles, yeah, I know. I commented earlier :) | 16:53 |
* beagles scrolls up | 16:54 | |
beagles | whoops sorry | 16:54 |
beagles | doh | 16:54 |
*** jmasud has quit IRC | 16:54 | |
amoralej | weshay|ruck, i'll add more jobs to your review | 16:54 |
cgoncalves | hmm. parameter to disable auto-enabled providers like amphora and now ovn makes sense | 16:54 |
cgoncalves | like if someone is on a commercial LB solution, probably doesn't care about amphora and ovn | 16:55 |
beagles | cgoncalves, true | 16:55 |
beagles | cgoncalves, it would also be an option for the case we were discussing earlier this week | 16:55 |
beagles | s/option/solution | 16:55 |
*** jmasud has joined #oooq | 16:56 | |
cgoncalves | correct, even though I can think of one person that is strongly against taking that path even if temporary | 16:56 |
beagles | :) | 16:56 |
beagles | regardless of context - it seems like it might be a mechanism that would be useful in times like this - we'd just tweak scenario environment file | 16:57 |
cgoncalves | +1 | 16:59 |
rlandy | what's the timeframe on ^^? | 17:00 |
rlandy | we're deciding whether to let the current code hit the integration pipeline | 17:01 |
beagles | rlandy, I'm working on it now.. should have a patch proposed in the next hour or so | 17:01 |
rlandy | beagles: great - we can test it as soon as you have it | 17:02 |
*** marios|out has quit IRC | 17:07 | |
weshay|ruck | arxcruz|rover, you have to follow up your own patches man https://review.opendev.org/#/c/713676/ | 17:08 |
arxcruz|rover | weshay|ruck: my bad, I saw the error, but got distracted by something else | 17:08 |
arxcruz|rover | :/ | 17:08 |
weshay|ruck | arxcruz|rover, do a check-rdo | 17:08 |
rlandy | periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset030-master - why is that job waiting? | 17:26 |
*** rascasoft has quit IRC | 17:29 | |
rlandy | arxcruz|rover: weshay|ruck: did zuul have a stroke on rdo? | 17:30 |
rlandy | 17 hr 20 | 17:30 |
rlandy | waiting for fs030 to kick' | 17:30 |
weshay|ruck | http://dashboard-ci.tripleo.org/d/cockpit/cockpit?orgId=1&fullscreen&panelId=399 | 17:31 |
* weshay|ruck looks at rdo | 17:31 | |
weshay|ruck | ah.. the periodic job | 17:32 |
rlandy | weshay|ruck: asking on #rhos-ops | 17:33 |
rlandy | it's hanging there | 17:33 |
rlandy | I can't tell why from our view | 17:34 |
amoralej | weshay|ruck, scenario001 fails because ceph-ansible desn't support 2.8: | 17:51 |
amoralej | 2020-03-19 17:38:21 | - 'fatal: [standalone]: FAILED! => changed=false ' | 17:51 |
amoralej | 2020-03-19 17:38:21 | - ' msg: Ansible version must be 2.8!' | 17:51 |
amoralej | the others are failing with same issue | 17:52 |
sshnaidm | raukadah, fyi https://github.com/containers/libpod/pull/5449 | 17:55 |
amoralej | same issue == https://bugs.launchpad.net/tripleo/+bug/1867608 | 17:55 |
openstack | Launchpad bug 1867608 in tripleo "Error in keystone resources in tripleo-ansible with ansible 2.9" [Undecided,New] | 17:55 |
*** derekh has quit IRC | 18:05 | |
weshay|ruck | rlandy, https://review.opendev.org/#/c/712665/ | 18:15 |
*** jpena is now known as jpena|off | 18:16 | |
*** holser has quit IRC | 18:18 | |
*** dsneddon_ has joined #oooq | 18:19 | |
rlandy | fails on openstack/tripleo-heat-templates only | 18:20 |
*** dsneddon has quit IRC | 18:20 | |
*** dsneddon has joined #oooq | 18:21 | |
*** dsneddon has quit IRC | 18:22 | |
*** dsneddon has joined #oooq | 18:22 | |
*** dsneddon has quit IRC | 18:22 | |
*** dsneddon has joined #oooq | 18:22 | |
*** holser has joined #oooq | 18:39 | |
zbr | https://review.opendev.org/#/c/713963/ | 18:55 |
beagles | rlandy, this is the patch I'm testing atm https://review.opendev.org/#/c/713960/ | 19:09 |
beagles | rlandy, so far I've only been able to pep8 it so it might bomb miserably | 19:09 |
beagles | (having some trouble getting an env up and running) | 19:09 |
*** dsneddon has quit IRC | 19:15 | |
*** dsneddon has joined #oooq | 19:16 | |
rlandy | beagles: let's give it a try | 19:27 |
rlandy | as a depends on | 19:27 |
*** holser has quit IRC | 19:32 | |
*** holser has joined #oooq | 19:33 | |
*** sshnaidm is now known as sshnaidm|off | 19:33 | |
*** Trevor_V has quit IRC | 19:41 | |
rlandy | beagles: this is the deploy failure the test returned so far: https://logserver.rdoproject.org/25/25325/20/check/periodic-tripleo-ci-centos-8-scenario010-standalone-network-master/fcc7285/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz | 20:00 |
rlandy | https://review.rdoproject.org/r/#/c/25325/ | 20:00 |
rlandy | ^^ testproject review | 20:00 |
rlandy | feel free to rekick it if you want to test another change to that patch | 20:00 |
beagles | rlandy, thanks! | 20:08 |
*** chem has quit IRC | 20:09 | |
rlandy | weshay|ruck: https://review.rdoproject.org/r/26015 Dlrn-role condition should be switched on os version | 20:11 |
rlandy | ^^ quick review | 20:12 |
rlandy | for bm jobs | 20:12 |
*** TrevorV has joined #oooq | 20:12 | |
*** chem has joined #oooq | 20:13 | |
*** TrevorV has quit IRC | 20:18 | |
*** jmasud has quit IRC | 20:54 | |
*** jmasud has joined #oooq | 20:56 | |
*** dsneddon_ has quit IRC | 20:57 | |
weshay|ruck | rlandy, added +2 | 20:59 |
rlandy | weshay|ruck: thanks ... | 21:01 |
rlandy | baremetal in play ... | 21:01 |
rlandy | https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/5da6ff2792784407916407ef0ef0f025?logfile=console.log | 21:01 |
weshay|ruck | reallly.. nice | 21:01 |
rlandy | let's see how far we get | 21:01 |
beagles | rlandy, I have to step away for a bit - I'm getting stumbled by some heat template syntax atm but will get bak at it later | 21:40 |
*** holser has quit IRC | 21:43 | |
rlandy | beagles: no worries | 21:47 |
rlandy | if we need to, we can skip this test and merge the content | 21:48 |
*** jfrancoa has quit IRC | 22:01 | |
*** holser has joined #oooq | 22:11 | |
*** holser has quit IRC | 22:12 | |
*** holser has joined #oooq | 22:13 | |
*** dpawlik has quit IRC | 22:38 | |
*** jbadiapa has quit IRC | 22:50 | |
*** apetrich has quit IRC | 22:50 | |
*** holser has quit IRC | 23:04 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!