| *** florianf has joined #oooq | 00:01 | |
| *** florianf has quit IRC | 00:08 | |
| *** rlandy has quit IRC | 00:08 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/560445 | 00:41 |
|---|---|---|
| *** agopi|zzz has joined #oooq | 01:19 | |
| *** hamzy has joined #oooq | 02:00 | |
| *** hamzy has quit IRC | 02:29 | |
| *** hamzy has joined #oooq | 02:36 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/560445 | 02:41 |
| *** gkadam has joined #oooq | 02:51 | |
| *** gkadam has quit IRC | 02:58 | |
| *** gkadam has joined #oooq | 03:01 | |
| *** ykarel|away has joined #oooq | 03:38 | |
| *** ykarel|away has quit IRC | 03:57 | |
| *** holser__ has joined #oooq | 04:17 | |
| *** holser__ has quit IRC | 04:22 | |
| *** skramaja has joined #oooq | 04:36 | |
| *** jaganathan has joined #oooq | 04:39 | |
| *** jaganathan has quit IRC | 04:40 | |
| *** jaganathan has joined #oooq | 04:40 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/560445 | 04:41 |
| *** marios has joined #oooq | 04:44 | |
| *** ykarel|away has joined #oooq | 04:47 | |
| *** ykarel|away is now known as ykarel | 04:48 | |
| *** links has joined #oooq | 05:04 | |
| *** links has quit IRC | 05:08 | |
| *** links has joined #oooq | 05:13 | |
| *** links has quit IRC | 05:13 | |
| *** pgadiya has joined #oooq | 05:26 | |
| *** pgadiya has quit IRC | 05:26 | |
| *** ratailor has joined #oooq | 05:28 | |
| *** quiquell|off is now known as quiquell|rover | 05:31 | |
| *** agopi|zzz has quit IRC | 05:44 | |
| *** udesale_ has joined #oooq | 05:44 | |
| *** udesale_ has quit IRC | 05:44 | |
| quiquell|rover | Damn just fs020 prevent us to have master/queens promotion :-/ | 05:46 |
| *** jfrancoa has joined #oooq | 05:48 | |
| weshay | quiquell|rover, was looking at that :) | 05:49 |
| weshay | quiquell|rover, I may skip it for master | 05:49 |
| weshay | quiquell|rover, 2018-04-27 01:33:01 | "Error: Could not set 'file' on ensure: No such file or directory - /etc/keepalived/keepalived.conf20180427-8265-s5p98d.lock", | 05:50 |
| quiquell|rover | Force a promotion ? | 05:50 |
| weshay | that seems transient | 05:50 |
| weshay | however it may be worth running a recreate on that job | 05:50 |
| *** jfrancoa has quit IRC | 05:50 | |
| weshay | maybe I'll kick one off | 05:50 |
| quiquell|rover | have you check queens too to ensure that it the same ? | 05:51 |
| weshay | quiquell|rover, not yet.. will look now | 05:51 |
| weshay | hrm.. quiquell|rover the queens job timed out it looks like | 05:52 |
| quiquell|rover | at te-broker ? let me check | 05:53 |
| weshay | ya.. 143 https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/4fe481e/console.txt.gz#_2018-04-27_05_17_06_965 | 05:53 |
| weshay | quiquell|rover, no.. it failed in the overcloud deploy | 05:53 |
| quiquell|rover | masters too | 05:54 |
| quiquell|rover | Let's see if it's the vxlan ping | 05:54 |
| weshay | quiquell|rover, aye.. good to check.. and FYI from time to time I'll consider bypassing fs20 because it's not used in the upstream check or gate jobs | 05:56 |
| weshay | and it's good to refresh everyone's packages | 05:57 |
| quiquell|rover | pikes and ocata too ? | 05:58 |
| weshay | quiquell|rover, I can look, but n-2 and n-3 are not as urgent :) | 05:58 |
| quiquell|rover | no problem pike is running now | 05:59 |
| quiquell|rover | Let's do a master/queen promotion ? | 06:00 |
| quiquell|rover | Before periodic kicks in ? | 06:00 |
| *** jfrancoa has joined #oooq | 06:01 | |
| quiquell|rover | But fs020 has been failing consistently since last succcess | 06:01 |
| quiquell|rover | 22th | 06:01 |
| quiquell|rover | A lot of days to not test and break things | 06:01 |
| weshay | quiquell|rover, ok.. /me takes a look | 06:03 |
| ykarel | some failures in fs20 are due to infra issues | 06:03 |
| *** matbu has quit IRC | 06:03 | |
| ykarel | latest one failure have a fix upstream | 06:03 |
| weshay | ykarel, master? | 06:03 |
| quiquell|rover | ykarel: the puppet one ? | 06:03 |
| ykarel | so i think it's good to have a promotion to refresh patches | 06:03 |
| ykarel | weshay, yes master | 06:03 |
| weshay | ykarel, which review? | 06:03 |
| ykarel | quiquell|rover, keeplived one | 06:03 |
| ykarel | weshay, ^^ | 06:04 |
| quiquell|rover | Do you have the review around ? | 06:04 |
| weshay | ya /me looks for the patch | 06:04 |
| weshay | https://bugs.launchpad.net/tripleo/+bug/1767135 | 06:04 |
| openstack | Launchpad bug 1767135 in tripleo "Missing keepalived package breaks nonha deployments" [High,In progress] - Assigned to Alex Schultz (alex-schultz) | 06:04 |
| weshay | https://review.openstack.org/#/c/564531/ | 06:04 |
| ykarel | yup this one | 06:04 |
| weshay | https://review.openstack.org/#/c/564531/1/elements/overcloud-controller/install.d/package-installs-overcloud-controller | 06:05 |
| ykarel | and the patch is in gate, and last run it resets so job restarted | 06:05 |
| weshay | ykarel, ugh what issue reset the gate? | 06:06 |
| ykarel | weshay, not checked that | 06:06 |
| weshay | we may want to consider a non-ha job.. the patch that broke us was https://review.openstack.org/#/c/563608/ | 06:07 |
| ykarel | and don't know where to look | 06:07 |
| ykarel | yeah good to have a preventive action | 06:07 |
| weshay | was http://logs.openstack.org/31/564531/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/e30abff/ | 06:08 |
| weshay | ykarel, you can find the history of the update check job http://cistatus.tripleo.org/ | 06:11 |
| ykarel | weshay, what to check in -updates job | 06:12 |
| weshay | ykarel, looks like that failure has only happened twice in the last 50 or so runs | 06:12 |
| ykarel | ohh you mean this reset the gate? | 06:13 |
| weshay | ykarel, /home/stack/http://logs.openstack.org/31/564531/1/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/e30abff/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz | 06:13 |
| ykarel | ahh Error while pulling image | 06:14 |
| weshay | ykarel, I think a package update | 06:15 |
| weshay | oh no.. ur right | 06:16 |
| weshay | u'fatal: [centos-7-inap-mtl01-0003754634]: FAILED! => {"changed": true, "cmd": ["docker", "pull", "192.168.24.1:8787/tripleomaster/centos-binary-mariadb:5cba8063c232a8fbc5a44abc0322bb689a08d3e1_310b64d1"], "delta": "0:00:00.512044", "end": "2018-04-26 17:06:29.078333", "failed": true, "msg": "non-zero return code", "rc": 1, "start": "2018-04-26 17:06:28.566289", "stderr": "Error while pulling image: Get http://192.168.24.1:8787/v1/repositori | 06:16 |
| weshay | es/tripleomaster/centos-binary-mariadb/images: dial tcp 192.168.24.1:8787: getsockopt: connection refused", "stderr_lines": ["Error while pulling image: Get http://192.168.24.1:8787/v1/repositories/tripleomaster/centos-binary-mariadb/images: dial tcp 192.168.24.1:8787: getsockopt: connection refused"], "stdout": "Trying to pull repository 192.168.24.1:8787/tripleomaster/centos-binary-mariadb ... \\nPulling repository 192.168.24.1:8787/tripleo | 06:16 |
| weshay | master/centos-binary-mariadb", "stdout_lines": ["Trying to pull repository 192.168.24.1:8787/tripleomaster/centos-binary-mariadb ... ", "Pulling repository 192.168.24.1:8787/tripleomaster/centos-binary-mariadb"]}', | 06:16 |
| *** marios has quit IRC | 06:16 | |
| weshay | quiquell|rover, ok.. so two attempts to create a stack in rdo-cloud have failed.. any known issues w/ rdo-cloud atm? | 06:17 |
| quiquell|rover | Not that we know off | 06:17 |
| *** marios has joined #oooq | 06:17 | |
| quiquell|rover | The only thing was the vxlang ping, that sometimes fails | 06:17 |
| quiquell|rover | Also | 06:18 |
| quiquell|rover | We show some timeouts at te-broker | 06:18 |
| weshay | k.. | 06:19 |
| quiquell|rover | Also manual cleaning of stacks on personal cloouds having timeouts | 06:19 |
| quiquell|rover | Gooing to do a cleanup just in case of the nodepool | 06:19 |
| quiquell|rover | weshay: How important is fs021 ? | 06:19 |
| *** saneax-_-|AFK is now known as saneax | 06:21 | |
| ykarel | weshay, quiquell|rover so considering promotion without fs020? | 06:29 |
| quiquell|rover | ykarel: master coming | 06:32 |
| weshay | ykarel, panda|ruck|off quiquell|rover yes.. but just for master | 06:32 |
| ykarel | weshay, ack | 06:32 |
| ykarel | queens looks good, can be promoted in next run | 06:33 |
| ykarel | so we are just left with pike | 06:33 |
| weshay | yes.. and tbh.. n-2 going red is not something anyone should get too stressed about | 06:34 |
| weshay | master and n-1 are more important | 06:34 |
| ykarel | ack, but atleast to take care once it passes n days(don't know what n is) | 06:35 |
| weshay | ykarel, they go red after 7 days | 06:36 |
| weshay | by default | 06:36 |
| ykarel | Okk | 06:36 |
| weshay | maybe I'll adjust that for older releases | 06:36 |
| *** holser__ has joined #oooq | 06:37 | |
| *** holser__ has quit IRC | 06:39 | |
| *** holser__ has joined #oooq | 06:40 | |
| *** holser__ has quit IRC | 06:41 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/560445 | 06:41 |
| weshay | quiquell|rover, ykarel hrm.. we have a tox job failure on stable/ocata | 06:44 |
| weshay | that should be looked at.. | 06:44 |
| weshay | quiquell|rover, ykarel thanks for the help, I'm going to jump on my flight now.. Have a good day and weekend | 06:44 |
| weshay | will catch up with guys on Monday | 06:45 |
| ykarel | weshay, link? | 06:45 |
| ykarel | weshay, np | 06:45 |
| ykarel | weshay, Have a safe trip | 06:45 |
| quiquell|rover | weshay: Ok, have a good trip (not acid one) | 06:45 |
| weshay | http://logs.openstack.org/91/564291/2/check/openstack-tox-pep8/4aaf3c5/ | 06:45 |
| ykarel | looks like i have seen it in master as well | 06:46 |
| ykarel | and should not be a problem | 06:46 |
| quiquell|rover | Promoting http://38.145.34.55/master.log | 06:48 |
| *** quiquell|rover is now known as quique|rover|afk | 06:49 | |
| ykarel | so tox failure is in queens as well https://review.openstack.org/#/c/560452/, and is GATE-CHEck patch specific | 06:49 |
| *** yolanda__ is now known as yolanda | 06:59 | |
| *** jtomasek has joined #oooq | 07:15 | |
| *** ccamacho has quit IRC | 07:17 | |
| *** ccamacho has joined #oooq | 07:18 | |
| *** kopecmartin has joined #oooq | 07:21 | |
| *** jaosorior has joined #oooq | 07:21 | |
| *** tesseract has joined #oooq | 07:27 | |
| *** saneax is now known as saneax-_-|AFK | 07:31 | |
| *** quique|rover|afk is now known as quiquell|rover | 07:40 | |
| *** holser__ has joined #oooq | 07:40 | |
| *** kopecmartin has quit IRC | 07:43 | |
| *** gkadam has quit IRC | 07:43 | |
| *** holser__ has quit IRC | 07:44 | |
| *** holser__ has joined #oooq | 07:45 | |
| *** bogdando has joined #oooq | 07:47 | |
| *** ykarel is now known as ykarel|away | 07:47 | |
| ykarel|away | so all ocata jobs passed, it would also be promoting | 07:48 |
| *** amoralej|off is now known as amoralej | 07:52 | |
| *** tesseract has quit IRC | 07:53 | |
| *** tesseract has joined #oooq | 07:56 | |
| *** lucas-afk is now known as lucasagomes | 07:58 | |
| quiquell|rover | niiice... | 08:01 |
| quiquell|rover | btw do you know what's the issue here ? | 08:01 |
| quiquell|rover | https://logs.rdoproject.org/75/564475/3/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Zd66ddac4c1384ba2a91caaee110cc261/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 08:01 |
| quiquell|rover | I think I am tired and not able to find it | 08:01 |
| quiquell|rover | %gatestatus | 08:01 |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/560445 | 08:01 |
| *** gkadam has joined #oooq | 08:07 | |
| *** tosky has joined #oooq | 08:10 | |
| quiquell|rover | ykarel|away: We have master promotion !!! | 08:28 |
| quiquell|rover | https://dashboards.rdoproject.org/master | 08:28 |
| quiquell|rover | and ocata | 08:29 |
| *** saneax-_-|AFK is now known as saneax | 08:35 | |
| arxcruz | panda|ruck|off: sshnaidm|off https://review.openstack.org/#/c/564429 | 08:39 |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/560445 | 08:41 |
| quiquell|rover | arxcruz: That's why sendmail is not working in tempest ? | 08:42 |
| arxcruz | quiquell|rover: one of the reasons, the other reason is the service that send the email isn't running, but it's already configured, I just did a few tests and found this problem | 08:43 |
| arxcruz | so in order to proper work, this patch is needed :) | 08:43 |
| quiquell|rover | There is no bug for it that can we close ? | 08:44 |
| arxcruz | quiquell|rover: not that I am aware | 08:44 |
| quiquell|rover | ok | 08:46 |
| arxcruz | I just notice this and when I tried to bring back the tempest-sendmail service | 08:47 |
| arxcruz | so i didn't open a but | 08:47 |
| arxcruz | bug* | 08:47 |
| quiquell|rover | Cool col | 08:47 |
| *** kopecmartin has joined #oooq | 08:48 | |
| quiquell|rover | you have my friendly +1 | 08:49 |
| *** dtantsur|afk is now known as dtantsur | 08:55 | |
| *** florianf has joined #oooq | 08:57 | |
| *** florianf has quit IRC | 09:21 | |
| *** panda|ruck|off is now known as panda|ruck | 09:21 | |
| *** zoli is now known as zoli|lunch | 09:29 | |
| quiquell|rover | Do we have a Related-Bug at zuul ? | 09:32 |
| quiquell|rover | Or it's Relate-Bug ? | 09:32 |
| arxcruz | quiquell|rover: yes, there's a Related-Bug: XXXXXXXXX | 09:36 |
| panda|ruck | quiquell|rover there's also Partial-Bug | 09:38 |
| quiquell|rover | only the Close-Bug appears at launchpad ? | 09:54 |
| quiquell|rover | panda|ruck: https://review.openstack.org/#/c/564475/ +2 +1w | 09:55 |
| quiquell|rover | please | 09:55 |
| quiquell|rover | the vxlan thing | 09:55 |
| panda|ruck | ah, yeah, I was wondering where it went | 09:58 |
| panda|ruck | quiquell|rover: I'd prefer you put Partial-bug instead of closess bug, it doesn't solve it for rdocloud | 09:58 |
| panda|ruck | well, for the nodepool part of rdocloud | 09:59 |
| panda|ruck | even if it seems to be intermittent | 09:59 |
| quiquell|rover | panda|ruck: What do you mean ? | 10:00 |
| panda|ruck | quiquell|rover: we saw this error in bot reproduced installation and in the periodic job. This patch solves it only for the reproducer part, but the bus is also for the promoter part, even if no other job failed in that way today. | 10:03 |
| quiquell|rover | Also it's impossible to really reproduce it at a periodic. | 10:04 |
| quiquell|rover | Agree with the Partial | 10:04 |
| panda|ruck | quiquell|rover: a change in the commint doesn't need another +2, so as soon as you change it, I can approve | 10:05 |
| quiquell|rover | panda|ruck: Updated | 10:06 |
| quiquell|rover | Damn I have lose the +2 from Wes | 10:06 |
| quiquell|rover | :-/ | 10:06 |
| panda|ruck | quiquell|rover: ok, sorry to not have warned youm you could have use the gerrit edit, and you would not have lost anything. | 10:08 |
| quiquell|rover | Na no problem I will move it to get another +2 | 10:08 |
| panda|ruck | uhm, master promoted ? | 10:09 |
| panda|ruck | when ? | 10:09 |
| quiquell|rover | Yep, Wes was only | 10:09 |
| quiquell|rover | Only fs020 was failing and there is already a patch for it | 10:09 |
| quiquell|rover | also fs020 is not used for check or gate upstream | 10:09 |
| panda|ruck | yeah but it wasn't merged into that hash ... | 10:10 |
| quiquell|rover | So he decide to skip fs020 and promote master and also skip it for queens | 10:10 |
| panda|ruck | so we have a hash with a problem that may cause gates to fail ? | 10:10 |
| quiquell|rover | Only for fs020 | 10:10 |
| panda|ruck | happy weekend! | 10:10 |
| quiquell|rover | It's not going to block check or gates | 10:11 |
| panda|ruck | why not ? | 10:11 |
| quiquell|rover | becouse fs020 is not there | 10:11 |
| quiquell|rover | and only fs020 is affected | 10:12 |
| panda|ruck | it affects all OVB nonha deployments | 10:12 |
| *** dtantsur is now known as dtantsur|brb | 10:13 | |
| quiquell|rover | Going to re-check OVB gates | 10:15 |
| quiquell|rover | But non was affected by it | 10:15 |
| quiquell|rover | panda|ruck: blue jeans ? | 10:18 |
| *** apetrich has quit IRC | 10:19 | |
| panda|ruck | quiquell|rover: I'm in my pyjamas | 10:20 |
| quiquell|rover | Ok, I am not prepare for that | 10:21 |
| quiquell|rover | You still have some dignity in my mind | 10:21 |
| panda|ruck | quiquell|rover: what, I have still some dignity to your eyes ? | 10:22 |
| quiquell|rover | panda|ruck: I am a newbie :-P | 10:22 |
| panda|ruck | quiquell|rover: ok, bj, but I'll stay naked for the whole meeting. | 10:23 |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291, master: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/560445 | 10:41 |
| *** zoli|lunch is now known as zoli | 10:52 | |
| quiquell|rover | panda|ruck: Got the +2 https://review.openstack.org/#/c/564475/ | 10:54 |
| quiquell|rover | Give it a +1w | 10:54 |
| panda|ruck | arxcruz: chandankumar approved 561920 do you need some other reviews? | 10:59 |
| quiquell|rover | panda|ruck: also re-activating fs020 at masters https://review.rdoproject.org/r/#/c/13572/ | 10:59 |
| panda|ruck | quiquell|rover: approved | 10:59 |
| quiquell|rover | cool | 11:00 |
| quiquell|rover | Also the fs020 to reactivate it this weekend | 11:00 |
| arxcruz | panda|ruck: https://review.openstack.org/#/c/562992/ and https://review.openstack.org/#/c/564429/ | 11:02 |
| quiquell|rover | Humm some knows a problem with collect logs like this ? | 11:05 |
| quiquell|rover | AttributeError: 'Blueprint' object has no attribute 'json_encoder' | 11:05 |
| quiquell|rover | [2018-04-27 09:21:23,193] ERROR in app: Unexpected status '500 INTERNAL SERVER ERROR' on URL /reports/ajax/parameters/565dcecb-a952-415e-ba7d-16521eb582eb.txt | 11:05 |
| *** dtantsur|brb is now known as dtantsur | 11:06 | |
| *** adarazs has quit IRC | 11:13 | |
| chandankumar | panda|ruck: there are lots in quee | 11:21 |
| chandankumar | panda|ruck: https://review.openstack.org/#/c/560374/ -> https://review.openstack.org/#/c/554522/ | 11:22 |
| *** ykarel|away is now known as ykarel | 11:22 | |
| chandankumar | second set | 11:22 |
| chandankumar | https://review.openstack.org/#/c/560006/ -> https://review.openstack.org/#/c/559281/ | 11:22 |
| ykarel | quiquell|rover, cool | 11:24 |
| chandankumar | panda|ruck: https://review.openstack.org/#/c/564675/ and https://review.openstack.org/#/c/564677/ | 11:26 |
| chandankumar | panda|ruck: total 6 patches | 11:26 |
| panda|ruck | ykarel: ever seen this ? https://bugs.launchpad.net/tripleo/+bug/1767343 | 11:30 |
| openstack | Launchpad bug 1767343 in tripleo "gate tripleo-ci-centos-7-containers-multinode fails to update packages in cron container" [Critical,Triaged] - Assigned to Quique Llorente (quiquell) | 11:30 |
| ykarel | panda|ruck, no | 11:33 |
| panda|ruck | it's failing at the top of the queue, so everything will be rerun again | 11:34 |
| ykarel | ahh | 11:35 |
| *** lucasagomes is now known as lucas-hungry | 11:38 | |
| *** quiquell|rover is now known as quique|rover|lch | 11:40 | |
| panda|ruck | arxcruz: approved https://review.openstack.org/564429, with a bit of historical explanation here https://review.openstack.org/564737 | 11:42 |
| arxcruz | panda|ruck: thanks! | 11:44 |
| panda|ruck | chandankumar: chak-rdo is not working ? | 11:47 |
| panda|ruck | chandankumar: nevermind | 11:50 |
| panda|ruck | chandankumar: reviewed all that I could | 11:56 |
| chandankumar | panda|ruck: cool sir, thanks :-) | 11:58 |
| panda|ruck | myoung: ping me when you're up | 11:59 |
| panda|ruck | myoung: please ping me , oh yeah, like I ping youuuu | 11:59 |
| *** jtomasek has quit IRC | 12:01 | |
| *** jtomasek has joined #oooq | 12:02 | |
| *** panda|ruck is now known as panda|ruck|lunch | 12:08 | |
| *** quique|rover|lch is now known as quiquell|rover | 12:22 | |
| *** trown|outtypewww is now known as trown | 12:23 | |
| chandankumar | panda|ruck|lunch: does the reproducer script work with virt vms? | 12:27 |
| panda|ruck|lunch | chandankumar: that's the topic of this sprint | 12:29 |
| *** atoth has quit IRC | 12:29 | |
| panda|ruck|lunch | chandankumar: so it will work at the end of the sprint, hopefully | 12:29 |
| *** atoth has joined #oooq | 12:29 | |
| chandankumar | panda|ruck|lunch: cool! | 12:29 |
| *** lucas-hungry is now known as lucasagomes | 12:38 | |
| *** rlandy has joined #oooq | 12:39 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 12:41 |
| *** saneax is now known as saneax-_-|AFK | 12:47 | |
| *** ratailor has quit IRC | 12:50 | |
| rlandy | rfolco: myoung: I | 12:51 |
| rlandy | ll follow up on your test boxes today | 12:51 |
| rlandy | haven't heard back from eng-ops yet | 12:52 |
| rfolco | rlandy, yeha... I checked 3 times already no updates | 12:52 |
| rfolco | thanks :) | 12:52 |
| *** skramaja has quit IRC | 12:52 | |
| rlandy | trown: hey - anything more sprint-wise that needs doing/testing? other than "make POC nicer" stuff? | 13:02 |
| *** quiquell|rover is now known as quiquell|roger | 13:02 | |
| quiquell|roger | hehe | 13:02 |
| *** quiquell|roger is now known as quiquell|rover | 13:02 | |
| quiquell|rover | panda|ruck|lunch, myoung: Fixed the problem with the SSL cerificate of jjb | 13:03 |
| *** panda|ruck|lunch is now known as panda|ruck | 13:03 | |
| panda|ruck | quiquell|rover: what happened ? | 13:03 |
| quiquell|rover | wget/curl was working fine without ignoring it | 13:03 |
| quiquell|rover | But I did have to enforce REQUESTS_CA_BUNDLE | 13:04 |
| quiquell|rover | To make python requests work | 13:04 |
| quiquell|rover | Maybe they have update the cert and python doens't get it | 13:04 |
| panda|ruck | quiquell|rover: ok, so you're ingoring the cert or not ? | 13:04 |
| quiquell|rover | panda|ruck: no, just point it to the right now | 13:05 |
| quiquell|rover | right one | 13:05 |
| panda|ruck | quiquell|rover: anything to review ? | 13:05 |
| quiquell|rover | Give me a min | 13:06 |
| quiquell|rover | btw, passing | 13:07 |
| quiquell|rover | https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-environments-jjb-push/67/ | 13:07 |
| trown | rlandy: I think that is it | 13:08 |
| trown | rlandy: getting things merged of course :P | 13:08 |
| panda|ruck | trown: rlandy: I'm thinking about splitting the TOC card into smaller cards, so other people can work on it. We have three days left to at least have set of reviews ready | 13:08 |
| rlandy | yeah - bribe the zuul officials :) | 13:09 |
| rlandy | panda|ruck: yep - was waiting to talk with myoung about that | 13:09 |
| rlandy | I think he has some work donw | 13:09 |
| rlandy | done | 13:09 |
| rlandy | maybe just not committed yet | 13:09 |
| panda|ruck | rlandy: can I merge https://review.openstack.org/#/c/563319/ or you wanted to wait for something ? | 13:10 |
| rlandy | but yeah, I can pick up some of those items | 13:10 |
| rlandy | panda|ruck: I am fine with merging it - my only concern is that the "make the POC nicer" will change some stuff | 13:10 |
| rlandy | and we will have to rework the reproducer | 13:10 |
| panda|ruck | rlandy: I'd prefer to merge it and then deal witht he consequences, the "nice" card is still WIP anyway | 13:11 |
| rlandy | so if we merge, we need another item in that open card to update the reproducer to mathc name changse etc. | 13:11 |
| rlandy | fine | 13:11 |
| panda|ruck | rlandy: approving | 13:11 |
| rlandy | panda|ruck: I just recheck it so zuul can hopefully remove its -1 | 13:12 |
| quiquell|rover | panda|ruck: https://code.engineering.redhat.com/gerrit/137148 | 13:13 |
| quiquell|rover | If I retrigger a green tripleo-environments-jjb-push build | 13:15 |
| quiquell|rover | It will destroy the world ? | 13:15 |
| myoung | panda|ruck, rlandy, o/ | 13:17 |
| myoung | panda|ruck, rlandy, re POC card, I haven't started yet on injecting the key's into VM's...i suspect it's just a VC and done, but haven't probed it yet | 13:17 |
| chandankumar | myoung: https://review.openstack.org/#/c/564677/ it needs your attension | 13:17 |
| myoung | panda|ruck, rlandy, other stuff i was doing some work on last night, will post in a bit | 13:18 |
| myoung | chandankumar: /me looks | 13:18 |
| rlandy | myoung: ok - I have some time today if you want me to work on some items or test | 13:19 |
| panda|ruck | myoung: rlandy let's chat when you have time, soe we can update the card | 13:21 |
| myoung | chandankumar: looks great, I can try things later, I had blocked off time in late afternoon to actually run it following these directions. is QE for the doc card to review the docs, or to actually try all the (newly) documented options? the latter is quite a bit more time, and will need to happen next week... | 13:21 |
| chandankumar | myoung: actual doc will come on monday | 13:22 |
| arxcruz | rfolco: too late on LOT_PATH :D | 13:22 |
| chandankumar | myoung: that one is for validate-tempesr ole | 13:22 |
| chandankumar | myoung: below is the list of things coming in docs https://review.rdoproject.org/etherpad/p/tripleo-tempest-doc | 13:23 |
| chandankumar | if you have any more points to cover, i will add that while writting | 13:23 |
| myoung | panda|ruck, rlandy, can chat now if it's quick...i wanted to blast out some of the actual POC card work :) | 13:23 |
| panda|ruck | myoung: on your channel! | 13:25 |
| rlandy | ok | 13:25 |
| rfolco | arxcruz, ack | 13:25 |
| myoung | panda|ruck: inc | 13:25 |
| *** trown is now known as trown|brb | 13:26 | |
| *** Goneri has joined #oooq | 13:29 | |
| quiquell|rover | panda|ruck: Do we have issues with tripleo-ci-centos-7-scenario001-multinode-oooq-container ? | 13:39 |
| quiquell|rover | I have a timeout executing tempest | 13:40 |
| quiquell|rover | Humm | 13:45 |
| quiquell|rover | Infinite loop at | 13:45 |
| quiquell|rover | 2018-04-27 13:02:28.511 4988 INFO urllib3.poolmanager [-] Redirecting http://192.168.24.11:8004/v1/acee2ddc7945462b8e8beb5e89632bcb/stacks/integration_test -> http://192.168.24.11:8004/v1/acee2ddc7945462b8e8beb5e89632bcb/stacks/integration_test/d2e19298-cad8-48d0-b67a-213ea3fcdc99 | 13:45 |
| quiquell|rover | arxcruz, chandankumar sounds familiar ? | 13:45 |
| myoung | quiquell|rover: the run of your test job pushed current HEAD of tripleo-env --> jenkins, so we now have https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/osp-director-promote-13-puddle, don't need to re-run. | 13:48 |
| quiquell|rover | myoung: Want to re-run just to check the commit in the script | 13:49 |
| myoung | quiquell|rover: thanks for fixing the cert issue! The idea is that jjb-push job can be run at any time, all changes to jobs for that jenkins server are in JJB | 13:49 |
| myoung | quiquell|rover: yup...merging now | 13:49 |
| quiquell|rover | ok | 13:49 |
| myoung | quiquell|rover: in the past we would have lots of "hand edits" to jobs in jenkins via jenkins UI and it became a PITA | 13:49 |
| quiquell|rover | We have to change the config after merged | 13:49 |
| quiquell|rover | There is the concept of "declarative pipelines" in the last jenkins version | 13:49 |
| myoung | so now even if we want to make minor changes to debug jobs, it's much easier to just commit them, can always roll back a review. harder with hand edits :) also can make hand edits and someone pushes JJB, and *poof* they are gone | 13:50 |
| quiquell|rover | myoung: Remove human factor is alwasy a win :-) | 13:50 |
| myoung | quiquell|rover: yes...that's the jenkins pipeline v2 stuff w/ ruby. QE uses this, these jobs all pre-date that, and were based on ci.centos.org RDO jobs with a goal to match what made sense and change what we had to in order to test OSP and RDO in the same way internally | 13:50 |
| quiquell|rover | myoung: Long run is to move everything to zuul ? | 13:51 |
| myoung | quiquell|rover: https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/tripleo-environments-jjb-push/68/ | 13:52 |
| myoung | quiquell|rover: long run is to move things to software factory / zuul, correct (rfolco has WIP, https://trello.com/c/vUzB3qzq/583-create-ci-config-to-run-on-sf) | 13:52 |
| myoung | (internal SF / zuul) | 13:52 |
| quiquell|rover | Then upstream will change to other stuff :-) | 13:53 |
| *** saneax-_-|AFK is now known as saneax | 13:55 | |
| *** trown|brb is now known as trown | 13:56 | |
| myoung | quiquell|rover: as is always the case...but at least rfolco is looking at zuul v3, which (I think?) is a leapfrog, and we're working directly with SF folks...so we might end up ahead of upstream :) | 13:57 |
| myoung | (caveat: i'm not sure offhand about upstream timelines) | 13:57 |
| myoung | rfolco would have a better view i'm sure | 13:58 |
| myoung | quiquell|rover: the test job is good. thanks again! | 13:58 |
| quiquell|rover | myoung: zuul v3 would be sweeet :-), btw first time a read the "leapfrog" concept | 13:58 |
| quiquell|rover | myoung: Going to clean up the config. | 13:58 |
| myoung | quiquell|rover: ahh sorry shouldn't use wierd expressions. leapfrog := kids game (https://www.youtube.com/watch?v=Hj0JBiknaPg) | 13:59 |
| quiquell|rover | myoung: no no use them, it what keep us reading this chat :-) | 13:59 |
| * myoung puts on blinders for an hour. this->DoWork(); | 14:00 | |
| *** apetrich has joined #oooq | 14:00 | |
| *** ratailor has joined #oooq | 14:03 | |
| arxcruz | quiquell|rover: nope | 14:05 |
| quiquell|rover | recheck also runs the gate jobs ? | 14:06 |
| quiquell|rover | I mean gerrit 'recheck' comment | 14:06 |
| *** dtantsur is now known as dtantsur|brb | 14:07 | |
| myoung | rasca: the journey begins... https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/osp-director-promote-13-puddle/1/ | 14:09 |
| *** holser__ has quit IRC | 14:09 | |
| rasca | myoung, WOW the FAIL journey I bet! :D :D :D | 14:09 |
| myoung | rasca: i have zero cycles this week to look at results...just kicked it for the first time now that quiquell|rover fixed the SSL issue | 14:10 |
| myoung | quiquell|rover: could you please (if not already) update IT ticket and let them know we're unblocked? would also be good to understand what changed on their side that caused this (somewhere between 4/10 and yesterday) just for posterity | 14:11 |
| quiquell|rover | myoung: Already did, they also explain why | 14:12 |
| quiquell|rover | Looks like python-jenkins have go to use requests | 14:12 |
| quiquell|rover | as python library | 14:12 |
| myoung | quiquell|rover: cool cool. thx | 14:12 |
| rasca | quiquell|rover, myoung, why https://rhos-dev-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/osp-rhos-13-promote-puddle-get-build/1/console is not finding the build number? | 14:14 |
| quiquell|rover | rasca: I am learning downstream now, so no clue yet | 14:15 |
| * myoung knows and fetches a link | 14:15 | |
| quiquell|rover | panda|ruck: What do we do with this ? | 14:16 |
| quiquell|rover | https://bugs.launchpad.net/tripleo/+bug/1766884 | 14:16 |
| openstack | Launchpad bug 1766884 in tripleo "Gate jobs are failing in different ways related to network issues" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 14:16 |
| myoung | http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/jenkins/jobs/tripleo-quickstart/scripts/trigger-getbuild.sh#n178 | 14:16 |
| myoung | ^^ needs an update (or a total re-write...depending on ambion) | 14:17 |
| myoung | ambition* | 14:17 |
| myoung | rasca, quiquell|rover ^^ | 14:17 |
| rasca | myoung, quiquell|rover, /me prepares a review | 14:18 |
| quiquell|rover | Closing now guys, have a good weekend | 14:19 |
| panda|ruck | quiquell|rover: you too | 14:19 |
| *** quiquell|rover is now known as quiquell|off | 14:19 | |
| panda|ruck | quiquell|rover: I think I\ll set the bug as invalid | 14:19 |
| panda|ruck | quiquell|off: | 14:19 |
| myoung | quiquell|off: ciao! | 14:19 |
| *** agopi|zzz has joined #oooq | 14:23 | |
| *** agopi|zzz is now known as agopi | 14:23 | |
| rasca | myoung, panda|ruck, https://code.engineering.redhat.com/gerrit/#/c/137155/ c'mon folks, put a smile on my face and make rhos-13 work! :D :D :D | 14:23 |
| rasca | it will be the best weekend evah | 14:24 |
| panda|ruck | rasca: I like when you're sad | 14:24 |
| rasca | panda|ruck, you like when *EVERYONE* is sad | 14:25 |
| rasca | panda|ruck|myfavoriteapocalypseknight | 14:25 |
| myoung | rasca: merged. KICK IT | 14:25 |
| rasca | myoung, that's the spirit! | 14:26 |
| rasca | panda|ruck, myoung, https://www.youtube.com/watch?v=YYOKMUTTDdA | 14:26 |
| myoung | rasca: lol, i was about to send you https://www.youtube.com/watch?v=YYOKMUTTDdA | 14:27 |
| myoung | err no...this https://www.youtube.com/watch?v=ZaI2IlHwmgQ | 14:27 |
| * myoung ignores IRC for 30 (or will try to) | 14:27 | |
| rasca | myoung, LOL | 14:28 |
| myoung | rasca: (black eyed peas take on rhos-13 ci-iteration) /me giggles | 14:28 |
| rasca | myoung, Best friday evah :D | 14:29 |
| *** ratailor has quit IRC | 14:31 | |
| *** atoth has quit IRC | 14:36 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 14:41 |
| *** atoth has joined #oooq | 14:51 | |
| *** dtantsur|brb is now known as dtantsur | 14:56 | |
| myoung | panda|ruck, as weshay is travelling shall we postpone bug triage? | 15:00 |
| rlandy | myoung: not sure if I understood the task correctly - https://review.openstack.org/#/c/564796/ - does that look right to you? | 15:00 |
| panda|ruck | myoung: it's just you and me | 15:01 |
| myoung | i thought was tc, ua, pm, ruck, rover | 15:01 |
| panda|ruck | myoung: yes, rover is gone, we is flying, ua is ruck | 15:02 |
| panda|ruck | myoung: so i'ts just you and me | 15:02 |
| panda|ruck | myoung: I'd rather skip | 15:02 |
| myoung | panda|ruck: ack and same | 15:03 |
| myoung | panda|ruck: will reschedule to next week monday/tues | 15:04 |
| panda|ruck | myoung: thanks. | 15:04 |
| myoung | rlandy: I was a little confused at this task as well. currently we do --root-password random (http://libguestfs.org/virt-builder.1.html#users-and-passwords) which generates a random password and prints to stdout, which we register and dump at the end of the playbook if -vv is turned on. | 15:04 |
| myoung | you patch generates a password and injects it...I don't know if the goal was to take a password as an (optional) input/variable?, generating it if not specified? | 15:05 |
| myoung | i wasn't here last week when we made this card, just going on the one-liner in the card | 15:05 |
| d0ugal | Which of the quickstart logs has the register node output? | 15:06 |
| myoung | or if there was an issue with VC's default "random" optionk, such that we need to create a (better/stronger?) password and inject *that* ? | 15:06 |
| rlandy | trown: can you comment on the the root-password requirement for fixing the POC? I did this ... https://review.openstack.org/#/c/564796/ but I am not sure of that matches what was supposed to be | 15:07 |
| rlandy | I am not even sure why we need a root password set | 15:07 |
| myoung | rlandy, trown: I had assumed that the task meant "as a developer I like 'fluffythecat' to be my root password, so I'll specify it when running this playbook, and if I don't specify gen a random password" but not siure | 15:08 |
| * myoung shrugs | 15:08 | |
| trown | rlandy: i should try that out, but that looks reasonable to me | 15:13 |
| trown | myoung: rlandy: I dont think we need a random password fallback, just have a default for vm_pass and we are good... we probably dont really even need a root password, but it is useful if something goes wrong before ssh is up | 15:14 |
| trown | rlandy: found one nitty thing actually, can get rid of the debug lines at the end | 15:15 |
| *** saneax is now known as saneax-_-|AFK | 15:23 | |
| rlandy | fixing | 15:24 |
| *** jfrancoa has quit IRC | 15:25 | |
| rlandy | trown: so debug does no print out the password - just the file path - which is deleted | 15:28 |
| rlandy | not | 15:28 |
| rlandy | review updated | 15:30 |
| *** kopecmartin has quit IRC | 15:31 | |
| *** myoung is now known as myoung|email-unl | 15:32 | |
| *** myoung|email-unl is now known as myoung|emailplz | 15:32 | |
| *** agopi is now known as agopi|brb | 15:34 | |
| *** jaosorior has quit IRC | 15:35 | |
| *** agopi|brb is now known as agopi | 15:38 | |
| agopi | trown, do you happen to know how the jjb clones the git repository which is defined in the job ? | 15:39 |
| agopi | ^ to give more context when cloning browbeat repo https://github.com/openstack/browbeat/blob/master/setup.cfg#L38 no longer is copying files into playbooks directory | 15:39 |
| *** marios has quit IRC | 15:48 | |
| *** kopecmartin has joined #oooq | 15:52 | |
| *** kopecmartin has quit IRC | 15:56 | |
| *** lucasagomes is now known as lucas-afk | 15:58 | |
| *** dtantsur is now known as dtantsur|afk | 16:15 | |
| trown | agopi: do you have logs of the pip install somewhere? | 16:25 |
| trown | agopi: have you tried to pip install locally and check it out? | 16:25 |
| trown | agopi: you could do something like: http://paste.openstack.org/show/720030/ | 16:27 |
| trown | agopi: when I do that, I see the playbooks copied to .venv/playbooks/ ... so I need more context to understand the problem/question | 16:29 |
| agopi | hey trown so when i did a local install it'd copy files to home directory of venv, | 16:29 |
| agopi | yes trown same issue. | 16:29 |
| agopi | i'm looking for logs | 16:29 |
| trown | agopi: my best guess is that it is not actually getting pip installed for some reason... that is the only thing that would make sense | 16:30 |
| trown | agopi: then just have to figure out what that reason is | 16:30 |
| *** ccamacho has quit IRC | 16:35 | |
| *** zoli is now known as zoli|gone | 16:36 | |
| agopi | oh okay thanks trown. | 16:38 |
| *** trown is now known as trown|lunch | 16:40 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 16:41 |
| *** bogdando has quit IRC | 16:55 | |
| *** gkadam has quit IRC | 17:08 | |
| *** tesseract has quit IRC | 17:09 | |
| panda|ruck | trown: before I go for the weekend, do yu need help to finish the snapshotting change ? | 17:15 |
| ykarel | so master and queens are going to be promoted | 17:28 |
| ykarel | panda|ruck, https://review.rdoproject.org/r/#/c/13572/ | 17:28 |
| panda|ruck | ykarel: yep | 17:29 |
| ykarel | we would need 1 more promotion to unblock phase 1 | 17:29 |
| *** brault has quit IRC | 17:30 | |
| panda|ruck | ykarel: we should revert also queens | 17:30 |
| ykarel | queens alreday has fs20 in criteria,no? | 17:36 |
| panda|ruck | ykarel: oh, yeah, I understood that was removed from queens too, nevermin | 17:40 |
| panda|ruck | d | 17:40 |
| ykarel | Okk | 17:41 |
| *** amoralej is now known as amoralej|off | 17:41 | |
| panda|ruck | ykarel: do you see any error here, beside the one from ar a? | 17:41 |
| ykarel | panda|ruck, phase 1? | 17:41 |
| panda|ruck | ykarel: http://logs.openstack.org/19/563319/5/gate/tripleo-ci-centos-7-scenario007-multinode-oooq-container/8ef0cf1/job-output.txt.gz | 17:41 |
| panda|ruck | ykarel: gate job | 17:41 |
| panda|ruck | failing, so another queue reset | 17:42 |
| ykarel | can see only ara issue | 17:44 |
| ykarel | so then all jobs should be affected with ara issue | 17:44 |
| panda|ruck | but they're not ... the failures have lost any sense in the last days | 17:46 |
| ykarel | hmm | 17:47 |
| *** ykarel is now known as ykarel|away | 17:49 | |
| *** trown|lunch is now known as trown | 17:50 | |
| * myoung|emailplz heads to doctor, back in 90 | 17:51 | |
| panda|ruck | trown: before I go for the weekend, do yu need help to finish the snapshotting change ? | 17:53 |
| *** ykarel|away has quit IRC | 17:54 | |
| trown | panda|ruck: no it is just docs... I keep getting distracted from it... aka procrastinating | 17:55 |
| panda|ruck | trown: ok so the rest could be reviewed already. | 17:56 |
| trown | panda|ruck: ya it 100% works as is, and instructions are in the etherpad | 17:57 |
| panda|ruck | trown: ok, if it helps with procrastination, bear in mind taht we have only 2 days and half until the end of the sprint. | 18:00 |
| *** jaganathan has quit IRC | 18:02 | |
| *** panda|ruck is now known as panda|ruck|off | 18:07 | |
| panda|ruck|off | time to go | 18:07 |
| *** holser__ has joined #oooq | 18:22 | |
| *** ykarel|away has joined #oooq | 18:23 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 18:41 |
| *** holser__ has quit IRC | 18:53 | |
| *** myoung|emailplz is now known as myoung | 19:18 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 20:41 |
| rlandy | myoung: hi | 20:49 |
| myoung | rlandy: howdy | 20:49 |
| rlandy | myoung: two reviews added to the card | 20:49 |
| rlandy | let me know what you think | 20:50 |
| rlandy | myoung: anything else for me to do? | 20:50 |
| myoung | rlandy: ack, will have a look tonight | 20:50 |
| rlandy | cool | 20:50 |
| *** trown is now known as trown|outtypewww | 21:01 | |
| *** jtomasek has quit IRC | 21:02 | |
| *** rfolco is now known as rfolco|off | 21:11 | |
| *** ykarel|away has quit IRC | 21:15 | |
| *** rlandy has quit IRC | 22:04 | |
| *** holser__ has joined #oooq | 22:14 | |
| *** agopi is now known as agopi|out | 22:20 | |
| *** Goneri has quit IRC | 22:35 | |
| hubbot | FAILING CHECK JOBS on stable/ocata: openstack-tox-pep8, tripleo-ci-centos-7-undercloud-upgrades @ https://review.openstack.org/564291 | 22:41 |
| *** agopi|out has quit IRC | 22:44 | |
| *** holser__ has quit IRC | 23:03 | |
| *** tosky has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!