Tuesday, 2018-11-20

*** tosky has quit IRC00:14
*** vinaykns has joined #oooq00:44
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario008-multinode-oooq- (1 more message)00:52
*** rlandy is now known as rlandy|bbl00:52
*** dsneddon has quit IRC01:55
*** dsneddon has joined #oooq01:56
*** dsneddon has quit IRC02:26
*** dsneddon has joined #oooq02:27
*** ykarel|away has joined #oooq02:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-centos-7-scenario007-multinode-oooq-container @ https://review.openstack.org/560445, stable/queens: tripleo-ci-centos-7-scenario008-multinode-oooq- (1 more message)02:52
*** ykarel|away has quit IRC02:55
*** dtrainor_ has quit IRC03:09
*** dtrainor has joined #oooq03:10
*** dtrainor has quit IRC03:13
*** vinaykns has quit IRC03:49
*** ykarel|away has joined #oooq03:54
*** dsneddon has quit IRC04:01
*** dsneddon has joined #oooq04:02
*** udesale has joined #oooq04:09
*** dsneddon has quit IRC04:17
*** dsneddon has joined #oooq04:18
*** dsneddon has quit IRC04:25
*** dsneddon has joined #oooq04:26
*** dsneddon has quit IRC04:34
*** dsneddon has joined #oooq04:38
*** dsneddon has quit IRC04:46
*** ykarel|away has quit IRC04:47
*** dsneddon has joined #oooq04:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/queens: tripleo-ci- (1 more message)04:52
*** chandankumar has joined #oooq04:55
*** chandankumar is now known as chkumar|ruck04:56
*** ykarel|away has joined #oooq05:04
*** dsneddon has quit IRC05:11
*** dsneddon has joined #oooq05:16
*** rlandy|bbl is now known as rlandy05:26
*** dtrainor has joined #oooq05:26
*** ratailor has joined #oooq05:46
*** ykarel|away is now known as ykarel05:51
*** dsneddon has quit IRC05:59
*** dsneddon has joined #oooq06:04
*** dsneddon has quit IRC06:12
*** dsneddon has joined #oooq06:15
*** dsneddon has quit IRC06:19
*** dsneddon has joined #oooq06:21
*** saneax has joined #oooq06:22
*** ccamacho has quit IRC06:24
*** dsneddon has quit IRC06:26
*** dsneddon has joined #oooq06:29
*** dsneddon has quit IRC06:34
*** dsneddon has joined #oooq06:36
*** chkumar|ruck has quit IRC06:37
*** rlandy has quit IRC06:39
*** dsneddon has quit IRC06:44
*** dsneddon has joined #oooq06:47
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/queens: tripleo-ci- (1 more message)06:52
*** dsneddon has quit IRC06:53
*** dsneddon has joined #oooq06:55
*** dsneddon has quit IRC06:59
*** apetrich has quit IRC07:00
*** dsneddon has joined #oooq07:01
*** gkadam has joined #oooq07:02
*** dsneddon has quit IRC07:07
*** dsneddon has joined #oooq07:09
*** chandankumar has joined #oooq07:14
*** chandankumar is now known as chkumar|ruck07:14
*** apetrich has joined #oooq07:17
*** dsneddon has quit IRC07:18
chkumar|ruck%gatestatus07:19
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/queens: tripleo-ci- (1 more message)07:19
*** dsneddon has joined #oooq07:19
*** jtomasek has joined #oooq07:23
*** kopecmartin|off is now known as kopecmartin07:25
*** dsneddon has quit IRC07:26
*** dsneddon has joined #oooq07:26
quiquell|offssbarnea|bkp2: good morning, second review here https://tree.taiga.io/project/tripleo-ci-board/task/390?kanban-status=144727607:27
*** quiquell|off is now known as quiquell07:28
quiquellssbarnea|bkp2: Moved https://tree.taiga.io/project/tripleo-ci-board/task/395?kanban-status=1447275 to in progress there is no review tehere07:29
quiquellchkumar|ruck: Good morning, do you have a minute ?07:33
*** ykarel is now known as ykarel|lunch07:35
*** dsneddon has quit IRC07:35
*** dsneddon has joined #oooq07:36
*** skramaja has joined #oooq07:36
*** dsneddon has quit IRC07:44
*** dsneddon has joined #oooq07:44
*** quiquell is now known as quiquell|brb07:46
*** jfrancoa has joined #oooq07:51
*** chandankumar has joined #oooq07:52
*** dsneddon has quit IRC07:53
*** ccamacho has joined #oooq07:54
*** chkumar|ruck has quit IRC07:56
*** chandankumar is now known as chkumar|ruck07:57
*** dsneddon has joined #oooq07:58
*** dsneddon has quit IRC08:03
*** dsneddon has joined #oooq08:04
*** quiquell|brb is now known as quiquell08:06
*** dsneddon has quit IRC08:10
*** dsneddon has joined #oooq08:11
*** holser_ has joined #oooq08:13
*** dsneddon has quit IRC08:16
*** dsneddon has joined #oooq08:18
*** bogdando has joined #oooq08:29
*** dsneddon has quit IRC08:32
*** ykarel|lunch is now known as ykarel08:35
*** dsneddon has joined #oooq08:38
*** amoralej|off is now known as amoralej08:40
*** dsneddon has quit IRC08:45
*** dsneddon has joined #oooq08:48
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/queens: tripleo-ci- (1 more message)08:52
*** tosky has joined #oooq09:11
quiquellmarios: Good morning, have a minute ?09:18
marioso/ quiquell09:21
mariosquiquell: wana use phone?09:21
quiquellmarios: It's just about writting stuff at zuul-in-a-box task, I am very bad at that09:22
quiquellmarios: A user guide would be enough or if more of a DoD ?09:22
mariosquiquell: yeah its not easy as it sounds to write those tasks. imo a really good contribution would be a section in tripleo-docs that explains how you can use this thing for testing.09:23
mariosso that would be a good dod09:23
quiquellmarios: to the DoD is 'add a user guide for this zuul thingyÂÂ' and then put a review in place with the change at docs ?09:25
mariosquiquell: sounds like a good start :)09:28
mariosquiquell: yeah that's it i think09:28
*** jfrancoa has quit IRC09:29
*** derekh has joined #oooq09:37
*** dsneddon has quit IRC09:43
*** dsneddon has joined #oooq09:44
*** jfrancoa has joined #oooq09:47
sshnaidmquiquell, had to add https://github.com/sshnaidm/zuul-in-a-box/blob/master/playbooks/setup.yaml#L25-L26  and ":z" in all mounts: https://github.com/sshnaidm/zuul-in-a-box/blob/master/docker-compose.yaml09:50
sshnaidmquiquell, and it worked \o/09:50
quiquellsshnaidm: Yep I am testing f28 now there09:50
quiquell:-)09:50
sshnaidmquiquell, at least gerrit part09:50
quiquellsshnaidm: you can use zuul autohold for forensics09:51
sshnaidmquiquell, yeah, it's cool feature09:51
quiquellsshnaidm: also activated again the toci_gate_test.sh execution, yuesterday was testing the post stuff09:51
quiquellsshnaidm: super powerful09:51
quiquellsshnaidm: Are you able to run stuff ?09:51
quiquellsshnaidm: where are you now ?09:51
sshnaidmquiquell, I think nodepool connection doesn't work, I don't see my tenant in zuul tenants, will work on this today09:52
quiquellsshnaidm: It takes a while09:53
quiquellsshnaidm: what I doo is start with docker-compose -d --build09:53
quiquellsshnaidm: then docker-compose logs -f --tail=10 scheduler09:53
quiquellsshnaidm: And wait until it's ready09:53
quiquellsshnaidm: Did you manage to upload the image ?09:54
sshnaidmquiquell, going to do it soon09:55
*** dtantsur|afk is now known as dtantsur09:58
quiquellsshnaidm: ack09:59
*** jfrancoa has quit IRC10:00
quiquellsshnaidm: btw if you manage to do the clouds.yaml thing10:00
quiquellsshnaidm: the you can do stuff like openstack --os-cloud=rdo-cloud server list10:00
quiquellsshnaidm: so no need to source stuff10:00
quiquellsshnaidm: you can even do export OS_CLOUD=rdo-cloud and then openstack server list10:00
*** jfrancoa has joined #oooq10:01
quiquellsshnaidm: Is a nice way to have the credentials organize10:01
quiquellmarios: what do you think now ? https://tree.taiga.io/project/tripleo-ci-board/task/348?kanban-status=144727510:04
sshnaidmquiquell, yeah, right10:06
*** chkumar246 has joined #oooq10:13
*** chkumar|ruck has quit IRC10:15
*** chkumar246 is now known as chkumar|ruck10:18
ssbarnea|bkp2sshnaidm: chkumar246 marios : https://review.openstack.org/#/c/618809/ -- separated as requested, covering only flake8 part.10:18
sshnaidmssbarnea|bkp2, discovered yesterday typo in tox.ini line 4: skipdist = True, it should be skipsdist10:21
sshnaidmssbarnea|bkp2, would be nice to fix it in this patch or next one10:21
quiquellssbarnea|bkp2: good morning, do you have a minute ?10:21
arxcruzssbarnea|bkp2: can i have 5 minutes without a change from you? :P10:22
arxcruzssbarnea|bkp2: btw, bkp2 ?10:22
ssbarnea|bkp2sshnaidm: i will fix in next (the ansible-lint one) because when this merges I need to rebase the other (and I already passed the check on this one).10:23
ssbarnea|bkp2sure. send numeric bj link.10:24
ssbarnea|bkp2yeah... i will remove the suffix from irc nickname... didn't had time.10:24
ssbarnea|bkp2i am trying to setup a docker container with synapse matrix to use it for persistent irc but... lack of time.10:25
quiquellssbarnea|bkp2: my choise was just a raspberry pi running weechat and tmux10:25
quiquellssbarnea|bkp2: I know is low tech :-)10:25
quiquellssbarnea|bkp2: Need you just a minute for the paunch stuff10:26
ssbarnea|bkp2https://bluejeans.com/265541792810:27
quiquellssbarnea|bkp2: http://logs.openstack.org/37/618537/6/check/tripleo-ci-fedora-28-standalone/e4bb1d0/10:35
quiquellssbarnea|bkp2: http://logs.openstack.org/37/618537/6/check/tripleo-ci-fedora-28-standalone/e4bb1d0/logs/undercloud/var/log/extra/rpm-list.txt.gz10:35
quiquellssbarnea|bkp2: DLRN repos for fedora https://trunk.rdoproject.org/api-fedora-master/api/report.html10:36
chkumar|rucksshnaidm, marios https://review.openstack.org/#/c/618946/1 needs +w on this10:40
chkumar|rucksshnaidm, what we can do to move it forward https://bugs.launchpad.net/tripleo/+bug/180158710:43
openstackLaunchpad bug 1801587 in tripleo "[master/Rocky]Fs035 job fails in promotion becasue of heat stack timeout" [Critical,Triaged]10:43
sshnaidmchkumar|ruck, I think we need to wait for new OVS on rdo cloud firstly10:44
sshnaidmchkumar|ruck, not sure it's worth to try anything before..10:45
ssbarnea|bkp2quiquell: https://review.openstack.org/#/c/618830/10:45
*** sshnaidm has quit IRC10:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci-fedora-28-standalone @  (1 more message)10:52
amoralejpanda|rover, are you around? wrt new oooq jobs for rdoinfo10:53
*** chkumar246 has joined #oooq10:54
*** chkumar|ruck has quit IRC10:57
panda|roveramoralej: morning10:59
amoralejpanda|rover, how far are we to have zuulv3 jobs working?11:01
amoraleji'm thinking in adding a couple of jobs for stable releases11:01
amoralejqueens and rocky11:01
amoralejusing rdo-trunk11:01
amoraleji was wondering if it'd be worthy to start with multinode11:01
amoralejbut using the v3 stuff you are working on11:01
panda|roveramoralej: I have the patches in order to make it work, but there's a particular patch taht may take a whiel to review11:02
panda|roveramoralej: start with standalone :)11:02
panda|roveramoralej:  https://review.openstack.org/607288 it's a noop but changes the playbooks a lot,11:03
amoralejpanda|rover, so current blocker is upstream change, right?11:04
amoralejhttps://review.openstack.org/#/c/607288/11:04
panda|roveramoralej: yep, not really blocked, craving fro reviews mainly11:05
amoralejyeah, right11:05
amoralejpanda|rover, but standalone will be master only, right?11:05
amoralejnot for standalone11:05
panda|roverbut this is what enables other playbooks to call the main workflow while making also other things11:05
amoralejnot for stable, i meant11:05
panda|roveramoralej: he, yep11:05
amoralejfor queens and rocky i still need multinode11:06
panda|roveryes11:06
amoralejok, so i may start using current legacy stuff and once you have standalone ready, move to your nice v3 stuff11:06
amoralejand using run-test role11:06
panda|roveramoralej: cool, I'm sorry you can use the shiny zuulv3 right now11:08
panda|rovercan't*11:08
amoralejno problem11:08
*** udesale has quit IRC11:11
*** chkumar246 is now known as chkumar|ruck11:30
*** sshnaidm has joined #oooq11:41
ykarelchkumar|ruck, the 3 failures in rocky: https://trunk-primary.rdoproject.org/api-centos-rocky/api/civotes_detail.html?commit_hash=b1bb2bd24fae5c5a981688898d759f7a8fc71763&distro_hash=9ed865e1f6937967f87b3b09969966f2a50bf1b611:42
ykarelchkumar|ruck, fs017,fs001,fs035 all seems intermittent, running those again, we might have promotion if these passes with recheck11:43
ykarel1 failed with mirrorlist network, 1 failed with swift Bad URl and other tempest accessing 172.21 network11:43
chkumar|ruckykarel, which one is 172.71 network11:45
chkumar|ruck?11:45
chkumar|ruckykarel, fs35 is known for mirror issue11:45
ykarelchkumar|ruck, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/36d341b/logs/undercloud/home/zuul/tempest.log.txt.gz11:45
chkumar|ruckykarel, if you re run it will vanished11:46
ykarelchkumar|ruck, what will vanish?11:46
chkumar|ruckykarel, the same issue we are seeing in vexhost11:46
ykarelbut intermittent right? not always11:46
chkumar|ruckykarel, always11:46
chkumar|ruckykarel, I trued to reproduce on rdocloud but no luck11:47
chkumar|ruckykarel, let me try this fs00111:47
ykarelchkumar|ruck, i recently deployed fs00111:47
chkumar|ruckykarel, is it reproducable?11:47
ykarelfor me it was successful11:48
ykarelchkumar|ruck, what will vanish if i rerun?11:48
chkumar|ruckykarel, python-tempestconf url retry error11:48
chkumar|ruckykarel, can you check tempest.log if this error is there or not?11:49
ykarelchkumar|ruck, https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/36d341b/logs/undercloud/home/zuul/tempest.log.txt.gz11:49
ykarel?11:50
*** chkumar246 has joined #oooq11:53
chkumar246ykarel, https://bugs.launchpad.net/tripleo/+bug/180315911:53
openstackLaunchpad bug 1803159 in tripleo "Config_tempest fails to gather extensions from nova in featureset001/53 OVB job in master" [High,Triaged] - Assigned to Gabriele Cerami (gcerami)11:53
chkumar246ykarel, can you check if these are still present in your deployment https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/36d341b/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-11-20_08_20_5311:54
ykarelchkumar246, but don't think it's vexxhost related as seeing sometimes in rdo-cloud11:54
chkumar246ykarel, the same thing is happening in the logs you passed11:55
ykarel"sometimes" that's why saying to rerun those job, but can't get what will vanish?11:55
*** chkumar|ruck has quit IRC11:56
*** gkadam has quit IRC11:56
*** dtantsur is now known as dtantsur|brb11:59
*** dsneddon has quit IRC12:00
*** dsneddon has joined #oooq12:02
*** amoralej is now known as amoralej|lunch12:08
chkumar246ykarel, sorry12:10
*** dsneddon has quit IRC12:11
ykarelchkumar246, for?12:11
chkumar246ykarel, let me say this way, on vexhost  on fs001 with this error WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, temepstconf was able to generate the conf and passed the job yesterday12:12
chkumar246I am asking in your current fs01 rocky deployment in tempest.log does this error vanished?12:12
*** dsneddon has joined #oooq12:13
ykarelchkumar246, i can see https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky/36d341b/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-11-20_08_20_5312:13
*** gkadam has joined #oooq12:14
ykarelchkumar246, my question is ^^ Error is not happening always in rdo-cloud, right?12:15
*** chkumar has joined #oooq12:15
*** chkumar246 has quit IRC12:18
*** ratailor has quit IRC12:23
*** panda|rover is now known as panda|rover|lch12:48
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci- (1 more message)12:52
quiquellmarios: Found something weird regarding standalone_env file override12:52
marioso/ quiquell ?12:53
quiquellmarios: humm will lunch first, long story short12:53
mariosquiquell: ack we can talk later np12:54
*** chkumar246 has joined #oooq12:54
quiquellmarios: my custom job for the zuul-in-a-box is generating this override https://paste.fedoraproject.org/paste/ss4kdGEK9nz6HD8vlPSPog12:54
*** chkumar246 is now known as chkumar|ruck12:54
quiquellmarios: this is the zuul config I am using https://paste.fedoraproject.org/paste/DBTJtW4Zd0QNGkPxssQplw12:54
*** quiquell is now known as quiquell|lunch12:55
quiquell|lunchI don't see the issue12:55
quiquell|lunchgo back in a few12:55
*** chkumar has quit IRC12:57
*** dsneddon has quit IRC12:57
*** dsneddon has joined #oooq12:58
*** dtantsur|brb is now known as dtantsur13:00
chkumar|rucksshnaidm, Hello13:06
*** dsneddon has quit IRC13:06
chkumar|rucksshnaidm, I have holed a node for debugging vexhost13:06
chkumar|rucksshnaidm, on overcloud it is giving this http://paste.openstack.org/show/735726/13:07
chkumar|ruckafter running any openstack commands13:07
*** dsneddon has joined #oooq13:11
*** jrist has quit IRC13:11
sshnaidmchkumar|ruck, looking13:12
*** jrist has joined #oooq13:13
sshnaidmchkumar|ruck, seems like problem with auth_url?13:14
ykarelsshnaidm, have you also hold overcloud nodes, seems they might be deleted by tebroker13:15
ykarelsorry chkumar|ruck ^^13:15
sshnaidmchkumar|ruck, try to add v2.0 or v3.0 after https://10.0.0.5:13000/13:16
chkumar|rucksshnaidm, I have also exported it explicitly but no liuch http://paste.openstack.org/show/735727/13:16
chkumar|rucksshnaidm, yes doing that13:17
ykarelchkumar|ruck, running those three jobs https://review.rdoproject.org/r/#/c/13943/13:17
sshnaidmchkumar|ruck, if it's overcloud then ykarel is right, nodes could be deleted by tebroker13:17
chkumar|ruckykarel, sshnaidm checking13:18
quiquell|lunchmarios: Don't we need just to merge two lists ?13:18
*** dsneddon has quit IRC13:20
*** dsneddon has joined #oooq13:24
chkumar|rucksshnaidm, ykarel http://paste.openstack.org/show/735728/ I think overcloud nodes got deleted13:24
sshnaidmchkumar|ruck, yeah13:26
chkumar|rucksshnaidm, so there is no way to debug13:26
sshnaidmykarel, do you know if autohold works only after a job?13:26
sshnaidmykarel, or there is a way to enter a node during a job13:26
ykarelsshnaidm, what i heard is: whenever autohold is requested, when the job fails it will be hold13:27
sshnaidmchkumar|ruck, if it's possible enter a node during the job, it can help you.. by increasing timeout and adding infinite sleep to job we can keep it13:27
sshnaidmykarel, ok, so it must finish..13:28
sshnaidmchkumar|ruck, what about reproducer?13:28
chkumar|rucksshnaidm, from reproducer I am not able to reproduce13:28
ykarelchkumar|ruck, u have vexxhost creds?13:29
chkumar|rucktill yesterday I faced different issue during overcloud deploy need to try to one again13:29
chkumar|ruckykarel, yes13:29
ykarelchkumar|ruck, ack13:29
ykarelwhat error u faced13:29
ykarelsshnaidm, may be u can try same on rdo-cloud with the same config tebroker launches13:29
ykarellike network=private etc13:30
ykarelin openstack-nodepool tenant13:30
chkumar|ruckpanda|rover|lch, around?13:30
sshnaidmykarel, to try what?13:30
ykarelsshnaidm, reproducing ovb failures on rdo-cloud13:30
sshnaidmykarel, you mean autohold?13:31
ykarelsshnaidm, with autohold u will need to hack tebroker to not delete overcloud nodes13:31
*** amoralej|lunch is now known as amoralej13:32
sshnaidmykarel, so only way I can think it's to inject chkumar|ruck's ssh keys into undercloud node and hold it with sleep13:32
ykareli was saying if it can be reproduced using reproducer script(using same environment config as tebroker: openstack-nodepool tenant, private network)13:32
chkumar|rucksshnaidm, ykarel pass your ssh keys and I will add to this undercloud node13:33
ykarelsshnaidm, but there may be timeout limit i think in that case13:33
chkumar|ruckbut there is no overcloud i think it will not help13:33
ykarelchkumar|ruck, but this is not usable i think when overcloud  nodes are deleted13:33
sshnaidmykarel, I don't think it's possible to use te-broker in reproducing..13:33
chkumar|ruckyes13:33
sshnaidmykarel, we can increase timeout13:33
*** rlandy has joined #oooq13:33
chkumar|rucksshnaidm, ykarel I have asked tristinc to hold overcloud nodes or not on #sf-ops13:34
ykarelchkumar|ruck, that's not under their control13:34
sshnaidmchkumar|ruck, I don't believe Tristan can help here, overcloud nodes are managed by te-broker13:34
chkumar|ruckhmm13:35
sshnaidmchkumar|ruck, and it can't hold them, I was checking this with bnemec13:35
chkumar|rucksshnaidm, we can hold overcloud node too13:35
sshnaidmchkumar|ruck, nope13:35
ykarelsshnaidm, hmm directly not possible , but i think by hacking https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/te-broker/destroy-env script seems possible13:35
chkumar|rucksshnaidm, http://paste.openstack.org/show/735729/13:36
chkumar|rucksshnaidm, let's wait for nhicher to appear13:36
sshnaidmykarel, maybe, but it will affect all jobs13:36
*** quiquell|lunch is now known as quiquell13:36
ykarelsshnaidm, yup adding not to delete a particular ENVNUM13:36
ykarelsshnaidm, and by using reproducer script i meant: https://github.com/openstack-infra/tripleo-ci/blob/master/scripts/te-broker/create-env#L42-L5613:37
ykarelbecause using reproducer we are creating multiple networks, etc while with tebroker a network named private is used13:37
*** panda|rover|lch is now known as panda|rover13:37
panda|roverchkumar|ruck: hey13:38
chkumar|ruckpanda|rover, can we sync?13:38
sshnaidmchkumar|ruck, ykarel well, hacking delete-env is good idea, I think we can do it13:38
panda|roverchkumar|ruck: yep13:39
chkumar|rucksshnaidm, ykarel ok then let's do it tomorrow in the morning then13:39
sshnaidmchkumar|ruck, start a job and tell me its ENVNUM number, or just point to logs13:39
ykarelsshnaidm, i heard few days back panda|rover was trying something, not sure if that helped13:39
ykarelmay be panda|rover already have some news with the ovb failures13:39
chkumar|ruckpanda|rover, https://redhat.bluejeans.com/1571313919/6145/?src=meet_now13:39
sshnaidmchkumar|ruck, ok, if you need to hold them, ping me, we can try something..13:40
panda|roverykarel: mine wasn't a hack, was a double axe on the tenat, and kforde was trying to debug the network issues13:40
ykarelpanda|rover, ack so any luck with that?13:41
quiquellmarios: found the issue we are deleting the standalone_environment_file from the stuff to combine13:41
quiquellmarios: will put a review to fix it13:41
ykarelchkumar|ruck, sshnaidm ack13:41
*** skramaja has quit IRC13:43
*** dsneddon has quit IRC13:44
mariosquiquell: o/ sorry was getting something to eat, we can talk about it after tripleo? in community call?13:45
chkumar|ruckpanda|rover, I think I lost you13:46
*** dsneddon has joined #oooq13:46
quiquellmarios: ack13:46
mariosquiquell: but it seems to work fine for scen113:46
mariosquiquell: i mean the standalone_environment_file13:46
quiquellmarios: Yep, we have to remove it before combine13:47
nhicherhello chkumar|ruck13:48
*** chkumar246 has joined #oooq13:49
quiquellmarios: only fails to me in the job definition that pasted to you13:50
quiquellmarios: also doing this dummy test https://paste.fedoraproject.org/paste/yMbaI3FwgZmaHH~bo2KYvA13:50
*** chkumar|ruck has quit IRC13:52
mariosquiquell: well fwiw it was passed correctly in scen 1 i mean the deploy command has -e /standalone-scenario001.yaml13:53
mariosquiquell: but lets talk on phone in a bit anyway13:53
quiquellmarios: Hummm have to be doing something wrong in the job definition locally, at upstream zuul is working fine there13:53
quiquellmarios: the standalone.sh is correctly generated13:53
mariosquiquell: right sec13:53
quiquellmarios: na forget about it, have dig it deeper13:54
mariose.g. v35 @ http://logs.openstack.org/72/616872/35/check/tripleo-ci-centos-7-scenario001-standalone/fe774cb/logs/undercloud/home/zuul/standalone.sh.txt.gz13:54
mariosquiquell: ^13:54
quiquellmarios: DonÂ't we have to merge it with the one already present at fs052 ?13:55
quiquellmarios: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset052.yml#L1913:55
mariosquiquell: that is done in the run-v3 though?13:56
quiquellmarios: It's not merging the two lists, just replacing it13:57
mariosquiquell: ah you're right13:57
quiquellmarios: we are supose to replace or merge them ?13:57
mariosquiquell: its a bug then. i don't see low-memory in my standalone.sh13:57
quiquellmarios: but we have to run with low-memory is it right ?13:57
mariosquiquell: we need to update https://github.com/openstack-infra/tripleo-ci/blob/6034dc8f05dc880a6dc06af22348d1893d029ba6/playbooks/tripleo-ci/run-v3.yaml#L68-L7313:58
quiquellmarios: Yep I am into it13:58
mariosquiquell: well if that's in the featureset then yeah13:58
quiquellmarios: playing with it13:58
mariosquiquell: we need to merge them13:58
quiquellmarios: also I think the order is important :-/13:58
mariosquiquell: err13:58
mariosquiquell: hold on13:58
mariosquiquell: yeah13:58
mariosquiquell: it IS same featureset13:58
quiquellwhat do you mean ?13:59
mariosquiquell: momentarily confused that we were using a new fs but we aren't that's the whole point of featureset_override13:59
mariosquiquell: ignore carry on yes we need to merge13:59
*** chkumar246 has quit IRC14:01
mariosquiquell: yeah order definitiely does matter when it comes to heat and precedence of any defined variables however i don't think its up to us to sort that, i.e. if they try to make incompatible envs14:03
mariosquiquell: go say hello in tripleo14:04
quiquellalways at my bubble14:05
*** agopi is now known as agopi|brb14:06
rlandymarios: hi ... wrt https://review.openstack.org/#/c/616993/ ... I've been working there ... mind if I remove the depends on?14:07
rlandyfails zuul14:07
*** agopi|brb has quit IRC14:10
mariosrlandy: sure it was just so i could run it but go ahead.14:10
rlandythanks14:11
mariosrlandy: i applied the inventory stuff like https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=862d863a89cac7587b2e7b877d5e1e9404ad5f60&context=3&ignorews=0&dt=0 and applied your patch from https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=9ee4c8abb6a83d9962bf66cb89fae39d355da773&context=3&ignorews=0&dt=0 and running nodepool-setup like in https://review.openstack.org/#/c/616993/10/roles/creat14:11
mariosrlandy: but fighting paths still14:11
mariosrlandy: we can catch up on community call?14:11
rlandymarios: sure - almost finished patching together the reproducer script - then it will be a one shot run14:12
mariosrlandy: cool14:12
ykarelpanda|rover, can u also check my comment on https://review.openstack.org/#/c/618669/3/playbooks/tripleo-ci/post.yaml and fix to get logs available, /me out for today14:14
*** dsneddon has quit IRC14:15
*** Goneri has joined #oooq14:15
panda|roverykarel: yup thanks14:16
*** dsneddon has joined #oooq14:20
*** saneax has quit IRC14:21
quiquellmarios: Ok I think I got it14:30
quiquellmarios: ansible and dict+list is a pain14:30
*** ykarel is now known as ykarel|away14:31
*** agopi|brb has joined #oooq14:34
quiquellmarios: https://review.openstack.org/61901014:34
*** agopi|brb is now known as agopi14:34
*** dsneddon has quit IRC14:36
*** ykarel|away has quit IRC14:37
weshay_ptoanyone need anything from me, while I'm here :)14:38
weshay_pto?14:38
*** dsneddon has joined #oooq14:40
mariosweshay_pto: go do thanksgiving man srsly stop harrassing us14:41
mariosor14:41
mariosgive me your pto14:41
mariosand come work14:41
* marios shrugs14:41
weshay_ptololz14:42
weshay_ptomarios, if you give me your island14:42
mariosoislan14:42
jaosoriorlol14:42
jaosoriorweshay_pto: you're on PTO this whole week, right?14:42
weshay_ptojaosorior, ya.. but it's pretty chill so I can log in and chat if needed14:42
jaosoriorhaha sure, and I appreciate that. Though I don't wanna poke you too much on your PTO dude. Gotta chill too.14:43
jaosoriorweshay_pto: I'll sync up with marios tomorrow, and we'll catch up with you when you're back. Thanks for stepping up!14:45
weshay_ptojaosorior, np man14:45
weshay_ptojaosorior, go run ur meeting ptl14:45
quiquelljaosorior, marios: Everytime someone talks to weshay_pto with _pto suffix a cat dies14:47
*** vinaykns has joined #oooq14:47
*** weshay_pto is now known as weshay14:47
weshaycan't be responsible for that.. pita will kill me14:48
* jaosorior not fond of cats. As long as it's not a dog it's cool.14:48
quiquelljaosorior: Then you can poke him14:48
weshayarxcruz, need anything14:48
weshayquiquell, 3min14:48
jaosoriorlol14:48
arxcruzweshay: beer, but i can't14:48
quiquellweshay: ack14:48
jaosoriorarxcruz: or can you?14:49
arxcruzjaosorior: my liver needs a break from alchool :P14:49
jaosoriorlol, yeah, last week was rough14:49
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci- (1 more message)14:52
*** chkumar246 has joined #oooq14:53
rfolcomarios, rlandy sshnaidm arxcruz quiquell ssbarnea|bkp2 ping ci mtg14:54
mariosack rfolco was waiting for tripleo to close14:54
mariosdid i miss it ?14:54
marios;/me joining14:54
sshnaidmrfolco, can you send a link?14:55
rfolcohttps://bluejeans.com/4113567798/14:55
rfolcowes bj14:55
rfolcoits in rhos mtg calendar14:55
sshnaidmrfolco, can you resend the link please to mail?14:55
sshnaidmrfolco, I don't have rhos cal enabled by default, it's too noisy..14:56
rfolcosshnaidm, you can copy it to your cal14:56
arxcruzrfolco: is it in your bj ?14:56
rfolconooooooooo14:56
arxcruzi'm there with panda|rover14:56
rfolcohttps://bluejeans.com/4113567798/14:57
rfolcowes bj14:57
rfolcopanda|rover, arxcruz ^14:57
panda|roverI'll stay here14:57
panda|roverit's cozy14:57
*** udesale has joined #oooq15:05
*** Goneri has quit IRC15:15
rlandypanda|rover: how goes the centos 7.6 testing?15:28
rlandypanda|rover: work out for you with ovb tenant?15:28
*** dsneddon has quit IRC15:29
*** dsneddon has joined #oooq15:29
arxcruzmarios: rlandy quiquell https://review.openstack.org/#/c/607077/ the rest api for zuul15:30
rlandyarxcruz: cool - thanks15:32
mariosrlandy: wana syncup?15:36
mariosrlandy: or ping me when you want?15:36
mariosrlandy: * my bluejeans: https://redhat.bluejeans.com/7661925373/15:37
mariosrlandy: if anyone else interested in zuul3/repro15:37
mariosthanks arxcruz15:38
rfolcomarios, too late ? nobody is there15:48
*** dsneddon has quit IRC15:48
mariosrfolco: we didn't join yet i'll ping you fi you like15:49
rfolcomarios, I like :)15:49
*** dsneddon has joined #oooq15:49
mariosack rascasoft15:50
marioserr sorry rfolco15:50
mariossorry rascasoft ping by tab apologies for noise15:51
rascasoftmarios, no worries, I need friends15:51
rascasoftrlandy, hey, so yesterday's job went fine on my tenant15:52
rascasoftrlandy, I wonder now how to pass --zuul-changes since we also have RDO reviews in there15:53
rlandymarios: ok - let's chat15:57
rlandyjoining your bj15:57
mariosrlandy: ack joining ping rfolco15:57
rfolcoo/ thx15:57
*** dsneddon has quit IRC15:57
rlandyrascasoft: you can define ZUUL_CHANGES15:57
rlandylook at a CI job15:57
rlandyyou can copy it from there15:58
rlandyeven if the job failed15:58
rascasoftrlandy, ok, but will it consider both rdo and openstack reviews?15:58
rascasoftthat's my doubt15:58
*** dsneddon has joined #oooq16:01
rascasoftrlandy, and by the way: which one should be the log I need to refer to here http://logs.rdoproject.org/82/16882/7/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset041/ad1ac59/logs/ ?16:06
*** dsneddon has quit IRC16:11
*** udesale has quit IRC16:11
*** dsneddon has joined #oooq16:12
quiquellmarios: Are you still there ?16:13
*** hamzy has quit IRC16:14
mariosquiquell: yeah in call16:14
quiquellmarios: ack, then let's talk tomorrow, looks like to merge we need to include tht at project-requirements at job16:14
quiquell:-/16:14
mariosquiquell: k16:14
mariosquiquell: lets talk16:15
quiquellmarios: ack see you tomorrow16:15
*** quiquell is now known as quiquell|off16:15
*** weshay is now known as weshay_pto16:15
marioshave a good one quiquell|off16:16
*** chkumar246 has quit IRC16:16
*** dsneddon has quit IRC16:20
*** ccamacho has quit IRC16:20
*** dsneddon has joined #oooq16:21
*** dsneddon has quit IRC16:31
*** jaganathan has quit IRC16:31
*** dsneddon has joined #oooq16:32
*** dsneddon has quit IRC16:41
*** agopi is now known as agopi|brb16:44
*** dsneddon has joined #oooq16:45
*** gkadam has quit IRC16:50
rfolcomarios, still there?16:52
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci- (1 more message)16:52
rfolcomarios, wondering if you know latest error/issue with scen001-standalone job... perhaps its current state is more or less the same as scen003 -- http://logs.openstack.org/61/618761/2/check/tripleo-ci-centos-7-scenario003-standalone/0cedb65/job-output.txt.gz16:53
rfolcowhich I have hard time in finding what went wrong16:54
*** dsneddon has quit IRC16:54
*** dsneddon has joined #oooq16:57
*** amoralej is now known as amoralej|off17:04
rlandyrascasoft: depends on the review ... https://logs.rdoproject.org/82/16882/7/check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset041/ad1ac59/zuul-info/inventory.yaml17:07
rlandythe change you need is refs17:07
*** dsneddon has quit IRC17:14
*** dsneddon has joined #oooq17:17
*** dtantsur is now known as dtantsur|afk17:19
*** jfrancoa has quit IRC17:23
ssbarnea|bkp2any tempest expert around? is this a specific test failure or infra related: http://logs.openstack.org/22/617022/1/gate/tripleo-ci-centos-7-standalone/0d48c04/logs/undercloud/home/zuul/tempest.log.txt.gz ?17:24
ssbarnea|bkp2btw: regarding categorization of failures: i can help you adding log stash queries as long you point me to the string to identify and a LP bug.17:25
*** dsneddon has quit IRC17:26
*** dsneddon has joined #oooq17:27
*** bogdando has quit IRC17:28
*** kopecmartin is now known as kopecmartin|off17:28
rascasoftrlandy, so are you saying that the file I need to check is zuul-info/inventory.yaml17:28
rascasoftrlandy, all the info are always there, right?17:28
rlandyyes17:28
rlandyor search the console log for ZUUL_CHAGES17:28
rlandywill be the same17:29
rascasoftrlandy, ok, makes sense17:29
*** trown is now known as trown|lunch17:30
rascasoftrlandy, ok, this is my command line: ./reproducer-quickstart-ha-utils-integration.sh -v true -a --zuul-changes "openstack/tripleo-ha-utils:master:refs/changes/70/610570/2^rdo-jobs:master:refs/changes/02/16902/4^config:master:refs/changes/03/16903/3^testproject:master:refs/changes/82/16882/7"17:32
rascasoftlet's see how it breaks17:32
*** dsneddon has quit IRC17:36
*** dsneddon has joined #oooq17:37
*** agopi|brb is now known as agopi17:42
*** dsneddon has quit IRC17:49
*** dsneddon has joined #oooq17:49
*** jtomasek has quit IRC17:56
*** dsneddon has quit IRC17:57
*** derekh has quit IRC18:00
*** dsneddon has joined #oooq18:01
*** dsneddon has quit IRC18:20
*** dsneddon has joined #oooq18:21
*** jrist has quit IRC18:22
*** dsneddon has quit IRC18:29
*** dsneddon has joined #oooq18:34
*** dsneddon has quit IRC18:42
*** dsneddon has joined #oooq18:43
*** dsneddon has quit IRC18:50
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci- (1 more message)18:52
*** trown|lunch is now known as trown18:53
*** dsneddon has joined #oooq18:55
*** holser_ has quit IRC18:58
*** apetrich has quit IRC19:04
*** jrist has joined #oooq19:11
*** apetrich has joined #oooq19:16
ssbarnea|bkp2rlandy: can you workflow https://review.openstack.org/#/c/618809/ ? (already got 2x2)19:16
*** amoralej|off is now known as amoralej19:18
*** dsneddon has quit IRC19:20
*** dsneddon has joined #oooq19:21
*** hamzy has joined #oooq19:25
*** jrist has quit IRC19:27
*** sshnaidm has quit IRC19:32
*** sshnaidm has joined #oooq19:40
*** sshnaidm is now known as sshnaidm|afk19:40
*** jrist has joined #oooq20:23
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci- (1 more message)20:52
*** dsneddon has quit IRC20:58
*** dsneddon has joined #oooq21:00
*** apetrich has quit IRC21:02
*** hamzy has quit IRC21:42
*** dsneddon has quit IRC21:46
*** dsneddon has joined #oooq21:48
*** apetrich has joined #oooq21:51
*** dsneddon has quit IRC21:54
*** dsneddon has joined #oooq21:55
*** trown is now known as trown|outtypewww22:08
*** dsneddon has quit IRC22:33
*** dsneddon has joined #oooq22:35
*** apetrich has quit IRC22:47
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario002-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-scenario001-multinode-oooq-container, tripleo-quickstart-extras-gate-master-delorean-full-featureset052, tripleo-ci- (1 more message)22:52
*** agopi is now known as agopi|brb22:54
*** agopi|brb has quit IRC22:59
*** dsneddon has quit IRC23:04
*** dsneddon has joined #oooq23:22
*** dsneddon has quit IRC23:27
*** dsneddon has joined #oooq23:28
*** agopi|brb has joined #oooq23:43

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!