Monday, 2019-02-18

*** weshay|ruck is now known as weshay_PTO00:28
*** fultonj has joined #oooq02:04
*** fultonj has quit IRC02:06
*** ykarel|away has joined #oooq03:10
*** udesale has joined #oooq03:44
*** skramaja has joined #oooq03:47
*** ykarel|away is now known as ykarel04:00
*** agopi has quit IRC04:15
*** ykarel has quit IRC04:58
*** ykarel has joined #oooq05:19
*** jbadiapa has joined #oooq06:02
*** chandankumar is now known as chkumar|ruck06:17
chkumar|ruckquiquell|off: Hello06:26
chkumar|ruckykarel: Hello06:27
ykarelchkumar|ruck, hi06:27
chkumar|ruckykarel: please have a look at this bug https://bugs.launchpad.net/tripleo/+bug/181602606:27
openstackLaunchpad bug 1816026 in tripleo "multinode promotion jobs timing out at overcloud deploy" [Critical,In progress] - Assigned to chandan kumar (chkumar246)06:27
chkumar|ruckykarel: it is kind of weired, when I tried to run it in test project, it failed with tempest docker error and overcloud deploy passed06:28
ykarellooking06:28
chkumar|ruckykarel: I am not sure what is leading to this error https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset019-master/cf698b1/logs/undercloud/var/log/containers/mistral/engine.log.txt.gz?level=ERROR06:28
ykarelchkumar|ruck, so tempest is failing because it's not using podman06:38
ykarelit' using docker which is removed from undercloud :- https://review.openstack.org/#/c/636442/06:39
ykareland as overcloud is passing with the mistral revert, that mistral patch is somewhat related06:40
ykarelgood to ask someone from mistral as issue seems around https://github.com/openstack/tripleo-common/blob/master/workbooks/deployment.yaml#L40406:42
ykarel^^ is returning True now somehow06:43
ykarelchkumar|ruck, the error you pointed is obvious when it tries to access a blank list06:44
ykarelrunning_config_download_workflows = []06:44
*** jtomasek has joined #oooq06:49
chkumar|ruckykarel: yes that is weired07:04
chkumar|ruckykarel: regading tempest tempest can run on podman07:04
chkumar|ruckykarel: support was already added for the same07:04
ykarelwhere?07:04
chkumar|ruckykarel: https://review.openstack.org/#/c/605356/07:05
ykarelchkumar|ruck, and where you enabled it07:06
ykarelthe patch is just adding support07:06
ykarelhttps://review.openstack.org/#/c/605356/55/roles/validate-tempest/templates/run-tempest.sh.j2@6807:07
chkumar|ruckykarel: it depends upon undercloud_container_cli07:08
ykarelyup07:08
ykarelwhere you set it to podman for master07:08
chkumar|ruckykarel: if it is defined in fs everything is taken care of07:09
ykarelchkumar|ruck, but most of the fs don't have it07:09
ykarelso docker is used07:09
ykareland it's failing07:09
chkumar|ruckykarel: then it is abug in fs side07:09
ykarel:)07:10
chkumar|ruckykarel: http://codesearch.openstack.org/?q=undercloud_container_cli&i=nope&files=.yml&repos=tripleo-quickstart07:10
chkumar|ruckykarel: let me check fs1607:10
chkumar|ruckykarel: let me fix that part07:14
chkumar|ruckif there is a containerized uc there we need to set undercloud_container_cli07:15
*** jfrancoa has joined #oooq07:17
*** sanjayu__ has joined #oooq07:17
*** apetrich has joined #oooq07:19
*** quiquell|off is now known as quiquell|rover07:23
quiquell|roverchkumar|ruck: o/07:23
quiquell|roverchkumar|ruck: https://github.com/gnocchixyz/gnocchi/pull/1017 merged07:25
quiquell|roverbut we need a promotion07:25
chkumar|ruckquiquell|rover: we have to wait till they create a new tag07:25
chkumar|ruckquiquell|rover: gnocchi is pinned in rdoinfo07:25
chkumar|ruckquiquell|rover: we have a serious bug in our fs07:26
quiquell|roverchkumar|ruck: Ahh yep, rdoinfo maybe we can unpin it ?07:26
quiquell|roverchkumar|ruck: what happend ?07:26
quiquell|roverthe timeout ?07:26
chkumar|ruckquiquell|rover: where containerized_undercloud is defined, there undercloud_container_cli is not defined07:26
chkumar|ruckin most of the fs07:26
quiquell|roverchkumar|ruck: maybe we can unpin it from rdoinfo so we just depend on promotions ?07:26
chkumar|ruckquiquell|rover: let me try that for rdoinfo07:27
*** jtomasek has quit IRC07:27
quiquell|roverchkumar|ruck: the cli is docker by default I think07:27
quiquell|roverchkumar|ruck: humm no this is standalone07:27
chkumar|ruckquiquell|rover: I am taking from fs1607:27
quiquell|roverchkumar|ruck: so we remove the pin or we update the pin ?07:27
apetrichmorning chkumar|ruck07:27
*** jtomasek has joined #oooq07:27
apetrichwhat can I do to help you with https://bugs.launchpad.net/tripleo/+bug/1816026 ?07:28
openstackLaunchpad bug 1816026 in tripleo "multinode promotion jobs timing out at overcloud deploy" [Critical,In progress] - Assigned to chandan kumar (chkumar246)07:28
quiquell|roverchkumar|ruck:f fs016 is not mistral ?07:28
chkumar|ruckquiquell|rover: check my https://bugs.launchpad.net/tripleo/+bug/1816026 comment 607:28
chkumar|ruckquiquell|rover: tempest error you will get it07:28
quiquell|roverchkumar|ruck: ack so is not mistral07:29
chkumar|ruckquiquell|rover: if I run the job seperatly it passes the overcloud07:29
chkumar|ruckbut in promotion it is returning timedout07:29
quiquell|roverchkumar|ruck: if you run it at check it uses diferent release file07:31
quiquell|roverchkumar|ruck: And it means different mistral version07:31
quiquell|roverchkumar|ruck: and different vars07:32
chkumar|ruckquiquell|rover: https://github.com/gnocchixyz/gnocchi/pull/1018 already backported to stable/4.307:32
chkumar|ruckquiquell|rover: let me just re-run the job07:32
quiquell|roverchkumar|ruck: you have to hack this https://github.com/openstack-infra/tripleo-ci/blob/master/roles/common/vars/main.yaml#L1407:34
quiquell|roverchkumar|ruck: to use same release file at periodic07:34
*** quiquell|rover is now known as quiquell|rover|b07:36
*** dtantsur|afk is now known as dtantsur07:42
*** sshnaidm|off is now known as sshnaidm07:56
chkumar|ruckapetrich: Hello07:57
chkumar|ruckapetrich: regarding mistral issue https://bugs.launchpad.net/tripleo/+bug/1816026/comments/807:57
openstackLaunchpad bug 1816026 in tripleo "multinode promotion jobs timing out at overcloud deploy" [Critical,In progress] - Assigned to chandan kumar (chkumar246)07:57
*** ykarel is now known as ykarel|lunch07:58
chkumar|ruckapetrich: in the check job old commit is used https://github.com/openstack/mistral/commit/58d6634702014eb3c2bf70dbe487fe5983cef8e9 but when this https://github.com/openstack/mistral/commit/a39db2d3dc21afbe8b9e1d6f8dd870da272b9671 is used, it is leading to timedout07:58
chkumar|ruckin periodic pipeline07:58
chkumar|ruckapetrich: i think revert fixes the issue or we need other fix07:59
chkumar|ruckapetrich: please have a look07:59
*** holser_ has joined #oooq07:59
apetrichchkumar|ruck, I am. I'm trying to talk to other mistral folk but I think it is better to revert first08:01
quiquell|rover|bI think issue is just python cannot discover type of empty list08:04
quiquell|rover|bDoes not have to be difficult to put a patch for it with a unit test08:04
quiquell|rover|bSo we don't have to revert08:04
*** kopecmartin|off is now known as kopecmartin08:04
*** quiquell|rover|b is now known as quiquell|rovee08:05
quiquell|roveeDamn08:05
*** quiquell|rovee is now known as quiquell|rover08:05
*** gkadam has joined #oooq08:06
*** gkadam has quit IRC08:09
*** panda|off is now known as panda08:09
*** rascasoft has joined #oooq08:09
pandazbr: marios rfolco my user story is Done, so I'm available to help for the promotion stuff08:19
mariospanda: ack thanks. my job merged but i didn't see it yet (can't quickly find in https://review.rdoproject.org/zuul/builds yet but didn't dig too much08:21
mariospanda: so i'll be doing that today and then debug08:21
mariospanda: folcos change need reviewing we said we'd merge those08:22
mariosand zbr was gonna incorporate his findings into folco change08:22
marioss08:22
pandamarios: your job is periodic, will run 4 times a day. When did it merge ?08:22
mariospanda: yah but friday08:23
*** amoralej|off is now known as amoralej08:23
mariosoh08:23
mariospanda: i think friday lemme check again08:23
mariospanda: yeah https://review.rdoproject.org/r/#/c/18729/08:23
pandamarios: weird, I don't see it too. https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/08:26
pandaI'm afraid it fails in a way that it cant even output logs08:28
quiquell|rovermarios: I think is here the place to look https://softwarefactory-project.io/zuul/t/rdoproject.org/builds08:29
pandathere's a pipeline build right now in https://review.rdoproject.org/zuul/status08:30
pandabut it's not there, so it's not running08:30
quiquell|roverthe job is there https://softwarefactory-project.io/zuul/t/rdoproject.org/job/periodic-tripleo-centos-7-master-containers-build-push08:30
mariosquiquell|rover: panda thanks just finishing some reviews and switching to in a bit08:31
mariosquiquell|rover: thanks for pointer there08:31
quiquell|roverpanda, marios: Have to be something with the config08:32
pandaquiquell|rover:yep, I think it has not been picked up properly ...08:33
quiquell|rover Going to ask around08:33
quiquell|rovermaybe they can check scheduler logs08:33
*** tosky has joined #oooq08:33
quiquell|roverpanda, marios: here we have "Â3" attempts08:35
quiquell|rover3 attempts08:35
quiquell|roverperiodic-tripleo-centos-7-master-containers-build-push08:35
quiquell|rover ahh this is config08:37
*** chem has joined #oooq08:37
mariosquiquell|rover: panda check https://review.openstack.org/637490 if you have 2 mins please (e.g. we should see scen1/4 running in https://review.openstack.org/#/c/637196/08:39
arxcruzpanda: sshnaidm can one of you guys +w https://review.openstack.org/#/c/636579/ ?08:40
quiquell|rovermarios: commented08:42
pandaarxcruz: done08:42
*** bogdando has joined #oooq08:42
arxcruzpanda: ty08:42
mariosquiquell|rover: thanks done08:47
zbrrfolco: panda did you had time to look at my change at https://review.openstack.org/#/c/636144/ ? i have no problems incorporating my changes to folco one but I want to get you to check it first.08:49
pandazbr: checking if I checked08:50
zbri still have one weird issue where apparently openstack overcloud container image build returns 0 on case of total failure. see http://logs.openstack.org/44/636144/22/check/tripleo-build-containers-fedora-28/5f850ce/job-output.txt.gz#_2019-02-17_18_35_35_49182008:51
*** jpena|off is now known as jpena08:52
*** ykarel|lunch is now known as ykarel08:59
zbrrfolco: are you also encountering http://logs.openstack.org/44/636144/19/check/tripleo-build-containers-fedora-28/fe7b613/job-output.txt.gz#_2019-02-17_13_09_43_706478 ?09:03
marios[docs - standalone jobs] o/ folks check it if you have 10 mins isn't too long :) comments please and lets merge it if you like it thanks https://review.openstack.org/#/c/635881/ http://logs.openstack.org/81/635881/2/check/openstack-tox-docs/597378b/html/ci/standalone_scenario_jobs.html09:04
mariosquiquell|rover: panda rfolco weshay_PTO sshnaidm zbr ** ^09:04
zbrit is a bit weird because I am cherry picking the change that adds supports for "base=fedora" and installing kolla with it. still at runtime, I get this. is like another copy of kolla would have being used.09:05
quiquell|rovermarios: talking with amoralej and #zuul looks like base job have "files" section so it does not do periodic, but this can be a zuul bug09:08
quiquell|rovermarios: let me reproduce something simmilar09:08
quiquell|rovermarios: can you test files: [] in the push job ?09:11
*** ccamacho has joined #oooq09:13
chkumar|ruckquiquell|rover: apetrich any other places https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/7e19d72/ I can look for more info like all the info related to overcloud deploy apart from mistral09:19
quiquell|roverhumm09:26
quiquell|roverwe didnt' have nothing in the gates this weekend09:26
quiquell|roverweird09:26
chkumar|ruckquiquell|rover: another weird http://zuul.openstack.org/status09:26
chkumar|ruckquiquell|rover: check tripleo queue09:26
chkumar|ruckquiquell|rover: 81 hr 46 min09:26
chkumar|ruckhttps://review.openstack.org/#/c/636934/ on this patch09:27
quiquell|roverchkumar|ruck: where do you see 81 hr ?09:27
quiquell|roverahh yep09:27
quiquell|rover...09:27
quiquell|roverwell is the new job09:28
quiquell|rovergood catch though09:28
chkumar|ruck:-)09:28
mariosquiquell|rover: thanks ! sure can add it09:29
marioschkumar|ruck: ack added for next reviews cos i really need to switch now !09:29
quiquell|rovermarios: For me this is a zuul bug, periodic have to run always no mather what do we have at files09:29
quiquell|roverchkumar|ruck: can you put that job to the wiki ?09:33
zbrrfolco: are you working on addresing comments on https://review.openstack.org/#/c/637212 -- if not I can help.09:36
mariosquiquell|rover: but i don't see any files09:36
mariosquiquell|rover: panda at https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleo-rdo-base.yaml https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/jobs.yaml09:36
mariosquiquell|rover: ah we need it empty09:37
mariosi get it now09:37
marios:) k adding09:37
zbrpanda: please give some advise regarding ^^ -- regarding if it would not be better to just have --mirror instead of variable params on each base os: like --centos-mirror and --fedora-mirrors ....09:37
quiquell|rovermarios: ack, let me know and we open a  zuul bug09:37
mariosquiquell|rover: well we won't know until we merge it09:37
*** derekh has joined #oooq09:37
mariosquiquell|rover: luckilly its override https://zuul-ci.org/docs/zuul/user/config.html#attr-job.files not merging so should work if that is indeed the problem09:38
quiquell|rovermarios: we can reproduce at reproducer09:40
quiquell|rovermarios: want to try ?09:40
quiquell|rovermarios: we can put in place a fake periodic pipeline09:40
quiquell|roverlot of fun :-)09:40
mariosquiquell|rover: yah i need to re-setup (migrated laptop) so i should do that09:41
mariosquiquell|rover: but i am kinda liking not having selinux alerts :D ;)09:41
quiquell|rovermarios: you can run it at quickvm https://operations.cee.redhat.com/#/quickvm09:42
quiquell|rovermarios: super fast to startup a f28 image09:42
mariosquiquell|rover: nice i didn't know that (yah was thinking of grabbing a  beaker box might do that)09:42
quiquell|rovermarios: no need for beaker, well those image at quickvm are not very powerful09:43
quiquell|rovermarios: but zuul is attacking your RDO tenant so no issue09:43
mariosquiquell|rover: yah i mean some of the beaker boxes are pretty small like even 4gb ram and they come on a timer (like 5 days is max /99 hours but you can extend once for another 99)09:44
pandazbr: yes, distro is automatically detected, but how would you specify different mirrors for fedora or centos ?10:00
zbrpanda: you can never specify multiple base OS mirrors, you can specify only one, for current OS. so it makes no sense to add one extra argument for each OS.10:01
zbryou just specify --distro --mirror10:01
zbrin fact distro is optional, and the safe mirror can also cover two OS-es at the same time. which means that tripleo-repos could be make to work on multiple platforms without having to call it with different parameters.10:03
zbrpanda: makes sense what I say? i can make a poc change to demostrate it in few minutes.10:05
rfolcozbr, can you help with https://review.openstack.org/#/c/637212 please10:06
zbrrfolco: sure!10:07
rfolcozbr, please keep in mind --centos-mirror may need to exist for compatibility10:09
zbrrfolco: i know, not dropping any old behavior.10:10
rfolcozbr, thx for your help :)10:11
quiquell|roverrfolco: the f28 containers jobs has 82 hr at zuul queue10:14
quiquell|roverrfolco: https://review.openstack.org/63693410:15
rfolcoquiquell|rover, will abandon that change, I have no idea why10:15
quiquell|roverrfolco: ack10:16
rfolcoquiquell|rover, done thanks10:16
rfolcozbr, as for https://review.openstack.org/#/c/636160, the ci job is defined in a separate change... I will merge into a single change10:20
rfolcomarios, panda ^10:20
zbrrfolco: yep, i do not see a reason to keep them separated, import it form my job, it avoids repetitio with yaml anchors.10:21
mariosrfolco: ack i commented and revoted10:25
rfolcothx hope this gets in good shape today for merging10:25
pandazbr: rfolco wanna meet ?10:41
rfolcopanda, up to you, I am making changes to the patch based on marios/zbr comments.10:42
rfolcopanda, can we meet 1 hour before scrum ?10:42
pandarfolco: ok, the fact is taht I'm not sure about the interaction between all your patches now, so I wanted to understand what we need to review10:45
rfolcopanda, ok, give me more few minutes, I can point you to the right sequence10:46
*** holser_ is now known as holser|lunch10:48
pandamarios: sorry I'm lost with your topic, do you have reviews for me ?10:50
mariospanda: which topic. https://review.rdoproject.org/r/18889 or https://review.openstack.org/#/c/635881 maybe10:51
pandamarios: the first one, ok I see. That's weird. Anyway now that I had more time to spend on it, I have to ask something that may be missing. But if it's missing we'll need to address it probably next sprint10:53
*** chem has quit IRC10:53
pandamarios: from which hash is that job building containers ?10:54
*** chem has joined #oooq10:55
pandarfolco: it can be in IRC if it's better for you10:56
mariospanda: latest i think. or at least "whatever the upstream job does"10:57
mariospanda: but i'm guessing that's wrong :)10:58
mariospanda: and yeah we can add override into the job definition but will not make it for wednesday10:58
mariospanda: but this is precisely why i want to see the thing run. thats what we discussed merge and debug till wednesday but quickly losing time10:59
quiquell|roverpanda, marios: I am preparing a test on periodic + files11:01
pandamarios: absolutely, agreee. We need to start from "tripleo-ci-testing" to build, but we'll deal with that in the next sprint, now focus on moving this forward. Unless of course building from a different hash it needs a complete refactor.11:03
pandaquiquell|rover: waht kidn of test ?11:03
quiquell|roverpanda: locally fake periodic pipeline and test the files job attribute and inheritance11:04
quiquell|roverpanda: to see if we are affected by it11:04
quiquell|roverpanda: so we don't have to merge unneeded stuff11:05
quiquell|roverpanda: at config11:05
pandaquiquell|rover: you're trying to adapt the reproducer to run periodic jobs ? Seems difficult especially to get the exact environment we need in rdoproject.11:06
mariospanda: well we're currently using a fake tag for push11:06
mariospanda: like tripleo-ci-testing11:06
marioserr11:06
mariosci-squad-testing11:06
rfolcopanda, marios zbr: so this is the change: https://review.openstack.org/#/c/636160/11:07
rfolcozbr is helping with https://review.openstack.org/#/c/637212, which not a blocker... tripleo-repos works with --centos-mirror <fedora_mirror>11:08
pandarfolco: ok so zbr will converge in that change 636160 and that will be the main focus11:09
mariosack rfolco will check afain11:09
mariosagain11:09
pandamarios: mmhh, that is the tafg used bu container after they are built11:10
pandamarios: maybe the right hash is pulled directly from tripleo-repos, but we needt o tell it to fetch that particular hash, so it may not even somethign on your patches that needs to be changed.11:11
pandamarios: focus on your task an thfour reviews at me, I'll explore that problem, and share the results.11:11
quiquell|roverpanda: no, just exercise periodic + files11:13
quiquell|roverpanda: I don'w want to test our periodic jobs there11:14
mariospanda: ack which 4 reviews?11:14
mariospanda: am looking briefly here too but gonna update it for the merge conflict/don't want to get stuck here. wrt my task, ideally i want to test the files: [] locally, it seems to be a perfect use-case for the reproducer quiquell|rover - not sure how i'll run the periodic yet but i'll ask you quiquell|rover once i get there. panda do you think its not even possible?11:16
pandamarios: (reproducer) I don't know, but since it's outside of MVP, and still not in RC1 even, I would not spend too much on it.11:17
mariospanda: well right now what else am gonna do? otherwise we need to merge https://review.rdoproject.org/r/18889 and hope it runs :)11:18
mariospanda: k might spend a bit more on the pacemaker stuff then11:18
* rfolco takes car to repair service brb11:18
mariosrfolco_work_from_shop11:18
pandamarios: yes ...11:20
pandaok, diving into https://review.openstack.org/636160 now11:22
zbrrfolco: panda marios: adding --mirror to tripleo-repos ready for review: https://review.openstack.org/#/c/637212/11:24
pandazbr: wait11:25
zbralso I think we need to talk on BJ about the cherry picking aproach, ok at xx:45 (in ~20min from now)11:26
pandazbr: how is this going to be used ?11:26
pandazbr: ok we can talk about that then11:26
zbrpanda: https://review.openstack.org/#/c/636144/ is using it, running now. but change also. needs to be backward compatible as we don't want to break that.11:27
quiquell|roverpanda, marios: files: [] does not work also - ".*" either11:33
pandaso files is not the problem11:33
quiquell|roverpanda: it is11:33
quiquell|roverpanda: does not trigger if base has files at periodic11:33
*** udesale has quit IRC11:33
quiquell|roverpanda: if I remove files it works11:34
*** udesale has joined #oooq11:34
mariosquiquell|rover: :/ but the override fails?11:34
quiquell|rovermarios: I see the value11:34
quiquell|rovermarios: Could be that files get aggregated not overriden11:34
quiquell|rovergoing to try again so I don't fuck up11:35
mariosquiquell|rover: not according to docs11:35
marioshttps://zuul-ci.org/docs/zuul/user/config.html#attr-job.files11:35
quiquell|rovermarios: do you have time fo rblue ?11:35
pandaquiquell|rover: where do we have files: specified ?11:36
quiquell|roverI want another pair of eyes11:36
mariosquiquell|rover: sure11:36
quiquell|roverpanda: blue ?11:36
quiquell|roverso we test it ?11:36
pandaquiquell|rover: ok11:36
quiquell|roverI can be doing something wrong11:36
mariosquiquell|rover: where11:36
quiquell|rovermarios, panda: I am at gcerami11:36
pandachez me11:37
mariospanda: number pls11:37
quiquell|roverjust gcerami11:37
quiquell|rovermarios: https://bluejeans.com/u/gcerami/11:38
pandamarios: 349250866911:38
mariosquiquell|rover: thanks panda yah i meant the nmeric id so i don't have to look in source :)11:38
*** jpena is now known as jpena|brb11:42
chkumar|ruckquiquell|rover: once the tripleo common works, then we can test telemetry stuff11:52
rfolcozbr, thanks for helping with --mirror patch11:52
rfolcozbr, marios panda I'm ready to talk about f28 patch. When is good time?11:56
zbrjoin meeting above ^ and wait for them to finish current subject.11:56
mariosrfolco: we are talking righ tnow about the periodic build contgainers job and issue running it https://review.rdoproject.org/r/#/c/18889/11:57
rfolcooh11:57
mariosrfolco: at 13:38 < quiquell|rover> marios: https://bluejeans.com/u/gcerami/11:57
marios13:38 < panda> marios: 349250866911:57
marios rfolco join us for radio quiquell|rover11:58
zbrbtw, i managed to fix my BJ and now I can click normal URLs and opening the client, not sure if others still have similar issue.11:58
mariosrfolco: we cant hear you12:01
mariosrfolco: can you hear us?12:01
rfolcoyes I hear you12:02
rfolcorejoining12:02
rfolcohell12:03
mariosrfolco: lemme know if you wana jump on a blue but sort it out firts :) gonna comment on your review now12:03
mariosrfolco: brb 2 mins12:03
zbrrfolco: marios panda we still need to talk on BJ about f28 work.12:04
rfolcomarios, zbr panda can you pls join https://bluejeans.com/u/rfolco I hope my mic works12:05
pandazbr: rfolco ok let's review it live12:05
mariosrfolco: ack gimme number please?12:05
rfolcohttps://www.google.com/url?q=https%3A%2F%2Fbluejeans.com%2F5878458097&sa=D&ust=1550662302494000&usg=AFQjCNFOk55G73RcR1s_4lauOylkXNqymA12:06
rfolcooops12:06
rfolco587845809712:06
rfolcomarios,12:06
mariosrfolco: joining thx12:06
*** apetrich has quit IRC12:11
*** skramaja_ has joined #oooq12:15
*** skramaja has quit IRC12:15
zbrhttps://review.openstack.org/#/c/636144/12:18
*** apetrich has joined #oooq12:25
quiquell|roversshnaidm: commented https://review.openstack.org/#/c/63658812:28
sshnaidmquiquell|rover, I mean not buildable projects like tripleo-ci, ovb repo, etc12:29
sshnaidmquiquell|rover, not every project can be built, but it doesn't mean we need to fail if it's in zuul changes12:29
quiquell|roversshnaidm: humm12:30
*** apetrich has quit IRC12:30
quiquell|roversshnaidm: we inject them at other ways12:30
quiquell|roversshnaidm: that's right12:30
quiquell|roversshnaidm: maybe we need a list for those12:30
sshnaidmquiquell|rover, yeah, we have this list, but not sure it's full12:30
quiquell|roversshnaidm: there is no transparent way to do it12:30
quiquell|roversshnaidm: skip ?12:30
chkumar|ruckquiquell|rover: sshnaidm https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/defaults/main.yml#L15 is the list na?12:31
sshnaidmquiquell|rover, https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/build-test-packages/defaults/main.yml#L1512:31
quiquell|roversshnaidm: but better to have false negative if the list is not full12:31
quiquell|roversshnaidm: than false positive12:31
quiquell|roversshnaidm: we can change the message and say that if project is being used by job it has to be added to the skip list12:32
quiquell|roversshnaidm: what do you think ?12:32
sshnaidmquiquell|rover, it started with problems with github packages, right?12:32
quiquell|roversshnaidm: yep12:32
sshnaidmquiquell|rover, and we can't build them..12:32
quiquell|roversshnaidm: like people adding Depends-On the CI is green12:32
quiquell|roversshnaidm: but not even the review itself is used12:33
quiquell|roversshnaidm: nope, at least for now more though have to be done about them12:33
sshnaidmquiquell|rover, I see.. ok, then you're right, better to fail if project is not in the list and not buildable12:33
quiquell|roversshnaidm: Going to change the error message and reference the skiplist12:34
sshnaidmquiquell|rover, but it requires a testing with different dummy patches for repos12:34
quiquell|roversshnaidm: totally will do12:34
chkumar|ruckquiquell|rover: what if they adds a depends on rdo spec file in the ci?12:35
quiquell|roverchkumar|ruck: we do test that12:35
quiquell|roverchkumar|ruck: This is new feature I thing12:35
quiquell|roversshnaidm: ^ ?12:35
quiquell|roverchkumar|ruck: also mapping is going to fail12:36
quiquell|roverchkumar|ruck: I will put a test for that too12:36
chkumar|ruckquiquell|rover: cool!12:36
quiquell|roverTo check that we fail at all possibilities12:36
sshnaidmyeah, I think it was something about it.. maybe jpena|brb or ykarel know better12:36
mariosrfolco: here https://review.openstack.org/#/c/635222/ https://review.openstack.org/#/c/635222/22/playbooks/tripleo-buildcontainers/templates/kolla-build.conf.j212:38
marioszbr: that was the 5 minutes ringing12:39
marios;D12:39
chkumar|ruckquiquell|rover: do we need to make some exceptions there also while failing github projects for example someday if we want to ceph-ansible depends on then?12:40
quiquell|roverchkumar|ruck: Now that I think of we have mapping with puppet-rabbitmq, this review is not enough12:41
ykarelsshnaidm, quiquell|rover what's up?12:42
quiquell|roverchkumar|ruck: for ceph-ansible if we want we fix12:42
chkumar|ruckquiquell|rover: ack!12:42
sshnaidmykarel, is it possible to test rdo spec file in CI?12:42
quiquell|roverchkumar|ruck: Do you know anything about this queen tempest issue ? http://logs.rdoproject.org/openstack-periodic-24hr/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/b273262/logs/tempest.html12:43
ykarelsshnaidm, u mean via depends-on on rdo spec in an upstream project?12:43
mariospanda: the base change is this https://review.openstack.org/#/c/636160/25/zuul.d/build-containers.yaml and https://review.openstack.org/#/c/636188/26/zuul.d/layout.yaml (though second review will merge into the first)12:43
mariospanda: fwiw/if it helps12:44
sshnaidmykarel, yes12:44
mariospanda: i.e. new base & then this stuff that folco is showing will run under the f28 one only12:44
ykarelsshnaidm, i remmeber mwhahaha added some support, let me find it12:44
chkumar|ruckquiquell|rover: it is the same error which is coming for rocky and master for telemetry tempest plugin12:44
sshnaidmykarel, yeah, also think there was something added..12:44
quiquell|roverchkumar|ruck: ack so we ignore it ?12:44
quiquell|roverchkumar|ruck: or we state somewhere that it affect queens too ?12:45
chkumar|ruckquiquell|rover: better to state it affects queens too12:45
chkumar|ruckquiquell|rover: but from today it is start showing na?12:45
quiquell|roverchkumar|ruck: Yep today I think12:45
ykarelsshnaidm, https://review.openstack.org/#/c/591865/12:45
chkumar|ruckquiquell|rover: need to check what got changed there12:45
chkumar|ruckquiquell|rover: may be we can open a seperate bug for this12:46
sshnaidmykarel, cool12:46
*** holser|lunch is now known as holser_12:46
sshnaidmchkumar|ruck, quiquell|rover we have this for testing rdo spec files: https://review.openstack.org/#/c/591865/3/roles/build-test-packages/tasks/dlrn-build.yml12:47
quiquell|roversshnaidm: ack12:47
chkumar|ruckack12:47
*** apetrich has joined #oooq12:48
arxcruzchkumar|ruck: please take a look at https://tree.taiga.io/project/tripleo-ci-board/task/664?kanban-status=1447274 and update if you have any other information when you have time please12:49
quiquell|roverchkumar|ruck: open the bug if you see it's needed12:49
chkumar|ruckquiquell|rover: sure12:49
*** bogdando has quit IRC12:50
*** bogdando has joined #oooq12:50
*** rlandy has joined #oooq12:54
marioszbr: sorry i don't mean to be rude i have to go look at this parenting before scrum13:05
marioszbr: talk again in less than an hour ;)13:05
*** trown|outtypewww is now known as trown13:06
zbrpanda: let's wf one of the changes, so we would have only one to cherry pick: https://review.openstack.org/#/c/629679/2013:06
zbronce of them is not problematic13:06
rfolcopanda, marios I do not need to change layout...       templates include - tripleo-build-containers-jobs13:09
rfolcozbr, ^13:09
mariosack rfolco13:16
*** amoralej is now known as amoralej|lunch13:17
rfolcomarios, point me to the no files patch pls13:18
rlandymarios: well, if you hack haproxy enough, ... tripleo-ci-centos-7-scenario009-standaloneSUCCESS in 1h13:20
rlandypanda: ^^13:20
pandarlandy: wow13:20
pandarlandy: and this was approved by one of the shiftonstack folks ?13:21
rlandypanda: it gets to be a bigger wow - will go through it in the meeting13:21
pandarlandy: so the task was not parked13:21
pandarlandy: ok13:21
rlandypanda: thetask was parked but I didn't give up on it13:21
rlandyit wasn't my choice to park it13:21
trownsshnaidm: never seen anyone talk on #openshift-dev so I am responding here :P13:23
trownsshnaidm: openshift-ansible is the only way to install openshift on openstack atm, but after the new installer[1] has support for openstack, you are correct that openshift-ansible wont be supported anymore [1]https://github.com/openshift/installer13:24
trownsshnaidm: at least that is my understanding...13:25
quiquell|rovertrown: they are going to use coreos operators ?13:25
trownquiquell|rover: ya, we have a operator for openstack: https://github.com/openshift/cluster-api-provider-openstack13:26
quiquell|rovermake sense13:26
trownor really a plugin for the cluster-api operator...13:27
*** jpena|brb is now known as jpena13:27
mariosrfolco: ack working on it13:28
*** quiquell|rover is now known as quique|rover|eat13:28
mariosrlandy: thanks will check in a bit my scen3 was also green (no pcs) the others failed 1/4 on the redis vip. i have to revisit but somethign else came up first13:28
rlandymarios: np - I have some info that may be helpful - will talk about it in meeting13:31
apetrichquique|rover|eat, I updated your patch to fix some pep8 errors and added more comments.13:41
*** skramaja_ has quit IRC13:44
*** skramaja has joined #oooq13:45
sshnaidmtrown, ack, thanks13:47
*** agopi has joined #oooq13:59
*** skramaja has quit IRC14:04
mariosrfolco:  15:53 < openstackgerrit> Marios Andreou proposed openstack-infra/tripleo-ci master: Add new base for the containers-build and periodic without files:  https://review.openstack.org/63754314:06
mariosnew change neededi just spoke about is here ^14:07
mariosand another one will follow14:07
*** quique|rover|eat is now known as quiquell|rover14:09
quiquell|roverapetrich: thanks!14:09
apetrichno worries14:09
zbrpanda: afaik, use of yaml constructs to avoid duplication in zuul is no-risk because that's if 100% implemented by yaml parser at load time, zuul has no idea about what you put inside, it just gets the expanded dictionary.14:13
*** ykarel is now known as ykarel|away14:14
*** amoralej|lunch is now known as amoralej14:14
*** ykarel|away is now known as ykarel14:15
*** vinaykns has joined #oooq14:24
quiquell|roverpanda: Now that I am thinking about periodics14:26
quiquell|roverpanda: why do we need hubbot rebasing if we have periodics ?14:27
quiquell|roverpanda: we can implement noop with periodics too14:27
chkumar|ruckquiquell|rover: timed to head home14:28
chkumar|rucksee ya tomorrow14:28
*** chkumar|ruck is now known as chandankumar14:28
quiquell|roverchandankumar: bye, read you tomorrow14:29
pandaquiquell|rover: test on the rebase are launched in ci.centos, and we don't have periodic there14:31
quiquell|roverpanda: ack14:32
pandazbr: yes, but it's the first time we are using them, and all your patch is doing is replicating all the contents of the parent, and changing a couple of keys, which is exaclty what zuul does in the end.14:32
zbrpanda: not the parent, the official job (centos-7). anyway, i am open for any approach as long we do not endup duplicating the list of file-patterns.14:34
zbri do not love yaml expansion myself, i just hate duplication more ;)14:35
pandamarios: another comment for your patch. We have all the base job in the same file14:36
pandamarios: don't know if you're comfortable in doing so too14:36
pandazbr: that's more reasonable and I think we can accomplish that with a single anchor containing the list of files for both.14:38
zbrpanda: i am adraid that not because yaml does  anchors work with dicitonaries and not with lists. i think i tried this first.14:39
zbri may be wrong14:39
mariospanda: will have a look momentarily14:39
quiquell|roversshnaidm: added test reviews as you commented https://review.openstack.org/63658814:41
*** ykarel is now known as ykarel|away14:42
*** ykarel|away has quit IRC14:52
*** dougbtv has joined #oooq14:53
arxcruzpanda: thanks for the review, I'll fix those :)14:57
mariospanda: i don't see a new comment in https://review.openstack.org/637543 did you forget to post it15:04
mariospanda: i am replying to your and zbr other comments now thanks and will update15:04
mariospanda: please add your comment and i'll do in the update15:04
pandamarios: added15:05
mariospanda: k not sold completely ont he base thing15:05
mariospanda:  i mean is nice that all the container build stuff is in one file15:06
mariospanda:and all the standalone stuff in another file?15:06
mariospanda: instead of also base.yaml15:06
pandamarios: it's also nice that we have all the bases in one file :)15:06
mariospanda: i guess its a matter of opinion;)15:06
pandamarios: trade offs, not a -1 from me15:06
quiquell|roverpanda: Can I workflow this ? https://review.openstack.org/#/c/63472515:06
quiquell|roverpanda: I see the job passin gthere15:06
mariospanda: ack lemme see what it lookslike and will update we can change if people feel strongly enough to -1 for a particular way15:07
quiquell|roverpanda: is the scenario012 new job15:07
pandaquiquell|rover: I wanted Derek to confirm it's legit success15:08
pandaquiquell|rover: I'll +1 it, we arelady have quorum15:09
pandaquiquell|rover: bot only after I see Derek commment positively15:09
quiquell|roverpanda: ack15:09
*** quiquell|rover is now known as quiquell|off15:11
jfrancoapanda: marios: as the ruck and rover are not online, just a heads up https://bugs.launchpad.net/tripleo/+bug/1816453 . The standalone upgrade job is failing, I've opened a lp for it and I'll investigate where the problem is15:21
openstackLaunchpad bug 1816453 in tripleo "Standalone upgrade job failing in task [Extra migration for nova tripleo/+bug/1656791]" [Medium,New] - Assigned to Jose Luis Franco (jfrancoa)15:21
mariosjfrancoa: thanks15:22
pandajfrancoa: are you afraid of quiquell|off  ? :) You waited for him to go offline. Behold! The Wrath of the rover!15:23
jfrancoaahahha, it was just a coincidence, I saw it now...but even though yes, he's a bit scary...too much hair for me :-D15:27
zbrjust got out of bj session with alex, he will try to merge the simple one and for the kolla one we can try current approach with one note: we may update it to run on rdo instead of upstream (if needed...)15:28
marioszbr: thanks for doing that call15:33
marioszbr: do you mean current approach just cherrypick/checkout as rfolco is doing now15:34
marioszbr: or do you mean fork as we discussed earlier15:34
zbrno fork15:34
zbrno fork, no branch15:34
mariosfork me . k so we'll just ask for forgiveness later if there are objections i guess :D15:35
marioszbr: rfolco panda ^15:35
zbri already integrated most of my changes to folco, now waiting for test results. also rebased change.15:35
zbri will ping you once I get the of from zuul15:36
*** udesale has quit IRC15:41
sshnaidmtrown, and where are you talking about openshift? :)15:42
trownsshnaidm: for openshift on openstack vms... downstream #shiftstack channel15:43
*** sanjayu__ has quit IRC15:44
rfolcozbr, we may update to rdo or we should ?15:49
*** ykarel|away has joined #oooq15:50
*** ykarel|away is now known as ykarel15:51
rfolcozbr, will rebase https://review.openstack.org/#/c/636160/ on https://review.openstack.org/#/c/637543/, but will wait for zuul job to run first16:03
rfolcomarios, panda fyi ^16:03
mariosrfolco: ack thanks16:07
*** ykarel is now known as ykarel|away16:13
rlandyrfolco; marios: of the cards in Ready to Review, afaict, only review.openstack.org/636160 is outstanding, correct?16:18
rfolcorlandy, this is the main change, yes. There a few ones around it.16:19
mariosrlandy: yeah and rfolco will reparent that onto my new base https://review.openstack.org/63754316:20
rfolcorlandy, I'm still waiting on zbr to tell me what are the kolla/tripleo-common patches that will be used16:20
rlandyrfolco: k - ping when ready for review16:20
rfolcorlandy, thanks16:21
zbrrfolco: rlandy : this needs to merge https://review.openstack.org/#/c/629679/ -- i removed depends-on from.16:27
pandaarxcruz: comments on https://review.rdoproject.org/r/1879516:47
pandaany last review before I leave for the day ?16:47
arxcruzpanda: no, thanks16:47
arxcruzpanda: i'll reply tomorrow, i'm already late for my german class16:48
rlandyzbr: I don;t have +2 in tripleo-common16:49
zbrrlandy: once i get it verified from zuul i will ping those that do have. it should be very easy as it previously already had 2x+216:50
*** panda is now known as panda|off16:58
*** chem has quit IRC17:14
*** jfrancoa has quit IRC17:16
*** bogdando has quit IRC17:22
zbrmarios: please check https://review.openstack.org/#/c/637543/3 now -- if I would have -1 on wofkflow i would block it. affected job is not even triggered by the change.17:30
zbrmarios: anyway, I did incorporate the idea from that change into https://review.openstack.org/#/c/636160/ -- so we should be ok.17:45
*** kopecmartin is now known as kopecmartin|off17:56
*** trown is now known as trown|lunch18:01
*** derekh has quit IRC18:02
chandankumarsshnaidm: Hello18:06
chandankumarsshnaidm: is there a way to enable ara from tripleo-modify-image role?18:06
chandankumar*for18:06
sshnaidmchandankumar, iirc it's difficult because of parallel queries to sqlite from ara, this roles runs multiple times in parallel18:07
*** amoralej is now known as amoralej|off18:07
sshnaidmchandankumar, for enabling it there we need to use a different database18:08
chandankumarsshnaidm: what about trying these stuff https://github.com/openstack/ara-server ?18:08
*** jpena is now known as jpena|off18:09
* chandankumar takes a look for different db18:09
sshnaidmchandankumar, ara-server looks like nice solution too18:09
chandankumarsshnaidm: one server then having client to fetch the data for each node18:10
chandankumarsshnaidm: I will create an epic for that18:10
sshnaidmchandankumar, cool18:10
*** ykarel|away has quit IRC18:16
*** ykarel|away has joined #oooq18:17
*** ykarel|away has quit IRC18:22
*** holser_ has quit IRC18:38
*** dtantsur is now known as dtantsur|afk18:47
*** dsneddon has joined #oooq18:59
*** trown|lunch is now known as trown19:06
rfolcozbr, around?19:23
zbrrfolco: yep. still working on that change.19:23
rlandyrfolco: how did it go with the reproducer?19:23
rfolcoso you saw the tripleo-repo error ?19:24
zbrrfolco: https://review.openstack.org/#/c/637608/ is supposed to fix it.19:24
rfolcorlandy, on hold for now... will resume asap19:24
zbranother example of what happens when we do not add unittesting around our modifications.19:25
rfolcozbr, before you submit can I please rebase on top of marios change ?19:26
zbrrfolco: no, because I already included that change.19:27
zbrrfolco: also no need to worry that it will merge too soon, already has two unmerged depends-on... ;)19:27
rfolcozbr, ok then, can we comment other jobs until we get a green run, then we just re-add overnight ?19:28
rfolcoso we speed up test19:28
zbrrfolco: smart move! can you do it?19:28
rlandypanda|off: ping if you become panda on again19:29
rfolcozbr, sure, you want me to do with this patchset ?19:29
rfolcorlandy, to turn panda|off on, you have to do this panda|off panda|off panda|off19:30
* rfolco hides19:30
* rlandy needs to switch time zones19:30
zbrrfolco: lets disable other jobs only if this run fails, most jobs already started so we should not loose our position in queue.19:32
rfolcozbr, ok19:33
zbrwhile reading discussions about sprints and productivity I realised that we may have a much better way to improve productivity: start shipping beer to people doing reviews. what if people, at the end of week, would receive an amount of beer direct proportional with their reviewing contribution?19:37
zbrprobably we need to put a cap on this as we may endup with one or two that will never be sober ;)19:37
rfolcozbr, if any other non-blocking error, please ignore and remove from the patch (like tripleo-repos)19:41
rfolcozbr, let's go straight to the point, we can fix tripleo-repos later19:42
zbrrfolco: no, you cannot. this bug is blocker.19:42
rfolcozbr, probably because you auto-detect, if you select -d fedora, it is not blocker19:43
zbrrfolco: look at last zuul execution on fedora job, this is how i found it. thus adding it with depends-on19:43
zbrrfolco: last version of the patch no longer has the --distro parma. but no need to worry, this patch will be merged quickly on tripleo-repos. is has very few tests on it.19:43
rfolcoError: 'don't worry' does not exist in your dict.19:44
rfolcozbr, ^19:44
rfolcorlandy, reproducer stuck... openstacksdk is required for this module19:45
rfolcoclean centos7 vm in rdocloud19:45
zbri think we may need to add a trick to lower the use of fedora-28 nodes: make it a child of centos-7, so it will only run after the other one, if the other one passes. it will not make the final execution longer because there are plenty of other jobs that take longer than sum of these two.19:54
rlandyrfolco: what are you running on?19:57
rlandytrown: hi ... I'd like to test this out on the baremetal hardware we have ... https://review.openstack.org/#/c/632052/2/doc/source/install/advanced_deployment/deploy_openshift.rst20:04
rlandytrown: do you have a ready defined test I could use - what's the assumed starting point here?20:06
rfolcorlandy, in a clean centos vm, created cloud.yaml and then ran reproducer-zuul-based-quickstart.sh20:18
rfolcorlandy, (sorry for the delay, somebody knocked my door)20:19
rlandyugh - weshay did this centos testing - we should have fixed that20:19
* rlandy checks20:19
rlandyhttps://github.com/rdo-infra/ansible-role-tripleo-ci-reproducer/blob/master/requirements.txt20:20
rlandyrfolco: can you tell me where it stopped20:20
trownrlandy: there is no automated tests for baremetal openshift ... the docs you pointed to should be correct though20:20
trownrlandy: starting point is an undercloud as far as I can tell20:20
rfolcorlandy, ASK [ansible-role-tripleo-ci-reproducer : Open needed ports at default security group]20:21
rlandytrown: k, so as long as I can get by undercloud install, we should be good to go20:21
rfolcorlandy, will start over20:21
rlandyrfolco:k - I also ran into that  - but I had openstacksdk installed via rpm20:21
rlandytry install it via pip using --user20:22
rfolcook, do you run any requirements.txt before starting ? r20:23
rfolcook, do you run any requirements.txt before starting ? rlandy20:23
rlandyrfolco: no20:23
rlandyyou should not need to run anything20:24
rlandybut to check try the install vai pip with --user20:24
rfolcorlandy, ok thanks20:24
*** agopi has quit IRC20:37
panda|offsomebody rang ?20:47
rlandypanda|off: I did20:50
rlandypanda|off: but I can't really remember why :)20:51
rlandyI think to ask if you were planning to keep the pacemaker stuff in scenario01220:51
panda|offrlandy: for now yes. Then it depends on derek, we'll probably discuss optimization before migrating test to standalone.20:53
rlandypanda|off: ack20:54
panda|offrfolco: last patchset in https://review.openstack.org/636160 is the good one ?21:05
rfolcopanda|off, waiting for zuul to tell if it is good, previous one was bad, zbr fixed with a depends on21:06
panda|off:/ I wanna review stuff. Tomorrow then.21:15
rlandytrown: would I use a CNS or non-CNS deployment?21:29
*** jtomasek has quit IRC21:40
trownrlandy: probably non-CNS is less to go wrong21:54
rlandyk - I'm good with less going wrong21:55
*** trown is now known as trown|outtypewww22:02
*** rascasoft has quit IRC23:06
*** rascasoft has joined #oooq23:44
*** agopi has joined #oooq23:47
*** rascasoft has quit IRC23:50

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!