Monday, 2018-12-17

hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages)00:03
*** dsneddon has joined #oooq00:16
*** dsneddon has quit IRC00:21
*** dsneddon has joined #oooq00:53
hubbot1FAILING CHECK JOBS on master: tripleo-ci-centos-7-scenario008-multinode-oooq-container, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages)02:03
*** ykarel has joined #oooq03:43
*** udesale has joined #oooq03:49
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages)04:03
*** ykarel has quit IRC04:40
*** ratailor has joined #oooq04:48
*** chandan_kumar is now known as chandankumar04:55
*** ykarel has joined #oooq05:01
chandankumarquiquell|off: Yes, seen ping from thursday05:04
*** brault has quit IRC05:06
*** saneax has joined #oooq05:07
*** ykarel has quit IRC05:10
*** ykarel has joined #oooq05:12
*** agopi has quit IRC05:15
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages)06:03
*** quiquell|off has quit IRC06:09
*** jfrancoa has joined #oooq06:41
*** apetrich has joined #oooq07:16
*** quiquell has joined #oooq07:24
quiquellchandankumar: o/07:24
*** jtomasek has joined #oooq07:33
*** yolanda has joined #oooq07:50
chandankumarmarios_|ruck: quiquell \o/07:55
*** jbadiapa has joined #oooq07:57
*** ykarel is now known as ykarel|lunch07:58
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages)08:03
*** skramaja has joined #oooq08:04
*** amoralej|off is now known as amoralej08:31
*** gkadam has joined #oooq08:35
*** gkadam has quit IRC08:35
*** tosky has joined #oooq08:36
*** ykarel|lunch is now known as ykarel08:39
*** ccamacho has quit IRC08:42
*** skramaja_ has joined #oooq08:53
*** skramaja has quit IRC08:53
*** agopi has joined #oooq09:01
*** afazekas is now known as afazekas|sick09:08
*** skramaja_ is now known as skramaja09:13
*** chem has joined #oooq09:26
quiquellsshnaidm: Hello, do you have a minute ?09:27
*** tosky has quit IRC09:29
*** ccamacho has joined #oooq09:31
*** tosky has joined #oooq09:33
*** ssbarnea|rover has joined #oooq09:47
*** derekh has joined #oooq09:58
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, master: tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: tripleo-ci-centos-7-scenario002-multinode-oooq-container, tripleo-ci-centos-7-scenario000-multinode-oooq- (2 more messages)10:03
sshnaidmquiquell, hi, yep10:18
sshnaidmchandankumar, arxcruz|next_yr what do you think about https://review.openstack.org/#/c/509728/ ?10:19
chandankumarsshnaidm: checking10:23
chandankumarsshnaidm: looks good, my last comment is about adding the further functionality in python-tempestconf10:24
sshnaidmchandankumar, can it be added later?10:24
chandankumarfor ironic so that anyone can consumed, yes it can be added later10:24
*** derekh has quit IRC10:46
*** derekh has joined #oooq10:47
*** dsneddon has quit IRC10:52
sshnaidmquiquell, btw, zuul containers stopped to work for me. Seems like I pulled a new problematic container. Gerrit fails with java errors, can't be configured (11:05
sshnaidmquiquell, need to set up our containers versions11:05
quiquellsshnaidm: for gerrit we can, I am working in a review for them to tag it11:07
quiquellsshnaidm: maybe we can tag with the sha1 of the docker hash ?11:07
*** udesale has quit IRC11:10
quiquellsshnaidm: gerrit is not zuul, it has version at your container11:11
quiquellsshnaidm: it's using upstream gerrit I think11:11
ssbarnea|rovermarios_|ruck: o/11:22
sshnaidmquiquell, yeah, but I didn't note version change in gerrit, weird thing11:26
quiquellsshnaidm: let inspect mine11:27
quiquellit's working11:27
sshnaidmquiquell, which tag do you  have for gerrit?11:27
marios_|rucko/ ssbarnea|rover11:28
*** dsneddon has joined #oooq11:28
quiquellsshnaidm: a6301c529fea11:29
quiquellsshnaidm: but it's the latest... I think issue is at zuul iamges11:30
ssbarnea|rovermarios_|ruck: i have some bad news regarding the post failure, it just didn't work.11:32
ssbarnea|roverhttp://logs.openstack.org/81/624381/17/check/tripleo-ci-centos-7-containers-multinode-queens/2cc30b0/job-output.txt.gz11:33
sshnaidmquiquell, maybe.. trying now older version of gerrit image11:33
quiquellsshnaidm: we can pin image id at zuul too, let me pass mine11:34
quiquellsshnaidm: https://paste.fedoraproject.org/paste/aiDxdf894HKHjyJQ0qKpTw11:35
quiquellsshnaidm: If you are able to pin to that, make a review and we merge it11:35
marios_|ruckssbarnea|rover: looks like https://bugs.launchpad.net/tripleo/+bug/1806699 ? http://logs.openstack.org/81/624381/17/check/tripleo-ci-centos-7-containers-multinode-queens/2cc30b0/job-output.txt.gz#_2018-12-16_19_18_28_52591811:36
openstackLaunchpad bug 1806699 in tripleo "HTTPConnectionPool(host='tempest-sendmail.tripleo.org', port=8080): Max retries exceeded with url" [Undecided,Triaged]11:36
sshnaidmquiquell, ha! previous version of gerrit worked11:37
*** rfolco has joined #oooq11:37
quiquellsshnaidm: can you try to pin all of them ?11:37
quiquellwith image id at least11:37
quiquellwe can investigate later on about taggin zuul and the like11:37
ssbarnea|rovermarios_|ruck: yep the sendmail issue I reported last week, I was told that there is some spam protection in place,... clearly is buggy.11:37
sshnaidmquiquell, yeah, will do11:39
sshnaidmquiquell, need to think where to upload our containers that we'll build for reproducer with SF11:39
sshnaidmquiquell, maybe to create a namespace in docker.io for that11:39
quiquellsshnaidm: take a look here https://tree.taiga.io/project/tripleo-ci-board/task/512?kanban-status=144727411:40
quiquellsshnaidm: experiment of using sf RPMs11:40
quiquellsshnaidm: with our repro11:41
quiquellsshnaidm: Was not working, but I think it's not difficult to solve the issues11:41
sshnaidmquiquell, cool, will look11:41
quiquellsshnaidm: TL;DR baze zuul image from centos and sf RPMs11:42
quiquellhas attached the docker-compose.yaml and the Dockerfile of the base image11:42
*** dsneddon has quit IRC11:45
rfolcopanda, sshnaidm can you please review these quick ones ? https://review.rdoproject.org/r/#/c/17739 and https://review.rdoproject.org/r/#/c/1780711:59
rfolcothe full series: https://review.rdoproject.org/r/#/q/status:open+branch:master+topic:f28_promotion11:59
*** dsneddon has joined #oooq12:01
quiquellsshnaidm: Do  you know what 's the vest way to discover if we are in a mixed fedora+centos scenario ?12:02
quiquellpanda: ^ Â?12:02
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: (2 more messages)12:03
sshnaidmquiquell, what do you mean?12:04
quiquellsshnaidm: Going to bypass container prep for fedora+centos mixed scenarios, it does not make sense12:05
quiquellsshnaidm: to do so I ahave to check if we are in that scenario at the tqe standalone role12:05
quiquellsshnaidm: Like looking at docke rimage name or something12:05
sshnaidmquiquell, I think you can compare docker base image (centos) and /etc/redhat-release, just first thought..12:06
quiquellsshnaidm: but that's inside docker image, I mean even before it pulls12:06
pandaquiquell: I think what rlandy is doing is to set a specific docker namespace12:07
sshnaidmquiquell, well, it should be different place12:07
pandaquiquell: so you could compare you current version with the docker_namespace12:07
quiquellpanda: ack12:08
quiquellpanda: well I can use "name_prefix" from container-prepare-parameters for now I think12:10
pandaquiquell: uhm, another thing to consider for when we have the pure fedora container job12:11
pandachange "name_prefix" in the containers prepare12:11
quiquellpanda: yep12:11
pandarfolco: ^12:11
quiquellpanda, rfolco: http://codesearch.openstack.org/?q=%3A%20centos-binary-&i=nope&files=&repos=12:12
rfolcopanda, name_prefix should be an override ? yes, I said "oooovvvveeeerrrrriiiiiddddeeeee"12:15
pandawe'll also need to change it to centos7-binary12:16
pandaif we're goign to maintain both c7 and c8 versions for stable branches12:16
pandarfolco: then we'll have to override the override12:16
sshnaidmrfolco, need to rebase manually https://review.rdoproject.org/r/#/c/1773912:17
pandaand override the fact that we overriding an overridden overridable12:17
quiquellpanda: I was thinking on adding complexity to exercise container prep for fedora28 just for the tripleo updateing12:18
quiquellpanda: but I think It does not worth it, at the fedora+centos thing is temporary12:18
quiquellpanda: is that right ?12:19
pandaquiquell: I will kill all the jobs myself in two months if we don't have the containers.12:19
quiquellpanda: With a gun and all ?12:20
pandaquiquell: nuke.12:20
ssbarnea|roverquiquell: i have some doubts about its temporary status. sooner or later we will have another fedoraXX to check, who known made another distro name.... It would be better to find a way to leverage the effort.12:20
pandaI want the soil to burn, nothing should grow on that area for ages12:21
quiquellssbarnea|rover: Let's hope kolla infra for fedora is back again when that happends so they have the images out of the box12:23
quiquellssbarnea|rover: I think problem now with fedora and kolla is like Alex is resurrecting it12:23
quiquell:-)12:23
quiquellLet's bypass for now so we can merge all the f28 stuff12:24
pandawhy check another fedoraXX ? the only reason we are focusing on 28 is that it's upstream for RHEL812:24
chandankumarquiquell: can we move this card to done https://trello.com/c/o1LIqEuH/841-cixtripleo-fedora-28-python-tempestconf-adjustments-for-python3 ?12:28
chandankumarrfolco: Hello12:30
quiquellchandankumar: let's move it it's working now12:30
rfolcochandankumar, o/12:30
chandankumarquiquell: thanks :-)12:30
chandankumarrfolco: I wanted to talk about these two patches https://review.openstack.org/624081 and https://review.openstack.org/62455012:31
rfolcochandankumar, refreshing my mind... looking12:32
chandankumarrfolco: here we are removing https://review.openstack.org/#/c/624081/ test_white_regex from fs05212:32
ssbarnea|roverpanda: what i was trying to say is that at some point in the future, we will need to do the same kind of work for RHEL9 which would be based on FedoraXX. Hopefully that time we will be able to reuse our investment.12:33
chandankumarrfolco: ok12:34
rfolcochandankumar, my brain is in loop looking for a clean solution there12:36
chandankumarrfolco: like?12:36
rfolcochandankumar, I don't think this is going to be better than overriding both regex and whitelist.12:37
chandankumarrfolco: so in sceanario standalone review, we are overriding test_white_regex and tempest_whitelist, so you mean to say we donot need above two review?12:38
*** udesale has joined #oooq12:39
rfolcochandankumar, yes, because my first suggestion is technically incorrect: if whitelist is not empty, ignore regex.12:39
chandankumarrfolco: ok, abandoning two reviews12:39
rfolcochandankumar, sorry about the confusion. Thanks for looking at this.12:40
chandankumarrfolco: nope, it is always provides space to learn and experiment things :-)12:41
rfolcopanda, name_prefix... should I override in the job definition for f28 promotion jobs ?12:41
*** ratailor has quit IRC12:45
*** kopecmartin|off is now known as kopecmartin12:47
chandankumarpanda: rfolco kopecmartin please have a look at this issue https://tree.taiga.io/project/tripleo-ci-board/issue/52112:51
chandankumarpanda: rfolco we need to transfer it to proper dfg maintaining heat12:52
*** rlandy has joined #oooq12:57
rlandysshnaidm: hi - can we chat for a few after the meeting. I was testing ovb on friday12:58
mariosscrum?13:00
pandassbarnea|rover: the point in the future is 5 years. I can't even imagin how many things will have changed at that point13:00
rfolcoping scrum13:01
sshnaidmrlandy, sure13:02
*** amoralej is now known as amoralej|lunch13:12
mariosssbarnea|rover: call13:30
pandachandankumar: is this only for heat or we need to involve other DFG for the others component as well ?13:34
chandankumarpanda: nope13:34
chandankumarpanda: just heat13:34
chandankumarsshnaidm: quiquell ssbarnea|rover please have a look at this patch https://review.openstack.org/622865 I need some help13:37
quiquellchandankumar: all jobs railing there13:42
pandarfolco: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/tripleo-upstream/upload-cloud-images.sh13:43
rfolcopanda, cool, thanks13:43
quiquellchandankumar: I think we are nog goint to be able to have a py3 job with devstack until we canno tchange ansible_python_interpreter at zuul13:44
sshnaidmrlandy, let's talk?13:44
pandarfolco: I can modify the script in the same way I modified the other13:45
rlandysshnaidm: sure - just opening a window to my test box13:45
pandarfolco: to be distro-aware13:45
rfolcopanda, L3 should be changed ?13:46
ssbarnea|roverrlandy: regarding reproducer, wans't it expected to self-configure the machine where it is run? like installing deps.13:46
rlandyssbarnea|rover: ack13:46
rlandysshnaidm: k - ready to talk  when you are13:46
rlandymy blue?13:47
sshnaidmrlandy, https://bluejeans.com/u/sshnaidm/13:47
rfolcopanda, can't find what is the correct url13:48
pandarfolco: L3 should maube be change to account for dnf13:48
rfolcopanda, images.rdoproject.org/images/centos7/trunk_rdo13:48
pandarfolco: the correct url for the images  in images.r.o ?13:48
rfolco?13:48
rfolcooops13:49
rfolcohttp://images.rdoproject.org/master/trunk_rdo13:49
chandankumarquiquell: sure, will wait for job to finish13:49
rfolcono, not this one either :(13:49
quiquellchandankumar:ack13:49
pandarfolco: right now we have https://images.rdoproject.org/master/rdo_trunk/13:49
*** quiquell is now known as quiquell|lunch13:49
rfolcooh https13:49
pandarfolco: this should be changed to something like  https://images.rdoproject.org/Fedora28/master/rdo_trunk/13:49
rfolcostupid me13:50
pandamaybe we can leave master as a symlink to CentOS7/master13:50
rfolcopanda, can I try making this ? I know you know how to know to know the knowledge of knowing the know-how to change this13:50
pandarfolco: get out.13:51
pandarfolco: get out now.13:51
rfolcolol13:51
rfolco:(13:51
rfolcopanda, panda panda13:51
pandarfolco: it's all yours13:51
rfolcopanda, thanks, will add you to the review13:52
pandamy home, my car. Yours.13:52
pandarfolco: since you thought this was easy, I'm adding some complexity13:57
panda1. you need to modify also https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/tripleo-upstream/convert-upload-undercloud.sh13:57
rlandypanda: latest test run got as far as a repo setup failure - which may be legit ... https://logs.rdoproject.org/55/17855/1/check/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/c4df844/job-output.txt.gz#_2018-12-17_13_47_19_30837713:58
pandarfolco: 2. you'll need to work with ronell to modify https://review.openstack.org/#/c/625034/4/config/release/tripleo-ci/Fedora-28/promotion-testing-hash-master.yml@26 and get the proper image url14:00
pandarlandy: progress!14:00
pandarlandy: https://review.openstack.org/#/c/625034/4/config/release/tripleo-ci/Fedora-28/promotion-testing-hash-master.yml@31 perhaps ?14:01
*** skramaja has quit IRC14:02
pandachange yum to dnf14:02
rlandyquiquell|lunch: pls  see #rhos-ops14:02
rlandypanda: ack - just setting up a meeting re: nodepool image and then will rerun14:03
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: (2 more messages)14:03
ssbarnea|roverdoes anyone knows if is possible to have a single change on gerrit with two depends-on one pointing to upstream gerrit and one to rdo one?14:04
*** amoralej|lunch is now known as amoralej14:06
rlandyrfolco: I am setting up a meeting re:public nodepool images14:09
rlandyrfolco: ca we use your bjs14:09
rlandyas it will be after the community call14:10
rfolcorlandy, yes of course14:10
rfolcopanda, ack, adding these to the task14:11
rfolcossbarnea|rover, rdo gerrit supports depends-on to upstream review.openstack.org14:13
rfolcossbarnea|rover, the other way around probably doesn't14:14
*** quiquell|lunch has quit IRC14:15
*** quiquell has joined #oooq14:15
quiquellrlandy: ack14:15
rlandyquiquell: no worries - just sent you an invite for thurs14:16
rlandyre: public nodepool images14:16
quiquellyey !!! they are going to have them public ?14:18
rlandyquiquell: we're going to talk about how to do that14:19
quiquellrlandy: ok ok14:19
rlandyquiquell: kforde thinks it's doable but need some thought/process as the image updates daily14:19
quiquellok, if there is no stopper then we are good14:20
rfolcoamoralej, can we make https://images.rdoproject.org/master/rdo_trunk/ distro based ? like https://images.rdoproject.org/centos7/master/rdo_trunk/ symlink to https://images.rdoproject.org/master/rdo_trunk/ and create https://images.rdoproject.org/fedora28/master/rdo_trunk/14:23
*** jpena has joined #oooq14:24
amoralejjpena, ^14:24
amoraleji'm ok with it14:24
jpenaI just joined the channel, so I'm missing context14:25
rfolcojpena, <rfolco> amoralej, can we make https://images.rdoproject.org/master/rdo_trunk/ distro based ? like https://images.rdoproject.org/centos7/master/rdo_trunk/ symlink to https://images.rdoproject.org/master/rdo_trunk/ and create https://images.rdoproject.org/fedora28/master/rdo_trunk/14:27
rfolcokeep existing default one14:27
rfolcoadd centos7 and fedora28 dirs, symlink centos7 to existing one. We'll change scripts to upload f28 images to fedora28 url14:28
rfolcojpena, sounds reasonable?14:28
jpenarfolco: sounds good to me. It'll require some small changes to our playbooks and cleanup scripts, but it's ok14:29
jpenawhile at it, do you have plans to add multiarch support?14:29
rfolcojpena, not me. This may be of interest for tbreeds14:33
jpenarfolco: I'm going to create the directories14:36
rfolcojpena, thank you14:37
rlandypanda: ha ... https://review.rdoproject.org/zuul/stream/f8f143dc92244fed9a4137764bcd3cf8?logfile=console.log .. getting there14:37
pandaweeeee14:41
pandadeploying the standalone ... expect fireworks14:41
rfolcorlandy, https://media.giphy.com/media/26ybvOkkmathUXss0/giphy.gif14:43
quiquell\o/ !!!14:43
quiquellIt's going to fail ta container prep :-)14:44
rfolcodon't screw up rlandy's moment14:44
* quiquell goes back to his corner14:44
rfolco /kick quiquell14:45
rfolco:)14:45
rlandyit was nice while it lasted14:45
rlandyquiquell: is that an expected failure?14:46
rlandyor you were just predicting?14:46
quiquellrlandy: yep, fixing it here https://review.openstack.org/#/c/62558014:46
quiquellrlandy: is the container prep stuff for mixed fedora+centos14:46
rlandyquiquell: cool - will rerun with your change14:46
quiquellrlandy: not working give a sec, have just put in place without local testing it14:48
rlandyquiquell: k - ping when it's ready to do a test run14:48
rfolcopanda, I think https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/tripleo-upstream/upload-cloud-images.sh#L3 stays as is... look http://logs.openstack.org/20/619520/21/check/tripleo-ci-fedora-28-standalone/864b99c/logs/undercloud/etc/yum.repos.d/delorean.repo.txt.gz14:54
pandathere's no /etc/dnf.repos.d ? :)14:58
quiquellrlandy: should work now https://review.openstack.org/62558014:59
rlandyk - trying14:59
marios /win 815:02
panda /lnx 415:04
rlandyno dice15:10
pandaquiquell: docker_prep_prefix is search(ansible_distribution|lower15:13
pandaquiquell: should be "in" ?15:13
pandadocker_prep_prefix is "centos-binary-" ?15:13
pandanot even in15:14
quiquellpanda: nope is "is"15:14
quiquellsearch is binary15:14
quiquellpanda: used a pipe "|" with search but ansible yell at me with obsolet stuff15:15
marios_|ruck315:15
marios_|ruckgerrit down for everyone or just me .com15:15
pandawhat's the sintax of serch ?15:15
pandajustme.com works for me15:16
quiquellpanda: "centos-binary-" | search("fedora")15:16
quiquellpanda: give you a warning15:16
pandaquiquell: is it the same as regex_search ?15:16
pandaquiquell: what's the warning ?15:17
pandabecause I don't undertand how would it work without the |15:17
quiquellpanda: "[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|search` use `result is search`."15:18
* rlandy look s- was just updating other reviews15:21
*** saneax has quit IRC15:22
*** derekh has quit IRC15:24
ssbarnea|roverrlandy: one side-effect of running reproducer: https://review.openstack.org/#/c/625621/15:25
rlandyoh the old reproducer15:25
rlandythat's probably fine15:26
*** chandankumar is now known as chkumar|out15:26
ssbarnea|roveri don't really know old/new, is the one i got when I tried to run one job from the gate :D15:26
*** ykarel is now known as ykarel|away15:26
pandaquiquell: I think this deployment is failing on a task in tht, not from quickstart-extras15:27
pandaquiquell:  tripleo-container-image-prepare lo15:27
pandaquiquell: https://logs.rdoproject.org/55/17855/1/check/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/f8f143d/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-12-17_14_44_2515:27
*** derekh has joined #oooq15:27
quiquellpanda: this one is failing because is using python2 there15:28
quiquellpanda: you have two options use the review tha disable container prep (this is the one I paste it here)15:28
panda/o\15:28
quiquellpanda: or use the review that use ansible-playbook-3 there https://review.openstack.org/#/c/625513/15:29
quiquellpanda: but executing container prep of centos7 images using fedora28 is not something that we want :-/15:29
ssbarnea|roverrlandy: i tried to run it on a freshly installed f28 machine and I had to run it many times just to find more and more missing system deps. my original impression what that it would try to auto-install deps if it detects them as missing.15:29
quiquellpanda: we can do a blue if you want so I explain15:30
pandaquiquell: how does your patch disable container-prep ?15:31
ssbarnea|roverrlandy: should I continue to report issues and/or attempt to fix them or not really?15:32
pandaquiquell: I'm blue15:32
ssbarnea|roverfor example I had to do: yum install python-virtualenv ansible python2-openstackclient python2-heatclient python2-lxml15:32
rlandyssbarnea|rover: sure - absolute;y15:32
rlandyuntil we have  a replacement15:32
quiquellpanda: updating the containre prep params yaml15:33
ssbarnea|roverand git, missing too (really fresh machine)15:33
quiquellpanda: I mean "NOT" updating it15:33
quiquellpanda: In case you don't want15:33
ssbarnea|roverrlandy: ok, just wanted to know I would not invest effort in something to be dropped very soon.15:33
rlandyssbarnea|rover: I think unless it's a huge effort - yes15:34
*** ykarel|away has quit IRC15:35
pandaquiquell: blue blue blue15:38
quiquellok ok15:38
ssbarnea|roverrlandy: panda it is weird because I do remember we had some CR that was installing dependencies in reproducer but I don't see any code to installed in in the script. wan't this merged, was this feature removed?15:38
quiquellpanda: I am at your blue15:39
ssbarnea|roverthe bootstrapping code only checks for dependencies, and missing to spot "git", for example.15:39
quiquellpanda: you there ? I have to drop soon15:42
pandaquiquell: givve 1 minute15:42
pandaquiquell: https://logs.rdoproject.org/55/17855/1/check/periodic-tripleo-ci-fedora-28-centos-7-containers-standalone-master/f8f143d/job-output.txt.gz#_2018-12-17_14_36_53_90765215:51
quiquellpanda: https://review.openstack.org/#/c/625513/15:52
quiquellpanda: https://review.openstack.org/#/c/624656/15:52
*** ykarel|away has joined #oooq15:52
ssbarnea|roverdoes anyone remember ther resize error with reproducer? guess what, it still reproduce on clean machine, even when running as root. Proof: https://seashells.io/v/mjMHHxsz15:55
*** udesale has quit IRC15:57
ssbarnea|roverI tried to run the failing task using my root use (same user used to run reproducer), and it worked: https://github.com/openstack/tripleo-quickstart/blob/55580cc7ad2123a003f05547b30d47ee8246d135/roles/libvirt/setup/overcloud/tasks/libvirt_nodepool.yml#L1216:01
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: (2 more messages)16:03
pandarlandy: added two more depends-on on the periodic test patch16:05
rlandypanda: looking16:05
*** quiquell is now known as quiquell|off16:06
marios_|rucksshnaidm: o/ can you recheck here please when you get a chance https://review.openstack.org/#/c/620596/3/zuul.d/layout.yaml16:06
marios_|ruckykarel|away: when you're next around thanks ^16:07
*** ykarel|away is now known as ykarel16:12
rlandypanda: I updated the pipeline review  .. updating the  promotion criteria review16:14
ssbarnea|roverrlandy: the irony is that if I run reproducer as non-root user, it does suceeds resizing the image. only as root it fails.16:17
ssbarnea|roverrlandy: my guess is that the image resize needs to be run by the same user as the one running reproducer. so for root to have become, and for non root not to have become.16:18
rlandyI have hit that issue before16:19
rlandyI am just trying to think how I got around it16:19
*** ccamacho has quit IRC16:19
*** ccamacho has joined #oooq16:21
rfolcorlandy, f28 periodic should use fs99 ?16:34
rfolcorlandy, both pure and mix I mean16:34
rlandyrfolco: only mixed periodic16:35
rlandypure should use 05216:35
rfolcorlandy, I left a comment on https://review.openstack.org/#/c/625034/4/config/release/tripleo-ci/Fedora-28/promotion-testing-hash-master.yml@26 anyway, this will need https://review.rdoproject.org/r/1788816:37
marios_|ruckssbarnea|rover: o/ see you tomorrow me out16:38
rlandywait, wait , it ran16:39
rlandypanda: ^^16:39
pandait ran ?16:41
pandasuccess ?16:43
pandazuul stop lying to me!16:43
rfolcozuul is now known as pinocchio16:44
rfolcozuul left the channel16:44
pandait didn't lie16:45
pandait really passed16:45
rfolcozuul joined the channel16:45
pandanow we only need to merge those 10 patches16:45
pandaand counting16:45
tosky"I'm not lying" and the queue increases16:45
rlandypanda: updated reviews ... https://review.rdoproject.org/r/#/c/17780/16:45
rlandyhttps://review.rdoproject.org/r/#/c/17719/3/zuul.d/tripleo.yaml16:45
pandaoh, la fata toskyna16:45
rlandypanda: k - where to from here, oh UA?16:46
pandato infinity .. and bayond!16:51
pandawe need to merge all the dependencies, then add the pipeline modification, then the scripts16:52
pandapiece of cacke16:52
pandacake16:52
rfolcorlandy, panda SIGSYNC16:52
* panda SEGSEGV16:52
rlandyeek16:52
rlandypanda: k - let's talk about the order of merging16:53
rlandypanda: we have two core votes, we could get quiquell|off's patches in16:55
* rlandy checks failures there16:55
pandarechecking 624656, post-run timed ou16:58
rlandyyeah17:01
rlandyf28 failures there17:01
panda625513  is working, but it's a miracle :)17:04
rlandywell - t'is the season, right?17:12
*** derekh has quit IRC17:21
*** jtomasek has quit IRC17:21
*** trown is now known as trown|lunch17:25
*** dsneddon has quit IRC17:26
rfolcopanda, rlandy: should I make an override for the pedigree job ? https://review.openstack.org/#/c/625034/4/config/release/tripleo-ci/Fedora-28/promotion-testing-hash-master.yml17:28
rlandyrfolco: no fix that review17:30
rlandyit is meant to be for the pure pipeline17:30
rlandyif we need to override for mixed, that should be in the fs17:30
ssbarnea|roverrlandy: panda : rfolco regarding post timeouts, i need some support to make this happen https://review.openstack.org/#/c/625573/17:30
rlandywhat are you changing17:30
rfolcorlandy, https://review.rdoproject.org/r/#/c/17888/1/ci-scripts/tripleo-upstream/upload-cloud-images.sh@1317:32
rfolcopanda, rlandy: and the change that enables distro dir in the image url is https://review.rdoproject.org/r/#/c/17886/ from jpena17:34
*** ykarel is now known as ykarel|away17:37
pandarfolco: ssbarnea|rover or we can also move this forward https://review.openstack.org/617617 , just sayin'17:45
rfolcopanda, I'll stop nitpicking there, +117:46
ssbarnea|roverpanda: here is the deal, i think i found the culprit of the timeouts already so we don't need so much boiler place... which will not work. I already tested.17:47
ssbarnea|roverpanda: i already tried to do something very similar. in fact i am now almost sure this is the fix https://review.rdoproject.org/r/#/c/17732/17:49
ssbarnea|roveryou remember the warning from ansible about ignoring ansible.cfg file due to permissions?17:49
ssbarnea|roverguess what,... our ansible.cfg has this https://github.com/openstack/tripleo-quickstart/blob/457e61fb73eb55153cd4b8105c6090b9730c13be/ansible.cfg#L2117:50
ssbarnea|roverServerAliveInterval is needed to prevent ansible to get frozen... exactly what happens to us.17:50
ssbarnea|roverbut because of wrong file permissions, our jobs are not loading the ansible.cfg file.17:50
ssbarnea|rover... an working that nobody cared about.17:51
rfolcossbarnea|rover, why does it happen randomly then?17:52
ssbarnea|roverrfolco: betcause networking, google for it and you will see that is random.17:53
pandaalso, my patch ensure taht whatever the reason is, we get at least some logs in place ..17:53
pandadamn networking!17:53
pandathat's it, no more networks from now on.17:53
ssbarnea|roverrfolco: panda read https://review.rdoproject.org/r/#/c/17732/ -- with all proposed workarounds.17:53
ssbarnea|rover:D17:54
pandajust usb keys sent through FedEx17:54
rfolcossbarnea|rover, fair and IMHO panda's patch make it more robust17:54
ssbarnea|roverrfolco: without fixing the frozen issue his patch is not going to help in any way.17:54
ssbarnea|roverwhen it happens ansible gets frozen, for good.17:55
pandammhh, pabelanger comment ...17:55
ssbarnea|roverjust to be noted: i am not against that patch, but first we need to sort the timeouts. after this we can get that patch.17:55
pandawhat patch ? we need numbers.17:56
*** tosky has quit IRC17:57
ssbarnea|roverhttps://review.openstack.org/#/c/620902/ -- is pointing to the two changes that are supposed to address the problem of ansible.cfg file.17:58
ssbarnea|roveronce we no longer see warnings about being skipped we should be free of the frozen bug.17:58
ssbarnea|roverpanda: what is curious is that the frozen part happens only and always on the same shell task. using "timeout" has zero value as it doesn't help. One alternative could be to use async.18:00
ssbarnea|roverbut i prefer to see ansible.cfg issue sorted first,  before looking for async.18:01
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: (2 more messages)18:03
rfolcopanda, rlandy I am stuck on chicken-n-egg for pure f28 jobs. If we can merge rdo-jobs definition at least, even though not having all the pieces, it does not break anything.18:04
rfolcohttps://review.rdoproject.org/r/#/q/status:open+branch:master+topic:f28_promotion18:04
ssbarnea|rovergoing offline for the next hour18:06
rlandyrfolco: what is defined as DISTRO_NAME?18:13
rlandythe answer just says pipeline18:14
*** jpena is now known as jpena|off18:14
rfolcorlandy, override var here https://review.rdoproject.org/r/#/c/17807/4/zuul.d/ovb-jobs.yaml@218 that is exported here https://review.rdoproject.org/r/#/c/17740/7/playbooks/tripleo-ci-periodic-base-upload/post.yaml@2618:15
rfolcoand used here https://review.rdoproject.org/r/#/c/17888/18:15
rfolcopanda, ^18:15
pandaoverride ?18:15
rfolconah... var set in job definitio18:16
rfolcolol I am addicted to the word 'overrrrrideeeeeeee'18:16
pandarfolco: -118:17
pandarfolco: no need to pass a variable18:17
pandarfolco: you can autodetect18:17
rfolcok18:17
pandarfolco: did you identify the point that calls this script ?18:17
rfolcoyes https://review.rdoproject.org/r/#/c/17740/7/playbooks/tripleo-ci-periodic-base-upload/post.yaml@2618:18
rfolcopanda, I think its time to create a discovery role at the beginning and set ansible var for distro... then we just use it (exporting to bash if needed)18:19
rfolcopanda, otherwise people will start auto-detecting distro on their bash scripts18:19
rlandyrfolco: panda: maybe we should bj for a few and see what can be merged?18:20
rfolcoyes yes18:20
pandarfolco: {{ ansible_distribution }}18:20
rfolcopanda, this is not what I need. This returns Fedora, not fedora2818:21
rlandypanda: including your reviews for the promoter18:21
rlandy28 is major release18:21
pandarfolco: {{ansible_distribution|lower}}{{ansible_distribution_major_version }}18:21
rlandycorrect - version18:22
pandaansible got us covered18:22
pandaI'm ready to bj whenever18:22
rfolcopanda, you are right. I left.18:22
* rlandy as well18:22
rfolco+118:22
rlandyrfolco: your bj?18:23
pandarfolco: you're left and I'm right ?18:23
rfolcomy blue18:23
rfolcopanda, :)18:23
rfolcopanda, get used to my fool puns18:23
pandarfolco: I'm rubber, you're glue.18:24
* rlandy waits on bj18:25
*** ykarel|away has quit IRC18:25
rfolcopanda, waiting 4 u18:25
*** jfrancoa has quit IRC18:27
ssbarnea|roveri am back18:28
ssbarnea|roverpanda: regarding pabelanger comment, what can i reply? AFAIK, most of tripleo jobs do use that role.18:29
*** amoralej is now known as amoralej|off18:33
pandassbarnea|rover: is there another way to do the same in non legacy roles ?18:35
pandassbarnea|rover: otherwise reply what you replied to me18:35
pandassbarnea|rover: lots of things need to be updated until we can get rid of legacy code18:35
ssbarnea|roverpanda: i will do so, i asked because I do not know the reasons and history behind. meanwhile we need to fix old roles in order to move on. i hope that testing will prove to fix that problem.18:36
*** trown|lunch is now known as trown18:41
hubbot1FAILING CHECK JOBS on master: tripleo-ci-fedora-28-standalone @ https://review.openstack.org/604298, stable/pike: tripleo-ci-centos-7-scenario001-multinode-oooq-container @ https://review.openstack.org/602248, master: tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-rocky-branch, tripleo-ci-centos-7-containers-multinode-rocky, tripleo-ci-fedora-28-standalone @ https://review.openstack.org/560445, stable/rocky: (2 more messages)20:03
sshnaidmssbarnea|rover, marios_|ruck I changed some points in OVB config, please keep eye on it and tell me if you see something suspicious20:07
*** sshnaidm is now known as sshnaidm|off20:07
ssbarnea|roversshnaidm|off: any hints around a specific area? i am going out in few minutes anyway but i will check in the morning.20:08
sshnaidm|offssbarnea|rover, if it doesn't execute in patches - it's suspicious20:08
sshnaidm|offssbarnea|rover, or it passes a lot - it's critically suspicious20:09
sshnaidm|offit means something is wrong20:09
*** kopecmartin is now known as kopecmartin|off20:43
*** dsneddon has joined #oooq20:55
*** hubbot1 has quit IRC21:04
*** dmellado has quit IRC21:05
*** gouthamr has quit IRC21:06
*** gouthamr has joined #oooq21:17
*** yolanda has quit IRC21:18
*** gouthamr_ has joined #oooq21:30
*** gouthamr_ has quit IRC21:37
rlandyyay - I am compliant21:38
*** gouthamr_ has joined #oooq21:41
*** gouthamr_ has quit IRC21:46
*** gouthamr_ has joined #oooq21:51
*** gouthamr_ has quit IRC21:56
*** trown is now known as trown|outtypewww21:58
*** holser_ has joined #oooq22:39
*** holser_ has quit IRC22:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!