Monday, 2019-12-16

*** tosky has quit IRC00:02
*** d0ugal has quit IRC01:06
*** d0ugal has joined #oooq01:08
*** ysandeep has joined #oooq01:10
*** fmount has quit IRC04:08
*** dmellado has quit IRC04:08
*** baha has quit IRC04:08
*** jjoyce has quit IRC04:08
*** mjturek has quit IRC04:08
*** openstackstatus has quit IRC04:11
*** openstackstatus has joined #oooq04:12
*** ChanServ sets mode: +v openstackstatus04:12
*** bhagyashris has joined #oooq04:41
*** ykarel has joined #oooq04:53
*** holser has joined #oooq04:54
*** skramaja has joined #oooq05:08
*** epoojad1 has joined #oooq05:16
*** raukadah is now known as chkumar|rover05:22
*** surpatil has joined #oooq05:33
*** soniya29 has joined #oooq05:33
*** marios|ruck has joined #oooq06:11
*** holser has quit IRC06:17
*** saneax has joined #oooq06:42
*** sanjayu_ has joined #oooq06:48
*** saneax has quit IRC06:51
*** jfrancoa has joined #oooq06:53
chkumar|rovermarios|ruck: Good morning07:03
chkumar|rovermarios|ruck: Current rdo trunk dlrn is not yet consistent, waiting for these patches https://review.rdoproject.org/r/#/c/24178/ and https://review.rdoproject.org/r/#/c/24165/07:04
chkumar|roverto get merged first07:04
marios|ruckchkumar|rover: morning mate07:04
marios|ruckpanda: morning o/ https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-wednesday-weekend queens looks green (fs20 not in criteria) do you think we can run a promote on it today ?07:07
*** akahat has joined #oooq07:08
*** soniya29 has quit IRC07:26
*** ykarel is now known as ykarel|lunch07:37
*** tesseract has joined #oooq07:48
*** bhagyashris has quit IRC07:59
*** tosky has joined #oooq08:02
*** tesseract has quit IRC08:11
marios|ruckchkumar|rover: should be able to clear quite a few of thos bugs today with selinux08:15
marios|ruckchkumar|rover: ( i mean ruck rover ether)08:15
chkumar|rovermarios|ruck: yes08:16
chkumar|rovermarios|ruck: we can update this patch https://review.opendev.org/#/c/698883/08:16
chkumar|roverit will fix one of the issue08:17
marios|ruckchkumar|rover: yeah i was planning to update it. needs conditional when rhel08:17
marios|ruckchkumar|rover: we already have such conditionals there08:17
marios|ruckchkumar|rover: https://review.opendev.org/#/c/698883/1/roles/oooci-build-images/templates/build-images.sh.j2@3308:18
marios|ruck# this is only set in rhel8 jobs dib_rhel_image -> DIB_LOCAL_IMAGE08:18
marios|ruckchkumar|rover: doing08:18
*** amoralej|off is now known as amoralej08:18
chkumar|rovermarios|ruck: currently the oooci-build-images is used for rhel8 only, fs002 for centos08:19
marios|ruckchkumar|rover: updated08:24
chkumar|rovermarios|ruck: commented08:29
marios|ruckchkumar|rover: so for fs1 something is overriding that https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/RedHat-8/master.yml#L121 ?08:29
*** ykarel|lunch is now known as ykarel|08:30
*** ykarel| is now known as ykarel08:30
chkumar|rovermarios|ruck: checking that one08:30
chkumar|rovermarios|ruck: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/overcloud-deploy/defaults/main.yml#L18508:31
marios|ruckchkumar|rover: ack08:31
marios|ruckchkumar|rover: that would do it08:31
chkumar|rovermarios|ruck: let me propose a patch for that08:31
marios|ruckchkumar|rover: sure updating for you comment adding var08:32
chkumar|rovermarios|ruck: Related-Bug: 185302808:40
openstackbug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged] https://launchpad.net/bugs/185302808:40
chkumar|rovermarios|ruck: sorry08:40
chkumar|rovermarios|ruck: https://review.opendev.org/69917008:40
*** sshnaidm|off is now known as sshnaidm08:41
marios|ruckchkumar|rover: i include it there https://review.rdoproject.org/r/#/c/23919/08:42
marios|ruckchkumar|rover: done updated thanks for checking again https://review.opendev.org/#/c/699170 (testing there ^)08:44
marios|ruckhmm seems False not false?08:46
*** jpena|off is now known as jpena08:52
chkumar|rovermarios|ruck: false09:00
*** d0ugal has quit IRC09:00
*** d0ugal has joined #oooq09:01
*** tesseract has joined #oooq09:03
*** tesseract has quit IRC09:04
*** tesseract has joined #oooq09:04
chkumar|rovermarios|ruck: from here http://logs.rdoproject.org/19/23919/7/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-master/cc0fb48/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz09:04
chkumar|roverit is already permissive09:05
chkumar|roverercloud_selinux_enabled": "permissive",09:05
*** apetrich has joined #oooq09:06
*** bhagyashris has joined #oooq09:08
chkumar|rovermarios|ruck: moving to wework, will back in half hour09:10
marios|ruckchkumar|rover: ack09:11
marios|rucksshnaidm: can you check again? https://review.opendev.org/#/c/698883/409:14
sshnaidmmarios|ruck, if negative, then " if not dib_...|bool"09:15
marios|rucksshnaidm: k can do that one what about true True09:15
sshnaidmmarios|ruck, lowercase is a nit, but we keep the same style.. at least try09:16
marios|rucki think it would work either way.i had it lower first, but as per comments looks like in jinja tests doc they reference upper09:16
sshnaidmmarios|ruck, it will work09:16
marios|rucksshnaidm: eg https://jinja.palletsprojects.com/en/master/templates/#whitespace-control09:16
marios|rucksshnaidm: but also they have something like "{% set iterated = false %}" there too so...09:17
marios|rucksshnaidm: k updating again09:17
*** akahat has quit IRC09:17
sshnaidmmarios|ruck, maybe better to add "sudo getenforce" after "setenforce" to ensure it did the job?09:19
marios|rucksshnaidm: i had it in debug didn't think we wantted to keep it for the patch tho09:19
marios|rucksshnaidm: not sure09:19
sshnaidmk, not critical09:19
marios|rucksshnaidm: done revote09:20
marios|rucksshnaidm: reckicked it too https://review.rdoproject.org/r/#/c/23919/09:20
marios|rucksshnaidm: thanks :)09:20
sshnaidmmarios|ruck, I think build jobs should run on your patch09:21
sshnaidmtripleo-rhel-8-buildimage-overcloud-full and tripleo-rhel-8-buildimage-ironic-python-agent are running09:22
marios|rucksshnaidm: ack but we have some bugs that needs periodic run for closing cos we didn't get green rhel runs for loooong time09:24
sshnaidmmarios|ruck, chkumar|rover nice interface to set of periodic runs: https://review.rdoproject.org/zuul/buildset/0b91f89f95d44351b5e12e27d80eec5c09:24
marios|rucksshnaidm: nice but i guess the buildset changes on each run (i.e. i can't have it open and refresh like that one https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master09:25
sshnaidmmarios|ruck, I see, although it's dangerous09:25
marios|rucksshnaidm: which bit?09:26
sshnaidmmarios|ruck, if we run periodic with patch and promote this hash, but patch for example is not merged.. we promoted something broken09:26
marios|rucksshnaidm: ah i see ... well not for promotion so much in this case. but more like 'we have green run on fs1 finally after fixes for these other things so we can now close this one"09:28
*** yolanda has quit IRC09:28
marios|ruckkind of thing09:28
*** yolanda__ has joined #oooq09:28
marios|rucksshnaidm: but i get your point about the depends-on09:28
sshnaidmyeah, in this case that's ok09:31
chkumar|roversshnaidm: yes it looks awesome09:32
*** derekh has joined #oooq09:44
*** chem_ has joined #oooq09:50
*** jbadiapa has joined #oooq09:52
*** akahat has joined #oooq09:59
*** mjturek has joined #oooq10:05
*** jjoyce has joined #oooq10:05
*** ysandeep has quit IRC10:06
*** dmellado has joined #oooq10:07
*** irclogbot_2 has quit IRC10:08
*** irclogbot_3 has joined #oooq10:09
*** fmount has joined #oooq10:10
*** fmount has quit IRC10:13
sshnaidmchkumar|rover, should it be abandoned? https://review.opendev.org/#/c/698876/10:13
chkumar|roversshnaidm: Done10:14
*** jaosorior has joined #oooq10:16
*** fmount has joined #oooq10:18
marios|rucksshnaidm: chkumar|rover: so it doesn't like the https://review.opendev.org/#/c/698883/6/roles/oooci-build-images/templates/build-images.sh.j2 @ https://review.rdoproject.org/r/#/c/23919/  going to try the "is False" again?10:20
marios|rucknot sure that is the problem yet10:20
sshnaidmmarios|ruck, where do you see it "doesn't like"?10:20
marios|rucksshnaidm: http://logs.rdoproject.org/19/23919/8/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/78121a3/build.log10:21
sshnaidmmarios|ruck, I see permissive is set10:21
sshnaidmmarios|ruck, http://logs.rdoproject.org/19/23919/8/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/78121a3/logs/script_build.log10:21
marios|rucksshnaidm: fails with 2019-12-16 10:09:22.170 | failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network: No such file or directory10:21
sshnaidmmarios|ruck, http://logs.rdoproject.org/19/23919/8/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/78121a3/logs/script_build-err.log10:21
marios|rucksshnaidm: which is the bug error (checking)10:21
sshnaidmmarios|ruck, seems like not selinux problem10:22
sshnaidmmarios|ruck, "is False" won't help here10:22
marios|rucksshnaidm: was green with that https://review.opendev.org/#/c/698883/1/roles/oooci-build-images/templates/build-images.sh.j210:22
marios|ruckchkumar|rover: i have fantastic news \o ^^10:22
sshnaidmmarios|ruck, it's transient error, sometimes it passes, sometimes not10:22
chkumar|roversshnaidm: we selinux enforcing it does not work, with permissive it works10:23
chkumar|roversshnaidm: we also updated the image there na?10:23
sshnaidmchkumar|rover, no, it doesn't, see my links above, it's set to permissive in job10:23
sshnaidmchkumar|rover, well, I don't know, did weshay change it finally?10:24
*** bhagyashris has quit IRC10:25
marios|ruckoh ffs10:25
chkumar|roversshnaidm: https://review.rdoproject.org/r/#/c/24160/10:25
chkumar|roversshnaidm: and https://review.rdoproject.org/r/#/c/24161/ I think he forgot to update the rhui rpm there10:25
marios|ruckpanda: o/ do you think you can run a Q promote today looks good there https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-wednesday-weekend10:27
marios|ruckpanda: yesterdays run was green fs20 not in criteria10:27
marios|ruckpanda: please thanks. also lets discuss today about plan to stop doing this and move over promoter? this week?10:28
marios|ruckchkumar|rover: ^10:28
marios|ruckpanda: (scrum i think is good time)#10:28
chkumar|rovermarios|ruck: Yes10:31
chkumar|rovermarios|ruck: we also need to discuss about pipeline timing10:31
marios|ruckchkumar|rover: sshnaidm: so wtf we now no longer sure its selinux ? https://bugs.launchpad.net/tripleo/+bug/185302810:31
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]10:31
marios|ruckchkumar|rover: sshnaidm: since we get fail at https://review.rdoproject.org/r/#/c/23919/10:31
marios|ruckchkumar|rover: yes on pipeline timing too10:32
pandamarios|ruck it has to be today, I'll be on PTO from tomorrow. Or you'll have to do it without me10:32
marios|ruckpanda: ack ok can you please try run a Q today then anyway so we at least get the very latest there10:32
marios|ruckpanda: and we discuss plan on scrum10:33
sshnaidmmarios|ruck, I'm not sure how dib works, maybe need to disable selinux inside the image? Still seems for me like not selinux problem though10:33
chkumar|roversshnaidm: we need to disable selinux on box itself10:33
sshnaidmchkumar|rover, what is a "box"?10:34
chkumar|roversshnaidm: box == rhel8 nodepool vm10:34
sshnaidmchkumar|rover, it's in permissive10:34
sshnaidmchkumar|rover, http://logs.rdoproject.org/19/23919/8/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/78121a3/logs/script_build.log10:35
chkumar|roversshnaidm: then we need to hold the node again and see what is going on there I think10:35
sshnaidmchkumar|rover, what do you mean?10:36
sshnaidmchkumar|rover, "autohold" in zuul?10:36
chkumar|roversshnaidm: yes10:37
sshnaidmchkumar|rover, was it escalated in cix calls?10:38
sshnaidmchkumar|rover, I wonder if somebody from RHEL can look at it10:38
chkumar|roversshnaidm: yes https://trello.com/c/MFhqvOGt/1239-cixlp1853028tripleociproa-build-overcloud-image-for-rhel8-fails-sometimes-on-intargetd-post-installd-51-enable-network-service10:38
sshnaidmchkumar|rover, Alex has there a long comment, looks like this can be the point10:40
sshnaidmchkumar|rover, I think worth to open BZ10:42
sshnaidmat least to get more opinions10:42
*** bhagyashris has joined #oooq10:44
chkumar|roversshnaidm: lookign for proper component to file the bz10:45
marios|ruckthanks chkumar|rover (fwiw apart from rhel promotions are lookign great ... train 4 days in row . stein promoting now and i think after master should promote that https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master (but master/train currently ongoing for new hash)10:50
*** bhagyashris_ has joined #oooq10:51
chkumar|roverawesome10:51
marios|ruckchkumar|rover: sshnaidm: i just recheck the test. if we get at least one green run there we confirm its def not selinux and some race10:51
*** bhagyashris has quit IRC10:53
sshnaidmmarios|ruck, chkumar|rover I'll rerun some tests depending on https://review.opendev.org/#/c/699178/10:55
jfrancoamarios|ruck: for your knowledge (although maybe you already have another lp tracking the issue) https://bugs.launchpad.net/tripleo/+bug/185655111:20
openstackLaunchpad bug 1856551 in tripleo "[rocky to stein] Upgrade fails with error: dnf: command not found" [High,Triaged]11:20
jfrancoamarios|ruck: it's impacting only to CI jobs as we don't have dnf upstream11:20
marios|ruckjfrancoa: thanks was not aware of that one11:21
*** ysandeep has joined #oooq11:30
*** SurajPatil has joined #oooq11:34
*** surpatil has quit IRC11:36
*** surpatil has joined #oooq11:40
*** surpatil has quit IRC11:40
*** SurajPatil has quit IRC11:43
*** epoojad1 is now known as epoojad1|afk11:56
*** epoojad1|afk has quit IRC12:01
*** rfolco has joined #oooq12:03
*** ysandeep has quit IRC12:08
marios|ruckfood brb12:18
rfolcozbr, help me fix this linter issue please12:35
rfolcohttp://logs.rdoproject.org/35/24135/7/check/tox-linters/f85b1cd/job-output.txt12:35
*** jpena is now known as jpena|lunch12:39
marios|ruckchkumar|rover: did you file bz for rhel/selinux issue?12:39
pandamarios|ruck: queens promoted12:53
marios|ruckthank you panda12:54
rfolcozbr, nm, I fixed it12:55
*** sanjayu_ has quit IRC12:56
chkumar|rovermarios|ruck: not yet12:56
marios|ruckchkumar|rover: sshnaidm: on rcm guest now looks like [centos@rcn-share rcm-guest]$ ll -Z images/rhel_base/rhel-8.0-base/12:56
marios|ruck-rwxrwxr-x. centos centos system_u:object_r:httpd_sys_content_t:s0 rhel-8.0-guest-rhui.qcow212:56
marios|ruckmaybe weshay didn't fix it yet as per the email12:56
marios|ruckchkumar|rover: sshnaidm: perhaps that is why we have selinux issue even though we http://logs.rdoproject.org/19/23919/8/check/periodic-tripleo-rhel-8-buildimage-overcloud-full-master/9af4f7a/build_images.sh12:57
marios|rucksetenforce ^12:57
sshnaidmmarios|ruck, can we just switch off the selinux there?12:57
marios|rucksshnaidm: i can try a virtcustomize but12:57
sshnaidmmarios|ruck, I mean on rcn-share12:57
marios|rucksshnaidm: not on rcm guest (don't want to start installing stuff)12:57
marios|rucksshnaidm: yea12:57
marios|rucksshnaidm: i can do it12:57
marios|rucksshnaidm: with virt-customize12:57
marios|rucksshnaidm: did it last week but problem is12:58
sshnaidmmarios|ruck, cool, idk why we have it there at all.. it's internal resource anyway12:58
marios|rucksshnaidm: takes couple hours for upload from laptop12:58
*** rlandy has joined #oooq12:58
marios|rucksshnaidm: cos you have to do everything through rdo vm12:58
marios|rucksshnaidm: is the only way to get onto that host12:58
sshnaidmomg12:58
marios|rucksshnaidm: gonna wait a few for weshay12:58
marios|rucksee if he can fix it as per his email maybe he has faster way12:58
marios|rucksshnaidm: scrum in 2 mins12:59
sshnaidmyeah12:59
*** amoralej is now known as amoralej|lunch12:59
rfolcoscrum time zbr weshay rlandy panda sshnaidm marios|ruck arxcruz|pto12:59
rfolcohere https://meet.google.com/oiv-geho-mai13:00
rfolcook we'll start13:02
rfolcochkumar|rover, ^13:02
marios|ruckchkumar|rover: o/ join scrum please13:02
*** ykarel is now known as ykarel|afk13:05
chkumar|rovermarios|ruck: https://bugzilla.redhat.com/show_bug.cgi?id=178400113:06
openstackbugzilla.redhat.com bug 1784001 in chkconfig "chkconfig network on returing failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network" [High,New] - Assigned to lnykryn13:06
*** holser has joined #oooq13:06
*** ysandeep has joined #oooq13:13
*** ykarel|afk has quit IRC13:15
sshnaidmmarios|ruck, chkumar|rover https://review.opendev.org/#/c/699178/13:17
sshnaidmlet's check it in the image itself13:18
*** bhagyashris_ has quit IRC13:19
*** ysandeep has quit IRC13:21
chkumar|rovermarios|ruck: I will be skipping ci escalation call13:24
marios|ruckchkumar|rover: ack13:25
*** marios|ruck is now known as marios|ruck|call13:26
chkumar|roversshnaidm: Are we testing above patches a s a depends on?13:30
*** jpena|lunch is now known as pjena13:31
*** pjena is now known as jpena13:32
sshnaidmchkumar|rover, which ones? my patch doesn't need it, build jobs are running on tripleo-puppet-elements patches13:32
chkumar|roversshnaidm: ok13:32
chkumar|roversshnaidm: anyway added depends-on here https://review.rdoproject.org/r/#/c/24177/13:32
weshaychkumar|rover, marios|ruck|call13:33
weshayhey13:33
chkumar|roverweshay: Hello13:34
chkumar|roverweshay: we are confused on RHEL8 image on rcm server13:34
weshaychkumar|rover, which one.. 8.0 or 8.113:34
chkumar|roverweshay: please have a look at last reply of marios13:34
chkumar|roverweshay: I think mages/rhel_base/rhel-8.0-base/13:35
chkumar|roverweshay: Earlier you have updated the image from 8.0 to 8.1 and renamed it to 8.0 na?13:35
weshaychkumar|rover, no.. no13:35
chkumar|roverI mean last friday13:36
weshaychkumar|rover, I just mounted it ro.. and looked at the certs13:36
weshaythat caused selinux context to change13:36
chkumar|roverweshay: ok, we also filed a bug on rhel side https://bugzilla.redhat.com/show_bug.cgi?id=178400113:36
openstackbugzilla.redhat.com bug 1784001 in chkconfig "chkconfig network on returing failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network" [High,New] - Assigned to lnykryn13:36
weshaychkumar|rover, was that caused by selinux?13:37
chkumar|roverweshay: we are still investigating13:37
chkumar|roverweshay: https://review.opendev.org/#/c/698883/ making selinux to permissive we again saw the error13:38
chkumar|roverthen https://review.opendev.org/699178 is looking through what is actually going on there via listing file13:38
sshnaidmchkumar|rover, etenforce: SELinux is disabled http://logs.rdoproject.org/78/699178/2/openstack-check/tripleo-rhel-8-buildimage-overcloud-full/99e4f0b/build.log13:40
sshnaidmchkumar|rover, so no selinux problem :(13:40
chkumar|roverok, will take a look heading home13:41
*** Goneri has joined #oooq13:44
*** amoralej|lunch is now known as amoralej13:50
weshaymarios|ruck|call, chkumar|rover should we jump on?13:53
marios|ruck|callweshay: yes13:53
* marios|ruck|call runs to coffee machine13:53
weshayhttps://meet.google.com/tie-xnpb-keu?authuser=113:53
rfolcorlandy, as soon as you get a chance pls https://review.rdoproject.org/r/2413513:54
rfolcorlandy, I can walk you through this change if needed13:54
rlandyrfolco: sec - just working on downstream dlrn13:54
weshaychkumar|rover, hey13:56
weshayyou avail?13:56
*** jaosorior has quit IRC14:02
*** ykarel|afk has joined #oooq14:06
sshnaidmzbr, https://review.opendev.org/#/c/693310/14:07
*** epoojad1 has joined #oooq14:09
weshaymarios|ruck|call, chcon -t httpd_sys_content_t rhel-8.0-guest-rhui.qcow214:12
marios|ruck|callsshnaidm: chkumar|rover: the rhel8 guest image has selinux enforcing ( weshay confirmed just now on rcm gues we are in call)14:16
marios|ruck|callsshnaidm: chkumar|rover: he is re-setting and we can rechecks14:16
sshnaidmmarios|ruck|call, ok, but we disabled it and it didn't help much14:17
weshaymarios|ruck|call, sshnaidm chkumar|rover https://review.rdoproject.org/r/#/c/24161/ fyi14:20
chkumar|roverweshay: I am here14:20
marios|ruck|callhttps://review.opendev.org/#/c/699170/14:23
chkumar|roverweshay: marios|ruck|call https://bugs.launchpad.net/tripleo/+bug/185365214:24
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Triaged]14:24
chkumar|rovermarios|ruck|call: weshay https://review.rdoproject.org/r/2418914:24
sshnaidmFailed to connect to rhui-cds port 443: Connection refused14:24
weshaymarios|ruck|call, chkumar|rover http://git.app.eng.bos.redhat.com/git/tripleo-environments.git/tree/config/release/rhos-15.yml14:26
chkumar|roverweshay: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/RedHat-8/master.yml#L12114:27
*** saneax has joined #oooq14:29
*** bhagyashris has joined #oooq14:33
*** epoojad1 has quit IRC14:34
*** ykarel|afk is now known as ykarel|away14:34
chkumar|roversshnaidm: https://bugzilla.redhat.com/show_bug.cgi?id=1784001 How can I hook strace there?14:42
openstackbugzilla.redhat.com bug 1784001 in chkconfig "chkconfig network on returing failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network" [High,New] - Assigned to lnykryn14:42
chkumar|roverdo I need to make a change in tripleo puppet elements?14:43
sshnaidmchkumar|rover, well, let's run "strace chkconfig ..." in https://review.opendev.org/#/c/699178/4/elements/overcloud-base/post-install.d/51-enable-network-service14:44
sshnaidmchkumar|rover, but strace -o ~/some.log14:45
sshnaidmchkumar|rover, if not, we can run strace on whole build command14:46
*** akahat has quit IRC14:46
chkumar|roversshnaidm: what about let's do both?14:46
sshnaidmchkumar|rover, you don't need both, one is included in second14:46
chkumar|roverok14:46
sshnaidmchkumar|rover, I'll add strace to my patch when it finishes14:46
chkumar|roversshnaidm: sure14:47
chkumar|roversshnaidm: thanks :-)14:47
sshnaidmchkumar|rover, and you can actually create a patch with strace on build command14:47
sshnaidmchkumar|rover, if you meant this14:47
chkumar|roversshnaidm: creating14:48
*** marios|ruck|call is now known as marios|ruck14:49
* marios|ruck ignore irc for a bit focus on rhel8.114:56
* chkumar|rover ignoreirc14:57
*** chkumar|rover is now known as ignoreirc14:57
*** ignoreirc is now known as chkumar|rover14:58
*** baha has joined #oooq14:58
marios|ruckchkumar|rover: why/how do we know "its just missing rhui rpm " rhel 8.1 guest? doing it currently, am just going to virt-customize install http://(redacted)/~apevec/OSP/rdo-rhui-2.2-1.noarch.rpm (latest afaics) on that and re-upload15:03
marios|ruck(sanity checking while i wait for download)15:03
marios|ruckweshay: ? ^15:03
chkumar|rovermarios|ruck: just do yum repolist15:03
chkumar|roverit will tell15:03
rlandyrfolco:don;t shoot me - but I need to make a change to promote-hash that will impact your patch15:04
rlandythe user is hardcoded for the rdoproject user15:04
rlandyI need to make that into  a var so we can override it downstream15:04
rlandyrfolco: putting up patch for your review15:05
rfolco:(15:06
rfolco:(15:06
rfolco:(15:06
rfolco:(15:06
rfolco:(15:06
rlandyrfolco: there are soooo many assumptions in our upstream code15:07
rlandyneed to updo them all to work with rhel/ops/downstream15:07
rfolcoI know, but can we fix this role first ?15:08
rfolcoall following changes in downstream will be easier to make15:08
rlandyrfolco: let me put up the patch and we can look at it15:08
rfolcoI understand you want to get things done in downstream, but it makes my job harder just rebasing and rebasing and resolving conflicts without having progress15:10
rlandyrfolco: fine15:11
rfolcohttps://review.rdoproject.org/r/#/c/24135/ and https://review.rdoproject.org/r/#/c/24192/ are ready to review15:11
rlandyI'll make the changes after your patch15:11
rfolcoI appreciate15:11
*** TrevorV has joined #oooq15:17
rlandyrfolco: I'm leaving comments15:21
rfolcorlandy, ack15:21
rlandyrfolco: this impacts promotions15:21
rlandyrfolco: pls get another core to sign off as well15:21
rfolcorlandy, k15:22
*** ykarel|away has quit IRC15:23
rlandyrfolco: left comments - possible to test any of this anywhere?15:26
rlandyI used the component promote-to-tripleo-ci-testing jobs in testproject to run right after merge15:27
marios|ruckchkumar|rover: ack i see the codeready issue with repolist... but same issue trying to yum install rhui :/15:27
rlandyexport DLRNAPI_SERVER="{{ dlrn_server_url | default('trunk-staging.rdoproject.org') }}"15:27
rlandy^^ mostly I object to that default15:27
rlandyrest looks ok15:27
chkumar|rovermarios|ruck: oh no15:27
chkumar|rovermarios|ruck: I will take a look tomorrow morning15:27
rfolcorlandy, I am not sure how we can test it without merging15:28
chkumar|rovermarios|ruck: want to leave the notes somewhere, will pick up early morning15:28
rlandyrfolco: of you want to go ahead and merge this, pls testproject test right afterwards15:28
rlandyif15:28
rlandyso ruck/rovers see a working pipeline when they come in tomorrow15:29
rfolcorlandy, this is config, I need some other core to do it. maybe marios|ruck or sshnaidm or panda can review it ? https://review.rdoproject.org/r/#/c/24135/15:29
marios|ruckchkumar|rover: i think ok15:29
marios|ruckchkumar|rover: i did explicit remove first then new command for install15:29
marios|ruckchkumar|rover: seems to have completed15:30
marios|ruckchkumar|rover: will start upload in a sec sanity checking15:30
chkumar|rovermarios|ruck: ack, great, thanks :-)15:30
chkumar|rovermarios|ruck:++15:30
rlandyrfolco: if you make the default change, I'll +2 it15:30
rfolcodefault change ? rlandy15:31
rlandyrfolco; see comments15:31
marios|ruckchkumar|rover: we'll see :/15:31
rfolcook ok15:31
rlandyrfolco: otherwise we can merge it, but then we need someone to keep watch and check promotions/get-hash keeps working15:32
rlandyand revert if it breaks15:32
rfolcorlandy, let me fix default dlrnapi server15:32
marios|ruckchkumar|rover: nope same problem after clean install of rdo rhui 2.2.115:32
sshnaidmrfolco, will look15:33
rfolcosshnaidm, nice, appreciate15:34
chkumar|roversshnaidm: marios|ruck rfolco rlandy weshay 5 years of PythonPune https://pbs.twimg.com/media/EL0ROKvVAAIHUOI?format=jpg&name=4096x409615:34
marios|ruckchkumar|rover: debugging might be the hosts15:34
chkumar|rovermarios|ruck: yes, but rpm installs it15:35
rlandyrfolco: oh - one more comment - pls put back component_release15:35
*** bhagyashris has quit IRC15:35
sshnaidmchkumar|rover++15:37
marios|ruckchkumar|rover: hosts missing for sure15:37
chkumar|roversshnaidm: thanks :-)15:37
chkumar|rovermarios|ruck: http://file.rdu.redhat.com/~apevec/OSP/rdo-rhui.hosts15:38
marios|ruckchkumar|rover: yes15:38
marios|ruckchkumar|rover: its not there in the 8.1 guest15:38
marios|ruckchkumar|rover: trying to add and re-check repolist15:38
*** fmount has quit IRC15:40
chkumar|roverack15:41
*** fmount has joined #oooq15:43
marios|ruckchkumar|rover: i think now ok http://paste.openstack.org/show/787634/15:44
rlandyrfolco: ping me when that patch is ready to go15:45
rfolcok trying to understand component_release15:45
rfolcowhere this is set up ?15:45
rfolcoin job definition its release only isn't it15:45
chkumar|rovermarios|ruck: yes looks perfect15:46
chkumar|rovermarios|ruck: we may need to email to this paste to apevec so that he can take care of that next time.15:46
rfolcorlandy, I don't see any release_component15:47
rfolcorlandy, its release instead15:47
chkumar|roversshnaidm: How we can we add job-output.txt in logstash output collected by collect-logs role?15:47
chkumar|rovercontext https://bugs.launchpad.net/tripleo/+bug/185652015:48
openstackLaunchpad bug 1856520 in tripleo "pypi errors in CI" [Critical,Triaged]15:48
sshnaidmchkumar|rover, what do you mean?15:48
rlandyrfolco: see the file you changed ... # release value than the rest of the product15:48
rlandy      export RELEASE="{{ component_release | default(release) }}"15:49
sshnaidmjob-output is sent to logstash by default15:49
chkumar|roversshnaidm: for each job/15:49
chkumar|rover?15:49
sshnaidmchkumar|rover, yes15:49
chkumar|roversshnaidm: then we need to narrow down my search queries on logstash15:50
*** fmount- has joined #oooq15:50
*** fmount has quit IRC15:50
*** fmount- is now known as fmount15:51
rfolcorlandy, component_release is defined in downstream only ?15:52
rfolcorlandy, why a different var ?15:52
rfolcorlandy, why can't use just release ?15:52
*** ykarel|away has joined #oooq15:56
rlandyrfolco: because the get-hash script expects that15:58
rlandyit is currently defined in downstream only15:58
rlandybut it can come up in any job where the component and the rest of the rpms do not match15:59
rlandyit's an option we are required to support15:59
rlandyit defaults to release anyways15:59
rlandyand was there and working before15:59
rlandymarios|ruck: do you guys need help splitting fs020?16:00
marios|ruckrlandy: not even looking at that today16:00
marios|ruckrlandy: will do tomorrow16:01
marios|ruckrlandy: need focus on rhel8 for remainder today16:01
rlandyok16:01
chkumar|roverrlandy: if interested take a look at this one https://zuul.opendev.org/t/openstack/build/92332f0e85724a0c98f59aab935b50ba/log/job-output.txt16:02
chkumar|roversorry16:02
chkumar|roverrlandy: https://bugzilla.redhat.com/show_bug.cgi?id=1784001 and few reviews https://review.opendev.org/699221 and https://review.opendev.org/69917816:02
openstackbugzilla.redhat.com bug 1784001 in chkconfig "chkconfig network on returing failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network" [High,New] - Assigned to lnykryn16:02
rlandyk16:03
chkumar|roversorry for bumping on that, we got screwed on this16:03
chkumar|roverfor more help check with sshnaidm16:03
chkumar|roverindirectly related with this https://bugs.launchpad.net/tripleo/+bug/1853652 and https://review.opendev.org/#/c/699170/16:05
openstackLaunchpad bug 1853652 in tripleo "openstack overcloud node provide --all-manageable timing out and failing periodic rhel-8-ovb-3ctlr_1comp-featureset001-master" [Critical,Triaged]16:05
chkumar|roverrlandy: please drop a note to us if you get something, thank you for helping on this :-)16:05
rfolcorlandy, get-hash expects RELEASE16:05
rfolcorlandy, component_release is defined where? I only see release in the job definition16:06
rfolcorlandy, am I missing anything?16:06
*** jaosorior has joined #oooq16:10
chkumar|rovertime to leave now16:10
chkumar|roversee ya16:10
chkumar|roverrfolco: rlandy panda marios|ruck sshnaidm Good night, see ya!16:10
chkumar|rovermarios|ruck: focus on finishing mOzito man! you are rocking today :-)16:11
marios|ruckchkumar|rover: living the dream \o/16:11
marios|ruckchkumar|rover: have a good one16:11
chkumar|rovermarios|ruck: want to spoil rlandy and sshnaidm holidays?16:11
* chkumar|rover looks at rlandy , she will kill me16:12
marios|ruckchkumar|rover: promotions looking great not sure why master didn't promote but will check that tomorrow (and rhel obviously... hopefully anyway )16:12
chkumar|rovermarios|ruck: rdo trunk got consistent during escalaiton call16:12
chkumar|roverwill take time16:12
marios|ruckchkumar|rover: ah k. cos there was at least one green master run today16:13
rlandychkumar|rover: no worries - not my holiday season anyways16:13
chkumar|roverpanda: Happy Holidays from #tripleo team, please break the ice if you see it :-)16:14
chkumar|rover(see ya)16:14
*** chkumar|rover is now known as raukadah16:14
rlandyERROR: No matching distribution found for six===1.13.0 (from -c https://releases.openstack.org/constraints/upper/master (line 538))16:15
marios|ruckrlandy: yes this one is transient16:15
marios|ruckrlandy: we think/hope16:15
marios|ruckrlandy: peaked yesterday16:15
raukadahrlandy: https://bugs.launchpad.net/tripleo/+bug/185652016:15
openstackLaunchpad bug 1856520 in tripleo "pypi errors in CI" [Critical,Triaged]16:15
marios|ruckrlandy: recheck16:15
marios|ruckhttp://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22No%20matching%20distribution%20found%20for%5C%2216:15
rlandyfix is not on our side then16:15
marios|ruckrlandy: no something something pypi mirrors ?16:15
rlandymarios|ruck: ok  - no record of the error in the last two hours16:17
marios|ruckrlandy: yeah we saw different cases for it wasn't always six package and then if you set logstash to 2 days it peaks around 15th16:18
rlandyrfolco: right now we only have one occurrence of the diff componnet release to release and it's downstream. if you object to changing that, np, you can leave release, I'll amend after we merge your patch16:19
rlandymarios|ruck: sorry - just following chandan's posting to me above16:19
*** ysandeep has joined #oooq16:19
marios|ruckrlandy: which bit16:20
marios|ruckrlandy: are you apologising for)16:20
rlandymarios|ruck: its' a cultural thing, I apologize for everything16:20
marios|ruckin that case16:20
* marios|ruck bows16:20
marios|rucki apologise too rlandy16:21
marios|ruckam i doing it right?16:21
rlandyperfectly16:21
rlandyyou can be an honorary south african16:21
marios|ruck\o/16:22
sshnaidmaren't that canadians that say sorry instead of spaces16:23
marios|rucklol16:24
sshnaidmI think canadian layout should have "sorry" key16:24
* rlandy wants a 'sorry' key16:25
raukadahrlandy: time for a RFC it might come with new macbook :-)16:27
rfolcorlandy, if I change to component_release I will have to change the 2 component job definitions (currently w/ vars: release: master)16:27
rlandyrfolco: just leave it - I'll modify afterwards16:27
rlandyit will only break downstream atm16:27
rlandyit defaults to release if component release is not defined16:28
rlandyyou shouldn't need to change anything in your jobs16:28
rlandybut forget that16:28
*** ysandeep has quit IRC16:34
*** dtrainor has quit IRC16:36
*** ykarel|away has quit IRC16:38
rfolcorlandy, https://review.rdoproject.org/r/#/c/24135/ updated and ci +116:41
rlandyrfolco: I think you are missing a default ... dlrnapi_user https://review.rdoproject.org/r/#/c/24135/10/roles/promote-hash/defaults/main.yaml16:46
*** ykarel|away has joined #oooq16:47
rlandyrfolco: dlrnapi_user should default to review_rdoproject_org16:48
rlandyas in16:49
rlandyhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/dlrn-report/defaults/main.yml#L316:50
rfolcorlandy, ok16:51
rfolcorole default you mean16:51
rlandyyes16:53
*** dtrainor has joined #oooq17:01
*** ykarel|away has quit IRC17:01
*** ykarel|away has joined #oooq17:02
marios|rucksshnaidm: weshay: rlandy: k lets merge that ready please https://review.rdoproject.org/r/#/c/24197/17:03
marios|ruckcan't test in https://review.rdoproject.org/r/#/c/23919/ because its config repo ^^17:03
marios|rucksshnaidm: weshay: rlandy: finished uploading the updated rhel8.1 image17:03
sshnaidmmarios|ruck, what is the link?17:03
sshnaidmjust will test it if it downloads17:03
sshnaidmmarios|ruck, ok, tested :)17:04
marios|rucksshnaidm: you can't i mean you need rdo vm17:04
marios|rucksshnaidm: i just updated chmod so is same as old one17:04
marios|rucksshnaidm: i backed up orig -rwxr-xr-x. 1 centos centos 848363520 Dec 16 17:01 rhel-8.1-x86_64-kvm.qcow217:04
marios|ruck-rwxr-xr-x. 1 centos centos 789118976 Dec 13 16:23 rhel-8.1-x86_64-kvm.qcow2.ORIG17:04
sshnaidmmarios|ruck, that's fine, I can download it17:05
marios|rucksshnaidm: ack17:05
*** saneax has quit IRC17:05
rlandyrfolco: ok +2'ed your patch17:07
marios|ruckweshay: rlandy: sshnaidm: raukadah: https://bugs.launchpad.net/tripleo/+bug/1853028/comments/15 /me ready to go17:10
openstackLaunchpad bug 1853028 in tripleo "Build overcloud image for rhel8 fails sometimes on in_target.d/post-install.d/51-enable-network-service" [Critical,Triaged]17:10
sshnaidmmarios|ruck, raukadah opened a bz: https://bugzilla.redhat.com/show_bug.cgi?id=178400117:11
openstackbugzilla.redhat.com bug 1784001 in chkconfig "chkconfig network on returing failed to glob pattern /etc/rc0.d/[SK][0-9][0-9]network" [Urgent,New] - Assigned to lnykryn17:11
sshnaidmmarios|ruck, I updated it with needed info17:11
sshnaidmit's unlikely selinux issue17:12
marios|rucksshnaidm: tomorrow17:12
sshnaidmmarios|ruck, sure, will update lp17:13
marios|rucksshnaidm: rechecked https://review.rdoproject.org/r/#/c/23919 too should use the new image now17:13
*** udesale has joined #oooq17:14
*** ykarel|away has quit IRC17:16
*** marios|ruck is now known as marios|ruck|out17:18
rfolcosshnaidm, still need your review https://review.rdoproject.org/r/#/c/24135/17:43
*** jfrancoa has quit IRC17:49
*** marios|ruck|out has quit IRC17:51
*** derekh has quit IRC18:02
rlandyrfolco: I think just merge the patch and revert if necessary18:07
rlandywatch it today18:07
rlandymake sure it didn;t break anything18:07
rlandywe need to move on18:07
rfolcorlandy, I just need you to w+ then18:07
rfolcorlandy, its hard to move on w/ +1 only :)18:08
*** jpena is now known as jpena|off18:08
rlandyrfolco: are you available to check jobs this afternoon to make sure stuff is still working?18:08
rfolcorlandy, yes I can do that. But need one core to revert it anyway18:09
rlandyrfolco: fine - if you ping me, I'll revert18:09
rfolcorlandy, ok18:09
rfolcothx18:09
rlandyrfolco: done18:09
rlandypls testproject check after merge18:09
rfolcook18:10
sshnaidmrfolco, seems like you lost "---" in files beginning, but otherwise looks fine to me18:10
sshnaidmanyway I'm not so familiar with that18:11
rfolcosshnaidm, linters didn't complain, I assumed for roles its not required18:11
sshnaidmI think linters just don't check them.. << zbr18:11
rfolcosshnaidm, rlandy depends-on needs to be merged18:13
rfolcohttps://review.rdoproject.org/r/#/c/24080/18:13
rfolcoyaml file for promotion criteria ^18:13
*** sshnaidm is now known as sshnaidm|afk18:26
rlandyrfolco: merging https://review.rdoproject.org/r/#/c/24192/18:33
*** amoralej is now known as amoralej|off18:38
rlandyrfolco: error18:42
rlandy2019-12-16 18:41:37.400391 | rdo-centos-7 | /home/zuul/src/review.rdoproject.org/config/ci-scripts/tripleo-upstream/get-hash.sh: line 37: PROMOTE_NAME: unbound variable18:42
rfolcorlandy, looking18:44
rlandy: ${PROMOTE_NAME:=""}18:44
rlandyrfolco: ^^ you need to add that to get-hash18:45
rlandyhttps://review.rdoproject.org/r/#/c/21853/18:45
rlandytestproject job18:45
rfolco      export PROMOTE_NAME"{{ promote_source }}"18:45
rfolcoits being exported18:45
rlandyrfolco: that is the fix18:47
rfolcorlandy, its already in get-hash.yaml18:47
rfolcorlandy, oh you mean : ${PROMOTE_NAME:=""} in bash18:49
rlandyyes18:49
rfolcoinside the script18:49
rfolcook18:49
rfolcoactually I think its something else... RELEASE is also expected by the script and we don't have it there18:51
*** tesseract has quit IRC19:06
*** rfolco has quit IRC19:09
*** rfolco has joined #oooq19:10
rlandybrb19:22
*** rlandy is now known as rlandy|brb19:22
rfolcorlandy|brb, https://review.rdoproject.org/r/2420019:29
*** rlandy|brb is now known as rlandy19:45
rlandyrfolco: there is nothing wrong with that change19:45
rlandybut I don;t see how it will make a difference19:45
*** jbadiapa has quit IRC19:46
rlandyrfolco: you around?19:51
rfolcorlandy, o/19:51
*** udesale has quit IRC19:52
rlandyrfolco: how sure are you that this will fix the error?19:52
rlandyrfolco: get-hash is run in other jobs, do those work?19:52
rlandyany periodic19:52
*** Goneri has quit IRC19:53
rlandyrfolco: I mean w/o your current fix19:53
* rlandy tests19:53
rfolcorlandy, I think this is an ansible issue19:53
rfolcorlandy, thats why set_fact or include_vars may work19:54
rlandyrfolco: testing non-promote job19:55
rlandyI want to see what a job running get-hash does19:55
rlandyhttps://review.rdoproject.org/zuul/stream/c000a4c8ae6244ce8c68566da520dffb?logfile=console.log19:55
rfolcorlandy, something like this issue https://github.com/ansible/ansible/issues/2189019:58
*** jaosorior has quit IRC19:59
rlandyrfolco: you don't have +2 on config?19:59
rfolcorlandy, no19:59
rfolcoonly in ci-config19:59
rlandythought we added that20:00
rlandyhttps://review.rdoproject.org/zuul/stream/c000a4c8ae6244ce8c68566da520dffb?logfile=console.log doe sok20:01
rlandydoes ok20:01
rlandyrfolco: so I think with include_role you have to define the vars with the role20:04
rlandyhttps://github.com/openstack/tripleo-quickstart-extras/blob/c556bcade9df039959aadeb859396ae826d7f639/roles/install-built-repo/tasks/inject_repo.yml#L320:04
rlandywe can try your patch20:04
rfolcoI am testing my patch locally20:04
rfolcogimme a sec20:04
rfolcothen we merge if this works20:04
rfolcorlandy, I fail to reproduce the issue, the var exposed win vars: actually works20:11
rlandyrfolco: so merge https://review.rdoproject.org/r/24200?20:12
rfolcorlandy, lets try20:12
rfolcounless you have any other ideas20:12
rlandywe'll revert if need be20:13
rfolcowait20:16
rfolcorlandy, I found it20:16
rfolcorlandy, sorry... we can revert it after20:18
rfolcorlandy, https://review.rdoproject.org/r/2420120:20
rlandyrfolco: reverting old patch first20:20
rfolcosure20:20
*** holser has quit IRC20:35
rlandyrfolco: another error: https://review.rdoproject.org/zuul/stream/b66e9e4f695043979cec4e6467beb972?logfile=console.log20:40
rlandyOSError: [Errno 13] Permission denied: '/dlrnapi_venv'20:40
rlandyTASK [promote-hash : Promote hash label from consistent to tripleo-ci-testing20:41
*** jfrancoa has joined #oooq20:44
rlandyrfolco: around?21:04
rlandyany fix for error above?21:04
rfolcoyes trying to fix this21:04
*** Goneri has joined #oooq21:06
rfolcoprobably WORKSPACE is empty $WORKSPACE/dlrnapi_venv21:11
rfolcorlandy, now this rings a bell21:14
rfolco environment: '{{ zuul | zuul_legacy_vars }}'21:14
rlandynote comment left on original review21:15
rfolcorlandy, do not throw things in my face, just makes things worse :)21:16
rfolcorlandy, https://review.rdoproject.org/r/2420321:22
*** dsneddon has joined #oooq21:25
*** TrevorV has quit IRC21:35
rlandyrfolco: I +2'ed the original patch21:36
rlandynot sure the second patch changes are needed21:37
rfolcorlandy, rdo gerrit out?21:37
rlandysorry?21:37
rlandyrfolco: probably ok21:38
rfolcorlandy, nm21:38
rfolcoyeah ps2 was to ensure workspace is exported to bash script21:38
rfolcowe do that in some places btw21:38
rlandyrfolco: it's fine - I'm +2'ing it21:39
rfolcok21:39
rfolcorlandy, w+ so i can recheck testproject21:41
rlandydone21:42
*** holser has joined #oooq21:44
rfolcorlandy, zuul is not cooperating21:52
rfolcohttps://review.rdoproject.org/zuul/status21:52
rfolco503 service unavailable21:52
*** rfolco is now known as rfolco|bbl21:58
*** jfrancoa has quit IRC22:09
rlandyrfolco|bbl: think rdocloud zuul is undergoing update22:15
rlandywill retry later22:15
*** holser has quit IRC22:25
*** rlandy is now known as rlandy|bbl22:51
rlandy|bblah zuul is back22:51
rlandy|bblrechecking22:52
rlandy|bbltenant looks ok22:55
rlandy|bblrfolco|bbl: reported the config check failure on sf-ops23:09
*** dsneddon has quit IRC23:09
*** dsneddon has joined #oooq23:16
*** dsneddon has quit IRC23:21
*** dsneddon has joined #oooq23:36
*** tosky has quit IRC23:43
*** dsneddon has quit IRC23:46
*** dsneddon has joined #oooq23:48
*** holser has joined #oooq23:52
*** dsneddon has quit IRC23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!