Thursday, 2020-01-30

*** holser has quit IRC00:22
*** holser has joined #oooq00:47
*** rfolco has joined #oooq00:55
*** holser has quit IRC01:06
*** rfolco has quit IRC01:29
*** ysandeep has quit IRC02:03
*** dsneddon has quit IRC02:16
*** ysandeep has joined #oooq02:41
*** dsneddon has joined #oooq02:45
*** ccamacho has quit IRC02:50
weshayowalsh, we have results from https://code.engineering.redhat.com/gerrit/#/c/189436/03:09
weshaysame job.. but runs baremetal.. so we can log in03:09
* weshay forwards03:09
owalshweshay: ack, I expect it to fail - we have 2 difference enabled_networks vars in ansible and I'm getting the wrong one :-(03:10
weshayowalsh, k.. just forwarded information on how to access the current live environment03:10
weshayowalsh, if you want to look, try now and I'll help if there are questions03:10
*** dsneddon has quit IRC03:12
weshayowalsh, I have to go to bed.. have to wake up at 4:30am03:14
weshaydid you get in?03:14
*** dsneddon has joined #oooq03:15
owalshweshay: yes, thanks!03:17
*** dsneddon has quit IRC03:21
*** dsneddon has joined #oooq03:52
*** dsneddon has quit IRC03:57
*** ykarel|away is now known as ykarel04:00
*** rlandy|bbl is now known as rlandy04:08
*** skramaja has joined #oooq04:25
*** udesale has joined #oooq04:40
*** dsneddon has joined #oooq04:48
*** dsneddon has quit IRC04:54
raukadahykarel, https://review.opendev.org/70492204:54
*** ysandeep has quit IRC04:54
*** ysandeep_ has joined #oooq04:54
*** ysandeep_ has quit IRC04:56
ykarelraukadah, ack05:21
*** dsneddon has joined #oooq05:25
*** dsneddon has quit IRC05:30
*** surpatil has joined #oooq05:44
*** raukadah is now known as chkumar|rover05:57
*** soniya29 has joined #oooq06:02
*** ratailor has joined #oooq06:34
*** surpatil has quit IRC06:44
*** marios has joined #oooq06:44
*** udesale_ has joined #oooq06:46
*** udesale has quit IRC06:47
chkumar|roverzbr, jpena|off when around please have  alook at this bug https://bugs.launchpad.net/tripleo/+bug/186137806:51
openstackLaunchpad bug 1861378 in tripleo "Multiple post_failure on master periodic pipeline" [Critical,Confirmed]06:51
chkumar|roverit is blocking promotion and check jobs06:51
chkumar|rovermay be log server got full06:51
*** apetrich has joined #oooq07:01
chkumar|roversshnaidm|afk, Hello, you have #tripleo channel access? We need to update the topic on not to workflow, gate queue is 10 hrs07:04
*** apetrich has quit IRC07:22
*** jtomasek has joined #oooq07:22
*** jfrancoa has joined #oooq07:24
*** ratailor has quit IRC07:26
*** dsneddon has joined #oooq07:27
*** ratailor has joined #oooq07:27
chkumar|roverheading wework07:28
*** dsneddon has quit IRC07:32
*** yolanda has quit IRC07:32
*** ykarel is now known as ykarel|lunch07:34
*** bogdando has joined #oooq07:41
*** chem has quit IRC07:46
*** jtomasek has quit IRC07:47
*** tesseract has joined #oooq07:47
*** chem has joined #oooq07:48
*** udesale_ has quit IRC07:58
*** udesale_ has joined #oooq07:58
*** jpena|off is now known as jpena08:10
jpenathe log server is full again :(. I'm checking08:11
*** sshnaidm|afk is now known as sshnaidm08:12
*** dpawlik has quit IRC08:14
*** ccamacho has joined #oooq08:19
sshnaidmjpena, I see ovb jobs don't gzip files again08:19
sshnaidmzbr, do you know why? ^08:20
jpenao_O08:20
sshnaidmnothing is gzipped: http://logs.rdoproject.org/17/704917/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/95fc484/logs/undercloud/home/zuul/08:20
*** jtomasek has joined #oooq08:25
*** udesale_ has quit IRC08:28
*** dmsimard5 has joined #oooq08:31
*** soniya29 has quit IRC08:32
*** dmsimard has quit IRC08:33
*** dmsimard5 is now known as dmsimard08:33
zbrsshnaidm: that file is used upstream08:38
sshnaidmzbr, look at #tripleo08:38
*** tosky has joined #oooq08:46
*** apetrich has joined #oooq08:48
*** jpena is now known as jpena|off09:00
*** holser has joined #oooq09:16
chkumar|roversshnaidm, zbr I am not sure this one is also related to gzip https://sf.hosted.upshift.rdu2.redhat.com/logs/36/189436/4/check/periodic-tripleo-ci-centos-7-bm_envD-1ctlr_2comp-featureset021-master/ed9ecfa/logs/09:21
chkumar|roveron downstream bm logs are not getting collected09:21
zbrchkumar|rover: you need to investigate, i do not have the energy to look at ansible debug logs.09:24
zbreven firefox crashes. not only me09:24
chkumar|roveraye09:25
zbrand luckly i escaped rovring last week ;)09:25
zbrchkumar|rover: but few days ago i think i discovered something with wes, that collect_logs.sh script does not not run artcl from the changeset, is probably an older version.09:26
*** dsneddon has joined #oooq09:27
zbrfor sure we need to avoid that shell script.   zull -> bash -> ansible is not correct.09:27
zbrwe can use import_playbook in zuul09:28
zbreven load vars from file09:28
zbrthat  -vvvv  there is just evil09:29
*** dtantsur|afk is now known as dtantsur09:29
*** holser has quit IRC09:30
*** dsneddon has quit IRC09:32
*** holser has joined #oooq09:32
*** derekh has joined #oooq09:33
*** ykarel|lunch is now known as ykarel09:33
*** udesale has joined #oooq09:35
matbusshnaidm: marios chkumar|rover hey all, do you know if there is a downstream oooq repo for OSP config ?09:37
matbuand rhel deployment09:37
sshnaidmmatbu, on internal irc09:39
sshnaidmmarios, did you see "Invalid regex: * in provided whitelist file" on http://logs.rdoproject.org/39/24339/9/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/45e0a19/job-output.txt09:41
mariossshnaidm: yeah i commmented -1 @ https://review.opendev.org/#/c/701016/15/config/general_config/featureset020.yml09:45
mariossshnaidm: i need to revisit that it needs an update09:45
mariossshnaidm: thanks for ping anout it09:46
mariosabout09:46
sshnaidmmarios, try to remove "|" from "tempest_test_blacklist: |"09:48
mariossshnaidm: i tried originally with >- but wanted it with newlines cos list. it failed same way with > but right i need to edit that syntaxt09:49
zbrsshnaidm: I had to block https://review.opendev.org/#/c/704933/1 because is messing upstream.09:59
zbralready got result09:59
sshnaidmzbr, no, don't block it09:59
sshnaidmzbr, we need logs server up in rdo server, and this is what it does09:59
zbrsshnaidm: we need to rebase https://review.opendev.org/#/c/704938/ on master instead borken on10:00
zbrbut that revert is not the solution10:00
zbrsolution is https://review.opendev.org/#/c/704938/110:00
sshnaidmzbr, please, let me do it, don't block anything right now10:00
ykarelyes please get that merged, no logs are bad then not browsable one10:00
zbrykarel: why not doing the correct fix?10:02
zbrthat revert is replacing one problem with another10:02
arxcruzhey guys, i'll be afk a little bit, solving some german visa issues10:02
zbrwe already know and have the right way of fixing it10:02
arxcruztl;dr there's a typo on my name10:02
ykarelzbr, that patch was wrong, so option 1 is revert to get things back, i am ok with revert + squash fix also10:03
ykarelwhichever is quicker10:03
chkumar|rovermigi, owalsh another bug https://bugs.launchpad.net/tripleo/+bug/1861393 related to volume tests10:12
openstackLaunchpad bug 1861393 in tripleo "Multiple volume related test failed due iscsiadm getting sessions: iscsiadm: and VolumeDeviceNotFound in nova compute" [Critical,Confirmed]10:12
zbrthat fixes the gzip problem directly: https://review.opendev.org/#/c/704952/10:14
zbralso putting a note, to avoid further regression, especially as we have artcl config there10:15
migichkumar|rover: getting coffee and will look in few min10:16
chkumar|rovermigi, fix already up and tested in the same box https://review.opendev.org/#/c/704805/10:17
sshnaidmzbr, we'll merge firstly patch that allows to log server to come to life, than patch that fixes all gzipping and not gzipping, if you want you can put your patches on top of these two10:17
chkumar|rovermigi, I will check the rest of the skip list before that10:18
migichkumar|rover: ok10:18
migichkumar|rover: btw why this? https://review.opendev.org/#/c/704805/1/zuul.d/layout.yaml10:18
migichkumar|rover: just so only one job triggers ?10:18
migichkumar|rover: cause I don't see 020 featureset there10:19
migisorry fs2110:19
migiunless it's depends-on10:19
*** ratailor has quit IRC10:24
zbrsshnaidm: after the fires settle we need to rethink the use of this parameter, i think we need to make the upload server define its value and not us10:26
*** ratailor has joined #oooq10:26
zbrregardless where a job runs, it does upload to a destination, and the destination should decide which kind of storage it wants.10:26
*** ratailor has quit IRC10:26
*** ratailor has joined #oooq10:27
zbrand we can avoid confusions like this, by linking artcl_gzip to the upload server instead of individual environments10:27
*** ratailor has quit IRC10:29
*** ratailor has joined #oooq10:30
*** ratailor has quit IRC10:31
*** ratailor has joined #oooq10:32
*** ratailor has quit IRC10:40
*** ratailor has joined #oooq10:41
*** dsneddon has joined #oooq10:42
mariospanda: fyi https://bugs.launchpad.net/tripleo/+bug/1861342/comments/110:43
openstackLaunchpad bug 1861342 in tripleo "tripleo-ci promotion failing on "pull ppc64le tagged containers"" [Critical,Triaged]10:43
mariospanda: not sure if there is something else we need to do to handle the _ppc? so it is reproducible with podman pull, don't think there is something wrong in https://github.com/rdo-infra/ci-config/blob/79bcc9c64b82f1c6806139118e5b9a3663dcdb76/ci-scripts/container-push/roles/containers-promote/tasks/manifest-push.yml#L40-L49 or i can't se it yet10:44
*** ratailor has joined #oooq10:44
chkumar|rovermigi, it was just a test review it works, in order to avoid wasting resources, we removed those jobs and tested in downstream review10:45
*** apetrich has quit IRC10:47
*** dsneddon has quit IRC10:49
*** dsneddon has joined #oooq10:52
*** zbr has quit IRC10:52
*** udesale has quit IRC10:53
*** ratailor_ has joined #oooq10:55
*** dsneddon has quit IRC10:57
*** ratailor has quit IRC10:58
chkumar|roverHey #oooq, please wait for few mins to push or recheck changes on review.rdoproject.org rdo logserver is full, it will take so time to get it cleaned, Sorry for the inconvenience11:06
owalshchkumar|rover: hey, is that what caused the POST_FAILUREs in https://review.opendev.org/70488011:22
*** ykarel is now known as ykarel|afk11:22
chkumar|roverowalsh, rdo log server is full we are freeing up space there11:23
chkumar|roverowalsh, https://bugs.launchpad.net/tripleo/+bug/186137811:23
openstackLaunchpad bug 1861378 in tripleo "Multiple post_failure on master periodic pipeline" [Critical,Confirmed]11:23
chkumar|roverit will take some time11:23
*** zbr has joined #oooq11:23
*** dsneddon has joined #oooq11:25
owalshack, thanks11:27
owalshmigi: the healthcheck failure suggests the sudo config has changed on the containers - could you create an LP for it?11:33
dtantsurhi folks! what does quickstart use instead of plain ssh nowadays? I'm getting Could not resolve hostname11:33
dtantsurfor a host that can be perfectly used with `ssh hostname`11:34
dtantsurdoes it require a globally resolvable name?11:35
*** zbr has quit IRC11:35
owalshdtantsur: IIRC it uses a different ssh config from ~/.quickstart with aliases e.g virthost etc...11:35
sshnaidmssh.config.ansible maybe11:35
*** zbr has joined #oooq11:35
dtantsursigh11:35
dtantsurthat's why we cannot have nice things :)11:36
dtantsuractually, it does respect ~/.ssh/config in some aspects, just not in aliases11:36
sshnaidmdtantsur, but that's weird11:36
dtantsuryeah. HostName directive doesn't work, User - does11:36
*** ykarel|afk is now known as ykarel11:36
sshnaidmdtantsur, it uses ansible, so if it doesn't work, most likely ansible is to blame11:37
migiowalsh: will do11:37
* dtantsur gladly blames ansible11:38
dtantsursshnaidm: how naive is my attempt to use RHEL 8 as a virthost oS?11:38
* owalsh blames ssh11:38
dtantsur"No package libvirt-python available.",11:38
dtantsur        "No package python-lxml available."11:39
dtantsurI've gotten the answe11:39
sshnaidmdtantsur, this may help: https://review.opendev.org/#/c/70210011:39
sshnaidmdtantsur, checked it today and it worked for me11:40
dtantsurthanks!11:40
*** soniya29 has joined #oooq11:42
owalshmigi: that suggest something has changed in the container sudo conf... I wonder if it's causing any of the other issues?11:43
chkumar|roverbrb11:46
*** ratailor_ has quit IRC11:46
weshaymarios, /me looking at /home/centos/ci-config/ci-scripts/dlrnapi_promoter/dlrn-promoter.sh11:46
weshayand other spots.. not clear where that is turned off11:46
weshay Active: active (running) since Wed 2020-01-29 21:40:13 UTC; 14h ago11:47
weshay Main PID: 29711 (dlrn-promoter-s)11:47
weshay   Memory: 216.2M11:47
weshay   CGroup: /system.slice/dlrn-promoter.service11:47
weshay           ├─ 3555 /usr/bin/python2 /usr/bin/ansible-playbook -e manifest_push=true -e target_registries_push=true /home/centos/ci-config-newcode/ci-scripts/container-push/container-push.y.11:47
weshaypanda, ^11:47
mariosweshay: the problem is the ppc containers in rdo registry.11:48
mariosweshay: we can switch off manifest push but that is not the problem11:48
migiowalsh: is ss to be run in overcloud on in contianer?11:48
*** holser has quit IRC11:48
weshaymarios, aye.. agree11:48
weshayjust where does one make that switch11:48
mariosweshay: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini11:49
mariosmanifest_push: true11:49
owalshmigi: https://github.com/openstack/tripleo-common/blob/master/healthcheck/common.sh#L5511:49
weshaymarios, panda found it.. https://review.rdoproject.org/r/#/c/24750/11:49
weshaythanks11:49
mariosweshay: 13:49 < marios> weshay: https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/master.ini11:50
mariosweshay: yes that 13:49 < weshay> marios, panda found it.. https://review.rdoproject.org/r/#/c/24750/11:50
weshaythanks11:50
mariosweshay: have you even brushed your teeth yet11:50
marios:)11:50
weshaymarios, what I ddin't know was that it was in the promotion config11:51
weshay:)11:51
weshaylolz11:51
owalshmigi: so container11:51
weshaymarios, what teeth?11:51
mariosweshay: well it is in the role as a variable. panda added it into the config to enable it for master/train11:51
migiowalsh: so I did short test and was able to run ss -ntuap as nova user from container11:51
migiowalsh: on same host that complained..., but now the env is gone11:52
owalshmigi: and sudo -u nova ss -ntuap11:52
migiowalsh: that not11:52
migiand yes if sudo is broken then it will be one11:52
migiowalsh: will check again and see11:53
mariosweshay: (default false there https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/container-push/roles/containers-promote/defaults/main.yml )11:53
weshaymarios, ya.. but that's a default11:54
mariosweshay: right11:54
owalshmigi: it's probably just that command that's failing. I don't think it's white listed but I guess sudo was wide open in the containers until now...11:54
owalshor it's been failing for ages but nobody noticed :-)11:54
weshaymarios, thanks for the assit11:55
*** rfolco has joined #oooq12:08
*** ykarel is now known as ykarel|afk12:25
*** soniya29 has quit IRC12:40
chkumar|roverarxcruz, around, can you take a look at this https://53cfbe20d54fcd346846-7e165b3b1cbfb7e5b567389713168c5c.ssl.cf1.rackcdn.com/703953/4/check/tripleo-ci-centos-7-containers-multinode/94a29e1/logs/undercloud/var/log/tempest/tempest_run.log in free?12:41
*** chem has quit IRC12:42
marioszbr: can you please see my comment (and ideally reply there ) at https://review.rdoproject.org/r/#/c/24705/ - we can discuss on call in a few mins as well12:43
mariospanda: wdyt? ^12:43
chkumar|rovermatbu, Hello, Do we have tht tripleo-ansible bump patch up?12:46
*** chem has joined #oooq12:46
zbrmarios: done12:54
*** rlandy has joined #oooq12:59
*** marios is now known as marios|call13:01
matbuchkumar|rover: you mean with the new release number ?13:01
matbuchkumar|rover: your patch has been merged ?13:01
rfolcozbr, weshay arxcruz marios|call sshnaidm rlandy panda : scrum time13:01
matbuI didn't check my email after lunch13:01
chkumar|rovermatbu, yes13:02
matbucool, i will update mine13:02
chkumar|rovermatbu, you can create the patch, reviewers will take care of that13:02
matbuchkumar|rover: i think only the tripleo-common is enough13:02
weshaypanda, you around?13:03
chkumar|rovermatbu, we need a tht patch13:03
weshayarxcruz, is pto13:03
matbuchkumar|rover: with the version ? why ?13:03
matbuchkumar|rover: the point of Alex was to put a version in ooo-common so the package can be upgraded with ooo-common13:04
chkumar|rovermatbu, ok13:04
matbuiiuc13:04
matbuAlex is not only yet13:04
weshaychkumar|rover, can you join scrum for a hot minute13:05
weshaychkumar|rover, or tell me what the status of the centos-8 ceph repos13:05
chkumar|roverweshay, no updates just asked gfindete on tripleo13:06
chkumar|roverwaiting for reply13:06
weshaychkumar|rover, k13:07
weshaythanks13:07
*** ykarel|afk is now known as ykarel13:10
weshayhttps://review.rdoproject.org/r/#/c/24667/13:12
weshayhttps://review.rdoproject.org/r/#/c/24667/13:13
*** ysandeep has joined #oooq13:28
*** apetrich has joined #oooq13:29
chkumar|roverWhy we are running standalone on tripleo-ci-centos-7-scenario004-standalone on stable/stein branch?13:32
chkumar|roverah sorry, it is running in promotion pipeline13:33
*** holser has joined #oooq13:33
rlandyzbr: https://sf.hosted.upshift.rdu2.redhat.com/logs13:37
weshayhttps://code.engineering.redhat.com/gerrit/#/q/project:openstack/rrcockpit13:41
marios|callhttps://code.engineering.redhat.com/gerrit/#/q/topic:17-standup+status:open13:46
marios|callweshay: rlandy: ^13:46
*** chem has quit IRC13:47
chkumar|roverrlandy, weshay let me know when free, we can start the call early and finish it13:48
*** chem has joined #oooq13:48
marios|callrlandy: podman meeting? (should i join just cos weshay said 'discuss there ' is it about rhel8 or sthing else13:52
rlandymarios|call: you are welcome to join - about gating downstream podman-related changes13:53
rlandystarting on the hour13:53
rlandywill post meeting link13:53
marios|callrlandy: ack ok thanks - its ok i'd only join if it was relevant to the rhel8/containers stuff13:53
marios|callrlandy: no worries thanks13:53
rlandymarios|call: idk think so - but always happy to have your opinion on stuff13:54
*** marios|call is now known as marios13:56
zbrrlandy: the only thing needed is to add artcl_gzip: true -- can you do it?13:57
zbrif you see a artcl_gzip_only, you can remove it, no longer used, correct name "artcl_gzip"13:57
rlandyzbr: where does that get added?14:00
chkumar|roverrlandy, weshay meeting tome14:00
chkumar|rover*time14:00
rlandymarios: if you want ... meet.google.com/vnf-jdnm-tzs14:00
zbrrlandy: you have to add this to the internal specific config14:01
mariosrlandy: thanks , would rather progress on other tasks, ping me if i am needed for something otherwise i'll skip14:01
mariosrlandy: i was only asking because it was mentioned on scrum during the rhel8/containers discussion14:01
rlandysure14:02
chkumar|roverweshay, we are waiting for you14:02
weshayI'm coming14:03
chkumar|roverweshay, https://docs.google.com/document/d/1-3ohDnfcj1ptJ_EUb_UAjsz5cAuqLnzURwOqUs40GP0/edit14:04
*** sshnaidm is now known as sshnaidm|afk14:07
arxcruzweshay: rfolco  i'm back, turns out it was easier than I though14:09
weshayarxcruz, cool.. thank you14:09
*** dtantsur is now known as dtantsur|brb14:18
rfolcoarxcruz, are you in trouble with your name?14:19
arxcruzrfolco: as always14:19
arxcruzrfolco: my name is pereira, and they write in the visa peireira14:19
rfolcof* i14:20
rfolcothey put i in everything14:20
rfolcoin phone, pad...14:20
arxcruzrfolco: yeah, so, i was "i"legal14:20
rfolcoha14:21
rfolcoI see14:21
chkumar|roverweshay, rlandy https://sf.hosted.upshift.rdu2.redhat.com/logs/84/190384/17/check/podman-package-rhel-8/d22a1fe/job-output.txt14:21
chkumar|roverrlandy, https://code.engineering.redhat.com/gerrit/#/c/190384/21/playbooks/podman/pre.yaml14:21
arxcruzmarios: chkumar|rover i have a fs020 job with 4:20 minutes ;)14:22
arxcruzwith 41 min remain to timeout :D14:22
pandaback14:22
arxcruzunfortunately got a post failure...14:22
pandadid I miss anything ?14:23
rfolcopanda, need 5 min of your time14:24
rfolcopanda, to show you something and ask your opinion14:24
mariosarxcruz: k, so you mean we should leave all the tempest there?14:25
arxcruzmarios: I'm saying it's possible, it was a poc, all i did was increase the cpu and number of concurrency tests14:26
mariosarxcruz: ack14:28
pandarfolco: not that thing again. I already told you you would need a different measuring device.14:29
rfolcopanda, ?14:30
pandarfolco: jk14:31
pandarfolco: when ? where ?14:31
rfolcopanda, ok https://meet.google.com/oiv-geho-mai14:32
*** sshnaidm|afk is now known as sshnaidm14:32
marioszbr: ack replied there again. would you consider removing your -1 then if you don't want to block? https://review.rdoproject.org/r/#/c/24705/14:32
mariossimple one reviews please when you next have time thank you "Refresh start_named_hashes after promotion to prevent false positive" https://review.rdoproject.org/r/#/c/24665/14:33
zbrmarios: am i using -1 wrong? afaik, -1 translates to "i don't like something about it", is far from -2 :D14:34
zbrsadly we do not have a -0.514:35
zbrbut if others say i should not use it as "i have concerns", i can skip.14:35
zbrbad part is that there is no other way to indicate that i seen a review other than making a decision about +/-14:36
marioszbr: ok fair enough thanks14:37
zbri mentioned clearly, is "soft"14:37
zbrweshay: bj?14:38
weshayzbr, need a few.. I need to chat.. but hold 10min ok?14:40
zbrsure14:40
weshayzbr, actually ready now.. ping when ur avail14:42
weshayupdated mtg invite14:43
weshayI'm in https://meet.google.com/pqf-dxyz-djo?authuser=114:43
mariosreviews please around removing duplication in promoter molecule rlandy rfolco panda sshnaidm weshay arxcruz chkumar|rover when you next have some time thanks https://review.rdoproject.org/r/#/c/2470514:45
*** ysandeep has quit IRC14:46
*** udesale has joined #oooq14:58
*** ysandeep has joined #oooq15:12
chkumar|roverrlandy, one downstream question, when we checkout a particular branch like triple-modify image branch, does it actually builds the rpm from that?15:15
rlandyit should15:15
rlandysec15:15
zbrmarios: do you have 10min for me?15:16
chkumar|roverrlandy, can you check?15:17
rlandychkumar|rover: on meeting - one sec15:17
*** holser has quit IRC15:18
*** ysandeep has quit IRC15:20
*** jaosorior has joined #oooq15:21
chkumar|roversee ya people15:23
marioszbr: whatsup15:24
*** chkumar|rover is now known as raukadah15:24
*** jmasud has quit IRC15:24
*** jaosorior has quit IRC15:27
*** ccamacho has quit IRC15:27
zbrmarios: i want to explain what i proposed on https://review.opendev.org/#/c/703586/ -- easier on https://meet.google.com/nah-avgj-ice15:31
zbrothers welcome15:31
marioszbr: joining15:32
*** marios is now known as marios|call15:33
*** jmasud has joined #oooq15:37
zbrtx!15:42
*** jmasud has quit IRC15:43
*** marios|call is now known as marios15:43
*** TrevorV has joined #oooq15:44
*** Trevor_V has joined #oooq15:47
*** ratailor has joined #oooq15:48
*** ratailor_ has joined #oooq15:50
*** TrevorV has quit IRC15:51
*** ratailor has quit IRC15:54
*** holser has joined #oooq15:56
zbrweshay: take a look at http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/15:59
zbrdo you see something interesting?15:59
zbri think i mentioned to chandan this morning that I seen a very weird "-vvv" when collect logs was called by qs.16:01
zbri am curious who is brave enough to open the collect logs file in their browser16:01
zbrrlandy: by any change can you help be find out where does the -vvvv comes from on http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/ovb_collect_logs.sh16:06
rlandyyes - one sec - submitting change16:06
*** skramaja has quit IRC16:06
*** dtantsur|brb is now known as dtantsur16:06
*** jmasud has joined #oooq16:11
rlandyzbr: looking16:11
migirlandy: do you know any role that is executed in prepare stage of zuul run and it's copying the source code of the single git repo ? if not I will need to work on one, but maybe already used somewhere16:11
rlandymigi: in general or when we build a specific repo?16:12
migirlandy: in general16:13
rlandyzbr: it's here ... https://github.com/openstack/tripleo-ci/blob/master/roles/run-test/templates/toci_quickstart.sh.j2#L5016:13
*** aakarsh has joined #oooq16:14
rlandymigi: yeah ... I'd have to find it again - give me a few16:14
zbrrlandy: not sure who put it but that value is insane, have you seen it produced 400MB? it will crash your browser if you click the log16:14
zbrwe should never see more than double v on ci.16:14
rlandyzbr: git blames alex - 13 months ago16:15
migirlandy: something that is kind of this: https://pagure.io/zuul-distro-jobs/blob/master/f/tests/local.yaml.example16:15
zbrnopb, i wasn't looking for a culprit, thanks for pointing. making fix now.16:15
rlandymigi: yeah - it's zuul code itself - what do you need exactly - maybe we can help?16:16
zbrmainly logging of log-collection spamming, bit ironic?16:16
*** aakarsh has quit IRC16:18
migirlandy: so I have already idea how to do this, but maybe you already use it. I need to run gate job with time trigger, meaning that zuul.executor.src_root is not there and git repo is not copied + the branch is missing, so I need a role which can be run (probably as part of prepare-workspace part) that clones specific git repo on a specific branch and uses it in the later run16:18
rlandymigi: I think so ...16:19
*** jmasud has quit IRC16:19
rlandylike http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/zuul.d/projects.yaml#n1416:19
rlandymigi: ^^16:19
rlandythat triggers off a change16:19
rlandybut you can run that periodically16:20
rlandywrt git repo ...16:20
migirlandy: yes16:20
migirlandy: so how to make this to run periodicly16:20
rlandymigi or what raukadah and I just added for podman ...16:20
*** jmasud has joined #oooq16:20
rlandymigi: add the job to the periodic pipeline16:20
rlandymigi: let's chat for 5, I'll explain16:21
migirlandy: yes, but it will be missing the repo, right ?16:21
migirlandy: later, on the call :)16:21
rlandymigi: k - ping me when ready16:21
migirlandy: thx16:21
rlandywill post link here16:21
rlandyhttp://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-config.git/tree/zuul.d/pipelines.yaml#n9416:21
rlandymigi: ^^ periodic pipeline - runs once a day16:22
migirlandy: correct, this I know but adding it to the job requires some change of job itself16:22
rlandymigi: job added to it ... http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d/podman-jobs.yaml and http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d/projects.yaml#n3916:22
rlandy^^ job added16:22
rlandywith git build dependency16:23
migirlandy: ah cool, perfect16:23
migirlandy: anyway will have one question around this later16:24
rlandymigi: k - I'm here16:24
migirlandy: around job - as I would like to move them to tripleo-ci-internal-jobs rather then keep them in osp-jobs16:24
rlandyfine16:24
rlandyI will need to do some config work to enable that16:25
*** jmasud has quit IRC16:29
*** jmasud has joined #oooq16:33
sshnaidmweshay, raukadah is it known? https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_fb8/704938/5/check/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/fb8d3f0/logs/undercloud/home/zuul/overcloud_update_run_Controller.log16:33
sshnaidmfails in gates16:34
weshaysshnaidm, yes16:34
weshaysshnaidm, I think this is the revert to fix https://review.opendev.org/#/c/704885/16:35
weshaysshnaidm, then Cedric just broke scen1016:35
weshayjust another day16:35
weshaysshnaidm, https://review.opendev.org/#/c/705051/16:36
*** jmasud has quit IRC16:40
zbrweshay: sshnaidm: an 1yr old bug that can hog log servers: https://review.opendev.org/#/c/70505916:42
zbrdiscovered one 450MB log file caused by it, and that was on the job that did not fail in POST.16:43
*** udesale has quit IRC16:43
sshnaidmzbr, it's not a bug and done intentionally16:44
sshnaidmzbr, so we can see errors that can't see with regular run16:45
zbruse of -vvvv is unresonable, and you have the proof on https://review.opendev.org/#/c/703586/16:46
zbrin fact the original toci_quickstart.sh file has the correct value, only the j2 has the debug level.16:47
sshnaidmzbr, unreasonable is not an argument16:47
weshayrfolco, can you get in touch w/ mjturek and try to discover what happened here https://bugs.launchpad.net/tripleo/+bug/186134216:47
openstackLaunchpad bug 1861342 in tripleo "tripleo-ci promotion failing on "pull ppc64le tagged containers"" [Critical,Triaged]16:47
weshayOMG.. it's only 9:4716:47
weshayjebus16:47
rlandylol16:47
rlandythe day is but young16:48
sshnaidmzbr, you need more convincing arguments than "unreasonable" to make our life harder and investigation less possible16:48
zbrsshnaidm: sure, please debug this http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/quickstart_collect_logs.log16:49
sshnaidmzbr, it's your patch fault, so you need to fix it16:50
*** holser__ has joined #oooq16:50
*** ykarel is now known as ykarel|away16:51
weshayzbr, the collect log console was 450 MB?16:52
*** holser has quit IRC16:52
zbrhere is how I see it: putting debug level logging in production code without even putting a cap on it, is the cause of this issue.16:52
zbruse of -vvvv in http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/ovb_collect_logs.sh16:53
weshayzbr, show me the 450mb file16:53
weshaythen continue16:53
zbrweshay: look ^16:54
rfolcoweshay, ack... mjturek lets chat :)16:54
sshnaidmzbr, is this log from this patch? https://review.opendev.org/#/c/703586/116:54
weshayk.. 405 http://logs.rdoproject.org/86/703586/12/openstack-check/tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset001/398d3d2/logs/quickstart_collect_logs.log16:55
* weshay looking16:55
weshayzbr, next question.. why is it not compressed?16:55
zbryes, where on rdo job choked and another one managed to product 450MB file16:55
sshnaidmzbr, so what did you do in your patch that generated so big file?16:57
weshayzbr, go talk to Alex https://opendev.org/openstack/tripleo-ci/commit/8d117bb228c8a79f4c67375a1f990239fa718cd316:58
*** marios is now known as marios|out16:59
weshayI can't read the log w/ -vvvv, looks like it was moved to -vvvv to debug a particular issue16:59
zbrweshay: i think alex will say that he forgot to revert it, i am sure.16:59
weshayzbr, probably.. better to ask16:59
migirlandy: so I confirmed it's not possible to use artefacts. Also pipelines, jobs can be shared between tenants16:59
weshayzbr, and to sshnaidm's point.. let's add more facts to commit msg's and less opinions.. I think that would help17:00
rlandymigi: ok - so can you build the rpm in tripleo-ci-internal?17:00
weshayzbr, for example...17:00
weshay-vvvv is producing a very large log file.. 405 ( + link );  looks like this was moved to -vvvv for a particular bug. Let's move it back to -vv17:01
migirlandy: there are 2 options, but let's discuss this tomorrow17:01
rlandymigi: ack17:01
*** bogdando has quit IRC17:04
*** ratailor_ has quit IRC17:05
*** tesseract has quit IRC17:08
*** marios|out has quit IRC17:12
*** jmasud has joined #oooq17:30
*** jmasud_ has joined #oooq17:34
*** jmasud has quit IRC17:34
zbrrlandy: a simple one https://review.opendev.org/#/c/669223/  when you can, thanks.17:36
*** holser__ has quit IRC17:38
*** holser has joined #oooq17:39
weshayzbr, help me out w/ https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a83/702754/8/gate/tripleo-ci-centos-7-containers-undercloud-minion/a83eed9/logs/undercloud/home/zuul/undercloud_install.log.txt.gz17:40
*** jmasud_ has quit IRC17:49
zbrwhat is the problem? i see a success.17:50
*** dtantsur is now known as dtantsur|afk17:52
mjturekrfolco panda: isn't https://bugs.launchpad.net/tripleo/+bug/1861342 happening because the last promoted hash wasn't built by the ppc job (the job was down, it's restored now)17:55
openstackLaunchpad bug 1861342 in tripleo "tripleo-ci promotion failing on "pull ppc64le tagged containers"" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami)17:55
mjtureklet me know if I'm wrong!17:55
weshayzbr, it's not rendering17:57
weshayit's downloading17:57
zbryeah, that is correct. gz files are supposed to downloaded.17:58
rfolcomjturek, try to run this17:58
rfolcopodman pull trunk.registry.rdoproject.org/tripleomaster/centos-binary-aodh-api:03e7a3a58585eb6d751cb6ea765973241f12f2f9_6e3b098e_ppc64le17:58
rfolcoin ppc box17:58
weshayzbr, https://meet.google.com/evr-euer-gad?authuser=117:59
mjturektrying17:59
*** derekh has quit IRC18:00
*** holser has quit IRC18:01
mjturekrfolco: unexpected EoF18:03
rfolcohmm18:03
rfolcofor that hash... did you notice something in the job run ? any unexpected errors ?18:03
rfolcoweird18:03
mjturekdo you know when that hash was?18:03
rfolconot sure18:04
rfolcolets try to find out18:04
mjturekI think I found it18:05
mjturekhttps://centos.logs.rdoproject.org/tripleo-upstream-containers-build-master-ppc64le/1972/logs/logs/containers-successfully-built.log18:05
mjturekbut no errors18:05
mjtureklets try to pull an earlier hash18:05
rfolcoyeah, compare sizes for example18:05
mjturekrfolco we built on that hash twice18:07
mjturekis that okay?18:07
bahaPotential collision?18:08
rfolcodon't see any problems18:08
mjturektrying to pull this now a5f1d5c6e280f12d048ba7ebd2c38189ec9c5070_6e3b098e_ppc64le18:09
mjtureksame error18:09
mjturekrfolco seeing it on other containers as well18:12
rfolcomjturek, this is weird18:13
mjturekvery18:13
rfolcomjturek, docker pull, did you try?18:13
zbrweshay: sshnaidm: https://review.opendev.org/#/c/704938/ would be the correct fix, but as I said this morning should have being be based directly on master18:14
zbrbecause its current parrent failed at the gate, and 8h we are back where we started from.18:14
weshayzbr, still have something borked though.. compression looks good upstream....18:16
weshayhttps://review.rdoproject.org/zuul/build/7454de5906c34f1ebab2e009b41622a918:16
weshayhas no undercloud or overcloud logs18:16
weshaybut the patch is right.. and not the cause18:17
mjturekrfolco working on it18:17
weshayzbr, jobs are fucked now .. and we can't merge this until some other patches land18:17
* weshay steps away18:17
weshayhungry18:18
rfolcomjturek, same hash for _x86_64 works18:19
rfolcomjturek, see if you can find something, I'm working on something else, if need anything just ping me18:20
*** sshnaidm is now known as sshnaidm|afk18:20
mjturekrfolco: same issue with a docker pull18:20
rfolco:(18:22
weshayrlandy,  can you do the honors of another +2 https://review.opendev.org/#/c/705051/18:25
rlandyweshay: done18:26
rlandyweshay: ok - 29 minutes late ... but this works on baremetal testing ... https://review.opendev.org/#/c/705052/18:29
rlandyyou can check my run on the tmate18:29
rlandyTASK [tripleo-inventory : add overcloud node to ansible] *************************************************************************************************************************************$│···············18:30
rlandychanged: [localhost] => (item={'value': u'10.9.122.137', 'key': u'overcloud-novacompute-0'})                                                                                                   │···············18:30
rlandychanged: [localhost] => (item={'value': u'10.9.122.134', 'key': u'overcloud-controller-2'})                                                                                                    │···············18:30
rlandychanged: [localhost] => (item={'value': u'10.9.122.135', 'key': u'overcloud-controller-1'})                                                                                                    │···············18:30
rlandychanged: [localhost] => (item={'value': u'10.9.122.146', 'key': u'overcloud-controller-0'})18:30
weshayrlandy,  we should really switch the depends on18:31
rlandyzbr: looking18:31
weshayrlandy, https://review.opendev.org/#/c/680571 depends on https://review.opendev.org/#/c/705052/518:31
weshayrlandy, please make that change18:31
weshaywe need to merge the inventory patch first18:32
rlandyweshay: I wanted to see the ovb test run18:32
weshayrlandy, you would either way18:32
rlandytrue18:32
rlandyweshay: ugh - in the same repo18:34
rlandyrebase18:34
*** sshnaidm|afk is now known as sshnaidm|off18:53
migiFYI upshift is now refusing to spawn instances19:00
rlandycharming19:12
rlandyis the floating ips again>19:12
migirlandy: yep19:16
weshayrlandy, migi .. you guys haven't heard?19:22
weshayIT'S NOTHING WORKS THURSDAY!!!!!19:23
weshaywoot woot19:23
mjturekrfolco: so some debugging it seems to be an issue with the base container19:24
mjturekthis one blob of the base container seems to be problematic19:25
mjturek23bd9eb8fdc019:25
rfolcomjturek, is the source container a real centos ppc base one ?19:25
mjtureki am not sure19:26
mjturekhmmm let's see where it comes from I guess?19:26
rfolcomjturek, nah ignore me. its the rhel8 that we build a local base and then the rest on top of it.19:31
rfolcomjturek, for ppc, base should follow kolla dockerfile https://github.com/openstack/kolla/tree/master/docker/base19:31
*** jmasud has joined #oooq19:31
rfolcojust like x8619:31
rfolcomjturek, hmmmmmmmm19:33
rfolcolook for arch in that dockerfile...19:33
rfolcohmmmmmm19:33
rfolcohmmmmmm19:33
rfolcoI've got a "cow" moment now19:34
rfolcohmmmmm19:34
rfolcomaybe its ok, just rabbitmq special case19:35
rfolcomjturek, what did make you think base is wrong/broken?19:38
*** jmasud has quit IRC19:42
baharfolco: When we attempt to grab containers, the first blob (whose hash matches up with the first blob of the base container) has to re-attempt downloading many times. It ends up being the last blob to download, and as soon as the progress bar completes, we get the unexpected EOF error20:11
*** hamzy_ has joined #oooq20:11
*** hamzy has quit IRC20:14
rfolcobaha, mjturek: don't know what to try... maybe build only the base one manually and push to dockerhub, then pull20:16
rfolcoI got the steps documented for centos8, you can ignore the specifics and just run the commands20:18
rfolcohttps://hackmd.io/dSagCbocQ4KSVEZR1uf8Tw20:18
rlandymigi: is it expected for codeng to pass for a password when checking out patches?20:19
rlandytrying to pull these patches https://code.engineering.redhat.com/gerrit/#/q/topic:17-standup+status:open20:19
rlandyweshay: ^^ any idea?20:26
weshaysec20:28
mjturekrfolco baha: we could try building locally and then seeing if the base image works as a container20:31
mjturekthoughts?20:31
rfolcomjturek, yes, and you can even use openstack command by tweaking /usr/share/tripleo-common/container-images/overcloud_containers.yaml.j220:33
rfolcofrom tripleo-common20:33
rlandyha - git review orked20:33
rlandyworked20:33
rlandyweird20:33
rfolcomjturek, like pick one that does not have much dependencies and remove the rest from that file20:37
rfolcomjturek, this file comes from https://github.com/openstack/tripleo-common/blob/master/container-images/overcloud_containers.yaml.j220:37
*** jmasud has joined #oooq20:37
mjturekrfolco we were just gonna run the job and build all the containers20:38
mjturekso we guarantee we're not missing something20:38
*** jtomasek has quit IRC20:38
rfolcomjturek, ok, this was for the manual steps to test quicker20:38
mjturekgotcha gotcha20:39
*** jtomasek has joined #oooq20:43
*** jfrancoa has quit IRC20:43
*** jmasud has quit IRC20:44
*** jmasud has joined #oooq20:45
*** jmasud has quit IRC20:47
*** holser has joined #oooq20:55
*** Trevor_V has quit IRC20:55
*** apetrich has quit IRC20:55
*** rfolco has quit IRC21:24
*** rfolco has joined #oooq21:24
weshayrlandy, didn't we say no patches first pass?21:25
weshayrlandy, baby steps21:26
rlandyno patches ==failure21:26
rlandytried that baby step21:26
rlandyin big school now21:26
rlandynvm - I managed to git-review the patches to my box21:27
*** zbr_ has joined #oooq21:31
rlandyugh21:35
rlandyok container build experts ... what does this mean? http://pastebin.test.redhat.com/83201321:36
rlandyrfolco: ^^?21:36
*** zbr_ has quit IRC22:02
*** ssbarnea- has joined #oooq22:08
rlandyweshay: ping re: post failure22:19
rlandywe have no way to qualify inventory change22:19
*** apetrich has joined #oooq22:55
weshayrlandy, oh because of the current state of logging?23:06
weshayrlandy, https://trello.com/c/0pT1zkSe/1316-cixlp1861378tripleociproa-multiple-postfailure-on-master-periodic-pipeline23:08
*** jmasud has joined #oooq23:12
rlandyok23:18
*** jmasud has quit IRC23:18
rlandyweshay: the patches marked as the fix also show the post failure23:23
*** ysandeep has joined #oooq23:40

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!