Monday, 2020-01-27

*** dsneddon has joined #oooq00:06
*** dsneddon has quit IRC00:11
*** jmasud has quit IRC00:12
*** jmasud has joined #oooq00:14
*** ysandeep has joined #oooq00:31
*** dsneddon has joined #oooq00:38
*** dsneddon has quit IRC00:43
*** rfolco has joined #oooq00:52
*** dsneddon has joined #oooq01:14
*** dsneddon has quit IRC01:19
*** dsneddon has joined #oooq01:42
*** rfolco has quit IRC01:45
*** dsneddon has quit IRC01:47
*** dsneddon has joined #oooq02:20
*** dsneddon has quit IRC02:25
*** ysandeep has quit IRC02:28
*** ysandeep has joined #oooq02:48
*** dsneddon has joined #oooq02:54
*** mschuppert has quit IRC02:56
*** dsneddon has quit IRC02:59
*** dsneddon has joined #oooq03:34
*** dsneddon has quit IRC03:39
*** ykarel|away is now known as ykarel04:07
*** dsneddon has joined #oooq04:07
*** jmasud has quit IRC04:11
*** dsneddon has quit IRC04:13
*** jmasud has joined #oooq04:14
*** udesale has joined #oooq04:25
*** dsneddon has joined #oooq04:43
*** raukadah is now known as chkumar|rover04:45
*** dsneddon has quit IRC04:48
*** skramaja has joined #oooq04:49
*** ratailor has joined #oooq04:55
*** ratailor_ has joined #oooq04:56
*** ratailor has quit IRC04:59
*** dsneddon has joined #oooq05:23
*** soniya29 has joined #oooq05:26
*** dsneddon has quit IRC05:27
*** jtomasek has joined #oooq05:28
*** jtomasek has quit IRC05:53
*** dsneddon has joined #oooq05:54
*** dsneddon has quit IRC05:58
*** marios has joined #oooq06:17
*** surpatil has joined #oooq06:21
*** dsneddon has joined #oooq06:27
*** dsneddon has quit IRC06:34
*** dsneddon has joined #oooq07:07
*** dsneddon has quit IRC07:12
*** ykarel is now known as ykarel|lunch07:33
*** dsneddon has joined #oooq07:47
*** dsneddon has quit IRC07:51
*** tosky has joined #oooq07:56
*** jtomasek has joined #oooq08:08
*** ykarel|lunch is now known as ykarel08:11
*** jmasud has quit IRC08:12
*** tesseract has joined #oooq08:13
*** jmasud has joined #oooq08:14
*** dsneddon has joined #oooq08:18
*** dsneddon has quit IRC08:23
*** jbadiapa has joined #oooq08:27
*** ratailor_ has quit IRC08:37
*** amoralej|off is now known as amoralej08:44
*** dsneddon has joined #oooq08:51
*** jpena|off is now known as jpena08:51
*** ratailor has joined #oooq08:51
*** dsneddon has quit IRC08:56
*** apetrich has joined #oooq08:56
*** chem has joined #oooq09:08
zbro/09:13
zbrkopecmartin: ping me once you fix https://review.opendev.org/#/c/682042/ -- especially the ignore_errors note09:13
zbrand use of lookup('env', TMP) to get temp directory09:14
kopecmartinzbr: that should be done, see patchset 2109:16
*** bogdando has joined #oooq09:23
*** ccamacho has joined #oooq09:25
*** dsneddon has joined #oooq09:29
*** jfrancoa has joined #oooq09:30
*** dsneddon has quit IRC09:33
*** derekh has joined #oooq09:37
*** saneax has joined #oooq09:59
zbrkopecmartin: look at https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_15f/682042/21/check/openstack-tox-molecule/15fb782/tox/reports.html -- i still see one red task: "Uninstall ansible-role-collect-logs plugin"10:07
zbrmake it green, without cheating ;)10:08
zbrkopecmartin: overall it looks much much nicer.10:09
*** dsneddon has joined #oooq10:09
zbrbtw, i already merged few new defaults in molecule for ansible.cfg that should solve the infrared warnings, a new release will be probably later this week.10:10
zbrone that I di dnot add is timeout=30, and I wonder if I should or not.10:12
zbrsshnaidm: what do you think about this? should molecule have an internal default of timeout=30 or not?10:13
*** dsneddon has quit IRC10:14
sshnaidmzbr, timeout of what?10:17
zbrsshnaidm: ansible.cfg, is timeout for ssh connections.10:18
zbrafaik, we have this set in most places.10:18
zbrbut molecule by itself never uses the in-repo ansible.cfg file, always generates one, based on its internal defaults + what you may override in the schenario10:19
zbrsee https://github.com/ansible-community/molecule/pull/2514/files10:19
zbrthis is in order to isolate the testing grounds from local settings.10:19
sshnaidmzbr, and what is the default for ansible?10:19
zbryep.10:20
zbrnope10:20
zbransible default timeout is 10s, https://docs.ansible.com/ansible/latest/reference_appendices/config.html#default-timeout10:20
zbrand I am not sure if we should make it bigger or not.10:21
zbri can prove useful when dealing with clouds, where provisioning a vm can take more than 30s, or networking glitches.10:22
*** dtantsur|afk is now known as dtantsur10:23
sshnaidmzbr, I'm not sure provisioning uses this timeout10:24
sshnaidmzbr, anyway, it doesn't seem like a biggie10:24
zbri will defer for the moment, I do not want to add defaults specially for infrared.10:25
zbrkopecmartin: please add a timeout value to molecule.yaml so infrared does not complain about it not being defined.10:26
kopecmartinzbr: ok10:27
*** sshnaidm is now known as sshnaidm|afk10:27
zbrin fact you can define all values needed for making it happy.10:27
kopecmartinok, will do10:28
*** jfrancoa has quit IRC10:28
*** sanjayu_ has joined #oooq10:32
*** surpatil has quit IRC10:32
*** saneax has quit IRC10:34
*** dsneddon has joined #oooq10:43
*** ysandeep has quit IRC10:47
*** jtomasek has quit IRC10:50
*** dsneddon has quit IRC10:50
pandamarios: I've been working on the next steps too https://review.rdoproject.org/r/#/q/status:open+AND+topic:%22transition_prepare%22 There's a lot there again to see, but I just hope people understand what's the idea behind the transition. Moreover, I hit a bump with the check hash function, the global var is driving me crazy, I think I'll need to transition those functions sooner than I expected, so I'll help10:56
pandayou merge the bug fix now.10:56
*** udesale has quit IRC10:58
pandamarios: reviewed the bugfix.11:01
*** holser has joined #oooq11:02
*** holser__ has joined #oooq11:05
*** holser has quit IRC11:05
*** saneax has joined #oooq11:06
chkumar|roverzbr, please have a look at this http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train/5ea58d2/job-output.txt11:06
chkumar|roverI am not sure where logs folder went11:06
chkumar|rover*vanished11:06
*** sanjayu_ has quit IRC11:07
mariospanda: o/ sorry distracted by other/rhel8 task right now... i will switch to that later and update https://review.rdoproject.org/r/#/c/24665/ and work on the https://tree.taiga.io/project/tripleo-ci-board/task/148411:19
mariospanda: thanks for comment i'll look more closely ^11:19
*** dsneddon has joined #oooq11:22
*** sshnaidm|afk is now known as sshnaidm11:34
zbrchkumar|rover: for ovb, i need to change old collect logs, working on it.11:36
zbri should have a CR ready before tomorrow11:36
kopecmartinzbr: i set the timeout and forks in ansible.cfg , but it didn't help, those values for some reason didn't get to /home/*/.cache/molecule/.*/infrared/ansible.cfg, molecule automagically generated that file without those options: http://paste.openstack.org/show/788822/11:51
zbrkopecmartin: https://molecule.readthedocs.io/en/stable/configuration.html - search for "Modifying ansible.cfg."11:56
kopecmartinOMG11:56
zbrthere is a design reason why molecule does creates and manages the file itself.11:57
zbrtest-isolation11:57
*** dsneddon has quit IRC11:57
chkumar|roveramoralej, marios, fyi, ceph centos8 repos will be available by End of wednesday11:58
amoralejvery cool!11:58
amoralejchkumar|rover, i already did uwsgi in CBS11:58
amoralejetcd will be the hardest11:59
amoralejcan we start running ci without etcd?11:59
amoralejiirc it's not used in njobs11:59
marioschkumar|rover: thx11:59
*** rfolco has joined #oooq12:00
arxcruzkopecmartin: meeting ;)12:01
weshay_kopecmartin, hey12:01
kopecmartinoh, a second12:01
sshnaidmzbr, can you +w https://review.opendev.org/#/c/704059/ please12:02
weshay_zbr, ping.. re collect logs12:06
*** holser__ has quit IRC12:06
*** holser has joined #oooq12:07
zbrweshay_: yeah, i did not forget.12:10
*** jmasud has quit IRC12:11
*** jmasud has joined #oooq12:13
*** holser has quit IRC12:22
weshay_kopecmartin, I need to grab my charger.. will step away for a second12:25
*** jpena is now known as jpena|lunch12:28
*** jfrancoa has joined #oooq12:30
*** dsneddon has joined #oooq12:31
*** dsneddon has quit IRC12:36
*** jfrancoa has quit IRC12:37
weshay_zbr, it's going on now12:38
*** jfrancoa has joined #oooq12:40
*** rlandy has joined #oooq12:50
*** Goneri has quit IRC12:53
*** ratailor has quit IRC13:00
rfolcozbr, weshay_ arxcruz marios rlandy sshnaidm panda: scrum time13:00
mariosnick marios|call13:00
arxcruzrfolco: in another meeting, just a sec13:00
*** marios is now known as marios|call13:00
weshay_rfolco, be there in 3 min13:00
rfolcowhere is chandan13:01
chkumar|roverrlandy, Hello13:01
chkumar|roverrlandy, https://sf.hosted.upshift.rdu2.redhat.com/logs/84/190384/5/check/podman-package-rhel-8/ce1f5fb/job-output.txt13:01
chkumar|roverrlandy, we made some progress there, need to enable one repo and figure out how to get go-compilers-golang-compiler13:01
chkumar|roverpackages13:01
pandaoh13:02
rlandycool - will check it out13:02
weshay_chkumar|rover, 0/13:02
weshay_got our promotion for master :)13:02
chkumar|roverweshay_, yes sir :-)13:02
rlandychkumar|rover: yeah ... I really didn't want to duplicate the playbooks entirely13:04
rlandychkumar|rover: can we put that work into roles?13:04
rlandychkumar|rover: then we will need playbooks but they will be much lighter13:04
rlandythe other option is to all jobs from rdo-jobs13:04
rlandychkumar|rover: ^^ that may open a can of worms13:04
rlandybut that is fine until we get the job working13:05
*** weshay_ is now known as weshay|ruck13:05
pandamarios|call: I don't see  ppc containers in the master promotion :(13:05
*** ykarel is now known as ykarel|afk13:06
*** dsneddon has joined #oooq13:07
*** dsneddon has quit IRC13:14
weshay|ruckchkumar|rover, let's drop at the bottom of the hour and go to cix together13:15
chkumar|roverweshay|ruck, yes13:15
chkumar|roverrlandy, will make it working in first goal then move it to role13:15
rlandychkumar|rover: ack13:21
marios|callpanda: o/ you mean they are not being pushed to registry yet? don'tknow i thought mjturek said they had already pushed some ppc tagged containers13:21
*** skramaja has quit IRC13:22
pandamarios|call: they are in rdo, I don't see them in docker.io13:24
weshay|ruckzbr, added and assigned13:25
*** soniya29 has quit IRC13:26
zbrweshay|ruck: which one? i am lost in taiga13:26
weshay|ruckzbr, sorry... your cards should be in tech debt user story13:28
* weshay|ruck fixes13:28
marios|callpanda: hmmm we can check logs from manifest push ... it should retrieve them from rdo and push to docker.io ... maybe we didn't get a hash yet for which there are both x86 and ppc?13:30
zbrimho, we could review/wf the archival code even if it does not work on the other jobs, because it does not work because is not called, not because it would be deffective.13:30
*** jpena|lunch is now known as jpena13:31
*** ykarel|afk is now known as ykarel13:35
pandamarios|call: I'm looking at this https://hub.docker.com/r/tripleomaster/centos-binary-nova-api/tags?page=1&ordering=last_updated13:37
pandamarios|call: they are in order, and the latest hash that promoted doesn't have any _ppc, I'll look at the logs13:38
marios|callpanda: ack as we just spoke about let's sync later /tomorrow would be good if you could check re the hash13:39
marios|callpanda: otherwise i will check it in my morning tomorrow and we can catch up13:39
*** dsneddon has joined #oooq13:40
*** Goneri has joined #oooq13:44
*** dsneddon has quit IRC13:45
*** jtomasek has joined #oooq13:48
*** chem has quit IRC13:49
*** chem has joined #oooq13:51
*** jtomasek has quit IRC13:54
chkumar|rovermarios|call, can we shift the pune bootcamp call to 30 mins, otherwise I might miss13:55
chkumar|roversorry need to leave out13:58
zbrhow about making the standup time limites to 15" ?13:58
*** holser has joined #oooq13:58
*** marios|call is now known as marios13:58
marioschkumar|rover: bit too late for that ...14:00
*** jtomasek has joined #oooq14:00
marioschkumar|rover: rather not try to change the call 2 mins before it starts ... hard to get a time everyone agrees on14:00
*** udesale has joined #oooq14:03
zbrmy 3mo old ticket is getting hot at https://bugs.centos.org/view.php?id=16389#c36135 :D14:12
*** dsneddon has joined #oooq14:14
zbrweshay|ruck: chkumar|rover: hotfix needed for tci repo: https://review.opendev.org/#/c/704326/ -- already passed what it matters, better to wf now.14:14
zbrthis means, nothing will get merged to tci, in the next 3ho.14:14
zbrnot sure exactly how it happened, probably due the lack of pinning.14:15
*** marios is now known as marios|call14:15
zbreven if these checks where eons old, i wonder if someone bypassed the gate.14:16
*** dsneddon has quit IRC14:19
zbrmaybe someone can tell me how can I see all stories assigned to me in taiga... i was not able to find such a fiew14:20
zbronly issues, which are not the same14:20
*** holser has quit IRC14:29
*** jtomasek has quit IRC14:40
arxcruzsomething happens, i need to get back home, brb14:44
*** jtomasek has joined #oooq14:45
pandaI sense a disturbance in the force14:47
pandazbr: https://tree.taiga.io/project/tripleo-ci-board/taskboard/unified-sprint-21?kanban-status=1447274&assigned_to=29865014:48
zbrpanda: that is not a list of stories, I need to scroll 2 screens to find the first story, and works only for current sprint.14:49
pandazbr: https://tree.taiga.io/project/tripleo-ci-board/backlog?q=&assigned_users=29865014:50
*** dsneddon has joined #oooq14:53
zbrpanda: challenge for you, find a "list" repor that shows the task #1462 that wes assigned it to me during this sprint14:53
*** TrevorV has joined #oooq14:53
zbrclearly is not listed under your link, neither under "issues" report.14:53
zbrafaik, the only way to find it, is to open the current sprint board14:54
zbrwhich requires horizontal and vertical scrolling to find that.14:54
zbrmainly, on taiga is impossible to get a user "todo" list.14:54
zbrtheir decision to consider epics, stories, tasks and issues something that cannot be listed together created a maintenance chaos.14:56
*** dsneddon has quit IRC14:58
zbrthey are similar in nature, granularity being the main differentiator, still no way to deal with them in one place.14:58
pandazbr: use raw api call for more customized views14:59
pandamjturek: hey, can you check if any ppc containers were successfully built for hash 9548111e24c5cb35e4d4990b9707d462e53692f2_b7487084 ?14:59
zbrzbr: sure. let me go back to DIY mode, back in 3 months with a15:03
zbra taiga cli gui15:03
pandazbr: echo GET /api/tasks | nc  www.taiga.io15:05
zbrpanda: i would be much happier to switch to github.15:05
*** marios|call is now known as marios15:06
sshnaidmweshay|ruck, rlandy should we sync about OVB topics to cover?15:07
rlandysshnaidm: ack15:07
rlandymarios: want to meet now or after company meeting?15:07
mariosrlandy: yeah rather after if ok?15:08
*** marios is now known as marios|call15:08
rlandymarios|call: sure - pine me when you are ready15:09
rlandyping15:09
marios|callrlandy: trying to summarize my stuff perhaps we can avoid the call (will update taiga and see if it helps/we take from there )15:10
rlandymarios|call: fine by me ... going o do some training stuff this afternoon anyways15:10
rlandymight not make a lot of container progress15:11
marios|callrlandy: ack will ping you either way in a bit anyway15:11
*** ykarel is now known as ykarel|away15:11
mjturekpanda: Job is down because of cico nodes being unavailable15:15
mjturekseems like the people who resolve them are out15:15
mjturekso I doubt we built them15:16
mjturekahhh actually them should be back now15:16
pandamjturek: /o\15:18
mjturekpanda: looks like we're still having trouble checking out from cico despite them handling the issue15:18
mjturekso we did not build that hash15:19
pandamjturek: we promoted master, but the ppc containers were not there, so they whre not pushed to docker.io15:19
mjturekuhg, great timing on cico's part :)15:19
mjturekwill there be another promotion in a few days?15:20
chkumar|roverI am back now15:20
chkumar|roverweshay|ruck, rlandy https://review.opendev.org/#/c/704268/ merge this please15:20
weshay|ruckchkumar|rover, yup :)15:21
pandamjturek: who knows ? getting a master promotion is like finding a prime number.15:21
mjturekright right15:21
mjturekhopefully now that we have the job restored there will be one soonish15:21
mjturekpanda let me know again if/when it happens15:22
pandamjturek: sure15:22
chkumar|roversshnaidm, Hello15:28
chkumar|roversshnaidm, http://logs.rdoproject.org/82/24682/1/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train/0f570b0/job-output.txt15:28
chkumar|roverseen this issue earlier15:29
*** dsneddon has joined #oooq15:29
chkumar|roverKeyError: u'baremetal-24682-1-63190-extra_0'15:29
weshay|ruckrlandy, I still have a free pass on env D right?15:29
weshay|ruckfree pass to use it15:29
rlandyweshay|ruck: yeah ...15:29
weshay|ruckrlandy, thanks!15:29
rlandyuntil bootcamp15:30
rlandyweshay|ruck: ^^ I need it back then15:30
weshay|ruckk15:30
*** holser has joined #oooq15:30
chkumar|roverweshay|ruck, fs039 is broken on master and train with same error15:30
chkumar|roverhttp://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train/d3a6490/logs/undercloud/var/lib/mistral/overcloud/ansible.log15:31
chkumar|roverpuppet-user: Error: Could not find group qemu15:31
weshay|ruckchkumar|rover, don't worry about master for that job..15:32
weshay|rucklet's look at train15:32
weshay|ruckchkumar|rover, https://review.rdoproject.org/zuul/builds?job_name=%09periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train15:32
*** dsneddon has quit IRC15:33
arxcruzback15:40
chkumar|roverweshay|ruck, http://logs.rdoproject.org/83/24683/1/check/distgit-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset021-train/e6c4a3d/logs/undercloud/var/log/tempest/tempest_run.log15:42
chkumar|roverweshay|ruck, arxcruz http://logs.rdoproject.org/83/24683/1/check/distgit-periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset021-train/e6c4a3d/logs/undercloud/var/log/tempest/tempest_run.log15:42
chkumar|roverif we update to temoest-23.0.0 in train, we get 15 passes15:42
*** dsneddon has joined #oooq15:45
arxcruzchkumar|rover: let's do it :D15:48
arxcruzchkumar|rover: 23 is the latest version to support python 2 iirc15:48
weshay|ruckchkumar|rover, AH nice15:48
weshay|ruckdo it :)15:48
chkumar|roverarxcruz, but ssh timeout still coming15:48
chkumar|roversomething real issue there15:48
arxcruzchkumar|rover: yeah... network is a problem15:48
arxcruzi know there were some changes in the ssh but not sure if it was included in v2315:49
chkumar|roverarxcruz, https://review.opendev.org/#/c/703072/15:50
chkumar|roverif needed, I can backport it15:50
arxcruzchkumar|rover: yeah, do you think this might fix the problem? worht we do some tests15:51
chkumar|roverarxcruz, may be poke tomorrow on that15:51
chkumar|roverweshay|ruck, arxcruz we need some training from network dfg on how to dig ssh timeout issue, multiple factors involved there15:52
arxcruzchkumar|rover: yeah, i need to read the openstack networking book I got on the summit, and i didn't even open xD15:53
weshay|ruckchkumar|rover, most of the failures in envD fs021.. were ssh timeouts15:54
weshay|ruckchkumar|rover, re: 039 qemu group exists http://logs.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train/f6b44c4/logs/overcloud-novacompute-0/etc/group15:56
chkumar|roverbut puppet is ignoring the changes15:57
sshnaidmchkumar|rover, fixing 039..15:59
chkumar|roversshnaidm, there are two issues we are discussing currently once we run periodic job in check and another overcloud deploy qemu issue16:00
sshnaidmchkumar|rover, I'm fixing this key error16:01
chkumar|roversshnaidm, ok, thanks :-)16:01
weshay|rucksshnaidm, you know what's up w/ 039?16:02
sshnaidmweshay|ruck, yes16:03
chkumar|roverweshay|ruck, http://logs.rdoproject.org/82/24682/1/check/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train/0f570b0/job-output.txt16:03
*** marios|call is now known as marios16:04
weshay|rucksshnaidm, nice.. /me watches for patches16:04
weshay|ruckmust be tripleo-ansible16:04
*** jtomasek has quit IRC16:05
*** soniya29 has joined #oooq16:06
zbrweshay|ruck: are you avl for some debugging on job deps? rdo seems not to find the role, but upstream finds it.16:08
zbrit all starts with https://review.opendev.org/#/c/704044/6/zuul.d/base.yaml16:08
zbrwhich tells zuul to install this role, with this name.16:09
zbrupstream job was able to find the role, but rdo, nope.16:09
mariosrlandy: added notes to https://tree.taiga.io/project/tripleo-ci-board/task/1500 feel free to move to hackmd or update there ... should get you to built base image16:10
weshay|ruckzbr, look for     nodeset: two-centos-7-nodes16:10
weshay|ruck in rdo job config16:10
zbrjpena: ^ maybe you know what could prevent it.16:10
mariosrlandy: i will check it tomorrow and try to progress before you com in16:10
*** saneax has quit IRC16:10
rlandymarios: thanks - looking16:10
weshay|ruckchkumar|rover, stein just promoted :)16:11
chkumar|roveryay!16:11
chkumar|roverwe need to work on train tomorrow16:11
weshay|ruckchkumar|rover, don't worry about rocky... just train / master16:11
weshay|ruckchkumar|rover, all we need is fs03916:11
weshay|ruckfor train16:12
zbrweshay|ruck: tripleo-build-containers-rhel-8-master has tripleo-ci-base as parent, it should find the role.16:12
*** jmasud has quit IRC16:12
zbrthey all inherit from the same place.16:12
kopecmartinzbr: weshay|ruck marios new changes ready, jobs passed, please re-review https://review.opendev.org/#/c/682042/16:15
weshay|ruckzbr, k.. then go grab the engineer-on-duty in #sf-ops internal and ask.. about that being cloned by zuul16:15
chkumar|roversshnaidm, this us https://tree.taiga.io/project/morucci-software-factory/us/3222 might be interesting for you dmsiard is working on that this sprint16:16
sshnaidmchkumar|rover, indeed, let's see how it's going16:17
chkumar|roversshnaidm, https://tree.taiga.io/project/morucci-software-factory/us/3283 and https://tree.taiga.io/project/morucci-software-factory/us/3278 regarding vexxhost , prodchain is planning to move to vexxhost, jpena want to know to the status related to smoke test vexxhost testing, when is the good time to move some jobs there16:17
chkumar|rovermarios, zbr, rlandy https://tree.taiga.io/project/morucci-software-factory/us/3314 it might be interesting for us16:18
*** udesale has quit IRC16:19
weshay|ruckkopecmartin, why can't this be temp?         infrared_location: "{{ lookup('env', 'HOME') or '~' }}/.infrared"16:20
weshay|ruckmktemp16:21
kopecmartinweshay|ruck: it can, i guess, it's about preferences, we can have either a global variable defined in molecule.yml for all playbooks or a tempfile task in every playbook16:22
kopecmartini don't care which one16:22
sshnaidmchkumar|rover, well, I'm not sure I'm a right address for this question, where are you talking?16:22
sshnaidmchkumar|rover, from my side - do it today :D16:23
weshay|ruckkopecmartin, I think the concern zbr  and I shared was running molecule locally would impact someone's infra venv16:23
weshay|ruckzbr, right?16:23
mariosack kopecmartin but tomorrow16:23
zbrweshay|ruck: yep, this was one of my concerns16:24
marioschkumar|rover: ack nice16:24
zbrweshay|ruck: but devil is in the details, I know that due to shebang limitations infrared was forced to use home directory instead tmp.16:25
weshay|rucksshnaidm, rlandy how does one create remote branches in testproject16:26
weshay|ruckI need to test stable/train but it's a not a remote branch16:26
zbrweshay|ruck: FYI, on macos, mktemp returns something like /var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/tmp.RDyYiWN8 -- if you create a virtualenv inside it... you may discover some tools may not be able to run due to broken shebangs.16:26
weshay|ruckcan I do a branch_override or something?16:26
sshnaidmweshay|ruck, admin I suppose, rdo folks16:26
*** chem has quit IRC16:26
weshay|ruckugh16:26
weshay|ruckk16:26
rlandyweshay|ruck: testproject has stable/train, no?16:26
weshay|ruck[whayutin@localhost testproject]$ git branch -a16:27
weshay|ruck  foo16:27
weshay|ruck* master16:27
weshay|ruck  remotes/gerrit/liberty-patches16:27
weshay|ruck  remotes/gerrit/master16:27
weshay|ruck  remotes/gerrit/stable-4.016:27
weshay|ruck  remotes/gerrit/stable/liberty16:27
weshay|ruck  remotes/gerrit/stable/pike16:27
weshay|ruck  remotes/gerrit/stable/queens16:27
weshay|ruck  remotes/gerrit/stable/rocky16:27
*** chem has joined #oooq16:29
sshnaidmhttps://review.rdoproject.org/r/#/admin/projects/testproject,branches16:32
chkumar|roverrlandy, magic is coming soon, wait few mins16:34
*** ccamacho has quit IRC16:34
rlandychkumar|rover: podman downstream ?16:34
rlandymarios: small typo - otherwise instructions worked :)16:36
rlandyweshay|ruck: ^^ difference being 'latest' image rather than 'released'16:36
chkumar|rovermarios, please something special for me to pune16:36
chkumar|rover*please bring16:36
marioschkumar|rover: ? what you need16:37
rlandychkumar|rover: what would you like from the US?16:37
marioschkumar|rover: will bring some candy :)16:37
chkumar|roverdonot know, what you make it special, that will work16:37
rlandymarios: I got a failure on step 1916:38
rlandythink it's a dependency issue16:39
mariosrlandy: maybe i missed one of the files in my summary? trace?16:39
chkumar|roversshnaidm, rlandy marios weshay|ruck zbr arxcruz panda https://sf.hosted.upshift.rdu2.redhat.com/logs/84/190384/12/check/podman-package-rhel-8/0ccbab9/buildset/16:39
chkumar|roverhere is the magic first podman rpm build downstream16:40
chkumar|roverjpena, ^^16:40
rlandychkumar|rover++16:40
rlandynice16:40
zbranother broken SSL...16:40
chkumar|roverweshay|ruck, rlandy Now we can tell the world, how podman is broken or viceversa16:40
chkumar|roverrlandy, https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/c2ed979974874a8b925c76ddbbe6f6ce?logfile=console.log standlone running here16:41
jpenachkumar|rover: nice!16:41
rlandychkumar|rover: good luck with that16:41
chkumar|roverrlandy, feel free to mess up in night16:41
rlandyI tried to tell them last time and nobody listened :(16:42
mariosrlandy: ah i missed the 'typo' bit ack so i'll check there /or email if you get further update and i can pickup16:42
* marios shutdown sequence16:42
rlandymarios: no worried - I fixed the typo16:42
weshay|ruckchkumar|rover++16:42
mariosrlandy: ack16:42
weshay|ruckwho wants to help their dumb boss on the proper way to create remote gerrit branches16:43
jpenazbr: I've seen your issue with the ansible-roles-collect-logs role not seen by review.rdo. Is that project included in the review.rdo config?16:44
rlandyjpena: thanks for helping with downstream podman work16:44
zbrneed to check, but my assumption is that the entire openstack/ namespace would have being whitelisted.16:44
zbralso more interesting is that I did not see an error from zuul complaining that it cannot install the rule, instead I seen import_role failing during job execution.16:45
zbror maybe zuul is very silent about failures in "roles:" definitions.16:45
*** marios is now known as marios|out16:48
*** dsneddon has quit IRC16:49
chkumar|roverrlandy, jpena https://sf.hosted.upshift.rdu2.redhat.com/logs/84/190384/12/check/tripleo-podman-integration-rhel-8-standalone-rhos-16/c2ed979/job-output.txt it failed "msg": "AnsibleUndefinedVariable: 'buildset_artifacts_url' is undefined"16:51
chkumar|roveri think you know the fix, we will poke in the morning16:51
*** jtomasek has joined #oooq16:51
rlandychkumar|rover: zuul should know that var16:51
rlandyConfig error: Parsing file "/etc/dnf/dnf.conf" failed: Parsing file '/etc/dnf/dnf.conf' failed: IniParser: Missing section header at line 116:52
rlandylooks ok to me16:52
chkumar|rovermay be jpena ^6 knows16:53
*** soniya29 has quit IRC16:54
chkumar|roversee ya tomorrow, have a nice day & night ahead :-)16:54
*** chkumar|rover is now known as raukadah16:54
weshay|ruckin retrospect.. I guess master would have been fine.. given the job is branched16:56
weshay|ruckDUHHH16:56
weshay|ruckmore COFFEE16:56
weshay|rucksshnaidm, https://review.rdoproject.org/r/#/c/24688/ raukadah16:56
*** dtantsur is now known as dtantsur|afk16:57
rlandyweshay|ruck: https://review.rdoproject.org/r/#/c/23558/ worked for me16:58
rlandycreated local stable/train16:58
weshay|ruckrlandy, that is topic not branch :)16:59
weshay|ruckbut.. the jobs are branched.. so .. it's moot I guess but good to have17:00
*** jtomasek has quit IRC17:00
rlandyweshay|ruck: with osp container builds I get as far as ...17:02
rlandySTEP 19: RUN dnf -y update     && dnf -y install ca-certificates curl findutils iproute iscsi-initiator-utils lvm2 ncurses procps-ng socat sudo tar which python3 cronie crudini dumb-init openstack-tripleo-common-container-base puppet-tripleo rsync iptables-services     && dnf clean all && rm -rf /var/cache/dnf17:02
rlandyFailed to set locale, defaulting to C.UTF-817:02
rlandyConfig error: Parsing file "/etc/dnf/dnf.conf" failed: Parsing file '/etc/dnf/dnf.conf' failed: IniParser: Missing section header at line 117:02
rlandyerror building at STEP "RUN dnf -y update     && dnf -y install ca-certificates curl findutils iproute iscsi-initiator-utils lvm2 ncurses procps-ng socat sudo tar which python3 cronie crudini dumb-init openstack-tripleo-common-container-base puppet-tripleo rsync iptables-services     && dnf clean all && rm -rf /var/cache/dnf": error while running runtime: exit status 117:02
rlandyetc/dnf/dnf.conf looks fine to me17:03
weshay|ruckhrm17:03
weshay|rucktmate?17:03
rlandygetting17:04
*** marios|out has quit IRC17:07
weshay|ruckrlandy, can you launch the ubi image as is17:11
weshay|ruckwe have to exec into that17:11
rlandyweshay|ruck: we downloaded multiple17:12
weshay|rucksshnaidm, are you fixing http://pastebin.test.redhat.com/830719 that issue?17:12
weshay|ruckthat makes more sense to me17:12
rlandymaybe should get rid of the first ones we downloaded17:12
sshnaidmweshay|ruck, yep17:12
weshay|rucksshnaidm, oddly we're not hitting that in periodic17:12
weshay|rucksshnaidm, ya.. two diff thing17:12
weshay|rucks17:12
weshay|ruckrlandy, ya.. pick one.. launch it.. let's look17:12
sshnaidmweshay|ruck, oh, right, in periodic it's different17:12
sshnaidmweshay|ruck, will check if it's not breaking periodic job..17:13
weshay|rucksshnaidm, periodic is failing on a puppet issue afaict17:13
*** dsneddon has joined #oooq17:13
sshnaidmweshay|ruck, well, this is a different issue17:13
weshay|rucksshnaidm, aye17:13
weshay|ruckraukadah, ^17:13
weshay|ruckmiscommunication17:14
weshay|ruckraukadah, but since it's fubar'd .. going to promote w/o it17:14
*** bogdando has quit IRC17:16
zbrjpena: so https://review.rdoproject.org/r/#/c/24689/ is what i need to do to make it usable by zuul.17:16
weshay|ruckrlandy, so it's up?17:20
weshay|ruckshall we exec?17:20
weshay|ruckrlandy, hrm.. both exited17:21
rlandyweshay|ruck: let's check marios instance17:21
rlandyto see if he got it running successfully17:22
rlandyweshay|ruck: hmm ... not letting me ssh ... he used a diff image to launch the instance though cloud-rhel-8-1-157998791817:24
rlandyugh ... got some other work to complete ... will get back to this17:24
rfolcoweshay|ruck, get-hash needs to keep backwards compat mode right?17:29
rfolcoweshay|ruck, for centos7, parsing commit.yaml into distro_hash + commit_hash = full_hash17:30
rfolcowhich will function the same as centos8 component one17:30
rfolcoand the 3rd case is centos8 integration, which gets full_hash=md5. End of story.17:31
zbrweshay|ruck: rfolco : wf https://review.opendev.org/#/c/704326/2 now17:38
*** tesseract has quit IRC17:38
weshay|ruckzbr, that's unrelated to the rdo rhel8 tox right?17:47
weshay|ruckor same issue?17:47
*** amoralej is now known as amoralej|off17:49
*** jpena is now known as jpena|off17:57
weshay|ruckarxcruz, you still around?18:12
weshay|ruckrlandy, check this out18:12
weshay|ruck2020-01-26 20:14:52.627422 | primary | TASK [os_tempest : Ensure private network exists] ******************************18:12
weshay|ruck2020-01-26 20:14:52.724417 | primary | Sunday 26 January 2020  20:14:52 -0500 (0:00:00.164)       0:54:08.343 ********18:12
weshay|ruck2020-01-27 03:01:09.342407 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]18:12
weshay|ruck2020-01-27 03:01:09.343809 | POST-RUN START: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/post.yaml@master]18:12
weshay|ruckhttps://sf.hosted.upshift.rdu2.redhat.com/logs/periodic/code.engineering.redhat.com/openstack/tripleo-ci-internal-jobs/master/tripleo-ci-rhel-8-standalone-rhos-16/4bfda7e/job-output.txt18:12
weshay|ruckrlandy, so.. fyi.. osp-16 container deployment is working.. osp-15 is still busted18:13
weshay|ruckrlandy, ping me later re: psi18:13
rlandyweshay|ruck: what is this from?18:28
weshay|ruckrlandy, OSP-16 standalone18:28
weshay|ruckrlandy, just fyi to you.. the container issue is resolved18:29
weshay|ruckbut hitting something new18:29
weshay|ruckI'll debug18:29
weshay|ruckw/ chandan18:29
rlandyweshay|ruck: oh yeah18:29
rlandyI kicked a restproject run18:29
rlandyweshay|ruck: we have success on osp15 running branches18:30
* rlandy gets18:30
rlandyrhos-release-rhel-8-rhos-15rhos-releasemastergate190454,1542020-01-27T16:15:45SUCCESS18:30
rlandyrhos-release-rhel-8-rhos-15rhos-releasemastercheck190454,1662020-01-27T16:12:35SUCCESS18:30
rlandyweshay|ruck: ^^18:31
weshay|ruckhuh.. k18:31
weshay|ruckdepends-on w/ those?18:31
weshay|ruckor from passed phase 118:31
weshay|ruckbtw.. OMG18:31
weshay|ruckI looked at osp-15,14 etc..18:31
weshay|ruckosp phase 1/2 hasn't run in like a month18:32
rlandyoh wait wrong job18:32
weshay|ruckwe need to pick a diff pin18:32
weshay|ruckthey have all lasers pointed at 1618:32
weshay|ruckrfolco, re: centos base images18:34
weshay|ruckand the name collision18:34
weshay|ruckany thoughts on that?18:34
rlandyweshay|ruck: ok - so the osp-16 testproject was the first pass we saw18:34
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=tripleo-ci-rhel-8-standalone-rhos-1618:34
*** sshnaidm is now known as sshnaidm|afk18:35
weshay|ruckrlandy, so.. the timed_out jobs18:35
weshay|ruckright?18:35
weshay|ruckgot through deploy but failed in os_tempest network18:35
weshay|ruckzbr, ping me re: power supply18:36
rlandywell ... this is good though ..18:36
zbrweshay|ruck: i received it.18:36
zbrtwo days ago18:36
weshay|ruckoh18:36
weshay|ruckwell good18:36
weshay|ruckur welcome :)18:36
zbrzbr: UPS to be correct. thanks.18:36
rlandytripleo-ci-rhel-8-standalone-rhos-16ansible-role-tripleo-modify-imagerhos-16.0-trunk-patchescheck190206,4111902020-01-24T21:19:11TIMED_OUT18:36
rlandytripleo-ci-rhel-8-standalone-rhos-16openstack/tripleo-ci-internal-jobsmasterperiodicmaster29862020-01-24T00:02:34FAILURE18:36
rfolcoweshay|ruck, didn't think on a solution for name collision18:37
rlandyso basically yes, we have a timeout now18:37
weshay|ruckrfolco, what happens if we pull the base image from rdo-cloud w/ a different name?18:37
rfolcoweshay|ruck, I can add this to community mtg agenda18:37
weshay|ruckrfolco, yes please18:37
rfolcok18:37
weshay|ruckrfolco, thanks for sending the summary18:38
rlandyhttps://github.com/openstack/openstack-ansible-os_tempest/blob/stable/train/tasks/tempest_resources.yml#L14618:39
rfolcoweshay|ruck, hope its all good, I set a timeout and assume its ok to send18:39
rfolcoweshay|ruck, can eventually send an errata18:39
rfolcoerratum?18:40
rlandythat's probably a settings thing18:40
*** holser has quit IRC18:48
weshay|ruckzbr, fyi.. https://review.opendev.org/#/c/704326/218:56
rlandyweshay|ruck: want t look at psi resources now?19:13
weshay|ruckrlandy, sure19:13
weshay|ruckhttps://meet.google.com/ztq-eppm-xvj?authuser=119:13
zbrweshay|ruck: thanks, fixed. another issue, unrelated to my patch.19:23
sshnaidm|afkzbr, any ideas how can I limit molecule runs for the role only if I change this role? https://review.rdoproject.org/r/#/c/24691/19:47
zbrsshnaidm|afk: very risky business because a role may and will also break due to external factors, like ansible, nodes, ... time.20:02
zbrafaik, is like saying that you want to run unittests only when specific classes are modified.20:04
zbrif done, it would be unreliable20:04
zbri am not saying is not possible, but may require extra thinking, pros/cons20:05
zbri would write some python code that looks at touched files in last commit and decides which roles to run.20:06
zbrit could be a new feature for pytest-molecule20:06
zbrsome kind of "skip-if-not-touched"20:06
sshnaidm|afkzbr, well, it takes 25 min and it's too much for every patch, that doesn't touch this code at all20:11
zbrsshnaidm|afk: 🤔... but running 10-20x jobs of 1-3h each is not too much.20:12
sshnaidm|afkzbr, it's config repo, not product..20:13
zbrmake another job and start doing conditional sharding.20:14
zbrthe gate in tripleo-ci was broken because of these kind of optimizations, see https://review.opendev.org/#/c/704326/3/zuul.d/layout.yaml20:15
zbrthat was today, one optimization went wrong.20:15
zbrpep8 was removed in another change previous weeks.20:16
zbrtox accepts params, and you can tell it to run a single scenario, or to skip specific scenario, so sharding can be implementing 100% in zuul.20:18
*** rlandy is now known as rlandy|brb20:19
*** jbadiapa has quit IRC20:25
*** openstackstatus has joined #oooq20:26
*** ChanServ sets mode: +v openstackstatus20:26
*** rlandy|brb is now known as rlandy20:29
rlandyweshay|ruck: just to confirm ... ci.m1.large 4 8 GB 40 GB 40 GB 0 GB  ... that is a big enough flavor?20:39
rlandyor20:39
rlandym1.large 4 8 GB 60 GB 60 GB20:39
weshay|ruckrlandy, does the 40 + 40 end up in /20:39
weshay|ruckor / and /home20:39
rlandy VCPUS RAM Total Disk Root Disk Ephemeral Disk20:40
rlandylet me check my vm20:40
rlandydev/vda1        40G20:41
rlandydevtmpfs        3.8G     0  3.8G   0% /dev20:41
rlandytmpfs           3.9G  168K  3.9G   1% /dev/shm20:41
rlandytmpfs           3.9G   25M  3.8G   1% /run20:41
rlandytmpfs           3.9G     0  3.9G   0% /sys/fs/cgrou20:41
* rlandy tries it out20:42
rlandywe need another network20:52
rlandyweshay|ruck: ping re: instances20:54
weshay|ruckrlandy, aye20:54
rlandyweshay|ruck: you are not expecting the participants to create vms on this instance20:55
weshay|ruckrlandy, those will be standalone only20:55
rlandyweshay|ruck: ok - so in that case we need an instance with two nics20:55
rlandyright now, we get eth020:55
rlandywe need to get eth0 and eth120:56
rlandyI can add another network and attach and interface20:56
rlandyhoping this doesn't bring anything down20:56
rlandyputting link in pvt20:57
weshay|ruckrlandy, the second interface does not need an ip20:57
weshay|ruckrlandy, we may want to prep one box.. as we want it.. and save the volume20:58
weshay|ruckand boot the rest from the volume20:58
rlandyweshay|ruck: yeah - I am doing a manual test20:58
weshay|ruckk20:58
rlandyget the system righ20:58
rlandyso - in the past, when I wanted to add a second, interface, it told me the instance only had one nic20:59
rlandyunless I created it with a port on two networks20:59
rlandyweshay|ruck: can we gm for a few? just so I get the plan right21:00
weshay|ruckrlandy, sure21:00
weshay|ruckhttps://meet.google.com/cvm-usfd-cyv?authuser=121:00
zbrweshay|ruck: marting change is ok, https://review.opendev.org/#/c/682042/2221:03
zbryou can remove the -1, is better this way.21:04
weshay|ruckwrong review21:19
*** TrevorV has quit IRC21:36
*** Goneri has quit IRC21:39
*** jfrancoa has quit IRC21:58
arxcruzweshay|ruck: hey, sorry, i wasn't around, something i can help you ?23:08
rlandyweshay|ruck: how long should the deploy take?23:21
rlandysitting on :23:22
rlandyTASK [debuging something] **************************************************************************************************************************************************************************************************23:22
rlandyok: [undercloud] => {23:22
rlandy    "msg": "sudo /usr/bin/tripleo-container-image-prepare \\\n  --roles-file /tmp/ansible.6jj573hk-role-data \\\n  --environment-file /tmp/ansible.ce2362uc-prepare-param \\\n  --cleanup partial \\\n  --log-file /var/log/tripleo-container-image-prepare.log --debug\n"23:22
rlandy}23:22
rlandyMonday 27 January 2020  18:12:30 -0500 (0:00:00.051)       0:02:39.599 ********23:22
rlandyceph_namespace: 10.1.11.200:8787/ceph23:26
*** ysandeep has joined #oooq23:27
rlandylooking for someplace to pull ceph23:30
*** tosky has quit IRC23:45

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!