Tuesday, 2021-03-02

« Monday, 2021-03-01 Index Wednesday, 2021-03-03 »
*** jmasud has joined #oooq00:03
*** jmasud has quit IRC00:15
*** tosky has quit IRC00:17
*** jmasud has joined #oooq00:18
*** jmasud has quit IRC00:32
*** rlandy has quit IRC00:43
*** saneax has quit IRC01:16
*** jmasud has joined #oooq02:11
*** jmasud has quit IRC02:16
*** jmasud has joined #oooq02:37
*** jmasud has quit IRC02:54
*** jmasud has joined #oooq03:45
*** jmasud has quit IRC04:01
*** saneax has joined #oooq04:03
*** ykarel|away has joined #oooq04:09
*** ykarel|away is now known as ykarel04:14
*** udesale has joined #oooq05:08
*** marios has joined #oooq06:05
*** jpodivin has joined #oooq06:38
*** jpodivin has joined #oooq06:40
*** slaweq has joined #oooq07:08
*** amoralej|off is now known as amoralej07:26
*** ykarel_ has joined #oooq07:27
*** ykarel has quit IRC07:30
*** ykarel_ is now known as ykarel07:51
*** jbadiapa has joined #oooq08:35
*** tosky has joined #oooq08:35
arxcruzakahat: hey man08:54
arxcruzsup with https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/720705/08:54
arxcruzare you still working on that ?08:54
*** jpena|off is now known as jpena08:57
akahatarxcruz, hey o/09:05
akahatarxcruz, we need to redesign it.09:05
akahatarxcruz, i think we should discuss it in tomorrow's tempest meeting.09:05
akahatchandankumar, what you think? ^09:06
chandankumarakahat: let's discuss in tomorrow's meeting cc soniya29|ruck09:15
*** skramaja has joined #oooq09:28
soniya29|ruckchandankumar, ack09:55
*** derekh has joined #oooq10:05
*** udesale_ has joined #oooq10:15
*** udesale has quit IRC10:19
marioschandankumar: can you vote on that so we can merge please  ? https://review.opendev.org/c/openstack/tripleo-common/+/77757711:29
mariosor anyone else around ^^^11:29
mariosthank you11:29
mariosthanks chandankumar11:34
*** udesale_ has quit IRC11:40
*** udesale has joined #oooq11:43
*** saneax has quit IRC12:00
*** saneax has joined #oooq12:20
marioszbr: any idea bout that one https://zuul.opendev.org/t/openstack/build/ac8966befe634bd282c5dcad4ec53c2d TypeError: find_module() missing 1 required positional argument: 'path'12:26
marioszbr: it must be something with test-requirements/tox.ini and versions but wondering if it is a 'known one' or something you came across before/recently12:27
zbrbump linters, i no longer use hacking. likely a newer version of hacking deps broke it.12:28
zbri would also assure we do NOT use py35 !12:28
zbrbe sure we run on py36 or newer.12:28
zbrmost linters (and libs) dropped support for py3512:29
marioszbr: thanks noting/will check12:29
zbrmarios: easy fix would be to override nodeset used by that job, i think.12:30
*** jpena is now known as jpena|lunch12:31
*** rlandy has joined #oooq12:39
*** rlandy has quit IRC13:01
*** rlandy has joined #oooq13:02
pojadhavweshay|ruck, sshnaidm : please review https://code.engineering.redhat.com/gerrit/#/c/229257/13:07
weshay|ruckaye13:07
*** amoralej is now known as amoralej|lunch13:08
* pojadhav brb13:08
chandankumarweshay|ruck: rlandy we deployed downstream promoter but there is no hash to promoter13:11
rlandychandankumar: there is an open BZ issue13:12
*** rlandy is now known as rlandy|mtg13:12
chandankumarweshay|ruck: rlandy http://10.0.11.190/promoter_logs/13:12
chandankumarrlandy|mtg: weshay|ruck please pass the pub key, I will add to the box13:13
rlandy|mtgchandankumar: https://github.com/rlandy.keys13:14
mariossshnaidm: sanity check https://review.rdoproject.org/r/#/c/32144/4/zuul.d/tripleo.yaml13:15
sshnaidmmarios, it's both 8 and 7, 8 is below13:15
chandankumarrlandy|mtg: done13:15
mariossshnaidm: ack just saw it and replied thanks13:16
chandankumarneed to find a way to push dockerfiles with image push also13:16
bhagyashri|roverweshay|ruck, plz add in your review list https://review.opendev.org/c/openstack/tripleo-ansible/+/77814613:19
bhagyashri|roverarxcruz, zbr, sshnaidm, rlandy, marios, ysandeep, svyas, soniya29|ruck , pojadhav, akahat, weshay|ruck , chandankumar, frenzy_friday13:22
bhagyashri|roverTripleO CI Community Meeting in 8 mins13:22
bhagyashri|roverfeel free to add topics here https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#Agenda13:23
rlandy|mtgbhagyashri|rover:  in meeting - may be late for community call13:23
bhagyashri|roverrlandy|mtg, ack, np13:23
akahatzbr, o/13:27
akahatzbr, hey.. i'm stuck here: https://review.rdoproject.org/r/#/c/32007/9/ci-scripts/container-push/roles/containers-promote/defaults/main.yml@1213:27
zbrlooking13:27
zbrakahat: add {{-  and -}} (and for %) everywhere to avoid accidental newlines in final result.13:28
akahatzbr, okay.. trying now.13:29
zbryou should never create vars that contain newlines13:30
akahathow to avoid this?13:30
zbrusually using {{- {%- -- the minus avoids it. test with a simple playbook to get it right, and fake input13:32
zbris a very common mistake with jinja conditions.13:32
*** jpena|lunch is now known as jpena13:33
weshay|ruckchandankumar, https://github.com/weshayutin.keys13:39
chandankumarweshay|ruck: rlandy|mtg try ssh centos@10.0.11.19013:40
weshay|rucktmux a or "new"13:42
chandankumarweshay|ruck: then sudo su -, su - promoter, tmux a13:44
mariosarxcruz: https://etherpad.opendev.org/p/tripleo-meeting-items13:44
weshay|ruckaye.. there13:44
chandankumarweshay|ruck: enjoy :-)13:44
*** skramaja has quit IRC13:47
marioszbr: sshnaidm: wdyt... makes pep8 non voting for ocata os-apply-config 'cos its broken' https://review.opendev.org/c/openstack/os-apply-config/+/77753313:49
mariosweshay|ruck: rlandy|mtg: zbr: sshnaidm: chandankumar: please add to your reviews list when you next have time thank you ^^13:50
sshnaidmmarios, fine with me, maybe just to delete there all jobs, to leave check-requirements only13:50
sshnaidmidk how that is still alive13:51
mariossshnaidm: :D13:51
mariossshnaidm: thanks sure i'm up for any suggestions please comment if you feel strongly either way13:51
mariossshnaidm: remember main aim is to remove those 2 deprecated templates13:51
mariossshnaidm: we could even make the pep8 revoting again after we remove those, if someone objects ;)13:52
mariossshnaidm: last commit there was 201913:52
sshnaidmyou can make revoting, because they are broken :)13:52
sshnaidms/can/can't/13:52
mariossshnaidm: ah right ::D13:52
mariossshnaidm: " voting in check "13:52
mariossshnaidm: ;)13:52
marioswell not even that13:53
marios:/13:53
mariosk13:53
*** rlandy|mtg is now known as rlandy13:56
*** amoralej|lunch is now known as amoralej13:57
akahatzbr, hey.. your solution worked.. but problem still there... i'm not able to access /tmp dir.14:07
akahathttps://logserver.rdoproject.org/07/32007/10/check/tripleo-ci-promotion-staging-integration-pipeline-centos-8/d9a7d79/job-output.txt14:07
akahat"/tmp/generate_list.sh: line 31: target: Permission denied",14:07
zbrakahat: plain use of /tmp is not safe, neither portable. depending on where this runs you may want to use a more reliable method, like https://stackoverflow.com/questions/52014202/access-local-tmp-from-a-playbook14:13
zbrif multiple users (current user, or root) put stuff into /tmp, you will get permission errors.14:15
zbrnow i do not exactly know what is in that file, but you got the idea, likely trying to access a file owned by another user (root?)14:15
akahatzbr, this file is just small shell script... using for multiple purpose. I just want to use zuul log dir if it is running in CI else /tmp .14:17
akahati got what your point is14:17
zbri would personally add one or two tasks at the top of tasks/main.yml to deal with the case where zuul is not defined.14:22
zbrinstead of using jinja magic to try to come up with a fallback, we can do it using ansible, and the cost is close to zero as the task will run only when variable is not defined.14:23
akahatzbr, okay.. let me try this14:29
rlandyfrenzy_friday: want to meet to review patches for stream .. will bring in weshay|ruck when we have the list together15:10
* bhagyashri|rover dinner brb15:14
*** ykarel has quit IRC15:17
weshay|ruckwoot https://review.opendev.org/c/openstack/project-config/+/77799115:21
weshay|ruckmerged15:21
arxcruzweshay|ruck: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/778157 ;)15:28
frenzy_fridayrlandy, I added the change for integration line as well in the hackmd15:43
rlandyfrenzy_friday: ok- just gor off prod chain call - let's meet after the all hands15:49
frenzy_fridaythanks15:50
* bhagyashri|rover back15:59
*** amoralej is now known as amoralej|off16:05
*** chem has quit IRC16:05
*** chem has joined #oooq16:07
weshay|ruckarxcruz++16:33
weshay|ruckchandankumar, https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/77815716:33
*** jpodivin has quit IRC16:33
weshay|ruckarxcruz, would be neat to have a dnm patch ... that removes the cirros image from local and check if the fail over works :)16:34
weshay|ruckarxcruz, I wish I could tell you that all our images are built the same way.. but they are not :(16:34
weshay|ruckrlandy, ^16:34
weshay|ruckrlandy, if the failover does not work.. that could impact internal jobs arxcruz16:35
weshay|ruckbhagyashri|rover, akahat looks like stein promoted ya?16:37
weshay|ruck3 days ago.. acc8216:37
weshay|ruckwelll.. sorry.. a hash from three days ago16:38
weshay|ruckakahat, bhagyashri|rover testing w/ https://review.opendev.org/c/openstack/tripleo-heat-templates/+/656935/16:39
akahatweshay|ruck, acc82 is the new hash. promoted 3 days back.. yesterday when i was trying it was promoting same hash.16:41
weshay|ruckakahat, k.. looks like the overcloud images promoted current-tripleo/2021-02-27 17:35-16:42
weshay|ruck[DIR]acc82755f1e8df8bb8f0..>2021-02-27 17:35-16:42
weshay|ruckhttp://images.rdoproject.org/stein/rdo_trunk/?C=M;O=D16:42
weshay|ruckakahat, so.. we should be good to go if all the containers were pushed to quay :)16:43
weshay|ruckafaict16:43
weshay|ruckthank you16:43
weshay|ruckarxcruz, fyi .. stein issue resolved16:43
weshay|ruckakahat, so.. we'll leave that promoter running w/ stein16:43
akahatweshay|ruck, are we not running other branch promotions on that server?16:44
akahatonly stein?16:44
weshay|ruckakahat, let's look.. I only disabled promotions on the prod c7 prmotion server for stein16:45
weshay|ruckso.. yes.. I think so..16:45
bhagyashri|roverweshay|ruck, looking at the log16:45
bhagyashri|roverhttp://10.0.111.180/promoter_logs/container-push/20210302-025248.log16:45
bhagyashri|roversomething is failing16:45
weshay|ruck2021-03-02 04:00:26,344 p=2491677 u=promoter n=ansible | localhost                  : ok=5    changed=0    unreachable=0    failed=1    skipped=2    rescued=0    ignored=016:45
bhagyashri|roverhttp://10.0.111.180/promoter_logs/centos7_stein.log16:45
weshay|ruckbhagyashri|rover, akahat I did not see centos-binary-base on that acc8216:46
weshay|ruckin quay16:46
bhagyashri|roverakahat, ^16:46
weshay|ruckoh wait16:46
weshay|ruckit's there16:46
weshay|ruckhttps://quay.io/repository/tripleostein/centos-binary-base?tab=tags16:46
weshay|ruckbhagyashri|rover, akahat do you folks want to chat for a sec?16:46
akahatweshay|ruck, yeah.. sure.16:46
bhagyashri|roverweshay|ruck, yes16:47
weshay|ruckmeet.google.com/teo-qntk-xhi16:47
*** apetrich has quit IRC16:48
weshay|ruckakahat, bhagyashri|rover https://review.opendev.org/c/openstack/tripleo-heat-templates/+/656935/16:49
*** apetrich has joined #oooq16:55
*** sshnaidm is now known as sshnaidm|afk17:00
*** marios is now known as marios|out17:01
*** udesale has quit IRC17:04
*** marios|out has quit IRC17:06
akahatweshay|ruck, http://10.0.11.190/promoter_logs/17:09
weshay|ruckthanks17:09
*** jmasud has joined #oooq17:18
*** saneax has quit IRC17:18
rlandyweshay|ruck: have a few minutes to review the stream plan for tomorrow?17:19
rlandyotherwise we can meet later17:20
rlandywe dropped the call - pls ping when you have time17:22
*** rlandy is now known as rlandy|lunch17:23
arxcruzweshay|ruck: that won't be a problem, i'm working on the failsafe on tempestconf, so even if the image isn't in the image, it will try to download from a few sources17:53
weshay|ruckarxcruz, boom.. perfecto!17:53
weshay|ruckthanks17:53
*** rlandy|lunch is now known as rlandy|brb17:58
rlandy|brbdropping of package - brb17:58
*** derekh has quit IRC18:01
*** jpena is now known as jpena|off18:02
*** rlandy|brb is now known as rlandy18:37
*** jmasud has quit IRC18:45
*** jmasud has joined #oooq19:06
*** jmasud has quit IRC20:10
rlandyweshay|ruck: why do you think the overcloud deploy issue on baremetal is mistral?20:33
weshay|ruckrlandy, I worked through the trace, but it was last week..20:34
rlandyhttps://opendev.org/openstack/python-tripleoclient/src/branch/stable/train/tripleoclient/workflows/deployment.py#L40020:34
rlandy assigned it to mistral - will see20:35
weshay|ruckI think that message comes from mistral20:35
weshay|ruckpayload['message']20:35
*** jmasud has joined #oooq20:42
rlandyweshay|ruck: https://bugzilla.redhat.com/show_bug.cgi?id=191041620:43
openstackbugzilla.redhat.com bug 1910416 in python-tripleoclient "Stack update reports failure even though everything is o.k. due to encoding issue - UnicodeEncodeError: 'latin-1' codec can't encode character '\u2192' in position 1702: ordinal not in range(256)" [High,Closed: notabug] - Assigned to rhos-maint20:43
rlandyweshay|ruck: according to that - not a bug20:44
weshay|rucknot a bug?20:45
rlandyLC_CTYPE20:46
rlandyThis issue stems from the use of LC_CTYPE=latin-1 instead of a utf8 lang time (e.g. en_US.UTF-8).20:46
rlandyweshay|ruck: maybe we define that before we deploy?20:47
rlandyLC_CTYPE=en_US.UTF-8 python320:47
weshay|ruckbetter to see what it's set to before changing anything20:49
weshay|ruckwhere's our log?20:50
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/83a4e36/logs/undercloud/home/zuul/overcloud_deploy.log20:51
rlandynot there20:51
weshay|ruckrlandy, perhaps we testproject w/ something to print locale and log the .bash files20:52
weshay|ruckor20:53
weshay|ruckadd something to collect logs to print out the set command20:53
rlandysource /home/zuul/stackrc20:54
rlandy^6 add to ther20:54
rlandythere20:54
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/83a4e36/logs/undercloud/home/zuul/stackrc20:55
rlandyweshay|ruck: tack on a setting20:55
weshay|ruckno no20:55
weshay|ruckfirst we should find out what the shell lang is set to20:55
rlandyit's bm - let me see if I can still access it20:58
weshay|ruckrlandy, https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/77830621:00
weshay|ruckrlandy, oh ya.. love that bm21:00
rlandyhold on - may not need it21:01
rlandy[zuul@undercloud ~]$ locale21:01
rlandyLANG=en_US.UTF-821:01
rlandyLC_CTYPE="en_US.UTF-8"21:01
rlandythat's right on the undercloud21:02
weshay|ruckrlandy, k.. do we have access to the overcloud? or did it not come up?21:02
rlandychecking if I can ssh there21:02
rlandyyep on overcloud-controller-121:03
rlandy[heat-admin@overcloud-controller-1 ~]$ locale21:03
rlandyLANG=en_US.UTF-821:03
rlandyLC_CTYPE="en_US.UTF-8"21:03
rlandysame21:03
rlandyidk21:03
rlandycontainers>21:04
*** bhagyashri|rover has quit IRC21:06
rlandyhttps://opendev.org/openstack/mistral/src/branch/master/mistral/service/coordination.py#L4121:06
rlandyhttps://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L668821:07
rlandyall over21:07
rlandyhttps://codesearch.opendev.org/?q=latin-1&i=nope&files=&excludeFiles=&repos=21:07
*** bhagyashris has joined #oooq21:07
weshay|ruckrlandy, osp-17 issue https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/233c3f0/logs/container-builds/448542c9-6ac8-498b-bc39-9d58eb8ceeb1/base/os/os-build.log21:08
weshay|ruckError: Unable to find a match: python3-pymemcache21:08
weshay|ruckwriting it up21:08
rlandyok - cool21:09
rlandyweshay|ruck: latin-1 is defined all over21:09
weshay|ruckwhere?21:09
rlandybut interestingly enough not train21:09
rlandyhttps://codesearch.opendev.org/?q=latin-1&i=nope&files=&excludeFiles=&repos=21:09
rlandyself.to_json(result).encode('latin-1')21:10
rlandyisk - all correctly set21:15
*** jbadiapa has quit IRC21:35
slaweqhi oooq experts21:48
slaweqI need some help with tripleo-standalone21:49
slaweqI'm trying to deploy it on centos vm, with script like http://paste.openstack.org/show/803165/21:49
slaweqall works fine until the moment when it fails ;)21:49
slaweqand it fails every time on:21:50
slaweqFATAL | Check containers status | tripleo-centos | error={"changed": false, "msg21:50
slaweq": "Failed container(s): ['mysql_wait_bundle'], check logs in /var/log/containers/stdouts/"}21:50
slaweqis it some known issue maybe? or maybe I have some obvious problem in my script?21:50
slaweqI would be grateful if someone could take a look :)21:50
slaweqthx in advance21:50
weshay|ruckslaweq, try this one https://hackmd.io/WlFvuBB7TReVVyMuULvcLg21:51
weshay|ruckwe just re-went through the instructions.. we need to post these some where perm21:51
weshay|ruckslaweq, but those have been recently verified21:51
weshay|ruckslaweq, which branch?21:52
slaweqweshay|ruck++ thx, I will try21:52
slaweqI'm trying to deploy master21:52
weshay|ruckslaweq, you may also want to pull the containers first.. or use quay.. beware of ratelimit21:53
weshay|ruckslaweq, you also have containers here https://registry.rdoproject.org:8443/oapi/v1/namespaces/tripleomaster/imagestreamtags/21:54
slaweqweshay|ruck: thx, I will try21:54
rlandyweshay|ruck: fs001 did ok - except for expected tempest fails .. https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/b85a5d3ae4ea4094baad141c738cc302?logfile=console.log21:54
rlandyweshay|ruck: ok - image builds should be fixed now in component line22:02
rlandydib_dnf_module_streams: "virt:av container-tools:2.0"22:02
rlandy^^ updated22:02
weshay|ruckcool22:02
rlandytesting baremetal line22:08
* rlandy check dependency line now22:08
rlandyweshay|ruck: so - attempted the w/o of putting the export LANG in stackrc  - just to check - got past step 122:20
slaweqweshay|ruck: it seems that for some reason my mysql server won't start :/22:37
slaweqI tried with Your guide and it won't work for me too22:37
slaweqmaybe it's something on my vm, idk22:37
weshay|ruckslaweq, hrm.. maybe it's an issue in master22:37
slaweqI'm done for today, I will try tomorrow morning :)22:38
slaweqthx for help weshay|ruck22:38
weshay|ruckslaweq, 0/  I'll try as well22:38
slaweqgood night22:38
slaweqo/22:38
weshay|ruckI'm out22:43
*** jmasud has quit IRC23:32
*** jmasud has joined #oooq23:45
« Monday, 2021-03-01 Index Wednesday, 2021-03-03 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!