Wednesday, 2020-05-06

*** waleedm has quit IRC00:02
*** jmasud has joined #tripleo00:07
*** rlandy has quit IRC00:20
*** rfolco has quit IRC00:35
openstackgerritMerged openstack/tripleo-common master: buildah: exclude build and push, not only build  https://review.opendev.org/72561701:08
openstackgerritMerged openstack/puppet-tripleo master: Rremove manifests for ceilometer-expirer  https://review.opendev.org/72402001:08
openstackgerritMerged openstack/tripleo-heat-templates master: Deprecating Old Dell EMC Xtremio Iscsi Volume Config  https://review.opendev.org/72304001:08
openstackgerritMerged openstack/tripleo-quickstart-extras master: Consume oooci-build-images role in build-image  https://review.opendev.org/71945301:08
*** jmasud has quit IRC01:17
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/train: buildah: exclude build and push, not only build  https://review.opendev.org/72574801:32
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: buildah: allow images without type  https://review.opendev.org/72572101:33
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Add a 600s timeout when creating enable-ssh-admin workflow  https://review.opendev.org/72515201:33
*** jmasud has joined #tripleo01:39
*** matbu has quit IRC01:50
openstackgerritMerged openstack/tripleo-ci master: ensure we can override the registry namespace  https://review.opendev.org/71843302:01
*** jmasud has quit IRC02:11
*** jmasud has joined #tripleo02:49
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [wip] focal ubuntu-minimal testing  https://review.opendev.org/72575202:49
*** nethans has quit IRC03:16
*** jmasud has quit IRC03:16
openstackgerritMerged openstack/tripleo-heat-templates master: Use python3 in tools scripts  https://review.opendev.org/72408303:37
*** ramishra has joined #tripleo03:37
*** psachin has joined #tripleo03:37
openstackgerritRabi Mishra proposed openstack/tripleo-common stable/queens: Match hostname when doing node delete  https://review.opendev.org/72562003:52
*** ykarel|away is now known as ykarel03:54
*** rh-jelabarre has quit IRC04:02
*** openstack has joined #tripleo04:21
*** ChanServ sets mode: +o openstack04:21
*** matbu has quit IRC04:29
*** evrardjp has quit IRC04:35
*** evrardjp has joined #tripleo04:36
*** ratailor has joined #tripleo04:37
*** udesale has joined #tripleo04:40
openstackgerritRabi Mishra proposed openstack/tripleo-docs master: Cleanup contributor guide for new developers  https://review.opendev.org/72545004:47
openstackgerritRabi Mishra proposed openstack/tripleo-docs master: Update doc for manual overcloudrc file generation  https://review.opendev.org/72575504:48
*** jdandrea has quit IRC04:48
*** bandini has joined #tripleo04:59
*** jmasud has joined #tripleo05:00
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Explicitly set port numbers used in swift storage  https://review.opendev.org/72575705:12
*** sanjayu__ has joined #tripleo05:14
*** sanjayu_ has quit IRC05:17
*** cgoncalves has joined #tripleo05:38
*** skramaja has joined #tripleo05:44
*** sanjayu__ has quit IRC05:48
*** ysandeep|away is now known as ysandeep05:51
*** saneax has joined #tripleo05:54
*** marios has joined #tripleo05:58
*** dpawlik has joined #tripleo06:03
*** abdysn has joined #tripleo06:10
*** ysandeep is now known as ysandeep|away06:12
akahat|ruckramishra, hello o/06:14
akahat|ruckramishra, are you working on this issue: https://bugs.launchpad.net/tripleo/+bug/1875352 ??06:15
openstackLaunchpad bug 1875352 in tripleo "keystone container failed to start in scenario000" [High,Triaged] - Assigned to amolkahat (amolkahat)06:15
*** jmasud has quit IRC06:17
*** waleedm has joined #tripleo06:19
*** jmasud has joined #tripleo06:19
*** lmiccini has joined #tripleo06:21
ramishraakahat|ruck: nope, I just had a look at the logs earlier. It's probably some job configuration issue...06:22
ramishraIf there is no green run of the job ever06:22
akahat|ruckramishra, okay.. can you tell me who i need to bother for this issue... because it is not in good state since long time... need to fix it ASAP.06:24
Tengukeystone is for security (xek), mysql pidone ( lmiccini or bandini I think - at least they'll know who to point for that service)06:26
ramishraakahat|ruck: May be someone who configured the job.. else as Tengu mentioned06:27
akahat|ruckTengu, ramishra thank you..06:29
ramishraakahat|ruck: btw, queens tripleo-ci-centos-7-undercloud-upgrades job seems broken too06:29
ramishrahttps://c2a55200c3e5568ee82f-eb83075bad77f107ed9b57803fd20c1f.ssl.cf5.rackcdn.com/725620/2/check/tripleo-ci-centos-7-undercloud-upgrades/3dd932f/logs/undercloud/home/zuul/install_packages.sh.log06:29
akahat|rucklmiccini, bandini hello o/06:29
lmicciniakahat|ruck: \o06:29
*** ysandeep|away has quit IRC06:29
lmicciniakahat|ruck: which job should we be looking at?06:29
akahat|rucklmiccini, https://bugs.launchpad.net/tripleo/+bug/18753506:30
openstackLaunchpad bug 187535 in gnome-control-center (Ubuntu) "gnome-appearance-properties crashed with signal 5 in g_array_free()" [Low,Invalid] - Assigned to Mirco Müller (macslow)06:30
akahat|rucklmiccini, sorry.. wrong link06:30
lmiccinisorry I use sway can't help with gnome :P06:30
akahat|rucklmiccini, : https://bugs.launchpad.net/tripleo/+bug/187535206:30
openstackLaunchpad bug 1875352 in tripleo "keystone container failed to start in scenario000" [High,Triaged] - Assigned to amolkahat (amolkahat)06:30
Tenguramishra: ah... that's probably for gfidente|afk seeing the ceph-ansible thing at the end06:30
Tenguor rdo in order to provide the right ansible version...06:31
akahat|ruckramishra, yes. it's work in progress.. y'day they in tripleo meeting they propose to block podman updates. it will fix the issue.06:31
Tenguakahat|ruck: that's another one :]06:31
ramishraakahat|ruck: podman? I guess it's ansible version required by ceph-ansible-3.0.31-1.el7.noarch06:32
lmicciniakahat|ruck: do you have the link to the logs of the last failure?06:33
akahat|ruckramishra, sorry.. i got confused, this is another.. i was talking about this. https://review.opendev.org/#/c/725614/06:33
*** jfrancoa has joined #tripleo06:34
*** matbu has joined #tripleo06:35
ramishraakahat|ruck: np:) raised a bug https://bugs.launchpad.net/tripleo/+bug/187703106:35
openstackLaunchpad bug 1877031 in tripleo "queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version" [High,Triaged]06:35
*** ysandeep has joined #tripleo06:35
akahat|rucklmiccini, let me check.06:35
akahat|ruckramishra, thank you :)06:36
akahat|rucklmiccini, looks like all the job logs are gone. But i have one form check pipeline: https://logserver.rdoproject.org/19/26919/6/check/tripleo-ceph-integration-centos-7-scenario001-standalone-train/f326712/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz06:36
akahat|ruckit's the same issue.06:37
*** ysandeep is now known as ysandeep|brb06:38
*** ysandeep|brb is now known as ysandeep06:39
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-ansible stable/train: DNM - test on train related to a current LP  https://review.opendev.org/72554406:40
lmicciniakahat|ruck: I think there is something different between containers and host, so mysql cannot start because it cannot talk to pacemaker06:44
lmiccinihttps://logserver.rdoproject.org/19/26919/6/check/tripleo-ceph-integration-centos-7-scenario001-standalone-train/f326712/logs/undercloud/var/log/containers/stdouts/mysql_init_bundle.log.txt.gz06:44
lmicciniakahat|ruck: I think bandini is almost done with a patchset to move these actions out of the container to avoid such situations06:45
*** jmasud has quit IRC06:51
akahat|rucklmiccini, that's great.. but job will run or not that is another question which depends on bandini's path.06:51
*** maciejjozefczyk has joined #tripleo06:52
*** pojadhav has quit IRC06:53
abdysnHi all, so i was trying to deploy an overcloud with ovs-dpdk, but it is failing on step3, waiting for the containers to be up, debuged a bit, and it seams that there is a problem with haproxy bind point being unreachable, any idea anyone?06:53
lmicciniakahat|ruck: if that's the real issue it will be red until the containers are rebuilt with the same rpms as the base image (or the image updated)06:53
abdysnskramaja, -^^, have any idea?06:54
*** jmasud has joined #tripleo06:55
Tenguabdysn: I've seen jobs where haproxy wasn't listening on the expected IP06:55
*** slaweq has joined #tripleo06:56
akahat|rucklmiccini, okay. got it.06:57
akahat|rucklmiccini, thank you :)06:57
lmicciniakahat|ruck: np. I'll see if we can confirm my suspicion06:58
akahat|rucklmiccini, yes. please..06:59
*** jbadiapa has joined #tripleo06:59
*** ccamacho has joined #tripleo07:04
abdysnTengu, oh, i checked the netstat command, and it is as expected see http://paste.openstack.org/show/793166/, and they are as the configuration pointed, but no connection to mysql, even thought the mysql is running07:06
*** saneax has quit IRC07:06
*** waleedm_ has joined #tripleo07:06
Tenguabdysn: so in the case I've seen, haproxy was configure to listen on IP_A, while the /etc/hosts was stating that the host is on IP_B - thus the issue. in your case, there isn't enough info to debug causes so far.07:07
abdysnTengu, let me check the etc hosts07:08
abdysnTengu, the /etc/hosts have this: 172.16.2.11  overcloud.internalapi.localdomain, and the error in paunch is http://paste.openstack.org/show/793169/07:10
*** pojadhav has joined #tripleo07:10
*** waleedm_ has quit IRC07:11
*** saneax has joined #tripleo07:11
lmicciniakahat|ruck: https://bugs.launchpad.net/tripleo/+bug/1875352/comments/4 confirmed07:11
openstackLaunchpad bug 1875352 in tripleo "keystone container failed to start in scenario000" [High,Triaged] - Assigned to amolkahat (amolkahat)07:11
abdysnTengu, have any ideas? thanks07:12
Tenguabdysn: you should be able to find some interesting logs in /var/log/containers/stdouts/ in regards to mysql state.07:12
Tenguabdysn: maybe it's related to what lmiccini was discussing like 2 minutes ago.07:13
abdysnTengu, oh will check07:13
Tenguthough haproxy being listening, it should report something else than EHOSTUNREACH07:14
*** yolanda has joined #tripleo07:14
Tenguabdysn: would be interesting to run some curl and check the firewall as well...07:14
*** jmasud has quit IRC07:14
abdysnTengu, yea, and the weird thing is there is no logs at all in the haproxy07:14
Tenguyeah well, for the haproxy log........ it's something a bit annoying.07:15
Tenguthat nasty boy isn't able to write anything, we have to play with rsyslog07:15
abdysnTengu, will check the firewall and see07:15
*** flfuchs has joined #tripleo07:15
*** sanjayu_ has joined #tripleo07:17
*** jcapitao has joined #tripleo07:18
*** jmasud has joined #tripleo07:19
*** saneax has quit IRC07:19
*** bhagyashris has quit IRC07:23
*** amoralej|off is now known as amoralej07:23
openstackgerritDaniel Bengtsson proposed openstack/tripleo-validations master: Remove the post-introspection group.  https://review.opendev.org/72460307:25
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: WIP DNR test standalone with separate vips  https://review.opendev.org/72577107:29
*** tosky has joined #tripleo07:30
*** rpittau|afk is now known as rpittau07:31
*** bhagyashris has joined #tripleo07:31
*** cgoncalves has quit IRC07:34
*** cgoncalves has joined #tripleo07:40
*** jmasud has quit IRC07:40
*** dtantsur|afk is now known as dtantsur07:40
*** bogdando has joined #tripleo07:43
*** openstackstatus has quit IRC07:46
*** openstack has joined #tripleo07:48
*** ChanServ sets mode: +o openstack07:48
*** suuuper has joined #tripleo07:50
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade master: Support new Rhel Enforcement in CI.  https://review.opendev.org/72478107:52
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/train: Support new Rhel Enforcement in CI.  https://review.opendev.org/72478207:53
*** bogdando_ has joined #tripleo07:56
*** bogdando has quit IRC07:57
*** bogdando_ is now known as bogdando07:57
*** jpena|off is now known as jpena07:58
*** dmellado has quit IRC07:58
*** jpich has joined #tripleo07:58
akahat|rucklmiccini, ack :)07:59
*** ysandeep is now known as ysandeep|lunch08:03
*** dmellado has joined #tripleo08:04
*** dpawlik has quit IRC08:09
*** dpawlik has joined #tripleo08:09
*** jpich has quit IRC08:10
*** jpich has joined #tripleo08:11
*** lucasagomes has joined #tripleo08:12
*** alexmcleod has joined #tripleo08:19
*** pkopec has joined #tripleo08:20
abdysnTengu, sorry for being late, so tried to stop the firewall and do a curl, ended with no route to host, checked the mysql.log from /var/log/containers/stdouts/mysql.log and i think it is good, not sure though, see http://paste.openstack.org/show/793174/08:25
openstackgerritMichele Baldessari proposed openstack/tripleo-quickstart-extras master: wip test standalone with separate vipfor ha  https://review.opendev.org/72578208:25
abdysnTengu, it is like the haproxy is not running, so i wanted to check the ovs-dpdk configuration, maybe it is causing problems.08:27
abdysnskramaja, if you may have a look at our configuration of the ovs-dpdk, would appreciate it08:39
*** xek has joined #tripleo08:39
skramajaabdysn: sorry.. missed your earlier ping.. going through them now..08:40
abdysndeploy command: paste.openstack.org/show/793159/, ovs-hw-offload: http://paste.openstack.org/show/793160/, compute-dpdk: http://paste.openstack.org/show/793161/, controller: http://paste.openstack.org/show/793162/, network:http://paste.openstack.org/show/793164/08:41
openstackgerritLucas Alvares Gomes proposed openstack/tripleo-heat-templates stable/train: Changes in env files due to new SRIOV roles  https://review.opendev.org/72579308:41
abdysnskramaja, np, did not send them actually, here they are -^^, thanks in advance.08:42
*** ykarel is now known as ykarel|lunch08:45
skramajaabdysn: on the nic configs for dpdk, i have never been sucessful in creating a vlan interface on top of the ovs_user_bridge. so single-nic templates cannot be used..08:45
skramajaabdysn: it is possible to have tenant network on ovs_user_bridge, but insted of vlan interface, vlan will be added to the port using ovs config.. let me get a sample..08:46
abdysnwaleedm, -^^08:47
skramajaabdysn: waleedm - here is an example of how to apply it for tenatnt network with vlan - http://paste.openstack.org/show/793177/08:50
abdysnskramaja, thanks alot for the info08:51
skramajasure.08:51
*** paramite has joined #tripleo08:52
abdysnskramaja, will try to fix that and see what will happen and update you.08:53
skramajasure. let me know.08:53
waleedmskramaja: Hi Sarvanna, so we are just change the bridge to be ovs_bridge, should it be ok08:55
skramajayes waleedm, that will work. its just the limitaiton with ovs_user_bridge08:55
waleedmskramaja, but I see a limitaion that all the bridges should be ovs_user_bridge in compute nodes08:56
waleedmskramaja, or thi's only a performance limitaion08:57
skramajawaleedm: yes, thats the reason, why we put the others on the linux_bond08:57
waleedmskramaja, ok so firtly we will check ovs_bridge beside ovs_user_bridge, as we don't need to test performance there08:59
skramajawaleedm: do you need dpdk deployment or offload deployment or both?08:59
waleedmskramaja, it's for virtio-forwarder, so the same dpdk interface will also have switchdev vfs09:00
skramajawaleedm: it is like using the vfs for both openstack services and vms?09:02
*** jmasud has quit IRC09:02
waleedmskramaja, no no all the vfs for vms09:02
waleedmskramaja, btw the failure in controller node not compute node, and in controller we don't have ovs_user_bridge !!09:03
skramajawaleedm: ahh.. that should be something else then..09:04
skramajacontroller does not have any dpdk functionality..09:04
waleedmskramaja, yes no need to for that09:07
*** gfidente|afk is now known as gfidente09:13
Tenguodyssey4me: heya! guess I'll just drop the patch about container removal. the podman version currently shipped is just too faulty - it's the one dropping the container reference from the DB even if the container removal fails..... "yay". and weshay|ruck patch excluding podman from centos repo is in gate right now.09:16
odyssey4meTengu: yeah, makes sense - working around a bad component is not a great way to solve this issue09:19
Tenguodyssey4me: indeed. so I'll abandon the change. hopefully the situation will be better. there are also discussions to lock the podman version in the dependencies so that it solves the issue outside of CI as well.09:19
*** ysandeep|lunch is now known as ysandeep09:20
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-operator-ansible master: New operator: validator_run  https://review.opendev.org/72522509:20
Tenguodyssey4me: guess we can abandon https://review.opendev.org/725545 as well09:22
odyssey4meTengu: heh, I see your comment on https://review.opendev.org/#/c/724802/4 - but perhaps you missed that it was rebased on top of https://review.opendev.org/#/c/725545/ so it should have picked up the depend-on09:26
*** ykarel|lunch is now known as ykarel09:27
Tenguodyssey4me: oh. I didn't see the rebase. meh.09:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/tripleo-upgrade stable/train: Simplify ffwd2 CIP stein parameters.  https://review.opendev.org/72480209:29
Tenguwondering if the "failed_when" thingy wasn't preventing the retries in fact.09:30
Tenguanyway. patch is dropped and burried for the better hopefully.09:30
odyssey4meTengu++09:30
Tenguso let's wait for https://review.opendev.org/#/c/725636/ to be integrated.09:31
Tenguhopefully it will solve the podman issue in train.09:31
Tengunext step after gate is back online is to make some better dependencies locking I guess.09:31
*** hkominos has joined #tripleo09:37
openstackgerritSlawek Kaplonski proposed openstack/tripleo-heat-templates master: Neutron ML2/OVS: add support to enable of IGMP snooping  https://review.opendev.org/72538609:37
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-operator-ansible master: New operator: validator_run  https://review.opendev.org/72522509:38
*** rpittau is now known as rpittau|bbl09:44
*** lbragstad has quit IRC09:45
*** owalsh has quit IRC09:46
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Wait for compute services before sanity check.  https://review.opendev.org/72016909:50
openstackgerritMerged openstack/puppet-tripleo master: Do not set cache parameter for openidc  https://review.opendev.org/72487210:08
openstackgerritMerged openstack/puppet-tripleo master: Accept missing memcached_node_ips  https://review.opendev.org/72502010:08
*** owalsh has joined #tripleo10:09
*** pcaruana has quit IRC10:31
*** hkominos has quit IRC10:33
*** hkominos has joined #tripleo10:34
*** jcapitao is now known as jcapitao_lunch10:37
openstackgerritMerged openstack/tripleo-quickstart master: exclude podman from centos-extras  https://review.opendev.org/72563610:38
openstackgerritMerged openstack/tripleo-ansible master: Use shared volume for side-car wrapper locks  https://review.opendev.org/72425910:41
openstackgerritMerged openstack/tripleo-heat-templates master: Change Collectd ports type to numbers.  https://review.opendev.org/71820010:43
*** maciejjozefczyk_ has joined #tripleo10:43
*** maciejjozefczyk has quit IRC10:44
*** maciejjozefczyk has joined #tripleo10:44
*** derekh has joined #tripleo10:46
*** hrw has joined #tripleo10:48
hrwmorning10:48
*** maciejjozefczyk_ has quit IRC10:48
hrwI looked at rabbimq failure in one of kolla jobs: https://logserver.rdoproject.org/32/725032/4/openstack-check/tripleo-build-containers-centos-8-master/53de6d3/logs/buildah-builds/kolla-builds/c3d37f08-9902-4107-ae20-42a7887d473b/docker/rabbitmq/rabbitmq-build.log10:48
*** maciejjozefczyk_ has joined #tripleo10:49
hrwdo you use some repo for rmq?10:50
*** pcaruana has joined #tripleo10:50
*** maciejjozefczyk has quit IRC10:50
*** maciejjozefczyk has joined #tripleo10:50
*** maciejjozefczyk_ has quit IRC10:53
ykarelmarios, chandankumar can u review https://review.opendev.org/#/c/724075/10:54
*** maciejjozefczyk_ has joined #tripleo10:55
*** maciejjozefczyk has quit IRC10:55
*** pcaruana has quit IRC10:57
openstackgerritTakashi Kajinami proposed openstack/puppet-tripleo master: Add monitor interval for ovn dbs  https://review.opendev.org/71040710:57
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Deprecate service ipv6 params  https://review.opendev.org/72389810:57
mariosack ykarel11:00
*** pcaruana has joined #tripleo11:02
Tenguhrw: what rabbitmq failure?11:02
*** sanjayu__ has joined #tripleo11:03
Tenguakahat|ruck: did you see this issue already? stderr: 'Error: error getting image "docker.io/tripleomaster/centos-binary-rabbitmq:pcmklatest": unable to find ''docker.io/tripleomaster/centos-binary-rabbitmq:pcmklatest'' in local storage: no such image'    happening on master, for instance11:03
Tenguhttps://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_d8f/724603/4/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/d8f0017/logs/undercloud/home/zuul/overcloud_update_run_Controller.log11:03
hrwTengu: https://review.opendev.org/#/c/725032/ patch - tripleo-build-containers-centos-8-master job11:03
Tenguhrw: ok - so not related to the thing I just hit in the CI then11:04
hrw:)11:04
*** maciejjozefczyk_ has quit IRC11:04
*** maciejjozefczyk_ has joined #tripleo11:05
*** sanjayu_ has quit IRC11:05
Tengubandini or lmiccini care to check with hrw about rabbitmq container building and repositories?11:06
*** ysandeep is now known as ysandeep|brb11:16
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Explicitly set port numbers used in swift storage  https://review.opendev.org/72575711:18
lmicciniTengu: hrw  I see a rabbitmq-server package available for example here http://centos.mirror.garr.it/centos/8/messaging/x86_64/rabbitmq-38/Packages/r/ but  I am not sure it is the place where we should pull it from11:18
hrwlmiccini: centos-release-rabbitmq-3811:18
hrwlmiccini: it adds messaging/rabbitmq-38 repo11:19
hjensasAny Manila expert that can take a look at https://review.opendev.org/#/c/723898/5/environments/network-isolation-v6.j2.yaml@59 ?11:22
lmiccinihrw: I found thins https://review.opendev.org/#/c/721800/1 but I probably miss 99% of the context11:24
Tengulmiccini: since you're here - placeholder LP for a possible CI issue on master that might be related to rabbitmq not being able to start during an update job: https://bugs.launchpad.net/tripleo/+bug/1877094 (recheck running right now, maybe it's nothing, but...)11:26
openstackLaunchpad bug 1877094 in tripleo "rabbitmq_init_bundle failed to start" [Medium,Triaged]11:26
hrwlmiccini: kolla/c8 uses external rabbitmq/erlang - not ones from c811:27
hrwlmiccini: this patch waits for centos 8.2 so I can enable it for aarch6411:28
*** rfolco has joined #tripleo11:30
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-ci master: Use registry_namespace for pulling out successful containers  https://review.opendev.org/72583111:32
*** raildo has joined #tripleo11:37
*** jcapitao_lunch is now known as jcapitao11:39
*** jpena is now known as jpena|lunch11:39
cloudnullmornings11:49
hkominosmorning!11:50
akahat|ruckTengu, hey... I never seen this issue.11:52
*** leanderthal has joined #tripleo11:52
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Ensure the ansible.cfg file presists  https://review.opendev.org/72560211:53
*** ysandeep|brb is now known as ysandeep11:54
*** waleedm has quit IRC11:55
openstackgerritTakashi Kajinami proposed openstack/tripleo-heat-templates master: Remove support for Neutron FUJITSU pluging  https://review.opendev.org/72583511:57
*** ekultails has joined #tripleo11:57
*** rlandy has joined #tripleo11:58
Tenguakahat|ruck: ok... I've created a "placeholder" LP while re-checking the patch. I hope being able to move it to "invalid" :)12:00
Tenguakahat|ruck: https://bugs.launchpad.net/tripleo/+bug/1877094 fyi12:00
openstackLaunchpad bug 1877094 in tripleo "rabbitmq_init_bundle failed to start" [Medium,Triaged]12:00
*** rh-jelabarre has joined #tripleo12:02
openstackgerritTakashi Kajinami proposed openstack/puppet-tripleo master: Remove support for Neutron FUJITSU pluging  https://review.opendev.org/72583712:02
*** rh-jelabarre has quit IRC12:02
*** rh-jelabarre has joined #tripleo12:03
*** amoralej is now known as amoralej|lunch12:03
*** holser_ has joined #tripleo12:04
*** holser_ has quit IRC12:09
*** holser_ has joined #tripleo12:10
jfrancoao/ EmilienM: mwhahaha: we would like to have your blessing on https://review.opendev.org/#/c/717320/ , please have a look when you have some time12:12
EmilienMjfrancoa: ok12:13
jfrancoamerci12:14
*** rpittau|bbl is now known as rpittau12:18
*** abdysn has quit IRC12:21
*** udesale_ has joined #tripleo12:23
*** udesale has quit IRC12:26
EmilienMjfrancoa: trading a review, thanks: https://review.opendev.org/#/c/725152/12:30
*** tkajinam has joined #tripleo12:34
*** ykarel is now known as ykarel|afk12:38
*** Goneri has joined #tripleo12:38
jfrancoaEmilienM: traded!12:41
EmilienMjfrancoa: yeah i failed to updated all the commit mesage. To be fair it's in the title12:42
jfrancoaEmilienM: yes, it's just that it is a little confusing. But I won't be blocking the patch for such a thing, the rest looks good and solid12:43
jfrancoaEmilienM: ok, I arrived late. weshay|ruck fired first :-)12:44
openstackgerritTakashi Kajinami proposed openstack/puppet-tripleo master: Add monitor interval for ovn dbs  https://review.opendev.org/71040712:44
*** jpena|lunch is now known as jpena12:45
*** sanjayu_ has joined #tripleo12:47
EmilienMjfrancoa, weshay|ruck : thx12:47
*** sanjayu__ has quit IRC12:50
*** fultonj has joined #tripleo12:51
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Add openstack tripleo container image build  https://review.opendev.org/72414712:52
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add role for Container file(s) generation  https://review.opendev.org/72255712:56
*** chem_ is now known as chem13:01
*** ratailor has quit IRC13:02
hkominosjistr . Quick ping regarding https://github.com/openstack/tripleo-heat-templates/commit/ea24fc57d025d6ae89db30cbd699e1ea20b1461e ?13:06
jistrhkominos: hi, please go ahead13:08
jistri mean feel free to ask... not sure if "please go ahead" is the best English here :)13:08
*** lbragstad has joined #tripleo13:10
openstackgerritTakashi Kajinami proposed openstack/puppet-tripleo master: Remove EC2 API  https://review.opendev.org/72585113:11
openstackgerritChandan Kumar (raukadah) proposed openstack/puppet-tripleo stable/train: Remove duplicate entry of collectd-python package  https://review.opendev.org/72585213:11
*** cdearborn has joined #tripleo13:11
hkominosjistr hi! I wanted to ask about the upgrades. Do you by any chance recall the pacemaker version on which the patch was tried upon ? For example on https://github.com/openstack/tripleo-heat-templates/blob/stable/stein/deployment/cinder/cinder-volume-pacemaker-puppet.yaml#L274 This CLI command does not work on the pacemaker available on (updates)13:12
*** lbragstad has quit IRC13:13
hkominospcs resource show maybe ??? ( which does work)13:13
*** rfolco is now known as rfolco|rover13:13
*** akahat|ruck is now known as akahat13:13
hkominosSo I wanted to know which pcm was used. was it Centos 8 ? etc..13:13
*** pojadhav is now known as pojadhav|afk13:16
bhagyashrisoye mala pn nav change karav lagel ka aata13:16
ysandeepbhagyashris, ?? :D13:17
bhagyashrisohh sorry13:17
bhagyashrismistakenly send this msg13:17
*** skramaja has quit IRC13:18
*** bhagyashris is now known as bhagyashris|ruck13:18
jistrhkominos: i'm not sure about the pcs version, we tested predominantly with RHEL 8 (CentOS 8 wasn't released yet IIRC)13:19
*** cgoncalves has quit IRC13:19
hkominosjistr: OK. this is very helpful . I can only assume that Centos 8 repos had a newer version of pcs where that command actually works.13:20
hkominosIs stein even supported on Centos7 ?13:20
*** lbragstad has joined #tripleo13:21
jistrhkominos: that task also has "failed_when: false" meaning any errors should be treated as non-fatal. I vaguely recall that the task is probably not meant to kick in when cinder-volume resource is not yet defined in pacemaker. So the cinder_volume_image_current fact would then be '' and later there are tasks conditioned with when: cinder_volume_image_current != ''13:21
jistrhkominos: re stein supported on CentOS 7: i'm a bit disconnected from TripleO proper right now, but i think it is, at least we CI it that way i think13:23
jistrhkominos: but we actually don't CI upgrade this way, generally it's quite difficult to do any meaningful CI of upgrades within the upstream time frame of ~3 hours13:24
*** cgoncalves has joined #tripleo13:25
hkominosjistr it is difficult anyway. Lots of changes. Regarding what you mentioned above. failed_when: false I think might be misleading. Because that CLI command does not exist basically " pcs resource config ..." just outputs the --help of pacemaker. Then you grep it, and end up with an empty cinder_volume_image_current_res13:26
hkominosAt least on my pacemaker. And while trying upgrading I am ending up wiht a few empty variables (like https://github.com/openstack/tripleo-heat-templates/blob/stable/stein/deployment/cinder/cinder-volume-pacemaker-puppet.yaml#L350) where is this being populated i have no idea13:28
jistrhkominos: ah indeed, probably a difference between pcs 0.9 and 0.10: https://github.com/ClusterLabs/pacemaker/blob/master/doc/pcs-crmsh-quick-ref.md#display-a-resource13:28
hkominos!!!13:29
openstackhkominos: Error: "!!" is not a valid command.13:29
hkominosaha!13:29
hkominosfinally some progress!13:29
*** TrevorV has joined #tripleo13:31
hkominosjistr possibly pcs 0.10 will require python 3++ as well ?13:32
*** jistr is now known as jistr|mtg13:32
hkominosSo it might be that, that commit might just not work on Stein, despite that it is backported13:33
*** dmacpher has joined #tripleo13:33
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Enable collectd-ceph plugin only where needed  https://review.opendev.org/72585413:37
*** hrw has left #tripleo13:37
*** maciejjozefczyk_ is now known as maciejjozefczyk13:37
*** dmacpher_ has joined #tripleo13:39
*** dmacpher has quit IRC13:41
*** amoralej|lunch is now known as amoralej13:44
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Disable checking of image url for master  https://review.opendev.org/72585813:50
ade_lee__mwhahaha, hjensas , EmilienM https://review.opendev.org/#/c/724282/ please13:53
*** jistr|mtg is now known as jistr13:53
*** ykarel|afk is now known as ykarel13:54
jistrhkominos: maybe... maybe it could be possible substitute (sed) for the `pcs show --full` variant13:54
ade_lee__mwhahaha, hjensas , EmilienM cloudnull also -- https://review.opendev.org/#/c/725262/ please13:54
ade_lee__cloudnull, mwhahaha thanks13:58
hkominosjistr. It does (at least temporarily but I think it is that whole playbook that is problematic. As I said for example. cinder_volume_node_names --> seems to always be empty but I have no idea why. Do you know what that variable represents ?13:58
ade_lee__cloudnull, if you can check out https://review.opendev.org/#/c/724282/ as well please13:58
cloudnulldone13:58
ade_lee__cloudnull, thanks!13:59
*** jdandrea has joined #tripleo13:59
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart master: Disable checking of image url for master  https://review.opendev.org/72585813:59
*** dmacpher__ has joined #tripleo14:00
*** ysandeep is now known as ysandeep|away14:02
*** dmacpher_ has quit IRC14:03
*** bogdando_ has joined #tripleo14:04
*** bogdando has quit IRC14:06
*** bogdando_ is now known as bogdando14:06
openstackgerritMerged openstack/tripleo-upgrade master: Support new Rhel Enforcement in CI.  https://review.opendev.org/72478114:26
*** hkominos has quit IRC14:27
*** hkominos has joined #tripleo14:28
*** sanjayu__ has joined #tripleo14:30
openstackgerritMerged openstack/tripleo-ansible stable/train: tripleo-redhat-enforce new role to enforce RHOSP constraints.  https://review.opendev.org/72169214:31
*** sanjayu_ has quit IRC14:33
*** flfuchs has quit IRC14:35
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Update roles path tripleo-ansible and tripleo-ipa repos  https://review.opendev.org/70741914:36
*** udesale_ has quit IRC14:38
*** ccamacho has quit IRC14:39
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates master: Fix typo in setting octavia wsgi server name  https://review.opendev.org/72587714:40
openstackgerritCédric Jeanneret (Tengu) proposed openstack/tripleo-quickstart master: Correct typo introduced in I1074e0204910726d82950496ac4813a06e315252  https://review.opendev.org/72153814:43
*** dmacpher__ has quit IRC14:45
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add role for standalone on multinode with IPA  https://review.opendev.org/72075714:46
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add role for standalone on multinode with IPA  https://review.opendev.org/72075714:46
*** dmacpher has joined #tripleo14:49
openstackgerritJavier Peña proposed openstack/tripleo-ci master: [TEST DNM] Testing molecule-rhel8 nodeset change  https://review.opendev.org/72588114:52
openstackgerritMerged openstack/tripleo-heat-templates master: Make /var/lib/mistral traversable by all users  https://review.opendev.org/71732014:57
odyssey4meweshay|ruck: with https://review.opendev.org/725636 merging 4 hours back, what's the time-frame for us to get good job results after it? I've tried a few after the merge and they're still failing14:59
*** jcapitao is now known as jcapitao|afk15:06
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/train: Make /var/lib/mistral traversable by all users  https://review.opendev.org/72588415:10
*** leanderthal has quit IRC15:11
*** ccamacho has joined #tripleo15:17
*** ykarel is now known as ykarel|away15:23
*** alexmcleod has quit IRC15:24
weshay|ruckodyssey4me, can you pass me the reviews that are failing?15:29
weshay|ruckodyssey4me, what you want to check is that podman 1.6 is NOT installed and that you have 1.515:33
weshay|ruckI see in other jobs 1.5 did not resolve TASK [tripleo-container-rm : Stop and remove container] ************************15:34
weshay|ruckin the upgrade15:34
odyssey4meweshay|ruck: yeah, that same task is still bugging out15:35
weshay|ruckso folks were thinking it was podman 1.6 but perhaps there is more to that bug15:36
weshay|ruckodyssey4me, worked here https://review.opendev.org/#/c/722284/15:38
weshay|ruckfailed here https://review.opendev.org/#/c/724295/15:38
weshay|ruckand has a failure trend still https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containerized-undercloud-upgrades15:39
*** jcapitao|afk is now known as jcapitao15:39
*** jmasud has joined #tripleo15:41
odyssey4meweshay|ruck: it's these that I'm looking at https://bit.ly/2zWNKfh15:45
odyssey4meweshay|ruck: sorry, it took a while to craft the search15:45
*** dmacpher has quit IRC15:49
weshay|ruckodyssey4me, two days :( http://dashboard-ci.tripleo.org/d/3sUHfh9Wz/jobs-exploration?orgId=1&var-influxdb_filter=job_name%7C%3D%7Ctripleo-ci-centos-7-containerized-undercloud-upgrades&var-influxdb_filter=branch%7C%3D%7Cstable%2Ftrain15:50
*** cdearborn has quit IRC15:51
*** ccamacho has quit IRC15:51
*** dmacpher has joined #tripleo15:51
odyssey4meweshay|ruck: yep, well aware of that so it seems that https://bugs.launchpad.net/tripleo/+bug/1876893 is not fixed, and we need to continue triage now that one issue is out of the way15:52
openstackLaunchpad bug 1876893 in tripleo "Error: error removing container - device or resource busy" [Critical,Fix released] - Assigned to wes hayutin (weshayutin)15:52
weshay|ruckaye15:53
*** jmasud has quit IRC15:55
tbarronhjensas: added myself and gouthamr to https://review.opendev.org/#/c/723898/15:58
*** maciejjozefczyk has quit IRC16:01
*** panda|pto has quit IRC16:03
*** jmasud has joined #tripleo16:04
*** maciejjozefczyk has joined #tripleo16:06
*** panda has joined #tripleo16:06
*** maciejjozefczyk_ has joined #tripleo16:07
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Run all noop opeartions in molecule testing.  https://review.opendev.org/69852616:09
*** dtantsur is now known as dtantsur|afk16:11
*** maciejjozefczyk has quit IRC16:11
*** jmasud has quit IRC16:21
*** jmasud has joined #tripleo16:23
*** jmasud has quit IRC16:23
*** rpittau is now known as rpittau|afk16:24
*** bandini has quit IRC16:27
*** marios is now known as marios|out16:27
*** jpich has quit IRC16:28
*** alexmcleod has joined #tripleo16:33
*** evrardjp has quit IRC16:36
*** evrardjp has joined #tripleo16:36
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Add openstack tripleo container image build  https://review.opendev.org/72414716:36
openstackgerritMichele Baldessari proposed openstack/tripleo-quickstart-extras master: wip test standalone  https://review.opendev.org/72578216:36
*** sanjayu__ has quit IRC16:38
weshay|ruckcloudnull, is https://opendev.org/openstack/tripleo-ansible/blame/branch/master/tripleo_ansible/roles/tripleo_container_rm/tasks/tripleo_podman_container_rm.yml#L87-L92 used in stein->train upgrades?16:38
weshay|ruckseems so https://zuul.opendev.org/t/openstack/build/26de0d3e01544542b26caaeda8459277/log/logs/undercloud/home/zuul/undercloud_upgrade.log#4640-464216:39
weshay|ruckbut wanted to confirm16:39
weshay|ruckchem, ^ ?16:39
openstackgerritMerged openstack/openstack-virtual-baremetal master: ipv6-radvd - resolve mtu from provisioning network  https://review.opendev.org/72565516:40
cloudnullweshay|ruck - https://opendev.org/openstack/tripleo-ansible/src/branch/stable/train/tripleo_ansible/roles/tripleo_container_rm/tasks/tripleo_podman_container_rm.yml it doesn't seem like that task file exists in train ?16:40
cloudnullmaybe its been moved around?16:40
cloudnullbut i dont remember specifcally what / where that's used16:40
weshay|ruckya.. so /me just confused a little because it's an undercloud upgrade from stein to train but using a master tripleo-ansible role.. which fine.. just wanted to be sure16:41
*** psachin has quit IRC16:41
weshay|ruckor maybe that task exists some where else16:41
*** bogdando has quit IRC16:43
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Add openstack tripleo container image build  https://review.opendev.org/72414716:49
*** alexmcleod has quit IRC16:54
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Add openstack tripleo container image build  https://review.opendev.org/72414716:55
*** lucasagomes has quit IRC16:58
*** marios|out has quit IRC17:01
*** lmiccini has quit IRC17:01
*** jcapitao has quit IRC17:04
*** jdandrea has quit IRC17:04
*** jdandrea has joined #tripleo17:06
*** jpena is now known as jpena|off17:08
openstackgerritLance Bragstad proposed openstack/tripleo-docs master: Add a new introduction for TLS documentation  https://review.opendev.org/69238017:09
*** holser_ has quit IRC17:09
*** holser has joined #tripleo17:09
*** holser_ has joined #tripleo17:13
*** holser has quit IRC17:14
*** pojadhav|afk is now known as pojadhav17:15
*** amoralej is now known as amoralej|off17:15
*** maciejjozefczyk has joined #tripleo17:18
cloudnullweshay|ruck hum. that does seem wrong, train upgrade using master tripleo-ansible17:19
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible stable/train: Stop and remove container with module  https://review.opendev.org/72593917:19
sshnaidmweshay|ruck, cloudnull kind of ^17:19
cloudnull-cc odyssey4me17:19
*** maciejjozefczyk_ has quit IRC17:20
*** irclogbot_1 has quit IRC17:20
*** maciejjozefczyk has quit IRC17:21
cloudnullsshnaidm that's fine , but does that need to be a train only review ? should it hit master first?17:21
*** maciejjozefczyk has joined #tripleo17:21
weshay|ruckanother coming up17:21
openstackgerritMerged openstack/tripleo-common master: buildah: allow images without type  https://review.opendev.org/72572117:22
*** maciejjozefczyk_ has joined #tripleo17:22
sshnaidmcloudnull, I think can be in both17:22
openstackgerritwes hayutin proposed openstack/tripleo-ansible stable/train: add retry/delay in podman rm container  https://review.opendev.org/72594017:23
weshay|rucksshnaidm, ^17:23
*** maciejjozefczyk_ has quit IRC17:23
openstackgerritwes hayutin proposed openstack/tripleo-ansible stable/train: add retry/delay in podman rm container  https://review.opendev.org/72594017:24
sshnaidmweshay|ruck, ack17:24
weshay|rucksshnaidm, Partial-Bug: #187689317:24
openstackbug 1876893 in tripleo "Error: error removing container - device or resource busy" [Critical,Triaged] https://launchpad.net/bugs/1876893 - Assigned to wes hayutin (weshayutin)17:24
*** irclogbot_3 has joined #tripleo17:24
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Stop and remove container with module  https://review.opendev.org/72594217:25
sshnaidmcloudnull, for master ^17:25
*** derekh has quit IRC17:26
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible stable/train: Stop and remove container with module  https://review.opendev.org/72593917:26
*** maciejjozefczyk has quit IRC17:26
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Stop and remove container with module  https://review.opendev.org/72594217:26
sshnaidmbbl17:26
*** sshnaidm is now known as sshnaidm|afk17:26
weshay|ruckodyssey4me, fyi ^17:27
weshay|ruckcloudnull, adding undercloud-upgrades to tripleo-ansible checks17:29
odyssey4meweshay|ruck: while I think it's fair to say there should be an upgrade test there, it doesn't need to be a full tripleo upgrade test - perhaps something smaller and quicker would work just as well to validate function17:30
weshay|ruckodyssey4me, right.. so pick ur poison ... standalone-upgrade, undercloud-upgrade multinode-oooq-container-updates17:31
weshay|ruckundercloud-upgrade should ALWAYS work.. so leaning towards that17:32
odyssey4meyeah, fair enough17:32
odyssey4mecloudnull weshay|ruck by now I hope you realise that https://opendev.org/openstack/tripleo-ansible/src/branch/stable/train/tripleo_ansible/roles/tripleo_container_rm/tasks/tripleo_podman_container_rm.yml doesn't work because of the role renames from - to _17:34
odyssey4mecloudnull: weshay|ruck https://opendev.org/openstack/tripleo-ansible/src/branch/stable/train/tripleo_ansible/roles/tripleo-container-rm/tasks/tripleo_podman_container_rm.yml works just fine17:35
weshay|ruckya.. that tripped me up when I flipped branches17:36
weshay|ruckphun17:36
openstackgerritwes hayutin proposed openstack/tripleo-ansible stable/train: add undercloud-upgrades job to check/gate  https://review.opendev.org/72594417:36
weshay|ruckodyssey4me, question now is why isn't this triggering https://opendev.org/openstack/tripleo-ansible/src/branch/stable/train/zuul.d/layout.yaml#L45-L5117:39
odyssey4meweshay|ruck: that is a good question... the path is right, and the exclusion seems ok17:42
weshay|ruck725944 is running in zuul. .w/ undercloud-upgrades17:43
weshay|ruckworks in master but not train https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-standalone-upgrade&project=openstack%2Ftripleo-ansible&branch=stable%2Ftrain17:44
openstackgerritLance Bragstad proposed openstack/tripleo-heat-templates master: trivial: fix a minor grammatical nit in ipaservices  https://review.opendev.org/72594717:47
odyssey4meweshay|ruck: this would be why - https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L7717:47
odyssey4meweshay|ruck: stable/train should have this job - https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L10617:47
weshay|ruckdoh17:47
openstackgerritPooja Jadhav proposed openstack/tripleo-heat-templates master: Trigger a job to run a standalone deploy with IPA on multinode  https://review.opendev.org/72525317:48
odyssey4meweshay|ruck: or, perhaps stable/train should have the upgrade to train and the upgrade from train... but I doubt it's been implemented that way17:48
*** gfidente is now known as gfidente|afk17:48
weshay|ruckodyssey4me, https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L94-L11417:49
weshay|ruckwill update the tripleo-ansible layout17:49
weshay|ruckodyssey4me, the master one is right17:50
odyssey4meja, that makes sense - the train one just never got adjusted17:50
openstackgerritwes hayutin proposed openstack/tripleo-ansible stable/train: add undercloud-upgrades job to check/gate  https://review.opendev.org/72594417:51
weshay|ruckodyssey4me, ok.. 725944 has it now17:51
weshay|ruckupgrade jobs are harder17:52
*** hamzy_ has quit IRC17:52
weshay|ruckI think that's right now .. lolz17:52
*** nethans has joined #tripleo17:53
*** hamzy has joined #tripleo17:54
*** aedc_ has joined #tripleo17:55
*** aedc has quit IRC17:58
*** waleedm has joined #tripleo18:00
*** aedc_ has quit IRC18:01
EmilienMweshay|ruck: did I miss something or I can't get any logs from rdo jobs?18:02
EmilienMhttps://review.rdoproject.org/zuul/build/c2198981c3ae412b86bede734dbe9a7818:02
EmilienMah zuul is down18:03
EmilienM#rdo18:03
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add role for standalone on multinode with IPA  https://review.opendev.org/72075718:07
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add role for standalone on multinode with IPA  https://review.opendev.org/72075718:07
weshay|ruckEmilienM, [11:42:22] <tristanC> ok so after running `zuul enqueue-ref --tenant local --pipeline release --project python-dciclient --ref refs/tags/1.0.1` the scheduler got stuck18:08
weshay|ruckEmilienM, <tristanC> and github rate limit is preventing the scheduler for doing the reconfiguration, it's going to be stuck for 20 more minutes18:08
EmilienMk18:09
*** waleedm has quit IRC18:17
openstackgerritMichele Baldessari proposed openstack/tripleo-quickstart-extras master: wip test standalone  https://review.opendev.org/72578218:20
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: WIP - new container images layout  https://review.opendev.org/72248618:20
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: WIP - new container images layout  https://review.opendev.org/72248618:21
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Add openstack tripleo container image build  https://review.opendev.org/72414718:22
*** pkopec has quit IRC18:22
*** hkominos has quit IRC18:24
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/train: buildah: allow images without type  https://review.opendev.org/72596318:31
*** hamzy_ has joined #tripleo18:39
*** hamzy has quit IRC18:42
openstackgerritJohn Fulton proposed openstack/tripleo-docs master: Replace accidental references to ScaleUp with ScaleOut  https://review.opendev.org/72596418:46
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Add openstack tripleo container image build  https://review.opendev.org/72414718:55
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: WIP - new container images layout  https://review.opendev.org/72248618:56
hjensasreviews please, https://review.opendev.org/#/q/topic:port_physnet_hook+(status:open+OR+status:merged)19:00
*** suuuper has quit IRC19:01
*** jmasud has joined #tripleo19:04
openstackgerritLance Bragstad proposed openstack/tripleo-docs master: Introduce skeleton TLS-e guide  https://review.opendev.org/72560719:06
*** nethans has quit IRC19:18
*** jmasud has quit IRC19:18
weshay|ruckcloudnull, https://review.opendev.org/#/c/725944/19:25
cloudnulldone19:26
EmilienMweshay|ruck: I'm not sure adding the tripleo-ci-centos-7-containerized-undercloud-upgrades job in tripleo-ansible is efficient19:28
EmilienMe.g. if I change a file in molecule, it'll run the tripleo-ci-centos-7-containerized-undercloud-upgrades job19:28
EmilienMwhich we don't care about in that case19:28
EmilienMoh I see it inherits from standalone_upgrade layout19:28
EmilienMmhh ok19:28
weshay|ruckfiles:19:28
weshay|ruck          - ^tripleo_ansible/roles/tripleo-hieradata/.*19:28
weshay|ruck          - ^tripleo_ansible/roles/tripleo-upgrade-hiera/.*19:28
weshay|ruck          - ^tripleo_ansible/roles/tripleo-container-manage/.*19:28
weshay|ruck          - ^tripleo_ansible/roles/tripleo-container-rm/.*19:28
weshay|ruck          irrelevant-files: *irrelevant_scenario_files19:28
weshay|ruck:)19:28
EmilienMthen I'm curious why this job is needed19:29
EmilienMif we have tripleo-ci-centos-7-standalone-upgrade-train19:29
EmilienMI see it's related to https://bugs.launchpad.net/tripleo/+bug/187689319:29
openstackLaunchpad bug 1876893 in tripleo "Error: error removing container - device or resource busy" [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)19:29
weshay|ruckEmilienM, I'm ok w/ one or the either19:29
weshay|ruckneither were triggering previously19:29
EmilienMbut imo this error could have been hit in standalone upgrade too19:29
weshay|ruckya.. it wasn't running19:29
weshay|ruckI can update it to be just the standalone-upgrade..19:30
weshay|rucklet me do that AFTER we get this bug fixed19:30
weshay|rucksince it's blocking19:30
EmilienMwe're just wasting CI resources19:30
EmilienMthe job is doing the same thing wrt containers19:30
EmilienMthe bug in launchpad could have hit the standalone-upgrade IMO19:30
openstackgerritMerged openstack/tripleo-heat-templates master: Make sure IdMServer is optional  https://review.opendev.org/72526219:32
openstackgerritMerged openstack/tripleo-heat-templates master: Add an option to disable the DNS record modification in FreeIPA  https://review.opendev.org/72428219:33
*** yolanda has quit IRC19:33
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Run all noop opeartions in molecule testing.  https://review.opendev.org/69852619:34
openstackgerritwes hayutin proposed openstack/tripleo-docs master: update list of TripleO core teams  https://review.opendev.org/72597119:48
weshay|ruckEmilienM, ^19:48
*** jamesdenton has quit IRC19:48
EmilienMweshay|ruck: +219:49
*** ayoung has joined #tripleo19:54
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP: Modify  get-hash when running with downstream components  https://review.opendev.org/72597419:55
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Run all noop opeartions in molecule testing.  https://review.opendev.org/69852619:59
*** jfrancoa has quit IRC19:59
openstackgerritMerged openstack/tripleo-heat-templates master: Fix missing OctaviaClientCert* parameters  https://review.opendev.org/72268320:01
weshay|ruckcloudnull,  not sure what's happening w/20:03
weshay|ruck                * Error initializing source docker://podman-legacy-container2:latest: Error reading manifest latest in docker.io/library/podman-legacy-container2: errors:20:03
weshay|ruckhttps://opendev.org/openstack/tripleo-ansible/src/branch/stable/train/tripleo_ansible/roles/tripleo-container-rm/molecule/podman/prepare.yml#L3720:04
weshay|ruckhttps://560bd206b1c54122f604-8bc1c0202523f17b73621207314548bd.ssl.cf1.rackcdn.com/725939/2/check/tripleo-ansible-centos-7-molecule-tripleo-container-rm/52288cb/reports.html20:04
weshay|ruckhiccup? or real20:04
*** jamesdenton has joined #tripleo20:04
weshay|ruckhttps://review.opendev.org/#/c/72593920:04
weshay|ruckEmilienM, FYi.. undercloud-upgrade is failing w/ standalone upgrade is working against head20:06
weshay|ruckso .. maybe both for a bit :)20:06
EmilienMif it's a transient issue it means nothing20:06
EmilienMbut ok20:06
EmilienMhaving more jobs isn't going to help20:06
weshay|ruckit's not transient20:06
EmilienMyes it'll make it more visible20:06
openstackgerritMerged openstack/tripleo-validations stable/train: Remove fwaas_driver.ini from neutron sanity checks  https://review.opendev.org/72154120:07
EmilienMi still want to know why it fails in one and not the other20:07
EmilienMshow me logs20:07
weshay|ruckaye20:07
weshay|ruckthey will post here shortly https://review.opendev.org/#/c/725944/20:07
weshay|ruckhttps://zuul.opendev.org/t/openstack/stream/8f8ea1375fb64d828cc14338043203ef?logfile=console.log https://zuul.opendev.org/t/openstack/stream/b341a15dc65f4ba995c24b036baa54cd?logfile=console.log20:07
*** jamesden_ has joined #tripleo20:08
* weshay|ruck is not sure how to fix tripleo-ansible-centos-7-molecule-tripleo-container-rmFAILURE in 10m 10s20:08
weshay|ruckhttps://review.opendev.org/#/c/72593920:08
EmilienMlooking20:09
weshay|ruckdon't see that container in docker.io20:09
EmilienMdocker://podman-legacy-container1:latest ?20:10
EmilienMyes of course it's not in docker.io20:10
EmilienMit's a local one used by molecule20:10
EmilienMlet me look too20:10
*** jamesdenton has quit IRC20:11
EmilienMit tries to pull podman-legacy-container1 while we want fedora20:11
weshay|ruckhttps://opendev.org/openstack/tripleo-ansible/src/branch/stable/train/tripleo_ansible/roles/tripleo-container-rm/molecule/podman/prepare.yml#L34-L4120:12
EmilienMoh I know20:12
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_*/molecule: pull fedora instead of centos8  https://review.opendev.org/72597820:17
EmilienMweshay|ruck: ^ should help20:17
EmilienMnow looking at https://review.opendev.org/#/c/725944/20:18
weshay|ruckthanks!20:19
weshay|ruckI did not know that20:19
EmilienMweshay|ruck: -1 https://review.opendev.org/#/c/725944/20:19
EmilienMhonestly you should give up until you know the exact root cause20:20
EmilienMthe problem is for some reason the memcached container couldn't be stopped20:20
EmilienMbut if we deploy memcached in standalone we could catch it too20:20
EmilienMadding more job isn't a good idea20:20
EmilienMwe have seen where it lead us in the past20:20
weshay|ruckEmilienM, I agree in theory.. for sure20:21
weshay|ruckhttps://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-containerized-undercloud-upgrades20:21
weshay|ruckbut that proves otherwise possibly20:21
EmilienMwhat I'm saying is that you don't even have an RCA20:21
EmilienMand you already adding more jobs20:21
*** jamesdenton has joined #tripleo20:21
EmilienM... that's my point20:21
EmilienMyou don't know what happen but you see green in zuul so add more jobs20:21
EmilienMit's not the way it should work IMO20:21
weshay|ruckEmilienM, I do think there should be a retry on the remove though20:21
EmilienMwe have to find what's the bug20:21
EmilienMfind why we don't hit it in standalone-upgrade20:22
weshay|ruckwe pinned the podman package.. because that was the first theory20:22
*** jamesden_ has quit IRC20:22
weshay|ruckworks fine here https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-standalone-upgrade&branch=stable%2Ftrain20:22
weshay|ruckEmilienM, while we explore a root cause.. can I move it to non-voting?20:23
weshay|ruckhttps://review.opendev.org/#/c/725614/20:23
weshay|ruckwe're blocking a lot of stuff atm20:24
EmilienMnon voting yes20:24
EmilienMadding more jobs no20:24
weshay|ruckhrm.. k.. I need a new patch20:24
weshay|ruckk20:24
EmilienMnow let me RCA this thing20:24
EmilienMweshay|ruck: hard to debug when you don't have journal logs :/20:24
EmilienMhttps://a8a564ccb03789f73602-795792c1f104e79962e44448ab55e3f1.ssl.cf5.rackcdn.com/725944/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/b341a15/logs/undercloud/var/log/20:24
weshay|ruckEmilienM, we do20:25
EmilienMhttps://a8a564ccb03789f73602-795792c1f104e79962e44448ab55e3f1.ssl.cf5.rackcdn.com/725944/2/check/tripleo-ci-centos-7-containerized-undercloud-upgrades/b341a15/logs/undercloud/var/log/extra/journal.txt20:25
EmilienMok it moved20:25
weshay|ruckright20:25
EmilienMyeah I know what's wrong (I think)20:25
EmilienMit's a race condition in the ansible role20:25
weshay|ruckso retries would help in that case .. no?20:26
EmilienMyes20:27
EmilienMI'm going to patch20:27
EmilienMlet me a fes20:27
EmilienMfew*20:27
EmilienMdo you have launchpad?20:27
weshay|ruckya20:27
*** xek has quit IRC20:27
EmilienM187689320:27
EmilienMk20:28
weshay|ruckhttps://bugs.launchpad.net/tripleo/+bug/187689320:28
openstackLaunchpad bug 1876893 in tripleo "Error: error removing container - device or resource busy" [Critical,In progress] - Assigned to Sagi (Sergey) Shnaidman (sshnaidm)20:28
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: New command "overcloud node extract provisioned"  https://review.opendev.org/67414120:29
openstackgerritwes hayutin proposed openstack/tripleo-ansible stable/train: standalone-upgrade was not triggering  https://review.opendev.org/72594420:32
weshay|ruckupdated https://review.opendev.org/#/c/725944/20:32
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: WIP - new container images layout  https://review.opendev.org/72248620:36
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: WIP - new container images layout  https://review.opendev.org/72248620:38
EmilienMweshay|ruck: it's a bit more complex than I thought but i'm on it20:47
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: WIP - new container images layout  https://review.opendev.org/72248620:48
cloudnullweshay|ruck I've never encountered that error20:48
openstackgerritwes hayutin proposed openstack/tripleo-ansible stable/train: tripleo_container_*/molecule: pull fedora instead of centos8  https://review.opendev.org/72598320:48
weshay|ruckrock on king kong20:50
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible stable/train: Stop and remove container with module  https://review.opendev.org/72593920:50
*** ayoung has quit IRC20:51
cloudnullweshay|ruck odyssey4me ^ updated to change the change-id so they match20:52
weshay|ruckyou are an officer and a gentleman20:53
cloudnulllets not resort to calling names20:54
cloudnull:P20:54
*** ayoung has joined #tripleo20:55
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: WIP - new container images layout  https://review.opendev.org/72248621:01
*** Goneri has quit IRC21:05
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_rm: more robust tasks to remove a container  https://review.opendev.org/72599121:11
EmilienMweshay|ruck: ^ WIP I need to look a bit further something else21:11
openstackgerritJames Slagle proposed openstack/python-tripleoclient master: Initially set rc to 0, not 1  https://review.opendev.org/72599221:12
openstackgerritJames Slagle proposed openstack/python-tripleoclient master: Deprecate --standalone argument  https://review.opendev.org/72599321:12
openstackgerritJames Slagle proposed openstack/python-tripleoclient master: Add --transport argument to tripleo deploy  https://review.opendev.org/72599421:12
openstackgerritJames Slagle proposed openstack/python-tripleoclient master: Skip hostname check when running in a container  https://review.opendev.org/72599521:12
openstackgerritJames Slagle proposed openstack/python-tripleoclient master: Use os.path.expanduser to get home dir  https://review.opendev.org/72599621:12
weshay|ruckEmilienM, https://review.opendev.org/#/c/725940/2/tripleo_ansible/roles/tripleo-container-rm/tasks/tripleo_podman_container_rm.yml21:12
weshay|ruckEmilienM, you should use my until:21:13
EmilienM?21:14
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_rm: more robust tasks to remove a container  https://review.opendev.org/72599121:16
EmilienMweshay|ruck: ^ should be more reliable21:17
weshay|ruckEmilienM, is it not possible w/ retries.. it gets removed and is gone.. w/o a 0 return21:17
weshay|ruckand just is missing21:17
EmilienMwhat?21:18
EmilienMthere is until: podman_rm_cmd.rc == 021:18
EmilienMI don't understand what you mean, sorry21:18
weshay|ruckuntil: ( podman_rm_cmd.rc == 0 ) or ("'Failed to find container' in removed_container.stderr")21:18
weshay|ruckuntil: ( podman_rm_cmd.rc == 0 ) or ("'Failed to find container' in podman_rm_cmd.stderr")21:19
weshay|ruckthat is a safer operation right?21:19
weshay|ruckbecause if it does delete.. and ansible doesn't catch it.. it will not return 021:19
EmilienMoh i see21:20
EmilienMyes indeed21:20
* weshay|ruck thinks EmilienM makes me earn my keep21:20
openstackgerritmathieu bultel proposed openstack/python-tripleoclient stable/train: Add missing download_ansible_playbooks function  https://review.opendev.org/70906621:20
EmilienM"no such container" is the correct stderr I think21:20
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: tripleo_container_rm: more robust tasks to remove a container  https://review.opendev.org/72599121:21
EmilienMweshay|ruck: ^ better?21:21
EmilienMwe'll see if it passes molecule testing21:22
EmilienMI need to run, bbl21:22
weshay|ruckEmilienM++21:22
EmilienMmwhahaha: https://review.opendev.org/#/c/725978/ plz21:23
EmilienMmwhahaha: oh wait21:24
EmilienMweshay|ruck: https://review.opendev.org/#/c/725983/21:24
EmilienMit's failing, dang21:24
weshay|rucklolz21:24
weshay|ruckwhy not use the module man?21:24
EmilienMdon't know if we can21:24
weshay|ruckEmilienM, go away.. I'll mark it non-voting21:24
EmilienMwhere is sshnaidm|afk21:24
weshay|ruckhe's hopefully sleeping21:24
EmilienMa novel concept21:25
weshay|ruckindeed21:25
* sshnaidm|afk is sleeping, yeah21:25
weshay|rucksshnaidm|afk, Evilien is being a pita :)21:25
* weshay|ruck runs21:25
EmilienMweshay|ruck: I know21:25
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible stable/train: tripleo_container_*/molecule: pull fedora instead of centos8  https://review.opendev.org/72598321:25
EmilienMthe backport was missing that ^21:25
EmilienMhttps://review.opendev.org/#/c/725983/1..221:26
EmilienMsshnaidm|afk: lol HI21:26
EmilienMmwhahaha: you can go ahead plz21:26
weshay|ruckEmilienM thanks21:26
weshay|ruckGO DUDE21:26
weshay|ruckwe gotcha :)21:26
EmilienMok I have to go21:26
*** slaweq has quit IRC21:26
sshnaidm|afkwhat's with module though?21:27
openstackgerritAlex Schultz proposed openstack/tripleo-ansible master: Add tripleo dense formatting  https://review.opendev.org/72600121:28
openstackgerritJames Slagle proposed openstack/tripleo-ansible master: Allow unsafe_writes for /etc/hosts  https://review.opendev.org/72600221:28
*** jmasud has joined #tripleo21:28
openstackgerritwes hayutin proposed openstack/tripleo-heat-templates stable/train: tripleo-ci-centos-7-containerized-undercloud-upgrades -> NV  https://review.opendev.org/72600321:30
openstackgerritwes hayutin proposed openstack/python-tripleoclient stable/train: tripleo-ci-centos-7-containerized-undercloud-upgrades -> NV  https://review.opendev.org/72600421:34
openstackgerritwes hayutin proposed openstack/tripleo-common stable/train: tripleo-ci-centos-7-containerized-undercloud-upgrades -> NV  https://review.opendev.org/72600521:35
openstackgerritMerged openstack/tripleo-ansible master: tripleo_container_*/molecule: pull fedora instead of centos8  https://review.opendev.org/72597821:41
openstackgerritMerged openstack/tripleo-docs master: Replace accidental references to ScaleUp with ScaleOut  https://review.opendev.org/72596421:42
*** raildo_ has joined #tripleo21:45
*** raildo has quit IRC21:48
*** _mmethot_ has joined #tripleo21:49
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible stable/train: Stop and remove container with module  https://review.opendev.org/72600821:50
sshnaidm|afkweshay|ruck, ^21:50
*** mmethot_ has quit IRC21:51
*** _mmethot_ has quit IRC21:56
*** mmethot has joined #tripleo21:56
openstackgerritAlex Schultz proposed openstack/tripleo-ansible master: Add tripleo dense formatting  https://review.opendev.org/72600122:01
*** raildo_ has quit IRC22:01
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Add openstack tripleo container image build  https://review.opendev.org/72414722:05
openstackgerritMerged openstack/tripleo-docs master: Remove mistral related developer documentation  https://review.opendev.org/72544822:06
*** ayoung has quit IRC22:16
*** jmasud has quit IRC22:17
*** ayoung has joined #tripleo22:17
*** jmasud has joined #tripleo22:18
*** threestrands has joined #tripleo22:20
*** morazi_ has quit IRC22:25
*** jmasud has quit IRC22:31
*** jmasud has joined #tripleo22:32
*** ekultails has quit IRC22:40
*** jmasud has quit IRC22:46
*** TrevorV has quit IRC22:54
*** tosky has quit IRC22:56
*** jmasud has joined #tripleo23:03
*** rh-jelabarre has quit IRC23:07
*** holser_ has quit IRC23:09
EmilienMweshay|ruck: do you want to expediate https://review.opendev.org/#/c/725944 ? we can ping mwhahaha if yes23:29
openstackgerritMerged openstack/tripleo-quickstart master: Update roles path tripleo-ansible and tripleo-ipa repos  https://review.opendev.org/70741923:39

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!