Monday, 2021-08-23

*** pmannidi|AFK is now known as pmannidi01:29
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Move grubenv to EFI dir  https://review.opendev.org/c/openstack/diskimage-builder/+/80400002:12
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Support grubby and the Bootloader Spec  https://review.opendev.org/c/openstack/diskimage-builder/+/80400202:12
opendevreviewSteve Baker proposed openstack/diskimage-builder master: RHEL/Centos 9 does not have package grub2-efi-x64-modules  https://review.opendev.org/c/openstack/diskimage-builder/+/80481602:12
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Add policycoreutils package mappings for RHEL/Centos 9  https://review.opendev.org/c/openstack/diskimage-builder/+/80481702:12
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Add reinstall flag to install-packages, use it in bootloader  https://review.opendev.org/c/openstack/diskimage-builder/+/80481802:12
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Add DIB_YUM_REPO_PACKAGE as an alternative to DIB_YUM_REPO_CONF  https://review.opendev.org/c/openstack/diskimage-builder/+/80481902:12
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Use non-greedy modifier for SUBRELEASE grep  https://review.opendev.org/c/openstack/diskimage-builder/+/80554502:12
opendevreviewTakashi Kajinami proposed openstack/puppet-tripleo stable/train: [Train only] Update the authtoken class for undercloud services  https://review.opendev.org/c/openstack/puppet-tripleo/+/80532703:22
*** pmannidi is now known as pmannidi|brb04:11
opendevreviewBrendan Shephard proposed openstack/python-tripleoclient master: Fix update run for Ephemeral Heat  https://review.opendev.org/c/openstack/python-tripleoclient/+/80557505:17
*** pmannidi|brb is now known as pmannidi05:42
opendevreviewMerged openstack/puppet-tripleo stable/train: [train-only] Use tripleo-undercloud-jobs-pipeline  https://review.opendev.org/c/openstack/puppet-tripleo/+/80548505:53
opendevreviewJiri Podivin proposed openstack/python-tripleoclient master: Enabling cover job in the zuul  https://review.opendev.org/c/openstack/python-tripleoclient/+/80541406:09
opendevreviewRabi Mishra proposed openstack/tripleo-heat-templates master: WIP Add http basic auth support for neutron api  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80459806:11
opendevreviewRabi Mishra proposed openstack/tripleo-heat-templates master: Deploy standalone ironic and neutron in undercloud  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/79491206:11
dciabrinykarel, morning o/ mind having a look at this backport https://review.rdoproject.org/r/c/openstack/tripleo-common-distgit/+/35029 ?06:25
ykareldciabrin, looking06:26
ykareldciabrin, you planning to backport to wallaby?06:27
ykarelthe linked review is of master06:28
ykarelwhich is merged06:28
dciabrinykarel, yes i it's broken on wallaby as well and i think it would benefit both upstream and downstream06:28
opendevreviewJiri Podivin proposed openstack/python-tripleoclient master: New test case and documentation for the export_stack function.  https://review.opendev.org/c/openstack/python-tripleoclient/+/80534706:29
opendevreviewJiri Podivin proposed openstack/python-tripleoclient master: Eliminating a race condition in the export_stack function  https://review.opendev.org/c/openstack/python-tripleoclient/+/80540906:29
ykareldciabrin, ack06:29
*** pmannidi is now known as pmannidi|mtg06:30
*** ysandeep|away is now known as ysandeep06:32
opendevreviewShnaidman Sagi (Sergey) proposed openstack/tripleo-ansible stable/wallaby: Convert all OS module imports to collections imports  https://review.opendev.org/c/openstack/tripleo-ansible/+/80486906:33
*** sshnaidm|afk is now known as sshnaidm06:34
sshnaidmTengu, can you review please in your time? It's backports: https://review.opendev.org/q/topic:%25222.11-stable/wallaby%2522+status:open06:35
Tengusshnaidm: gimme a sec06:36
Tengusshnaidm: hmmmmm with https://review.opendev.org/c/openstack/tripleo-ansible/+/804869 did you correct the pacemaker thing already?06:37
dciabrinykarel, thx!06:37
sshnaidmTengu, pacemaker thing is in tht06:37
Tenguor is it unrelated...06:37
Tengusshnaidm: ah, right. sorry - Monday, morning, brain's still trying to boot.06:38
Tenguhopefully it won't kernel-panic ^^'06:38
Tengusshnaidm: done :). have to reboot my node, I'll be offline a couple of minutes06:39
opendevreviewFrancesco Pantano proposed openstack/tripleo-heat-templates master: Consider GlanceMultistoreConfig when setting glance_pool  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80502906:43
Tenguback.06:43
opendevreviewFrancesco Pantano proposed openstack/tripleo-heat-templates master: DNM - Test Pending Ceph Pacific daemons in CI  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/77891506:49
opendevreviewMerged openstack/puppet-tripleo stable/train: Set authtoken memcached_servers in undercloud services  https://review.opendev.org/c/openstack/puppet-tripleo/+/80532406:50
opendevreviewFrancesco Pantano proposed openstack/tripleo-heat-templates master: DNM - Test Pending Ceph Pacific daemons in CI  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/77891506:51
*** pmannidi|mtg is now known as pmannidi|brb07:29
opendevreviewMichele Baldessari proposed openstack/tripleo-ansible stable/wallaby: Add BGP router-id and resolve-via-default options  https://review.opendev.org/c/openstack/tripleo-ansible/+/80555507:31
opendevreviewJiri Podivin proposed openstack/validations-common master: Reducing the sensitivity of the coverchange job  https://review.opendev.org/c/openstack/validations-common/+/80269307:33
opendevreviewMichele Baldessari proposed openstack/tripleo-heat-templates stable/wallaby: Add /run/frr volume mount  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80555607:36
opendevreviewMichele Baldessari proposed openstack/tripleo-heat-templates stable/wallaby: Support both frr 7.x and frr-8.0 concurrently  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80555707:36
*** jpena|off is now known as jpena07:37
opendevreviewMichele Baldessari proposed openstack/tripleo-ansible stable/wallaby: Fix BGP IPv4 unicast disablement  https://review.opendev.org/c/openstack/tripleo-ansible/+/80555807:38
opendevreviewJiri Podivin proposed openstack/validations-common master: DNM  https://review.opendev.org/c/openstack/validations-common/+/80559207:50
*** rpittau|afk is now known as rpittau07:51
*** pmannidi|brb is now known as pmannidi07:54
opendevreviewMatthias Runge proposed openstack/tripleo-heat-templates stable/wallaby: Example configurable cap add  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80555908:23
*** ykarel is now known as ykarel|lunch08:23
opendevreviewJing Li proposed openstack/diskimage-builder master: Add new element rocky  https://review.opendev.org/c/openstack/diskimage-builder/+/80560108:27
opendevreviewMerged openstack/tripleo-ansible master: Reimplement os_tripleo_baremetal_node_introspection  https://review.opendev.org/c/openstack/tripleo-ansible/+/76411508:28
opendevreviewJiri Podivin proposed openstack/validations-common master: Reducing the sensitivity of the coverchange job  https://review.opendev.org/c/openstack/validations-common/+/80269308:44
opendevreviewJiri Podivin proposed openstack/python-tripleoclient master: Overriding the log path argument for tripleoclient cli  https://review.opendev.org/c/openstack/python-tripleoclient/+/79457708:51
opendevreviewJiri Podivin proposed openstack/validations-common master: Reducing the sensitivity of the coverchange job  https://review.opendev.org/c/openstack/validations-common/+/80269309:22
*** ykarel|lunch is now known as ykarel09:36
opendevreviewJing Li proposed openstack/diskimage-builder master: Add new element rocky  https://review.opendev.org/c/openstack/diskimage-builder/+/80560909:50
opendevreviewJiri Podivin proposed openstack/validations-common master: Reducing the sensitivity of the coverchange job  https://review.opendev.org/c/openstack/validations-common/+/80269309:56
opendevreviewJesse Pretorius proposed openstack/tripleo-upgrade master: Add master upper constaints to testenv and linters  https://review.opendev.org/c/openstack/tripleo-upgrade/+/79727909:58
opendevreviewKamil Sambor proposed openstack/openstack-tempest-skiplist master: Revert "basic_ops mac_spoof fails upstream"  https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/80343410:01
opendevreviewHarald Jensås proposed openstack/python-tripleoclient master: Dynamic defaults for roles, networks, vips and bm  https://review.opendev.org/c/openstack/python-tripleoclient/+/79797910:01
opendevreviewHarald Jensås proposed openstack/python-tripleoclient master: Deprecate network data v1 definition  https://review.opendev.org/c/openstack/python-tripleoclient/+/79864210:01
opendevreviewMerged openstack/tripleo-common stable/victoria: Don't install dhcp-all-interfaces for redhat 8 onwards  https://review.opendev.org/c/openstack/tripleo-common/+/78837110:16
opendevreviewMerged openstack/python-tripleoclient stable/wallaby: Use 'all' for ANSIBLE_VARS_PLUGIN_STAGE  https://review.opendev.org/c/openstack/python-tripleoclient/+/80499210:16
opendevreviewMerged openstack/puppet-tripleo master: qdr: Add SSL support  https://review.opendev.org/c/openstack/puppet-tripleo/+/79694510:16
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: tripleo_push_artifacts: ensure destination exists  https://review.opendev.org/c/openstack/tripleo-ansible/+/80528110:44
opendevreviewMerged openstack/tripleo-heat-templates master: qdr: Add SSL support for oslo.messaging RPC  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/79713910:44
opendevreviewJiri Podivin proposed openstack/validations-common master: Reducing the sensitivity of the coverchange job  https://review.opendev.org/c/openstack/validations-common/+/80269310:48
*** ykarel is now known as ykarel|afk10:53
*** arxcruz|off is now known as arxcruz11:01
opendevreviewShnaidman Sagi (Sergey) proposed openstack/tripleo-ansible stable/wallaby: Convert all OS module imports to collections imports  https://review.opendev.org/c/openstack/tripleo-ansible/+/80486911:06
opendevreviewJiri Podivin proposed openstack/validations-common master: DNM  https://review.opendev.org/c/openstack/validations-common/+/80559211:19
*** dviroel|out is now known as dviroel|ruck11:31
*** jpena is now known as jpena|lunch11:33
opendevreviewCedric Jeanneret proposed openstack/tripleo-heat-templates master: Disable all container healthchecks by default  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/79996011:34
Tengufultonj: heya! I just took some time to check about the --network-port vs --network-config in https://github.com/fultonj/xena/tree/main/networkv2#modify-tripleo-lab-script-for-network-configuration11:36
Tengufultonj: it's "just" a matter of toggling a boolean in tripleo-lab :)11:36
Tengufultonj: basically, if you pass -e network_config_before_stack=true  you'll get the --network-config; Since it implies --network-ports, there's no conflict, and both can be passed without issue.11:37
Tengufultonj: hope this helps :)11:37
Tengu(sorry for the delay, had to re-read a bit and check that specific part in tripleo-lab, I didn't write it actually ^^'11:38
fultonjnetwork_config_before_stack11:39
fultonji see11:39
fultonjthanks Tengu 11:39
fultonjhttps://github.com/cjeanner/tripleo-lab/commit/da4972c5a6ea0a6fa56ae6dff9827e595164e4c3#diff-4f9ff34184d4268e67aad2f6db4b32f8de5807b17d9b57c63f71ae592af427e411:40
Tengufultonj: exactly. That's a contrib from ramishra when network-v2 was added11:41
Tenguand ready11:41
*** rlandy is now known as rlandy|rover11:43
Tengufultonj: if you find other issues like that, please feel free to open an issue against the project github (or ping me ;))11:43
fultonjTengu: sure, no prob11:44
Tengufultonj: I'll soon check for some centos-9 support11:44
Tenguat least for the VM provisionning and, maybe, deploy. Though that's another whole story.11:44
fultonji didn't see it as an issue originally11:44
Tengufultonj: having to manually edit a generated file is an issue ;)11:44
fultonjmore of a matter of scope11:44
fultonji was just using it wrong :)11:45
Tengu:)11:45
opendevreviewJohn Fulton proposed openstack/python-tripleoclient master: Introduce "openstack overcloud ceph deploy"  https://review.opendev.org/c/openstack/python-tripleoclient/+/80034111:48
Tenguhmmmmm.... weird.11:48
Tenguthe "provide" command is failing now on tripleo-lab, preventing the OC node to be ready...11:49
opendevreviewArx Cruz proposed openstack/openstack-tempest-skiplist master: Add featureset001 group into tempest-allow  https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/80439611:49
Tengudoes it ring any bells? Make nodes available | localhost | error={"changed": false, "msg": "Timeout waiting for node 65a17855-069f-40d8-beff-390fbc47d743 to have bridge_mappings set in the ironic-neutron-agent entry"}   hjensas maybe?11:50
opendevreviewMikolaj Ciecierski proposed openstack/tripleo-upgrade stable/train: Allow applying workaround patches in containers.  https://review.opendev.org/c/openstack/tripleo-upgrade/+/79959411:54
hjensasTengu: Have not seen that, here is the method that fails https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/ansible_plugins/modules/os_baremetal_provide_node.py#L379-L38911:54
hjensasTengu: Check logs for the ironic-neutron-agent? You can also manually check with "openstack network agent list", "openstack network agent show $agent_uuid" in case it happened, but too late for the 60s timeout?11:56
hjensasTengu: /var/log/containers/neutron/ironic-neutron-agent.log would be the log to check.11:57
opendevreviewJiri Podivin proposed openstack/validations-common master: DNM  https://review.opendev.org/c/openstack/validations-common/+/80559212:06
opendevreviewRabi Mishra proposed openstack/tripleo-heat-templates master: Add http basic auth support for neutron api  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80459812:07
opendevreviewRabi Mishra proposed openstack/tripleo-heat-templates master: Deploy standalone ironic and neutron in undercloud  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/79491212:07
jpodivinhi there, The log path argument for validator in tripleclient is freshly rebased and ready for review. Anyone willing to take a look ? https://review.opendev.org/c/openstack/python-tripleoclient/+/794577 12:11
Tenguhjensas: checking, but not seeing things :/12:13
Tenguhjensas: may it be poor resources on my UC?12:14
Tenguhjensas: dang.... now it passes. I'm seeing multiple messages in the agent talking about "Reporting state for host: 65a17855-069f-40d8-beff-390fbc47d743 with configuration: {'bridge_mappings': {'ctlplane': 'yes'}"12:15
Tenguhjensas: I have the feeling it's a matter of too short timeout at some point.12:15
Tenguwould that make any sense ?12:16
hjensasTengu: I'm not sure tbh, have not seen it fail before. The timeout is twice the time of the loop checking for new nodes and registering. Usually thats enough, but yeah.12:16
Tenguhjensas: didn't see that one before either.12:17
Tenguhjensas: it's a fresh deploy, on the same hardware I've been using for the last months12:17
Tengumaybe making that timeout a bit bigger would help. can't really say either.12:18
Tenguwould be interesting checking with other tripleo-labs users. maybe it's something in it, going too fast between tasks?12:18
Tenguhjensas: I'll finish my current deploy, crash it, and test a patch raising this timeout. Since it's steadily crashing (3 times in a row) here, it should be a good pointer.12:21
Tenguhjensas: fyi, it's running on a 12-core + 27G RAM virtual machine. Should be more than enough of an UC for a one-controller OC...12:22
hjensasTengu: Yes, that should be plenty. I'm not aware of any changes lately in the ironic-neutron-agent code, but it does depend on ironic and neutron api. If they are slow for some reason, I guess this could happen.12:23
Tenguhjensas: yeah.... who knows -.-. I'm using containers built on Aug-1812:24
Tenguusing-errr... lemme check the repositories.12:24
Tengucurrent-tripleo repositories.12:24
*** jpena|lunch is now known as jpena12:29
opendevreviewMikolaj Ciecierski proposed openstack/tripleo-upgrade master: Allow applying workaround patches in containers.  https://review.opendev.org/c/openstack/tripleo-upgrade/+/70312112:36
opendevreviewJiri Podivin proposed openstack/validations-common master: DNM  https://review.opendev.org/c/openstack/validations-common/+/80559212:44
opendevreviewAnanya proposed openstack/ansible-role-collect-logs master: Update url for sova query source  https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/78750212:44
Tenguhjensas: test is running. let's see if a 120s timeout is better. Maybe it should be an accessible parameter?12:55
hjensasTengu: maybe, but it worries me a bit that we now suddenly need to increase it. When it has been at 60s for quite some time? i.e something got slower ...12:58
Tenguhjensas: yep. lemme do some tests with a higher timeout - maybe it won't solve the issue at all. You can access the node if you want12:58
cloudnullmornings 12:59
opendevreviewShnaidman Sagi (Sergey) proposed openstack/tripleo-puppet-elements master: Install yum plugin priorities only on centos7 and fedora  https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/80563513:05
opendevreviewCedric Jeanneret proposed openstack/tripleo-specs master: Healthchecks: a new hope  https://review.opendev.org/c/openstack/tripleo-specs/+/80315213:06
Tengumrunge: -^^  updated :). Thanks for the reviews!13:07
*** amoralej is now known as amoralej|lunch13:07
opendevreviewJiri Podivin proposed openstack/validations-common master: Reducing the sensitivity of the coverchange job  https://review.opendev.org/c/openstack/validations-common/+/80269313:12
opendevreviewBrendan Shephard proposed openstack/python-tripleoclient master: Fix update run for Ephemeral Heat  https://review.opendev.org/c/openstack/python-tripleoclient/+/80557513:16
opendevreviewJing Li proposed openstack/diskimage-builder master: Add new element rocky  https://review.opendev.org/c/openstack/diskimage-builder/+/80563913:27
opendevreviewJing Li proposed openstack/diskimage-builder master: Add new element rocky  https://review.opendev.org/c/openstack/diskimage-builder/+/80564113:28
Tenguhjensas: hmmmm what agents should I see in the openstack network agent list output?13:33
Tenguhjensas: also: with a 120s timeout, it fails as well. so yeah. there's something weird.13:34
Tenguuho. ok. I see what I should get:   "ad27978d-a635-4054-b798-c45d6d92da3f | Baremetal Node     | 1b82598d-5c0d-447e-b0e8-05c5cc8b2b63 | None              | :-)   | UP    | ironic-neutron-agent"   but that one comes far, far, far later.13:36
Tenguguess I'm good to open an issue.13:37
*** amoralej|lunch is now known as amoralej13:46
Tenguhjensas: if you have some time: https://bugs.launchpad.net/tripleo/+bug/1940838   I'm a bit lost :/. if needed, the env is available (check your DM)13:46
opendevreviewJiri Podivin proposed openstack/validations-common master: Reducing the sensitivity of the coverchange job  https://review.opendev.org/c/openstack/validations-common/+/80269313:48
opendevreviewShnaidman Sagi (Sergey) proposed openstack/tripleo-ansible stable/wallaby: Convert all OS module imports to collections imports  https://review.opendev.org/c/openstack/tripleo-ansible/+/80486913:58
opendevreviewBrent Eagles proposed openstack/tripleo-ansible master: DNM: WIP: working on some cert repair playbooks for octavia  https://review.opendev.org/c/openstack/tripleo-ansible/+/80544214:07
*** pojadhav- is now known as pojadhav14:14
opendevreviewBrendan Shephard proposed openstack/tripleo-heat-templates master: Adjust HAProxy script to allow for ID change  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80547514:14
rlandy|roverPSI: vexxhost was having login issues ... just FYI if you see failures there14:17
Tengurlandy|rover: meh.. thanks for the info14:19
rlandy|roverTengu: vexxhost should be back now and job recovering14:20
Tengurlandy|rover: too easy :)14:20
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: Add BGP router-id and resolve-via-default options  https://review.opendev.org/c/openstack/tripleo-ansible/+/80555514:24
opendevreviewMichele Baldessari proposed openstack/tripleo-ansible stable/wallaby: tripleo_frr: set default version and profile  https://review.opendev.org/c/openstack/tripleo-ansible/+/80556914:31
weshay|ruckdviroel|ruck, odyssey4me may have some insight into the issue14:37
weshay|ruckodyssey4me, I noticed during an upgrade.. we see the mysql service shutdown, but we're not getting the logs where it starts back up14:38
weshay|ruckis that a known thing or maybe we're not grabbing the right logs14:39
weshay|ruckdviroel|ruck, can add detail.. re: https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-ussuri14:39
weshay|ruckstarted to fail on the 18th14:39
opendevreviewMerged openstack/tripleo-heat-templates stable/wallaby: Add /run/frr volume mount  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80555614:44
dviroel|ruckodyssey4me: weshay|ruck LP Bug https://bugs.launchpad.net/tripleo/+bug/1940844 - will continue to investigate 14:56
*** pmannidi is now known as pmannidi|AFK14:56
odyssey4medciabrin sounds like something you may be interested in :)14:57
dciabrinodyssey4me, weshay|ruck let me look at it14:58
opendevreviewMerged openstack/tripleo-docs master: Document "deployed ceph"  https://review.opendev.org/c/openstack/tripleo-docs/+/80424515:01
dciabrin<weshay|ruck> do you have a specific job failure where mariadb is not restarted? When I pick a random one from the failed jobs, i can see galera up and running https://4139e1adba54aef54cd3-90b48a3106a793a661c7495bcec7bb56.ssl.cf5.rackcdn.com/804797/1/check/tripleo-ci-centos-8-standalone-upgrade-ussuri/225239a/logs/undercloud/var/log/extra/podman/podman_allinfo.log15:04
dciabrinodyssey4me, looks like the br-ctlplane is down in the failed standalone jobs https://4139e1adba54aef54cd3-90b48a3106a793a661c7495bcec7bb56.ssl.cf5.rackcdn.com/804797/1/check/tripleo-ci-centos-8-standalone-upgrade-ussuri/225239a/logs/undercloud/var/log/extra/network.txt15:05
weshay|ruckdciabrin, so in both passing and failing.. I didn't see mysql start logs after the upgrade shutdown https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-standalone-upgrade-ussuri&pipeline=periodic15:05
*** amoralej is now known as amoralej|off15:08
opendevreviewJiri Podivin proposed openstack/python-tripleoclient stable/wallaby: Moving the 'export_data' dict into constants  https://review.opendev.org/c/openstack/python-tripleoclient/+/80535515:09
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: Fix BGP IPv4 unicast disablement  https://review.opendev.org/c/openstack/tripleo-ansible/+/80555815:09
opendevreviewMichele Baldessari proposed openstack/tripleo-ansible stable/wallaby: Add BGP EVPN support to tripleo_frr  https://review.opendev.org/c/openstack/tripleo-ansible/+/80567015:13
opendevreviewMichele Baldessari proposed openstack/tripleo-heat-templates stable/wallaby: Add BGP EVPN support to the FRR service  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80567115:14
dciabrinweshay|ruck, not sure i got the question :) when I pick a random job, I can see under /var/long/containers/mysql/mysqld.log that galera gets stopped during the upgrade. Then the DB upgrade task runs a temporary server and logs under /var/log/containers/mysql/mysqld-upgrade.log. Then the standalone restarts and logs continue in /var/log/containers/mysql/mysqld.log15:14
weshay|ruckdciabrin, ok.. I'll look again perhaps the timestamps will make it more clear to me.. sorry for that15:15
weshay|ruckdciabrin, dviroel|ruck will be your primary poc15:16
weshay|ruckI may just be noise15:16
weshay|ruckI may be just noise... rather15:16
dciabrinweshay|ruck, ack i'll continue with him them :)15:16
opendevreviewMichele Baldessari proposed openstack/tripleo-heat-templates stable/wallaby: Add BGP EVPN support to the FRR service  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80567115:16
dviroel|ruckbrb after lunch15:17
*** dviroel|ruck is now known as dviroel|ruck|lunch15:17
rlandy|roverramishra: hi - looking at the 16.2 backports15:32
rlandy|roverare there tests we could add to the downstream pipelines to help catch this?15:33
rlandy|roverysandeep: ^^?15:33
rlandy|roverwe have fs035 ovb15:33
ysandeepyes, fs035 is ipv6 job, we should have catch that issue in upstream train itself.15:34
* ysandeep reading bug15:34
rlandy|roverysandeep: either way - downstream backport15:34
rlandy|roverso we need to catch this in downstream only case as well15:34
ramishrarlandy|rover: It came from downstream CI15:34
ramishrahttps://rhos-ci-staging-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/hardware_provisioning/view/rqci/job/DFG-hardware_provisioning-rqci-16.2_director-virthost-3cont_2comp_2ironic-ipv6-geneve-IR-OC_Ironic_ipv6_provision-20201207-2258/15:34
ysandeepramishra, thanks! We will compare this job vs fs035 and where we can add coverage in fs035 to capture this issue early15:36
ramishraIt seems like a new job15:36
*** jpena is now known as jpena|off15:36
rlandy|roverramishra: yep - was looking for what testing could be caught earlier15:36
rlandy|roverupgrade15:37
rlandy|roverysandeep has it :)15:37
*** ysandeep is now known as ysandeep|dinner15:51
*** rpittau is now known as rpittau|afk15:58
opendevreviewMerged openstack/python-tripleoclient stable/wallaby: Implement protected resource types  https://review.opendev.org/c/openstack/python-tripleoclient/+/80394515:58
opendevreviewMerged openstack/python-tripleoclient stable/wallaby: Deprecate --network-ports, enable it by default  https://review.opendev.org/c/openstack/python-tripleoclient/+/80394615:58
*** ysandeep|dinner is now known as ysandeep16:10
*** dviroel|ruck|lunch is now known as dviroel|ruck16:19
opendevreviewMerged openstack/tripleo-heat-templates stable/wallaby: Support both frr 7.x and frr-8.0 concurrently  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80555716:23
opendevreviewMerged openstack/python-tripleoclient master: Enable py38 test  https://review.opendev.org/c/openstack/python-tripleoclient/+/80511316:36
opendevreviewFrancesco Pantano proposed openstack/tripleo-heat-templates stable/wallaby: Enable CephDashboard for scenario001  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80528016:38
opendevreviewMerged openstack/tripleo-ci master: add tripleo-common checkout to content-provider  https://review.opendev.org/c/openstack/tripleo-ci/+/80506117:42
opendevreviewBrent Eagles proposed openstack/tripleo-heat-templates master: Run some designate tempest tests  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/79809318:11
opendevreviewBrent Eagles proposed openstack/tripleo-ansible stable/ussuri: Use different name to distinguish between facts and vars in octavia  https://review.opendev.org/c/openstack/tripleo-ansible/+/80569218:15
opendevreviewBrent Eagles proposed openstack/tripleo-ansible stable/train: Use different name to distinguish between facts and vars in octavia  https://review.opendev.org/c/openstack/tripleo-ansible/+/80569318:16
weshay|ruckFACT: beagles knows what's up  VAR: does_he_like_this_joke18:26
eagleslol18:27
eaglesafter several days of "fun with jinja filters", I'm open to all sorts of humor18:27
*** ysandeep is now known as ysandeep|away19:05
opendevreviewSergii Golovatiuk proposed openstack/tripleo-upgrade master: Add master upper constaints to testenv and linters  https://review.opendev.org/c/openstack/tripleo-upgrade/+/79727919:24
*** slaweq is now known as slaweq_19:50
opendevreviewMerged openstack/python-tripleoclient master: Overriding the log path argument for tripleoclient cli  https://review.opendev.org/c/openstack/python-tripleoclient/+/79457719:53
opendevreviewMerged openstack/tripleo-ansible master: Remove stack_action from Octavia deployment  https://review.opendev.org/c/openstack/tripleo-ansible/+/80129920:02
opendevreviewMerged openstack/tripleo-heat-templates master: Remove StackAction from Octavia deployment  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80130120:02
opendevreviewDouglas Viroel proposed openstack/openstack-tempest-skiplist master: Skip TestSnapshotPattern and TestVolumeBootPattern  https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/80570820:58
opendevreviewMerged openstack/openstack-tempest-skiplist master: Skip TestSnapshotPattern and TestVolumeBootPattern  https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/80570821:13
-opendevstatus- NOTICE: The Gerrit service on review.opendev.org has been restarted for a patch version upgrade, resulting in a brief outage21:44
opendevreviewSteve Baker proposed openstack/tripleo-heat-templates master: Configure ironic to do UEFI boot by default  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80573222:23
*** dviroel|ruck is now known as dviroel|out22:37
opendevreviewSteve Baker proposed openstack/python-tripleoclient master: Add --boot-mode to import and configure  https://review.opendev.org/c/openstack/python-tripleoclient/+/80573322:43
*** chem is now known as Guest520122:51
opendevreviewMerged openstack/puppet-tripleo stable/train: [Train only] Update the authtoken class for undercloud services  https://review.opendev.org/c/openstack/puppet-tripleo/+/80532723:08
opendevreviewMerged openstack/puppet-tripleo stable/train: Revert "Deprecate warnings for Ceilometer and Gnocchi"  https://review.opendev.org/c/openstack/puppet-tripleo/+/80550823:08
opendevreviewSteve Baker proposed openstack/tripleo-quickstart master: Do not hard-code overcloud-full image name  https://review.opendev.org/c/openstack/tripleo-quickstart/+/80170323:08
opendevreviewSteve Baker proposed openstack/tripleo-quickstart master: Refactor inject_images for whole disk images  https://review.opendev.org/c/openstack/tripleo-quickstart/+/80140823:08
opendevreviewSteve Baker proposed openstack/tripleo-quickstart master: Switch to whole disk image on all master jobs  https://review.opendev.org/c/openstack/tripleo-quickstart/+/80140923:08
opendevreviewMerged openstack/puppet-tripleo stable/victoria: Revert "Deprecate warnings for Ceilometer and Gnocchi"  https://review.opendev.org/c/openstack/puppet-tripleo/+/80552723:44
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: Disable mlockall if postcopy is enabled  https://review.opendev.org/c/openstack/tripleo-ansible/+/80421423:44
opendevreviewMerged openstack/tripleo-ansible stable/victoria: Disable mlockall if postcopy is enabled  https://review.opendev.org/c/openstack/tripleo-ansible/+/80421523:44
opendevreviewMerged openstack/tripleo-ansible stable/ussuri: Disable mlockall if postcopy is enabled  https://review.opendev.org/c/openstack/tripleo-ansible/+/80421623:44
*** pmannidi|AFK is now known as pmannidi23:49
opendevreviewMerged openstack/tripleo-ansible stable/train: Disable mlockall if postcopy is enabled  https://review.opendev.org/c/openstack/tripleo-ansible/+/80421723:53
opendevreviewMerged openstack/validations-libs master: Add validation config file mechanism  https://review.opendev.org/c/openstack/validations-libs/+/79246023:53
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: tripleo_push_artifacts: fix argument parsing  https://review.opendev.org/c/openstack/tripleo-ansible/+/80528223:53
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: tripleo_frr: set default version and profile  https://review.opendev.org/c/openstack/tripleo-ansible/+/80556923:53
opendevreviewMerged openstack/tripleo-ansible stable/wallaby: Add BGP EVPN support to tripleo_frr  https://review.opendev.org/c/openstack/tripleo-ansible/+/80567023:53
opendevreviewMerged openstack/tripleo-heat-templates stable/wallaby: Add BGP EVPN support to the FRR service  https://review.opendev.org/c/openstack/tripleo-heat-templates/+/80567123:53
opendevreviewMerged openstack/tripleo-upgrade master: Improve node_upgrade_pre.sh  https://review.opendev.org/c/openstack/tripleo-upgrade/+/80523123:53

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!