Tuesday, 2020-03-24

*** tosky has quit IRC00:01
*** ramishra has quit IRC00:40
*** cdearborn has quit IRC00:44
*** jmasud has quit IRC00:55
*** jmasud has joined #tripleo00:56
*** ysandeep|away is now known as ysandeep00:59
*** Goneri has quit IRC01:13
openstackgerritJuan Badia Payno proposed openstack/tripleo-ansible master: BaR Always do MySQL dumps even the services wont be stopped  https://review.opendev.org/71145801:14
*** rlandy has quit IRC01:21
*** lbragstad__ has joined #tripleo01:33
*** lbragstad_ has quit IRC01:36
*** hjensas_ has joined #tripleo01:51
*** hjensas has quit IRC01:54
openstackgerritEmilien Macchi proposed openstack/ansible-role-redhat-subscription master: Improve support for HTTP proxy  https://review.opendev.org/71452402:02
*** jmasud has quit IRC02:03
*** jmasud has joined #tripleo02:04
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deprecate Keepalived service  https://review.opendev.org/65706702:04
*** ramishra has joined #tripleo02:07
ramishraweshay|ruck, cloudnull: https://bugs.launchpad.net/tripleo/+bug/1868632 is due to missing missing mistral-extra02:08
openstackLaunchpad bug 1868632 in tripleo "overcloud-deploy fails w/ The Workflow errored and no messages were received. " [Critical,Triaged]02:08
ramishraNot sure why it was reverted earlier.. revert does not have enough details02:09
ramishraykarel|away: ^^02:09
*** lbragstad__ has quit IRC02:13
openstackgerritMerged openstack/tripleo-common master: Add mistral workflow module path to mistral ansible config  https://review.opendev.org/71449202:14
openstackgerritMerged openstack/tripleo-ansible stable/train: Add role/playbook to manage nova image cache  https://review.opendev.org/71366102:35
weshay|ruckthanks ramishra02:44
*** rh-jelabarre has quit IRC02:49
ykarel|awayramishra, looking02:50
ykarel|awayramishra, issue was  CentOS8 build job failing, failures likely ignored due to non-voting.02:51
ykarel|awayramishra, issue is mentioned in original patch https://review.rdoproject.org/r/#/c/25280/02:51
ykarel|awayramishra, recently mistral-extra was added as depenedency for mistral, is this also needed for tripleo-common?02:52
ykarel|awayramishra, with https://review.rdoproject.org/r/#/c/25780/02:53
ramishraykarel|away: so we've not packaged it yet?02:54
ramishra" nothing provides python3-mistral-extra" ?02:54
ykarel|awayramishra, checking the logs, but with that mistral patch, that job passed02:54
ykarel|awayso possibly some repo issue02:54
ramishraykarel|away: mistal does not provide mistral-extra, it should be a dep for tripleo-common02:56
ykarel|awayramishra, where u seeing " nothing provides python3-mistral-extra"?02:57
ramishraykarel|away: https://github.com/openstack/mistral/blob/master/requirements.txt02:57
ykarel|awayramishra, i think mistral guys were involved in this change, and they said it to be added for mistral02:57
ramishraykarel|away: your build trace in https://review.rdoproject.org/r/#/c/25280/02:57
ramishraso surely need mistral-extra as a requirement in tripleo-common02:57
ykarel|awayramishra, there issue was due to no space around02:58
ykarel|awaynothing provides python3-mistral-extra>= needed by python3-tripleo-common-12.1.1-0.20200224125754.bbb3d19.el8.noarch02:58
ykarel|awaymissing space around >=02:58
ykarel|awayohhk i think i got issue02:59
ykarel|awaylet me confirm it02:59
ramishraykarel|away: ack thanks for looking, but we need that req AFAICT03:00
ykarel|awayramishra, as per logs mistral-extra is installed in container https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-tripleo-master/2031709/logs/undercloud/var/log/extra/podman/containers/mistral_engine/podman_info.log.txt.gz03:01
ramishraif it was the space issue, I've updated https://review.rdoproject.org/r/#/c/26072/03:01
ykarel|awayramishra, more is  missing there03:02
ykarel|awaythe original patch missed python2 also iirc03:02
ykarel|awaybut now it can be ignored03:02
ramishrathe current issue is surely due to missing mistral-extra03:03
ykarel|awayramishra, i think issue is due to outdated mistral03:04
ramishrawe seem to have added it in mistral package03:06
ramishraso yeah, it could be old mistral package03:06
ykarel|awayhttps://github.com/openstack/mistral/commit/8a5d35ac6f7a8677b33354d8a8fe8ea6a3738ea6 is included so mistral is also good03:06
weshay|ruckhow are we getting old mistral packages out of consistent?03:07
ramishraykarel|away: it should have been included in tripleo-common and not mistral.. but whatever03:07
ramishramistral-extra is 'extra' and not a dep for mistral..03:07
ykarel|awayramishra, ohhk we can do that but if u see logs it's installed there03:08
ykarel|awayso something else is wrong03:08
*** ykarel|away is now known as ykarel03:09
*** mmethot has joined #tripleo03:09
*** apetrich has quit IRC03:10
*** mmethot_ has quit IRC03:11
ramishraykarel: may be mistal-extra downstream package missing https://github.com/openstack/mistral-extra/commit/bb2af2e2be4ce39a621c222750b2dc7e360fa23803:22
ramishraI mean rdo packages03:24
ykarelramishra, ^^ is there in container03:24
ramishraykarel: the commit is there in the container?03:24
ramishraor the package?03:24
ykarelpython3-mistral-extra.noarch                   10.0.1-0.20200307102243.b615bd4.el803:25
ykarel^^ is installed in container03:25
*** dking_desktop has quit IRC03:27
ykarelramishra, do u know since when it started happeing03:28
ykareli see there is recent promotion for centos8 master03:28
ykarelok me checks03:29
ykarellast run passed with latest promotion03:30
ykarelbut yes good to understand why it happened to avoid further recurrence of it03:32
ykarelstrange it failed on 21st with same error, so something basic is wrong in those jobs03:37
ramishraoh, the last promotion was green. Is that the only job which has issues..03:37
ramishraIf the recent runs are green, we can probably update the bug and downgrade it from 'critical '03:39
ykarelramishra, from last run it seems good, but still i see something wrong as it failed on 21st with same error03:45
ykarelwill try to find what's going on with these03:46
ramishraykarel: ack, I just updated the bug with the current status..03:46
ykarelramishra, and wrt to mistral-extra as dep for tripleo-common03:48
ykareli don't see mistral as dep for tripleo-common03:48
ykarelso don't get how switch of actions to mistral-extra needs now as dep for tripleo-common03:49
ramishraykarel: we've mistral-lib that would provide the mistral api and mistral-extra for the actions we use03:53
ykarelramishra, yes i see mistral-lib03:54
ramishrawe've mistral package providing the action entrypoints earlier with mistral service03:54
ramishranot the case when they moved it to mistral-extra03:55
ykarelramishra, so mistral-extra also needed on host03:56
ykareloutside of container03:56
ramishranope, in the container03:56
ykarelokk, but it's there already03:56
ramishraykarel: yeah I can see that.. not sure why it fails at times..I've not yet understood that03:57
ykarelramishra, okk03:57
ramishraykarel: we'll get rid of mistral in undercloud soon.. but we've to make sure the packaging is right and the mistral containers have mistral-extra too03:59
ykarelramishra, okk04:11
*** skramaja has joined #tripleo04:13
openstackgerritRabi Mishra proposed openstack/python-tripleoclient master: Create *-messages container when setting deployment status  https://review.opendev.org/71456704:27
openstackgerritRabi Mishra proposed openstack/tripleo-heat-templates master: Remove most of the mistral related undercloud post config  https://review.opendev.org/71435404:29
*** jamesdenton has quit IRC04:30
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-heat-templates master: Enable ceph-dashboard on multinode jobs  https://review.opendev.org/71456804:30
*** bdodd has quit IRC04:52
*** jmasud has quit IRC04:53
*** jmasud has joined #tripleo04:56
*** ratailor has joined #tripleo04:58
*** jamesdenton has joined #tripleo05:03
openstackgerritMerged openstack/tripleo-heat-templates master: Neutron ML2/OVN: Add support to enable IGMP Snooping  https://review.opendev.org/71446205:04
*** gchamoul has quit IRC05:11
ykarelramishra, i got why it failed in those jobs05:13
ykarelramishra, mistral-api container don't have latest mistral package05:14
ykarelmistral-engine was fine05:16
openstackgerritMerged openstack/tripleo-heat-templates master: Introduce environments/disable-swift.yaml  https://review.opendev.org/71374705:17
*** udesale has joined #tripleo05:19
ramishraykarel: Ah, ok. sounds resonable05:21
ykarelramishra, but still its not clear why it didn't got updated05:24
ykarelall container images should be updated same05:25
ramishraykarel: not sure why engine container use api container packages05:25
ykarelramishra, actually issue is during mistral-db populate05:26
ramishramay be we bind mount stuff from host.. I can't see anything though....05:26
ykareland mistral-db populate usage api image05:26
ykareland api image is outdated05:26
ykarelfrom container prepare logs it's missed why it's missing https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-tripleo-master/2031709/logs/undercloud/var/log/tripleo-container-image-prepare.log.txt.gz05:27
ykarelthere is other bug in logging05:27
*** abdysn has joined #tripleo05:28
*** gchamoul has joined #tripleo05:29
*** bfournie has quit IRC05:30
ykarelramishra, with logging bug i meant https://bugs.launchpad.net/tripleo/+bug/186667605:35
openstackLaunchpad bug 1866676 in tripleo "Container updates logs are no longer logged to /var/log/tripleo-container-image-prepare-ansible.log" [Medium,Triaged]05:35
*** evrardjp has quit IRC05:36
ramishraykarel: if mistral_db_sync is using mistral-api container that should have mistral-extra package.. I think it's missing there05:36
*** evrardjp has joined #tripleo05:36
ykarelramishra, yup everything outdated there05:37
ramishraI think that's the issue... db_sync should not use api container IMO05:37
ramishrabut it's history05:38
ykarelin current situation, all mistral-lib/mistral-extra/mistral should be in sync05:38
ykarelnow everything should be ok after recent promotion05:38
ramishrasync in what?05:38
ramishraapi does not need those actions05:38
ykarelramishra, not api, it's about db05:38
ykarellet me paste links05:38
ramishrayes, db_sync should use an image with mistral-extra05:39
*** shyamb has joined #tripleo05:39
ykarelnot just mistral-extra, but also need mistral and mistral-lib updated05:40
ykarelhttps://review.opendev.org/#/c/713873/ from mistral and https://review.opendev.org/#/c/709963/ from mistral-lib needed to get work with mistral-extra05:40
shyambHi ykarel:05:40
shyambWill this document work for osp16 too?05:40
shyambI want to enable rhosp repos on all overcloud containers05:41
shyambI saw this rhosp13 doc05:41
shyambin rhosp16 I am not able to find it05:41
ramishraykarel: yep, they should be in sync for this kind of change, very difficult to do that you do that when using from source05:45
ykarelshyamb, looks like in 16 method changed https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/advanced_overcloud_customization/ansible-based-registration05:46
ykarelbut /me not aware about background, may be ramishra knows ^^?05:47
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: ControlPlaneSubnetCidr in net_vip_map_external  https://review.opendev.org/71354905:47
openstackgerritMerged openstack/python-tripleoclient stable/train: Add --work-dir option to container image build command  https://review.opendev.org/71443705:47
openstackgerritMerged openstack/tripleo-ci master: Adding dummy epel and epel-modular repo  https://review.opendev.org/71367605:47
ykarelramishra, yes05:47
ykarellooks like tooling issue05:48
ykarelbut without logs it's hard to find what went wrong in those jobs05:48
shyambykarel: Okay, thank you05:48
shyambI will try it05:48
openstackgerritMerged openstack/tripleo-heat-templates stable/train: ControlPlaneSubnetCidr in net_vip_map_external  https://review.opendev.org/71354805:54
shyambykarel: document looks perfect05:56
shyambthank you05:56
*** ykarel is now known as ykarel|afk05:58
*** shyamb has quit IRC06:00
*** abdysn has quit IRC06:11
*** Abdallahyas has joined #tripleo06:11
*** shyamb has joined #tripleo06:11
*** abdysn has joined #tripleo06:12
*** ykarel|afk is now known as ykarel06:14
*** Abdallahyas has quit IRC06:16
*** Abdallahyas has joined #tripleo06:16
*** abdysn has quit IRC06:17
*** saneax has joined #tripleo06:20
*** abdysn has joined #tripleo06:22
*** waleedm has joined #tripleo06:23
*** lmiccini has joined #tripleo06:24
*** Abdallahyas has quit IRC06:25
*** bfournie has joined #tripleo06:26
openstackgerrityatin proposed openstack/tripleo-ci master: [WIP] Enable tripleo-image-serve for CentOS8  https://review.opendev.org/71343906:30
openstackgerritRabi Mishra proposed openstack/python-tripleoclient master: Create *-messages container when setting deployment status  https://review.opendev.org/71456706:42
*** shyamb has quit IRC06:49
*** marios has joined #tripleo06:58
*** bandini has joined #tripleo07:03
*** marios_ has joined #tripleo07:08
*** pcaruana has joined #tripleo07:09
*** marios has quit IRC07:10
*** marios_ is now known as marios07:12
*** dpawlik has joined #tripleo07:16
openstackgerritMerged openstack/tripleo-validations master: Drop eventlet from lower constraints  https://review.opendev.org/71428307:24
openstackgerritMerged openstack/tripleo-validations master: Don't write playbook stats in binary mode  https://review.opendev.org/71428007:35
*** bogdando has joined #tripleo07:46
openstackgerritMerged openstack/python-tripleoclient master: Use ansible playbook for plan deploy  https://review.opendev.org/71377307:50
Tenguhjensas_: heya! apparently we want to put a |lower filter for the /etc/hosts generation :).07:50
Tengualthough I always thought it WAS case-insensitive, a new BZ says otherwise. probably regression with the move to tripleo-ansible tripleo_hosts_entries07:51
Tenguhjensas_: pushing a test on my lab, we should be fixed shortly.07:53
*** jcapitao has joined #tripleo07:57
abdysnTengu, hjensas_, good morning, so i am curious about this new patch, why was it not like this since the beginning? why was it render if it is possible to do it directly?07:58
*** shyamb has joined #tripleo07:58
Tenguabdysn: frankly, no idea, I didn't work on that role when it was created07:59
Tenguabdysn: and since I'm not really "fluent" with all the vars in t-h-t, I didn't see that hosts_entry thingy... and didn't think about checking if there was something already there :/.08:00
Tenguthis is why I pushed my other changes, before hjensas_ had a look08:01
abdysnTengu, my point is, maybe there is a point why it is rendered08:01
Tenguwe will need to get cloudnull and Luke reviews on it anyway08:02
*** cylopez has joined #tripleo08:02
abdysnTengu, great, was just curious. thanks for your hard work :D08:03
Tengunp - sorry for the delays though ^^'08:03
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-heat-templates master: Include Ansible host key and private key for ceph  https://review.opendev.org/71410408:04
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-heat-templates master: Enable ceph-dashboard on multinode jobs  https://review.opendev.org/71456808:04
*** tkajinam has quit IRC08:05
*** slaweq has joined #tripleo08:06
shyambMy provided list of repositories got configured on compute host08:07
shyambBut repositories inside container are different than host08:07
shyambWhat I am missing here?08:07
shyambI need rhsm repos that I provide in env file during overcloud deployment should be available inside all openstack containers08:08
*** apetrich has joined #tripleo08:09
shyambTengu: Overcloud update is failing with this error08:14
shyambWe are mounting a nfs share inside "/var/lib/nova"08:14
shyambDo you see any issue from these logs?08:15
*** xek has joined #tripleo08:15
shyamb""stderr: Error: relabel failed \"/var/lib/nova\": operation not supported","08:15
shyambthis selinux context we are changing on "/var/lib/nova/triliovault-mounts"08:18
Tengushyamb: well, there's selinux, of course, and mounting something in the location you pointed will create issues since we have "some" parts in tripleo that ensures we have the right contexts...08:21
shyambTengu: This issue looks similar https://bugzilla.redhat.com/show_bug.cgi?id=172726008:22
openstackbugzilla.redhat.com bug 1727260 in openstack-tripleo-heat-templates "[OSP15] Deployment with NovaNfsEnabled: True to enabled NFS share for nova ephemeral storage fails" [High,Closed: errata] - Assigned to mschuppe08:22
shyambin this case user is using NFS as nova backend08:23
Tengushyamb: well, you want to check if you have https://review.opendev.org/#/c/669317/08:23
Tenguguess that's it08:23
shyambYes, I am checking it08:24
shyambbut we are not using nfs for nova backend08:24
shyambwe are mounting our NFS for trilio backend08:24
shyambso, I need to check the root cause08:24
Tenguso the fix will probably not apply to you since you're doing it from another place.08:24
Tenguso the nova_nfs_enabled won't be set to true, so the z flag is still applied, so it will fail.08:25
Tengubecause NovaNfsEnabled is set to false08:25
*** paramite has joined #tripleo08:26
ykarelshyamb, looking,08:26
ykarelshyamb, so on 13, ur containers repos also got updated when u update repos on host?08:26
*** rpittau|afk is now known as rpittau08:26
shyambykarel: In osp13, we have not tried this scenario08:27
ykarelshyamb, okk, i don't think changing repos on host will auto change repos in containers08:28
ykarelbut there may be some way available to do this08:28
shyambykarel: Okay08:29
shyambykarel: I think I can skip this08:30
shyambI can install packages while building container08:30
shyamband I can enable subscription there itself08:30
shyambI needed this for testing on a running container08:30
shyambCustomer should also do not need any repos enabled on container, right?08:31
ykarelshyamb, no idea08:31
shyambykarel: okay, np. For now, I am all set08:33
shyambThank you08:33
*** pkopec has joined #tripleo08:36
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/python-tripleoclient master: Limit ansible playbook execution with timeout  https://review.opendev.org/71380708:37
shyambTengu: Regarding that nfs issue08:39
shyambThis is the fix08:39
shyambOne doubt I have is, why we are listing the volume two times?08:40
shyambwhat is it's meaning08:40
openstackgerritDaniel Bengtsson proposed openstack/tripleo-common stable/stein: Fix path of log file.  https://review.opendev.org/71458208:40
*** amoralej|off is now known as amoralej08:40
Tengushyamb: it's not 2 times - there's an "if" just before.08:41
*** tosky has joined #tripleo08:41
Tenguso that's what I just said earlier: the fix won't apply to your case because you're not using the NovaNfsEnabled thing, while still mounting an NFS volume in that location (even sub-dir).08:41
Tengucan't work like that.08:41
*** jbadiapa has quit IRC08:43
shyambso if condition is true then first volume statement will true?08:44
shyambor second?08:44
Tengufirst one08:45
shyambThis is how we mount the "/var/lib/nova"08:45
Tenguthink about it as ternary08:45
shyambin the same fashion08:45
shyambI do not use that z flag08:46
shyambDo you know what it is for?08:46
*** dtantsur|afk is now known as dtantsur08:47
openstackgerritMerged openstack/tripleo-common master: config-download timeout should KILL the process.  https://review.opendev.org/71370708:47
*** sshnaidm|pto is now known as sshnaidm08:48
Tengushyamb: "z" is for relabelling.... it's even written in the error message you posted08:48
*** ccamacho has joined #tripleo08:48
Tenguand, with NFS volume, you can't relabel anything - nfs just doesn't support it08:48
shyambI got the overall issue08:50
shyambI will try few things08:50
shyambthank you08:50
*** jmasud has quit IRC08:55
*** jmasud has joined #tripleo08:56
*** suuuper has joined #tripleo09:03
*** jpena|off is now known as jpena09:04
*** jpich has joined #tripleo09:06
*** lucasagomes has joined #tripleo09:06
bogdandohjensas_, Tengu: hi, do you know if we can migrate scripts/hosts-config.sh in t-h-t to tripleo-ansible?09:12
bogdandonot sure why it still sits there09:13
bogdandoalso afaik passing it via environment var may be limited in bash09:13
Tengubogdando: I'd say "probably"09:13
*** ccamacho has quit IRC09:15
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade master: Missing variable redifition during cleanup.  https://review.opendev.org/71458509:21
*** panda|off is now known as panda09:21
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade master: Missed variable redefinition during cleanup.  https://review.opendev.org/71458509:23
openstackgerritAthlan-Guyot sofer proposed openstack/tripleo-upgrade stable/train: Missed variable redefinition during cleanup.  https://review.opendev.org/71458709:24
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-heat-templates master: Drop unused hosts-config.sh in deploy steps  https://review.opendev.org/71458809:25
*** jfrancoa has joined #tripleo09:27
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Remove plan_management workflows and unused actions  https://review.opendev.org/71431409:29
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Remove swift_backup workbook and some related actions  https://review.opendev.org/71431509:29
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Remove deployment workbook and some unused actions  https://review.opendev.org/71431609:29
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Remove unused support.yaml workbook  https://review.opendev.org/71431709:29
*** bandini has quit IRC09:29
*** bandini has joined #tripleo09:30
*** avivgt has joined #tripleo09:31
*** shyamb has quit IRC09:36
*** alexmcleod has joined #tripleo09:42
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-heat-templates master: Drop unused remnants of the hosts-config bits  https://review.opendev.org/71458809:45
bogdandocjloader, hjensas_: ^^09:53
*** leanderthal has joined #tripleo09:53
bogdandothe next move would be to ditch all the hosts list -> str -> list dances09:54
*** leanderthal has quit IRC09:54
bogdandoand just use json all the places09:54
bogdandoor a list maybe09:54
bogdandowe have like 20 places we do join/split/join09:54
bogdandoslagle: ^^ WDYT?09:54
*** leanderthal has joined #tripleo09:55
Tengua list would be sufficient09:56
Tengujson for the hosts entries has no real use imho09:56
bogdandook I'll take a loot that09:57
bogdandolook and loot09:57
*** ykarel is now known as ykarel|lunch09:57
*** shyamb has joined #tripleo09:59
*** lbragstad__ has joined #tripleo10:14
*** holser has joined #tripleo10:19
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: FFWD2 Overcloud upgrade automation.  https://review.opendev.org/71352810:19
hjensas_bogdando: Tengu: how about storing the hostname/dnsname on the port? (where they belong - https://opendev.org/openstack/neutron-lib/src/branch/master/api-ref/source/v2/samples/ports/port-show-response.json#L17-L18 ?10:24
bogdandohjensas_: unsure how this stacks with the plans to remove neutron from uc10:25
*** hjensas_ is now known as hjensas10:25
hjensasbogdando: why would we ever want to do that? :)10:25
bogdandoI think there is a plan for that10:25
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Implement validations-libs List method for tripleoclient  https://review.opendev.org/71195710:26
bogdandoand use ansible network automation instead10:26
bogdandohjensas: anyway, the idea is nice10:26
bogdandobut would it also work with multiple overclouds?10:26
bogdandocustom ports may have arbitrary dns domains or not?10:27
hjensasbogdando: multiple overclouds still have their own ports created? the dns domain and hostname can be set per port. So I don't see why arbitrary dns domains would'nt work.10:28
bogdandook then10:28
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Implement validations-libs Run method for tripleoclient  https://review.opendev.org/71196410:29
bogdandohjensas: IIRC there had been issues with some dns bits may be only a singletone admin config in ini files10:29
bogdandoand users weren't being able even to update that config w/o admin rights for cloud10:30
bogdandoin neutron conf files10:30
bogdandowas that naneservers?10:30
hjensasbogdando: I think that's historical by now. https://opendev.org/openstack/neutron-lib/src/branch/master/api-ref/source/v2/ports.inc#L43-L5510:31
bogdandois still a quite new so indeed may be nice for tripleo to catch it up10:32
*** udesale_ has joined #tripleo10:36
hjensasI did'nt look at it a lot in THT context, but putting the data there seems easy. If that can help us reduce some of the list, map, string templating dancing in THT. I would have to look into.10:36
*** hjensas_ has joined #tripleo10:37
*** udesale has quit IRC10:38
*** hjensas has quit IRC10:41
*** ykarel|lunch is now known as ykarel10:42
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: FFWD2 Overcloud upgrade automation.  https://review.opendev.org/71352810:45
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Remove mistral dependency for password generation completely  https://review.opendev.org/71459810:50
bogdandohjensas_: wrt missing tht patch for the hosts_entry rework, just drop https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/common/deploy-steps.j2#L393 +310:57
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: Remove mistral dependency from password rotate module  https://review.opendev.org/71459910:57
openstackgerritRabi Mishra proposed openstack/tripleo-ansible master: Hide passwords from ansible output  https://review.opendev.org/71460010:57
bogdandoand pass only hosts_entry at https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/common/deploy-steps.j2#L44910:57
bogdandohjensas_: ^^10:58
bogdandoand that patch in tht should require your change in tripleo ansible10:58
bogdandohjensas_: > reduce some of the list, map, string templating dancing in THT11:00
bogdandoHostsValue (and VipHosts maybe) are string heat values that ideally should be stored in ansible inventories by config download directly, as lists11:00
bogdandoslagle: wdyt^11:00
*** jbadiapa has joined #tripleo11:00
bogdandothat would also affect the export dances documented for external data manipulations in multistack11:00
bogdandointernal heat values are not a part of user visible "APIs" so we are free to move them to ansible from heat w/o upgrade impact IMO11:01
bogdandohjensas_: so that could be a good starting point11:02
bogdandosame to moving HostsEntryValue from heat to inventory11:05
*** s0ru has joined #tripleo11:08
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: Set tempest_images path for periodic  https://review.opendev.org/71460911:09
bogdandohjensas_: btw, your current patches with hosts_entry rework, may also let VipHostsEntries go in tht11:10
*** s0ru has left #tripleo11:10
bogdandoafaik, it becomes not needed once you drop those 3 unused params11:10
bogdandohjensas_: I hope that's clear. If not, I can propose that patch for you, to complete https://review.opendev.org/#/c/714409/11:11
hjensas_bogdando: I think get it, please propose the patch. I'm pre-occupied with the inspection issue in RHEL 8.2 and 16.1 downstream atm.11:13
*** hjensas_ is now known as hjensas11:13
*** holser has quit IRC11:17
openstackgerritDaniel Bengtsson proposed openstack/tripleo-ansible master: Fix condition tuned profile.  https://review.opendev.org/71461011:18
*** holser has joined #tripleo11:18
openstackgerritSagi Shnaidman proposed openstack/tripleo-ci master: Sync vexxhost and rdo cloud settings  https://review.opendev.org/71150011:18
*** hjensas has quit IRC11:19
*** shyamb has quit IRC11:19
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-heat-templates master: Use global hosts entries to configure /etc/hosts  https://review.opendev.org/71461111:19
openstackgerritDaniel Bengtsson proposed openstack/tripleo-ansible master: Fix condition tuned profile.  https://review.opendev.org/71461011:20
*** hjensas has joined #tripleo11:22
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-heat-templates master: Use global hosts entries to configure /etc/hosts  https://review.opendev.org/71461111:24
*** shyamb has joined #tripleo11:28
openstackgerritDaniel Bengtsson proposed openstack/tripleo-ansible master: Fix condition tuned profile.  https://review.opendev.org/71461011:29
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-heat-templates master: Use global hosts entries to configure /etc/hosts  https://review.opendev.org/71461111:31
*** dpawlik has quit IRC11:33
*** dpawlik has joined #tripleo11:34
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Only set haproxy::crl_file once  https://review.opendev.org/71359311:38
*** rlandy has joined #tripleo11:40
*** psachin has joined #tripleo11:44
*** waleedm_ has joined #tripleo11:46
*** waleedm has quit IRC11:48
*** shyamb has quit IRC11:54
Tenguw00t! validation framework are now under TripleO governance!11:55
*** derekh has joined #tripleo11:56
*** jcapitao is now known as jcapitao_lunch11:59
*** jpena is now known as jpena|lunch11:59
*** rh-jelabarre has joined #tripleo12:00
*** rfolco has joined #tripleo12:02
*** rh-jelabarre has quit IRC12:04
*** jcoufal has joined #tripleo12:04
*** shyamb has joined #tripleo12:07
*** rh-jelabarre has joined #tripleo12:10
openstackgerritCarlos Goncalves proposed openstack/tripleo-ci master: Add Octavia standalone upgrade CI job  https://review.opendev.org/71047312:13
openstackgerritCarlos Goncalves proposed openstack/tripleo-ci master: Add Octavia standalone upgrade CI job  https://review.opendev.org/71047312:15
*** raildo has joined #tripleo12:18
*** ysandeep is now known as ysandeep|brb12:23
*** weshay|ruck changes topic to "CI status: RDO-Cloud CTRL plane going down today. monitoring mirror failures| community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs"12:32
shyambWe are facing an issue during upgrade of triliovault on osp1612:35
shyambIt's related to "/var/lib/nova" relabelling12:35
shyambHere is the error12:36
shyambWe mount "/var/lib/nova" in our container12:36
shyambHere are the mount options we use: https://github.com/shyam-biradar/triliovault-cfg-scripts/blob/master/redhat-director-scripts/docker/services/trilio-datamover-osp16.yaml#L17412:36
shyambThis bug looks close to our issue: https://bugzilla.redhat.com/show_bug.cgi?id=172726012:38
openstackbugzilla.redhat.com bug 1727260 in openstack-tripleo-heat-templates "[OSP15] Deployment with NovaNfsEnabled: True to enabled NFS share for nova ephemeral storage fails" [High,Closed: errata] - Assigned to mschuppe12:38
Tengushyamb: mention that you have an NFS mount that is managed by trilio in a sub-dir, and neither nova nor trilio knows about it, therefore mount the /var/lib/nova with the "z" flag, calling a relabelling...12:44
*** jpich has quit IRC12:44
Tenguthis is the actual issue in your service...12:45
Tenguor, at least, that's what I understood earlier.12:45
*** jpich has joined #tripleo12:45
shyambTengu: Okay12:46
shyambmy nova backend is not nfs12:46
shyambso it mounts the "/var/lib/nova" with z12:46
Tenguyep..... therefore the nfs_enable thing will be False, therefore it uses the "z" flag. but since you have a subdir on NFS there, it fails.12:46
Tengubecause z is recursive relabelling12:47
Tenguguess you'll need to override nova template in order to enforce the absence of "z" flag there.12:47
TenguUNLESS a new var/param is added in order to override the value in nova template (and in trilio)....12:48
Tengubut adding such a new param will probably raise some questions|concerns, and make the code more complicated for a single specific use-case.12:48
Tengunot sure it has a chance to make through reviews.12:48
shyambI can share my changes with you for review12:49
Tengushyamb: guess that nfs share *must* be known by nova?12:49
shyambThis was working fine on osp13, osp1412:49
Tenguif not, well, move it elsewhere. if yes, well.... you have two proposals that make sense12:49
Tenguyeah, osp-13 and osp-14 were using docker without any selinux support within containers.12:50
Tenguosp-15 and newer moved to podman, and selinux support is fully active.12:50
TenguI think this was already pointed in some earlier discussions.12:50
Tenguwith docker daemon, an option was passed in order to deactivate some selinux extensions in the daemon - since podman doesn't have such a daemon, it wasn't that easy to deactivate those extensions, and we took the opportunity to port the templates to a full selinux support instead.12:51
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart-extras master: Fix build-images for centos8 jobs set tripleoclient and image-yaml  https://review.opendev.org/71462712:51
Tenguthis is also better for customers, better security and all12:51
cloudnullif folks have a moment to review https://review.opendev.org/#/q/starredby:cloudnull+status:open,n,z it would be greatly appreciated.12:52
shyambTengu: Okay12:53
shyambI see12:54
*** jmasud has quit IRC12:54
shyambWhat are our options now?12:54
Tengushyamb: as said: either override the nova templates in your project, or propose a patch adding a new param allowing to NOT set the "z" flag - of course, the default value shouldn't change the default behavior.12:55
Tengushyamb: unless you can move that shared volume elsewhere, but if I understand, nova must have access to its content, right?12:55
shyambTengu: Yes, we can't move it12:55
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Implement validations-libs framework for ALL validator commands  https://review.opendev.org/71462912:55
shyambWe intentionally moved here12:56
Tengu'k. sooo... I don't see other options than the 2 above.12:56
*** lmiccini has quit IRC12:56
shyambHow can I override the nova template?12:56
*** jmasud has joined #tripleo12:56
shyambYou mean nova-compute heat template?12:56
Tengubasically you'll create a custom service, like trilio-nova-compute and push the needed things there.12:57
Tenguyou can copy-paste the original one and just drop the relevant flag|option|"if" and things12:57
Tenguand ensure all the templates mounting that same nova location are properly overridden as well.12:58
owalshTengu, shyamb: /var/lib/nova needs the z flag since it's mounted in multiple containers12:58
Tenguowalsh: n-yeah - but z fails to relabel when it has some subdir on NFS, which is shyamb case...12:58
Tenguit's an unusual thing.12:58
Tenguand a custom template|others.12:58
shyambowalsh: okay12:59
*** lmiccini has joined #tripleo12:59
Tengushyamb: MAYBE the right thing to do is to mount that nfs elsewhere on the host, and then bind-mount it to the location.12:59
Tenguthat should prevent the relabelling issues.12:59
shyambowalsh: We are checking the options to solve this one12:59
shyambTengu: It's complicated12:59
shyambWe are mounting our NFS share inside our container13:00
shyambunder "/var/lib/nova"13:00
*** ekultails has joined #tripleo13:00
shyambso that it gets reflected on nova_libvirt and nova_compute13:00
Tenguchicken, egg, and other things.13:00
Tenguissue is: with that sub-dir, you'll get relabelling issues, because NFS doesn't know about it.13:01
Tengu(in short: that's the issue you're hitting now)13:01
Tenguand, yes, if you drop that "z" flag, you might have other issues as mentionned by owalsh....13:01
shyambAny NFS options that may prevent this situation?13:01
*** jpena|lunch is now known as jpena13:01
owalshshyamb: does it have to be in /var/lib/nova?13:01
shyambowalsh: our goal is to make our nfs share available to nova_compute and nova_libvirt container13:02
owalshso no?13:02
shyambmounting under "/var/lib/nova" solved the problem13:03
shyambso we did13:03
shyambApproved by Redhat dev team13:03
Tengushyamb: you can mount it in other locations though... there's a whole filesystem in the containers in the end.13:03
chandankumarweshay|ruck, review rdoproject is down13:03
shyambWe mount it in our container13:04
*** weshay|ruck changes topic to "CI status: RDO-Cloud CTRL plane going down today!!!| community irc meeting Tues@1400 UTC - tripleo-ci-community meeting Tues@1330 UTC | https://docs.openstack.org/tripleo-docs"13:04
shyamband it gets available in other two containers through "var/lib/nova"13:04
shyambwhich is the only directory shared among three containers13:04
*** skramaja has quit IRC13:04
shyambwe do not have direct access to nova_compute and nova_libvirt13:04
*** skramaja has joined #tripleo13:05
shyambowalsh: Do you see any options here?13:05
Tengushyamb: though you might do specific services for the nova-compute and nova-libvirt in order to mount another volume, to another location, with the right flags (i.e. no z since NFS)13:05
owalshshyamb: how about NovaComputeOptVolumes13:05
Tenguah, even better13:05
*** psachin has quit IRC13:05
shyambowalsh: Okay, I am not aware of that container?13:06
shyambIt's used to mount additional volumes?13:06
shyambto nova_libvirt as well as nova_compute?13:07
*** ysandeep|brb is now known as ysandeep13:07
shyambour mount path is not static13:07
shyambthis is one more issue13:07
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/train: HA: check before restarting resource on stack update  https://review.opendev.org/71463413:08
Tengushyamb: ... any way to make it predectible at least?13:08
shyambWe create a hash for a give nfs share and mount under that hash directory13:08
shyambtechnically we can predict. but we need to use the algorithm again13:09
owalshshyamb: doubt that matters if the bind mount isn't relabeling13:09
Tengushyamb: well, so not that dynamic... you can probably generate a param file that will inject the needed info and use that NovaComputeOptVolumes thing..13:11
paramiteEmilienM, Thanks for the review on https://review.opendev.org/#/c/713627 . So I should just move the firewall record from puppet to HAProxy heat template IIUC13:12
Tenguowalsh: apparently that option only works for the nova_compute... ?13:12
owalshat the moment but...13:12
owalshratailor: are you around?13:12
openstackgerritAlan Bishop proposed openstack/puppet-tripleo stable/train: Handle ipv6 addresses in etcd and cinder's backend_url  https://review.opendev.org/71463513:13
Tenguyeah, there are other *OptVolumes in there - but not for libvirt.13:13
ratailorowalsh, yea.13:13
Tengushyamb: maybe https://linux.die.net/man/8/nfs_selinux  - not sure though.13:14
Tengubut it doesn't seem to actually add proper support for relabelling.13:15
shyambTengu: owalsh: If we get any nfs selinux option to solve this, then that would be the best way13:16
*** lmiccini has quit IRC13:17
owalshshyamb: can set the selinux context in the mount options13:17
owalshratailor: hey, trying to figure out an selinux vs NFS issue with shyamb. Do you know much about this?13:17
openstackgerritSorin Sbarnea proposed openstack/tripleo-ci master: lint: enable black  https://review.opendev.org/70398013:18
shyambowalsh: that would be great13:18
owalshshyamb: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-security-enhanced_linux-working_with_selinux-mounting_file_systems#sect-Security-Enhanced_Linux-Mounting_File_Systems-Mounting_an_NFS_File_System13:18
shyambowalsh: Thanks, I will go through this document13:19
shyambmeanwhile if you guys get any specific option please let me know13:20
*** amoralej is now known as amoralej|lunch13:20
shyambthanks a lot13:20
shyambThis is my end of the day13:20
shyambowalsh: If you get anything, please send me on shyam.biradar@trilio.io13:21
*** jpich has quit IRC13:22
owalshshyamb: ack, ttyl13:22
mariosfultonj: gfidente|pto: any idea about the defaults at http://codesearch.openstack.org/?q=v4.0.10-stable-4.0-nautilus-centos-7-x86_64&i=nope&files=&repos= tripleo-common13:22
mariosmainly what should we default to for centos8 fultonj13:22
*** jpich has joined #tripleo13:22
marios(in call with rlandy and chandankumar discussing now)13:22
* fultonj looking13:22
marios& zbr13:22
fultonjmarios: so you want to know what ceph-container to use13:23
fultonjmarios: unfortunately we don't yet have a cent8 based container. we are waiting for one13:23
zbri am it out of context13:23
openstackgerritFlavio Fernandes proposed openstack/tripleo-heat-templates master: Add Octavia OVN Provider configuration  https://review.opendev.org/70769513:23
fultonjmarios: i don't see any reason the cent7-based ceph container shouldn't work until it's available13:24
fultonjmarios: will that suffice until then we have a cent8-based one?13:24
mariosfultonj: ack13:25
openstackgerritBrent Eagles proposed openstack/tripleo-heat-templates master: Re-enable driver agent for scenario 10  https://review.opendev.org/71463913:25
*** lmiccini has joined #tripleo13:25
mariosfultonj: chandankumar: was poking at this waiting to hear if there was an error running the centos7 container (he will add details here we just got off the phone give couple mins )13:25
*** jcapitao_lunch is now known as jcapitao13:25
*** shyamb has quit IRC13:26
fultonjchandankumar: i expect the cent7 ceph-container will continue to work13:26
chandankumarmarios, fultonj on tobiko testing https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d6/683126/35/check/tobiko-tripleo-multinode-containers/6d69a07/logs/undercloud/home/zuul/overcloud_deploy.log13:26
chandankumarmarios, fultonj it looked for /usr/share/ceph-ansible/roles/ceph-container-engine/vars/RedHat.yml'13:27
chandankumarand  RETRYING: install container and lvm2 packages (3 retries left)13:27
chandankumarand from there I went to container image prepare file and saw c7 image so got worried, looked at the ceph-ansible code base it was correct but not sure why it is coming there13:28
*** ykarel is now known as ykarel|away13:29
fultonjchandankumar: what is the error message?13:30
chandankumarfultonj, FAILED - RETRYING: install container and lvm2 packages (3 retries left).13:30
fultonjchandankumar: we have a tripleo-validation for ceph to ensure lvm2 is installed on overcloud nodes13:30
chandankumarNo package python-docker-py available.'13:31
chandankumarfultonj, the question is why it is pulling this file https://github.com/ceph/ceph-ansible/blob/stable-4.0/roles/ceph-container-engine/vars/RedHat.yml13:31
*** derekh has quit IRC13:32
ratailorowalsh, do we have a BZ to know the context ?13:32
chandankumarsorry I think I got the issue13:32
fultonjchandankumar: i agree we don't want to be using that RedHat.yml13:33
fultonj09:32  chandankumar: sorry I think I got the issue13:33
*** mcornea has joined #tripleo13:33
fultonjchandankumar: you good then?13:34
chandankumarfultonj, https://github.com/ceph/ceph-ansible/blob/stable-4.0/roles/ceph-container-engine/tasks/pre_requisites/prerequisites.yml#L413:34
chandankumarI think I need to fix this part13:34
chandankumarfultonj, I need to use something like this {{ ansible_os_family | lower }}-{{ ansible_distribution_major_version | lower }}.yml13:35
fultonjchandankumar: we trigger ceph-ansible with a tag to make it not install any packages13:35
chandankumarfultonj, I need to use something like this {{ ansible_os_family }}-{{ ansible_distribution_major_version | lower }}.yml13:35
owalshratailor: not that I know of13:35
owalshratailor: IIUC trillovault container bind mounts /var/lib/nova and mounts NFS exports within that13:36
fultonjchandankumar: we've been assuming that the overcloud image should have what it needs (the lvm validation was only to figure out why the predeployed server scenario might have failed to simplify debug for user)13:36
openstackgerritRabi Mishra proposed openstack/python-tripleoclient master: Remove usage of swifthelper.SwiftTempUrlAction  https://review.opendev.org/71464013:36
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Remove unused support.yaml workbook  https://review.opendev.org/71431713:38
openstackgerritRabi Mishra proposed openstack/tripleo-common master: Remove mistral dependency for password generation completely  https://review.opendev.org/71459813:38
fultonjchandankumar: https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_ceph_common/defaults/main.yml#L6413:39
ratailorowalsh, ack. looks like I need to scroll through discussion above. will do it later. Thanks!13:39
chandankumarfultonj, sorry you mean we can set ceph_ansible_skip_tags: '' to skip the package installation step?13:41
fultonjchandankumar: no13:41
chandankumaror is it not going to check for lvm?13:41
fultonjchandankumar: no13:42
fultonjchandankumar: what i mean is that ceph-ansiuble shouldn't be trying to install anything13:42
fultonjchandankumar: and that we're already passing tags so that it shouldn't13:42
fultonjchandankumar: so...13:42
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Implement validations-libs framework for ALL validator commands  https://review.opendev.org/71462913:43
chandankumarfultonj, ok got it, Now I need to figure out why it is trying to install13:43
fultonjchandankumar: yeah13:43
fultonjchandankumar: by default it shouldn't be trying to install anything13:44
fultonjchandankumar: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d6/683126/35/check/tobiko-tripleo-multinode-containers/6d69a07/logs/undercloud/var/lib/mistral/config-download-latest/ceph-ansible/13:44
openstackgerritFlavio Fernandes proposed openstack/tripleo-heat-templates master: Add Octavia OVN Provider configuration  https://review.opendev.org/70769513:44
fultonjchandankumar: the problem is in ceph-ansible13:44
fultonjchandankumar: it simply shouldn't try to install it13:44
fultonjchandankumar: wait...13:45
fultonjchandankumar: interesting look at ceph_ansible_command.sh13:46
fultonjchandankumar: so problem isn't in ceph-ansible itself13:47
fultonjchandankumar: because we didn't pass it the tags it needs13:47
fultonjfmount: ^13:47
fultonjchandankumar: in other wrods...13:47
fultonjdefines ceph_ansible_skip_tags with 'package-install,with_pkg' to be skipped13:47
fultonjbut i don't see those in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d6/683126/35/check/tobiko-tripleo-multinode-containers/6d69a07/logs/undercloud/var/lib/mistral/config-download-latest/ceph-ansible/ceph_ansible_command.sh13:48
fultonjso that's a bug in the tripleo-ansible role that gfidente|pto fmount and i maintain13:48
fultonjchandankumar: OR your job overrode that var13:48
chandankumarfultonj, let me check the job definition13:48
chandankumarfultonj, https://review.opendev.org/#/c/683126/36/zuul.d/tobiko-tripleo.yaml job definition13:49
fmountfultonj: so the tags we have to skip pkg installation aren't passed to the playbook?13:49
fultonjfmount: yes, exactly13:50
chandankumarfultonj, job definition and playbooks used are ok13:51
*** ratailor has quit IRC13:52
chandankumarfultonj, I am opening a bug13:54
fultonjchandankumar: ok, that's fair13:54
fultonjfmount: i wonder what happended13:54
*** chem has quit IRC13:55
Tengumatbu, gchamoul did you add any reviews requiering some love in the doc?13:55
fultonjfmount: i see we had some commits land in tripleo-ansible/roles/ceph_* so maybe they broke something13:55
openstackgerritKevin Carter (cloudnull) proposed openstack/python-tripleoclient master: Switch to ansible for overcloud node (un)provision  https://review.opendev.org/70874913:55
*** chem has joined #tripleo13:57
fmountyeah fultonj I'm taking a look as well13:57
*** Goneri has joined #tripleo13:57
fultonjchandankumar: let me know the bug and fmount or i will take ownership of it13:57
*** tkajinam has joined #tripleo13:57
fultonjthanks fmount13:57
fmountwe should have some evidence in the previous job executions13:58
* fultonj checks if my local runs in my env had the same issue13:58
*** jdandrea has joined #tripleo13:59
weshay|ruck#topic agenda14:00
weshay|ruck* Review past action items14:00
weshay|ruck* One off agenda items14:00
weshay|ruck* Squad status14:00
weshay|ruck* Bugs & Blueprints14:00
weshay|ruck* Projects releases or stable backports14:00
weshay|ruck* Specs14:00
weshay|ruck* open discussion14:00
weshay|ruckAnyone can use the #link, #action and #info commands, not just the moderatorǃ14:00
weshay|ruckHey folks! who's around?14:00
ysandeepysandeep, o/14:02
weshay|ruck#topic review past action items14:02
weshay|ruckanyone need to raise anything from a previous meeting?14:02
weshay|ruck#topic one off agenda items14:03
weshay|ruck#link https://etherpad.openstack.org/p/tripleo-meeting-items14:03
weshay|ruckI added two items to the agenda..14:03
weshay|ruckplease add something if you have it..14:03
* weshay|ruck notes that rdo-cloud the 3rd party ci platform is in a scheduled maint window today14:03
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Refactorize test_tht_ansible_syntax to welcome new roles/modules  https://review.opendev.org/71366914:04
weshay|ruckthe ctrl plane is moving to vexxhost14:04
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: standalone/overcloud: enable the HA deployment by default  https://review.opendev.org/35906014:04
weshay|ruckshould be back by tomorrow14:04
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deprecate Keepalived service  https://review.opendev.org/65706714:04
weshay|ruckany questions?14:04
weshay|ruckEmilienM, stop working :)14:04
weshay|ruckleanderthal, you may want to get the testday on the agenda :)14:05
leanderthalooOOOooh good call14:05
weshay|ruckok.. the next topic is around bugs14:05
weshay|ruckwe were up to 60 untriaged bugs last week.. a lot of folks do NOT have access to self triage and that's fine14:05
weshay|ruckbut if are a member of the tripleo org in launchpad.. we'd hope that you do follow the triage process14:06
sshnaidmdon't we run a meeting bot?14:06
weshay|ruckSo Question for the group...14:06
weshay|rucksshnaidm, yes14:06
weshay|ruckbut maybe it's down.. ?14:06
weshay|ruckdon't know14:06
weshay|ruckQuestion... Any objects to closing any bugs in marked "New" ( untriaged ) older than 365 days?14:07
weshay|ruckobjection rather14:07
mariosweshay|ruck: sounds good to me14:07
mariosweshay|ruck: but 'latest updated ' date right?14:07
weshay|ruckya.. I'll account for that... good point14:07
mariosweshay|ruck: not filed? maybe doesn't matter 365 is long enough that it either /or14:07
chandankumarfultonj, fmount https://bugs.launchpad.net/tripleo/+bug/1868731 thanks for looking into that14:07
openstackLaunchpad bug 1868731 in tripleo "ceph_ansible_skip_tags is getting ignored during multinode overcloud deploy" [Critical,Triaged]14:07
beagles(wrong window) but yeah, that sounds reasonable14:07
*** florianf has joined #tripleo14:08
fultonjchandankumar: thanks14:08
weshay|ruckI'll send an email to the list to have folks reopen if they need it14:08
weshay|ruckok.. seeing no objections... we'll go ahead and clean it up :)14:08
weshay|ruck#topic one off agenda items14:08
weshay|ruck#link https://etherpad.openstack.org/p/tripleo-meeting-items14:08
weshay|ruck#topic Active Squad status14:08
weshay|ruck#link https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw?both14:08
weshay|ruck#link https://etherpad.openstack.org/p/tripleo-validations-squad-status14:08
weshay|ruck#link https://etherpad.openstack.org/p/tripleo-integration-squad-status14:08
weshay|ruck#link https://etherpad.openstack.org/p/tripleo-ansible-agenda14:08
weshay|ruck#link https://etherpad.openstack.org/p/tripleo-mistral-to-ansible14:09
*** TrevorV has joined #tripleo14:09
weshay|ruckany reviews needed by the squads?14:09
Tenguvalidations: we have our package under TripleO governance!14:09
Tenguit's been merged today.14:09
weshay|ruckrdo governance :)14:09
cloudnullweshay|ruck the following reviews would be great to get some eyes on - https://review.opendev.org/#/q/starredby:cloudnull+status:open,n,z14:09
weshay|ruckthanks cloudnull14:10
weshay|ruckfultonj, I mentioned this earlier.. but ansible 2.9 is being tested along side the latest ceph-ansible.. I don't have a ton of detail on that14:10
*** derekh has joined #tripleo14:11
weshay|ruckok.. let's move on14:11
fultonjweshay|ruck: ack14:11
weshay|ruck#topic Moderately Active Squads14:11
weshay|ruck#link https://etherpad.openstack.org/p/tripleo-upgrade-squad-status14:11
weshay|ruck#link https://etherpad.openstack.org/p/tripleo-edge-squad-status14:11
weshay|ruck#link https://etherpad.openstack.org/p/tripleo-networking-squad-status14:11
weshay|ruckprobably nothing new here14:11
weshay|ruck#link https://launchpad.net/tripleo/+milestone/ussuri-314:11
weshay|ruck5 New, 28 Incomplete, 4 Invalid, 4 Won't Fix, 4 Confirmed, 385 Triaged, 120 In Progress, 2 Fix Committed, 151 Fix Released14:12
weshay|rucknot bad on the bug counts.. I guess14:12
weshay|ruckStoryboard bugs.14:13
weshay|ruck#link https://storyboard.openstack.org/#!/project_group/7614:13
cloudnullweshay|ruck we've recently updated the mistral_to_ansible board - https://storyboard.openstack.org/#!/board/20814:13
weshay|ruckvery nice..14:14
weshay|ruckk.. will include that in the future14:14
weshay|ruckThere are two blueprints not started for ussuri14:14
weshay|ruckMediumtripleo-firewall-rule-reportingNewNot startedussuri14:14
weshay|ruckMediumremove-puppetDraftingNot startedEmilien Macchiussuri14:14
*** amoralej|lunch is now known as amoralej14:15
weshay|ruckEmilienM, maybe we move that to the next release or future at this point...14:15
weshay|ruckApril 20th is m3 I think14:15
weshay|ruckand dsneddon owns https://blueprints.launchpad.net/tripleo/+spec/tripleo-firewall-rule-reporting14:15
weshay|ruck#topic specs14:16
weshay|ruck#link https://review.openstack.org/#/q/project:openstack/tripleo-specs+status:open14:16
weshay|ruckreviews for luke please https://review.opendev.org/#/c/698828/ on14:16
weshay|ruckAdd spec for AWX integration.14:16
weshay|ruck#topic open discussion14:16
weshay|ruckAnything else that folks want to bring up to the meeting?14:16
EmilienMweshay|ruck: i'll probably abandon it for now14:17
weshay|ruckk.. thanks14:17
weshay|ruckanyone else ?14:17
weshay|ruckgoing once14:17
weshay|ruckgoing twice14:17
weshay|ruck#endmeeting tripleo14:17
mariosdo we wantto consider virtual ptg14:17
mariosweshay|ruck: ?14:17
weshay|ruckI'll save the meeting notes14:17
mariosweshay|ruck: next time then14:17
weshay|ruckmarios, that's still not clear14:17
weshay|ruckbut most likely14:18
cloudnull+1 for virt ptg14:18
mariosweshay|ruck: i think it is clear14:18
mariosweshay|ruck: and we should consider how we do it if we do it etc14:18
mariosweshay|ruck: or we do things via mailing list14:18
sshnaidmthis is final I think, virtual prg14:18
mariosnova had some good ideas today on the list14:18
mariosk weshay|ruck next week's meeting is still early enough for us to organise discuss/if not before then here etc14:19
EmilienMmarios: yeah14:20
EmilienM#link http://lists.openstack.org/pipermail/openstack-discuss/2020-March/013555.html14:20
EmilienMcheck this out ^ pretty good stuffs14:20
mariosEmilienM: ack saw some of the discussion go by today but didn't revisit there were some proposals in an equivalent nova thread14:20
mariosEmilienM: yes actually this is the one i saw proposal from gibi14:21
weshay|ruckah.. this is news to me..14:23
weshay|ruckPTG is canceled, will add to the next mtg14:23
mariosweshay|ruck: yeah was news for me too this morning14:24
leanderthalwell, ptg isn't cancelled, it's virtual.14:24
marioswell that's what we should decide formats/organise etc while we have lead time14:25
weshay|ruckMarch 20 I guess that news came out14:25
leanderthalif you'd like to help organize it, add your name here https://etherpad.openstack.org/p/Virtual_PTG_Planning14:25
mariosweshay|ruck: yeah as you know cos you gave me the day off, i was out yesterday :D14:25
leanderthalit came out.... last thursday14:25
mariosweshay|ruck: i guess i was working too hard on friday to notice ;)14:25
* marios runs14:25
ekultailsA virtual PTG should be interesting.14:26
leanderthalsorry, wednesday lists.openstack.org/pipermail/foundation/2020-March/002854.html14:26
weshay|ruckdidn't really hit the list afaict14:26
weshay|ruckodd.. must be a filter issue on my end14:27
cloudnullweshay|ruck did we ever figure out what was going on with mistral / mistral-extras ? -cc apetrich14:27
mwhahahathat didn't got to openstack-discuss14:27
mwhahahathat went to foundation14:27
mwhahaha(it should be forwarded)14:28
apetrichcloudnull, I'm looking into it since this morning but haven't found what can be causing it :(14:28
weshay|ruckcloudnull, ya.. updates in the bug.. yatin and Rabi were working on it last night14:28
weshay|ruckapetrich, thanks14:28
*** ysandeep is now known as ysandeep|afk14:28
*** waleedm_ has quit IRC14:31
openstackgerritLuke Short proposed openstack/tripleo-common stable/train: config-download timeout should KILL the process.  https://review.opendev.org/71465914:33
*** bdodd has joined #tripleo14:38
fultonjdid the meeting end?14:39
weshay|ruckbot was down14:39
fultonjchandankumar: https://bugs.launchpad.net/tripleo/+bug/1868731/comments/114:40
openstackLaunchpad bug 1868731 in tripleo "ceph_ansible_skip_tags is getting ignored during multinode overcloud deploy" [Critical,Triaged]14:40
fultonjchandankumar: standalone jobs are not affected14:40
fultonjchandankumar: is there a multinode job for train or cent7 ?14:41
chandankumarfultonj, fs016 was not used anywhere, we have started using right now for tobiko testing14:41
chandankumarfultonj, let me check train job14:41
*** waleedm_ has joined #tripleo14:42
*** skramaja has quit IRC14:48
*** dking_desktop has joined #tripleo14:50
fultonjchandankumar: https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/ceph-ansible/ceph-base.yaml#L80-L8314:51
*** jbadiapa has quit IRC14:52
chandankumarfultonj, https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/ci/environments/scenario001-multinode-containers.yaml#L14414:52
fultonjwho did that!14:52
chandankumarCephAnsibleSkipTags: ''14:52
fultonjthat's the root cuase14:52
fultonjfmount: ^ fyi14:52
fultonjchandankumar: nice find14:53
Tenguprobably some wrong copy-paste :)14:53
fultonjchandankumar: can you just remove that line in your job?14:53
fmountahahha yeah, so you're overriding the skip-tags w/ no value14:54
fmountyeah, in the standalone jobs we don't have CephAnsibleSkipTags14:54
chandankumarfultonj, yes sure14:54
fmountso the default value is used14:54
fultonjchandankumar: ok, thanks so it was the job definition14:54
chandankumarfultonj, yes14:54
chandankumarEmilienM, https://review.opendev.org/#/c/714104/ please +w it14:55
fmountand by default, in tripleo-ansible the parameter passed to ceph-ansible is https://github.com/openstack/tripleo-ansible/blob/master/tripleo_ansible/roles/tripleo_ceph_common/defaults/main.yml#L6414:56
fmountchandankumar: ^14:56
fultonjchandankumar: can you update your patch with Closes-Bug: #1868731 ?14:56
openstackbug 1868731 in tripleo "ceph_ansible_skip_tags is getting ignored during multinode overcloud deploy" [Critical,Triaged] https://launchpad.net/bugs/186873114:56
*** abdysn has quit IRC14:56
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-heat-templates master: Include Ansible host key and private key for ceph  https://review.opendev.org/71410414:58
chandankumarfultonj, ^^14:58
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add ffwd upgrade prepare role  https://review.opendev.org/71466914:58
*** abdysn has joined #tripleo14:59
chandankumarfultonj, https://review.opendev.org/#/c/714104/3/ci/environments/scenario001-multinode-containers.yaml@155 is it possible to set those values somewhere else14:59
chandankumarfmount, fultonj thanks :-)14:59
fultonjthanks chandankumar15:00
chandankumarfultonj, is there a way ANSIBLE_PRIVATE_KEY_FILE: '/home/zuul/.ssh/id_rsa_tripleo' to remove hardcoded zuul value?15:03
fultonjchandankumar: i'm thinking about that i saw mwhahaha 's comment15:03
chandankumarok cool, thanks !15:04
fultonjchandankumar: i think we could make your patch to tht depends-on a tripleo-ansible patch15:04
*** tkajinam has quit IRC15:04
fultonjand in that tripleo-ansible patch we could make it pass --overcloud-ssh-key + --overcloud-ssh-user to the ceph-ansible run15:04
fultonjchandankumar: i'll put a patch up for your to depends on and comment that in the review and we'll see if that's better15:05
fultonjfmount: fyi ^15:05
chandankumarfultonj, works for me15:05
EmilienMbogdando: have you seen the comment on https://review.opendev.org/714588 ?15:09
*** bogdando has quit IRC15:11
openstackgerritMerged openstack/tripleo-upgrade master: Missed variable redefinition during cleanup.  https://review.opendev.org/71458515:12
*** leanderthal has quit IRC15:15
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add centos/rhel based interface/bridge commands  https://review.opendev.org/71420815:17
openstackgerritEmilien Macchi proposed openstack/python-tripleoclient master: Clean unit tests from OS::TripleO::Hosts::SoftwareConfig  https://review.opendev.org/71467715:19
EmilienMhjensas: ^15:20
openstackgerritDaniel Bengtsson proposed openstack/tripleo-ci master: Fix the markup.  https://review.opendev.org/71467915:21
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add ffwd upgrade prepare role  https://review.opendev.org/71466915:35
*** ysandeep|afk is now known as ysandeep15:35
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Implement validations-libs framework for ALL validator commands  https://review.opendev.org/71462915:36
*** abdysn has quit IRC15:38
cloudnullanyone mind giving https://review.opendev.org/#/c/714520/ a look15:39
*** lbragstad__ has quit IRC15:40
*** abdysn has joined #tripleo15:41
weshay|rucksshnaidm, matbu FYI.. think I'm hearing opendev/x will be the best home15:41
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add ffwd upgrade prepare role  https://review.opendev.org/71466915:42
slaweqhi, is tripleo standalone working on centos 8 now?15:44
*** d0ugal has quit IRC15:44
slaweqI'm trying to deploy it and I have error like http://paste.openstack.org/show/791091/15:44
*** d0ugal has joined #tripleo15:44
slaweqand I don't know what exactly am I missing there15:44
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add ffwd upgrade prepare role  https://review.opendev.org/71466915:49
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud ffwd-upgrade run  https://review.opendev.org/71468315:49
mwhahahaslaweq: if you're internal you need to specify internal ntp servers15:53
mwhahahait failed trying to query ntp15:54
openstackgerritMerged openstack/tripleo-common stable/stein: Fix path of log file.  https://review.opendev.org/71458215:55
*** ykarel|away is now known as ykarel15:58
*** waleedm_ has quit IRC15:58
ykarelmwhahaha, slaweq looks like issue is something else not related to ntp server, it failed getting ntp servers from hiera Failed to start Hiera: RuntimeError: Config file /etc/puppetlabs/puppet/hiera.yaml not found15:59
ykarelmay be hiera/puppet installed from epel or some other repo?15:59
mwhahahaykarel: that may not be the fatal error16:00
mwhahahantp is fatal16:00
ykarelmwhahaha, looks like it didn't went in the loop due to that hiera error16:00
slaweqykarel: can be16:03
slaweqI will check16:03
*** hjensas_ has joined #tripleo16:04
*** hjensas has quit IRC16:07
*** jcapitao is now known as jcapitao_afk16:07
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Make the metalsmith module interactions more resilient  https://review.opendev.org/71468716:07
openstackgerritFlavio Fernandes proposed openstack/tripleo-heat-templates master: Add Octavia OVN Provider configuration  https://review.opendev.org/70769516:08
ykarelslaweq, ack16:09
*** florianf has quit IRC16:12
*** dciabrin has quit IRC16:18
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates stable/train: Introduce environments/disable-swift.yaml  https://review.opendev.org/71468816:21
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: FFWD2 Overcloud upgrade automation.  https://review.opendev.org/71352816:23
*** dtantsur is now known as dtantsur|afk16:25
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud ffwd-upgrade run  https://review.opendev.org/71468316:30
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add ffwd upgrade converge  https://review.opendev.org/71469016:30
*** jcapitao_afk is now known as jcapitao16:32
*** abishop_ has joined #tripleo16:33
*** abishop has quit IRC16:33
openstackgerritJohn Fulton proposed openstack/tripleo-docs master: Introduce Multibackend Storage Documenation  https://review.opendev.org/71307816:34
*** artom has quit IRC16:40
*** artom has joined #tripleo16:40
slaweqykarel++ it seems that I indeed had hiera from epel repo and that was the issue16:42
openstackgerritMerged openstack/puppet-tripleo stable/train: pacemaker-remote is broken on CentOS/RHEL < 8  https://review.opendev.org/71410716:43
ade_leeEmilienM, mwhahaha ever had an undercloud fail to install because of this error in puppet-container-ironic ?16:44
ade_leeuppet-user: Error: /Stage[main]/Ironic::Pxe/File[/var/lib/ironic/tftpboot/ipxe.efi]: Could not evaluate: Could not retrieve information from environment production source(s) file:///usr/share/ipxe/ipxe.efi16:44
mwhahahanope i've seen that error in bug reports tho16:45
* mwhahaha defers to bfournie, hjensas_, stevebaker & co16:45
ade_leeEmilienM, mwhahaha looks like its failing to set up pxe stuff ...16:45
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Fix readme issues  https://review.opendev.org/71471016:45
ade_leebfournie, stevebaker hjensas_ ^^ hjensas_ this is running the undercloud install in your setup16:46
*** abishop_ is now known as abishop16:47
*** ysandeep is now known as ysandeep|away16:50
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add ffwd upgrade converge  https://review.opendev.org/71469016:50
*** marios has quit IRC16:53
*** udesale_ has quit IRC16:53
ykarelslaweq, ack16:53
*** ykarel is now known as ykarel|away16:53
*** jmasud has quit IRC16:54
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add centos/rhel based interface/bridge commands  https://review.opendev.org/71420816:54
*** jmasud has joined #tripleo16:55
ekultailsCan I get a Workflow nudge on this one? https://review.opendev.org/#/c/707380/17:02
*** lbragstad__ has joined #tripleo17:02
*** lbragstad__ is now known as lbragstad17:04
*** lucasagomes has quit IRC17:09
*** waleedm_ has joined #tripleo17:10
*** sshnaidm is now known as sshnaidm|afk17:13
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud credentials  https://review.opendev.org/71471817:13
*** bandini has quit IRC17:14
*** jpich has quit IRC17:16
*** beagles has quit IRC17:16
*** rpittau is now known as rpittau|afk17:18
openstackgerritMerged openstack/tripleo-ansible master: tripleo_container_manage: improve logging for failed containers  https://review.opendev.org/71423417:20
openstackgerritMerged openstack/tripleo-ansible master: Force container deletion if namespace does not exist in service_kill  https://review.opendev.org/71451717:23
*** beagles has joined #tripleo17:29
*** abdysn has quit IRC17:31
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade master: Correct wrong template reference.  https://review.opendev.org/71472517:32
*** evrardjp has quit IRC17:36
*** evrardjp has joined #tripleo17:36
*** waleedm_ has quit IRC17:36
*** pkopec has quit IRC17:37
bfournieade_lee: sorry, in a meeting, which version are you seeing this on?17:43
*** elGamal has quit IRC17:45
*** fultonj has quit IRC17:45
*** jistr has quit IRC17:45
*** yolanda has quit IRC17:45
*** openstackstatus has quit IRC17:45
*** iurygregory has quit IRC17:45
*** dmsimard|off has quit IRC17:45
*** jistr has joined #tripleo17:45
ade_leebfournie, master I believe -- not sure if its actually causing a failure -- I tried adding some pxe related packages and errors went away17:45
ade_leebut I still have some error about Mar 24 17:21:03 undercloud.tls.example.com puppet-user[53922]: Error: Facter: error while resolving custom fact "rabbitmq_nodename": undefined method `[]' for nil:NilClass17:46
*** yolanda has joined #tripleo17:46
ade_leemwhahaha, EmilienM ^^ ya'll seen that one before17:46
mwhahahathat fact is bad17:47
mwhahahabut it doesn't stop anything17:47
*** irclogbot_0 has quit IRC17:47
mwhahahafact failures just spit output17:47
ade_leemwhahaha, ok - then something else is borking things ..17:48
*** openstackstatus has joined #tripleo17:48
*** ChanServ sets mode: +v openstackstatus17:48
mwhahahait's failing because rabbitmqctl status doesn't return expected output17:48
mwhahahabut it doesn't impact anything17:48
mwhahahaso if rabbitis installed but not up, it'll spit that out17:48
*** elGamal has joined #tripleo17:49
*** irclogbot_1 has joined #tripleo17:49
*** fultonj has joined #tripleo17:50
mwhahahaweshay|ruck: ade_lee is failing because he's getting centos7 containers on centos817:55
mwhahahawe need to better communicate how to get c8 stuff properly setup17:55
weshay|ruckmwhahaha, same setup17:55
weshay|ruckhow can I be of service today17:55
*** lmiccini has quit IRC17:55
openstackgerritMerged openstack/tripleo-heat-templates master: Drop unused remnants of the hosts-config bits  https://review.opendev.org/71458817:56
EmilienMweshay|ruck: hey, I'm working on enabling HA by default in TripleO https://review.opendev.org/#/c/359060 - and it's failing in the minor update (btw ansible-pacemaker issue is solved); when it tries to migrate rabbitmq from non-pacemaker to pacemaker managed containers17:56
mwhahahaweshay|ruck: so current-tripleo is now c8 containers right?17:56
EmilienMweshay|ruck: at this point I wonder if I should make the job passing or if I should make it non voting, land my stuff and make it voting again17:56
mwhahahaade_lee: you have centos7 running with centos8 containers, reprovision with centos817:56
EmilienMweshay|ruck: btw the jobname is tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates17:57
weshay|ruckEmilienM, updates should remain voting17:57
EmilienMyes, I know but this is a CI situation, not production17:58
weshay|ruckyes.. I'm familiar w/ the reivew17:58
weshay|ruckEmilienM, how so?17:58
EmilienMmigration from non-HA to HA isn't something we support AFIK17:58
EmilienMthe update job was designed without Pacemaker17:58
EmilienMand now I'm adding it to have better coverage (see the bug that I found with ansible-pacemaker)17:59
weshay|ruckEmilienM, so let's thread the needle.. I'll put up a patch for tht to mark it non-voting.. we'll mark a depends on the patch to that17:59
weshay|ruckand I'll put up a revert to get it back17:59
weshay|rucksound ok?17:59
EmilienMweshay|ruck: yes please17:59
weshay|ruckk.. ping you in a few17:59
mwhahahadon't be doing ci shenanigans to do this18:00
mwhahahafix it correctly18:00
weshay|ruckwhich is?18:00
EmilienMadding update tasks to migrate rabbitmq?18:00
mwhahahafigure out the teps to understandw hat it means to go non-ha to ha18:00
mwhahahaand write upgrade/update tasks18:00
mwhahahathe problem doesn't go away for upgrades18:00
*** derekh has quit IRC18:01
EmilienMgood point...18:01
EmilienMbut again, just for CI18:01
mwhahahait doesn't go away18:01
EmilienMnon-HA isn't something we support18:01
mwhahahafix it correctly18:01
*** jcapitao has quit IRC18:03
mwhahahaEmilienM: and non-ha is something we support in the undercloud18:07
mwhahahaEmilienM: which is exactly why this needs to be solved correctly18:07
openstackgerritMerged openstack/tripleo-ansible master: Add baremetal node providing module  https://review.opendev.org/71230318:08
*** xek_ has joined #tripleo18:08
EmilienMmwhahaha: this patch doesn't affect undercloud, we don't enable pacemaker18:09
EmilienMbut I'll solve it correctly18:10
*** xek has quit IRC18:11
*** jpena is now known as jpena|off18:12
fultonjCan anyone tell me how config-download implements --overcloud-ssh-key and --overcloud-ssh-user ? (as passed to 'openstack overcloud deploy')18:13
fultonjI'm tracing my way through the code, but maybe someone can save me some time?18:13
fultonjDoes it end up just setting ANSIBLE_PRIVATE_KEY_FILE before it executes ansible?18:14
fultonjekultails cloudnull ^ ?18:14
mwhahahafultonj: it's added to the host via enable_ssh_admin18:15
fultonjmwhahaha: i thought that's how the private key is installed on the host18:16
mwhahahabut that's only if not --config-download18:16
fultonji'm asking how ansible knows to use what key/user when it executes18:17
mwhahahaif you use --config-download, i think it's assuemd you've set up the keys correctly and we only specifyt eh user18:17
mwhahahayou specify it in ansible.cfg?18:17
mwhahahaor env vars?18:17
mwhahahaaka the normal way ansible knows these things?18:17
mwhahahaor you can pass cli params18:17
mwhahahaso i guess the question is whenyou say how does ansible know, who is executing ansible?18:17
fultonji know how to do all of that with ansible, that's not what i'm after18:18
fultonjand i'm not talking about --stack-only scenarios where the user runs config-download manually18:18
mwhahahaconfig-download is overloaded18:18
mwhahahaok so "in a normal deployment, how does..."18:18
ekultailsfultonj: TripleO creates it's own internal key-pair that gets setup for the `tripleo-admin` user on the Overcloud nodes.18:18
fultonjbut then...18:19
mwhahahawhen folks say config-download, i think --stack-only18:19
ekultailsThat --overcloud-ssh-{user,key} is only used for the initial connection to setup that tripleo-admin user and key it with the internal key-pair.18:19
*** iurygregory has joined #tripleo18:19
fultonjwhen mistral executes ansible18:19
mwhahahamistral uses tripleo-admin18:19
mwhahahaso liek ekultails said, we use it to enable_ssh_admin18:19
*** cylopez has quit IRC18:20
ekultails^ Yeap18:20
fultonjsorry... i'm having trouble expressing my question18:20
ekultailsfultonj: No worries. What exactly are you trying to do / figure out? Do you need to run the deployment as a different user?18:21
fultonjmy last comment in ^18:21
mwhahahaso in this case, ceph is having problems with the ssh key but that's an external deploy tasks so it should be inheriting the existing tripleo-admin bits18:22
mwhahahaseems like something else is broken18:22
mwhahahai think18:23
mwhahahaseems like an issue with the execution of ceph ansible as include in the deployment18:23
* mwhahaha steps away18:23
fultonjwell, yes. _that_ is what i'm trying to fix and thus my question18:24
fultonjmy initital thought was...18:25
fultonjIf `openstack overcloud deploy` passes --overcloud-ssh-key and --overcloud-ssh-user, then pass those the corresponding options and values to the ceph-ansible execution.18:25
ekultailsIt should be assumed that the you will always connect as the `tripleo-admin` user for deployment tasks. The --overcloud-ssh-{user,key} is only used very temporarily used before the deployment. It's never used again for updates/upgrades/anything. The tripleo-admin key-pair used to be stored in Mistral's database (in all non-master branches). In master I believe we have it stored on the file system somewhere.18:26
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add dynamic concurrency values  https://review.opendev.org/71473518:26
fultonjekultails: yes, i think that's the first think i wasn't understanding18:27
fultonjThe params --overcloud-ssh-key and --overcloud-ssh-user are used to install an ssh for future runs of config-download18:27
fultonjThe params --overcloud-ssh-key and --overcloud-ssh-user are used to install an ssh key for a user for future runs of config-download, and not used for each run18:28
fultonjso i don't want to look for those params...18:28
ekultailsYeap, not exactly what you want.18:28
fultonjhowever, there's something in the config-download run i want to pass to the ceph-ansible run18:28
fultonji can see this in chandan's job18:28
fultonjexport ANSIBLE_PRIVATE_KEY_FILE="/home/zuul/.ssh/id_rsa_tripleo"18:29
fultonjso i should just be able to use ANSIBLE_PRIVATE_KEY_FILE in my job18:30
fultonjnot job18:30
fultonjin my tripleo-ansible/ceph* patch18:30
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Dynamically adjust the retires value to fit the deployment  https://review.opendev.org/71473618:31
fultonjhttps://review.opendev.org/#/c/708252/ was too many vars18:31
*** amoralej is now known as amoralej|off18:31
ekultailsYou can probably assume the key is always at $HOME/.ssh/id_rsa_tripleo18:32
ekultailsAnd setting it via ANSIBLE_PRIVATE_KEY_FILE should work, yeah.18:32
fultonjekultails: ^18:33
fultonjekultails: my experience has been that config-download puts the key in its {{ playbook_dir }} and calls it ssh_private_key18:35
*** cdearborn has joined #tripleo18:35
fultonjI can expand thoser tasks to look for $HOME/.ssh/id_rsa_tripleo18:35
fultonjekultails: are you aware of any changes comeing for {{ playbook_dir }}/ssh_private_key ?18:36
cloudnullfultonj - https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_ceph_work_dir/tasks/get_ssh_private_key.yml#L21 - would be /var/lib/mistral/$CLOUD/ssh_private_key which should exist however, with the removal of the mistral get / store key  workflows that location should now be considered unreliable.18:37
cloudnull$HOME/.ssh/id_rsa_tripleo would be better IMO18:37
ekultailsfultonj: So I think $HOME/.ssh/id_rsa_tripleo is where the key lives in master. We definitely used to have /var/lib/mistral/overcloud/ssh_private_key and probably still use it (and just copy from ~/.ssh/id_rsa_tripleo).18:38
fultonjcloudnull ekultails right :)18:38
fultonjok, so that used to work, might still work for a little while more, but i should update to the new way18:38
*** saneax has quit IRC18:39
fultonjekultails cloudnull i'll update the path to use both but let $HOME/.ssh/id_rsa_tripleo win if both are set18:39
fultonjeventually i could remove the /var/lib/mistral/$CLOUD/ssh_private_key but supporting both should be safe for train backport too18:40
fultonjekultails cloudnull i'll add you both to the review since you're now familiar with what i'm trying to do. thanks for the advice18:40
Madkisshi everybody. quick Q: When I want to have networks in network_data.yaml for TripleO collapse, how do I do that? I want to use the same untagged vlan for Ctrlplane, tenant, storage-mgmt and internalapi.18:50
fultonjMadkiss: if you deploy without network isoloation all networks will run on the provisioning network.18:51
fultonjMadkiss: but you're trying to convert an existing deloyment by flattening, then i don't know how to do that18:51
*** jcoufal has quit IRC18:52
Madkissfultonj: no no, I deploy a new cloud, I kind of need a mixture though18:52
*** dpawlik has quit IRC18:53
MadkissThe basic design looks like this: All machines have two Mellanox Connect-X5 cards with two times 25G (ens1f0/ens1f1 and ens7f0/ens7f1). I want all machines to have bond0 out of ens1sf0/ens1f1 untagged in port 20, which is the management, storage-management, internal api and tenant transport layer.18:58
Madkisson a subset of nodes, i want ens7f0/ens7f1 to be bond1 and be present in three distinct VLANs (those are the network nodes). and on another subset of nodes, i want the same bond1-based bond to be present untagged in yet another vlan, which is the storag enetwork.18:59
Madkissi am looking at the documentation, but there are so many options and possibilities that I kind of don't see where to begin.18:59
EmilienMMadkiss: long time no see18:59
*** hberaud has quit IRC19:01
Madkisshello EmilienM, indeed :-)19:02
*** hberaud has joined #tripleo19:04
MadkissI am unsure as to whether when I specifiy the same ip rangds for the networks in network_data.yaml, will tripleo recognize that? i do not want a host to have multiple ips on the same interface in the same network.19:04
*** alexmcleod has quit IRC19:07
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add dynamic concurrency values  https://review.opendev.org/71473519:07
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Dynamically adjust the retires value to fit the deployment  https://review.opendev.org/71473619:08
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Add dynamic concurrency values  https://review.opendev.org/71473519:09
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Dynamically adjust the retires value to fit the deployment  https://review.opendev.org/71473619:09
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Dynamically adjust the retires value to fit the deployment  https://review.opendev.org/71473619:10
*** holser__ has joined #tripleo19:17
*** holser has quit IRC19:18
ekultailsHey :-)19:19
stevebakerweshay|ruck, rlandy: I have modified and uploaded a upstream-centos-8-1584907370 which has my key injected in it, and it works!19:20
rlandystevebaker: k - good news - we uploaded a new image to the openstack-nodepool with the same - however if your image works, that's fine19:21
rlandystevebaker: we're in a bit of flux today with the rdocloud/vexxhost move19:22
stevebakerrlandy: good luck!19:22
*** matbu has quit IRC19:23
*** holser has joined #tripleo19:32
*** holser__ has quit IRC19:34
weshay|ruckstevebaker, yay!19:46
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: standalone/overcloud: enable the HA deployment by default  https://review.opendev.org/35906019:50
*** rcernin|brb has quit IRC19:57
*** jfrancoa has quit IRC20:02
openstackgerritJohn Fulton proposed openstack/tripleo-ansible master: Make ceph-ansible run handle missing "{{ playbook_dir }}/ssh_key"  https://review.opendev.org/71476820:14
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Deprecate Keepalived service  https://review.opendev.org/65706720:16
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Normalize how we run validations  https://review.opendev.org/71476920:19
openstackgerritKevin Carter (cloudnull) proposed openstack/tripleo-ansible master: Normalize how we run validations  https://review.opendev.org/71476920:20
openstackgerritMerged openstack/tripleo-ansible stable/train: Insert hosts entries to the beginning of the file.  https://review.opendev.org/70738020:23
*** abishop has quit IRC20:37
*** abishop_ has joined #tripleo20:37
*** hjensas_ is now known as hjensas20:39
*** TrevorV has quit IRC20:47
*** asbishop has joined #tripleo20:49
*** abishop_ has quit IRC20:51
*** jmasud has quit IRC20:54
*** jmasud has joined #tripleo20:56
openstackgerritFlavio Fernandes proposed openstack/tripleo-heat-templates master: Add Octavia OVN Provider configuration  https://review.opendev.org/70769520:57
*** raildo_ has joined #tripleo20:59
*** raildo has quit IRC21:01
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud delete  https://review.opendev.org/71477621:02
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Add overcloud delete  https://review.opendev.org/71477621:04
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add centos/rhel based interface/bridge commands  https://review.opendev.org/71420821:08
*** rfolco has quit IRC21:12
EmilienMweshay|ruck: would be nice to get https://review.opendev.org/#/c/705775/21:16
weshay|ruckah cool21:17
*** cjloader has quit IRC21:18
*** holser has quit IRC21:18
*** holser has joined #tripleo21:19
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP: Remove connections added by Network Manager  https://review.opendev.org/71477721:19
*** abishop_ has joined #tripleo21:26
*** dciabrin has joined #tripleo21:26
cloudnullanyone around mind giving https://review.opendev.org/#/q/topic:bug/1868619+(status:open+OR+status:merged) a review w21:27
*** asbishop has quit IRC21:28
openstackgerritMerged openstack/tripleo-operator-ansible master: Fix readme issues  https://review.opendev.org/71471021:30
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP: Remove connections added by Network Manager  https://review.opendev.org/71477721:31
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: Fix population of username, public keys  https://review.opendev.org/71477821:32
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Switch to ansible for overcloud node (un)provision  https://review.opendev.org/70874921:33
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: overcloud deploy, use ansible for bm provision  https://review.opendev.org/71222121:33
*** holser has quit IRC21:36
Madkisswhere can I see what Ironic and Heat consider "nic1", "nic2" etc for a system?21:37
cloudnullweshay|ruck mwhahaha did either of you ever figure out what was going on with build images and cent8?21:37
openstackgerritSteve Baker proposed openstack/tripleo-ansible master: Fix population of username, public keys  https://review.opendev.org/71477821:37
cloudnullok :'(21:38
mwhahaharefresh my memory, what was wrong again?21:38
mwhahahabesides everything21:39
cloudnullits just that21:39
cloudnullwhen building images - https://pasted.tech/pastes/b8bf1466a4f64d2cb2eda4cde05e61ea2e09e95b21:39
cloudnullit seems to be failing in this new way ?21:39
mwhahahaodd let me give it a shot21:40
cloudnullthough this is the first ive tried it in a week or so21:40
cloudnullthis is the complete log - https://pasted.tech/pastes/37fc0b85854200c7ae29711c261cd5d4fc17c33e.raw21:41
cloudnullwhich I didn't find helpful21:41
*** abishop_ has quit IRC21:41
mwhahahayou're missing the HA repos21:41
mwhahaha2020-03-24 21:32:03.084 | No match for argument: pacemaker-remote21:41
*** abishop has joined #tripleo21:41
cloudnulli still see "Error: Unable to find a match: pacemaker-remote pacemaker pcs" though idk why ?21:42
cloudnulloh, hum... which repos has that ?21:42
mwhahahaCentOS-HA.repo on my system21:42
mwhahahaor we have a tripleo-centos-highavailability that tripleo-repos provides21:42
mwhahahaif you have uses a recent version of tripleo-repos it should configure it for you on centos821:43
mwhahahaif you're on a centos7 env trying to build a centos8, then you're missing it21:43
mwhahaha2020-03-24 21:32:02.534 | repo: using cache for: centos7-rt21:43
mwhahahaseems odd21:43
*** xek_ has quit IRC21:43
cloudnullim on cent8 , that repo wasn't enabled. will enable it and try again21:43
cloudnulldib should default to building cent8 on cent8 right ?21:44
mwhahahaprobably not but who knows21:44
mwhahahadepends on the cli params you're using21:44
mwhahahawe need to update it to default to centos8 now21:44
cloudnullmeh , running the things and will report back :D21:44
*** raildo_ has quit IRC21:45
cloudnullIm running `openstack --os-cloud undercloud overcloud image build --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-centos8.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-python3.yaml`21:45
cloudnullthough i vaguely remember us talking about py3 not being needed?21:45
mwhahahai usually flip the order21:46
mwhahahabecause overcloud-images.yaml is py2 by default21:46
mwhahahaso we have to specify overcloud-images-python3.yaml21:46
mwhahahato get the python3 opnes21:46
openstackgerritSteve Baker proposed openstack/tripleo-quickstart-extras master: Exit overcloud-deploy.sh on provision failure  https://review.opendev.org/71477921:46
Madkissdo OVS bridges in Ironic support LACP by now without dpdk?21:48
mwhahahahttps://access.redhat.com/solutions/3417681 seems to indicate yes, though not certain how to express it for os-net-config21:51
MadkissBondInterfaceOvsOptions seems to be the place for that.21:51
*** jcoufal has joined #tripleo21:53
*** jcoufal has quit IRC21:54
ade_leemwhahaha, have you tried --  export DIB_YUM_REPO_CONF="/etc/yum.repos.d/delorean*"21:56
ade_lee      openstack overcloud image build --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-centos8.yaml21:56
mwhahahayea you need to also specify the python3 image file21:57
ade_leefails for me with "Problem: The operation would result in removing the following protected packages: kernel-core"21:57
mwhahahaade_lee: hjensas hit that, if the system that your building on has an older kernel21:57
mwhahahaade_lee: update the local system to latest and reboot21:57
mwhahaha(as you should always do first thing before installing the undercloud)21:58
* hjensas needs to update the playbook to reboot ... and we probably want to merge https://review.opendev.org/713988 ?21:59
mwhahahai'm not aware of anyone from tripleo who merges stuff in dib22:00
mwhahahamaybe nudge ian22:00
* mwhahaha pokes ianw22:00
ianwumm, i'm happy with that change22:01
*** abishop_ has joined #tripleo22:02
*** suuuper has quit IRC22:05
*** abishop has quit IRC22:05
Madkissahum the controlplane that i define in the undercloud -- is that the same one as the former Management network in the overcloud?22:09
mwhahahaMadkiss: it's the provisioning network, a management network is usually different. Though if you consider the provisioning network as maangement (because the undercloud manages the overcloud nodes via it) then yes22:10
Madkissthing is, I only have one network where all nodes can have their ssh-login-interface (which is equivalent for me to the management network) in.22:12
mwhahahathen it'll need to be that network22:12
Madkissi have a ctlplane-subnet-entry in undercloud.conf and now I wonder whether I need to define the Management network in network_data.yaml anyway, or whether there are predefined Ctlplane variables22:12
mwhahahathough not certain what network compute uses22:12
mwhahahaso not completely sure that'll work22:12
Madkisshu? nova-compute requiring its own network would be a new one.22:13
mwhahahalet me check what network it uses for live migration by default22:13
weshay|ruckcloudnull, what's busted? http://zuul.openstack.org/builds?job_name=tripleo-buildimage-overcloud-full-centos-822:14
weshay|ruckthis review? https://review.opendev.org/#/c/714629/22:15
weshay|ruckcloudnull, show me what review and I'll look later, have to take my kids to the park22:16
mwhahahaMadkiss: so the libvirt live migration service uses the internal api or storage network by default (which also has an additional ssh). not certain if that conflicts with your bits because it's not the usual login ssh22:16
cloudnullweshay|ruck i didn't have a review in mind. I was attempting to build images locally, which seems to be busted.22:17
mwhahahaall these things are defined int he service net map so you can customize if you needed to22:17
Madkissinternal api and storage are both separate networks I define and that will be present.22:17
mwhahaharight but you said you had teh ssh limitation22:17
mwhahahais that for managing the network externally?22:17
mwhahahaif so you can define a management network for that ssh seperate from the ctlplane22:17
mwhahahaif the undercloud can ssh to the nodes over the ctlplane still22:18
Madkissoh, no, sorry, I misexpressed myself. I started with OpenStack in 2011, and back then, the "management network" simply was the main network where all traffic was going except for storage traffic ...22:18
mwhahahayou can move the traffic around to whatever networks you want22:18
mwhahahathe ctlplane is usually just the provisioning/monitoring bits22:18
mwhahahayou can then specify additional networks (internal_api/storage/etc) and move services to them22:19
mwhahahait's controlled via the service net map in tripleo22:19
mwhahahayou can see the defaults in THT/network/service_net_map.j2.yaml22:20
cloudnullweshay|ruck this is from a recent run - https://pasted.tech/pastes/ae10727972980eefb1f6bfb39b4da2a42c80a637.raw - trying to build local images22:20
*** gregwork has quit IRC22:21
cloudnullweshay|ruck this is the stacktrace - https://pasted.tech/pastes/5dfb26974f02102764438170ac462c1edba1016d22:21
Madkissmwhahaha: thank you -- perfect!22:21
cloudnullmwhahaha I tried re-running tripleo-repos to make sure all my repos were right, and I reversed the command args `openstack --os-cloud undercloud overcloud image build --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-python3.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-centos8.yaml`22:22
cloudnullsadly same result22:22
mwhahahak let me retry22:22
cloudnulldid you happen to have better luck?22:22
*** slaweq has quit IRC22:23
mwhahahayour repos are still bad22:23
mwhahahayou can seem them in  your log22:24
mwhahahayou forgot to export the repos bits22:24
mwhahahathe botom of https://pasted.tech/pastes/ae10727972980eefb1f6bfb39b4da2a42c80a637.raw shows you only have appstream/baseos/extras/centos7-rt22:24
mwhahahaexport DIB_YUM_REPO_CONF="/etc/yum.repos.d/*"22:24
mwhahahaand rerun22:25
Madkissone other question: documentation says "ctlplane interface is always nic1", but i want the ctlplane to be on a bond from ens1f0 and ens1f1. can I redefine that somewhere?22:25
mwhahahaMadkiss: you'd define that in your network config for the host22:25
mwhahahai cannot help you with network nic configs22:25
* mwhahaha avoids them like the plague22:25
Madkissmwhahaha: you mean the nic configs for the individual roles?22:26
Madkissgotcha, thank you22:26
* mwhahaha handwaves https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/network_isolation.html22:26
Madkisssome light at the end of the tunnel after i had to find out that my beloved director UI has died.22:26
ade_leemwhahaha, so my image build completed ..22:27
cloudnullmwhahaha running nw22:27
*** rh-jelabarre has quit IRC22:27
ade_leemwhahaha, specifically --  openstack overcloud image build --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-centos8.yaml22:28
mwhahahaif only there was some sort of ansible collection that wrapped all this stuff for you22:28
mwhahahaand had an example playbook22:28
* mwhahaha coughs22:28
mwhahahaline 55 needs tweaking tho22:28
mwhahahacause it needs python322:29
ade_leemwhahaha, cool - I22:30
ade_leeI'll try it -- I was following along what hjensas was doing ..22:30
mwhahahai got the missing pacemaker bits cause it didn't find my ha repo either22:30
mwhahahabecause i didn't specify /etc/yum.repo.d/*22:31
ade_leeI tried -- openstack overcloud image upload though -- and get the error -- Required file "./ironic-python-agent.initramfs" does not exist.22:31
mwhahahaif you specify only /etc/yum.repo.d/delorean* you get missing ha repos22:31
ade_leeah ..22:32
mwhahahathat means you only build the overcloud image and not the ipa22:32
ade_leewell actually it created ironic-python-agent.qcow222:32
mwhahahait shouldn't create a qcow2 for that22:33
mwhahahai've got ironic-python-agent.{initramfs,kernel,vmlinuz}22:33
ade_leehmm .. and yet it did ..22:34
ade_leeronic-python-agent.d  ironic-python-agent.log  ironic-python-agent.qcow2  overcloud-full.d  overcloud-full.log  overcloud-full.qcow222:34
* mwhahaha shrugs22:34
ade_leeis whats in my images directory22:34
mwhahahathis is one of the oldest processes we have22:35
mwhahahahasn't changed in years :D22:35
mwhahahamy images ar ebuild so we'll see22:35
mwhahahaare building22:35
*** lbragstad has quit IRC22:38
ade_leeyup - I see ..22:38
ade_lee2020-03-24 22:16:06.303 | Converting image using qemu-img convert22:38
ade_lee2020-03-24 22:17:15.633 | Image file ./ironic-python-agent.qcow2 created...22:38
ade_leeat the end of my build22:38
cloudnullhum... https://pasted.tech/pastes/0f5f928716f7245b720c162cdf4fc12512a13787.raw - it ran longer, but still seemed to fail :\22:39
ade_leemwhahaha, gotta dinner - but I'll check back in a bit -- thanks22:40
mwhahahacloudnull: 2020-03-24 22:36:00.917 |  Problem: The operation would result in removing the following protected packages: kernel-core22:40
mwhahahacloudnull: update system and reboot22:40
* mwhahaha broken record22:40
cloudnullI need to update the local system and reboot ?22:41
cloudnullis this the normal joys of DIB ?22:41
mwhahahawell you should always update your undercloud before you install after you configure the repos so you don't hit this22:41
mwhahahabut sure22:41
mwhahahaif you see teh conversation from about an hour ago with ade_lee, you can see why22:42
openstackgerritMerged openstack/diskimage-builder master: Use rpm -e instead of dnf for cleaning old kernels  https://review.opendev.org/71398822:42
mwhahahaspecifically because -^22:43
cloudnullthis was an already deployed cloud, which I was trying to build new images on just for grins.22:43
cloudnullis there a proceedure to get updated images without having to update/reboot the undercloud ?22:43
mwhahahayea it's a bug with dnf22:43
ade_leecloudnull, mwhahaha though in my case, my undercloud was already updated, but I was running an old kernel.  had to modify grub to boot into newer one.22:43
cloudnulloh ok22:43
mwhahahait popped up in fedora a long while ago but never hit centos722:44
mwhahahabecause yum vs dnf22:44
openstackLaunchpad bug 1623409 in diskimage-builder "Error: The operation would result in removing the booted kernel" [High,Fix released] - Assigned to Harald Jensås (harald-jensas)22:44
mwhahahaLucas Alvares Gomes (lucasagomes) on 2016-09-1422:44
mwhahahaso yea takes about 4 years to fix stuff22:44
*** rcernin has joined #tripleo22:45
mwhahahaalso you usually wouldn't build iamges w/o having updated your undercloud22:45
mwhahahathe process should be update undercloud to latest and then build22:46
mwhahahaespecially when dealing with upstream because dlrn repos may go away22:46
mwhahahaso you'd want the latest ones22:46
cloudnullOIC - this makes sense22:46
*** tkajinam has joined #tripleo22:53
mwhahahayea worked for me22:56
mwhahahadon't have an ipa .qcow2 either22:56
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP: Remove connections added by Network Manager  https://review.opendev.org/71477722:56
mwhahaha99-ramdisk-create should ahve generated all the ramdisk bits22:56
mwhahahaso the key is: export DIB_YUM_REPO_CONF="/etc/yum.repos.d/*"22:58
mwhahahaopenstack overcloud image build --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-python3.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-centos8.yaml22:59
mwhahahaade_lee: you might try flipping the config-file ordering to ensure python3 is specified first followed by centos822:59
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Update image URL links for CentOS 8  https://review.opendev.org/71479023:03
rlandyweshay|ruck: ^^23:03
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add centos/rhel based interface/bridge commands  https://review.opendev.org/71420823:04
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: WIP: Remove connections added by Network Manager  https://review.opendev.org/71477723:05
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Switch default images to py3  https://review.opendev.org/71479123:07
cloudnullmwhahaha thanks for the help . I learned a bit more and it works now. :D23:09
mwhahahacloudnull: k you're now the expert23:09
* mwhahaha runs away23:10
* cloudnull hold my beer 23:10
openstackgerritAlex Schultz proposed openstack/tripleo-operator-ansible master: Fix image build for python3  https://review.opendev.org/71479323:13
* mwhahaha fixes the defaults and runs away23:13
*** chem has quit IRC23:29
*** lbragstad has joined #tripleo23:46
*** mcornea has quit IRC23:47
openstackgerritMerged openstack/python-tripleoclient master: Add mistral workflow module path to ansible-runner  https://review.opendev.org/71449323:48
openstackgerritMerged openstack/tripleo-heat-templates master: DCN/Edge: Handle ipv6 address for local glance endpoint  https://review.opendev.org/71448223:48
*** rlandy has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!