Monday, 2019-10-14

*** dsneddon has quit IRC00:02
*** d0ugal has quit IRC00:05
*** cfontain has quit IRC00:06
*** d0ugal has joined #tripleo00:08
*** dsneddon has joined #tripleo00:29
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/stein: Randomize the container list for uploads  https://review.opendev.org/68678500:32
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/stein: Improve ThreadPoolExecutor usage  https://review.opendev.org/68678600:32
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/stein: Add support for uploading local image  https://review.opendev.org/67920700:33
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/stein: Always close src/dst sessions and raise to retry  https://review.opendev.org/68678700:33
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/stein: Prefix ansible output with timestamps  https://review.opendev.org/68678800:33
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/stein: Implement threading locks around layers  https://review.opendev.org/68678900:33
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/stein: Improving debug logging  https://review.opendev.org/68679200:34
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/stein: Use tenancity to backoff when layer is locked  https://review.opendev.org/68820000:34
openstackgerritEmilien Macchi proposed openstack/tripleo-common stable/stein: Make executor type dynamic  https://review.opendev.org/68820100:34
*** ayoung has quit IRC00:36
*** ayoung has joined #tripleo00:37
*** dsneddon has quit IRC00:39
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Temporarily disable Fedora 30 build jobs  https://review.opendev.org/68831500:43
*** dsneddon has joined #tripleo00:46
*** dsneddon has quit IRC00:50
*** dsneddon has joined #tripleo00:56
openstackgerritFeilong Wang proposed openstack/os-apply-config master: Avoid overwriting existing value  https://review.opendev.org/68831700:59
*** ysandeep has joined #tripleo01:14
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Convert container environment from a list to a dict  https://review.opendev.org/68729301:16
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - deploy steps without paunch  https://review.opendev.org/68765101:16
*** ayoung has quit IRC01:24
*** gbarros has quit IRC01:25
*** ayoung has joined #tripleo01:25
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP - Introduce tripleo-container-manage role  https://review.opendev.org/68619601:31
openstackgerritEmilien Macchi proposed openstack/tripleo-ansible master: WIP - Introduce tripleo-container-manage role  https://review.opendev.org/68619601:31
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - deploy steps without paunch  https://review.opendev.org/68765101:32
*** ayoung has quit IRC01:35
*** ayoung has joined #tripleo01:37
*** ayoung has quit IRC01:44
*** ayoung has joined #tripleo01:45
*** gbarros has joined #tripleo02:02
*** cfontain has joined #tripleo02:03
openstackgerritwes hayutin proposed openstack/tripleo-ci master: Add train jobs  https://review.opendev.org/68700402:12
openstackgerritwes hayutin proposed openstack/tripleo-ci master: Add train jobs  https://review.opendev.org/68700402:12
*** ayoung has quit IRC02:14
*** ayoung has joined #tripleo02:17
*** ramishra has joined #tripleo02:19
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: check both /etc/ and home for clouds.yaml  https://review.opendev.org/68814802:23
*** aakarsh|2 has quit IRC02:26
*** cfontain has quit IRC02:36
*** gbarros has quit IRC02:58
HardCaseAll, trying a keystone credential fix and when I mark resources unhealthy I get this message twice.  Could this be why my deployment is hanging and if so does anyone know how to fix it?03:11
HardCaseheat resource-list -n 5 overcloud |awk '$6 == "OS::Heat::SoftwareDeployment" || $6 == "OS::Heat::StructuredDeployment" || $6 == "OS::TripleO::SoftwareDeployment"{print "heat resource-mark-unhealthy", $12, $2}'   |bash03:11
HardCaseSorry wrong paste03:11
HardCaseERROR: Stack overcloud-AllNodesDeploySteps-v7kq2u5lkpt6-ControllerDeployment_Step2-hkagczx74x3m already has an action (UPDATE) in progress.03:11
*** ayoung has quit IRC03:14
openstackgerritMerged openstack/diskimage-builder master: Temporarily disable Fedora 30 build jobs  https://review.opendev.org/68831503:17
*** gbarros has joined #tripleo03:26
openstackgerritIan Wienand proposed openstack/diskimage-builder master: centos7; use numeric DIB_RELEASE  https://review.opendev.org/68832103:35
*** psachin has joined #tripleo03:36
*** gbarros has quit IRC03:39
*** gbarros has joined #tripleo03:42
*** dking_desktop has quit IRC03:45
*** gbarros has quit IRC03:47
*** ykarel has joined #tripleo03:55
*** dking_desktop has joined #tripleo03:58
*** gbarros has joined #tripleo04:00
*** gbarros has quit IRC04:05
*** udesale has joined #tripleo04:07
*** ricolin has joined #tripleo04:12
*** gbarros has joined #tripleo04:14
*** ricolin_ has joined #tripleo04:15
*** ricolin has quit IRC04:19
*** soniya29 has joined #tripleo04:33
*** cfontain has joined #tripleo04:33
*** dking_desktop has quit IRC04:45
*** dking_desktop has joined #tripleo04:51
openstackgerritMerged openstack/diskimage-builder master: Remove RA solicit delay  https://review.opendev.org/68821804:52
*** sanjayu_ has joined #tripleo04:53
*** surpatil has joined #tripleo05:04
*** ykarel has quit IRC05:04
*** cfontain has quit IRC05:06
*** ysandeep has quit IRC05:07
*** waleedm has joined #tripleo05:19
*** waleedm_ has joined #tripleo05:20
*** waleedm has quit IRC05:23
*** ykarel has joined #tripleo05:24
*** lmiccini has joined #tripleo05:27
*** jbadiapa has joined #tripleo05:36
*** marios has joined #tripleo05:36
*** marios is now known as marios|rover05:36
*** HardCase has quit IRC05:36
*** kopecmartin|off is now known as kopecmartin05:59
*** cfontain has joined #tripleo06:02
*** ksambor has joined #tripleo06:06
*** ccamacho has joined #tripleo06:10
openstackgerritMerged openstack/tripleo-common stable/stein: Randomize the container list for uploads  https://review.opendev.org/68678506:16
*** holser has joined #tripleo06:18
*** cfontain has quit IRC06:36
*** mschuppert has joined #tripleo06:39
openstackgerritMerged openstack/tripleo-common stable/stein: Improve ThreadPoolExecutor usage  https://review.opendev.org/68678606:40
openstackgerritMerged openstack/tripleo-common stable/stein: Add support for uploading local image  https://review.opendev.org/67920706:40
*** bogdando has joined #tripleo06:42
*** pcaruana has joined #tripleo06:43
*** epoojad1 has joined #tripleo06:54
*** apetrich has joined #tripleo07:04
*** slaweq has joined #tripleo07:04
*** tesseract has joined #tripleo07:04
*** rcernin has quit IRC07:11
*** raukadah is now known as chandankumar07:11
*** amoralej|off is now known as amoralej07:20
*** paramite has joined #tripleo07:25
*** jfrancoa has joined #tripleo07:30
*** hamdyk has joined #tripleo07:31
*** apetrich has quit IRC07:32
*** hamdykhader has joined #tripleo07:32
*** lucasagomes has joined #tripleo07:34
*** hamdyk has quit IRC07:36
*** hberaud|gone is now known as hberaud07:37
*** gfidente has joined #tripleo07:37
*** rpittau|afk is now known as rpittau07:40
*** jpena|off is now known as jpena07:46
*** ykarel is now known as ykarel|lunch07:48
*** jpich has joined #tripleo07:56
*** iurygregory has joined #tripleo07:58
*** gfidente has quit IRC07:59
*** lmiccini has quit IRC08:10
*** fmount has quit IRC08:14
*** lmiccini has joined #tripleo08:17
*** hamdyk has joined #tripleo08:19
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates stable/stein: Ensure nova-api is running before starting nova-compute containers  https://review.opendev.org/68834908:21
*** hamdykhader has quit IRC08:23
*** alexmcleod has joined #tripleo08:25
*** hamdykhader has joined #tripleo08:28
*** hamdyk has quit IRC08:30
*** hberaud has quit IRC08:31
*** fmount has joined #tripleo08:33
*** ykarel|lunch is now known as ykarel08:33
*** dtantsur|afk is now known as dtantsur08:34
*** hberaud has joined #tripleo08:34
openstackgerritHarald Jensås proposed openstack/python-tripleoclient master: Undercloud hosts entries in overcloud parameters  https://review.opendev.org/68734708:36
openstackgerritHarald Jensås proposed openstack/python-tripleoclient master: Use name for container registry  https://review.opendev.org/68738808:37
*** derekh has joined #tripleo08:38
*** gbarros has quit IRC08:38
*** ricolin_ is now known as ricolin08:41
*** elGamal has quit IRC08:44
*** lmiccini has quit IRC08:45
*** elGamal has joined #tripleo08:45
*** chem has joined #tripleo08:47
*** xek_ has joined #tripleo08:47
*** gfidente has joined #tripleo08:51
*** lmiccini has joined #tripleo08:57
*** bogdando has quit IRC08:59
*** bogdando has joined #tripleo09:00
*** sri_ has joined #tripleo09:07
openstackgerritHamdy Khader proposed openstack/tripleo-heat-templates master: Add compute role "ComputeSriovIB" for infiniband deployments  https://review.opendev.org/68835809:12
*** holser has quit IRC09:13
*** holser has joined #tripleo09:14
openstackgerritHamdy Khader proposed openstack/tripleo-heat-templates master: Add compute role "ComputeSriovIB" for infiniband deployments  https://review.opendev.org/68835809:15
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: Remove OpenDaylight templates and environments  https://review.opendev.org/68263009:20
*** skramaja has joined #tripleo09:20
*** apetrich has joined #tripleo09:39
*** ricolin has quit IRC09:41
*** pkopec has joined #tripleo09:43
*** hamdyk has joined #tripleo09:46
*** hberaud is now known as hberaud|lunch09:46
openstackgerritNatal Ngétal proposed openstack/python-tripleoclient master: Add test for the function check_deprecated_parameters.  https://review.opendev.org/67453109:47
*** hamdykhader has quit IRC09:48
*** hamdykhader has joined #tripleo09:50
*** hamdyk has quit IRC09:52
*** hamdykhader has quit IRC09:56
*** bfournie has joined #tripleo10:02
*** waleedm_ has quit IRC10:06
*** soniya29 has quit IRC10:15
*** sri_ has quit IRC10:17
*** surpatil has quit IRC10:25
*** surpatil has joined #tripleo10:25
openstackgerritzhulingjie proposed openstack/tripleo-quickstart-extras master: Sync Sphinx requirement  https://review.opendev.org/66291910:27
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: [reproducer]Set registry_login_enabled to false  https://review.opendev.org/68837710:29
*** pbandark has joined #tripleo10:31
openstackgerritChandan Kumar (raukadah) proposed openstack/tripleo-quickstart-extras master: [reproducer]Set registry_login_enabled to false  https://review.opendev.org/68837710:31
*** surpatil has quit IRC10:44
*** surpatil has joined #tripleo10:46
*** dsneddon has quit IRC10:48
*** mburned_out is now known as mburned10:50
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Use dynamic argument_spec from documentation  https://review.opendev.org/68812710:51
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: [WIP] Fix Ironic Authentication exception  https://review.opendev.org/68838310:51
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Hide password from Ansible run and logging  https://review.opendev.org/68838410:51
*** hamdyk has joined #tripleo10:53
*** waleedm has joined #tripleo10:58
*** hberaud|lunch is now known as hberaud11:12
*** udesale has quit IRC11:13
*** chem has quit IRC11:15
*** dsneddon has joined #tripleo11:19
*** hamdykhader has joined #tripleo11:25
*** sanjayu_ has quit IRC11:25
*** suuuper has joined #tripleo11:26
*** hamdyk has quit IRC11:28
*** chem has joined #tripleo11:29
*** jpena is now known as jpena|lunch11:31
*** epoojad1 has quit IRC11:32
*** pcaruana has quit IRC11:34
*** hamdyk has joined #tripleo11:34
*** hamdykhader has quit IRC11:36
*** waleedm has quit IRC11:38
*** pcaruana has joined #tripleo11:41
openstackgerrityatin proposed openstack/tripleo-ci master: Set rhcloud provider settings for reproducer  https://review.opendev.org/68531611:46
*** hamdykhader has joined #tripleo11:46
*** hamdyk has quit IRC11:49
*** hamdyk has joined #tripleo11:49
*** hamdykhader has quit IRC11:52
openstackgerritSaravanan KR proposed openstack/tripleo-heat-templates master: Move KernelArgs and OvS-DPDK deployment to ansible role  https://review.opendev.org/67772811:53
*** hjensas has quit IRC11:54
*** surpatil has quit IRC11:56
*** surpatil has joined #tripleo11:57
*** sri_ has joined #tripleo11:57
sri_cgoncalves, Hi quick question, when we enable octavia service I keep the amphora images in /usr/share/openstack-octavia-amphora-images/ but it didn't upload the image to glance12:03
sri_cgoncalves, http://paste.openstack.org/show/783608/12:03
*** rh-jelabarre has joined #tripleo12:05
*** amoralej is now known as amoralej|lunch12:06
sri_cgoncalves, is there anything i am missing ? maybe because of that my octavia dashboard is not working properly ! I am not sure is there any relation between those two12:06
*** dsneddon has quit IRC12:08
*** rfolco has joined #tripleo12:08
*** hamdykhader has joined #tripleo12:10
*** morazi has joined #tripleo12:11
*** rfolco is now known as rfolco|ruck12:12
*** hamdyk has quit IRC12:13
*** jroll has quit IRC12:14
cgoncalvessri_, could you please remind me on which tripleo version are you deploying?12:14
sri_cgoncalves, stable/rocky12:14
*** jroll has joined #tripleo12:15
cgoncalvessigh...12:16
cgoncalveslet me check12:16
cgoncalvessri_, try this. either a) rename amphora-x64-haproxy-centos.qcow2 to amphora-x64-haproxy.qcow2 or b) set OctaviaAmphoraImageFilename to /usr/share/openstack-octavia-amphora-images/amphora-x64-haproxy-centos.qcow212:18
*** hamdyk has joined #tripleo12:20
sri_cgoncalves, ack, is that the reason why  when i open the  load balancers tab it's getting  keep stuck at page reloading ?12:23
cgoncalvessri_, no12:23
sri_cgoncalves, I've uploaed the image via service account12:23
*** hamdykhader has quit IRC12:24
cgoncalvesthat is also an option, sure12:24
sri_cgoncalves, it's keep reloading at this page https://pasteboard.co/IBtAwl1.png12:24
*** devep has joined #tripleo12:25
cgoncalvessri_, right, same issue as last week. have you looked at the logs in the controller as well as browser?12:25
sri_cgoncalves, i didn't find the any error messages12:26
sri_cgoncalves,  It's very hard to read logs in the browser because it's keep reloading the page, I didn't find any error messages in http/horizon I've checked all the logs files (error_log  error_log.1  horizon_access.log  horizon_access.log.1  horizon_error.log  horizon_error.log.1)12:27
cgoncalvessri_, check what the browser is trying to load. there should be some HTTP 4xx or HTTP 5xx errors12:27
*** jpena|lunch is now known as jpena12:27
sri_cgoncalves, let me check12:27
cgoncalvessri_, does the command "openstack loadbalancer list" exits without errors?12:28
*** ekultails has joined #tripleo12:28
*** hamdyk has quit IRC12:28
sri_cgoncalves, Yes , i am able to create loadbalancer without any issues12:28
cgoncalvesok, that's good. I wanted us to check that the octavia api service was functional12:29
*** dsneddon has joined #tripleo12:31
*** hamdyk has joined #tripleo12:35
openstackgerritSlawek Kaplonski proposed openstack/tripleo-quickstart-extras master: Add playbook and role to run tobiko  https://review.opendev.org/65542312:35
*** dsneddon has quit IRC12:37
EmilienMykarel: thanks12:37
*** soniya29 has joined #tripleo12:38
sri_cgoncalves, screen shorts  https://ibb.co/bvJvM09, https://ibb.co/mGqFYpX, https://ibb.co/bvJvM0912:38
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Convert container environment from a list to a dict  https://review.opendev.org/68729312:39
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Do not repeat setting defaults for image layers  https://review.opendev.org/68839812:41
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates master: Fix placement_wait_for_service indent  https://review.opendev.org/68839912:44
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates stable/stein: Ensure nova-api is running before starting nova-compute containers  https://review.opendev.org/68834912:45
sri_cgoncalves, fyi I didn't run the openstack overcloud container image prepare -e ~/new-heat-templates/environments/services/octavia.yaml while i am deploying12:49
sri_cgoncalves, if nobody else have this issue can I assume it's because of manual image upload,12:50
cgoncalvessri_, no. two different problems12:51
chandankumargfidente: Hello12:52
gfidentechandankumar hey12:52
gfidenteI know12:52
gfidentethe ceph-ansible ci thread12:52
chandankumargfidente: We will be including ceph-ansible gating in this sprint12:52
gfidente++12:52
gfidenteI think this is awesome12:52
chandankumargfidente: please have a look on this https://hackmd.io/nnoFQSm7T-mDwa3wSl7zKg12:52
sri_cgoncalves, hmmm12:52
gfidenteI was trying to be more helpful12:53
gfidenteother than just giving ++12:53
gfidentebut thanks!12:53
gfidenteand sure I will12:53
chandankumargfidente: great, thanks!12:55
*** amoralej|lunch is now known as amoralej12:58
*** Guest31368 has joined #tripleo12:58
*** cdearborn has joined #tripleo12:58
*** hjensas has joined #tripleo13:00
*** Guest31368 is now known as redrobot_13:00
*** redrobot_ is now known as redrobot13:01
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart master: Update centos base_image to CentOS-7-x86_64-GenericCloud-1907.qcow2  https://review.opendev.org/68840113:02
sri_cgoncalves, 'enable_lb': False, in /var/lib/config-data/puppet-generated/horizon/etc/openstack-dashboard/local_settings13:03
sri_is that suppose to be true ?13:04
sri_cgoncalves, http://paste.openstack.org/show/783674/13:04
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Track uploaded layers for workers as global view  https://review.opendev.org/68728813:04
*** surpatil has quit IRC13:05
cgoncalvessri_, no. that setting is to enable the neutron-lbaas dashboard13:07
*** dsneddon has joined #tripleo13:08
sri_cgoncalves, ack13:14
*** openstackstatus has quit IRC13:15
*** hamdykhader has joined #tripleo13:16
*** sanjayu_ has joined #tripleo13:17
*** hamdyk has quit IRC13:19
*** slaweq_ has joined #tripleo13:21
*** slaweq has quit IRC13:23
openstackgerritMerged openstack/tripleo-upgrade master: Handle upgrade_prepare_extra_params as comma-separated string.  https://review.opendev.org/68647813:28
*** d0ugal has quit IRC13:28
*** surpatil has joined #tripleo13:28
*** d0ugal has joined #tripleo13:29
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Hide sensitive data from Ansible run and logging  https://review.opendev.org/68838413:30
sri_cgoncalves, any suggestions how to debug this further ?13:30
openstackgerritSlawek Kaplonski proposed openstack/tripleo-quickstart-extras master: Add playbook and role to run tobiko  https://review.opendev.org/65542313:30
cgoncalvessri_, paste the horizon logs13:31
*** bnemec has joined #tripleo13:31
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates stable/stein: Ensure nova-api is running before starting nova-compute containers  https://review.opendev.org/68834913:31
*** slaweq_ is now known as slaweq13:35
openstackgerritMartin Schuppert proposed openstack/tripleo-heat-templates stable/stein: Ensure nova-api is running before starting nova-compute containers  https://review.opendev.org/68834913:38
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates stable/queens: Remove hiera interpolation from ceph-ansible parameters  https://review.opendev.org/68840613:41
*** hamdyk has joined #tripleo13:41
*** lbragstad_ is now known as lbragstad13:44
*** hamdykhader has quit IRC13:45
*** sanjayu_ has quit IRC13:45
*** toure has joined #tripleo13:45
openstackgerritMerged openstack/tripleo-common stable/stein: Always close src/dst sessions and raise to retry  https://review.opendev.org/68678713:46
sri_cgoncalves, /var/log/containers/httpd/horizon/horizon_access.log http://paste.openstack.org/show/783697/13:50
*** bfournie has quit IRC13:50
cgoncalvessri_, also paste error log and container log13:53
*** epoojad1 has joined #tripleo13:53
*** dsneddon has quit IRC13:55
*** chandankumar is now known as raukadah13:56
sri_cgoncalves, http://paste.openstack.org/show/783698/13:57
*** gbarros has joined #tripleo13:57
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Fix Ironic Authentication exception  https://review.opendev.org/68838314:01
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Use dynamic argument_spec from documentation  https://review.opendev.org/68812714:01
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Hide sensitive data from Ansible run and logging  https://review.opendev.org/68838414:01
*** surpatil has quit IRC14:03
*** bogdando_ has joined #tripleo14:04
*** beekneemech has joined #tripleo14:06
*** bnemec has quit IRC14:06
*** mdbooth has joined #tripleo14:06
*** yolanda__ has joined #tripleo14:06
*** bogdando has quit IRC14:06
*** bogdando_ is now known as bogdando14:06
*** yolanda has quit IRC14:09
*** mburned is now known as mburned_out14:09
*** epoojad1 has quit IRC14:11
*** morazi has quit IRC14:12
openstackgerrityatin proposed openstack/tripleo-ci master: [DNM] Add missing files for standalone  https://review.opendev.org/68840814:12
*** hamdykhader has joined #tripleo14:13
*** ricolin has joined #tripleo14:13
cgoncalvessri_, I cannot spot anything, sorry. no clue14:15
*** hamdyk has quit IRC14:16
sri_cgoncalves, no worries, it's fun of openstack :)14:17
*** dtrainor has quit IRC14:17
gchamoulTengu: hey there, it's maybe too late for you but could you please take a look at this one https://review.opendev.org/#/c/688383/ today if possible ?14:18
Tengugchamoul: on my way14:20
openstackgerritKamil Sambor proposed openstack/tripleo-heat-templates master: Add posibilities to set ovn_openflow_probe_interval for controller  https://review.opendev.org/68329414:20
Tengugchamoul: hmm is it tested in the CI in some way? could you test it if not?14:21
gchamoulTengu: A gift from dtantsur ;-)14:21
* dtantsur hides14:21
gchamoulTengu: I tested and retested this afternoon cause no helps from CI, you know :/14:22
Tengugchamoul: ok :9.14:22
Tengudtantsur: you're OK with it I guess?14:22
Tengu(sorry, just wanting to ensure that patch won't break things ;))14:23
openstackgerritMerged openstack/tripleo-common master: Fix dynamic executor appending to objects  https://review.opendev.org/68754814:23
gchamoulTengu: no worries14:24
dtantsurre https://review.opendev.org/#/c/688383 ? are you sure you need that? the problem with tripleo-common was usage of ironicclient.v1.client (note v1)14:24
gchamouldtantsur: I am not sure but it fixed what I wanted to fix14:26
dtantsurthen it's okay14:26
gchamouldtantsur: that's why I put you on the review circle14:26
Tengu:)14:26
*** dsneddon has joined #tripleo14:28
*** lmiccini has quit IRC14:28
gchamouldtantsur: the original exception is "An unhandled exception occurred while running the lookup plugin 'ironic_nodes'. Error was a <class 'keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions'>, original message: Auth plugin requires parameters which were not given: auth_url"14:28
dtantsurgchamoul: mmm, right, you have to pass either session or os_auth_url14:30
gchamouldtantsur: to the ironicclient.client.get_client method?14:31
dtantsurmm, no wait14:31
sri_hi all can anybody confirm is octavia/horizon UI working in stable/rocky ?14:32
dtantsurgchamoul: okay, I had to dig into the code, but it seems that both ironic_url and os_auth_token are not correct. should be "endpoint" and "token". then you won't need a session. although passing a session is better and more future-proof.14:33
*** hamdykhader has quit IRC14:34
dtantsurgchamoul: it should be enough to do get_client(1, session=session)14:34
gchamouldtantsur: ack will make some test with that ^^^, thanks a lot14:34
dtantsurnp14:36
gchamoulyes the session is enough ...14:36
gchamoulso the get_endpoint is useless too14:37
*** leanderthal has joined #tripleo14:37
openstackgerritHervé Beraud proposed openstack/puppet-tripleo master: Ensure to remove cinder's outdated leftover lock files  https://review.opendev.org/68841414:40
*** dtantsur is now known as dtantsur|brb14:40
openstackgerritHervé Beraud proposed openstack/tripleo-heat-templates master: Ensure to remove cinder's outdated leftover lock files  https://review.opendev.org/68841514:40
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Fix Ironic Authentication exception  https://review.opendev.org/68838314:43
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Use dynamic argument_spec from documentation  https://review.opendev.org/68812714:43
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Hide sensitive data from Ansible run and logging  https://review.opendev.org/68838414:43
*** klemorali_ has quit IRC14:44
*** matbu has joined #tripleo14:44
*** klemorali has joined #tripleo14:45
*** dtrainor has joined #tripleo14:45
*** surpatil has joined #tripleo14:47
openstackgerritCarlos Camacho proposed openstack/tripleo-heat-templates master: Add missing step tag in updates/upgrades/ffu tasks  https://review.opendev.org/68809214:48
*** brault has quit IRC14:50
*** surpatil has quit IRC14:51
openstackgerritJose Luis Franco proposed openstack/tripleo-upgrade stable/stein: Handle upgrade_prepare_extra_params as comma-separated string.  https://review.opendev.org/68841714:52
*** bdodd has joined #tripleo14:56
*** ysandeep has joined #tripleo15:00
*** beekneemech is now known as bnemec15:01
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Revert "Use tenancity to backoff when layer is locked"  https://review.opendev.org/68842115:02
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Revert "Use tenancity to backoff when layer is locked"  https://review.opendev.org/68842115:08
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Track uploaded layers for workers as global view  https://review.opendev.org/68728815:10
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Track uploaded layers for workers as global view  https://review.opendev.org/68728815:11
*** udesale has joined #tripleo15:12
ramishraweshay: hey! can you please review https://review.opendev.org/#/c/688081/? This is voting in stable/stein at present and fails most of the time..15:15
ramishraand we don't care about scenario009 openshift stuff15:16
*** dsneddon has quit IRC15:16
weshayhttps://bugs.launchpad.net/tripleo/+bug/184738515:18
openstackLaunchpad bug 1847385 in tripleo "rhel 8 master fails to create cm-local-ca.pem, Error: 'openssl pkcs12 " [Critical,Triaged]15:18
weshayTengu, ^15:18
*** ykarel is now known as ykarel|afk15:21
*** sri_ has quit IRC15:24
*** xek_ has quit IRC15:26
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Track uploaded layers for workers as global view  https://review.opendev.org/68728815:27
weshayade_lee, hey15:27
weshayade_lee, so the creds for pki turns out are not being created..15:28
Tenguweshay: https://review.opendev.org/#/c/688081/15:28
weshayade_lee, check out my last comment in https://bugs.launchpad.net/tripleo/+bug/184738515:28
openstackLaunchpad bug 1847385 in tripleo "rhel 8 master fails to create cm-local-ca.pem, Error: 'openssl pkcs12 " [Critical,Triaged]15:28
ade_leeweshay, ok15:29
ade_leelooking15:29
Tenguweshay: thanks for the merge :).15:31
ade_leeweshay, right -- I mentioned that the other day I think ..15:31
Tenguade_lee: certmonger thingy :). Move to Ansible!15:32
weshayade_lee, ya.. some jobs have it some don't15:32
* Tengu runs away15:32
weshayade_lee, but at least it's logged now15:32
weshaymarios|rover, ^15:32
Tenguweshay: race, for sure. Would be good to know WHAT is creating this file.15:32
Tenguanyway. See you around folks!15:32
ade_leeweshay, Tengu - creds is supposed to be created when certmonger starts up15:34
*** alexmcleod has quit IRC15:34
ade_leeso question is -- is certmonger actually up?  does creds eventually get created?15:34
weshayade_lee, services are viewable in /var/log/extra/services15:35
ade_leeweshay, is creds eventually created?15:36
weshayno15:37
weshayfails due to  Notice: /Stage[main]/Tripleo::Certmonger::Ca::Local/Exec[extract-and-trust-ca]/returns: Can't open /var/lib/certmonger/local/creds for reading, No such file or directory",15:37
weshayError: 'openssl pkcs12 -in /var/lib/certmonger/local/creds -out /etc/pki/ca-trust/source/anchors/cm-local-ca.pem -nokeys -nodes -passin pass:'' && update-ca-trust extract' returned 1 instead of one of [0]",15:37
weshaywhen those files are there, that works for me15:38
ade_leeweshay, ack -- but eventually does it get created?15:38
weshayno15:38
*** jtomasek has quit IRC15:38
ade_leeweshay, I'm investigating a different issue with certmonger where a file its supposed to create gets created 3.5 hours later15:38
ade_leeweshay, and rcrit was looking into some certmonger startup issues - where is spins for awhile on startup15:39
*** jtomasek has joined #tripleo15:40
*** raildo has joined #tripleo15:40
weshay  certmonger.service                                                                                     loaded    active   running   Certificate monitoring and PKI enrollment15:40
weshaylooks like certmonger is running15:40
*** dsneddon has joined #tripleo15:40
*** soniya29 has quit IRC15:41
*** jtomasek_ has joined #tripleo15:43
openstackgerritMerged openstack/tripleo-common stable/stein: Prefix ansible output with timestamps  https://review.opendev.org/68678815:44
*** jtomasek has quit IRC15:44
*** lucasagomes has quit IRC15:45
ade_leeweshay, is this happening consistently ?15:45
ade_leeweshay, or just every so often?15:46
weshayade_lee, in rhel8 looks like most of the time.. ya..15:47
*** dsneddon has quit IRC15:47
*** suuuper has quit IRC15:47
*** ccamacho has quit IRC15:47
*** dking_desktop has quit IRC15:48
weshayade_lee, so do we know where the initial creds are created?15:48
ade_leeweshay, there is a bug in certmonger startup which is causing certmong to close bunches of fds when forking - causing huge delays in startup15:48
ade_leeweshay, certmonger calls get_roots on startup, which in the case of the local creds, created them if they do not exist15:49
*** dsneddon has joined #tripleo15:49
ade_leeit happens after all the forking is done.15:49
ade_leeweshay, there is a fix for the forking issue which has been committed upstream -- maybe we can try it?15:50
openstackgerritRafael Folco proposed openstack/tripleo-ci master: Skip retag/push w/ _arch suffix for rhel8/podman  https://review.opendev.org/68843315:50
ade_leeweshay, not sure how to do that here ..15:50
weshayade_lee, ya.. what change?15:50
weshayer.. review15:50
ade_leewell - I have a couple of commits in certmonger upstream ..15:51
marios|roverweshay: ack15:51
ade_leeweshay, how do we patch certmnger here?15:51
weshayade_lee, if the patches have merged in certmonger.. we'll need to update rdoinfo to pull in the latest15:52
*** dsneddon has quit IRC15:55
*** dsneddon has joined #tripleo16:00
openstackgerritBogdan Dobrelya (bogdando) proposed openstack/tripleo-common master: Track uploaded layers for workers as global view  https://review.opendev.org/68728816:01
*** rpioso is now known as rpioso|afk16:02
*** leanderthal has quit IRC16:04
*** openstackstatus has joined #tripleo16:04
*** ChanServ sets mode: +v openstackstatus16:04
*** leanderthal has joined #tripleo16:04
*** iurygregory has quit IRC16:06
*** rpittau is now known as rpittau|afk16:06
*** marios|rover has quit IRC16:07
*** dtantsur|brb is now known as dtantsur16:08
*** raildo has quit IRC16:11
*** raildo has joined #tripleo16:11
openstackgerritRyan McCabe proposed openstack/tripleo-heat-templates master: Disable legacy telemetry by default  https://review.opendev.org/68148516:12
*** raildo has quit IRC16:18
*** raildo has joined #tripleo16:19
*** ykarel|afk has quit IRC16:21
*** jpich has quit IRC16:25
*** ysandeep has quit IRC16:27
*** brault has joined #tripleo16:31
*** udesale has quit IRC16:34
*** chem is now known as chem|eod16:36
*** ramishra has quit IRC16:36
*** xek_ has joined #tripleo16:42
*** xek_ has quit IRC16:42
*** xek has joined #tripleo16:44
*** devep has quit IRC16:49
*** xek has quit IRC16:49
*** jpena is now known as jpena|off16:55
*** jtomasek_ has quit IRC16:58
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Convert container environment from a list to a dict  https://review.opendev.org/68729316:58
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - deploy steps without paunch  https://review.opendev.org/68765116:58
*** jtomasek has joined #tripleo16:59
*** derekh has quit IRC17:00
*** ykarel|afk has joined #tripleo17:04
*** hjensas has quit IRC17:08
*** hjensas has joined #tripleo17:20
*** raildo has quit IRC17:20
*** raildo has joined #tripleo17:20
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: Deep merge hiera keys for mysqld_options  https://review.opendev.org/68851417:23
*** tesseract has quit IRC17:25
*** bfournie has joined #tripleo17:30
*** dking_desktop has joined #tripleo17:30
*** ricolin has quit IRC17:33
*** hjensas has quit IRC17:34
*** leanderthal has quit IRC17:35
*** leanderthal has joined #tripleo17:36
*** dtantsur is now known as dtantsur|afk17:36
dking_desktopI just tried adding two new physical nodes, ran a discover on the range. It found both of the new nodes, and one of them was even moved into available. However, for the second, it was left at "enroll". The logs show that the ipmitool command didn't include the user, which caused it to fail. I confirmed that if the same command were run for it as for the other nodes, it would succeed. So, I removed it, and ran another17:37
dking_desktopn't see anything in the ironic-conductor.log that indicates it's even trying that IP again. Meanwhile, ipmitool still works just fine to find it.17:37
*** hjensas has joined #tripleo17:37
dking_desktopAny thoughts on what would cause that, or what would be preventing it from finding it again?17:37
openstackgerritMerged openstack/tripleo-upgrade stable/stein: Enable system upgrade and upgrade run per host.  https://review.opendev.org/68521817:38
ade_leeweshay, so after chatting with rcrit -- a couple of things come to mind17:44
weshayk17:44
ade_leeweshay, 1) certmonger uses lockf() before writing the creds file.17:44
ade_leeso that means that likely some other process is holding the lock to that file17:45
ade_leemost likely another certmonger thread/process17:45
ade_leeif we could get onto the system when it fails, we might be able to determine what thread is holding the lock and why.17:46
*** hberaud is now known as hberaud|gone17:46
ade_leeweshay, we can also increase logging for certmonger although there is not a lot of logging in that area17:47
ade_leewe do this by creating a file /etc/sysconfig/certmonger with contents OPTS=-d317:47
ade_leethat should log more stuff into the journal17:48
ade_leelastly of course we can try the latest certmonger changes to see if they magically solve the problem -- though its unlikely they will.17:49
ade_leeweshay, certmonger is in rhel/centos/fedora -- not rdo ..17:51
dking_desktopCorrection. I checked the logs again, and it is attempting to find the new node. But each time, it tires without a username. It has never included the username or port for this one IP, unlike the way it connects to all the others.17:51
weshayade_lee, ok.. can you put up a change w/ increased logging17:53
weshayand we'll run rhel8 on it17:53
weshayade_lee, I'll try and get you an environment17:53
ade_leeweshay, cool17:53
ade_leeweshay, trying to figure our where to add the code to add increased logging ..17:54
ade_leeweshay, ansible soemwhere?17:54
weshayit's run from puppet17:55
ade_leeweshay, ok - so something in puppet-certmonger or puppet-triplo maybe ..17:56
weshayya17:59
*** psachin has quit IRC18:05
openstackgerritAde Lee proposed openstack/puppet-tripleo master: DNM: add debug to certmonger  https://review.opendev.org/68852618:09
ade_leeweshay, ^^18:09
weshaythanks ade_lee18:09
ade_leeweshay, lmk how it goes18:09
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: Tighten up how we add and remove kernel modules  https://review.opendev.org/68741918:10
openstackgerritwes hayutin proposed openstack/tripleo-ansible master: Tighten up how we add and remove kernel modules  https://review.opendev.org/68741918:10
weshayade_lee, added a depends on to https://review.opendev.org/#/c/687419/ .. I'll try and get you a system..18:11
*** gbarros has quit IRC18:11
*** gbarros has joined #tripleo18:12
ade_leeweshay, cool thanks18:13
*** matbu has quit IRC18:13
dking_desktopDoes anybody know, when "openstack overcloud node discover..." is run, specifically on a range, what is run initially to discover the nodes?18:13
dking_desktopThe fact that ironic-conductor.log only seems to show entries for successful attempts makes me think it's running something else that it isn't logging.18:14
*** rpioso|afk is now known as rpioso18:15
openstackgerritMerged openstack/tripleo-common stable/stein: Implement threading locks around layers  https://review.opendev.org/68678918:24
*** holser has quit IRC18:27
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: only support x86_64 and centos  https://review.opendev.org/68853418:46
openstackgerritMerged openstack/tripleo-heat-templates stable/stein: Set scenario009 job voting to false  https://review.opendev.org/68808118:48
*** ekultails has quit IRC18:51
*** ekultails has joined #tripleo18:51
*** gbarros has quit IRC18:53
*** kopecmartin is now known as kopecmartin|off19:00
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: Fix booleans settings in podman_container  https://review.opendev.org/68854019:14
*** ykarel|afk has quit IRC19:14
sshnaidm|ptoEmilienM, cloudnull ^^19:14
*** holser has joined #tripleo19:20
*** bnemec has quit IRC19:22
*** brault has quit IRC19:24
*** jfrancoa has quit IRC19:30
weshaysshnaidm|pto, he can't resist the keyboard19:32
sshnaidm|ptoweshay, who? :o19:33
openstackgerritMerged openstack/tripleo-common stable/stein: Improving debug logging  https://review.opendev.org/68679219:33
openstackgerritMerged openstack/tripleo-common stable/stein: Use tenancity to backoff when layer is locked  https://review.opendev.org/68820019:33
weshay:)19:34
*** gbarros has joined #tripleo19:39
*** gbarros has quit IRC19:42
openstackgerritRafael Folco proposed openstack/tripleo-ci master: Attempt to fix slow cleanup of cached files  https://review.opendev.org/68854919:49
*** bnemec has joined #tripleo19:56
*** gfidente has quit IRC19:57
*** gbarros has joined #tripleo20:05
*** slaweq_ has joined #tripleo20:15
*** slaweq has quit IRC20:18
*** slaweq__ has joined #tripleo20:19
*** slaweq_ has quit IRC20:21
weshayade_lee, fyi http://logs.rdoproject.org/19/687419/10/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/62f4cfd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz20:22
weshayade_lee, it does try three times :)20:23
weshayer.. twice20:23
ade_leeweshay, ack - I'll take a look thanks20:24
weshayade_lee, it's going to be a little bit for the recreate env20:25
ade_leeweshay, ok - will look at these logs and see if we can glean anything20:25
* weshay looking through the journal http://logs.rdoproject.org/19/687419/10/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/62f4cfd/logs/undercloud/var/log/journal.txt.gz20:25
*** pcaruana has quit IRC20:26
weshayheh.. will revisit Request1 soon20:27
weshayOct 14 15:04:22 undercloud.localdomain certmonger[83957]: 2019-10-14 15:04:22 [83957] Request1('haproxy-external-cert') releasing writing lock20:30
dking_desktopCould somebody give me a pointer on where I could look to troubleshoot this issue with baremetal node discovery? I'm q bit at a loss, and I'm not really sure where to even look in the code, or if it's a tripleo issue or an ironic issue.20:30
weshaymaybe it can't remove the lock? http://logs.rdoproject.org/19/687419/10/openstack-check/tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001/62f4cfd/logs/overcloud-controller-0/var/lib/certmonger/20:30
*** pbandark has quit IRC20:31
weshayndercloud.localdomain python3[84428]: ansible-file Invoked with path=/opt/puppetlabs/facter state=absent recurse=False force=False follow=True modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S _original_basename=None _diff_peek=None src=None modification_time=None access_time=None mode=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None content=NOT_LOGGING_PARAMETER backup=None20:31
weshayremote_src=None regexp=None delimiter=None directory_mode=None unsafe_writes=None20:31
weshayOct 14 15:04:27 undercloud.localdomain certmonger[83957]: 2019-10-14 15:04:27 [83957] Will revisit Request1('haproxy-external-cert') in 86400 seconds.20:31
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Remove hiera interpolation from ceph-ansible parameters  https://review.opendev.org/68840620:38
*** slaweq has joined #tripleo20:50
*** slaweq__ has quit IRC20:51
*** pkopec has quit IRC20:53
*** rcernin has joined #tripleo20:58
*** holser has quit IRC21:01
*** zaneb has quit IRC21:04
*** zaneb has joined #tripleo21:04
*** ldumont has joined #tripleo21:04
*** LaurentDumont has quit IRC21:05
*** leanderthal has quit IRC21:12
*** leanderthal has joined #tripleo21:13
*** raildo has quit IRC21:17
*** raildo has joined #tripleo21:17
*** raildo has quit IRC21:17
*** ekultails has quit IRC21:19
openstackgerritSagi Shnaidman proposed openstack/tripleo-ansible master: WIP: idempotency for podman_container  https://review.opendev.org/68857421:23
openstackgerritMerged openstack/diskimage-builder master: centos7; use numeric DIB_RELEASE  https://review.opendev.org/68832121:31
*** brault has joined #tripleo21:46
*** jistr has quit IRC21:52
*** bnemec has quit IRC21:55
*** rcernin has quit IRC21:59
*** jistr has joined #tripleo22:01
*** jistr has quit IRC22:02
*** jistr has joined #tripleo22:03
*** holser has joined #tripleo22:13
EmilienMsshnaidm|pto: thx22:14
*** slaweq has quit IRC22:14
*** holser has quit IRC22:19
openstackgerritIan Wienand proposed openstack/diskimage-builder master: [dnm] testing dnf copr repo for f30 build  https://review.opendev.org/68858022:21
*** rainleander has joined #tripleo22:48
openstackgerritMerged openstack/tripleo-common master: Bump ceph/daemon container image to v4.0.4  https://review.opendev.org/68812022:49
*** leanderthal has quit IRC22:51
*** slaweq has joined #tripleo22:52
*** slaweq has quit IRC22:56
*** rcernin has joined #tripleo23:07
*** artom has joined #tripleo23:11
*** tkajinam has joined #tripleo23:17
*** tkajinam_ has joined #tripleo23:17
*** tkajinam has quit IRC23:21
*** morazi has joined #tripleo23:23
*** morazi has quit IRC23:34
*** brault has quit IRC23:36
*** brault has joined #tripleo23:39
*** cdearborn has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!