Wednesday, 2017-10-11

openstackgerritJames Slagle proposed openstack/tripleo-common master: Config download support for all deployments
*** salmankhan has joined #tripleo00:07
*** ooolpbot has joined #tripleo00:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1722640 in tripleo "pike promotion errors, overcloud images and containers were not promoted properly" [Critical,Triaged] - Assigned to John Trowbridge (trown)00:10
*** salmankhan has quit IRC00:11
*** toure is now known as toure_biab00:15
*** yamahata has quit IRC00:18
*** ecerquei has joined #tripleo00:26
*** ecerquei has quit IRC00:39
*** psahoo has joined #tripleo00:42
EmilienMslagle: nice work00:43
openstackgerritMerged openstack/puppet-tripleo stable/newton: Allow to configure snmpd_config
*** Goneri has quit IRC00:53
*** tongl has joined #tripleo01:05
*** ooolpbot has joined #tripleo01:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1722640 in tripleo "pike promotion errors, overcloud images and containers were not promoted properly" [Critical,Triaged] - Assigned to John Trowbridge (trown)01:10
*** liverpooler has quit IRC01:10
*** jkilpatr has quit IRC01:10
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: overcloud-deploy: add config-download + ansible run feature
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: Config download support for all deployments
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Config download support for standalone deployments
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: fs10: deploy steps with ansible
openstackgerritMerged openstack-infra/tripleo-ci master: Add logs for config-download
openstackgerritMerged openstack-infra/tripleo-ci master: Update document url and fix a spelling error
*** daidv has quit IRC01:24
*** daidv has joined #tripleo01:24
openstackgerritMerged openstack/tripleo-heat-templates master: Remove Heat Cloudwatch API during upgrade and disable by default
*** daidv has quit IRC01:26
*** daidv has joined #tripleo01:26
openstackgerritMerged openstack-infra/tripleo-ci master: Include gate jobs in the cistatus page
*** daidv has quit IRC01:27
*** daidv has joined #tripleo01:27
openstackgerritMerged openstack/puppet-tripleo stable/ocata: Disable SSH login for nova_migration user when migration over ssh is disabled.
openstackgerritMerged openstack/puppet-tripleo stable/newton: Disable SSH login for nova_migration user when migration over ssh is disabled.
*** dmacpher has joined #tripleo01:49
*** psachin has joined #tripleo01:51
*** owalsh_ has joined #tripleo02:08
*** ooolpbot has joined #tripleo02:10
openstackLaunchpad bug 1722640 in tripleo "pike promotion errors, overcloud images and containers were not promoted properly" [Critical,Triaged] - Assigned to John Trowbridge (trown)02:10
*** ooolpbot has quit IRC02:10
*** owalsh has quit IRC02:11
*** Shatadru|Gone is now known as Shatadru02:13
*** Shatadru is now known as Shatadru|coffee|02:13
*** catintheroof has joined #tripleo02:20
*** catintheroof has quit IRC02:22
*** Shatadru|coffee| is now known as Shatadru02:23
*** ramishra has joined #tripleo02:39
*** Shatadru is now known as Shatadru|afk02:47
*** marrusl has quit IRC02:53
*** links has joined #tripleo02:56
*** marrusl has joined #tripleo03:06
*** daidv has quit IRC03:08
*** ooolpbot has joined #tripleo03:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1722640 in tripleo "pike promotion errors, overcloud images and containers were not promoted properly" [Critical,Triaged] - Assigned to John Trowbridge (trown)03:10
*** yamahata has joined #tripleo03:17
*** tzumainn has quit IRC03:21
openstackgerritOpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements
*** pdeore has joined #tripleo03:54
*** noslzzp has quit IRC03:55
*** noslzzp has joined #tripleo03:56
*** aditya_r has joined #tripleo04:03
*** ykarel|afk has joined #tripleo04:04
*** ratailor has joined #tripleo04:05
*** mdnadeem has joined #tripleo04:06
*** sshnaidm|off is now known as sshnaidm04:06
*** ratailor_ has joined #tripleo04:07
*** daidv has joined #tripleo04:11
*** ratailor has quit IRC04:11
*** fpan has quit IRC04:11
*** zaneb has quit IRC04:17
openstackgerritMerged openstack/tripleo-heat-templates master: Add networking-sfc support
*** dpawar has joined #tripleo04:19
*** aditya_r has quit IRC04:20
*** ratailor__ has joined #tripleo04:21
*** jtomasek has joined #tripleo04:22
*** ratailor_ has quit IRC04:22
*** Shatadru|afk is now known as Shatadru04:23
*** dpawar has quit IRC04:24
EmilienMjaosorior: FYI
*** gbarros has joined #tripleo04:28
EmilienMjaosorior: yeah04:28
*** aditya_r has joined #tripleo04:28
EmilienMjaosorior: see my comment04:28
EmilienMjaosorior: maybe we can engage with the ec2-api community04:28
jaosoriorEmilienM: when we introduced it we only ran pingtest jobs04:28
EmilienMansiwen: ^ when you're up, please look04:28
jaosoriormaybe they'll find it more useful now that we run tempest.04:29
EmilienManyway I go to bed04:29
jaosoriorhave a good one04:29
* EmilienM will try to not dream about gate04:29
pdeoreEmilienM, need your review on .. could you please have a look on this?04:31
*** ratailor_ has joined #tripleo04:32
*** shreshtha has joined #tripleo04:33
pdeoreEmilienM, jaosorior , Thanks !!04:33
*** ratailor__ has quit IRC04:36
*** gbarros has quit IRC04:37
*** fpan has joined #tripleo04:40
*** zaneb has joined #tripleo04:41
*** dparkes has quit IRC05:00
*** ebarrera has quit IRC05:06
*** skramaja has joined #tripleo05:07
*** mdnadeem_ has joined #tripleo05:08
*** mdnadeem has quit IRC05:10
*** gkadam has joined #tripleo05:15
*** Guest2849 is now known as sdake05:20
*** sdake is now known as Guest9543005:21
*** Guest95430 is now known as sdake_fixing05:23
*** ratailor__ has joined #tripleo05:23
*** ratailor_ has quit IRC05:26
*** sdake_fixing has quit IRC05:29
*** sdake_fixing has joined #tripleo05:29
*** jaganathan has joined #tripleo05:30
*** sdake_fixing has quit IRC05:35
*** udesale has joined #tripleo05:38
*** yog_ has quit IRC05:38
*** aufi has joined #tripleo05:42
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: Fix IP address for ssh-tunnel for UI
*** mdnadeem_ has quit IRC05:44
openstackgerritOpenStack Proposal Bot proposed openstack/instack-undercloud master: Updated from global requirements
*** cshastri has joined #tripleo05:49
*** dpawar has joined #tripleo05:49
*** dparkes has joined #tripleo05:51
*** yog_ has joined #tripleo05:51
sshnaidm is down, let's go home05:52
*** sshnaidm is now known as sshnaidm|afk05:52
*** achadha has joined #tripleo05:53
*** marios has joined #tripleo05:55
*** janki has joined #tripleo05:55
*** yog_ has quit IRC05:56
*** achadha has quit IRC05:58
*** mdnadeem_ has joined #tripleo06:00
*** dmacpher has quit IRC06:02
*** jfrancoa has joined #tripleo06:04
*** aditya_ra has joined #tripleo06:04
*** aditya_r has quit IRC06:05
*** yprokule has joined #tripleo06:07
*** yog_ has joined #tripleo06:08
*** aditya_ra has quit IRC06:14
*** aditya_r has joined #tripleo06:15
*** fragatina has joined #tripleo06:16
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-common master: Add novajoin images
Tenguhello jaosorior :906:16
jaosoriorTengu: hey dude, how's it going?06:18
Tengufine and you?06:18
jaosoriorpretty good06:19
jaosoriorgetting my morning coffee06:19
Tengueh :). getting my morning tea in here. and preparing a new deploy for my ceph nodes - I found out the issue yesterday, the colleague who plugged the additional network card plugged it in the wrong socket - meaning the deploy failed because it didn't find the 4 interfaces we configured…06:20
Tenguso hopefully today will be THE day where all works as expected. wish me luck ;)06:21
jaosorioroooh damn06:21
jaosoriorbut good that you found it06:21
jaosoriorgood luck!06:21
Tengufunnily, this issue also reported and "unknown" code/message.06:21
Tenguhad to go on the node and check its logs directly.06:22
*** spectr has quit IRC06:27
*** spectr has joined #tripleo06:28
*** jprovazn has joined #tripleo06:30
*** nyechiel_ has joined #tripleo06:31
*** ratailor_ has joined #tripleo06:32
openstackgerritMerged openstack/tripleo-heat-templates stable/newton: Correctly set DEFAULT/dhcp_domain to empty string.
openstackgerritMerged openstack/tripleo-heat-templates master: Fix some missed hard-coded network references
*** ratailor__ has quit IRC06:35
*** ebarrera has joined #tripleo06:36
openstackgerritGael Chamoulaud proposed openstack/tripleo-common master: Make validation inputs configurable via Mistral
*** ccamacho has joined #tripleo06:47
*** owalsh_ is now known as owalsh06:49
*** threestrands has quit IRC06:50
*** dpawar has quit IRC06:54
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: EARLY WIP: Convert tags to when statements for Q major upgrade workflow
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Addition of Nuage as mechanism driver for ML2
*** aditya_r has quit IRC06:58
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates stable/pike: Remove Heat Cloudwatch API during upgrade and disable by default
*** sdake has joined #tripleo07:01
*** sdake has joined #tripleo07:01
*** aditya_r has joined #tripleo07:02
openstackgerritMerged openstack-infra/tripleo-ci master: set neutron mtu to match system mtu
*** openstackgerrit has quit IRC07:03
*** rcernin has joined #tripleo07:05
*** jlinkes has joined #tripleo07:05
*** jpena|off is now known as jpena07:08
Tenguoh, new issue now: deploy seems to be stuck.07:12
*** pcaruana has joined #tripleo07:13
*** ykarel_ has joined #tripleo07:14
*** ffiore has joined #tripleo07:15
*** ykarel|afk has quit IRC07:16
*** tesseract has joined #tripleo07:19
*** sid1 has joined #tripleo07:19
*** dpawar has joined #tripleo07:19
*** amoralej|off is now known as amoralej07:20
*** florianf has joined #tripleo07:20
*** ykarel_ is now known as ykarel07:21
*** ratailor__ has joined #tripleo07:21
*** ratailor_ has quit IRC07:24
*** tongl has quit IRC07:30
*** ffiore has quit IRC07:35
*** ratailor__ has quit IRC07:36
*** cylopez has joined #tripleo07:36
*** jpich has joined #tripleo07:39
lvdombrkrfolks, if i deploy my overcloud with dvr, how can i remove it?07:43
*** openstackgerrit has joined #tripleo07:43
openstackgerritMarios Andreou proposed openstack/tripleo-docs master: Update documentation for O to P upgrade and update
Tengulvdombrkr: I think you'd better redeploy it from 0 - we did try something similar but in the end we couldn't change the network setting because a log of things are done in a one-way way, meaning we should have searched everywhere in order to ensure all variables/configurations are properly reverted/managed07:46
*** openstackgerrit has quit IRC07:48
lvdombrkrTengu: thanks, i will redeploy from 0...but it if i want to remove it from existing cloud (just to know)  i need to remove from deploy command dvr templates?07:49
Tengulvdombrkr: you need to replace all dvr stuff by the network plan you want, but that won't be enough07:50
Tengubecause many configurations won't be reverted/removed, as they will be "unmanaged" once you drop dvr templates.07:50
lvdombrkrTengu: thanks now its more clear..07:51
*** itlinux has joined #tripleo07:52
*** ffiore has joined #tripleo07:54
*** openstackgerrit has joined #tripleo07:55
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Remove Heat Cloudwatch API
*** sid2 has joined #tripleo07:59
*** florianf has quit IRC08:02
*** sid1 has quit IRC08:03
*** ffiore has quit IRC08:04
*** itlinux has quit IRC08:04
*** ffiore has joined #tripleo08:05
*** florianf has joined #tripleo08:07
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DNM: test containers update
*** florianf has quit IRC08:13
*** florianf has joined #tripleo08:13
*** sid1 has joined #tripleo08:15
*** nyechiel_ has quit IRC08:15
*** ebarrera has quit IRC08:18
*** sid2 has quit IRC08:18
*** yamahata has quit IRC08:19
*** hjensas has joined #tripleo08:20
*** ebarrera has joined #tripleo08:20
Tengujaosorior: so, deploy failed, got stuck in some process without any output. I suspect I kind of broke the stack with all the failures I got lately trying to add the ceph storages… Last try now, else I'll drop the whole stack and re-create it from zero…08:25
jaosoriorTengu: or you could wait for shardy to log in. He could probably help you out with that.08:25
jaosoriorI just wrecked my development environment08:27
jaosoriorlooks like a good time to go for lunch08:27
*** pcaruana has quit IRC08:27
*** nyechiel_ has joined #tripleo08:28
*** shardy has joined #tripleo08:29
*** pcaruana has joined #tripleo08:31
*** gfidente has joined #tripleo08:32
*** openstackgerrit has quit IRC08:33
*** spectr has quit IRC08:40
*** egonzalez has joined #tripleo08:41
Tengujaosorior: duh. well, we have to change something deep inside the overcloud, so I think we will re-deploy all the stuff anyway.08:42
*** openstackgerrit has joined #tripleo08:42
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Use sub_nodes_private instead of node_private
*** nyechiel_ has quit IRC08:47
*** aditya_r has quit IRC08:48
*** spectr has joined #tripleo08:50
*** hjensas has quit IRC08:52
*** fzdarsky has joined #tripleo08:52
*** derekh has joined #tripleo08:52
*** salmankhan has joined #tripleo08:53
*** lucas-afk is now known as lucasagomes08:55
*** ykarel is now known as ykarel|lunch08:56
*** hjensas has joined #tripleo08:58
*** hjensas has quit IRC08:58
*** hjensas has joined #tripleo08:58
openstackgerritThomas Herve proposed openstack/instack-undercloud master: Handle relative hieradata_override
*** milan has joined #tripleo09:07
*** fzdarsky has quit IRC09:10
*** dtantsur|afk is now known as dtantsur09:11
*** suuuper has joined #tripleo09:14
*** ratailor has joined #tripleo09:14
*** salmankhan has quit IRC09:14
*** dr_gogeta86 has joined #tripleo09:16
*** dr_gogeta86 has joined #tripleo09:16
*** salmankhan has joined #tripleo09:20
*** tosky has joined #tripleo09:21
ramishratherve: tempest chnage has broken our gate, we probably need
jaosoriorsshnaidm|afk, adarazs: I'm trying to run quickstart and it fails the repo setup step with: Failed to parse dlrn hash09:25
jaosoriorany idea why?09:25
ramishraignore it here:)09:25
sshnaidm|afkjaosorior, no, but logs would help09:25
adarazsjaosorior: yeah, how are you running? where? what job? :)09:26
jaosorioradarazs: no job, locally09:26
jaosorioradarazs, sshnaidm|afk: This is what I'm running:  ./ --no-clone --teardown all --clean -p quickstart-extras.yml -N config/nodes/1ctlr_1comp.yml -c config/general_config/minimal.yml  -R master-tripleo-ci --tags "all" $HOST09:26
*** mcornea has joined #tripleo09:27
*** mcornea has quit IRC09:27
adarazsjaosorior: so probably your env is not clean and you have some variables defined from an older run of maybe devmode where you specified some changes that the playbook picks up?09:27
jaosorioradarazs, sshnaidm|afk: This is the step where it breaks
adarazsjaosorior: that would be my guess.09:28
jaosorioradarazs: ok, gonna try deleting ~/.quickstart09:28
*** hewbrocca_afk is now known as hewbrocca09:28
*** mcornea has joined #tripleo09:28
adarazsjaosorior: and also try to start in a fresh shell session.09:28
sshnaidm|afkjaosorior, and what is in /home/stack/ ?09:28
adarazs(from the virthost)09:29
*** psahoo has quit IRC09:29
*** mcornea has quit IRC09:31
adarazsjaosorior: hmm, that's weird.09:32
adarazsnot what I expected. so it tries to curl and it failed to parse the hash.09:32
adarazsnot sure why that would happen apart from random download error from the DLRN server.09:32
jaosorioradarazs: I've had that happen three times in a row now :/09:32
adarazsthat is weird.09:33
adarazswe should look at that code that tries to parse it.09:33
*** mcornea has joined #tripleo09:34
*** jaosorior has quit IRC09:36
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Fix to check playbooks
*** fragatina has quit IRC09:37
*** mdnadeem has joined #tripleo09:38
sshnaidm|afkadarazs, jaosorior now current and current-tripleo are different, that's why script fails09:39
sshnaidm|afknot sure why it's different and why it should be the same according to pabelanger, checking..09:39
*** mdnadeem_ has quit IRC09:39
*** dparkes has quit IRC09:41
sshnaidm|afkadarazs, jaosorior oops, sorry, it shouldn't be the same, just checking if it's not empty..09:41
openstackgerritRajesh Tailor proposed openstack/tripleo-heat-templates master: Enable TLS for ec2api service
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Add pre-deployment negative tests for validations
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Add post-deployment negative tests for validations
pdeoreshardy, gfidente, need your review on please...09:44
adarazssshnaidm|afk: yeah, it shouldn't be the same :)09:45
*** dparkes has joined #tripleo09:46
*** pdeore has quit IRC09:47
*** jaosorior has joined #tripleo09:47
Tengujaosorior: good news: deploy is working fine. I have all my seven nodes recognized, and they were already rebooted on the final system, now it's configuration time for their different roles.09:52
jaosoriorTengu: nice!09:53
ykarel|lunchjaosorior, to me it looks like there is some network issue(connectivity to in your environment, can you try hacking your master-tripleo-ci.yml09:54
*** ykarel|lunch is now known as ykarel09:54
*** sid1 has quit IRC09:55
ykareli mean the output of: curl --silent ${NODEPOOL_RDO_PROXY}/centos7/current/delorean.repo09:55
Tengujaosorior: small note though: might be good to redefine the specs for the undercloud node, as they are pretty wrong in the end. Especially regarding RAM. and mentionning it's an I/O-intensive task would be nice as well.09:55
jaosoriorykarel: trying to run it out again. restarted my router and such.09:57
jaosoriorTengu: where? in the docs?09:57
ykareljaosorior, Ok09:58
*** udesale has quit IRC09:58
Tengujaosorior: yup.09:58
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: EARLY WIP: Convert tags to when statements for Q major upgrade workflow
jaosoriorTengu: you could propose a patch for this
Tengujaosorior: hmm yup.10:03
Tenguwill do that now as I have some spare time during the deploy process. thanks for the link :)10:03
openstackgerritNuman Siddique proposed openstack/tripleo-heat-templates master: ci-ovn: Disable Swift services in scenario 007 container job
*** dpawar has quit IRC10:05
*** dpawar has joined #tripleo10:06
-openstackstatus- NOTICE: The CI system will be offline starting at 11:00 UTC (in just under an hour) for Zuul v3 rollout:
openstackgerritCédric Jeanneret proposed openstack/tripleo-docs master: added some remarks for undercloud requirements
Tengujaosorior: -^ :)10:11
*** mdnadeem has quit IRC10:12
*** dbecker has joined #tripleo10:12
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Do not override the docker image for scenario001
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server
openstackgerritSagi Shnaidman proposed openstack/tripleo-quickstart master: Update images location to new ones
Tengujaosorior: thank you :)10:16
sshnaidm|afkadarazs, ^^10:17
*** ebarrera has quit IRC10:19
*** ratailor has quit IRC10:21
*** ebarrera has joined #tripleo10:21
*** ratailor has joined #tripleo10:22
*** rhallisey has joined #tripleo10:22
adarazssshnaidm|afk: -- reviewed, realized some things are missing :)10:23
sshnaidm|afkadarazs, let's do it in different patch, this one is only for tripleo-ci10:24
Tenguoh. hmmm. the fact we will be able to add custom backends to haproxy makes me think we might have a nice way to get let's encrypt certificate in a way that won't require any external storage.10:24
openstackgerritDmitry Tantsur proposed openstack/tripleo-common stable/ocata: Take wwn_with_extension into account, when configuring a boot device
*** salmankhan has quit IRC10:26
*** mdnadeem has joined #tripleo10:26
adarazssshnaidm|afk: you know how difficult it is to push through a change nowadays, one less thing to recheck 20 times.10:27
adarazssshnaidm|afk: again it was just rebased so we don't lose any time with pushing another patchset now.10:28
sshnaidm|afkadarazs, ci will shutdown soon anyway, so no hurry..10:28
adarazssshnaidm|afk: still do you think it will be easier to push things through after? :D10:29
adarazssshnaidm|afk: I just mean the less change we need to merge the better.10:29
sshnaidm|afkadarazs, I'd like to do more atomic changes for easy reverting them in case we need to10:29
adarazsespecially for 3 line extra changes.10:29
*** dobson has quit IRC10:29
*** ffiore has quit IRC10:29
openstackgerritJuan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Use cacertfile option to get CA certificate
adarazssshnaidm|afk: well, your patch is mainly concerned with renaming all the ipa_images.tar mentions, so that fits in the style.10:29
adarazssshnaidm|afk: same thing just for 3 different files.10:30
adarazsif we need to revert it, it needs to be reverted for both.10:30
adarazsthose links are pointing to the same things.10:30
sshnaidm|afkadarazs, yeah, but these are patches for tripleo-ci anyway..10:31
adarazssshnaidm|afk: eh, okay.10:32
adarazsyou'll help me with the rechecks. :P10:32
*** dobson has joined #tripleo10:32
*** salmankhan has joined #tripleo10:32
sshnaidm|afkadarazs, it's not a new issue :)10:33
sshnaidm|afkadarazs, and with zuulv3 it will be much easier and better (as infra promises)10:33
sshnaidm|afknext quarter maybe..10:33
Tenguhmm so no need to stress the "recheck" command for CI stuff right now, right? :)10:34
Tenguuhu, I see people on other openstack related channels stressing their merge requests/reviews :Þ10:34
adarazsTengu: it's probably as useful as beating a dead horse.10:34
Tengupoor horse.10:35
Tengushall we eat it instead? :]10:35
adarazsI wouldn't dare, it's possessed by Zuul :P :)10:35
Tenguit's ok, I'm a daemon myself, I don't fear Zuul ;)10:36
Tengudeploy's almost done. pfiouu.10:39
Tengutook over an hour… thanks to our undercloud instance that's just… well. anyway. it's working.10:39
*** ratailor_ has joined #tripleo10:41
ansiwenEmilienM: ok, I looked. Yes, it's a pity if they remove it, thanks for your comment. on the other hand, I can understand that they feel uncomfortable that something is gating their job that they don't control. If tripleo CI job is broken, they can't merge...10:42
ansiwen*gating their project...10:42
*** fandrieu has quit IRC10:43
rookshardy hey - thanks for checking out the doc... the only concern I have is, I was able to scale out beyond 32 nodes w/o convergence.10:44
*** ratailor has quit IRC10:45
*** sid1 has joined #tripleo10:45
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Mirror images from RDO server
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart master: Fix ironic-python-agent.tar name for centosci
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: WIP Fix ConfigDebug for puppet host runs
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Migrate to the new Mistral context class
*** jkilpatr has joined #tripleo10:49
*** dpawar has quit IRC10:52
*** achadha has joined #tripleo10:53
*** achadha has quit IRC10:58
*** fzdarsky has joined #tripleo10:59
*** dpawar has joined #tripleo11:00
*** dbecker has quit IRC11:02
*** ffiore has joined #tripleo11:02
lvdombrkrfolks, someone see this error, when tried to update existing overcloud?
*** stee_3 has quit IRC11:04
*** brault has quit IRC11:04
*** stee_3 has joined #tripleo11:05
*** pkovar has joined #tripleo11:05
*** itlinux has joined #tripleo11:05
itlinuxhello all.. I have a question since I have a client that wants to use provider network only what's the best way to adopt that using OOO?11:06
mandrejistr: is green!!11:06
shardyrook: Hi, so your nova error went away after disabling convergence?11:07
rookshardy the nova/keystone-auth issue didn't happen with convergence disabled,no.11:07
rookshardy: I also find it interesting it happened during a scale-up operation, not during the initial deploy11:08
itlinuxalso has anyone used or checked cell2 config on OOO11:10
jistrmandre: whohooo \o/ +A'd both of them11:11
* jistr grabs tea11:11
mandrejistr: looks like the upgrade failures are valid bugs, "Could not find the requested service openstack-ceilometer-central"11:12
mandrewow a lot of sweat in these two patches, thanks jistr11:12
*** fandrieu has joined #tripleo11:13
*** sshnaidm|afk is now known as sshnaidm11:15
rookscaling up w/o convergence shardy11:16
shardyrook: Ok, would be good to see the nova logs, as currently I'm not clear why convergence would cause a nova error11:17
shardyin theory the calls to nova from heat should be about the same11:18
shardythe timing may be different but otherwise I'm unclear why it would make any difference to nova11:18
shardyunless we dos'd the DB or something?11:18
* shardy bbiab lunchtime11:18
*** shardy is now known as shardy_lunch11:19
sshnaidmfor whom it could be relevant, zuul-v3 status page is here:
sshnaidmzuul-v2 is here: http://cistatus.tripleo.org11:20
itlinuxhi.. all about zuul is there a module then to install zuul?11:21
*** udesale has joined #tripleo11:21
jaosorioritlinux: where do you want to install zuul?11:22
*** fzdarsky_ has joined #tripleo11:22
*** lucasagomes is now known as lucas-hungry11:23
*** bfournie has quit IRC11:26
jistrmandre: hm right, there's not a single recent upgrade job success...11:26
jistrmandre: and mainly thank *you* for those patches! :)11:26
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Add new SELinux validation check
rookshardy_lunch i don't see how the ddos of the db could cause this11:28
rookwe did see an additional CPU WAIT time of 2%11:28
rookbut i don't see how that would impact the scale11:28
rooki will look once I get back to add more information11:28
*** morazi has joined #tripleo11:29
*** mburned is now known as mburned_out11:31
*** mburned_out is now known as mburned11:33
lvdombrkrfolks, someone see this error, when tried to update existing overcloud?
jistrmandre: interesting -- the upgrade job on went *much* further than on other patches currently proposed11:37
jistrmandre: we're fixing things by accident, that's a new level11:37
*** ratailor__ has joined #tripleo11:38
*** ffiore has quit IRC11:38
*** Shatadru is now known as Shatadru|Gone11:38
jistreverything else on fails around 2 hour mark with some dbus permission issue it looks. I'm not even sure if it's worth reporting and investigating given that the problem might disappear shortly...11:40
*** shreshtha has quit IRC11:41
*** ratailor_ has quit IRC11:41
*** jpena is now known as jpena|lunch11:42
*** spectr has quit IRC11:44
*** skramaja has quit IRC11:44
*** spectr has joined #tripleo11:46
*** ffiore has joined #tripleo11:47
*** ratailor__ has quit IRC11:49
*** pdeore has joined #tripleo11:49
*** pdeore has quit IRC11:50
*** pdeore has joined #tripleo11:51
*** ffiore has quit IRC11:53
*** fzdarsky_ has quit IRC11:54
*** spectr-RH has joined #tripleo11:54
*** fzdarsky has quit IRC11:54
*** fzdarsky_ has joined #tripleo11:56
*** fzdarsky has joined #tripleo11:56
mwhahahaugh looks like v3 ovb jobs are failing11:57
*** athomas has joined #tripleo11:57
*** spectr has quit IRC11:57
*** jaganathan has quit IRC11:57
mwhahahasshnaidm: do you know why the v3 ovb jobs are failing or do we have a bug for it?11:58
sshnaidmwell first failures of v3 are coming..11:58
sshnaidmmwhahaha, "rm: cannot remove ‘/home/zuul/cache/files/cirros-0.3.5-x86_64-disk.vhd.tgz’: Permission denied"11:58
*** jcoufal has joined #tripleo11:58
sshnaidmmwhahaha, no, it's something new11:59
sshnaidmmwhahaha, I think related patch was merged just yesterday in infra, about cache dir..12:00
*** amoralej is now known as amoralej|lunch12:00
mwhahahak might want to let them know12:00
*** lucas-hungry is now known as lucasagomes12:01
*** bfournie has joined #tripleo12:01
*** dprince has joined #tripleo12:03
sshnaidmmwhahaha, yeah, we have an etherpad:  reported there12:03
*** abishop has joined #tripleo12:03
mwhahahasounds good12:04
sshnaidmseems like it's not for us only12:04
*** jlabarre has joined #tripleo12:04
sshnaidmmwhahaha, EmilienM btw, v3 jobs are here:
*** pdeore has quit IRC12:05
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: EARLY WIP: Convert tags to when statements for Q major upgrade workflow
*** fzdarsky has quit IRC12:10
*** fzdarsky_ has quit IRC12:10
*** itlinux has quit IRC12:10
*** shardy_lunch is now known as shardy12:11
*** ffiore has joined #tripleo12:11
*** spectr-RH has quit IRC12:13
*** spectr has joined #tripleo12:13
*** itlinux has joined #tripleo12:14
*** spectr-RH has joined #tripleo12:14
*** panda|off is now known as panda12:15
jaosoriormandre, shardy: Is there a way to run an ansible task immediately after a container is spawned?12:16
*** spectr has quit IRC12:18
*** openstackgerrit has quit IRC12:18
*** ebarrera_ has joined #tripleo12:19
*** openstackgerrit has joined #tripleo12:20
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: Implement stonith levels resource
*** aditya_r has joined #tripleo12:21
*** aputtur has joined #tripleo12:26
*** tzumainn has joined #tripleo12:26
jaosoriorjistr: the containerized undercloud currently doesn't support SSL, or does it?12:27
shardyjaosorior: you can run bootstrapping tasks via docker_config which could run a command/script but it's not integrated to run ansible tasks specifically atm12:27
shardyjaosorior: the ordering is controlled via the steps, so you could launch the container in step3 the do some docker_config in step412:28
shardyjaosorior: there is also docker_puppet_tasks which does bootstrapping operations (only on the bootstrap host) via puppet12:28
*** rlandy has joined #tripleo12:28
jaosoriorshardy: I would need to run a task on the host that runs the container :/12:29
*** trown|outtypewww is now known as trown12:29
jaosoriorshardy: so this is the case: Currently we use certmonger to auto-generate certificates for the undercloud (when you set generate_service_certificate to True). I was trying to do something similar for the containerized undercloud, which would use a containerized certmonger.12:31
shardyjaosorior: can you expand on why?  The normal pattern is to run tasks in a temporary init container12:31
shardyjaosorior: as we can no longer be sure any dependencies exist on the host12:31
thrashweshay: remember that issue I was telling you about quickstart where a tag occurs "after" my patch, and devmode won't install the dlrn package with my patch in it?12:31
thrashweshay: trown
openstackLaunchpad bug 1722783 in tripleo "devmode Quickstart does not install built package" [Medium,Triaged]12:31
*** ecerquei has joined #tripleo12:31
jaosoriorNow, while we can run certmonger containerized, and request certificates with it, this wouldn't be very useful, as the CA certificate needs to be trusted by the host, else we won't be able to use that certificate12:32
thrashweshay: trown I've hit it again.12:32
shardyjaosorior: so can you spin up a container via docker_config with some directory bind mounted from the host?12:32
weshaythrash, ok.. thanks for opening a bug12:32
weshaywe'll poke at it12:32
jaosoriorshardy: sure! but then I need to run a command to actually get the host to trust that cert12:32
*** sshnaidm is now known as sshnaidm|mtg12:32
shardyjaosorior: Ok so a command needs to run on the host, or just some config data be written?12:32
thrashweshay: sure thing12:32
shardyjaosorior: ack, I'm just not familiar with the command, what does it change, and can that be done via a bind mount?12:33
jaosoriorshardy: yep, a command; namely update-ca-trust12:33
*** liverpooler has joined #tripleo12:33
jaosoriorshardy: it grabs whatever CA certificates we add to /etc/pki/ca-trust/source/anchors/  and appends it to the trusted  bundle.12:33
jaosoriorshardy: if we don't do that, we're gonna have to specify the CA certificate for every command we do.12:34
jistrbut the bundle is also just a file in /etc isn't it?12:34
*** itlinux has quit IRC12:35
*** jpena|lunch is now known as jpena12:35
shardyyeah so we just bind the ca bundle file or etc dir into the bootstrapping container12:35
shardyand run that via docker_config?12:35
jaosoriorjistr: so you're suggesting I could bind-mount in r/w mode /etc/pki/ca-trust  and run the task12:35
*** itlinux has joined #tripleo12:35
jaosoriorwell yeah, I guess that could work12:35
shardynot super clean but I think that may work12:35
*** pchavva has joined #tripleo12:35
jaosoriorsounds like a way forward12:36
matbushardy: o/ hey, do you have any idea why the tests is failing there:
jaosoriorshardy, jistr: thanks12:36
jistrjaosorior, larsks, jbadiapa: btw just finished reviewing the logging spec and i love it :) That's exactly what we need for k8s. Can't wait to be able to access all logs centrally just via `kubectl logs`.12:37
*** aputtur has quit IRC12:37
jaosoriorjistr: yay :D12:37
jaosoriorjistr: thanks for taking a look12:38
shardymatbu: looks like an out of memory problem?12:38
shardyo/ btw :)12:39
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations stable/pike: Update services in the process count validation
matbushardy: yes, looks like but thats weird12:39
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Migrate to the new Mistral context class
openstackgerritDmitry Tantsur proposed openstack/tripleo-docs master: Rework documentation for ironic in the overcloud
*** catintheroof has joined #tripleo12:43
jaosoriormwhahaha: hey, if you have some time could you give this a read ?12:44
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data
mwhahahajaosorior: yea it's on my list today12:45
*** rodolof has joined #tripleo12:45
jaosoriormwhahaha: thanks12:45
*** jcoufal has quit IRC12:47
shardymatbu: perhaps try running the unit tests locally and monitor the memory usage?12:47
openstackgerritMerged openstack/tripleo-quickstart-extras master: Set NeutronGlobalPhysnetMtu=1350 for ovb based deployment
*** jcoufal has joined #tripleo12:48
lvdombrkrfolks who are using pacemaker and dvr same time?12:48
openstackgerritMerged openstack/instack-undercloud master: Updated from global requirements
*** jmelvin has joined #tripleo12:49
*** hewbrocca is now known as hewbrocca_afk12:49
*** fragatina has joined #tripleo12:51
*** hewbrocca_afk is now known as hewbrocca12:52
*** amoralej|lunch is now known as amoralej12:55
*** ykarel has quit IRC12:56
*** ykarel has joined #tripleo12:56
openstackgerritMarios Andreou proposed openstack/tripleo-heat-templates master: Add post_upgrade_tasks and upgrade_batch_tasks to deploy output
EmilienMsshnaidm|mtg: ok12:58
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates stable/pike: Fixes dynamic networks falling back to ctlplane
*** toure_biab is now known as toure13:00
*** ebarrera has quit IRC13:01
*** ebarrera has joined #tripleo13:02
*** openstackgerrit has quit IRC13:03
*** dpawar has quit IRC13:05
-openstackstatus- NOTICE: Due to unrelated emergencies, the Zuul v3 rollout has not started yet; stay tuned for further updates13:07
*** dpawar has joined #tripleo13:07
*** bobh has joined #tripleo13:07
*** nyechiel_ has joined #tripleo13:09
gfidentemwhahaha fultonj regarding my comment in
openstackLaunchpad bug 1722633 in tripleo "Puppet Duplicate declaration error when role is composed with both Ceph OSDs and Mons" [Low,Triaged] - Assigned to John Fulton (jfulton-org)13:09
gfidenteI am just not sure there is an easy way to figure if the ::mon profile is enabled at puppet compile time13:09
*** sdake has quit IRC13:11
*** sdake has joined #tripleo13:11
*** sdake has joined #tripleo13:11
*** ebarrera_ has quit IRC13:14
*** openstackgerrit has joined #tripleo13:14
openstackgerritJuan Antonio Osorio Robles proposed openstack/paunch master: Add option to configure uts namespace
*** dbecker has joined #tripleo13:15
*** sdake has quit IRC13:15
*** ebarrera has quit IRC13:15
*** ebarrera has joined #tripleo13:17
rookshardy: so updating the cloud /scaling up w/o convergence worked :/13:17
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Set metric procssing delay for metricd
shardyrook: ack, any more clues from the logs?  I'm still not really clear how convergence can cause errors in nova13:17
rooklooking now shardy13:17
*** nyechiel_ has quit IRC13:18
*** sdake has joined #tripleo13:19
*** sdake has quit IRC13:19
*** sdake has joined #tripleo13:19
openstackgerritThomas Herve proposed openstack/instack-undercloud master: Handle utf-8 when running python commands
rookshardy so Nova -> Neutron issues it seems... With Convergence does heat being to query things more often?
*** marrusl has quit IRC13:31
*** cdearborn has joined #tripleo13:35
ykarelslagle, hi13:35
*** tosky_ has joined #tripleo13:36
*** itlinux has quit IRC13:36
*** aditya_r has quit IRC13:37
*** aditya_r has joined #tripleo13:38
*** tosky has quit IRC13:38
*** jistr is now known as jistr|mtg13:38
*** pblaho has quit IRC13:38
*** aditya_r has quit IRC13:40
*** pblaho has joined #tripleo13:41
slagleykarel: hello13:41
ykarelslagle, actually i was trying trass,13:42
ykareland it was failing and now i got that it was because of default: tripleo_ci_remote13:42
*** lblanchard has joined #tripleo13:43
ykarelso will try with openstack-infra/tripleo-ci13:43
*** udesale has quit IRC13:43
ykarelactually this patch is not there in slagle/tripleo-ci that's why it was failing13:43
slagleykarel: how was it failing? the default value for the tripleo_ci_remote parameter should just work13:43
slagleykarel: oh i see. i may need to rebase my fork13:44
ykarelslagle, yes and in trass branch as that's the default13:44
Tenguuhu. I have a *doc* build that fails. I think I won't do anything for now, right? :)13:44
slagleykarel: the reason i have the fork is because this patch is needed:
slagleEmilienM: could you have another look at ?13:45
ykarelslagle, Ok13:45
EmilienMslagle: of course. Do you mind if I rebase it and make sure CI pass well, and also re-verify from where package is deployed?13:46
slagleEmilienM: i think i need to update the patch anyway because my history undo command there could cause a problem13:46
EmilienMslagle: yeah13:46
*** mdnadeem has quit IRC13:46
*** paramite has quit IRC13:47
rookshardy nothing in the neutron logs.13:48
rookshardy: i wonder if haproxy is killing the connection?13:49
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Remove workaround for LP#1701239 bug
openstackgerritJames Slagle proposed openstack-infra/tripleo-ci master: Enable repo for python-os-testr
slagleEmilienM: there, the logic should be correct now. just need to confirm where the rpm is coming from13:49
EmilienMslagle: the only problem I see is if we have a RDO outage, the job will fail because it doesn't use the AFS mirror13:53
*** mcornea has quit IRC13:53
*** mcornea has joined #tripleo13:53
EmilienMwhy not running `generate-subunit $(date +%s) 10 fail pingtest | gzip - > $WORKSPACE/logs/testrepository.subunit.gz` later ?13:53
*** ffiore has quit IRC13:53
EmilienMlike once we have the repos in place (with AFS mirrors)13:54
slagleEmilienM: not sure, i didn't add that part13:54
EmilienMI'm not trying to be picky, but I just don't want us to do recheck because RDO server was down13:54
slagleEmilienM: how do the AFS mirrors get configured?13:54
EmilienMand we have elastic queries that prove it's the case very often13:54
*** udesale has joined #tripleo13:55
EmilienMslagle: in quickstart for the oooq jobs and in tripleo-ci for legacy jobs13:55
EmilienMsshnaidm|mtg: you added this code, maybe we can discuss13:55
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Set host name explicitly for telemetry
sshnaidm|mtgEmilienM, on mtg now, is it urgent?13:55
slagleEmilienM: really, we shouldn't be using this openstack package (ostestr) until the repos have been setup13:56
*** links has quit IRC13:56
EmilienMsshnaidm|mtg: no13:56
*** janki has quit IRC13:56
EmilienMslagle: I can't agree more13:56
sshnaidm|mtgEmilienM, ok, I'll ping you then13:57
EmilienMsshnaidm|mtg: ack13:57
EmilienMslagle: we'll figure that out once Sagi is back - meantime I'll look where we can run that13:57
*** ffiore has joined #tripleo13:57
*** paramite has joined #tripleo13:57
*** Guest85006 is now known as sc13:59
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras
*** jistr|mtg is now known as jistr14:00
openstackgerritAdriano Petrich proposed openstack/tripleo-common master: WIP validate parameters before updating
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Add pre-deployment negative tests for validations
*** leitan has joined #tripleo14:00
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Add post-deployment negative tests for validations
openstackgerritAndreas Scheuring proposed openstack/diskimage-builder master: Fix rendering issues in DIB "building_an_image" doc
openstackgerritJason E. Rist proposed openstack/tripleo-quickstart master: SSH Tunnel wrongly including http/https
*** dpawar has quit IRC14:03
openstackgerritAndy Smith proposed openstack/puppet-tripleo master: WIP: Updates to separate messaging backends alternative
*** spectr-RH has quit IRC14:04
*** chlong has joined #tripleo14:04
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: overcloud-deploy: add config-download + ansible run feature
*** spectr has joined #tripleo14:04
*** lblanchard1 has joined #tripleo14:05
EmilienMslagle: please review
EmilienMslagle: I forgot something in the commit message14:05
*** lblanchard has quit IRC14:06
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: overcloud-deploy: add config-download + ansible run feature
*** tosky_ is now known as tosky14:10
openstackgerritAndy Smith proposed openstack/puppet-tripleo master: WIP: Updates to separate messaging backends alternative
*** dpawar has joined #tripleo14:12
*** aditya_r has joined #tripleo14:12
*** gbarros has joined #tripleo14:12
*** rajinir-afk is now known as rajinir14:14
EmilienMmwhahaha: have we found a way to track new timeouts on ovb-ha?14:14
EmilienMthe query doesn't seem to work14:14
EmilienMI remember we had a query for that14:14
mwhahahathere's a generic timeout one14:14
mwhahahathat one?14:14
*** artom has joined #tripleo14:14
EmilienMbut it doesn't work on some jobs14:15
mwhahahait should14:15
mwhahahamaybe elastic check is lagged14:15
mwhahahai saw it pop up a few time recently14:15
EmilienMok I'll investigate a bit14:16
EmilienMmwhahaha: this one? Bug 1686542 - Generic job timeout bug14:16
openstackbug 1686542 in OpenStack-Gate "Generic job timeout bug" [Low,Confirmed]
*** gbarros has quit IRC14:17
EmilienMouch indeed14:17
dalvarezegonzalez, ping14:17
egonzalezdalvarez, sup14:17
*** gbarros has joined #tripleo14:17
dalvarezegonzalez, re.   i'd like to fix the sources thing but i'm super new to kolla so im unsure about how to do it...14:18
dalvarezegonzalez, i took a look at:   so maybe i could use its {% elif install_type == 'source' %}  block?14:18
EmilienMmwhahaha: stats aren't good, 28 timeouts in 3h14:18
mwhahahai wonder if it's because of increased load from v2/v314:19
EmilienMmwhahaha: zuul or keystone?14:19
mwhahahasince we're 2x the ovb jobs14:21
mwhahahathough i thought we had an upper limit of total running ovb jobs14:21
* mwhahaha shrugs14:21
mwhahahawhat does bnemec say :D14:21
EmilienMmwhahaha: the queue should be bigger but not the load, iiuc14:21
egonzalezdalvarez, more like this example server-ovn uses plugins feature14:22
EmilienMslagle: any thoughts on the way we deploy "jq" in ? do you think of a better way?14:22
egonzalezdalvarez, and to add the source tarball14:22
openstackgerritmathieu bultel proposed openstack/tripleo-common master: WIP - POC on callback plugin for ansible to collect result
bnemecmwhahaha: I think we're still only running 70 jobs at once between 2 and 3 though, which is what we had before.14:23
dalvarezegonzalez, python-networking-ovn-metadata-agent is a subpackage of python-networking-ovn. Also neutron-server-ovn is installed from python-networking-ovn so that's why i thought I had to use it14:24
EmilienMmwhahaha, bnemec:
bnemecThey split them 55/15 during the rollback.14:24
slagleEmilienM: yes. we need a way to run some bootstrap style tasks14:24
slagleEmilienM: i plan to add that in a subsequent patch14:24
EmilienMbnemec, mwhahaha: it keeps being worse :(14:24
slaglethe first patch is already quite large14:24
EmilienMslagle: I was thinking to add "jq" as a dependency of something we deploy on the overcloud14:25
egonzalezdalvarez, source code for metadata-ovn comes in networking-ovn git14:25
dalvarezegonzalez, exactly14:25
mwhahahaEmilienM: plz don't hide the deps that way14:25
mwhahahait's better to be exliclit in the playbook than wedge a dep on a random package that may or maynot continue to be deployed14:26
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data
mwhahaharemember we want to get rid of the images so let's think about creating a dep task to capture some of these software requirements14:26
slaglewe need to add some bootstrap tasks anyway to do things like install the initial heat hooks, etc14:26
mwhahahayes that :D14:26
slagleso all that should just be done explicitally in an ansible task14:26
EmilienMslagle: in tripleo-common, right?14:27
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data
slagleEmilienM: yes14:27
*** paramite has quit IRC14:27
EmilienM(I just didn't want to see it in quickstart-extras)14:27
EmilienMslagle: perfect14:27
bnemecEmilienM: mwhahaha: I suppose it's possible convergence has increased the load to the point where we can't run as much at once, but the load on rh1 doesn't look that high.14:27
honzaHello folks, this patch seems to be stuck in CI (unknown time left).  Is there a way to kill it/reset it?
mwhahahabnemec: i'd hate to revert the convergence switch, should we lower the limit on rh1 to like 65 jobs and see if that clears it up?14:28
EmilienMbnemec, mwhahaha : so look at logstash:
mwhahahahonza: no need it's queued upstream14:29
mwhahahahonza: you don't want to reset it14:29
bnemecmwhahaha: We could try that.  We did lose a couple of compute nodes in the reboot so it's possible that is a factor too, which would be helped by reducing the job concurrency.14:29
EmilienMsorry that sounds stupid but we switched to heat convergence?14:29
mwhahahaon the undercloud14:29
honzamwhahaha: It says 'Remaining time unknown'.  Should I ignore that?  All things as they should be?14:29
EmilienMo_O when?14:30
honzamwhahaha: Moar patience required?14:30
mwhahahahonza: click on it you'll see two jobs are queued (waiting on nodes most likely)14:30
honzamwhahaha: ok, thanks14:30
EmilienMok bnemec did
EmilienMI didn't see it14:30
mwhahaha2 days ago14:30
mwhahahathat's when it spiked the 2nd time14:30
mwhahahalooks like we were timing out more starting on 10/114:31
mwhahahabnemec: when did you do the reboots?14:31
EmilienMyeah and it doubled14:31
*** sadasu has joined #tripleo14:31
*** udesale has quit IRC14:34
*** dpawar has quit IRC14:36
openstackgerritThomas Herve proposed openstack/instack-undercloud master: Use transport_url for Heat
*** sid1 has quit IRC14:39
gfidentefultonj, don't pay attention marios always finds additional work for me14:40
EmilienMmwhahaha: it sounds like we didn't have logs frm 5 days ago14:40
EmilienMmwhahaha: but timeouts started 5 days ago if we look logstash14:40
EmilienMah no, 10 days14:41
EmilienMweird logstash paging14:41
EmilienMI can't see hits after 6 days but the chart shows 10 days14:41
EmilienMit started to be serious on October 314:42
bnemecmwhahaha: According to my records, I finished the reboot on/around Sept. 26.14:43
*** spectr has quit IRC14:43
fultonjgfidente: ack14:45
mariosgfidente: fultonj i +2 man it was a _suggestion_  but really after this comment I think i have to revote.14:46
*** dpawar has joined #tripleo14:46
*** shardy has quit IRC14:47
gfidentemarios oh I was joking14:50
mariosgfidente: me too :)14:50
*** dpawar has quit IRC14:50
gfidentemarios thanks for pointing it out14:51
gfidente(you were joking too)14:51
mariosgfidente: did you just throw some kind of exception?14:51
gfidenteno I was actually writing some release notes14:51
gfidentegot distracted14:51
* marios stops harassing the faidentee14:52
mariosthough i should point out you started it14:52
gfidenteyes but you can take over the submission14:53
EmilienMso cute :)14:56
EmilienMbnemec, mwhahaha : ok so what's the plan for this timeout? we need to take a look and see if indeed heat convergence has an effect. We could compare overcloud deployment time now and 10 days ago14:57
bnemecI would love to, but I don't think we have those numbers in a consumable format anywhere these days.14:57
*** spectr has joined #tripleo14:57
EmilienMmwhahaha: re: - why jq would be a dep in package feature in ansible? ?14:58
EmilienMbnemec: we can look at the ansible logs though14:59
*** dbecker has quit IRC14:59
EmilienMbnemec: and see task run time14:59
EmilienMbnemec: I'll take a look...14:59
mwhahahaEmilienM: use proper ansible thank ansible == bash++14:59
* bnemec is going to start referring to Ansible as Bash++ now15:00
EmilienMI'm trying to have productive discussion here :)15:00
EmilienMthe timeouts are really bad, we need to do something15:01
mwhahahai'm not trolling15:01
mwhahahause proper ansible to do package installs15:01
*** cshastri has quit IRC15:01
mwhahahaas for CI, drop the capacity15:01
mwhahahastep 115:01
mwhahahastep 2 figure out why we had to do step 115:01
bnemecI'm sort of trolling but also serious.  The whole reason we had graphite stats was for debugging stuff like this.15:02
*** rhallisey has quit IRC15:02
EmilienMmwhahaha: I thought you were talking about the ansible logs, ok for package, it makes sense. slagle said we follow-up with jq in a later patch15:02
*** psachin has quit IRC15:02
EmilienMbnemec: we can still restore it15:02
EmilienMsomeone has to do it15:03
mwhahahaif only there was a team who should be focusing on ci15:03
* mwhahaha now starts trolling15:03
* mwhahaha pokes weshay & company15:03
mwhahahaby the way, when are we cutting over to rdo cloud for ovb15:03
slaglemwhahaha: EmilienM : i think what i will do is start an etherpad (with bugs) of stuff we need to clean up and improve around for the ansible effort15:04
weshaywhich timeout, upgrades?15:04
EmilienMweshay: not only, ovb15:04
EmilienMslagle: lgtm15:04
slaglei can see a situtation where this tripleo-common patch never gets merged b/c we can nitpick it to death15:04
weshayovb is migrating to rdo-cloud starting now15:04
weshaywe're walking through the planning on that as I type :)15:05
mwhahahaweshay: we're getting timeouts on ovb, seems like it might be capacity related. was wondering when rh1->rdo cloud15:05
EmilienMslagle: no, we'll merge it and iterate.15:05
weshaymwhahaha, yup.. in planning and process now15:05
EmilienMslagle: we want this thing in the gate asap15:05
mwhahahaslagle, EmilienM: yea i don't want to nit pick it, i just pointing out that we shouldn't be doing that. it can be a follow up but a lot of times we lose track of those follow ups and they become set in stone15:05
*** chlong has quit IRC15:06
mwhahahaslagle, EmilienM: i'm all for iteration but we have to follow through on the clean up15:06
mwhahahaour clean up track record is terrible15:06
*** yprokule has quit IRC15:06
*** salmankhan has quit IRC15:07
trownmwhahaha: we will be sending an email to the ML after our planning meeting to layout our plan for moving OVB jobs, but that is the focus of this sprint for the CI squad15:07
mwhahahaalso i wasn't -1 because of that either, but if we do have to touch it it would be good to swithc out15:07
*** salmankhan has joined #tripleo15:07
mwhahahatrown: k sounds good15:07
EmilienMmwhahaha: we will follow-up15:08
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Add a GetNetworksAction
EmilienMslagle: can I start an etherpad or you're already on it?15:08
slagleEmilienM: go4it15:08
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Add a Get Networks workflow
EmilienMslagle: ok15:09
EmilienMbnemec, mwhahaha: another option is to revert the convergence switch on the undercloud15:09
EmilienMgive me a reason not doing so15:09
mwhahahabe cause it started before the convergence switch15:09
mwhahahaso it's not going to 100% fix it15:09
*** ebarrera_ has joined #tripleo15:09
EmilienMplease look at logstash15:10
mwhahahai did15:10
EmilienMit doubled the day we switched15:10
mwhahahaand it started on 10/115:10
EmilienM85 hits in 12h15:10
mwhahahadrop the capacity of rh115:11
*** links has joined #tripleo15:11
* mwhahaha has 0 visibility into rh1's status 15:11
*** ebarrera has quit IRC15:11
mwhahahai'm all for revert if it's a complete fix15:12
mwhahahafor timeouts the revert is not one of those15:12
mwhahahait takes it from awful to slightly less awful15:12
*** aditya_r has quit IRC15:13
*** aditya_r has joined #tripleo15:14
bnemec is the current load on rh1, for the record.15:14
bnemecNone of the nodes are running above 50% utilization and most are far below that.15:14
mwhahahanetwork throughput maybe?15:14
*** aputtur has joined #tripleo15:15
*** ebarrera has joined #tripleo15:15
bnemecPossibly.  I haven't looked at that since we started doing containers.15:15
mwhahahathe ha job is the one that's timing out more regularly i think15:15
* mwhahaha starts digging into logs15:15
mwhahahaso it's taking ~30 mins to get to the undercloud install15:17
bnemecI wonder, did we lose the optimization I did to disable unused services on the undercloud?15:17
mwhahahai don't think so15:18
*** ebarrera_ has quit IRC15:18
mwhahahawe haven't really merged much in instack-undercloud15:18
mwhahahaunless that was a quickstart change15:18
*** aditya_ra has joined #tripleo15:19
*** aditya_r has quit IRC15:19
mwhahahaso it seems the modifying of the image is taking a decent chunk of time15:20
EmilienMmwhahaha: have you compared tasks with a run 10 days ago for ex?15:20
mwhahahano just spot checking timings right now looking for the big ones15:21
*** tongl has joined #tripleo15:21
mwhahahait's litterally like 30 mins of prep around everything before we even start installing the undercloud15:21
mwhahahathat seems crazy in general15:22
*** achadha has joined #tripleo15:22
mwhahahawell i can tell you that we're running a different ordering of stuff from ~2 weeks ago15:25
mwhahahaor at leas tit seems this way15:25
* mwhahaha supresses his hatred for ansible outputs15:26
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Add global deployment tasks executed on undercloud
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Deploy kubernetes using TripleO on the overcloud
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Support Kubespray installation via config download mechanism
openstackgerritBrad P. Crochet proposed openstack/tripleo-common master: Validate roles data and network data
* mwhahaha gives ARA a try15:26
mwhahahaso everything is taking slightly longer15:27
EmilienMjistr, flaper87: do we still need ?15:27
*** aditya_ra has quit IRC15:28
jistrEmilienM: we don't have another way to test in CI, but right now i'm not pursuing testing in CI... It would be preferable to have it packaged. I hear bogdando pursued that with centos community, i gotta sync up on it with him.15:28
*** jaganathan has joined #tripleo15:29
bnemecmwhahaha: No, I mean from the pre-quickstart days.  This:
jistrflaper87, gfidente, slagle, dprince: btw i successfully installed overcloud with external k8s installer using the config download + ansible-playbook mechanism
bnemec4.5 minutes per job adds up quite quickly.15:30
EmilienMjistr: it would be great to test in CI imho15:30
mwhahahabnemec: it's possible but I see that we're getting an image or something now that we weren't 2 weeks ago15:30
EmilienMjistr: and good job!15:30
* mwhahaha is checking quickstart15:30
jaosoriorbnemec: hey, could you give this a read?
jaosoriorjistr: nice!!!15:30
*** aputtur has quit IRC15:30
gfidentejistr nice how does the inventory looks like?15:31
*** jlinkes has quit IRC15:31
mwhahahabnemec, EmilienM: fetch-images : Get image is a new task that task ~1.5 mins15:31
jistrgfidente: see line 55
mwhahahabnemec, EmilienM vs
jistrgfidente: i reuse the outer inventory and essentially add aliases for what kubespray expects15:31
mwhahahabnemec, EmilienM: were doing a bunch of image stuff we weren't doing 2 weeks ago15:33
mwhahahamodify-image : make sure an image and script are provided15:33
*** dparkes has quit IRC15:33
mwhahahathat used to be true15:33
mwhahahais now false15:33
mwhahahaEmilienM: so there's yer problem15:33
dprincejistr: very nice on the kubespray patch15:34
*** mcornea has quit IRC15:34
EmilienMslagle: is safe to land even if gate-tripleo-ci-centos-7-ovb-ha-oooq timeouts? i think yes15:35
EmilienMmwhahaha: ok let me look git history15:35
mwhahahaEmilienM: hasn't changed15:36
slagleEmilienM: has it ever passed?15:36
mwhahahaso it must be that config15:36
EmilienMslagle: 6 days ago15:36
jistrgfidente: i'm not sure if that's directly applicable to ceph-ansible too? But even if it weren't, there should be ways to introspect the outer Ansible inventory and generate the inner one without including the file itself, we could try and figure out whatever you need.15:36
slagleEmilienM: actually, legacy-tripleo-ci-centos-7-ovb-ha-oooq passes15:37
slagleEmilienM: so i think it's good15:37
*** chlong has joined #tripleo15:37
slaglearen't those jobs currently the same?15:37
mwhahahayea they are15:38
gfidentejistr so ceph-ansible we used to have roles based on service_name15:38
EmilienMmwhahaha: maybe ?15:38
gfidenteand collect the list of ips where such a role is installed15:38
EmilienMslagle: they are same15:38
EmilienMslagle: I'll approve15:38
openstackgerritAlan Bishop proposed openstack/tripleo-heat-templates master: Enable Cinder as a backend for Glance
mwhahahaEmilienM: no i don't think so15:38
gfidentejistr like
mwhahahaEmilienM: did we bump ansible version? maybe it's that conditional15:39
*** rodolof has quit IRC15:39
jistrgfidente: we could generate that somehow i think, but it might be easiest to do what i did in the kubespray patch. tripleo-ansible-inventory script generates host groups based on the service_manifest names15:39
*** ebarrera has quit IRC15:40
jistrso you could do15:40
* jistr looking at code15:40
*** aufi has quit IRC15:40
jistrand that's it IIUC15:40
jistrsorta like an alias... :)15:41
EmilienMmwhahaha: which conditional sorry?15:41
mwhahahaEmilienM: modify-image : make sure an image and script are provided15:41
mwhahahaEmilienM: or we weren't previously updating the repos (a bug) and now we are (not a bug)15:42
openstackgerritRicardo Noriega proposed openstack/tripleo-heat-templates master: Using stevedore alias for BGPVPN Service Plugin
EmilienMmwhahaha: indeed, that could be a reason15:43
gfidentejistr is ceph_mon collected by tripleo-inventory15:43
gfidenteI mean, what does it represent?15:43
gfidenteI am not familiar with how the outer ansible inventory looks like15:44
*** gkadam has quit IRC15:44
EmilienMtrown, sshnaidm|mtg : please ping when you're back15:44
mwhahahaactually let me see maybe i'm reading this wrong15:44
jistrgfidente: yes it collects on which roles are which services, and generates an ansible host group for each service. here's a snippet
mwhahahaso the cache image doesn't exist anymore15:46
mwhahahawrong conditional15:46
mwhahahafetch-images : Ensure image cache directory exists15:46
mwhahahawhich i think is triggering all the stuff15:47
mwhahahait's creating /home/jenkins/images-cache15:47
mwhahahathat's what's triggering the fetch-images task15:47
jistrgfidente, slagle, dprince: also i reused bogdando's logging approach (like we use for puppet) for the inner ansible run, so the logs are nicely readable within the outer playbook. Snippet:
mwhahahainstead of
dprincejistr: are there any controls with that for Debug, vs Info logging levels?15:49
*** thrash is now known as thrash|biab15:51
*** chlong has quit IRC15:51
jistrdprince: not now i think (neither the inner ansible, nor puppet -- though i'm not sure about puppet). We could surely add a flag on the outer playbook and pass apppropriate flag or set a variable on the inner one. The only limit is whatever the inner tool supports.15:51
mwhahahaEmilienM, trown, sshnaidm|mtg: We're purging the image cache dir on ovb
sshnaidm|mtgmwhahaha, yes, so...?15:53
mwhahahasshnaidm|mtg: so every ovb run is rebuilding them15:53
numansEmilienM, hi, FYI - the job gate-tripleo-ci-centos-7-scenario007-multinode-oooq-container-nv is passing in
sshnaidm|mtgmwhahaha, no15:53
mwhahahasshnaidm|mtg: yes15:53
sshnaidm|mtgmwhahaha, no, we use cached images15:53
mwhahahasshnaidm|mtg: please read scroll back15:54
*** ratailor has joined #tripleo15:54
*** egonzalez has quit IRC15:54
mwhahahasshnaidm|mtg: we are running the fetch-image tasks where 2 weeks ago we were not15:54
mwhahahathis leads to longer ovb run times and timeouts15:54
sshnaidm|mtgmwhahaha, if nothing was changed recently we use cached images15:55
mwhahahahow often are we changing stuff in master?15:55
sshnaidm|mtgmwhahaha, will check after mtg15:55
EmilienMnumans: AWESOME! why did you disable swift? can you explain in commit message?15:55
EmilienMnumans: or at least in comment in the review15:55
sshnaidm|mtgmwhahaha, anyway removing image-cache is not related15:56
EmilienMnumans: also, please project a patch to project-config to make it voting once this THT patch is merged15:56
mwhahahasshnaidm|mtg: then what's the problem then15:56
*** marios has quit IRC15:57
*** pcaruana has quit IRC15:59
*** rcernin has quit IRC15:59
EmilienMshardy, gfidente, jistr: i've been asked why skydive patches haven't been reviewed, and - I was wondering if you could help, at least to make sure it's ok to do the same workflow thing as we do for ceph-ansible16:00
sshnaidm|mtgmwhahaha, do you have a link?16:01
EmilienMgfidente, jistr: or if no, they should do another way16:01
mwhahahasshnaidm|mtg: wait why aren't we building the images anymore16:01
gfidenteEmilienM didn't we have this conversation already16:01
gfidenteand decided with slagle to postpone it until the new mechanism was in place?16:01
EmilienMgfidente: yeah but the reviews didn't happen16:01
sshnaidm|mtgmwhahaha, we build always on tripleo-common and instack-undercloud16:01
EmilienMgfidente: we might want to write that in Gerrit16:01
EmilienMsafchain: ^ fyi16:02
*** rodolof has joined #tripleo16:02
*** lucasagomes is now known as lucas-afk16:02
gfidenteEmilienM I remember asking slagle to do so because he suggested waiting16:02
sshnaidm|mtgon these repos we always build16:02
mwhahahasshnaidm|mtg: it seems that the cached version is no faster than building the images in terms of timing16:02
mwhahahaso let me go get a different ci run to compare against16:03
*** ratailor has quit IRC16:03
safchainEmilienM, gfidente thx if could have a status on the patches16:03
mwhahahasshnaidm|mtg: for the record i was comparing vs
*** rodolof has quit IRC16:04
*** rodolof has joined #tripleo16:04
slaglegfidente: EmilienM : all I had said was that I was going to block any patches16:05
slagleerr, wasn't16:05
sshnaidm|mtgmwhahaha, 8 minutes to update the image, not sure it's a problem (first link)16:05
EmilienMI think these guys just want to know what happens16:05
slaglei honestly don't know what the urgency around skydive was, so i had not plans to block anything16:05
mwhahahasshnaidm|mtg: so now i'm going to compare against
mwhahahasshnaidm|mtg: 8 mins is significant when we're constantly bumping up against the wall clock16:06
*** rodolof has quit IRC16:06
sshnaidm|mtgmwhahaha, it always was 1016:06
*** ffiore has quit IRC16:06
*** rhallisey has joined #tripleo16:06
*** rodolof has joined #tripleo16:06
*** milan has quit IRC16:07
*** dbecker has joined #tripleo16:07
*** rodolof has quit IRC16:08
*** jpich has quit IRC16:08
*** rodolof has joined #tripleo16:08
mwhahahasshnaidm|mtg: yup, actually the image cache was longer previously. looks like the deploy is just taking long for some reason16:09
bnemecWe just got one of the compute nodes in rh1 back.16:10
mwhahahaEmilienM, bnemec: so the undercloud install is about 2 mins longer and the image prep is about 2 mins longer, but the overall time to get to the deploy is the same. the deploy is taking longer16:10
mwhahahaso it's something in the overcloud deployment that's going long16:10
bnemecThat gives us like 3.3% more capacity. :-)16:10
*** chlong has joined #tripleo16:10
EmilienMwe've lost alex16:10
EmilienMhe was a nice guy16:11
*** itlinux has joined #tripleo16:11
*** lebauce has quit IRC16:11
trownfrom canadian to french in 1 minute16:13
EmilienMyeah :)16:14
pabelanger2017-10-11 15:50:47 | 2017-10-11 15:50:47,205 INFO: [1;31mError: Evaluation Error: Error while evaluating a Function Call, Could not find class ::swift for centos-7-2-node-infracloud-vanilla-11334692 at /etc/puppet/manifests/puppet-stack-config.pp:329:1 on node centos-7-2-node-infracloud-vanilla-11334692[0m16:14
pabelangerknown issue?16:14
EmilienMslagle, gfidente : I discussed with the skydive team and they want it asap and probably iterate with the new framework later - but right now they don't know what to do to make progress and kind of wait on us16:15
EmilienMpabelanger: which change id?16:15
pabelangerthat is in gate16:15
pabelangerwhich just caused a reset16:15
EmilienMlet me check the puppet error16:16
gfidenteslagle EmilienM ok so while on one hand it looks like we might have to redo it within the queens cycle16:16
EmilienMsafchain: ^16:16
gfidenteif that helps testing we can merge it as is16:16
sshnaidm|mtgmwhahaha, but I see downloading image took 5 mins, when should take 1 maximum.. and it's suspicious16:16
*** sshnaidm|mtg is now known as sshnaidm16:16
mwhahahasshnaidm: i saw 10 on the old job16:17
sshnaidmmwhahaha, that's very bad16:17
mwhahahasshnaidm: it would be nice to get all the various prep peices < 15 mins. cause we spend ~30 mins messing with images16:17
mwhahahawhich is significant in the CI16:17
mwhahaha30 mins for prep, 30mins for undercloud install and that leaves like ~1hour for the overcloud deploy16:18
mwhahaha+/- various other things16:18
sshnaidmmwhahaha, actually it was about 15 mins or less before16:18
mwhahahanot sure what you mean by before16:18
mwhahahathe test results from the end of sept the timing is the same as it is now16:19
sshnaidmmwhahaha, when I counted it, month(?) ago16:19
itlinuxhello all.. does anyone have the puppet mani for ooo to push designate ?16:19
mwhahahapabelanger: EmilienM puppet-swift is missing from the rpm list16:19
mwhahahapabelanger: EmilienM:
sshnaidmmwhahaha, 5 mins for closing qcow2 image!16:19
sshnaidmmwhahaha, pabelanger that's vm disk performance sucks16:20
EmilienMmwhahaha: oh wait16:20
mwhahahabnemec: -^ io issues maybe?16:20
mwhahahaEmilienM: did you forget to add puppet-swift to the deps for puppet-tripleo16:20
mwhahahanot sure who that didn't fall out anywhere else16:21
mwhahahathat seems odd16:21
pabelangersshnaidm: feel free to donate more resources16:21
mwhahahasshnaidm: this is on rh1 anyway so that's not upstream's fault. /me points to rdocloud migration questions from ealier16:21
EmilienMmwhahaha: I'll double check we don't miss any other module now16:22
EmilienMmwhahaha: it's weird, I copy-pasted from OPM16:22
*** lebauce has joined #tripleo16:22
sshnaidmmwhahaha, next 2 weeks:
EmilienMmwhahaha: but I obviously did a mistake16:22
bnemecmwhahaha: Maybe?  That would be weird though because rh1 is configured with unsafe caching and SSDs specifically to avoid io performance issues.16:22
sshnaidmpabelanger, it seems more like disk problem, did something changed recently?16:23
*** rodolof has quit IRC16:23
sshnaidmpabelanger, sorry, it was for bnemec16:23
safchaingfidente, EmilienM in case something changed during the last few weeks we are going to restest it, otherwise that's ok for us to merge it, it will help to test16:23
bnemecmwhahaha: Is it happening on more than one job?  We do have one or two compute nodes that are missing SSDs, so it's possible that if a vm got scheduled to one of those it would be slow.16:23
bnemecBut even then it's usually not a problem because of the cache settings.16:24
mwhahahabnemec: the ha job more consistently16:24
akrivokajtomasek: do you remember if there's a bug open for the inconsistency issue in parameter names in tht (ObjectStorage vs SwiftStorage, etc)?16:24
mwhahahawhich is why i asked about io16:24
*** cylopez has quit IRC16:24
mwhahahadat galera16:24
EmilienMsafchain: it works for me, I'll have a look at the patches again today16:24
mwhahahabnemec: i dont' suppose there's a way to track nodepool vm to hypervisor?16:25
safchainEmilienM, great thx16:25
*** mugsie has joined #tripleo16:26
*** sadasu has quit IRC16:26
EmilienMmwhahaha: missed ovn as well16:27
bnemecEmilienM: Maybe, but I don't think one or even a couple of bad computes would explain the failure rates we're seeing.16:27
EmilienMmwhahaha: I'll patch and merge16:27
bnemecBut I need to run for a meeting like 10 minutes ago.16:27
bnemecI'll be back in ~1 hour.16:27
*** shreshtha has joined #tripleo16:28
*** yamahata has joined #tripleo16:30
*** Goneri has joined #tripleo16:31
sshnaidmmwhahaha, btw, overcloud deployment was taking about 1 hour, if it's more, it's also a problem..16:31
mwhahahayea but that could be io related16:31
sshnaidmmwhahaha, it may be network problem, especially if downloads take so much time16:31
mwhahahawhich is why i happens more on the ha than the non-ha16:31
openstackgerritGiulio Fidente proposed openstack/tripleo-heat-templates master: Update CephPools format in the docker templates to fit ceph-ansible
sshnaidmmwhahaha, yeah, either disk io or network16:32
*** weshay is now known as weshay|ruck16:32
mwhahahaif it was network i would assume it'd happen more consistently on all the jobs16:32
*** panda is now known as panda|rover16:32
mwhahahasince the non-ha job (As far as i've seen) is fine, but the scenario/ha job is timing out i think it's io16:33
*** dparkes has joined #tripleo16:33
mwhahahait's taking ~10 mins longer16:33
sshnaidmmwhahaha, which non-ha job? multinode?16:34
EmilienMmwhahaha: sorry for that, human error - can you review again plz?16:34
mwhahahasshnaidm: containers16:35
*** d0ugal has quit IRC16:35
sshnaidmyeah, it also had problems with timeouts, but less, as well as  gate-tripleo-ci-centos-7-ovb-1ctlr_1comp_1ceph-featureset024 job16:36
EmilienMmwhahaha: when ready, let's approve it16:36
mwhahahaEmilienM: k done16:37
EmilienMmwhahaha: we should have 72 modules, let me check16:38
sshnaidmmwhahaha, all these problems started from this Monday, btw16:38
*** thrash|biab is now known as thrash16:38
mwhahahasshnaidm: no we had timeouts starting on 10/116:39
mwhahahasshnaidm: it got worse starting on monday (probably related to the undercloud convergence switch)16:39
*** dtantsur is now known as dtantsur|afk16:39
sshnaidmmwhahaha, yeah, I see..16:39
*** yog_ has quit IRC16:40
*** dparkes has quit IRC16:40
sshnaidmmwhahaha, do you have a bug?16:40
mwhahahai don't think there is one yet16:40
sshnaidmweshay|ruck, can we handle this issue? ^^ timeouts in ovb jobs16:40
weshay|rucksshnaidm, with regards to this sprint?16:41
sshnaidmweshay|ruck, not sure, it still should work until we move..16:42
sshnaidmweshay|ruck, seems like disk io or network problem actually16:42
sshnaidmon rh116:42
weshay|rucksshnaidm, ya.. should be tracked in a bug16:42
weshay|rucksshnaidm, feel free to pick it up16:43
sshnaidmwhere is |rover ??16:43
openstackgerritAndy Smith proposed openstack/puppet-tripleo master: WIP: Updates to separate messaging backends alternative
weshay|ruckpanda|rover, <----16:44
mwhahahared rover, red rover send weshay over16:44
weshay|ruckmwhahah ahahahhaha :)16:44
*** tzumainn has quit IRC16:45
openstackgerritRonelle Landy proposed openstack/tripleo-common master: DO NOT MERGE - Add no-op change for containers update testing
weshay|ruckthat works anytime alex tells a joke16:45
openstackgerritAndy Smith proposed openstack/tripleo-heat-templates master: WIP Separate rpc and notify messaging backends
*** chlong has quit IRC16:48
*** dprince has quit IRC16:48
*** milan has joined #tripleo16:48
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Fix ConfigDebug for puppet host runs
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add the option to run the container-check script
*** dprince has joined #tripleo16:50
openstackLaunchpad bug 1722864 in tripleo "OVB frequent timeouts on rh1 cloud" [Critical,Triaged]16:51
*** sshnaidm is now known as sshnaidm|off16:52
*** yog_ has joined #tripleo16:53
*** links has quit IRC16:54
*** achadha has quit IRC16:55
honzaI'm getting errors in CI where some binary packages aren't installed.  Is this a known issue?
*** tzumainn has joined #tripleo16:57
*** d0ugal has joined #tripleo16:58
*** derekh has quit IRC16:58
weshay|ruckarxcruz, ok.. the doc looks good for the sprint end, please send to openstack-dev[tripleo]17:00
honzaWhat does 'see alerts' mean?17:01
EmilienMhonza: alerts are posted here on IRC by a bot17:03
EmilienMhonza: they represent launchpad bugs with "alert" tag, see
EmilienMthey usually are critical issues in our CI17:03
honzaEmilienM: thanks!17:04
*** MaxPC has joined #tripleo17:06
EmilienMhonza: sure, anytime17:06
*** paramite has joined #tripleo17:06
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: fs10: deploy steps with ansible
EmilienMslagle: ^ this one is also ready for review plz17:09
*** jpena is now known as jpena|off17:09
*** ooolpbot has joined #tripleo17:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1722864 in tripleo "OVB frequent timeouts on rh1 cloud" [Critical,Triaged]17:10
*** trown is now known as trown|lunch17:10
*** marrusl has joined #tripleo17:11
*** jfrancoa has quit IRC17:12
EmilienMhonza: this is an alert ^17:13
*** tesseract has quit IRC17:15
*** athomas has quit IRC17:15
*** aditya_r has joined #tripleo17:16
*** aditya_r has quit IRC17:17
openstackgerritAna Krivokapic proposed openstack/tripleo-validations master: Warn if there are not enough node IPs in pools
*** aditya_r has joined #tripleo17:18
*** hewbrocca is now known as hewbrocca_afk17:20
*** panda|rover is now known as panda|rover|off17:22
slagleEmilienM: we don't actually want to take over fs10 do we?17:24
EmilienMtrown|lunch, sshnaidm|off, mwhahaha: if some of you can review please17:24
EmilienMslagle: oh ok17:24
slaglei dunno17:24
EmilienMslagle: so where should we take over?17:24
trozethi what happened to stable directory in ?17:24
EmilienMslagle: all?17:24
slaglei guess i was thinking new job17:27
*** achadha has joined #tripleo17:27
slaglei dont know what job we'd get rid of17:27
*** links has joined #tripleo17:27
*** ebarrera has joined #tripleo17:28
EmilienMslagle: on previous fs10, deploying the overcloud took 31 minutes and 10 seconds. On the new fs10 with config-download and ansible steps, overcloud deployment takes 33 minutes17:28
EmilienMslagle: just fyi17:28
slaglethat's about what i'd expect17:29
EmilienMslagle: but we aren't testing the mistral workflow yet (not written yet)17:30
EmilienMslagle: I would keep an eye on these numbers and see17:30
*** achadha_ has joined #tripleo17:30
EmilienMslagle: you said "it might offset the agent sleep/poll times" - where can we see that?17:30
slaglesee what?17:31
slaglei said it might17:31
*** achadha has quit IRC17:31
slagleyou'd "see" it in the timing data17:31
*** achadha has joined #tripleo17:32
weshay|rucktrown|lunch, sshnaidm|off adarazs panda|rover|off
EmilienMslagle: of which tasks?17:32
slagleEmilienM: i dont know what youre talking about17:32
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Always pass docker-ha env when passing docker env
*** achadha has quit IRC17:33
*** achadha has joined #tripleo17:33
weshay|rucktrown|lunch, we also need to get your patch through
EmilienMslagle: I'm just asking which metrics can we monitor in these changes to see if there are improvements or not... that's all17:33
*** achadha has quit IRC17:33
*** achadha_ has quit IRC17:34
slagleEmilienM: are we collecting any metrics in the ci jobs?17:34
slagleEmilienM: if we're not, we can't see it17:34
*** achadha has joined #tripleo17:34
EmilienMslagle: ARA collects the lengh of Ansible tasks, that's all I'm aware of17:34
*** achadha has quit IRC17:34
EmilienMthe duration of tasks17:34
*** tosky has quit IRC17:34
dmsimardI believe Sagi might have something to feed task duration from ara into graphite17:34
*** fragatina has quit IRC17:35
dmsimardEmilienM: wtf17:35
EmilienMslagle: so I guess it's wip to have metrics, at this point17:36
dmsimardEmilienM: otherwise job-wide duration is available on grafana.o.o17:37
dmsimardalthough jeblair is still ironing out some issues from v3 for that17:37
EmilienMdmsimard: we're not interested by this data17:38
EmilienMdmsimard: we want details of what happens17:38
*** ykarel is now known as ykarel|afk17:38
EmilienMexactly what sshnaidm is doing17:38
*** achadha has joined #tripleo17:38
slagleEmilienM: we'd have to be collecting metrics around exactly what we want to measure17:38
dmsimardEmilienM: bah17:38
slagleEmilienM: we aren't measuring ansible tasks, we should be measuring overcloud deloy times, for example17:39
EmilienMslagle: re: config-download job - I'm not sure we want a new job17:39
*** fzdarsky has joined #tripleo17:39
EmilienMslagle: yes, it's exactly what sshnaidm is working17:39
slagleEmilienM: if it happens to correspond to 1 specific task, that might work17:39
EmilienMor at least I think17:39
*** fzdarsky_ has joined #tripleo17:39
dmsimardanother approach is that I believe tasks are pushed to firehose17:39
EmilienManyway, we could already discuss about what jobs we want to switch to use config-download17:40
*** itlinux has quit IRC17:40
EmilienMuntil we have a proper mistral workflow, I suggest to not move all the jobs using it17:40
EmilienMjust maybe one for now is enough17:40
EmilienMso we can keep continue the testing on the features and iterate17:40
EmilienMslagle: what do you think?17:41
slagleEmilienM: that's fine. just not sure which we can do without?17:41
trozetweshay|ruck: do you know where stable directory went?
slagleevery job we have, we currently need17:42
dmsimardEmilienM: fyi
slagleEmilienM: which one were you thinking we don't need coverage on any longer?17:42
EmilienMdmsimard: uh?17:43
*** dparkes has joined #tripleo17:43
*** suuuper has quit IRC17:43
EmilienMslagle: I proposed to switch fs010 (container-multinode) as a first iteration for now, as this job is run almost every time17:43
dmsimardEmilienM: just saying the task data might also be available through mqtt but it doesn't look like we're using it for v3 yet17:44
slagleEmilienM: ok, if you don't think we need coverage on that any longer17:44
weshay|rucktrozet, kind of.. I didn't realize those paths were changing. However they have.. they are now using the same directory structure as delorean17:44
trozetweshay|ruck: but ocata path still has stable17:45
EmilienMslagle: can you explain what do you mean by coverage? what coverage are you talking about?17:45
*** paramite has quit IRC17:45
EmilienMslagle: do you mean "classic overcloud deployment"?17:45
weshay|rucktrozet, ya next time it promotes it will look like pike17:46
slagleEmilienM: i mean what the job does today before the patch. that would no longer be covered (obviously) if it's changed to use config download17:46
trozetweshay|ruck: ok sounds like it will stay this way.  I'll change my patch and just remove stable.  thanks17:46
slagleEmilienM: so we've lost coverage on how we actually tell people to deploy conatiners17:47
slagleEmilienM: what other jobs even deploy containers? does ovb?17:47
EmilienMslagle: ok yeah, that's what I thought. So, I think we still have coverage on gate-tripleo-ci-centos-7-nonha-multinode-oooq for example but I agree this is not the exact same environment17:47
EmilienMslagle: gate-tripleo-ci-centos-7-ovb-containers-oooq17:47
EmilienMdprince: gate-tripleo-ci-centos-7-undercloud-containers-nv is green, you might want to propose a patch to make it voting, so we stop breaking it.17:48
EmilienMslagle: we still have gate-tripleo-ci-centos-7-scenario00x-multinode-oooq-container who run quite often every day, if we would break something I think we'll see it17:49
EmilienMslagle: so having said that, I would go with fs01017:49
slagleEmilienM: yea, it wfm, as I said,  if you don't think we need coverage on that any longer17:50
dprinceEmilienM: ack, thanks17:51
EmilienMslagle: cool - so let's targer fs010 until we have a mistral workflow17:53
EmilienMslagle: once we have it, maybe we can switch all the other jobs?17:53
*** links has quit IRC17:53
*** amoralej is now known as amoralej|off17:56
*** tosky has joined #tripleo17:58
*** fandrieu has quit IRC17:58
*** jmelvin has quit IRC17:59
slagleEmilienM: yea, i think once we have the mistral workflow ... it's a more transparent change, almost an implementation detail17:59
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates stable/pike: Fix some missed hard-coded network references
trozetweshay|ruck: i just realized...undercloud.qcow2 is missing18:02
trozetweshay|ruck: from
*** hrybacki is now known as hrybacki|trainin18:03
weshay|rucktrozet, which config/release file are you using?18:04
trozetweshay|ruck: im not using anything, just the url18:04
weshay|rucktrozet, so you don't need the undercloud if you have this set,
weshay|rucktrozet, let me look at the test jobs to see if someone botched something w/ the default release files18:06
*** cylopez has joined #tripleo18:06
trozetweshay|ruck: im not using oooqs18:06
*** salmankhan has quit IRC18:06
weshay|ruckk right but you guys were sharing the undercloud18:07
weshay|ruckso let me poke for a minute18:07
weshay|ruckI think I know what's up.. trown|lunch ping me when you are back18:07
*** trown|lunch is now known as trown18:07
trownweshay|ruck: back18:07
weshay|rucktrown, can you please jump on my blue18:08
trownweshay|ruck: sure, supposed to have 1-1 in 20min just fyi18:08
weshay|rucktrown, this won't take that long18:09
*** salmankhan has joined #tripleo18:09
*** shreshtha_ has joined #tripleo18:10
*** ooolpbot has joined #tripleo18:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1722864 in tripleo "OVB frequent timeouts on rh1 cloud" [Critical,Triaged]18:10
*** fzdarsky_ has quit IRC18:10
openstackgerritSai Sindhur Malleni proposed openstack/tripleo-heat-templates stable/newton: Bump fs.inotify.max_user_instances for scale
*** fzdarsky has quit IRC18:10
*** shreshtha has quit IRC18:11
*** sid1 has joined #tripleo18:12
*** salmankhan has quit IRC18:13
*** sid2 has joined #tripleo18:14
*** sid1 has quit IRC18:15
*** brault has joined #tripleo18:19
*** artom_ has joined #tripleo18:20
Tengujaosorior: for information, I'll try tomorrow to integrate the haproxy dynamic endpoint patch on the undercloud - I'll make a simple proxy for prometheus. Once this one is OK, I think it can be merged in master, then cherry-picked into stable/pike, and I'll provide another patch in order to support basic auth.18:20
*** pkovar has quit IRC18:23
*** artom has quit IRC18:23
*** brault has quit IRC18:23
*** d0ugal has quit IRC18:24
*** cylopez has quit IRC18:27
*** yolanda has quit IRC18:28
*** suuuper has joined #tripleo18:30
*** suuuper has quit IRC18:30
*** CaptTofu has quit IRC18:31
*** jaganathan has quit IRC18:32
*** jlabarre has quit IRC18:33
*** aditya_r has quit IRC18:33
*** vpickard is now known as vpickard_18:34
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates master: Fixes OpenDaylight deployments docker support
vkmchey guys, is the gate broken?18:36
*** tongl has quit IRC18:36
weshay|rucktrozet, I have to open a bug or two on this and start fixing it18:36
*** gfidente has quit IRC18:37
trozetweshay|ruck: ok cool.  In the meantime is it possible to manually upload the missing file to the repo?18:37
weshay|rucktrozet, we can ask dmsimard but we don't have the creds to the image server18:37
weshay|ruckbit of a disconnect between ci and infra there18:38
trozetweshay|ruck: ok.  Can you include me on the bugs please?18:38
trozetdmsimard: can you help me out?18:38
trozetweshay|ruck: thanks18:38
mwhahahaslagle, bnemec: When you get a minute can you review
EmilienMsshnaidm, trown : please review when you have time
EmilienMvkmc: we had some outage, should be back to normal. Do you have an example?18:40
*** dbecker has quit IRC18:41
EmilienMvkmc: October 5, yes you want to recheck. A bunch of things were fixed18:41
vkmcEmilienM, very well, I triggered the recheck a few hours ago18:42
vkmcwaiting for the results then18:42
dmsimardtrozet: reading, hang on18:45
dmsimardtrozet: upload what where ?18:45
*** rbrady has quit IRC18:45
trozetdmsimard: the missing undercloud.qcow2 file to
*** MaxPC has quit IRC18:48
openstackgerritAlex Schultz proposed openstack/tripleo-specs master: Remove python3 squad
*** milan has quit IRC18:51
*** rwsu has quit IRC18:51
dmsimardtrozet: well, the file is definitely not there18:53
dmsimardtrozet: I can put a file there, but I need to know where it would be18:53
*** yolanda has joined #tripleo18:54
dmsimardweshay|ruck, trown ^ was there an issue with image upload or something ?18:54
*** aputtur has joined #tripleo18:54
weshay|ruckdmsimard, no.. we created a bug that trozet just hit18:54
weshay|ruckand our jobs18:54
*** aputtur has quit IRC18:55
weshay|ruckdmsimard, we could do a one off and upload something for trozet but it's probably better to wait for the proper fix unless we're causing too much pain18:55
dmsimardweshay|ruck: so there was no issue with the upload or the symlink but there's a missing file ? o_O18:55
weshay|ruckI'm about to open a bug, details will be there18:56
dmsimardweshay|ruck: just tell me what to do and I'll do it :p18:56
dmsimardweshay|ruck: if you know what the hash before that one was, I can change the symlink18:56
weshay|ruckur fine right now, trozet was just asking if I could upload something for him18:56
* weshay|ruck goes to write a bug on the ci team18:56
dmsimardwhatever works, we can either upload something there or revert to a prior hash18:57
dmsimardping me if you want to do something18:57
*** milan has joined #tripleo18:57
trozetweshay|ruck, dmsimard: yeah would be good if you can manually fix it for the time being.  trying to test something and it kind of blocks me18:57
weshay|ruckthat will take a little time18:58
*** akrivoka has quit IRC18:58
trowndmsimard: could you give access to the image server and I can fix it ... and we can disable further promotes until we fix the bug weshay|ruck is filing?18:59
*** milan has quit IRC18:59
trownEmilienM: do we have any job actually showing that patch working?18:59
trownEmilienM: a little hard to review that... seems fine in that it wont break anything, but pretty hard to say whether it works19:00
*** jlabarre has joined #tripleo19:01
*** MaxPC has joined #tripleo19:04
*** brault has joined #tripleo19:09
*** ooolpbot has joined #tripleo19:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1722864 in tripleo "OVB frequent timeouts on rh1 cloud" [Critical,Triaged]19:10
*** brault has quit IRC19:13
EmilienMtrown: yes, in
*** brault has joined #tripleo19:22
EmilienMtrown: fs010 is container-multinode19:26
*** milan has joined #tripleo19:34
*** ykarel|afk has quit IRC19:50
trownEmilienM: thanks19:50
EmilienMtrown: seeking your feedback on the way we did in oooq and oooq-extras19:51
EmilienMtrown: we document the WIP here: and also the tech debt19:51
EmilienMtrown: since it's WIP, it will evolved during the next weeks in iterations19:52
trownEmilienM: ya seems implemented differently than most quickstart stuff19:54
trownEmilienM: seems like it should be done with more upfront collaboration from CI squad as well19:54
trownEmilienM: like bringing it up to ci squad at some point before ... hey review this :P19:54
EmilienMtrown: I guess we wanted a prototype working in CI before using your time19:56
EmilienMtrown: now we have something that work, we want to use your time and see what we did and see how we can do better19:56
EmilienMtrown: I would be happy to work on next patchsets with CI squad19:57
*** jprovazn has quit IRC19:57
*** liverpooler has quit IRC20:00
*** itlinux has joined #tripleo20:03
*** rbrady has joined #tripleo20:06
*** rbrady has joined #tripleo20:06
*** apetrich has quit IRC20:07
mwhahahafultonj: looks like scenario001-containers might be broken again
EmilienMtrown: to be honest, our prototype in oooq-extras is 70 lines of code, so it's not a big deal that we didn't interrupt you for that - we wanted a PoC, we made it working - now seeking for how to make it better20:07
EmilienMtrown: it would have been bad to implement a PoC of 300 lines of code or more and ask you to review20:08
EmilienMtrown: but that's not the case, our PoC is really basic, just some tasks and some doc - that we're happy to rethink20:08
fultonjmwhahaha: looking20:09
*** tongl has joined #tripleo20:09
*** apetrich has joined #tripleo20:10
*** ooolpbot has joined #tripleo20:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1722864 in tripleo "OVB frequent timeouts on rh1 cloud" [Critical,Triaged]20:10
*** tbonds has joined #tripleo20:10
fultonjmwhahaha: indeed ceph is broken there, no OSD, I am checking to see if container and ceph-ansible version are out of sync20:14
*** toure is now known as toure_biab20:18
*** jmelvin has joined #tripleo20:18
*** achadha_ has joined #tripleo20:21
*** achadha_ has quit IRC20:21
fultonjmwhahaha: the fix is to merge i can explain20:21
fultonjgfidenete isn't here20:21
*** achadha_ has joined #tripleo20:22
fultonjwe use the centos-storage-sig's -release repo to provide the ceph-ansible rpm20:22
*** achadha_ has quit IRC20:23
fultonja new ceph-ansible was recently promoted from -candidate to -release20:23
*** achadha_ has joined #tripleo20:23
fultonjit requires the new docker container (which will default) once we don't hard code it20:24
fultonjmwhahaha: what doyou think?20:24
mwhahahafultonj: sec meeting20:25
*** achadha has quit IRC20:25
fultonjthe alternative is to undo the promotion but we want the promotion ; we just landed tings out of sync breaking the scenario while we wait for the other20:25
* fultonj waits20:25
*** sid2 has quit IRC20:26
*** milan has quit IRC20:26
fultonjmwhahaha: i will make a bug documenting all this20:27
*** achadha_ has quit IRC20:27
fultonjeven when we fix we need better process20:28
*** fragatina has joined #tripleo20:31
*** achadha has joined #tripleo20:35
openstackLaunchpad bug 1722908 in tripleo "scenario001-containers fail on tempest run; missing OSD from ceph-ansible and ceph-docker being out of sync" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)20:36
weshay|rucktrozet, fyi you may be able to use
weshay|rucktrown, ^ fyi20:36
trozetweshay|ruck: trown said cbs was broken for a long time with a bug where it would not update.  Is that fixed now or should we keep using rdo images server?20:38
*** achadha has quit IRC20:39
*** dprince has quit IRC20:40
weshay|ruckI know it's very slow, but it's better than nothing atm20:40
weshay|rucknot sure if it's busted20:40
openstackgerritIhar Hrachyshka proposed openstack/tripleo-quickstart-extras master: Revert "Adding test_mtu_sized_frames to skip list"
*** gbarros has quit IRC20:47
openstackgerritwes hayutin proposed openstack/tripleo-quickstart master: update release config to use overcloud-as-undercloud
*** jkilpatr has quit IRC20:50
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: fs10: deploy steps with ansible
*** pchavva has quit IRC20:50
*** rodolof has joined #tripleo20:55
*** ccamacho has quit IRC20:55
dmsimardtrown: I totally missed your ping from earlier20:57
dmsimardstill need something done ?20:57
*** trown is now known as trown|outtypewww20:59
*** lblanchard1 has quit IRC20:59
*** dprince has joined #tripleo21:02
*** dprince has quit IRC21:02
*** catintheroof has quit IRC21:02
*** rodolof has quit IRC21:02
*** leitan has quit IRC21:03
*** rodolof has joined #tripleo21:03
mwhahahafultonj: so should we restore the unpin?21:04
fultonjmwhahaha: why not unabandon ?21:04
mwhahahayea that's what i mean21:05
fultonjmwhahaha: thanks21:05
mwhahaha never merged tho21:05
mwhahahathat might hav ebeen the problem21:05
fultonjit's true21:06
fultonjtechnically it doesn't have to depend on it21:06
fultonji can revise and remove the depends-on21:06
mwhahahafultonj: please do thanks21:06
fultonjmwhahaha: ok, coming up21:06
*** dprince has joined #tripleo21:06
*** rodolof has quit IRC21:08
*** rodolof has joined #tripleo21:08
*** achadha has joined #tripleo21:09
*** ooolpbot has joined #tripleo21:10
openstackLaunchpad bug 1722864 in tripleo "OVB frequent timeouts on rh1 cloud" [Critical,Triaged]21:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1722908 in tripleo "scenario001-containers fail on tempest run; missing OSD from ceph-ansible and ceph-docker being out of sync" [Critical,Triaged] - Assigned to John Fulton (jfulton-org)21:10
*** mwhahaha changes topic to "CI Status: YELLOWish - scenario001-containers and ovb timeouts (see alerts) | |"21:10
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates master: Hardcode tag-stable-3.0-jewel-centos-7 in scenario001-containers
*** florianf has quit IRC21:12
*** rodolof has quit IRC21:13
*** rodolof has joined #tripleo21:14
*** jtomasek has quit IRC21:14
*** achadha has quit IRC21:14
*** abishop has quit IRC21:15
fultonjmwhahaha: I went with instead ; explanation in comment #3 of bug21:16
mwhahahafultonj: so should I re-abandon the other one?21:17
fultonjmwhahaha: no, please don't21:18
fultonjmwhahaha: i like the other, but 511338 is quicker21:18
fultonjmerging two vs one21:18
fultonjbut two is good for long run21:18
fultonjhard code only in repos generating tool21:18
*** dprince has quit IRC21:19
fultonjmwhahaha: i am going to open another bug about improving the process21:19
mwhahahafultonj: sounds good21:19
fultonjit's about lock-stepping and centos-storage sig promotions21:20
fultonja hard problem to solve21:20
fultonjso perhaps hard coding in CI is the best fix21:20
fultonjthey don't always become incompatible but this is a case where they did21:21
*** MrRoot has joined #tripleo21:23
trozetweshay|ruck: did you ever file a bug?21:27
*** rodolof has quit IRC21:27
*** jkilpatr has joined #tripleo21:27
*** rodolof has joined #tripleo21:27
weshay|rucktrozet, yup21:28
* weshay|ruck gets21:28
openstackLaunchpad bug 1722885 in tripleo "default release configs in config/release point to an unavailable undercloud image" [Critical,In progress] - Assigned to wes hayutin (weshayutin)21:28
openstackLaunchpad bug 1722889 in tripleo "tripleo-quickstart public pipelines no longer generate an undercloud image" [High,Triaged]21:29
*** cdearborn has quit IRC21:30
*** ecerquei has quit IRC21:30
fultonjmwhahaha: do you need me to hang around right now to see that bug through?21:32
mwhahahafultonj: no we'll just see how the results come back later21:32
trozetweshay|ruck: thanks21:32
fultonjok, i'll check back later21:32
fultonji'm just going to go AFK for a few hours as nothing else I can do til then21:32
mwhahahafultonj: might send giulio an email about it and maybe he can look at it in the morning21:33
fultonjyes, i will21:33
fultonji can cc you21:33
* mwhahaha has to go afk for a bit21:33
mwhahahasounds good21:33
*** gbarros has joined #tripleo21:33
*** rodolof has quit IRC21:33
*** rodolof has joined #tripleo21:33
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add the option to run the container-check script
*** rodolof has quit IRC21:38
*** rodolof has joined #tripleo21:38
*** rodolof has quit IRC21:43
*** rodolof has joined #tripleo21:43
*** rodolof has quit IRC21:48
*** rodolof has joined #tripleo21:49
*** leitan has joined #tripleo21:50
*** gbarros has quit IRC21:53
*** threestrands has joined #tripleo21:56
*** jkilpatr has quit IRC21:57
*** jkilpatr has joined #tripleo21:57
*** rodolof has quit IRC21:58
*** rodolof has joined #tripleo21:58
*** rodolof has quit IRC22:03
*** rodolof has joined #tripleo22:04
EmilienMweshay|ruck: please join #openstack-infra-incident22:07
*** rlandy has quit IRC22:08
*** ooolpbot has joined #tripleo22:10
openstackLaunchpad bug 1722864 in tripleo "OVB frequent timeouts on rh1 cloud" [Critical,Triaged]22:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1722908 in tripleo "scenario001-containers fail on tempest run; missing OSD from ceph-ansible and ceph-docker being out of sync" [Critical,In progress] - Assigned to John Fulton (jfulton-org)22:10
*** bfournie has quit IRC22:11
*** bobh has quit IRC22:11
*** fzdarsky_ has joined #tripleo22:13
*** fzdarsky has joined #tripleo22:14
*** rodolof has quit IRC22:16
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Collect mistral-ansible execution files from /tmp"
EmilienMweshay|ruck: ^22:16
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: Revert "Collect mistral-ansible execution files from /tmp"
*** MaxPC has quit IRC22:17
dmsimardEmilienM, weshay|ruck: with the /tmp/ansible fire extinguished, I think it would be a nice improvement opportunity to avoid completely globbing /var/log and /etc by default in
dmsimardand pick the directories we're really interested in22:18
dmsimardwe do this in puppet-openstack-integration
weshay|ruckEmilienM, k.. re:
weshay|ruckEmilienM, k.. I thought we did that but apparently not22:19
openstackgerritSteve Baker proposed openstack/tripleo-common master: Action to always populate container image parameters
openstackgerritMerged openstack-infra/tripleo-ci master: Revert "Collect mistral-ansible execution files from /tmp"
openstackgerritMonty Taylor proposed openstack-infra/tripleo-ci master: Stop collecting ephemeral temp dirs
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: remove the collection of /tmp/* from upstream jobs
weshay|ruckEmilienM, ^22:24
EmilienMweshay|ruck: thanks22:25
openstackgerritMerged openstack-infra/tripleo-ci master: remove the collection of /tmp/* from upstream jobs
openstackgerritPradeep Kilambi proposed openstack/python-tripleoclient master: Support to drive undercloud deploy via undercloud.conf
*** pmannidi has joined #tripleo22:37
*** kbyrne has quit IRC22:48
*** kbyrne has joined #tripleo22:50
*** jcoufal has quit IRC22:53
openstackgerritLars Kellogg-Stedman proposed openstack/tripleo-specs master: Add blueprint for Logging to stdout and rsyslog
*** bfournie has joined #tripleo22:59
*** fzdarsky has quit IRC23:00
*** fzdarsky_ has quit IRC23:00
*** bfournie has quit IRC23:00
*** bfournie has joined #tripleo23:01
*** noslzzp has quit IRC23:04
*** dsneddon_afk has quit IRC23:06
*** dsneddon has joined #tripleo23:07
*** dsariel has quit IRC23:09
*** ooolpbot has joined #tripleo23:10
openstackLaunchpad bug 1722864 in tripleo "OVB frequent timeouts on rh1 cloud" [Critical,Triaged]23:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1722908 in tripleo "scenario001-containers fail on tempest run; missing OSD from ceph-ansible and ceph-docker being out of sync" [Critical,In progress] - Assigned to John Fulton (jfulton-org)23:10
openstackgerritAndy Smith proposed openstack/tripleo-heat-templates master: WIP Separate rpc and notify messaging backends
pabelangerEmilienM: at this point, you might consider abandoning the gate. That will give logs.o.o time to recover, for now, the pipeline is red23:19
*** morazi has quit IRC23:21
*** fandrieu has joined #tripleo23:24
*** tbonds has quit IRC23:26
EmilienMpabelanger: what does it mean abandon the gate?23:30
* EmilienM afk23:37
*** myoung|remote has joined #tripleo23:42
*** tosky has quit IRC23:43
*** myoung|remote has quit IRC23:43
*** fragatina has quit IRC23:52
*** itlinux has quit IRC23:57

Generated by 2.15.3 by Marius Gedminas - find it at!