Thursday, 2018-11-29

*** bfournie has joined #tripleo00:03
*** wolverineav has joined #tripleo00:04
*** slaweq has quit IRC00:05
*** Baggypants12000 has quit IRC00:10
openstackgerritYossi Ovadia proposed openstack/python-tripleoclient master: increasing the workflow timeout from 300 to 600 required in case of relatively large scale on physical servers  https://review.openstack.org/62075400:18
*** artom has quit IRC00:36
*** ansmith has joined #tripleo00:41
*** artom has joined #tripleo00:45
*** tosky has quit IRC00:50
*** jistr has quit IRC01:00
*** jistr has joined #tripleo01:01
*** pbandark has quit IRC01:09
*** zaneb has quit IRC01:10
*** boazel has joined #tripleo01:19
*** boazel_ has quit IRC01:22
*** mjturek has quit IRC01:28
*** etingof has quit IRC01:45
*** wolverineav has quit IRC02:21
*** mrsoul has joined #tripleo02:21
*** dsneddon has quit IRC02:23
*** mschuppert has quit IRC02:23
*** wolverineav has joined #tripleo02:24
*** wolverineav has quit IRC02:28
*** eck` is now known as eck`gone02:34
openstackgerritDan Radez proposed openstack/tripleo-heat-templates stable/queens: Add networking-ansible ML2 plugin support  https://review.openstack.org/62076802:36
*** dsneddon has joined #tripleo02:36
openstackgerritshanks yang proposed openstack/tripleo-heat-templates master: Correct the wrong word  https://review.openstack.org/62076902:37
*** dsneddon has quit IRC02:41
tonybI'm trying to debug an install where tripleo.parameters.update looks to fail and all I'm getting is: .../v2/action_executions: ('Connection aborted.', BadStatusLine("''",))02:41
tonybany idea where I can look to work out where it died?02:41
*** ramishra has joined #tripleo02:53
*** radeks has joined #tripleo02:55
tonybOh 'ERROR: The Parameter (GlanceBackend) was not defined in template' Now to work out why02:55
*** rcernin has quit IRC02:58
*** ramishra has quit IRC03:04
*** ramishra has joined #tripleo03:06
*** apetrich has quit IRC03:16
*** rlandy|bbl is now known as rlandy03:18
*** dsneddon has joined #tripleo03:25
*** dsneddon has quit IRC03:30
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common master: Derives NUMA aware vSwitches parameters  https://review.openstack.org/61715803:31
openstackgerritJaganathan Palanisamy proposed openstack/tripleo-common master: Derives NUMA aware vSwitches parameters  https://review.openstack.org/61715803:32
*** rcernin has joined #tripleo03:38
openstackgerritMerged openstack/diskimage-builder master: Fix unit tests for elements  https://review.openstack.org/61938703:53
*** jaganathan has joined #tripleo03:57
*** dsneddon has joined #tripleo04:30
*** dsneddon has quit IRC04:35
*** udesale has joined #tripleo05:06
*** janki has joined #tripleo05:09
*** tzumainn has quit IRC05:12
*** ykarel|away has joined #tripleo05:20
jankichkumar|away, hey05:22
chkumar|awayjanki: \o/05:28
jankichkumar|away, is this persistent http://logs.openstack.org/19/618719/1/gate/tripleo-puppet-ci-centos-7-undercloud-containers/5d7365a/job-output.txt.gz#_2018-11-28_17_00_11_804010?05:29
chkumar|awayjanki: the job got timedout05:30
chkumar|awayjanki: I need to recheck why it got timedout05:31
chkumar|awayjanki: can you do a recheck on the review05:33
chkumar|away?05:33
jankichkumar|away, ohk05:37
Tenguhello tere :)05:38
*** ratailor has joined #tripleo05:51
*** jaganathan has quit IRC06:00
*** ykarel|away has quit IRC06:02
*** redrobot has quit IRC06:02
*** redrobot has joined #tripleo06:04
*** saneax has joined #tripleo06:05
*** ykarel|away has joined #tripleo06:16
*** ykarel|away is now known as ykarel|lunch06:20
*** radeks has quit IRC06:20
*** ccamacho has quit IRC06:21
*** radeks has joined #tripleo06:26
*** dsneddon has joined #tripleo06:31
*** dsneddon has quit IRC06:36
openstackgerritRabi Mishra proposed openstack/python-tripleoclient master: Don't print message in payload if None  https://review.openstack.org/62078406:39
openstackgerritMerged openstack/python-tripleoclient master: Improve bulk symlink error handling  https://review.openstack.org/61936306:57
openstackgerritMerged openstack/tripleo-common master: Specify python interpreter for undercloud  https://review.openstack.org/61935706:57
*** etingof has joined #tripleo07:00
*** mschuppert has joined #tripleo07:00
*** ahosam has joined #tripleo07:02
*** ksambor has joined #tripleo07:03
*** yprokule has joined #tripleo07:03
*** slaweq has joined #tripleo07:03
*** jfrancoa has joined #tripleo07:03
*** quiquell|off is now known as quiquell07:07
*** marios|rover has joined #tripleo07:11
*** ccamacho has joined #tripleo07:12
*** chkumar|away has quit IRC07:13
*** chandan_kumar has joined #tripleo07:15
*** jaganathan_ has joined #tripleo07:20
*** marios_|ruck has joined #tripleo07:25
*** jaganathan__ has joined #tripleo07:25
*** rascasoft has quit IRC07:25
*** jaganathan_ has quit IRC07:26
*** apetrich has joined #tripleo07:26
*** ahosam has quit IRC07:26
*** ahosam has joined #tripleo07:26
hjensas2018-11-29 06:55:12.746504 | primary | + /home/zuul/workspace/.quickstart/bin/ansible-playbook /home/zuul/workspace/.quickstart/playbooks/collect-logs.yml -vv --extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-7/master.yml --extra-vars @/home/zuul/src/git.openstack.org/openstack/tripleo-quickstart/config/general_config/featureset-multinode-common.yml --extra-vars @/home/zuul/src/git.openstack.org/openstack/tripleo-quickstart07:28
hjensas/config/general_config/featureset010.yml --extra-vars @/home/zuul/src/git.openstack.org/openstack-infra/tripleo-ci/toci-quickstart/config/testenv/multinode.yml --extra-vars local_working_dir=/home/zuul/workspace/.quickstart --extra-vars virthost=127.0.0.2 --inventory /home/zuul/workspace/.quickstart/hosts --extra-vars tripleo_root=/home/zuul/src/git.openstack.org/openstack --extra-vars working_dir=/home/zuul --skip-tags tripleo-validations,teardown-all07:28
hjensas--extra-vars @/home/zuul/src/git.openstack.org/openstack/tripleo-ci/toci-quickstart/config/collect-logs.yml --extra-vars artcl_collect_dir=/home/zuul/workspace/logs --extra-vars @/home/zuul/workspace/logs/zuul-variables.yaml --tags all07:28
hjensasooops, how long does normally the collect logs playbook take to run?07:28
*** hakhande_ has joined #tripleo07:29
*** rcernin has quit IRC07:35
*** ccamacho has quit IRC07:39
*** kopecmartin|off is now known as kopecmartin07:39
*** rascasoft has joined #tripleo07:40
*** cylopez has joined #tripleo07:42
openstackgerritRadoslaw Smigielski proposed openstack/paunch master: Default log location failes for non-root users  https://review.openstack.org/61801407:50
*** skramaja_ has joined #tripleo07:51
dtantsur|mtgdtrainor: what you describe is how it is expected to work, yes. the only thing to be noted is that with this approach there is no "generic" flavor (like "baremetal"). What is in flavor must match what is in node.07:57
*** skramaja_ is now known as skramaja07:57
*** florianf|afk is now known as florianf07:57
*** ccamacho has joined #tripleo08:06
*** gkadam has joined #tripleo08:07
*** ccamacho has quit IRC08:07
*** ccamacho has joined #tripleo08:08
openstackgerritCédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get dedicated logging file for HAProxy  https://review.openstack.org/62060108:10
openstackgerritCédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get dedicated logging file for HAProxy  https://review.openstack.org/62060108:10
openstackgerritChristian Schwede proposed openstack/tripleo-heat-templates master: Fix Swift S3 API configuration  https://review.openstack.org/62065008:13
*** quiquell is now known as quiquell|brb08:13
openstackgerritDmitry Tantsur proposed openstack/tripleo-heat-templates master: Enable ironic-inspector support in ironic when enabling ironic-inspector  https://review.openstack.org/62080108:19
openstackgerritCédric Jeanneret proposed openstack/puppet-tripleo master: Allow to set log facility for HAProxy  https://review.openstack.org/62080208:21
*** yprokule has quit IRC08:22
*** yprokule has joined #tripleo08:23
*** jpena|off is now known as jpena08:31
*** ykarel|lunch is now known as ykarel08:31
*** amoralej|off is now known as amoralej08:31
*** dsneddon has joined #tripleo08:32
*** dsneddon has quit IRC08:37
*** quiquell|brb is now known as quiquell08:37
openstackgerritMerged openstack/tripleo-docs master: [Documentation] Fix link bugs.  https://review.openstack.org/62061508:37
*** hberaud|gone is now known as hberaud08:38
openstackgerritMarios Andreou proposed openstack/tripleo-docs master: RHSM: some updates for Rocky  https://review.openstack.org/62038208:39
*** bogdando has joined #tripleo08:40
openstackgerritMarios Andreou proposed openstack/tripleo-docs master: RHSM: some updates for Rocky  https://review.openstack.org/62038208:43
*** pcaruana has joined #tripleo08:48
*** Baggypants12000 has joined #tripleo08:48
*** apetrich has quit IRC08:52
*** dsneddon has joined #tripleo09:00
*** cfontain has joined #tripleo09:01
*** tosky has joined #tripleo09:03
*** shardy has joined #tripleo09:04
*** gfidente has joined #tripleo09:06
*** cfontain has quit IRC09:06
*** cfontain has joined #tripleo09:06
*** saneax has quit IRC09:07
*** ahosam has quit IRC09:10
*** ahosam has joined #tripleo09:10
dtantsur|mtga one-line fix if anybody has time: https://review.openstack.org/#/c/617987/09:12
*** shardy has quit IRC09:13
*** dsneddon has quit IRC09:14
*** dmacpher has joined #tripleo09:14
openstackgerritCédric Jeanneret proposed openstack/puppet-tripleo master: Allow to set log facility for HAProxy  https://review.openstack.org/62080209:14
*** jpich has joined #tripleo09:15
*** dsneddon has joined #tripleo09:15
openstackgerritYurii Prokulevych proposed openstack/tripleo-upgrade master: Assert workloads are spawned in different stages  https://review.openstack.org/60067409:19
*** mrunge has quit IRC09:21
marioscan we get another +2 here please folks we've decided to merge the envs to make the jobs testing easier this is for scen 2 https://review.openstack.org/#/c/618537/ (more context in the commit message, moving scenarios to standalone jobs)09:22
mariosthanks09:22
*** derekh has joined #tripleo09:22
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates master: Use correct permission on Gnocchi tmp dir  https://review.openstack.org/62083309:23
*** mrunge has joined #tripleo09:24
quiquellmarios: also scenario004 https://review.openstack.org/#/c/619504/09:25
*** paramite|clone is now known as mmagr09:25
quiquellTengu, bogdando, jfrancoa, ccamacho: ^ to get standalone + scenario00409:25
mariosthanks quiquell09:29
TenguI'll check that shortly :)09:30
quiquellmarios: only scn004 is missing all of them has proper votes now09:32
openstackgerritJanki Chhatbar proposed openstack/tripleo-heat-templates master: Don't mount data folder  https://review.openstack.org/62005309:32
openstackgerritCédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get dedicated logging file for HAProxy  https://review.openstack.org/62060109:33
*** holser_ has joined #tripleo09:33
Tengujbadiapa: -^^  complement to the other one allowing to set log facility for haproxy :).09:34
Tengupretty happy if it works :)09:34
bogdandoPTAL https://review.openstack.org/#/c/613300/09:38
bogdandojistr, jaosorior: ^^ please09:38
*** apetrich has joined #tripleo09:43
*** ssbarnea has quit IRC09:45
*** ssbarnea|rover has joined #tripleo09:45
*** shardy has joined #tripleo09:51
shardyMorning all, I need a second reviewer for the last two patches in this series https://review.openstack.org/#/q/topic:bug/1792613(+status:open)10:00
shardymuch appreciated if anyone has a moment to take a look :)10:00
Tengubandini: hey, are your here? :)10:01
*** jchhatbar has joined #tripleo10:02
openstackgerritRabi Mishra proposed openstack/tripleo-heat-templates master: Mount config-data/puppet-generated/nova for nova_api_ensure_default_cell  https://review.openstack.org/62086810:02
*** dsneddon has quit IRC10:03
*** skramaja_ has joined #tripleo10:04
*** janki has quit IRC10:05
*** hakhande_ has quit IRC10:05
*** skramaja has quit IRC10:05
*** numans_ has joined #tripleo10:05
*** numans has quit IRC10:05
*** hakhande_ has joined #tripleo10:05
jaosoriorHey folks! Do we have any validations that check the overcloud heat parameters that were given?10:06
*** ahosam has quit IRC10:06
shardyjaosorior: yes there are some, and heat itself does validation, what were you looking for specifically?10:08
jaosoriorshardy: I wanted to start checking that, if a certain feature is enabled (TLS everywhere), we should issue more explicit warnings for things that are easy to check (such as the DNS server not pointing to FreeIPA and stuff like this)10:08
openstackgerritMerged openstack/os-apply-config stable/pike: fixes how os-apply-config handles invalid json  https://review.openstack.org/62020710:10
shardyjaosorior: ack, well the closest to that I think is where we check for deprecated parameters, then warn the user https://github.com/openstack/tripleo-common/blob/master/workbooks/plan_management.yaml#L58510:12
shardyso we could have a more general case of that I guess10:12
*** akrivoka has joined #tripleo10:12
shardyjaosorior: it'd be nice if we could somehow express those constraints in the service templates though, vs hard-coding in the workbook10:12
shardywe'd probably have to invent some new key in the role_data to do that though I guess10:13
*** saneax has joined #tripleo10:13
shardyhttps://github.com/openstack/tripleo-common/blob/master/workbooks/derive_params.yaml has some similar patterns too I think10:15
jaosoriorTengu, florianf: The work that was posponed regarding validations. It was gonna be driven by playbooks and we could potentially pass the deployment's heat paramters there, right? similar to what derive_params.yaml does above ^^10:16
florianfjaosorior: The postponed work is about the CLI validations framework, but even if that is delayed, we can already add more validations that are run through the UI/mistral/ansible-playbook.10:23
jaosoriorflorianf: are there any validations that already can check the heat parameters that were given for a deployment?10:24
florianfjaosorior: we already have a dns check, but it's very generic and doesn't check any specifics (like freeIPA)10:24
jaosoriorflorianf: right, but I want to only check that IF TLS everywhere is enabled and the CA is FreeIPA10:24
jaosoriorwhich is specified via heat parametesr10:25
florianfjaosorior: nope, not yet, we currently check on a template level (network env for instance), but there is a patch up for review that fetches params from heat to use those for validations.10:25
jaosoriorflorianf: can you pass me the review?10:25
florianfjaosorior: sec...10:26
*** udesale has quit IRC10:27
florianfjaosorior: https://review.openstack.org/#/c/575121/ and https://review.openstack.org/#/c/547518/10:27
*** ahosam has joined #tripleo10:31
*** jchhatba_ has joined #tripleo10:37
*** akrivoka has quit IRC10:37
*** akrivoka has joined #tripleo10:37
*** hakhande_ has quit IRC10:39
jistrhi folks, anyone seen these in their overcloud pcmk recently?10:39
*** skramaja has joined #tripleo10:39
*** numans_ has quit IRC10:39
jistrgalera-bundle-docker-0_start_0 on overcloud-controller-2 'unknown error' (1): call=218, status=complete, exitreason='failed to pull image 192.168.24.1:8787/tripleomaster/centos-binary-mariadb:pcmklatest',10:39
jistr    last-rc-change='Thu Nov 29 10:19:17 2018', queued=0ms, exec=194ms10:39
*** skramaja_ has quit IRC10:39
*** jchhatbar has quit IRC10:39
*** hakhande_ has joined #tripleo10:39
*** gfidente has quit IRC10:39
* jistr retrying deploy to see if it was a one time failure10:40
*** cylopez has quit IRC10:41
*** shardy has quit IRC10:41
*** shardy has joined #tripleo10:43
*** numans has joined #tripleo10:44
florianfshardy: quick question: Do we seriously support using something other than /home/stack as the home dir or config dir in tripleo?10:44
amoralejmwhahaha, EmilienM https://review.rdoproject.org/r/#/c/17341 updating puppet, let us know if we hit any issue10:45
openstackgerritMerged openstack/tripleo-heat-templates master: Remove unnecessary logs due to misconfiguration from pacemaker  https://review.openstack.org/61304310:45
openstackgerritMerged openstack/tripleo-heat-templates master: Add ctlplane interface routes  https://review.openstack.org/60494310:45
florianfshardy: by "seriously" I mean if it's something that we support in theory but that doesn't really happen in the real world.10:48
shardyflorianf: yes it could be any home directory10:52
shardyor any directory for that matter10:52
shardythe stack user is a tripleo-quickstart and/or devstack thing AFAIK10:53
therveflorianf: CI doesn't use stack AFAIR10:53
shardytherve: yeah true, good point10:53
shardyit's just a default10:53
florianfshardy, therve: thanks, that's very useful information!10:56
*** aedc has joined #tripleo10:57
sri_shardy, Hi if you have min, I facing a very wired error with my overcloud compute nodes(particularly with Dell servers ), I am using nic4 for provisioning, overcloud created is failing at compute node NetworkDeployment, I've logged into the compute node to see what was the problem I've found in one of the compute node nic4 comes  in order 3: em4,  this is the first time i am using Dell servers, you can find complete information  http://paste.openstack.org/sho10:59
sri_w/736386/10:59
sri_shardy, http://paste.openstack.org/show/736386/10:59
shardysri_: I'd run os-net-config again with --debug (and maybe --noop) so you can see the sorted order of nics (which may not match ip a)11:06
shardysri_: you can then either adjust the nicN value in the templates to match, use the real interface name e.g em4, or write a mapping file11:07
shardysri_: note that the order is a sorted list of the active (link up) nics11:07
shardyhttps://github.com/openstack/os-net-config/blob/master/os_net_config/utils.py#L21411:07
shardysri_: ^^ that output should help see the order used internally11:08
sri_shardy, ack, i will try to use real interface names, how do i write a mapping file ? any links11:10
shardysri_: https://github.com/openstack/os-net-config/blob/master/etc/os-net-config/samples/mapping.yaml11:11
shardysri_: it's just a yaml file, you can write it via cloud-init or the PreNetworkConfig hook:11:12
*** vpickard_ is now known as vpickard11:12
shardyhttps://github.com/openstack/tripleo-heat-templates/blob/master/firstboot/os-net-config-mappings.yaml11:13
shardysri_: that's a more complex example for hetrogeneous hardware, but you could use a similar approach to write a simple file11:13
shardyif all machines are the same just using the real ifname in the templates is probably easier11:13
openstackgerritCédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get dedicated logging file for HAProxy  https://review.openstack.org/62060111:14
sri_shardy, understood, I've two types of hardware supermicro for controller nodes and dell for compute nodes11:15
shardysri_: Ack - the mapping approach is mostly useful if you've got different nic setups in the same group e.g compute etc11:16
sri_shardy, got it11:16
sri_shardy, Thank you :)11:17
shardynp! :)11:18
openstackgerritNatal Ngétal proposed openstack/tripleo-docs master: [Documentation] Specify the name of the irc server.  https://review.openstack.org/62088711:26
ssbarnea|roverjistr: can you please workflow this doc fix https://review.openstack.org/#/c/619518/ ?11:26
gchamoultherve, shardy: in tripleo-validations we relied on the TRIPLEO_UNDERCLOUD_CONF_FILE variable coming from hiera for some check. But this hiera var has been lost with the instack-undercloud retirement. Do you know if we may get this info from a different way?11:29
quiquellssbarnea|rover: Have done it, it's only doc11:29
quiquellwell Â"only"11:29
*** morazi has quit IRC11:30
shardygchamoul: sounds like we need to set the same hiera in t-h-t, do you happen to know what the key was?11:30
*** jchhatbar has joined #tripleo11:30
gchamoulshardy: the key was tripleo_undercloud_conf_file11:31
*** skramaja_ has joined #tripleo11:33
*** numans has quit IRC11:33
*** hakhande_ has quit IRC11:33
*** skramaja has quit IRC11:33
*** jchhatba_ has quit IRC11:33
openstackgerritMike Fedosin proposed openstack/tripleo-heat-templates master: Allow to skip docker reconfiguration during stack update  https://review.openstack.org/62062111:34
*** salmankhan has joined #tripleo11:35
shardygchamoul: ack - could you raise a bug? IMO it's a regression, and it should be fairly easy to get the path in tripleoclient and pass it to t-h-t where we can set the expected hiera key11:38
shardybogdando: ^^ FYI11:38
bogdandosounds like a bug indeed11:39
gchamoulshardy: ack thanks11:40
shardyIMO we should move the extraconfig/(pre|post)_deploy stuff in t-h-t to a composable service template, e.g puppet/services/undercloud.yaml then we could capture all the special configuration of the undercloud in one place11:40
shardyif we did that it'd be easy to add an UndercloudConfigPath parameter and set the hiera via config_settings11:41
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates stable/rocky: Remove unnecessary logs due to misconfiguration from pacemaker  https://review.openstack.org/62089011:41
openstackgerritMerged openstack/diskimage-builder master: Fix a typo in the help message of disk-image-create  https://review.openstack.org/61967911:42
ssbarnea|roverquiquell++11:42
quiquell:-)11:42
*** artom has quit IRC11:42
quiquellI needed one more today11:42
gchamoulshardy, bogdando: oh Sofer started to work on that https://review.openstack.org/#/c/614470/11:42
*** jchhatbar has quit IRC11:43
openstackgerritQuique Llorente proposed openstack/tripleo-quickstart-extras master: Install correct python version packages at tempest  https://review.openstack.org/61872111:43
*** akrivoka has quit IRC11:44
ssbarnea|rovermarios: gate of 40 hours?!11:44
*** akrivoka has joined #tripleo11:44
mariosssbarnea|rover: yeah but that is the python issue right11:44
mariosssbarnea|rover: i mean pypi mirror11:45
*** ahosam has quit IRC11:45
*** raildo has joined #tripleo11:45
*** hakhande_ has joined #tripleo11:46
shardygchamoul: ah, nice!  I reviewed it, it won't work as is but looks pretty close11:47
shardygchamoul, chem: let me know if you'd like any help fixing it up11:47
gchamoulshardy: great, raising the bug11:48
shardygchamoul: also, I think this needs to be set always, not only for upgrade, so we may want to look at renaming the file11:48
openstackgerritCédric Jeanneret proposed openstack/tripleo-heat-templates master: Ensure we get dedicated logging file for HAProxy  https://review.openstack.org/62060111:50
gchamoulshardy: yes this needs to be set always and ok for the renaming11:50
openstackgerritMike Fedosin proposed openstack/tripleo-heat-templates master: Allow to skip docker reconfiguration during stack update  https://review.openstack.org/62062111:53
*** trozet has quit IRC11:54
*** holser_ is now known as holser|burger11:54
EmilienMamoralej: ok11:55
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: Match node properties more strictly  https://review.openstack.org/62089211:55
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: Match node properties more strictly  https://review.openstack.org/62089211:57
*** amoralej is now known as amoralej|lunch11:58
*** jpena is now known as jpena|lunch11:59
*** dsneddon has joined #tripleo12:00
openstackgerritGael Chamoulaud proposed openstack/tripleo-heat-templates master: WIP: Restore tripleo_undercloud_conf_file hiera variable.  https://review.openstack.org/61447012:02
*** dsneddon has quit IRC12:05
*** saneax has quit IRC12:08
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870912:10
ykarelssbarnea|rover, can u get the container logs collection fixed https://review.openstack.org/#/c/618163, are u stuck on something?12:10
*** xek_ has joined #tripleo12:12
*** gfidente has joined #tripleo12:12
ssbarnea|roverykarel: just the rover role...12:13
ykarelssbarnea|rover, ack so u will fix it, right?12:14
ssbarnea|roverykarel: right now12:14
ykarelssbarnea|rover, okk cool12:14
*** salmankhan1 has joined #tripleo12:16
*** salmankhan has quit IRC12:16
*** salmankhan1 is now known as salmankhan12:16
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870912:19
*** jcoufal has joined #tripleo12:19
openstackgerrityatin proposed openstack-infra/tripleo-ci master: Fix overcloud node log collection in ovb jobs  https://review.openstack.org/61866912:19
*** xek_ has quit IRC12:21
openstackgerritJiri Stransky proposed openstack/python-tripleoclient stable/rocky: [Rocky] Fix backport of undercloud online upgrade  https://review.openstack.org/62089712:21
ssbarnea|roverykarel: damn, the entire code was refactored on that one and not in a good way, i am close to abandon mine and let someone else do it...12:22
ssbarnea|roverthe entire logic of building a list of commands in bash does not make any sense to me12:22
ssbarnea|roverthe code looks 10x more complex than it should be12:23
*** gkadam_ has joined #tripleo12:23
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: Add standalone scenario002 job  https://review.openstack.org/61870912:23
*** gkadam has quit IRC12:25
ssbarnea|roverykarel: sorry, i don't have time to fix it.12:26
*** gkadam_ has quit IRC12:28
*** gkadam has joined #tripleo12:28
*** hberaud is now known as hberaud|lunch12:30
*** ansmith has quit IRC12:31
*** jcoufal has quit IRC12:33
*** ratailor has quit IRC12:34
ykarelssbarnea|rover, :(12:34
ykarelssbarnea|rover, good to fix the issue atleast12:34
ykarelfixing the pattern may need some effort12:35
ssbarnea|roverykarel: if you do it, i will help. is just that i don't have the bandwith for the next 3 weeks due to ruck/rover.12:35
openstackgerritDerek Higgins proposed openstack/tripleo-docs master: Add documentation on using net-ansible in overcloud  https://review.openstack.org/62010112:35
ykarelssbarnea|rover, what's the issue in ur last patchset12:36
ssbarnea|roverit does not merge and there were too many changes around, mainly it needs reimplementaiton.12:36
*** rh-jelabarre has joined #tripleo12:37
*** bfournie has quit IRC12:37
ykarelssbarnea|rover, okk seeing merge conflict when rebasing12:38
ykarelssbarnea|rover, ok checking what changed, and fixing in ur approach12:38
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: DNM: testing adding pip fallback for pypi.org  https://review.openstack.org/62090212:39
ykarelokk looks like podman support is added12:39
*** panda|pto is now known as panda12:41
*** numans has joined #tripleo12:41
*** dprince has joined #tripleo12:50
openstackgerritRadoslaw Smigielski proposed openstack/paunch master: Default log location failes for non-root users  https://review.openstack.org/61801412:51
*** sshnaidm|afk is now known as sshnaidm12:51
Baggypants12000Hi, I'm trying to deploy tripleo quickstart but it's failing because the undercloud install can't reach the ntp servers, Am I correct in passing '-e ntp_servers=my.ntp.server' on the quickstart launch?12:53
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart-extras master: Add export ANSIBLE_LIBRARY to linting so we can find modules  https://review.openstack.org/62055612:53
marios_|ruckssbarnea|rover: this the kind of thing you had in mind? ^12:53
marios_|ruckssbarnea|rover: env is tidier but this lets us check and append without naked override12:53
*** morazi has joined #tripleo12:54
openstackgerritJiri Stransky proposed openstack/python-tripleoclient master: Stop undercloud upgrade procedure if deployment tasks fail  https://review.openstack.org/62090512:54
*** abishop has joined #tripleo12:54
ssbarnea|rovermarios_|ruck: hehe, kindof, but this is *not* the way of doing it in ansilbe, there is an "environment" parameter which is supposed to be used for that.12:54
ssbarnea|roverwhich takes a dictionary.12:55
ssbarnea|roverso you don't have to write bash around, and works with most ansible modules, not only shell/command.12:55
ykarelssbarnea|rover, should i just fix the issue and leave the reimplementation12:56
*** moguimar has quit IRC12:56
ssbarnea|roverykarel: yeah, if you want to have it merged this year ;)12:58
*** mcornea has joined #tripleo12:58
*** moguimar has joined #tripleo12:58
ykarelssbarnea|rover, :) so please restore12:59
ykareli will push the patch12:59
ssbarnea|rovermarios_|ruck: and this opens a new question: we provably need a story for implementing and documenting a way to pass envvars on tripleo jobs, just like tox_environment already works.12:59
*** cschwede has joined #tripleo13:00
marios_|ruckssbarnea|rover: that's what i had originally13:01
marios_|ruckssbarnea|rover: i mean environment13:01
marios_|ruckssbarnea|rover: and you suggested bash :D13:01
*** agurenko has joined #tripleo13:01
cschwedeHi! Can someone please review this tiny fix: https://review.openstack.org/#/c/620650/ ?13:01
ssbarnea|roverhaha... until i needed it.13:01
*** artom has joined #tripleo13:02
ssbarnea|rovermarios_|ruck: ahh, but I did had a special reason for it, because this param is a list.13:02
*** shardy has quit IRC13:03
marios_|ruckssbarnea|rover: so i just replied to your comment see https://review.openstack.org/#/c/620556/2/roles/standalone/tasks/main.yml13:03
ssbarnea|roverbut the newer implementation is not solving the problem I raise: what if the variable is already defined.13:03
ssbarnea|rovermarios_|ruck: yeah, is ok. my mistake.13:04
ssbarnea|rovermarios_|ruck: this variable is a bad example of using environment.13:04
ssbarnea|rovermarios_|ruck: does this mean that we can skip the scrum meeting?13:05
marios_|ruckssbarnea|rover: so looks like calendar error13:05
marios_|ruckssbarnea|rover: there wasno scrum me quiquell and sshnaidm joined and dropped13:05
marios_|ruckcall in 1 hour for planning13:06
ssbarnea|roverok, i will get luch now in this case.13:06
marios_|ruckomnomnom13:06
*** agurenko has quit IRC13:08
florianfakrivoka: can you have a look at this: https://review.openstack.org/#/c/620085/13:08
*** shardy has joined #tripleo13:10
*** moguimar has quit IRC13:10
florianfmandre: do you think it's possible to add a dynamic bind mount to a service template in t-h-t based on a variable passed by tripleoclient?13:11
*** rlandy has joined #tripleo13:11
*** moguimar has joined #tripleo13:11
openstackgerritDan Prince proposed openstack/tripleo-specs master: Config container image optimizations  https://review.openstack.org/62090913:12
dprincebogdando: an idea 13:14
*** pbandark has joined #tripleo13:15
*** bfournie has joined #tripleo13:16
hjensasssbarnea|rover: marios_|ruck: why does every other tripleo-common change in the gate fail with http://logs.openstack.org/39/619039/1/gate/openstack-tox-py27/d6ca374/testr_results.html.gz ?13:21
hjensasssbarnea|rover: marios_|ruck: or maby is is just https://review.openstack.org/619039, and I keep seeing it because of gate resets ...13:22
*** ansmith has joined #tripleo13:23
marios_|ruckhjensas: not seen that one we only just came on as ruck|rover. aware of a pypi mirror issue affecting all the things but not seen that trace yet we may need a new bug13:23
marios_|ruckhjensas: thanks for ping13:23
marios_|ruckhjensas: yeah but your trace seems different to https://bugs.launchpad.net/tripleo/+bug/180158713:24
openstackLaunchpad bug 1801587 in tripleo "[master/Rocky]Fs035 job fails in promotion because of heat stack timeout" [Critical,Triaged]13:24
marios_|ruckhjensas: so looking at https://review.openstack.org/#/c/619039/ it looks green?13:25
marios_|ruckat least voting things13:25
*** gkadam_ has joined #tripleo13:25
marios_|ruckoh gate13:25
*** rh-jelabarre has quit IRC13:27
*** tzumainn has joined #tripleo13:27
*** ade_lee has quit IRC13:27
*** gkadam has quit IRC13:28
hjensasmarios_|ruck: yes, check looks good. But that was Nov 20 ... I may have shouted a bit early, I have seen that happen in gate many times yesterday and today. But it may just be that one change over and over after gate resets.13:29
marios_|ruckhjensas: ack i went looking for more traces so i can file a bug13:29
*** saneax has joined #tripleo13:29
marios_|ruckhjensas: gonna hold for now but thanks for heads up anyway let me know if you get something to point to13:29
*** jpena|lunch is now known as jpena13:31
openstackgerritNatal Ngétal proposed openstack/tripleo-docs master: [Documentation] Replace newton by rocky.  https://review.openstack.org/62091013:32
*** hberaud|lunch is now known as hberaud13:33
openstackgerritThomas Herve proposed openstack/puppet-tripleo master: Increase websocket timeout  https://review.openstack.org/62091213:34
bogdandodprince: thanks, looking13:35
*** janki has joined #tripleo13:35
openstackgerritNatal Ngétal proposed openstack/tripleo-docs master: [Documentation] Replace newton by rocky.  https://review.openstack.org/62091013:35
*** salmankhan has quit IRC13:36
florianfakrivoka: oh, and this one: https://review.openstack.org/#/c/620545/ :)13:36
*** holser|burger is now known as holser_13:38
*** aufi has joined #tripleo13:39
jfrancoamwhahaha: EmilienM when you have some time, could you have a look at https://review.openstack.org/#/c/617974/ ? it's blocking us from creating a new version for ansible-redhat-subscription-manager. thanks13:39
EmilienMjfrancoa: looking right now sir13:39
jfrancoaEmilienM: merci13:39
EmilienMjfrancoa: SHIP IT13:40
EmilienMjfrancoa: get jaosorior (PTL) acking it13:40
jfrancoajaosorior: when you'll have a moment ^^^13:41
*** cylopez has joined #tripleo13:42
*** cylopez has quit IRC13:42
*** cylopez has joined #tripleo13:43
*** bogdando has quit IRC13:43
jaosoriordone13:43
*** amoralej|lunch is now known as amoralej13:43
*** bogdando has joined #tripleo13:44
EmilienMthx13:44
cylopezchem: EmilienM hello, If you have time to review, I add a simple param in puppet-keystone for LDAP usage https://review.openstack.org/#/c/620876/13:45
EmilienMcylopez: hey! looking now13:45
cylopezEmilienM: thx mate13:46
jfrancoajaosorior: thanks13:46
*** cfontain has quit IRC13:48
*** jcoufal has joined #tripleo13:48
EmilienMcylopez: I fixed the release note and +213:49
*** aufi has quit IRC13:49
cylopezEmilienM: great, there is a chance to backport this to queen ? a customer ask me that13:49
*** aufi has joined #tripleo13:49
EmilienMcylopez: yes, rocky and queens.13:49
EmilienMcylopez: wait for the patch to land13:49
cylopezEmilienM: of course13:49
cylopezEmilienM: thx again13:50
EmilienMcylopez: for jaosorior's team :D13:50
* EmilienM hides13:50
*** jcoufal has quit IRC13:50
jaosorior<_<13:50
jaosorior>_>13:50
EmilienM:)13:50
*** pbandark has quit IRC13:50
akrivokaflorianf: done13:59
*** eck`gone is now known as eck`14:00
*** zul has quit IRC14:00
*** dsneddon has joined #tripleo14:01
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Fix pacemaker tagging race condition  https://review.openstack.org/62091814:01
florianfakrivoka: thanks!14:02
*** salmankhan has joined #tripleo14:04
openstackgerritJason E. Rist proposed openstack/tripleo-heat-templates master: Remove containers-default-parameters.yaml from inclusion.  https://review.openstack.org/61846214:04
*** dsneddon has quit IRC14:05
weshaymarios_|ruck, can you update the status to RED14:07
weshayjaosorior, mwhahaha any new thoughts on the queue?14:10
weshaymarios_|ruck, and ssbarnea|rover are your ruck/rovers14:10
*** mjturek has joined #tripleo14:10
jaosoriormarios_|ruck, ssbarnea|rover: Are these timeouts again? or is there some other issue?14:12
ssbarnea|roverjaosorior: i am not sure yet but looking at http://status.openstack.org/elastic-recheck/ i begin to suspect it may be related to rax14:14
*** karthiks has joined #tripleo14:14
karthiksbfournie, Can you please review https://review.openstack.org/#/c/615488/14:14
bfourniekarthiks: yes, looking14:15
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-docs master: Move developer documentation to top level  https://review.openstack.org/62092114:16
jaosoriormarios_|ruck: ^^14:16
jaosoriorssbarnea|rover: let me know if you find anything else14:19
sri_shardy, using real interface name sloved that issue, but i've run into one more error this looks like cabling issue http://paste.openstack.org/show/736404/14:27
ssbarnea|roverjaosorior: i really don't know what is happening, but I see like tempest failing with timeout and zero longs as in http://logs.openstack.org/19/619519/1/gate/tripleo-ci-centos-7-standalone/1c111aa/job-output.txt.gz14:27
*** vinaykns has joined #tripleo14:28
ssbarnea|roverand by looking at http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 --- i only see a huge number of timeouts14:29
shardysri_: yes I'd check the cabling and the switch configuration14:31
sri_shardy, ack14:33
*** hjensas has quit IRC14:34
jristjpich: can you have a quick look at this for possible +2? https://review.openstack.org/#/c/616199/14:34
*** ade_lee has joined #tripleo14:38
*** rpioso|afk is now known as rpioso14:39
*** skramaja_ has quit IRC14:42
mwhahahaweshay: so yea we need to stop approving things. Also we need to know what is failing so we can form a plan14:44
*** mwhahaha changes topic to "CI Status: RED. Gate is 40+ hours behind. DO NOT APPROVE THINGS | https://docs.openstack.org/tripleo-docs/latest/"14:44
mwhahahaand the gate is about to reset14:45
*** rh-jelabarre has joined #tripleo14:45
openstackgerritMerged openstack/tripleo-common master: Fix broken YAQL expressions in heat_environments.yaml  https://review.openstack.org/61731114:47
jaosoriorjillr: around?14:50
*** jcoufal has joined #tripleo14:54
*** udesale has joined #tripleo14:58
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient stable/rocky: Remove timeout in config-download  https://review.openstack.org/62092515:00
*** xek has joined #tripleo15:00
slagle#startmeeting TripleO Edge Squad Meeting15:00
openstackMeeting started Thu Nov 29 15:00:45 2018 UTC and is due to finish in 60 minutes.  The chair is slagle. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: TripleO Edge Squad Meeting)"15:00
openstackThe meeting name has been set to 'tripleo_edge_squad_meeting'15:00
slagleping slagle, csatari, jaosorior, owalsh, fultonj, gfidente, hjensas, jtomasek,thrash, bogdando, dtantsur, rbrady, d0ugal, toure15:00
slagle#info remove or update your nick from the Meeting Template on the etherpad if you want (or don't want) to be ping'd for the start of the meeting15:01
jaosoriorslagle: o/15:01
dtantsur|mtgo/15:01
slagle#link https://etherpad.openstack.org/p/tripleo-edge-squad-status15:01
slagleAnyone can use the #link, #action, #help, #idea, #agreed, and #info commands, not just the moderatorǃ15:01
slagle#topic Agenda15:01
*** openstack changes topic to "Agenda (Meeting topic: TripleO Edge Squad Meeting)"15:01
slagle* Review past action items15:01
slagle* One off items15:01
slagle* Goals for the week15:01
slaglehi everyone, been a while since we've met15:01
fultonjo/15:01
toure0/15:02
toureo/15:02
owalsho/ shouldn't this have been an hour ago?15:02
toureowalsh I bet DST is messing with everyones schedule15:03
*** trown has quit IRC15:03
slagleowalsh: you make an excellent point15:03
fultonjas per etherpad it's Thursday 1400 UTC which was an hour ago15:03
fultonj(oh well - we're here now)15:03
bogdandoo/15:03
slaglehowever, given no one ran it :) i guess we'll continue on15:03
owalsh:-)15:04
slagle#info next week we shall meet an hour earlier to stick to the schedule of 1400UTC15:04
*** trown has joined #tripleo15:04
slagle#action slagle to practice timezone'ing15:04
*** holser_ has quit IRC15:05
slagle#topic Review past action items15:05
*** openstack changes topic to "Review past action items (Meeting topic: TripleO Edge Squad Meeting)"15:05
dtantsur|mtgslagle: you can just add the official ics from eavesdrop15:05
dtantsur|mtg* to your calendar15:05
openstackgerrityatin proposed openstack/tripleo-quickstart-extras master: Fix regression of collect-logs package listing  https://review.openstack.org/62092715:05
slagledtantsur|mtg: i'll look into that15:06
slaglefultonj: i saw the docs patch, but have not had a chance to run through it15:06
fultonjslagle: ok, thanks15:06
fultonji'm still cleaning it up15:06
slaglewill review in the coming week15:06
openstackgerritCédric Jeanneret proposed openstack/python-tripleoclient stable/rocky: Remove timeout in config-download  https://review.openstack.org/62092515:06
fultonjran into an issue, i wanted to research but go side tracked15:06
slagleany update on the CI?15:06
fultonjProgress on split-control-plane CI as it produces exported control plane15:07
fultonjhttp://logs.openstack.org/88/615988/4/check/tripleo-ci-split-controlplane-standalone/b6334f8/logs/undercloud/home/zuul/15:07
fultonjbut...15:07
fultonjstill need to coordinate CI deploying external compute which consumes it15:07
fultonjhttps://review.openstack.org/#/c/617368/9/playbooks/split-controlplane-standalone.yml15:07
slagle#link https://review.openstack.org/61463615:07
fultonj^ not doing what i want yet15:07
slagle#link https://review.openstack.org/#/c/617368/15:07
fultonj#link https://review.openstack.org/#/q/topic:bp/split-controlplane-ci+(status:open+OR+status:merged)15:08
slagle#link https://review.openstack.org/#/c/615988/15:08
slaglefultonj: great progress!15:08
fultonjthanks!15:08
slagleso we need to get more eyes on this15:08
slaglereviews, etc15:08
slagle#action all review split-controlplan CI / docs15:08
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: WIP: Add new tripleo-ci-centos-7-scenario001-standalone job  https://review.openstack.org/61950815:08
bogdandoall is included in the topic right?15:09
*** holser_ has joined #tripleo15:09
slaglebogdando: it's the patches i linked15:10
fultonjbogdando: yes, i linked the topic (3 patches)15:10
fultonjfor ci15:10
*** mjturek has quit IRC15:10
bogdandothanks15:10
fultonjchallenge at the moment is coordinating two standalone jobs15:11
fultonji've been working w/ tripleo-ci team for help15:11
*** mjturek has joined #tripleo15:11
*** ykarel is now known as ykarel|away15:11
slaglefultonj: you mean 2 standalone deployments in the same job?15:12
fultonjslagle: yes15:12
*** moguimar has quit IRC15:12
fultonjgood clarification15:12
*** moguimar has joined #tripleo15:12
Hobbestigrousomeone can review this patch please https://review.openstack.org/#/c/620910/ ?15:13
*** tzumainn has quit IRC15:13
*** hjensas has joined #tripleo15:13
slagleok, other action items was to continue to update https://etherpad.openstack.org/p/tripleo-edge-terms15:13
Hobbestigrouand this one also please https://review.openstack.org/#/c/618161/ ?15:13
slaglefwiw, "far edge" seemed pretty well accepted at the summit15:14
slagleany other comments about that?15:14
*** ykarel|away has quit IRC15:15
slagle#topic One Off / Goals15:16
*** openstack changes topic to "One Off / Goals (Meeting topic: TripleO Edge Squad Meeting)"15:16
dtrainordtantsur|mtg, thanks for the follow-up information.15:16
slaglecombining these items :)15:16
slagleas next steps for DCN, i plan to start taking a look at deploying with non-standalone.15:17
owalshslagle: what does that mean?15:17
slagleowalsh: right now, we've been using standalone to get the deployments stood up15:18
owalshso a standalone stack per site vs one big stack?15:18
slagleas opposed to creating a 2nd stack on the undercloud for a DCN deployment, then deploying that with config-download, etc15:18
slagleowalsh: it's not going to be one big stack. it'd be one stack for centralized control plane, then separate stack per DCN deployment15:19
slagle*opinion15:19
owalshgotcha15:19
slaglethen perhaps a workflow "deploy_dcn_stack"...that automates creating the dcn stack based off of a control plane stack15:20
slaglecreating the needed parameter values from the stack outputs, etc15:20
slaglebut will be manual at first to work out the kinks15:20
openstackgerritMarius Cornea proposed openstack/ansible-role-redhat-subscription master: DNM: Set up repos when using portal with activation key  https://review.openstack.org/62093715:22
slagleanyone else have any work or planned work to share?15:22
slagleok, well reach out if you'd like to get involved further with any of the ongoing work15:23
slaglethanks everyone! and see you 1 hour earlier next week!15:24
slagle#endmeeting15:24
*** openstack changes topic to "CI Status: RED. Gate is 40+ hours behind. DO NOT APPROVE THINGS | https://docs.openstack.org/tripleo-docs/latest/"15:24
openstackMeeting ended Thu Nov 29 15:24:08 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:24
openstackMinutes:        http://eavesdrop.openstack.org/meetings/tripleo_edge_squad_meeting/2018/tripleo_edge_squad_meeting.2018-11-29-15.00.html15:24
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/tripleo_edge_squad_meeting/2018/tripleo_edge_squad_meeting.2018-11-29-15.00.txt15:24
openstackLog:            http://eavesdrop.openstack.org/meetings/tripleo_edge_squad_meeting/2018/tripleo_edge_squad_meeting.2018-11-29-15.00.log.html15:24
owalshthanks slagle15:24
*** moguimar has quit IRC15:24
*** moguimar has joined #tripleo15:25
*** holser__ has joined #tripleo15:26
*** holser_ has quit IRC15:26
*** udesale has quit IRC15:29
*** udesale has joined #tripleo15:29
openstackgerritMarius Cornea proposed openstack/ansible-role-redhat-subscription master: DNM: Set up repos when using portal with activation key  https://review.openstack.org/62093715:30
*** kopecmartin is now known as kopecmartin|afk15:32
*** ykarel|away has joined #tripleo15:32
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: Do not enable ubuntu bionic just yet  https://review.openstack.org/62093915:33
*** ykarel|away is now known as ykarel15:34
*** panda is now known as panda|pto15:34
*** hakhande_ has quit IRC15:37
openstackgerritJanki Chhatbar proposed openstack/tripleo-heat-templates master: Don't mount data folder  https://review.openstack.org/62005315:37
*** itlinux has quit IRC15:38
*** ahosam has joined #tripleo15:39
*** ccamacho has quit IRC15:39
*** ramishra has quit IRC15:39
*** Vorrtex has joined #tripleo15:42
*** ccamacho has joined #tripleo15:49
*** quiquell is now known as quiquell|off15:55
*** cylopez has quit IRC15:59
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO  https://review.openstack.org/60429816:00
dtrainordtantsur|mtg, re: flavor's 'resource:' attribute, why is the resource named specifically CUSTOM_* and why is its value set to 1?  Is that just a boolean to enable or disable the resource for that flavor?16:01
dtantsur|mtgdtrainor: essentially. a bare metal node is one "instance" of a resource.16:01
dtrainorok, i follow16:01
*** dsneddon has joined #tripleo16:02
dtrainorbut why the 1 in e.g. resources:CUSTOM_BAREMETAL=116:02
dtantsur|mtgdtrainor: it's a request of one node per one nova instance16:03
dtrainoroh i see.  so 1 of this type16:04
dtrainor"A flavor can request exactly one instance of a bare metal resource class." https://docs.openstack.org/ironic/latest/install/configure-nova-flavors.html16:04
dtrainorthat makes sense now16:04
*** agurenko has joined #tripleo16:06
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: Invoke redhat-subscription role during upgrade.  https://review.openstack.org/61049316:06
*** agurenko has quit IRC16:07
*** tzumainn has joined #tripleo16:08
ssbarnea|roverweshay: can you confirm if https://review.openstack.org/#/c/620950/ and related bug makes sense? re: ERROR:dlrn:Received exception Error in build_rpm_wrapper16:10
dtrainordtantsur|mtg, I think the undercloud post install should set the CUSTOM_BAREMETAL_<flavor/role> on a flavor when the default flavor is created, if we expect the default flavors to be used16:10
dtrainorhttps://github.com/openstack/tripleo-heat-templates/blob/stable/rocky/extraconfig/post_deploy/undercloud_post.sh#L7816:10
dtantsur|mtgdtrainor: could be, but it will be a behavior change for the default "baremetal" flavor16:11
dtantsur|mtgso we only use "baremetal" everywhere16:11
dtrainori see16:11
weshayssbarnea|rover, sec16:12
jaosoriorstevebaker: around?16:12
dtrainordoing another deployment right now to get the exact error, but it indicated that no nodes could be found that match a resource class 'baremetal', because that's true16:13
jillrjaosorior: whats up?16:13
dtrainorso maybe set the nodes all to have a resource_class of CUSTOM_BAREMETAL to be able to consume the flavors they're supposed to be tagged in to16:13
dtrainori know that flavors / roles are different things but for the purpose of the defaults, they happen to be the same things16:14
weshayssbarnea|rover, what additional help can I provide to get you core on that repo ( elastic recheck )?16:15
*** yprokule has quit IRC16:15
weshaychandan_kumar, arxcruz ^ any contacts w/ upstream QE to help ssbarnea|rover get core there would be very helpful16:16
weshaymaybe attila f?16:16
ssbarnea|roverweshay: i think due to the nature of the changes we need confirmation from inside the team regarding the validity of the query. once we do get it ok, I can push infra guys to merge the change.16:17
jaosoriorjillr: needed help with the chronyd role, but I figured it out. thanks16:18
jillrjaosorior: ahk, np16:18
*** udesale has quit IRC16:20
*** gkadam_ has quit IRC16:20
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates stable/rocky: Invoke redhat-subscription role during upgrade.  https://review.openstack.org/62095316:21
weshayssbarnea|rover, mwhahaha http://logstash.openstack.org/#/dashboard/file/logstash.json?query=(message:%5C%22No%20matching%20distribution%20found%20for%5C%22%20OR%20message:%5C%22%5C%5CnNo%20matching%20distribution%20found%20for%5C%22)%20AND%20tags:console%20AND%20voting:1&from=864000s16:22
weshayhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=(message%3A%5C%22ERROR%3Adlrn%3AReceived%20exception%20Error%20in%20build_rpm_wrapper%5C%22)%20AND%20tags%3Aconsole%20AND%20voting%3A116:22
mwhahahaweshay: is that all the pypi problems?16:23
weshayssbarnea|rover, so the question is why did the number of these failures spike yesterday16:23
weshayno.. this is rpm build so maybe.. but looking at it16:23
mwhahahawe've had a bunch of tox jobs fail in tripleo-common, http://status.openstack.org/openstack-health/#/?groupKey=build_name&searchProject=tripleo16:23
mwhahahaso that would point to pypi problems16:23
weshaytripleo_common.tests.actions.test_deployment.OrchestrationDeployActionTest.test_run_failed ...    ERROR [tripleo_common.actions.deployment] Heat deployment failed for 'name'16:23
*** janki has quit IRC16:24
* weshay sees unit test failures16:24
weshayhttp://logs.openstack.org/39/619039/1/gate/tripleo-ci-centos-7-containers-multinode/56bf0b8/logs/delorean_logs/7f/63/7f638090124fd8cbacf9390ce5b26e72215a7761_dev/build.log.txt.gz16:24
ssbarnea|roverpip errors looks like the rax mirror issue... with my fix from yesterday that was not approved due to possible risks.16:24
weshayhttp://logs.openstack.org/39/619039/1/gate/tripleo-ci-centos-7-standalone/c0b3623/logs/delorean_logs/9d/2d/9d2d263ff06c9fb110c382dda464549733a33a3f_dev/build.log.txt.gz16:25
weshayssbarnea|rover, we need to be sure we're adding the unit test errors we see above to the logstash files in collect logs16:26
weshayand then build a query around the unit tests16:26
weshayssbarnea|rover, let me know if that is not clear16:26
ssbarnea|roverweshay: create a LP bug, assign it to me and be sure you mention the matching rule, I can take care of the rest.16:27
weshayrock16:27
mwhahahai thought we had a lp for the Oops\n thing16:27
ssbarnea|roveras long I know which bug to link to which query I can make them happen16:27
mwhahahabut i'd like ot kno where that's coming from16:27
mwhahahabecause it's not happening locally16:28
ssbarnea|roverfeel free to reopen old bugs if they are generic16:28
openstackgerritMarius Cornea proposed openstack/ansible-role-redhat-subscription master: Set up repos when using portal with activation key  https://review.openstack.org/62093716:28
mwhahahaweshay: so is that package failure happening in the gate right now? or is it cleared up16:29
*** hberaud is now known as hberaud|afk16:30
weshay05:00 utc was the last one16:30
ssbarnea|roverperfect zuul is about to die....16:30
weshaymwhahaha, /me looking for the bug you are referring to16:30
*** jcoufal has quit IRC16:31
mwhahahai'd seen it before, maybe no one reported the bug16:31
*** jcoufal has joined #tripleo16:31
weshayade_lee, did ntp work for you?16:32
openstackgerritMike Fedosin proposed openstack/tripleo-heat-templates master: Allow to skip docker reconfiguration during stack update  https://review.openstack.org/62062116:33
*** ykarel is now known as ykarel|away16:33
weshayapetrich, ping http://logs.openstack.org/39/619039/1/gate/tripleo-ci-centos-7-containers-multinode/56bf0b8/logs/delorean_logs/7f/63/7f638090124fd8cbacf9390ce5b26e72215a7761_dev/build.log.txt.gz16:34
weshayis that a mistral unit test?16:34
*** ccamacho has quit IRC16:34
apetrichweshay, looking 1 min16:34
mwhahahait's a return from mistral-lib16:34
mwhahahacause actions.Result comes from mistral-lib16:35
apetrichweshay, tripleo-common16:35
*** akrivoka has quit IRC16:35
*** akrivoka has joined #tripleo16:36
weshayhttps://bugs.launchpad.net/tripleo/+bug/180587016:36
openstackLaunchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged]16:36
weshayfyi16:36
openstackgerritYossi Ovadia proposed openstack/python-tripleoclient master: increasing the workflow timeout from 300 to 600 required in case of relatively large scale on physical servers resubmit  https://review.openstack.org/62075416:37
weshayssbarnea|rover, notes for you in the bug16:37
*** akrivoka has quit IRC16:38
weshayapetrich, does that appear to be something you can fix?16:39
*** akrivoka has joined #tripleo16:39
apetrichweshay, aye. <lastwords>should be a quick fix</lastwords>16:40
weshaymeh.. understand..16:40
weshayso it goes16:40
weshayapetrich, thanks for the help!16:40
apetrichweshay, I will do that before I close the lid16:40
jaosoriormandre: around?16:40
mandrejaosorior: sí señor16:41
jaosoriormandre: do you happen to know how this works https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/container_image_prepare.html#modify-with-custom-dockerfile ?16:42
openstackgerritMarius Cornea proposed openstack/ansible-role-redhat-subscription master: Set up repos when using portal with activation key  https://review.openstack.org/62093716:42
ade_leeweshay, yes and no. For rocky, I was able to deploy.  for master, there are still issues16:42
mandrejaosorior: yeah, what would you like to know?16:43
mandrejaosorior: the role lives at https://github.com/openstack/ansible-role-tripleo-modify-image/16:43
mandreit's pretty well documented16:44
ade_leeweshay, for some overcloud nodes, I end up with the chronyc sync failure even when the ntp server is set.  for others it works, and for my compute node, the ntp server was not set.16:44
jaosoriormandre: right, so I'm trying out that example (almost as-is, but just doing a yum install)16:44
ade_leeweshay, its kind of a mess16:44
jaosoriormandre: and the ansible task can't access the /home/stack/nova-custom directory16:44
ade_leeweshay, which is why I'm looking to disable the time sync in my deploy16:44
jaosoriormandre: what permissions do I need for the ansible task to be able to access it. I guess it's also running inside the container? does it even have access to the file system?16:45
jaosoriorade_lee: I think I fixed it. I had to set NtpPool in the overcloud deploy to be []16:45
jaosoriorade_lee: so... in order to fix it, we need to be able to set this from quickstart16:46
*** ahosam has quit IRC16:46
ade_leejaosorior, cool -- so how to do that ..16:46
mandrejaosorior: it needs to have access to the docker daemon, and perm to read the files16:46
*** hamzy has quit IRC16:46
weshayade_lee, hrm.. interesting16:46
mandrejaosorior: IIRC it's root in the container so it shouldn't hit perms issues16:47
ade_leejaosorior, patch to oooq?16:47
jaosoriorade_lee: it did :/16:47
jaosoriorade_lee: yeah16:47
ade_leejaosorior, you got patch?16:47
jaosoriorno16:48
jaosoriornot yet16:48
*** gfidente has quit IRC16:48
ade_leejaosorior, I'll be happy to kick it off once you do :)16:48
jaosoriormandre: also, are there readable logs for that ansible run?16:48
*** saneax has quit IRC16:48
jaosoriorthey get appended without formatting in the ansible.log file :(16:48
*** trown is now known as trown|lunch16:49
mandrejaosorior: are you modifying the image as part of your deployment? if so there should be a log at /var/log/tripleo-container-image-prepare.log16:50
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: Pin rubocop to 0.60.0  https://review.openstack.org/62095516:52
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: WIP, add dlrn build log to log stash  https://review.openstack.org/62095816:54
*** hberaud|afk is now known as hberaud16:54
weshayssbarnea|rover, something like https://review.openstack.org/#/c/620958/116:54
*** itlinux has joined #tripleo16:54
weshaysshnaidm, fyi ^16:54
weshaydoes that seem right to you?16:54
openstackgerritMike Fedosin proposed openstack/tripleo-heat-templates master: Allow to skip docker reconfiguration during stack update  https://review.openstack.org/62062116:54
ssbarnea|roverweshay: sorry busy on infra talking about pip.... issues.16:55
weshayssbarnea|rover, please do.. np16:55
ssbarnea|roveryesterday rex networking got its pip mirror down, followed by vexxhost mirror going down for other reasons, result: check len(queue)16:56
*** mjturek has quit IRC16:56
*** moguimar has quit IRC16:58
dtrainordtantsur|mtg, I16:59
dtrainorarg.16:59
dtrainorI'll follow up in a few :) thanks for your input.16:59
dtantsur|mtgsure :)17:00
*** mjturek has joined #tripleo17:00
*** mjturek has quit IRC17:01
karthiksdsneddon, can you please review https://review.openstack.org/#/c/615488/17:02
apetrichweshay, that's a very odd error. I cannot reproduce locally. If I fix to pass that gate tox fails locally. I'm not sure on what to do17:03
*** hjensas has quit IRC17:03
weshaymwhahaha, jaosorior looks like ovb is back on rdo-cloud17:04
weshaywill send an update17:04
*** mmagr has quit IRC17:04
openstackgerritMike Fedosin proposed openstack/tripleo-heat-templates master: Allow to skip docker reconfiguration during stack update  https://review.openstack.org/62062117:05
ssbarnea|roverweshay: how it comes that we have such huge gate queue but http://cistatus.tripleo.org/gates/ is as as green as it can be.17:07
weshayit's not as green as can be17:08
weshayoh wait17:08
weshayit wasn't yesterday17:08
weshaysshnaidm, ^17:08
weshayssbarnea|rover, yesterday was 93% pass.. that actually pretty bad17:08
weshaywe need to be at 97+17:08
jaosoriormandre: right, that has the logs of the container-image-prepare command, but not of the ansible tasks it tried to run17:08
weshay tripleo-ci-centos-7-containers-multinode17:09
*** shardy has quit IRC17:09
weshaylast fail is 11/2417:09
weshaysomething is not right there sshnaidm can we restart the containers?17:09
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180587017:10
openstackLaunchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich)17:10
*** ooolpbot has quit IRC17:10
sshnaidmweshay, will check17:12
weshaysshnaidm, thanks17:12
ssbarnea|roverweshay: this is what I was trying to say: the last failure I seen there was from 27th!  also the view shows only percentages, no idea about how many builds passed. 0/0 would likely count as 100$.17:12
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates stable/rocky: Added Dell EMC SC multipath support  https://review.openstack.org/62035517:12
Hobbestigrousomeone can review this patch please https://review.openstack.org/#/c/618106/ ?17:13
openstackgerritRajini Karthik proposed openstack/tripleo-heat-templates stable/queens: Added Dell EMC SC multipath support  https://review.openstack.org/62035817:14
sshnaidmweshay, which page are checking?17:14
*** akrivoka has quit IRC17:14
weshaysshnaidm, gates17:15
*** akrivoka has joined #tripleo17:15
sshnaidmweshay, I see it's working.. which job are wrong?17:15
weshaysshnaidm, old data man17:15
weshaycheck containers-multindoe17:16
weshaynode17:16
weshaysshnaidm, https://review.openstack.org/#/c/606064/17:16
weshayhttps://review.openstack.org/#/c/619383/17:16
weshayit's missing too much data17:16
dtrainorI'm not sure why my queens / downstream osp13 deployment is failing.  Ultimately in nova-scheduler I see:  'ceph-storage' does not match 'control'.  I have nodes tagged for each of these mentioned systems but they appear to be filtered out, and eventually:  Resource CREATE failed: ResourceInError: resources[0].resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500"17:19
*** hjensas has joined #tripleo17:19
*** agopi is now known as agopi|food17:21
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-quickstart-extras master: Add ability to set NtpPool for the overcloud  https://review.openstack.org/62096517:21
jaosoriorade_lee: ^^17:21
mwhahahajaosorior: i fixed that in the chrony role17:22
jaosoriormwhahaha: where?17:22
mwhahahano in tht17:22
jaosoriormwhahaha: ade_lee deployed yesterday (or today even), and it's not working :(17:23
mwhahahahttps://review.openstack.org/#/c/617001/17:23
jaosoriormwhahaha: sure17:24
jaosoriorand that bit works17:24
jaosoriorBUT17:24
jaosoriorNtpPool defaults to having a value17:24
jaosoriormwhahaha: which is what I'm seting in the review I posed to quickstart17:24
mwhahaharight but what is the problem with that17:24
mwhahahaif we want we can nuke the ntppool default value17:24
mwhahahabut i'd like to understand the actual problem17:24
mwhahahabecause configuring both ntppool and ntpservers should still give you some function ntp17:25
jaosoriormwhahaha: so, having a deployment in a VPN, we can't use that default. quickstart only exposes ntp_server. With my commit, you can set ntppool as empty if you pass ntp_server... or just pass ntp_pool and use that instead17:25
mwhahahabut why17:25
mwhahahawhat do you mean you can't use it17:25
jaosoriormwhahaha: why what?17:25
jaosoriorthere is no access to pool.ntp.org from inside this network17:26
mwhahaharight but if you set ntp servers to something you do have access to it should still work17:26
mwhahaha(i tested this internally)17:26
jaosoriorshould it?17:26
mwhahahayes17:26
jaosoriorIt keeps failing17:26
jaosorioragain and again17:26
mwhahahaon what17:26
jaosorioruntil I did that fact17:26
jaosorioron chrony17:26
* mwhahaha applies head to desk17:26
mwhahahaplz provide exact error messages17:26
jaosoriorade_lee: do you have the error message handy somewhere?17:27
mwhahahanot just "it doesn't work"17:27
jaosoriorsure17:27
jaosoriormwhahaha: isn't it the case that the pool takes precedence over the ntp_server option?17:27
mwhahahano17:27
mwhahahait configures all17:28
jaosoriorthat's what the description of the command says17:28
jaosoriorgotta update that then17:28
mwhahahaso you should end up with a pool config and a server list in the sources list17:28
jaosoriorthat is the case17:28
mwhahahahttps://review.openstack.org/#/c/617001/ didn't properly updated the relationship in the description as that removed that priority bit17:29
dsneddonkarthiks, Will review 615488 now17:29
jaosoriormwhahaha: I'll reproduce the error and hand you over the error message17:29
*** jpich has quit IRC17:29
*** dmacpher has quit IRC17:30
*** ksambor has quit IRC17:30
*** mjturek has joined #tripleo17:31
*** bogdando has quit IRC17:33
dsneddonkarthiks, Quick question. In line 443 of impl_ifcfg.py, shouldn't the 'data += "MACADDR=\"%s\"\n" % primary_mac' be run even if self.noop? We want to see the data that would be output into the file if doing a trial run without changing anything, right? https://review.openstack.org/#/c/615488/5/os_net_config/impl_ifcfg.py@44317:34
*** mjturek has quit IRC17:38
*** akrivoka has quit IRC17:39
*** mjturek has joined #tripleo17:39
*** akrivoka has joined #tripleo17:39
mwhahahaweshay, jaosorior: i think we should clear the gate and nuke the scenarios from the stable branches17:41
mwhahahaweshay, jaosorior: see http://zuul.openstack.org/builds?pipeline=gate&result=failure&result=timed_out&result=post_failure&project=openstack%2Ftripleo-common&project=openstack%2Ftripleo-heat-templates17:42
weshay5 timeouts from today17:42
weshaypretty typical17:43
mwhahahawhile there are failures with containers-multinode and standalone, i think we're really being hurt by the scenario jobs on rocky. When they were non-voting we had a much better queue17:43
mwhahahait wasn't until we landed the enabling of the voting bits that the gate started to go down hill17:43
weshaybut open to what ever man17:43
weshaythis is such a flippin mess17:43
mwhahahaweshay: unforutnately i don't think this captures failures that get cleared from a gate reset17:43
mwhahahaso i think it's worse than that17:43
* mwhahaha proposes patch to drop scnearios from rocky17:44
mwhahahai will also send an ML post17:44
weshay7 here http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen17:44
weshayfrom today17:44
weshaymwhahaha, all ovh bs1 and limestone17:44
weshayssbarnea|rover, ^17:44
weshaypinche limestone17:45
*** akrivoka has quit IRC17:46
weshaymwhahaha, I guess we're not allowed to have both ci systems working well at any one given time17:46
*** cfontain has joined #tripleo17:46
mwhahahaof course not17:46
*** akrivoka has joined #tripleo17:46
weshaySHIP IT!!17:46
weshaymwhahaha, what are the chances of back porting standalone to rocky?17:47
mwhahahathe standalone worked in rocky17:48
mwhahahaand is supported17:48
mwhahahaso if you wanted standalone scenarios, it should be ok17:48
weshayah k..17:48
weshaygood.. that would help a lot imho17:48
apetrichweshay, need help. wrote some comments on https://bugs.launchpad.net/tripleo/+bug/1805870 but I'm unable to find a reason why that is failing if I fix for the gates the local tests fail.17:48
openstackLaunchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich)17:48
weshayto have n and n-1 w/ standalone17:48
mwhahahaweshay: just need to have those scenarios files defined :D17:49
weshaymwhahaha, can we disable those tests ?17:49
mwhahahawhich tests?17:49
weshaymwhahaha, https://bugs.launchpad.net/tripleo/+bug/180587017:49
weshaythanks apetrich17:49
mwhahahathe ones in the package build? would be a distgit change17:49
mwhahahai also tried to reproduce it and could not17:49
mwhahahalet me nuke the scenarios first and then i'll try and figure out wtf is wrong with that test17:49
weshayk17:49
weshaymwhahaha, apetrich if I can repro you guys owe me a beer17:51
*** ykarel|away has quit IRC17:53
openstackgerritAlex Schultz proposed openstack/puppet-tripleo stable/rocky: Switch scenarios to non-voting  https://review.openstack.org/62097117:53
*** jpena is now known as jpena|off17:53
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/rocky: Switch scenarios to non-voting  https://review.openstack.org/62097217:54
apetrichmwhahaha, I tried forcing different versions of python-heatclient and could not see it happening. also looked at the code and there was nothing that I could see what could cause that17:56
mwhahahaapetrich: the only thing i could think of was the mistral_lib actions.Result's use of repr17:57
mwhahahaapetrich: because the string is converted to unicode17:57
apetrichweshay, deal! but out of spite it will be an IPA as bitter as I would feel17:57
mwhahahaapetrich: the other thing it might be related to the host charset or something17:57
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/rocky: Switch scenarios to non-voting  https://review.openstack.org/62097317:58
apetrichmwhahaha, but it happens on 3.5 also and that seems weird if it was the repr17:58
mwhahahaapetrich: yea which is why i'm wondering if it's a host charset thing of some sort17:58
mwhahahabecause it is cross versions17:58
apetrichmwhahaha, agreed17:58
mwhahahabut we only build python2.7 packages17:59
*** wolverineav has joined #tripleo17:59
mwhahahaand i'm not sure we saw the same error in tox?17:59
apetrichmwhahaha, mistral-lib last change was 21 days ago.18:01
*** akrivoka has quit IRC18:01
mwhahahayea i didn't see any changes to it that would have caused this either18:01
*** akrivoka has joined #tripleo18:02
apetrichmwhahaha, lets just hope that wes can reproduce and then we try to poison him with an artesanal IPA18:03
mwhahaha:D18:03
*** amoralej is now known as amoralej|off18:05
apetrichmwhahaha, one option would be that the internationalization for python-heatclient has the string OOPS translated as OOPS\n but that is a bit of a stretch18:05
mwhahahayea seems a bit much18:06
*** derekh has quit IRC18:06
*** aufi has quit IRC18:06
*** aufi has joined #tripleo18:07
*** trown|lunch is now known as trown18:08
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180587018:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich)18:10
weshayssbarnea|rover, alex is reseting the gates18:10
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/queens: Switch scenarios to non-voting  https://review.openstack.org/62097518:11
*** hberaud is now known as hberaud|gone18:12
openstackgerritAlex Schultz proposed openstack/puppet-tripleo stable/queens: Switch scenarios to non-voting  https://review.openstack.org/62097618:12
*** hamzy has joined #tripleo18:12
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates stable/queens: Switch scenarios to non-voting  https://review.openstack.org/62097818:13
ssbarnea|roverweshay: i would like to understand what is causing the piling up, as I see a trend regarding resetting gate action. We should start logging each date we had to do it.18:14
weshayless of a trend more of a way of life18:15
*** agopi|food is now known as agopi18:15
ssbarnea|roveri know that the two pip-mirror issues did had their effect, but this does not explain why after they were fixed it did not start to recover.18:15
weshayssbarnea|rover, maybe you can track resets in http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=118:15
weshayssbarnea|rover, we do need to triage the master immediately though18:16
weshaymaybe it's the same issue18:16
weshaybut we need a master build18:16
*** salmankhan has quit IRC18:16
weshayssbarnea|rover, 9 days old http://rhos-release.virt.bos.redhat.com:3030/rhosp18:16
*** wolverineav has quit IRC18:16
apetrichweshay, I'm clocking out now. Ping me if any news. I will see that in the morning18:18
weshayapetrich, aye18:18
weshaythanks18:18
*** wolverineav has joined #tripleo18:18
*** akrivoka has quit IRC18:18
*** jistr has quit IRC18:19
*** akrivoka has joined #tripleo18:19
*** jistr has joined #tripleo18:19
openstackgerritJuan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Default NtpPool to an empty list  https://review.openstack.org/62098118:20
jaosoriormwhahaha: ^^18:20
mwhahahawfm18:20
*** aufi has quit IRC18:21
mwhahahaso many patches in the gate18:24
jaosoriorare all of them timing our? or only some specific scenarios?18:24
mwhahahait's not specific scenarios18:25
*** akrivoka has quit IRC18:25
*** holser__ has quit IRC18:25
mwhahahawe're even seeing it in standalone which is it's own problem since that job finishes in like an hour18:25
*** akrivoka has joined #tripleo18:26
mwhahahai abandoned 50 patches that were in the gate18:26
mwhahahaand the 2nd one from the top was about to reset it all again18:27
mwhahahaso we need to figure out what the deal with the timeouts are18:27
mwhahahabe it capacity or whatever18:27
*** akrivoka has quit IRC18:29
weshaymwhahaha, should we consider pushing hard on landing the multinode -> standalone conversion for master18:29
weshayI'm hearing we're pretty much there, I need to go look18:30
mwhahahaweshay: yea i saw i abandoned some of those patches18:30
mwhahahalet me resurect those18:30
weshayssbarnea|rover, fyi ^18:30
*** jfrancoa has quit IRC18:31
openstackgerritMerged openstack/tripleo-common stable/rocky: Update swift_backup workflow success status message  https://review.openstack.org/61913018:31
weshayone should be given cocaine to do this job18:31
*** hamzy has quit IRC18:31
EmilienMand tea18:32
weshayOH HI18:32
mwhahahabut mostly tea18:32
EmilienMhi18:32
mwhahahaperhaps some biscuits18:32
EmilienMweshay: I'm with your boss18:32
weshayya.. not talking about anything crazy.. just a little18:33
weshayEmilienM, like you are married now?18:33
EmilienMtrying to catch up here, what's going on in short?18:33
EmilienMweshay: kinda18:33
hjensascamomile tea. :)18:33
weshayEmilienM, same shit, diff day18:33
mwhahahaEmilienM: gate was 45hrs behind, jobs timing out, the usual18:33
weshayblows in our stuff, blow ups in infra, blow ups in life18:33
EmilienMah so everything is fine18:33
weshayrightO18:33
weshaymwhahaha, ha.. dlrn failure did not recreate for me18:34
weshaylolz18:34
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/rocky: Switch scenarios to non-voting  https://review.openstack.org/62097318:34
mwhahahaweshay: see told you18:34
mwhahahaEmilienM, weshay: we'll need https://review.openstack.org/#/q/topic:scenarios-nv+(status:open+OR+status:merged)18:34
ssbarnea|roverweshay: that reminded me of cohen song which could easily be adapted ... give me back my broken build. https://genius.com/Leonard-cohen-the-future-lyrics18:35
* mwhahaha goes to lunch while zuul churns18:35
EmilienMmwhahaha: +2, I'll let weshay +A18:35
weshayEmilienM, https://review.openstack.org/#/c/620973/18:42
*** vpickard is now known as vpickard_18:43
EmilienMweshay: ?18:43
weshayEmilienM, nothin, I'll fix18:44
*** gfidente has joined #tripleo18:45
*** kopecmartin|afk is now known as mkopec|off18:45
*** mkopec|off is now known as kopecmartin|off18:45
*** pbandark has joined #tripleo18:47
openstackgerritwes hayutin proposed openstack/tripleo-common stable/rocky: Switch scenarios to non-voting  https://review.openstack.org/62097318:48
*** cfontain has quit IRC18:48
*** gfidente has quit IRC18:51
sshnaidmweshay, this should solve problem with missed jobs: https://github.com/sshnaidm/sova/commit/68885d2b60d85be908180709b931ed6e28907f8818:51
sshnaidmweshay, thanks for finding it18:51
weshaysshnaidm++18:52
weshaythanks man!18:52
*** hamzy has joined #tripleo18:52
weshaysshnaidm, thank mwhahaha18:52
sshnaidmthanks mwhahaha18:53
*** hamzy_ has joined #tripleo18:57
*** hamzy has quit IRC18:58
*** aufi has joined #tripleo18:59
*** abishop_ has joined #tripleo19:00
*** abishop has quit IRC19:02
*** wolverineav has quit IRC19:04
*** wolverineav has joined #tripleo19:05
*** wolverineav has quit IRC19:06
*** wolverineav has joined #tripleo19:06
openstackgerritAlex Schultz proposed openstack/tripleo-common stable/rocky: Switch scenarios to non-voting  https://review.openstack.org/62097319:07
*** morazi has quit IRC19:07
*** wolverineav has quit IRC19:07
*** wolverineav has joined #tripleo19:07
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180587019:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich)19:10
*** wolverineav has quit IRC19:19
*** wolverineav has joined #tripleo19:20
*** xek has quit IRC19:23
*** wolverineav has quit IRC19:24
*** wolverineav has joined #tripleo19:24
*** hjensas has quit IRC19:25
*** hjensas has joined #tripleo19:30
weshaymwhahaha, you have a sec to discuss a change to promotion criteria for upstream?19:33
mwhahahaweshay: i gotta dip for like 40 mins, can we do it at like 1:30?19:34
weshaymwhahaha, when ever.. is fine.. have a presentation w/ chris next week to mgmt.. your input would be ++19:34
mwhahahak i'll ping you when i get back19:35
weshay5min max19:35
weshayk.. thanks19:35
*** aufi has quit IRC19:35
*** dtrainor has quit IRC19:41
sshnaidmweshay, so let's make a switch? https://review.rdoproject.org/r/#/c/17583/19:43
* weshay wonders if anyone responded19:44
*** mjturek has quit IRC19:44
weshaysshnaidm, it has my +1 would be nice to see others but.. what ever19:44
*** mjturek has joined #tripleo19:45
stevebakermwhahaha: hey, do you have any idea which file I should edit to add a package to undercloud_rpm_dependencies for the job tripleo-ci-centos-7-undercloud-containers? This change did nothing https://review.openstack.org/#/c/62041319:46
*** dtrainor has joined #tripleo19:48
weshaystevebaker, that's the right change19:49
stevebakerweshay: the package wasn't added though, http://logs.openstack.org/13/620413/1/check/tripleo-ci-centos-7-undercloud-containers/9dbbf6e/logs/undercloud/home/zuul/install_packages.sh.txt.gz19:49
*** KingJ has quit IRC19:52
weshaylooking19:53
*** mjturek has quit IRC19:54
stevebakerweshay: if I knew how files like config/release/centosci/master-current-tripleo.yml were added to the job that would help, but I can't even figure that out19:54
weshaystevebaker, http://logs.openstack.org/13/620413/1/check/tripleo-ci-centos-7-undercloud-containers/9dbbf6e/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz19:54
weshay            "undercloud_rpm_dependencies": "python-tripleoclient ceph-ansible",19:55
*** mjturek has joined #tripleo19:55
weshaylooks like it didn't get passed19:55
*** wolverineav has quit IRC19:55
weshaystevebaker, can you try to run wget on the package in the "repo_cmd_after:" section right above19:55
*** KingJ has joined #tripleo19:56
weshaystevebaker, and then you can localinstall the rpm from a known dir19:56
*** wolverineav has joined #tripleo19:56
*** mjturek has quit IRC19:56
weshayvs.. https://url/rpm19:56
*** sshnaidm is now known as sshnaidm|afk19:56
stevebakerweshay: yeah ok, its just a temp hack for testing19:56
*** itlinux has quit IRC19:56
*** aufi has joined #tripleo19:59
*** morazi has joined #tripleo20:00
*** wolverineav has quit IRC20:00
*** mjturek has joined #tripleo20:01
*** wolverineav has joined #tripleo20:04
*** mjturek has quit IRC20:07
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180587020:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich)20:10
*** wolverineav has quit IRC20:10
*** slaweq has quit IRC20:11
*** jtomasek has quit IRC20:12
*** mjturek has joined #tripleo20:14
*** tzumainn has quit IRC20:15
*** holser_ has joined #tripleo20:18
*** florianf is now known as florianf|afk20:19
*** toure is now known as toure|biab20:21
*** abishop_ has quit IRC20:23
*** abishop_ has joined #tripleo20:25
*** abishop_ is now known as abishop20:26
*** dprince has quit IRC20:27
*** slaweq has joined #tripleo20:32
*** radeks has quit IRC20:34
*** radeks has joined #tripleo20:34
*** aufi has quit IRC20:39
mwhahahaweshay: ok i'm back20:39
weshaymwhahaha, ok.. 5 min you can time me20:39
weshayin blue20:40
weshayhttps://bluejeans.com/u/whayutin/20:40
mwhahahastevebaker: you might need to check the featureset to see if it's actually configured in there insteadh. those places are the defaults20:40
*** jcoufal has quit IRC20:46
stevebakermwhahaha: I think this is the key "QUICKSTART_DEFAULT_RELEASE_ARG": "--extra-vars @/home/zuul/workspace/.quickstart/config/release/tripleo-ci/CentOS-7/master.yml"20:46
mwhahahai wonder if it' snot getting the patch change20:48
*** hjensas has quit IRC20:48
openstackgerritSteve Baker proposed openstack/tripleo-quickstart master: DNM install tar-split  https://review.openstack.org/62041320:49
*** mjturek has quit IRC20:49
*** raildo has quit IRC20:50
*** wolverineav has joined #tripleo20:51
*** ansmith has quit IRC20:51
*** raildo has joined #tripleo20:51
*** raildo has quit IRC20:51
*** mjturek has joined #tripleo20:52
*** wolverineav has quit IRC20:53
*** wolverineav has joined #tripleo20:53
openstackgerritSteve Baker proposed openstack/python-tripleoclient master: Switch to stestr  https://review.openstack.org/61832820:54
*** holser_ has quit IRC20:55
jaosoriorhey folks! would appreciate the reviews for this https://review.openstack.org/#/c/620981/21:02
*** hjensas has joined #tripleo21:03
weshaybah21:04
weshayno soup for you21:04
*** dprince has joined #tripleo21:06
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180587021:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich)21:10
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: Remove unneeded spec testing file  https://review.openstack.org/61989221:12
*** mjturek has quit IRC21:15
*** pbandark has quit IRC21:20
*** agopi is now known as agopi|pto21:24
*** bfournie has quit IRC21:25
*** agopi|pto has quit IRC21:26
*** wolverineav has quit IRC21:34
*** wolverineav has joined #tripleo21:35
*** KingJ has quit IRC21:38
*** wolverineav has quit IRC21:38
*** wolverineav has joined #tripleo21:38
*** KingJ has joined #tripleo21:39
*** rlandy is now known as rlandy|biab21:44
*** hamzy_ has quit IRC21:47
*** hberaud|gone has quit IRC21:49
*** radeks has quit IRC21:52
*** rcernin has joined #tripleo21:57
openstackgerritJill Rouleau proposed openstack/tripleo-ansible master: Add podman_container module  https://review.openstack.org/62042322:00
*** aissat-mus has joined #tripleo22:01
*** trown is now known as trown|outtypewww22:05
openstackgerritMerged openstack/tripleo-common master: Set DeployIdentifier on update/upgrade/ffwd prepare actions  https://review.openstack.org/61975922:06
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180587022:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich)22:10
*** xek has joined #tripleo22:11
*** abishop has quit IRC22:12
*** rlandy|biab is now known as rlandy22:21
*** rh-jelabarre has quit IRC22:29
*** sshnaidm|afk is now known as sshnaidm|off22:37
*** agopi has joined #tripleo22:38
*** dprince has quit IRC22:39
*** slaweq has quit IRC22:41
*** xek has quit IRC22:44
*** Vorrtex has quit IRC22:46
*** aissat-mus has quit IRC22:58
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Revert "Make tripleo-buildimage-overcloud-full-centos-7 non-voting"  https://review.openstack.org/62020123:00
openstackgerritIan Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var  https://review.openstack.org/61911923:00
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912023:00
openstackgerritIan Wienand proposed openstack/diskimage-builder master: Revert "Make tripleo-buildimage-overcloud-full-centos-7 non-voting"  https://review.openstack.org/62020123:03
openstackgerritIan Wienand proposed openstack/diskimage-builder master: package-installs: provide for skip from env var  https://review.openstack.org/61911923:03
openstackgerritIan Wienand proposed openstack/diskimage-builder master: simple-init: allow for NetworkManager support  https://review.openstack.org/61912023:03
openstackgerritJohn Fulton proposed openstack/puppet-tripleo master: Allow user to define a custom tuned profile  https://review.openstack.org/61369823:03
*** artom has quit IRC23:08
openstackgerritMerged openstack/tripleo-quickstart master: [Configuration] Remove liberty files.  https://review.openstack.org/61951923:08
openstackgerritMerged openstack/tripleo-upgrade stable/queens: Add timeout to openstack command  https://review.openstack.org/62059723:08
*** artom has joined #tripleo23:09
*** artom has quit IRC23:09
*** artom has joined #tripleo23:09
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180587023:10
openstackLaunchpad bug 1805870 in tripleo "mistral unit tests failing in dlrn build of tripleo-common" [Critical,Triaged] - Assigned to Adriano Petrich (apetrich)23:10
*** ooolpbot has quit IRC23:10
*** artom has quit IRC23:10
*** artom has joined #tripleo23:10
*** agopi has quit IRC23:16
*** jamesdenton has quit IRC23:23
*** dciabrin has quit IRC23:24
*** dhellmann_ has joined #tripleo23:26
*** dhellmann has quit IRC23:26
*** dsneddon has quit IRC23:28
*** aedc has quit IRC23:28
*** dsneddon has joined #tripleo23:29
*** dhellmann_ is now known as dhellmann23:30
*** jamesdenton has joined #tripleo23:32
*** aedc has joined #tripleo23:36

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!