Tuesday, 2017-11-07

rdogerritrdo-trunk created rdoinfo master: Bump rdoinfo pike tags to latest upper-constraints  https://review.rdoproject.org/r/1043000:03
rdogerritrdo-trunk created rdoinfo master: Bump rdoinfo queens-uc tags to latest upper-constraints  https://review.rdoproject.org/r/1043100:03
rdogerritrdo-trunk created rdoinfo master: Promote CBS tags update for pike-testing  https://review.rdoproject.org/r/1043200:04
*** danpawlik has joined #rdo00:12
*** noslzzp has joined #rdo00:13
*** Son_Goku has quit IRC00:14
*** anilvenkata has quit IRC00:14
*** danpawlik has quit IRC00:18
*** julim has joined #rdo00:20
*** anilvenkata has joined #rdo00:27
*** fzdarsky has quit IRC00:35
*** Son_Goku has joined #rdo00:36
*** Son_Goku has quit IRC00:53
*** chlong has joined #rdo00:57
*** jkilpatr has joined #rdo00:58
*** owalsh_ has joined #rdo01:09
*** Son_Goku has joined #rdo01:10
*** ihrachys has quit IRC01:10
*** ihrachys has joined #rdo01:10
*** owalsh has quit IRC01:11
*** cshastri has joined #rdo01:12
*** jethro7 has joined #rdo01:14
*** owalsh_ is now known as owalsh01:15
jethro7Hi folks, it seems that Rally on Packstack is not working (I am seeing  ERROR rally ImportError: cannot import name VariableManager). Is it a bug and should I go ahead file a  report?01:16
*** jethro7 has quit IRC01:16
*** jethro7 has joined #rdo01:16
*** Son_Goku has quit IRC01:19
*** noslzzp has quit IRC01:22
*** liujiong has joined #rdo01:24
jethro7looks like something related to https://github.com/ansible/ansible-container/issues/54001:26
*** Son_Goku has joined #rdo01:31
rdogerritDavid Moreau Simard created rdo-infra/rdo-container-registry master: WIP: Improve docs  https://review.rdoproject.org/r/1043301:32
*** fragatina has quit IRC01:35
*** kellyp has quit IRC01:40
*** kellyp has joined #rdo01:41
*** kellyp has quit IRC01:45
*** sreichar has joined #rdo01:52
*** jlabarre has quit IRC01:56
*** rwsu has joined #rdo02:01
*** Son_Goku has quit IRC02:04
*** kellyp has joined #rdo02:09
*** kellyp has quit IRC02:13
*** Son_Goku has joined #rdo02:21
*** jkilpatr has quit IRC02:27
*** jethro7 has quit IRC02:28
*** jethro7 has joined #rdo02:28
*** Son_Goku has quit IRC02:31
*** suresh12_ has quit IRC02:42
*** openstack has joined #rdo02:44
*** ChanServ sets mode: +o openstack02:44
*** rodrigods has quit IRC02:50
*** links has joined #rdo02:52
*** rodrigods has joined #rdo02:58
*** paragan has joined #rdo03:07
*** apetrich has quit IRC03:09
*** sreichar has quit IRC03:16
*** jhershbe has joined #rdo03:17
*** gildub has joined #rdo03:20
*** gildub has quit IRC03:21
*** gildub has joined #rdo03:21
*** thrash is now known as thrash|g0ne03:24
*** liujiong has quit IRC03:28
*** liujiong has joined #rdo03:34
*** suresh12 has joined #rdo03:36
*** suresh12 has quit IRC03:40
*** jhershbe has quit IRC03:44
*** jhershbe has joined #rdo03:47
EmilienMihrachys: ack03:49
*** ihrachys has quit IRC03:49
*** brault has joined #rdo03:51
*** gildub has quit IRC03:51
*** brault has quit IRC03:55
*** jhershbe has quit IRC04:02
*** ykarel|afk has joined #rdo04:08
*** udesale has joined #rdo04:10
*** dbecker has quit IRC04:24
*** fragatina has joined #rdo04:36
*** chlong has quit IRC04:36
*** chlong has joined #rdo04:37
*** dbecker has joined #rdo04:38
*** fragatina has quit IRC04:38
*** fragatina has joined #rdo04:38
*** fragatina has quit IRC04:39
*** fragatina has joined #rdo04:40
*** shreshtha has joined #rdo04:41
*** paragan has quit IRC04:55
*** paragan has joined #rdo04:55
*** kellyp has joined #rdo04:59
*** ratailor has joined #rdo05:01
*** paragan has quit IRC05:01
*** kellyp has quit IRC05:03
*** cylopez has joined #rdo05:05
*** cylopez has quit IRC05:06
*** jatanmalde has joined #rdo05:06
*** cylopez has joined #rdo05:07
*** cylopez has quit IRC05:07
*** cylopez has joined #rdo05:09
*** suresh12 has joined #rdo05:12
*** paragan has joined #rdo05:13
*** janki has joined #rdo05:17
*** anilvenkata has quit IRC05:19
*** anilvenkata has joined #rdo05:19
*** imcleod has quit IRC05:24
*** sasha2 has quit IRC05:32
*** chlong has quit IRC05:39
*** anilvenkata has quit IRC05:40
*** anilvenkata has joined #rdo05:47
*** pchavva has joined #rdo05:48
*** rbartal has joined #rdo05:54
*** pchavva has quit IRC05:56
*** anilvenkata has quit IRC05:57
*** dpawar has joined #rdo05:58
*** paragan has quit IRC05:59
*** jackNemrod has joined #rdo06:01
*** fragatina has quit IRC06:02
*** pkovar has joined #rdo06:07
*** rbartal has quit IRC06:10
*** anilvenkata has joined #rdo06:16
*** pchavva has joined #rdo06:20
*** abregman has joined #rdo06:22
*** jackNemrod has quit IRC06:24
*** paragan has joined #rdo06:27
*** pkovar has quit IRC06:30
*** morazi has joined #rdo06:36
*** paragan has quit IRC06:37
*** morazi has quit IRC06:42
*** suresh12 has quit IRC06:46
*** paragan has joined #rdo06:50
*** mosulica has joined #rdo06:54
*** iranzo has joined #rdo06:55
*** Son_Goku has joined #rdo07:01
*** pgadiya has joined #rdo07:11
*** jhershbe has joined #rdo07:12
*** dpawar has quit IRC07:14
*** Son_Goku has quit IRC07:16
*** jprovazn has joined #rdo07:17
*** spectr has quit IRC07:22
*** zigo has quit IRC07:24
*** chasmo77 has joined #rdo07:24
*** egonzalez has joined #rdo07:28
*** zigo has joined #rdo07:28
*** dpawar has joined #rdo07:29
*** flepied has joined #rdo07:40
*** pchavva has quit IRC07:42
*** ykarel|afk is now known as ykarel|lunch07:47
*** emalikov has joined #rdo07:48
*** flepied has quit IRC07:51
*** hogepodge has quit IRC08:02
*** hogepodge has joined #rdo08:04
*** rbartal has joined #rdo08:04
*** morazi has joined #rdo08:05
*** apetrich has joined #rdo08:06
*** pcaruana has joined #rdo08:06
*** mcornea has joined #rdo08:07
*** tshefi has joined #rdo08:08
*** brault has joined #rdo08:12
*** aufi has joined #rdo08:15
*** brault has quit IRC08:16
*** toanju has joined #rdo08:20
*** tesseract has joined #rdo08:23
*** jackNemrod has joined #rdo08:23
*** bsurfer has joined #rdo08:26
*** trown|outtypewww has quit IRC08:26
*** trown has joined #rdo08:27
*** martinkopec has joined #rdo08:27
*** zoli is now known as zli08:28
*** zli is now known as zoli08:28
*** jpena|off is now known as jpena08:30
*** brault has joined #rdo08:30
jpenagood morning08:34
*** oidgar has joined #rdo08:35
*** martinkopec has quit IRC08:36
*** ccamacho has joined #rdo08:36
*** flepied has joined #rdo08:40
*** jackNemrod has quit IRC08:43
*** racedo has joined #rdo08:43
*** jackNemrod has joined #rdo08:43
*** amoralej|off is now known as amoralej08:47
amoralejgood morning08:47
*** gkadam has joined #rdo08:53
*** bsurfer has quit IRC08:53
*** ykarel|lunch is now known as ykarel08:53
*** dbrandon has quit IRC08:59
*** kellyp has joined #rdo08:59
*** jpich has joined #rdo09:02
*** kellyp has quit IRC09:04
*** bsurfer has joined #rdo09:13
*** lucas-afk is now known as lucasagomes09:17
*** sbulage has joined #rdo09:27
*** paragan has quit IRC09:27
*** liujiong has quit IRC09:30
openstackgerritJavier Peña proposed openstack/packstack master: [WIP] Multinode CI job for Packstack  https://review.openstack.org/51764409:31
openstackgerritJavier Peña proposed openstack/packstack master: [WIP] Multinode CI job for Packstack  https://review.openstack.org/51764409:38
rdogerritMerged rdoinfo master: Promote CBS tags update for pike-release  https://review.rdoproject.org/r/1037909:38
rdogerritAlfredo Moralejo proposed rdoinfo master: Promote CBS tags update for pike-testing  https://review.rdoproject.org/r/1043209:40
rdogerritAlfredo Moralejo proposed rdoinfo master: Promote CBS tags update for pike-testing  https://review.rdoproject.org/r/1043209:40
*** kellyp has joined #rdo09:42
*** derekh has joined #rdo09:42
*** tosky has joined #rdo09:46
*** kellyp has quit IRC09:46
*** vnogin has joined #rdo09:49
*** mvk has quit IRC09:52
*** vnogin has quit IRC09:53
*** vnogin has joined #rdo09:54
*** Son_Goku has joined #rdo09:56
*** gfidente has joined #rdo10:02
*** gfidente has quit IRC10:02
*** gfidente has joined #rdo10:02
*** Son_Goku has quit IRC10:07
*** wolverineav has joined #rdo10:07
*** toanju has quit IRC10:08
*** wolverineav has quit IRC10:13
rdogerritYatin Karel created rdo-infra/ci-config master: Use delorean_current_hash in DELOREAN_URL  https://review.rdoproject.org/r/1043410:18
ykarelamoralej, ^^10:18
*** mvk has joined #rdo10:19
*** jatanmalde has quit IRC10:22
*** jatanmalde has joined #rdo10:22
*** toanju has joined #rdo10:26
*** jchhatbar has joined #rdo10:26
*** janki has quit IRC10:26
*** dbrandon has joined #rdo10:26
*** paragan has joined #rdo10:29
*** abregman has quit IRC10:32
*** pgadiya has quit IRC10:33
*** janki has joined #rdo10:34
*** jchhatbar has quit IRC10:35
*** gfidente has quit IRC10:36
*** fragatina has joined #rdo10:39
*** toanju has quit IRC10:43
*** links has quit IRC10:45
*** lebauce has joined #rdo10:49
*** wolverineav has joined #rdo10:55
*** toanju has joined #rdo10:56
*** links has joined #rdo10:57
*** ykarel is now known as ykarel|afk10:57
*** racedo has quit IRC10:59
rdogerritJavier Peña proposed rdo-infra/puppet-dlrn master: Additional configuration to support SELinux=Enforcing  https://review.rdoproject.org/r/736911:03
*** mvk_ has joined #rdo11:03
*** mvk has quit IRC11:04
*** Son_Goku has joined #rdo11:13
*** rfolco has joined #rdo11:15
*** sfbender has joined #rdo11:15
sfbenderMatthieu Huin created rdopkg master: Split utils.cmd, create separate utils.git module  https://softwarefactory-project.io/r/1011411:15
*** pblaho has quit IRC11:20
sshnaidmdmsimard, hi, can we post to gerrit the pipelines name in comment? like it's done in infra now11:22
openstackgerritJavier Peña proposed openstack/packstack master: [WIP] Multinode CI job for Packstack  https://review.openstack.org/51764411:22
*** vnogin has quit IRC11:22
sshnaidmdmsimard, like "Zuul check-tripleo"11:23
*** vnogin has joined #rdo11:23
*** pkovar has joined #rdo11:24
*** Son_Goku has quit IRC11:25
*** ragiman has joined #rdo11:25
*** Amita has quit IRC11:26
*** Amita has joined #rdo11:27
*** udesale has quit IRC11:28
*** martinkopec has joined #rdo11:28
*** links has quit IRC11:30
*** Son_Goku has joined #rdo11:30
sfbenderJavier Peña proposed DLRN master: Improve filtering in civotes_detail.html  https://softwarefactory-project.io/r/1009511:31
*** shreshtha has quit IRC11:35
*** Amita has quit IRC11:35
*** Amita has joined #rdo11:37
*** Son_Goku has quit IRC11:40
*** links has joined #rdo11:43
openstackgerritAlfredo Moralejo proposed openstack/packstack stable/pike: Update Cirros image to 0.3.5  https://review.openstack.org/51830211:47
*** toanju has quit IRC11:47
*** dtantsur|afk is now known as dtantsur11:49
openstackgerritAlfredo Moralejo proposed openstack/packstack stable/ocata: Update Cirros image to 0.3.5  https://review.openstack.org/51830411:51
*** abregman has joined #rdo11:54
*** racedo has joined #rdo12:00
*** jpena is now known as jpena|lunch12:00
*** lucasagomes is now known as lucas-hungry12:04
*** jkilpatr has joined #rdo12:04
*** ykarel|afk is now known as ykarel12:07
*** julim has quit IRC12:08
*** julim has joined #rdo12:09
amoralejjobs keep failing with the jenkins slaves going offline12:16
*** Son_Goku has joined #rdo12:16
*** thrash|g0ne is now known as thrash12:18
*** brault has quit IRC12:18
*** gfidente has joined #rdo12:19
*** gfidente has quit IRC12:19
*** gfidente has joined #rdo12:19
*** toanju has joined #rdo12:20
*** iranzo has quit IRC12:20
*** jayg|g0n3 is now known as jayg12:20
*** morazi has quit IRC12:21
*** abregman is now known as abregman|mtg12:21
*** janki has quit IRC12:22
*** cshastri has quit IRC12:23
*** iranzo has joined #rdo12:24
*** iranzo has quit IRC12:24
*** iranzo has joined #rdo12:24
panda|ruckamoralej: ok, I'll remove the tag, and will have to look at the undercloud removal experimental job12:26
*** pblaho has joined #rdo12:26
amoralejpanda|ruck, i was looking for some review on tripleo-ci to run experimental12:26
amoralejdo you have any?12:26
*** flepied has quit IRC12:27
*** pkovar has quit IRC12:29
*** toanju has quit IRC12:32
*** pradk has joined #rdo12:33
panda|ruckamoralej:  only containers build fail in that way ?12:34
*** catintheroof has joined #rdo12:34
*** Son_Goku has quit IRC12:35
*** gszasz has joined #rdo12:35
*** links has quit IRC12:37
*** dpawar has quit IRC12:40
*** flepied has joined #rdo12:42
*** kellyp has joined #rdo12:42
*** toanju has joined #rdo12:44
*** catintheroof has quit IRC12:45
*** Son_Goku has joined #rdo12:47
*** kellyp has quit IRC12:47
amoralejpanda|ruck, not12:50
amoralejpanda|ruck, today at 10:37 12 jobs failed at the same time12:52
*** aditya_r has joined #rdo12:52
*** jlabarre has joined #rdo12:56
*** catintheroof has joined #rdo12:57
*** lucas-hungry is now known as lucasagomes12:57
*** imcleod has joined #rdo12:59
*** Son_Goku has quit IRC13:01
*** rlandy has joined #rdo13:02
*** eharney has quit IRC13:02
*** dpawar has joined #rdo13:03
*** jcoufal has joined #rdo13:04
panda|ruckamoralej: I think any change will work really, you can create a new one if you like. I'll have to look up a t the gates jobs now13:06
*** spr has quit IRC13:06
*** ratailor has quit IRC13:08
*** Son_Goku has joined #rdo13:08
pradkamoralej, Hi13:09
pradkamoralej, regarding https://review.rdoproject.org/r/#/c/10380/ .. seems like its fixed, but the packaging is not reflecting. Can we have a promotion ?13:09
*** shreshtha has joined #rdo13:11
*** spr has joined #rdo13:11
*** aditya_r has quit IRC13:15
amoralejdmsimard, according to kibana, from 10:37 to 10:40 we had ~ 40 jobs failing with the offlime message13:15
*** aditya_r has joined #rdo13:15
*** fragatina has quit IRC13:16
dmsimardwow wtf13:16
amoralejall of them tripleo related13:16
*** abregman|mtg is now known as abregman13:17
dmsimardneed to drop kid at school for now..13:18
dmsimardbtw hostpath is not as simple as it seemed, see my comment https://github.com/openshift/openshift-ansible/pull/6022#issuecomment-34234752213:19
amoralejdmsimard, ok13:19
amoralejhttps://review.rdoproject.org/app/kibana#/discover/Slave-nodes-failure-lat-4-hours?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-4h,mode:quick,to:now))&_a=(columns:!(build_name,build_change),filters:!(),index:'logstash-*',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'%22Slave%20went%20offline%20during%20the%20build%22')),sort:!('@timestamp',desc))13:19
dmsimardspent a lot of time on that yesterday...13:19
amoralejdmsimard, i thought kubernetes would take care of setting permissions and selinux context13:20
amoralejups, then we need to take care of it13:20
*** jmelvin has joined #rdo13:21
dmsimardIt does13:21
dmsimardBut you need to tell it what to do13:21
dmsimardAnd that's part of the scc13:22
dmsimardFilesystem permissions are also involved13:22
dmsimardbrb13:22
amoralejok13:22
*** pcaruana has quit IRC13:22
*** sreichar has joined #rdo13:23
*** rbartal has quit IRC13:24
*** panda|ruck is now known as panada|ruck|brb13:28
amoralejpradk, we promoted recently13:31
amoralejit should be13:31
amoralejlet me check13:31
*** ragiman has quit IRC13:32
*** MrAbaddon has joined #rdo13:32
amoralejpradk, in rdo trunk it's there already13:33
amoralejrpm -q --requires -p https://trunk.rdoproject.org/centos7-pike/current-tripleo-rdo/openstack-ceilometer-common-9.0.2-0.20171103153425.7060591.el7.centos.noarch.rpm|grep tooz13:33
*** brault has joined #rdo13:34
rdogerritMerged rdoinfo master: Promote CBS tags update for pike-testing  https://review.rdoproject.org/r/1043213:34
amoralejpradk, in cloudsig repo, it was promoted to testing in https://review.rdoproject.org/r/#/c/1036913:35
amoralejit will be in -release in next bot execution13:35
amoralejtoday13:35
*** cylopez1 has joined #rdo13:35
*** rbartal has joined #rdo13:36
*** cylopez has quit IRC13:36
*** julim has quit IRC13:36
rdogerritAlfredo Moralejo proposed rdoinfo master: Bump rdoinfo pike tags to latest upper-constraints  https://review.rdoproject.org/r/1043013:37
pradkamoralej, cool, thx!13:40
dmsimardamoralej: http://paste.openstack.org/raw/625724/13:41
dmsimardlooks like an i/o related xfs trace13:41
amoralejdmsimard, that's review.r.o?13:41
dmsimardyeah13:42
*** cylopez1 has quit IRC13:42
amoralejdmsimard, but timestamp doesn't match with observed job failures13:43
amoralejdmsimard, http://bit.ly/2hhdx9513:43
amoralejand yes, from 74 failed jobs in last 24hr, only one is non-tripleo13:44
amoralejmaybe connectivity with tripleo tenant?13:44
*** panada|ruck|brb is now known as panda|ruck13:47
dmsimardamoralej: there's something weird..13:48
dmsimardjournalctl -u jenkins stops at Nov 07 05:01:0413:48
dmsimarddate is now Tue Nov  7 13:47:51 UTC 201713:48
*** kgiusti has joined #rdo13:48
fbomaybe not related but we have some issues with our nodepool. Lot of ssh timeout access to built node.13:49
dmsimardamoralej: there are significant plugin updates available13:51
dmsimardamoralej: we should consider doing an update13:51
dmsimardssh agent from 1.4.1 to 1.15..13:51
*** thrash is now known as thrash|biab13:54
*** jpena|lunch is now known as jpena13:54
*** amoralej is now known as amoralej|lunch13:55
dmsimardalso only 1024MB for heap, we should increase that /usr/bin/java -Xmx1024m -Dhudson.model.ParametersAction.keepUndefinedParameters=true -jar /usr/lib/jenkins/jenkins.war --webroot=/var/cache/jenkins/war --httpPort=8082 --ajp13Port=-1 --prefix=/jenkins --httpListenAddress=127.0.0.113:57
*** kgiusti has left #rdo13:58
*** udesale has joined #rdo13:59
*** sasha21 has joined #rdo14:00
*** catintheroof has quit IRC14:03
*** eharney has joined #rdo14:03
openstackgerritJavier Peña proposed openstack/packstack master: [WIP] Multinode CI job for Packstack  https://review.openstack.org/51764414:04
dmsimardjpena: I wonder if the slave disconnections could be caused by memory pressure on the slaves14:05
dmsimardjpena: the jobs are designed to run within 8GB of ram upstream but they also don't have a jenkins java process running14:06
*** anshul has quit IRC14:09
*** catintheroof has joined #rdo14:10
rdogerritYatin Karel proposed rdo-infra/ci-config master: Store variables required by promotion script in properties file  https://review.rdoproject.org/r/1043414:12
dmsimardjpena, amoralej|lunch: I'll schedule a maintenance tonight to update the plugins in Jenkins.14:12
*** dpawar has quit IRC14:13
jruzickajschlueter, when I remove %{?dist} duplication, release is bumped 1.0%{?dist}.1 -> 1.1%{?dist}.1 instead of -> 1.0%{?dist}.2 as you suggested14:18
*** jmlowe has quit IRC14:18
jruzickajschlueter, is that OK?14:18
jpenadmsimard: mmm... do we get the memory statistics from the CI jobs? Let me check14:19
sfbenderMerged rdopkg master: tests: add topy to whitelist_externals  https://softwarefactory-project.io/r/1006714:20
dmsimardjpena: pabelanger did confirm that we are definitely running into the same kind of issues they were having with upstream jenkins..14:21
*** aufi has quit IRC14:21
dmsimardWe're stuck between a rock and a hard place14:22
jpenadmsimard: so our alternative is to move to zuul-launcher14:22
dmsimardIt's an all or nothing deal, we can't run both simultaneously and it breaks some implementations14:22
dmsimardLike postbuildscript which is a Jenkins plugin14:23
dmsimardAt this point we'd have to try to find a way to hold out until v3..14:23
*** aditya_r has quit IRC14:23
dmsimardI want to increase the Jenkins heap but I'm not finding where.. google says jenkins.xml but there's no such thing14:24
*** anshul has joined #rdo14:25
jpenadmsimard: it seems to be hardcoded in the systemd unit: /usr/bin/java -Xmx1024m -Dhudson.model.ParametersAction.keepUndefinedParameters=true ...14:26
jpenabtw, I can see this in /etc/sysconfig/jenkins, maybe it could help somewhere14:27
jpena# Maximum number of HTTP worker threads.14:27
jpena#14:27
jpenaJENKINS_HANDLER_MAX="100"14:27
*** thrash|biab is now known as thrash14:27
*** hewbrocc` is now known as hewbrocca14:30
sfbenderJakub Ružička created rdopkg master: spec: don't duplicate %{?dist}  https://softwarefactory-project.io/r/1012014:33
*** iOsX has joined #rdo14:33
*** dhill_ has quit IRC14:34
sfbenderJakub Ružička proposed rdopkg master: spec: don't duplicate %{?dist}  https://softwarefactory-project.io/r/1012014:34
iOsXHi everyone :) Is it possible to run packstack after sucessfull deployment and separately install only Panko service?14:35
jpenaiOsX: if you just want to add Panko, you can use the same answer file you got from the first execution, enable Panko there, then run "packstack --answer-file=<answer-file>" again14:36
*** aufi has joined #rdo14:36
iOsXgreat, thanks14:37
*** julim has joined #rdo14:37
iOsXjpena: what about if I have already changed a lot since last deployment?14:40
jpenaiOsX: then it's probably a bad idea. Packstack is not meant to install a single service in isolation14:41
iOsXok, then another question14:41
*** dhill_ has joined #rdo14:41
iOsXin official documentation I see a bit different approach. Seems like it wasn't written for RedHat/CentOS distributives14:42
iOsXIf I correct understand developers team has some kind of ansible playbooks or puppet modules to configure panko and other services. Is it possible to get them to investigate how to configure manually ?14:44
*** jcoufal has quit IRC14:45
*** pcaruana has joined #rdo14:47
*** aufi_ has joined #rdo14:50
*** amoralej|lunch is now known as amoralej14:51
jpenaiOsX: the puppet manifests that Packstack uses to configure Panko are: https://github.com/openstack/packstack/blob/master/packstack/puppet/modules/packstack/manifests/keystone/panko.pp and https://github.com/openstack/packstack/blob/master/packstack/puppet/modules/packstack/manifests/panko.pp14:51
iOsXthanks a lot, man14:51
*** brault has quit IRC14:51
jpenathey rely on the puppet-module (already set up for you under /usr/share/openstack-puppet/modules/panko), and they use some values from Hiera. You can find the hiera data for your setup at /var/tmp/packstack/latest/hieradata14:52
*** aufi has quit IRC14:54
amoralejjpena, dmsimard we may need to override ExecStart in jenkins unit file14:56
amoralejwe need to change -Xmx14:56
*** brault has joined #rdo14:58
*** jmlowe has joined #rdo14:59
*** artom has quit IRC14:59
*** spectr has joined #rdo15:01
*** apevec has joined #rdo15:01
*** apevec has quit IRC15:01
*** apevec has joined #rdo15:01
*** flepied has quit IRC15:05
*** flepied has joined #rdo15:06
jpenaamoralej: we can probably do it in a file under /etc/systemd/system, so we don't have to touch the rpm-provided file15:06
amoralejyeah, that's what i meant15:06
jpenadmsimard: have you seen any out of heap errors?15:06
*** ykarel is now known as ykarel|away15:06
rdogerritMerged rdoinfo master: Bump rdoinfo pike tags to latest upper-constraints  https://review.rdoproject.org/r/1043015:07
dmsimardjpena: not really tbh, just want to try and give it more breathing room if we can afford it15:08
*** anshul has quit IRC15:08
rdogerritrdo-trunk created openstack/mistralclient-distgit pike-rdo: Update to 3.1.4  https://review.rdoproject.org/r/1043515:09
rdogerritrdo-trunk created openstack/paunch-distgit pike-rdo: Update to 1.5.1  https://review.rdoproject.org/r/1043615:09
dmsimardthere https://wiki.jenkins.io/display/JENKINS/Consideration+for+Large+Scale+Jenkins+Deployment but seems fairly outdated15:09
*** martinkopec has quit IRC15:09
*** ykarel|away has quit IRC15:15
*** chlong has joined #rdo15:15
amoralejdmsimard, jpena it may be it's hitting nofile limits?15:16
amoralejlimit is 409615:16
amoralejright now it's using 137715:17
*** emalikov has quit IRC15:18
amoralejmmm, i don't think so15:20
*** anshul has joined #rdo15:20
rdogerritMerged openstack/mistralclient-distgit pike-rdo: Update to 3.1.4  https://review.rdoproject.org/r/1043515:31
dmsimardjpena: http://bit.ly/2hhdx9515:32
rdogerritMerged openstack/paunch-distgit pike-rdo: Update to 1.5.1  https://review.rdoproject.org/r/1043615:36
*** kgiusti has joined #rdo15:38
*** eharney has quit IRC15:40
*** suresh12 has joined #rdo15:45
*** toure_biab is now known as toure15:45
*** ccamacho has quit IRC15:49
*** iOsX has quit IRC15:49
dmsimardweshay, adarazs|rover, amoralej: FYI nodes running tripleo jobs are all heavily swapping...15:50
amoralejthat's bad15:51
dmsimardhttps://i.imgur.com/tvrWmff.png15:51
dmsimardlook at the upstream-* nodes15:51
dmsimardI took https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-nv/1950/console as an example: http://paste.openstack.org/raw/625752/15:52
dmsimardvm.swappiness = 30 so it shouldn't swap for fun unless required15:53
*** eharney has joined #rdo15:53
dmsimardmysql uses the most ram, after that it's heat15:54
dmsimardit seems like each service has 4 processes but heat has 815:54
dmsimardheat-engine15:54
*** jhershbe has quit IRC15:55
dmsimardmwhahaha: ^ just FYI, not sure if you're seeing the same memory starvation upstream15:55
mwhahahawouldn't be shocked if we were15:55
*** janki has joined #rdo15:56
dmsimardmwhahaha: is there a dstat or something ?15:56
mwhahahawe run dstat15:56
mwhahahaor at least we used to15:57
dmsimardcan't find it in the logs of a job like http://logs.openstack.org/82/479882/9/check-tripleo/legacy-tripleo-ci-centos-7-ovb-ha-oooq/36f5073/15:57
mwhahahayea i'm looking15:57
mwhahahait rusn15:58
mwhahahahttp://logs.openstack.org/18/516218/1/gate/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq/4aa4488/job-output.txt.gz#_2017-11-07_12_42_37_386776\15:58
mwhahahabut i bet we lost the logs in the capture15:58
dmsimardlol15:58
* mwhahaha sighs15:58
dmsimard/var/log/extra/dstat-csv.log15:58
dmsimardwe should have that15:58
mwhahahayou'd think wouldn't you15:59
mwhahahawe should have a lot of things we don't15:59
* mwhahaha is extra salty this morning15:59
dmsimardnothing here though http://logs.openstack.org/18/516218/1/gate/legacy-tripleo-ci-centos-7-scenario004-multinode-oooq/4aa4488/logs/undercloud/var/log/extra/15:59
dmsimardtrown: ^ do you know ?15:59
*** brault has quit IRC16:00
*** spectr has quit IRC16:00
*** artom has joined #rdo16:00
*** spr has quit IRC16:00
dmsimardmwhahaha: FWIW I did something else for dstat when I was ansibilizing p-o-i16:01
* dmsimard searches16:01
*** ihrachys has joined #rdo16:01
dmsimardmwhahaha: https://review.openstack.org/#/c/404471/37/tests/roles/setup/tasks/base.yml@2416:01
dmsimardand https://review.openstack.org/#/c/404471/37/tests/roles/setup/files/dstat.service16:02
*** mosulica has quit IRC16:02
mwhahahashould get that into quickstart16:02
dmsimardand then at the end you can do like16:02
dmsimardjournalctl -u dstat > somelogfile16:02
pradkamoralej, so what was the conclusion for the bson package depedency when we removed the --skip-metering-db?16:03
openstackgerritMerged openstack/packstack stable/ocata: Update Cirros image to 0.3.5  https://review.openstack.org/51830416:03
openstackgerritMerged openstack/packstack stable/pike: Update Cirros image to 0.3.5  https://review.openstack.org/51830216:03
pradkamoralej, i see it in https://review.openstack.org/#/c/515796/16:03
amoralejdmsimard, sounds strange to me that, if the problem is related to the load inside the instances, we have many jobs failing at the same time16:03
dmsimardamoralej: let's not put the blame on any single issue16:03
pradkamoralej, waiting on promotion i guess?16:04
amoralejpradk, we had promotions with new packages, so we should be able to merge the drop skip-metering-database reviews16:04
dmsimardamoralej: the fact that jobs are swapping (a lot) likely does not help with performance/timeout and ultimately the java slave process16:04
*** oidgar has quit IRC16:04
*** spr has joined #rdo16:04
pradkamoralej, hmm ok, i rebased lets see if it picks up new packages now16:04
pradkthx16:05
dmsimardamoralej: I'd like to attack this problem on multiple fronts for the things we can do something about -- we'll update the plugins, increase heap16:05
*** suresh12 has quit IRC16:05
dmsimardbut I mean, we have to highlight what are the other potential issues16:05
amoralejyes, sounds good to consider all potential issues16:06
amoralejand trying to grab as much data as possible16:06
*** dtrainor has quit IRC16:09
*** dtrainor has joined #rdo16:10
*** trown is now known as trown|brb16:11
*** fragatina has joined #rdo16:11
dmsimardmwhahaha: I'll send a patch for the quickstart dstat thing I guess16:11
dmsimardwasn't there a tool to generate a dstat graph ?16:12
*** egonzalez has quit IRC16:12
dmsimardmwhahaha: I found https://github.com/Dabz/dstat_graph16:13
dmsimard¯\_(ツ)_/¯16:13
mwhahahayea i've seen that16:14
*** trown|brb is now known as trown16:14
*** iranzo has quit IRC16:14
mwhahahaor rather it lets you drop in the data file and graphs it16:14
dmsimardoh wow it doesn't look half bad16:15
*** aufi_ has quit IRC16:15
*** jatanmalde has quit IRC16:17
*** Jack_Iv has joined #rdo16:21
dmsimardmwhahaha, amoralej: example dstat graph https://dmsimard.com/files/dstat.html16:21
*** Son_Goku has quit IRC16:24
openstackgerritAlfredo Moralejo proposed openstack/packstack master: Remove --skip-metering-database when calling ceilometer-upgrade  https://review.openstack.org/51575316:25
*** gvrangan has joined #rdo16:25
amoralejpradk, ^ testing it in packstac too16:26
pradkcool16:26
*** fragatina has quit IRC16:26
*** toanju has quit IRC16:26
*** thrash is now known as thrash|biab16:27
*** tshefi has quit IRC16:30
*** Jack_Iv has quit IRC16:30
*** vnogin has quit IRC16:36
*** pradk has quit IRC16:36
*** udesale has quit IRC16:38
*** iOsX has joined #rdo16:38
*** aludwar has quit IRC16:38
*** jhershbe has joined #rdo16:40
*** Jack_Iv has joined #rdo16:44
*** anshul has quit IRC16:44
*** suresh12 has joined #rdo16:46
*** abregman has quit IRC16:47
*** owlbot has quit IRC16:50
*** anilvenkata has quit IRC16:50
openstackgerritJavier Peña proposed openstack/packstack master: Add multinode CI job  https://review.openstack.org/51764416:51
*** mvk_ has quit IRC16:52
*** agurenko has quit IRC16:52
*** pradk has joined #rdo16:52
openstackgerritJavier Peña proposed openstack/packstack master: Add multinode CI job  https://review.openstack.org/51764416:53
*** hamzy has quit IRC16:55
*** owlbot has joined #rdo16:58
*** newmember has quit IRC16:58
*** vnogin has joined #rdo16:58
*** newmember has joined #rdo16:58
*** anilvenkata has joined #rdo16:59
sfbenderJavier Peña proposed DLRN master: Update Fedora default mock config to use Rawhide  https://softwarefactory-project.io/r/992617:00
*** gvrangan has quit IRC17:04
*** jackNemrod has quit IRC17:04
rdogerritJavier Peña proposed rdo-infra/puppet-dlrn master: Fixes for Fedora Rawhide builds  https://review.rdoproject.org/r/1003617:07
*** jpich has quit IRC17:08
*** oidgar has joined #rdo17:11
*** flepied has quit IRC17:14
*** imcsk8 has quit IRC17:15
*** dtantsur is now known as dtantsur|afk17:16
*** mcornea has quit IRC17:16
*** iOsX has quit IRC17:17
*** lucasagomes is now known as lucas-afk17:18
*** fragatina has joined #rdo17:18
*** jhershbe has quit IRC17:19
*** imcsk8 has joined #rdo17:20
*** kellyp has joined #rdo17:21
*** lebauce has quit IRC17:23
*** gvrangan has joined #rdo17:28
*** gkadam has quit IRC17:29
dmsimardmwhahaha: got something up real quick https://review.openstack.org/#/c/518374/ don't have time to push it through for now, need to wrap up other things17:30
*** tesseract has quit IRC17:30
*** janki has quit IRC17:30
*** pchavva has joined #rdo17:31
mwhahahadmsimard: ok thanks17:31
sfbenderJavier Peña proposed DLRN master: Update Fedora default mock config to use Rawhide  https://softwarefactory-project.io/r/992617:31
*** janki has joined #rdo17:31
*** janki has quit IRC17:32
*** iOsX has joined #rdo17:33
*** jmelvin has quit IRC17:36
*** tesseract has joined #rdo17:40
*** Jack_Iv has quit IRC17:40
dmsimardjpena: nodepool already knows about the new parameter of the ssh slaves plugin https://github.com/openstack-infra/nodepool/blob/881b2fcba6be3654c3ca7aa3eaced110c2414e3a/nodepool/jenkins_manager.py#L3217:41
dmsimardjust confirming we have that on review.rdo..17:41
dmsimardwe don't, actually17:41
dmsimardhttps://github.com/openstack-infra/nodepool/commit/56e0172805524d464ecb59285f9053ffdbfe348717:41
dmsimardthanks bkero :P17:42
dmsimardnhicher, tristanC: ^ this is a clean backport but doesn't exist in nodepool v3, I'm not sure how you handle the packaging for that17:43
*** shreshtha has quit IRC17:44
dmsimardI see https://github.com/softwarefactory-project/nodepool-distgit but at first glance that's not nodepool v317:44
*** shreshtha has joined #rdo17:44
*** pradk has quit IRC17:44
*** iOsX has quit IRC17:45
*** thrash|biab is now known as thrash17:46
*** dyasny has joined #rdo17:46
*** dyasny_ has quit IRC17:48
*** sbulage has quit IRC17:48
*** tesseract has quit IRC17:49
*** pradk has joined #rdo17:51
*** suresh12 has quit IRC17:53
*** trown is now known as trown|lunch17:54
*** gszasz has quit IRC18:00
*** julim_ has joined #rdo18:03
*** julim has quit IRC18:05
*** pchavva has quit IRC18:05
openstackgerritJavier Peña proposed openstack/packstack master: Add multinode CI job  https://review.openstack.org/51764418:05
*** racedo has quit IRC18:07
*** jpena is now known as jpena|off18:09
*** MrAbaddon has quit IRC18:10
*** amoralej is now known as amoralej|off18:11
*** flepied has joined #rdo18:15
sfbenderMerged rdopkg master: Split utils.cmd, create separate utils.git module  https://softwarefactory-project.io/r/1011418:16
*** mary_grace has joined #rdo18:19
*** gszasz has joined #rdo18:20
*** dneary has quit IRC18:21
*** oidgar has quit IRC18:28
*** suresh12 has joined #rdo18:29
*** panda|ruck is now known as panda|ruck|bbl18:30
*** suresh12 has quit IRC18:35
*** gszasz has quit IRC18:36
*** eharney has quit IRC18:37
*** suresh12 has joined #rdo18:38
*** hamzy has joined #rdo18:45
*** iOsX has joined #rdo18:48
*** derekh has quit IRC18:50
*** marrusl has quit IRC18:50
*** jackNemrod has joined #rdo18:51
*** eharney has joined #rdo18:53
*** iOsX has quit IRC18:57
*** jackNemrod has quit IRC18:57
*** jackNemrod has joined #rdo18:57
*** julim_ has quit IRC19:00
*** d0ugal has quit IRC19:01
*** julim has joined #rdo19:01
*** marrusl has joined #rdo19:02
*** trown|lunch is now known as trown19:07
*** gvrangan has quit IRC19:10
*** jhershbe has joined #rdo19:24
*** amoralej|off is now known as amoralej19:28
*** derekh has joined #rdo19:44
*** amoralej is now known as amoralej|off19:45
*** jackNemrod has quit IRC19:48
*** jackNemrod has joined #rdo19:48
*** derekh has quit IRC19:49
*** jhershbe has quit IRC19:51
*** eck` is now known as eck`gone19:54
*** eck`gone is now known as eck`19:54
*** mary_grace has quit IRC20:02
*** mary_grace has joined #rdo20:04
*** jackNemrod has quit IRC20:09
*** catintheroof has quit IRC20:15
*** dneary has joined #rdo20:16
*** lebauce has joined #rdo20:21
*** jprovazn has quit IRC20:26
*** pkovar has joined #rdo20:27
*** kellyp has quit IRC20:29
*** dhill_ has quit IRC20:31
*** pkovar has quit IRC20:32
*** jmelvin has joined #rdo20:38
*** dhill_ has joined #rdo20:38
*** artom has quit IRC20:41
*** catintheroof has joined #rdo20:46
*** catintheroof has quit IRC20:51
*** jkilpatr has quit IRC20:54
*** catintheroof has joined #rdo20:58
*** kellyp has joined #rdo20:59
*** catintheroof has quit IRC21:02
*** jackNemrod has joined #rdo21:04
*** chlong has quit IRC21:04
*** kellyp has quit IRC21:05
*** julim has quit IRC21:08
*** panda|ruck|bbl is now known as panda|ruck21:08
*** kellyp has joined #rdo21:08
*** catintheroof has joined #rdo21:11
*** jkilpatr has joined #rdo21:13
*** catintheroof has quit IRC21:15
*** dneary has quit IRC21:17
*** MrAbaddon has joined #rdo21:21
*** CosmicSound has joined #rdo21:25
*** jmlowe has quit IRC21:26
*** kgiusti has left #rdo21:30
*** kellyp has quit IRC21:32
*** kellyp has joined #rdo21:33
*** toure is now known as toure_biab21:33
*** jackNemrod has quit IRC21:33
*** eharney has quit IRC21:36
dmsimardapevec, jpena|off, amoralej|off: I managed to resolve the issue I had with 3.7, I respun the dev instance and it looks like it's good to go now. I'll put it under some load.21:39
apevecdmsimard, just pull and push all images from active registry?21:42
dmsimardapevec: don't really want to load the production registry more than it already is21:42
apevecpull would load it that much?21:43
apevecbut yeah, you could just run containers build role21:43
dmsimardI planned on forking https://ci.centos.org/view/rdo/view/all/job/rdo-tripleo-containers-build/21:44
dmsimardand making a periodic job and letting it go for a while21:44
dmsimardIt's the job we built containers with on ci.centos21:45
*** dsneddon has quit IRC21:51
apevecsounds good21:52
apevecdmsimard, so yesterday's outage was fat-fingered, lol: https://www.theregister.co.uk/2017/11/06/level3_comcast_outage/21:52
dmsimard:(21:53
dmsimardhappens to the best of us21:53
apevecwe just need more artificial inteligence21:54
*** trown is now known as trown|outtypewww21:54
*** sreichar has quit IRC21:59
*** thrash is now known as thrash|g0ne22:01
*** wolverineav has quit IRC22:07
*** wolverineav has joined #rdo22:07
*** eck` is now known as eck`gone22:13
*** gildub has joined #rdo22:14
dmsimardadarazs|rover, panda|ruck: you folks around ?22:24
panda|ruckdmsimard: more or less22:26
dmsimardpanda|ruck: do you use the ansible kolla build role to push to dockerhub ?22:27
dmsimardto push or to tag on dockerhub22:27
panda|ruckdmsimard: no22:27
dmsimardpanda|ruck: then we should replace https://github.com/rdo-infra/ansible-role-rdo-kolla-build/blob/master/tasks/tag.yml#L23-L24 with an "oc tag" command22:27
dmsimardit means the role won't really be able to push to dockerhub but we don,t care22:28
panda|ruckdmsimard: to make the tagging more atomic ?22:29
dmsimardpanda|ruck: not just atomic, it's more performant. One task instead of two (times 125 images) and it's just an API call instead of tag/push22:30
dmsimardwhen you push a new tag the registry needs to match the image layers sha122:30
*** jmlowe has joined #rdo22:31
panda|ruckdmsimard: yes, it's 5-10 seconds per image22:31
dmsimardI'll send a patch for that22:31
dmsimardI'm currently benchmarking builds/pushes against the new registry22:31
*** lebauce has quit IRC22:32
dmsimardpanda|ruck: btw why do we run toci-test in the container build job ?22:33
dmsimardhttps://github.com/rdo-infra/review.rdoproject.org-config/blob/master/jobs/tripleo-upstream.yml#L273-L27822:33
dmsimardit builds the undercloud image in it as well ?22:34
*** suresh12 has quit IRC22:37
dmsimardJobs will remain queued in Zuul for the next hour in preparation for the maintenance: https://lists.rdoproject.org/pipermail/users/2017-November/000017.html22:37
*** newmember has quit IRC22:42
*** dsneddon has joined #rdo22:43
*** jmelvin has quit IRC22:57
*** suresh12 has joined #rdo22:59
rdogerritDavid Moreau Simard created rdo-infra/ansible-role-rdo-kolla-build master: WIP: Threading  https://review.rdoproject.org/r/1043723:02
*** apevec has quit IRC23:18
*** jmlowe has quit IRC23:18
*** Son_Goku has joined #rdo23:24
*** vnogin has quit IRC23:30
*** vnogin has joined #rdo23:30
*** apevec has joined #rdo23:31
*** apevec has quit IRC23:31
*** apevec has joined #rdo23:31
*** gildub has quit IRC23:33
*** vnogin has quit IRC23:35
*** julim has joined #rdo23:37
*** jmlowe has joined #rdo23:38
*** paragan has quit IRC23:38
*** d0ugal has joined #rdo23:40
*** mvk has joined #rdo23:40
*** rodrigods has quit IRC23:51
*** rodrigods has joined #rdo23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!