Monday, 2017-03-20

*** larstobi has quit IRC00:27
*** larstobi has joined #tripleo00:31
*** limao has joined #tripleo00:33
*** oaken_chris has quit IRC00:50
*** oaken_chris has joined #tripleo00:51
*** oaken_chris has quit IRC00:58
*** dmacpher has quit IRC01:08
*** pmannidi has quit IRC01:44
*** pmannidi has joined #tripleo01:49
*** fragatina has joined #tripleo02:26
*** japestinho has joined #tripleo02:32
*** fzdarsky_ has joined #tripleo02:33
*** fragatina has quit IRC02:33
*** fzdarsky has quit IRC02:37
openstackgerritMerged openstack/tripleo-heat-templates stable/ocata: Make sure PrePuppet runs before any Deployment_Step  https://review.openstack.org/44722702:38
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: WIP, create a README / TOC for oooq logs  https://review.openstack.org/44734702:39
*** gbarros has joined #tripleo02:54
*** gkadam has joined #tripleo03:00
*** gkadam has quit IRC03:00
*** ramishra has joined #tripleo03:22
*** dmacpher has joined #tripleo03:43
*** gbarros has quit IRC04:04
*** udesale has joined #tripleo04:16
*** links has joined #tripleo04:20
*** ratailor has joined #tripleo04:30
*** rwsu has joined #tripleo04:51
*** saneax-_-|AFK is now known as saneax05:07
*** saneax is now known as saneax-_-|AFK05:08
*** Vijayendra has joined #tripleo05:17
*** saibarspeis has joined #tripleo05:19
*** skramaja has joined #tripleo05:19
*** saneax-_-|AFK is now known as saneax05:20
*** nmathew has joined #tripleo05:32
*** saibarspeis has quit IRC05:33
*** saibarspeis has joined #tripleo05:34
*** dsneddon has joined #tripleo05:36
*** pgadiya has joined #tripleo05:59
*** jaosorior has joined #tripleo06:04
*** prateek has joined #tripleo06:05
*** bfournie has joined #tripleo06:07
*** bfournie has quit IRC06:08
*** bfournie has joined #tripleo06:09
*** ealcaniz has joined #tripleo06:14
*** yamahata has joined #tripleo06:29
*** dparkes has joined #tripleo06:37
*** lmiccini has joined #tripleo06:38
*** jaosorior has quit IRC06:53
*** jaosorior has joined #tripleo06:55
*** masco has joined #tripleo06:56
*** karthiks_afk has joined #tripleo06:57
*** dmacpher has quit IRC06:59
*** ealcaniz has quit IRC07:05
*** florianf has joined #tripleo07:10
*** iranzo has joined #tripleo07:12
*** iranzo has joined #tripleo07:12
*** jaosorior has quit IRC07:14
*** jaosorior has joined #tripleo07:15
jaosoriorbandini: hey dude, if you have some time can you check this out https://review.openstack.org/#/c/444891/ ?07:15
*** jaosorior has quit IRC07:16
*** jaosorior has joined #tripleo07:19
*** bogdando has joined #tripleo07:22
*** jprovazn has joined #tripleo07:24
bandinijaosorior: morning! done07:25
jaosoriorbandini: thanks dude :D07:26
bandininp ;)07:26
jaosoriorbandini: by the way, all that stuff was moved to one central place. So it shouldn't be affecting anyone. That was done in a previous commit.07:27
bandiniack07:28
jaosoriorbandini: https://github.com/openstack/puppet-tripleo/commit/f3c157318d7797a42a8ccb57e8d38be288c67fbf07:28
bandinijaosorior: yeah I was kind of wondering if folks used the 'generate_service_certificates' parameter in general07:30
jaosoriorbandini: well... I could re-introduce it. I don't think anybody really used it, but might be a good idea.07:36
jaosoriorbandini: given that one of my tasks for this sprint is document all that shit :P I think it would be really hard for the regular deployer to figure out how to use all that stuff.07:36
chandankumarbandini: hello07:37
jaosoriorbandini: I just didn't think it as being so useful, since one can just explicitly include the certmonger_user profile if they want cert generation07:37
chandankumarbandini: please have a look on this review https://review.openstack.org/#/c/446868/ it does not triggered full mitaka tempest run. it was just a test job07:37
jaosoriorinstead of relying on the boolean07:38
chandankumarbandini: what i am doing wrong here?07:38
bandinijaosorior: yeah no worries, I think we should be moderately safe there07:38
bandinichandankumar: lemme see07:38
bandinichandankumar: so the mitaka jobs fail due to https://bugs.launchpad.net/tripleo/+bug/167342407:42
openstackLaunchpad bug 1673424 in tripleo "Mitaka - CI fails with --env errors" [Critical,In progress] - Assigned to Ben Nemec (bnemec)07:42
bandinilet's see the others07:43
*** karthiks_afk is now known as karthiks07:43
*** cylopez has joined #tripleo07:44
bandinichandankumar: the newton job seem to error out due to ""/home/jenkins/tempest-setup.sh: line 85: /home/jenkins/tempest_git/tools/configure-tempest-directory: No such file or directory"],"07:44
chandankumarbandini: taking a look on newton job07:46
*** dsariel has quit IRC07:47
chandankumarbandini: i got the known cause of newton failure07:48
openstackgerritChandan Kumar proposed openstack-infra/tripleo-ci master: [TEST JOB] test tempest-12.2.0 for Mitaka  https://review.openstack.org/44686807:49
bandinichandankumar: ack. for mitaka we need https://review.openstack.org/#/c/447116/ to land first07:50
chandankumarbandini: what about adding depends on my patch will work?07:50
bandinichandankumar: it should, I believe07:51
chandankumarlet me do that07:51
openstackgerritChandan Kumar proposed openstack-infra/tripleo-ci master: [TEST JOB] test tempest-12.2.0 for Mitaka  https://review.openstack.org/44686807:52
chandankumarbandini: done, thanks!07:52
bandinichandankumar: let's see how it goes ;)07:52
*** tesseract has joined #tripleo07:58
*** saibarspeis has quit IRC07:59
openstackgerritzenghui.shi proposed openstack/tripleo-heat-templates master: Disable core dump for setuid programs  https://review.openstack.org/44245807:59
*** zzzeek has quit IRC08:00
*** zzzeek has joined #tripleo08:00
*** brault has joined #tripleo08:01
*** jlinkes has joined #tripleo08:02
*** shardy has joined #tripleo08:04
*** shardy has quit IRC08:04
*** shardy has joined #tripleo08:07
*** ratailor is now known as ratailor|Lunch08:08
*** dsariel has joined #tripleo08:08
sshnaidm|offchandankumar, hi08:09
*** sshnaidm|off is now known as sshnaidm08:09
sshnaidmchandankumar, how can I add you to review? what's your main mail there?08:10
*** suuuper has joined #tripleo08:13
openstackgerritzenghui.shi proposed openstack/tripleo-heat-templates master: Restrict Access to Kernel Message Buffer  https://review.openstack.org/44740708:14
*** mcornea has joined #tripleo08:14
*** pcaruana has joined #tripleo08:14
*** suuuper has quit IRC08:17
*** ealcaniz has joined #tripleo08:17
*** suuuper has joined #tripleo08:18
*** paramite has joined #tripleo08:22
chandankumarsshnaidm: chkumar@redhat.com08:23
sshnaidmchandankumar, Chandan Kumar <chkumar@redhat.com> does not identify a registered user or group08:25
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart master: Allow custom undercloud initrd/vmlinuz  https://review.openstack.org/44740908:25
chandankumarsshnaidm: my gerrit userid is chkumar24608:27
chandankumarsshnaidm: please pass me the review link, i can add to it.08:27
chandankumarseems to be gerrit problem08:28
*** chem has quit IRC08:28
*** remix_tj has joined #tripleo08:30
bogdandofolks, PTAL this chain https://review.openstack.org/#/c/447409/ . My use case is try oooq on ubuntu host08:32
bogdandoI have yet much options but using my host as virthost =) and it's ubuntu08:32
bogdandobasically that would allow folks on arch linux or gentoo or another scary thing to try oooq as well. Which is kinda... nice08:32
sshnaidmchandankumar, https://review.openstack.org/#/c/446289/08:35
bandinibogdando: morning ;) what does PTAL stand for?08:36
bogdandoplease take a look08:36
bandiniHA! damn that was easy08:36
*** lmiccini has quit IRC08:36
bogdandoI carried that a nice acronym from the k8s world, it seems vety common across google folks08:37
bogdandos/from/out :)08:37
bandinieheh first time I heard it08:37
bogdandoit's really neat when asking for a code review08:38
bogdandoit attracts ppl by "em, what is PTAL???"08:38
bandinilol. awesome evil trick :)08:39
bandinitotally worked for me!08:39
jaosoriorapetrich: I'll check it out08:39
apetrichjaosorior, cheers08:39
*** ealcaniz has quit IRC08:45
*** sshnaidm is now known as sshnaidm|afk08:47
*** dtantsur|afk is now known as dtantsur08:47
*** pmannidi has quit IRC08:49
*** lmiccini has joined #tripleo08:49
*** jpich has joined #tripleo08:52
*** stendulker has joined #tripleo08:56
*** jtomasek|afk is now known as jtomasek08:57
*** athomas has joined #tripleo08:57
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates stable/ocata: Pick dynamically the first node for stack validation  https://review.openstack.org/44741608:57
*** pgadiya has quit IRC08:58
remix_tjshardy: i'd backport also to newton (it's here i found the issue)08:59
*** b00tcat has joined #tripleo09:01
*** openstackgerrit has quit IRC09:02
*** openstackgerrit has joined #tripleo09:03
openstackgerritPeng Liu proposed openstack/puppet-tripleo master: Add l2 gateway Neutron service plugin profile  https://review.openstack.org/44405009:03
*** aufi has joined #tripleo09:04
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates stable/newton: Pick dynamically the first node for stack validation  https://review.openstack.org/44741909:05
shardyremix_tj: ^^ done09:05
remix_tjthank you very much09:05
*** stendulker has quit IRC09:06
*** pgadiya has joined #tripleo09:07
*** ealcaniz has joined #tripleo09:19
*** akrivoka has joined #tripleo09:22
*** stendulker has joined #tripleo09:22
openstackgerritkavitha h r proposed openstack/tripleo-image-elements master: Prepare for using standard python tests  https://review.openstack.org/44581109:26
*** zoli|gone is now known as zoli09:27
*** morazi has quit IRC09:27
*** lucas-afk is now known as lucasagomes09:28
tbarronjprovazn: can we get another +2 and +A from someone for stable/newton backport https://review.openstack.org/#/c/423405 ?09:31
* tbarron is doing his Monday morning bug scrub ...09:31
tbarronjprovazn: oh, and welcome back from PTO :)09:32
jprovazntbarron: hi, hopefully yes09:32
jprovaznthanks :)09:32
openstackgerritJulie Pichon proposed openstack/tripleo-ui master: Remove '\n' linebreak from localised string  https://review.openstack.org/44723709:33
shardydone09:33
*** owalsh_ is now known as owalsh09:33
jprovaznshardy: thanks09:34
jpichjtomasek, florianf, akrivoka: Good morning! I'd appreciate your thoughts and reviews on https://review.openstack.org/#/c/447237/ to move it forward, it's blocking the Ocata translations from being completed at the moment09:35
jtomasekjpich: ack09:36
jpichjtomasek: Thanks! It's also a one liner with no visible difference for the end user :)09:38
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates stable/ocata: Don't try to run os-net-config from yum_update.sh  https://review.openstack.org/44742609:39
rookmorning shardy sorry to keep bothering you, however, the deployment I hvae going is pretty much stuck (09:39
rookhttps://gist.github.com/jtaleric/26f73d30f07f76ad3de99af81bbd55b2)09:40
*** nyechiel has joined #tripleo09:40
rookshardy my only thought to get around this is to remove compute 107 from the mix, and try to scale up yet again.09:40
* rook just looked, that cannot easily be done :/09:41
rookhttps://gist.github.com/jtaleric/e3d1ef49b2c19201aa0580d9af824d7a09:41
shardyrook: Ok, any clues in /var/log/messages as to why it's stuck?09:43
shardyit's trying to run the hosts element, so I'd guess there's an error related to that somewhere?09:43
rookshardy: nothing that means anything to me :09:43
rookhttps://gist.github.com/jtaleric/270ce10ee79ab36c58355288a8a2fbd909:43
openstackgerritPeng Liu proposed openstack/tripleo-heat-templates master: Add l2gw neutron service plugin support  https://review.openstack.org/44742909:43
florianfjpich: Done. I'm all in favor of removing something that does absolutely nothing *except* breaking i18n :-)09:44
shardyrook: is that everything?  It doesn't look like it's got any metadata from heat at all, is the network OK?09:44
*** dbecker has joined #tripleo09:44
shardyrook: and what's the contents of /etc/os-collect-config.conf?09:44
rookshardy yup, network is fine from what Ican tell / me tries the 169 addy09:44
florianfjpich: btw, do we actually want to allow adding newlines in localized strings?09:45
jpichflorianf: Thanks Florian! :)09:45
shardyrook: also try running curl against the metadata_url in the log09:45
rookshardy: i can curl the 169 address09:45
rookok09:45
rookhttps://gist.github.com/jtaleric/6d288310bcb3fa10cf6978de098a9cfa09:45
jpichflorianf: I think <br/> and other HTML codes should work fine in other strings, but there's something else going on with the tooltips? I mean I've used bold tags and stuff in localised strings before and they were picked up09:45
*** derekh has joined #tripleo09:45
shardyrook: ack, so I guess try the 192.168.2.1:8080 one09:45
rook[root@overcloud-openstack-compute-107 heat-admin]# 401 Unauthorized: Temp URL invalid09:46
*** gkadam has joined #tripleo09:46
florianfjpich: maybe the tooltips strip tags09:46
rookshardy: so, we got a 401? huh09:46
shardyrook: Ok, so that is the issue, now to figure out why...09:46
*** salmankhan has joined #tripleo09:47
jpichflorianf: When I tried to add <br/> it was shown as is. I'd need to double-check actually, I'm suddenly uncertain about my <b> statement because I ended up moving it out of the string in the end09:47
florianfjpich: Huh, html inside tooltips are not stripped, but escaped. So we end up with a literal <br>09:47
jpichflorianf: right, "escaped" was the word I was looking for :D09:48
rookshardy: when the metadata goes fubar, i typically run :  heat resource-list -n5 overcloud-openstack | egrep 'OS::Nova::Server|OS::TripleO::Server' | awk '{print $(NF-1) " " $2}' | while read line;do echo ""; echo $line; echo "==="; heat resource-metadata $line | grep metadata_url;done -- to compare the links09:48
shardyrook: haha, yeah I was just about to suggest exactly that09:48
shardyand you can prove the curl works on the undercloud09:48
*** yamahata has quit IRC09:48
florianfjpich: yeah, I just tested it. Same thing in other strings that are not inside of tooltips. So it's a limitation of the i18n mechanism in general.09:51
rookshardy: while that dumps out :), how could this metadata url get messed up? This was a previously deployed machine09:51
shardyrook: not sure - it should be the same as the initial deployment, so you could grep /var/log/messages and prove the URL hasn't changed09:54
jpichflorianf: Hrm... Ok, that's good to know! It should be easy to work around for new lines at least till we figure it out. It's weird though, since localised strings are not user input and are controlled... maybe the goal is to avoid a translated strings accidentally breaking the UI because of a tag mismatch09:55
rookshardy: i get the same message09:57
rookcurling from the UC09:57
florianfjpich: Still not sure we want to allow that. It makes sense to me to include punctuation that is handled differently in different languages (like our colon examples). But is the same true for paragraphs?09:57
shardyrook: to clarify, you're quoting the URL to avoid the & ?09:58
rookyes09:58
shardyhrm09:58
florianfjpich: And if newlines are needed to ameliorate issues of space, the problem is somewhere else anyway...09:58
rookshardy nm. I had two quotes09:58
rookok09:59
florianfjpich: (I don't have an opinion on this yet, just brainstorming...)09:59
rookshardy so, the URL is fine10:00
rookI can curl it on the UC and Node10:00
rookand get a ton of json10:00
*** athomas has quit IRC10:02
jpichflorianf: Likewise, I suspect it won't be an issue till we absolutely need a tag in a string for some reason :) I think if translators did need to make changes to a paragraph, they'd be able to even if it was broken down by line because they're already used to having to do funky stuff because an interface element doesn't quite fit in the target language10:02
shardyrook: Ok, well that sounds good, but it doesn't explain why your logs don't show os-refresh-config run and get stuck on 51-hosts10:02
jpichflorianf: (Thinking about the issues with the nodes count "role cards" that were brought up for Japanese :))10:02
florianfjpich: So let's assume this limitation is intended and we shouldn't try to override it until we know we really need it? :-)10:04
*** limao has quit IRC10:06
*** ealcaniz has quit IRC10:07
openstackgerritJiri Stransky proposed openstack/tripleo-heat-templates master: Keep existing data for containerized ironic-conductor  https://review.openstack.org/44317310:07
jpichflorianf: That'd be my current plan :-) Looking around a bit, it's definitely something react-intl does by design, and the bug tracker has some tips on how to work around needing tags (e.g. https://github.com/yahoo/react-intl/issues/137 )10:08
*** athomas has joined #tripleo10:09
*** sshnaidm|afk is now known as sshnaidm10:09
florianfjpich: Yeah, I agree it's probably by design. I'm just wondering if it's merely for technical reasons (to not break stuff with unclosed tags etc) or specifically with translation in mind (like: translations should not be able to emphasize different things). But since there are workarounds, we don't really need to solve this right now. So +1 on your current plan :-)10:13
d0ugaljtomasek: Could you test this from the GUI's point of view? https://review.openstack.org/#/c/444220/10:14
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Add dracut-regenerate elements  https://review.openstack.org/44704710:14
d0ugaljtomasek: I think it was you that reported the slowness originally10:14
jpichflorianf: I suspect there's a bit of both, but yay :)10:17
*** akrivoka has quit IRC10:17
*** akrivoka has joined #tripleo10:18
*** panda|Zz is now known as panda10:21
*** florianf has quit IRC10:25
*** florianf has joined #tripleo10:27
*** tosky has joined #tripleo10:27
*** pblaho has joined #tripleo10:28
*** zoli is now known as zoli|lunch10:31
chandankumarsshnaidm: https://review.openstack.org/#/c/446289/ gate-tripleo-ci-centos-7-ovb-ha-oooq-nv i am not getting why this job is failing?10:32
bandinihas anyone seen this issue with stable/newton jobs https://bugs.launchpad.net/tripleo/+bug/1674067 recently?10:32
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged]10:32
tbarronshardy: thanks10:32
sshnaidmchandankumar, http://logs.openstack.org/89/446289/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/baf6b53/console.html#_2017-03-16_05_30_07_90296610:34
sshnaidmchandankumar, http://logs.openstack.org/89/446289/1/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha-oooq-nv/baf6b53/logs/oooq/undercloud/home/jenkins/overcloud_validate.log.txt.gz10:35
chandankumarsshnaidm: Stack not found: pingtest_stack how to fix it?10:35
chandankumarsshnaidm: do recheck will work?10:35
pandaadarazs: ^10:36
adarazspanda, chandankumar: I'll look into it in a minute.10:36
chandankumaradarazs: cool!10:36
pandaadarazs: was trying to point you at https://review.openstack.org/#/c/446289/, delorean build problem10:37
adarazspanda: in which job?10:39
pandaadarazs: http://logs.openstack.org/29/447229/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/c1f6ba4/logs/quickstart_install.txt.gz10:39
sshnaidmchandankumar, the problem is in 504 Gateway Time-out: The server didn't respond in time. (HTTP N/A)10:39
pandaso basic multinode10:39
*** ealcaniz has joined #tripleo10:40
openstackgerritSaravanan KR proposed openstack/os-net-config master: Correct the test code to remove 'pass' where it is not needed  https://review.openstack.org/44744110:40
*** jkilpatr has quit IRC10:41
*** ratailor|Lunch has quit IRC10:42
chandankumarbandini: Depends on tag does not work https://review.openstack.org/#/c/446868/10:43
bandinichandankumar: I just +A that dependent patch, so maybe retry once it merges?10:46
chandankumarbandini: yes10:46
openstackgerritSaravanan KR proposed openstack/os-net-config master: Removed vlan port on ovs_user_bridge in dpdk bond samples  https://review.openstack.org/44744610:49
openstackgerritBogdan Dobrelya proposed openstack/tripleo-quickstart master: Allow custom undercloud initrd/vmlinuz  https://review.openstack.org/44740910:49
adarazschandankumar: so yeah, I'd recheck that job, might be just some temporary network hiccup. let me know if it happens again.10:49
*** ckyriakidou has joined #tripleo10:51
*** ealcaniz has quit IRC10:56
*** udesale__ has joined #tripleo10:56
*** udesale has quit IRC10:57
*** ealcaniz has joined #tripleo10:57
*** jkilpatr has joined #tripleo10:58
*** udesale has joined #tripleo10:59
*** udesale__ has quit IRC11:00
*** nmathew has quit IRC11:05
EmilienMhello11:06
*** stendulker has quit IRC11:07
*** thrash|wknd is now known as thrash11:07
adarazsEmilienM: good morning11:07
EmilienMstable/ocata upgrade jobs look very unstable11:10
apetrichd0ugal, about https://review.openstack.org/#/c/444220/7/tripleo_common/actions/base.py so the tox -epep8 locally fails if I move the json to the top. Should I expect a different result from the upstream gate?11:11
d0ugalapetrich: when you put it to the top did you add a space between it and the ones below?11:11
d0ugalapetrich: like in the link11:11
d0ugalapetrich: https://docs.openstack.org/developer/hacking/#import-order-template11:11
d0ugalthe \n is important :)11:11
apetrichd0ugal, I think so. let me restest fast11:11
d0ugalapetrich: if that doesn't then fair enough, leave it as it is - maybe a bug in hacking  we can look at later.11:12
apetrichd0ugal, yeah you are right. I probably missed the \n on my tests. fixing it11:15
d0ugalapetrich: thanks :) I feel bad nitpicking that sorta thing!11:16
apetrichd0ugal, np. I just need to recall how/why the put_object failed with None as the content11:17
d0ugalapetrich: I was wondering why it would ever be "None" :)11:17
*** udesale has quit IRC11:19
*** udesale has joined #tripleo11:19
apetrichd0ugal, just completeness of the API. We let it update cache with any value. Trying to save None is a delete11:19
adarazspanda, sshnaidm: can you guys review https://review.openstack.org/446075 when you have time? it is not as daunting as it looks from the amount of changed files. :)11:20
adarazsand it would be nice to get the timestamps in the logs.11:21
d0ugalapetrich: fair enough, we can keep it - I just wanted to make sure. Feel free to ignore that one.11:21
apetrichd0ugal, I'm commenting that there.11:21
d0ugalapetrich: thanks11:21
*** ealcaniz has quit IRC11:22
*** ealcaniz has joined #tripleo11:23
b00tcatHi - Does the storage network always need a storage_mgmt network?11:24
*** adarazs is now known as adarazs_lunch11:24
jtomasekd0ugal: will do!11:31
d0ugalthanks!11:32
shardyb00tcat: you could probably configure it to only use one, but by default net-iso enables both to isolate e.g user storage traffic from background replication and things11:32
openstackgerritDamien Ciabrini proposed openstack/puppet-tripleo stable/newton: Create /etc/my.cnf.d/tripleo.cnf with proper bind-address  https://review.openstack.org/44745311:34
*** zoli|lunch is now known as zoli11:35
*** gchamoul|afk is now known as gchamoul11:37
openstackgerritGael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Fix ansible-lint.sh to check playbooks  https://review.openstack.org/44652511:41
*** pkovar has joined #tripleo11:44
jpichjtomasek: Thanks for the review on that patch! I'll submit the backport once it merges :)11:45
*** gchamoul is now known as gchamoul|afk11:45
b00tcatshardy: ok thanks! makes sense11:47
*** udesale has quit IRC11:47
EmilienMsomeone looked at CI jobs today? I'm seeing some unstable jobs :/11:48
sshnaidmadarazs_lunch, I think you need to place "2>&1" before {{ timestamper_cmd }} so that stderr will by logged and "timestamped"11:48
*** gchamoul|afk is now known as gchamoul11:50
EmilienMsshnaidm: have you look at why we didn't have promotions?11:50
EmilienMotherwise I'll start looking in  afew11:50
EmilienMI see https://bugs.launchpad.net/tripleo/+bug/167423611:50
openstackLaunchpad bug 1674236 in tripleo "CI: periodics jobs fail with "No valid host" error" [High,Triaged]11:50
sshnaidmEmilienM, yeah, but nothing found yet :(11:52
sshnaidmEmilienM, if you or anybody can look too, would be nice11:53
EmilienMsshnaidm: I'll look shortly11:53
EmilienMbandini: any progress on https://bugs.launchpad.net/tripleo/+bug/1674067 ? Have you poke someone from RDO team maybe?11:53
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged]11:53
*** jayg|g0n3 is now known as jayg11:55
*** ckyriakidou has quit IRC11:58
jtomasekjpich: np!11:58
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Run introspection 20 nodes at a time  https://review.openstack.org/43956812:00
*** snecklifter has joined #tripleo12:02
bandiniEmilienM: no, not yet. it's been one of these mornings. will pick it up again this afteroon12:02
*** pgadiya has quit IRC12:06
*** abishop has joined #tripleo12:06
EmilienMbandini: no prob, I'll also look in a few, let's sync when we start12:08
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406712:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423612:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged]12:10
openstackLaunchpad bug 1674236 in tripleo "CI: periodics jobs fail with "No valid host" error" [Critical,Triaged]12:10
*** dprince has joined #tripleo12:10
*** rhallisey has joined #tripleo12:11
*** masco has quit IRC12:15
rookbandini rasca if pcs reports " Resource management is DISABLED " how do I reverse that?12:16
*** pradk has joined #tripleo12:18
*** bfournie has quit IRC12:21
bandinirook: pcs property set maintenance-mode=false12:26
jtomasekflorianf: could you please let this one through https://review.openstack.org/#/c/445525/6 I had to rebase it12:26
rookbandini++12:27
rookthanks12:27
*** snecklifter has quit IRC12:27
bandininp ;)12:27
dtantsurhey folks! do we have a place to put upgrade actions for undercloud? we will have to do something on Ocata->Pike or Pike->Queens upgrade to ironic nodes and flavors12:30
openstackgerritArx Cruz proposed openstack/tripleo-quickstart-extras master: Send mail tool  https://review.openstack.org/42334012:30
openstackgerritArx Cruz proposed openstack/tripleo-quickstart-extras master: Send mail tool  https://review.openstack.org/42334012:31
openstackgerritOliver Walsh proposed openstack/tripleo-heat-templates master: WIP: set libvirtd bind address  https://review.openstack.org/44746912:32
*** fultonj has joined #tripleo12:32
florianfjtomasek: done12:32
jtomasekflorianf: thanks12:32
shardydtantsur: Hey, it's mostly done via puppet, and there's a recently added undercloud_upgrade flag which can enable specific changes on upgrade vs update12:33
shardyhttps://github.com/openstack/instack-undercloud/commit/ad47a3c40fbf13427457e4b0b9fb7a684de6377a12:33
shardydtantsur: ^^ example12:33
*** remixtj has joined #tripleo12:33
*** remix_tj has quit IRC12:33
dtantsurshardy, thanks! for context: nova/ironic change how we schedule on bare metal nodes. we'll have resource classes, which is essentially what we call profiles.12:34
dtantsurwe'll stop scheduling based on memory/disk/etc in Queens. we have to migrate to this resource classes.12:34
dtantsurwe're working on a plan in the meantime12:34
EmilienMsshnaidm: I'm seeing a lot of unstability on oooq jobs, specially on stable/ocata... i'm looking at https://review.openstack.org/#/c/444338/12:35
shardydtantsur: Ok, good to know, thanks for the heads-up12:35
dtantsurshardy, np. I guess I cannot change objects like flavors or nodes from puppet, can I?12:35
shardydtantsur: FWIW I think only scheduling based on tags is a good thing12:35
dtantsuryep, very good for tripleo12:36
dtantsurprofile matching without capabilities hacks12:36
EmilienMsshnaidm: but it's not directly related to oooq this time: http://logs.openstack.org/38/444338/4/gate/gate-tripleo-ci-centos-7-scenario002-multinode-oooq/5d09552/logs/undercloud/home/jenkins/undercloud_install.log.txt.gz#_2017-03-20_09_53_15_10112:36
d0ugalthrash: thanks :)12:36
thrashd0ugal: sure... Don't think the ones that failed are your fault.12:36
*** jcoufal has joined #tripleo12:37
d0ugalthrash: agreed, the last one passed12:37
shardydtantsur: I think we could do it in puppet, but currently flavors are managed via python here:12:37
shardyhttps://github.com/openstack/instack-undercloud/blob/master/instack_undercloud/undercloud.py#L124912:37
dtantsurshardy, this is undercloud. now, for overcloud, should I update these small ansible bits in THT?12:37
shardydtantsur: yeah for overcloud you'll modify either puppet-ironic to do the right thing on upgrade, or add something to upgrade_tasks if puppet cant or won't handle it12:38
*** remixtj has quit IRC12:38
*** remix_tj has joined #tripleo12:38
dtantsurshardy, well, the same problem with puppet: it will have to do something unusual about nodes and flavors12:38
* dtantsur brb, thanks for links12:39
shardydtantsur: ack - well maybe upgrade_tasks would be best, but if you look at the stuff we did for nova and cells v2, there were changes in puppet and tht12:39
shardyso I guess it depends on the exact nature of the change12:39
dtantsuryeah12:39
jrollit's all API-based changes, shouldn't be any config changes12:40
shardydtantsur: for the overcloud you probably can't manage overcloud flavors via upgrade_tasks, it'll probably need to be done via puppet I think12:40
jrollso "do it in python" makes the most sense to me, with my limited knowledge12:40
shardydtantsur: the upgrade_tasks run on each node, so don't really have API access to the overcloud services, only e.g the db-sync commands and other admin interfaces12:41
*** snecklifter has joined #tripleo12:41
shardyjroll: hey, yeah if it's only overcloud flavors then probably some special-case code in that undercloud.py will be best12:41
jrollshardy: idk tripleo well, but it's likely under and overcloud12:42
jrollanything nova+ironic12:42
*** dsariel has quit IRC12:43
shardyjroll: ack, OK then I guess we'll need to update undercloud.py for the undercloud flavors, and the puppet configuration for the overcloud (or document/release-note what admin steps are required for their manually created flavors)12:43
jkilpatrmandre, hey I'm looking at tripleo-ansible-inventory trying to add support for listing bare metal hosts on the undercloud, this should be the same as poking nova for a list of instances. But I'm not sure where these configs.list or configs.host values get populated. Pointer please?12:43
jrollshardy: nod, makes sense12:44
*** shardy is now known as shardy_lunch12:44
*** rlandy has joined #tripleo12:45
*** janki has joined #tripleo12:45
*** ealcaniz has quit IRC12:45
openstackgerritJiri Tomasek proposed openstack/tripleo-specs master: Deployment plan management changes  https://review.openstack.org/43891812:46
*** adarazs_lunch is now known as adarazs12:47
Ngshardy_lunch: thanks for your comment on #442497 - am I understanding correctly that gerrit isn't merging it because the ocata branch of the Depends-On change, hasn't merged?12:49
*** vpickard has quit IRC12:50
openstackgerritThomas Herve proposed openstack/tripleo-heat-templates master: Use httpd in Zaqar docker service  https://review.openstack.org/44747612:50
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Add dracut-regenerate elements  https://review.openstack.org/44704712:52
numansbeagles, Hi, when ever you get some time can you please add this to your review queue - https://review.openstack.org/#/c/433025/, https://review.openstack.org/#/c/372274/ - these are related to OVN.12:53
*** ealcaniz has joined #tripleo12:53
*** bfournie has joined #tripleo12:53
*** trown|outtypewww is now known as trown12:53
dtantsurshardy_lunch, tbh, just documenting it might be the best way12:53
beaglesnumans: ack . can you make sure I'm added as a reviewer to them so I don't lose track?12:53
numansbeagles, sure12:54
* dtantsur writes an draft blueprint12:54
adarazssshnaidm: thanks for the review, nice catch with the stderr redirect.12:54
beaglesnumans: thx12:54
dtantsurjroll, what happens in Queens if we don't fill in a resource class for both nodes and flavors? will a random node be picked?12:55
jrolldtantsur: I think we'll ignore nodes without resource classes in queens ("we" being the resource tracker)12:56
jrolldtantsur: that's just a proposal right now, but I suspect nova people will agree12:56
dtantsuroh, this is bad.. but if a flavor has no resource class, will it match any node?12:57
jrolldtantsur: I think so, including kvm hosts12:58
mandrejkilpatr: these values are set when you pass the '--list' or '--host' options to the CLI13:00
*** remixtj has joined #tripleo13:01
*** remix_tj has quit IRC13:01
*** salmankhan has quit IRC13:01
*** salmankhan has joined #tripleo13:02
*** liverpooler has joined #tripleo13:02
dtantsurjroll, ok. so I have to set a dummy resource class even for nodes that don't have our "profile"13:03
dtantsurbut skip it for our generic flavor13:03
* dtantsur is mostly thinking aloud13:03
jkilpatrmandre, ah so it does work on undercloud instaces I just needed to pass --list. What's the default flow supposed to do?13:04
*** jmelvin has joined #tripleo13:04
jrolldtantsur: the empty flavor thing is interesting, I'm not sure what we'll do with this13:04
openstackgerritRyan Brady proposed openstack/tripleo-common master: Create or update deployment plan from git repository  https://review.openstack.org/41728013:05
dtantsurjroll, it's very important for us here to understand13:05
dtantsurjroll, currently the default mode for us is "just pick whatever node"13:05
*** ealcaniz has quit IRC13:05
*** ansmith has joined #tripleo13:06
*** noslzzp has quit IRC13:06
*** noslzzp has joined #tripleo13:07
mandrejkilpatr: yep, that's what ansible expects, see http://docs.ansible.com/ansible/dev_guide/developing_inventory.html13:08
trozetEmilienM: this patch never kicked off gate jobs https://review.openstack.org/#/c/435081/ Is it because there was a comment when workflow was set?13:08
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart-extras master: Collect the last 4 hours from the journal in text form  https://review.openstack.org/44709913:08
mandrejkilpatr: usually you won't run the script yourself, but only point ansible to it13:09
*** remixtj has quit IRC13:09
*** remix_tj has joined #tripleo13:09
mandrejkilpatr: you needed the expand the script?13:10
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406713:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423613:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged]13:10
openstackLaunchpad bug 1674236 in tripleo "CI: periodics jobs fail with "No valid host" error" [Critical,Triaged]13:10
*** vpickard_afk has joined #tripleo13:10
jkilpatrmandre, hmm clearly I need to play around with this more. I might, frankly I expected to have to which is why I went straight to debugging/hacking13:10
*** vpickard_afk is now known as vpickard13:10
EmilienMtrozet: remove the depends-on13:11
jtomasekdtantsur, jroll: I'd love to see the spec for these changes once it is ready13:11
*** limao has joined #tripleo13:11
dtantsurjroll, jtomasek, shardy_lunch, makes sense so far? https://blueprints.launchpad.net/tripleo/+spec/ironic-resource-classes13:11
trozetEmilienM: but the dependency merged?13:11
*** jprovazn has quit IRC13:11
*** links has quit IRC13:11
mandrejkilpatr: it would probably help if we print a message when running the script without option, or make '--list' the default action13:11
mandrejkilpatr: I'll gladly accept patches :)13:12
jkilpatrmandre, that's what I was thinking because I thought it was broken, I'll have a patch up after playing around with it some more.13:12
*** snecklifter has quit IRC13:13
* dtantsur brb finally for real13:13
*** dtantsur is now known as dtantsur|brb13:13
EmilienMtrozet: not in stable/newton13:13
trozetEmilienM: the dependency covers across branches?13:13
jtomasekdtantsur|brb: looks pretty clear to me except for flavors, so flavor won't have resoucre_class? it will have 'resources'?13:14
jkilpatrmandre, also just -i tripleo-ansible-inventory doesn't work /usr/bin/tripleo-ansible-inventory runs it then errors out is there a minimum ansilbe version I should be aware of?13:14
EmilienMtrozet: yes13:15
EmilienMtrozet: that's why we don't backport before master patches are landed...13:15
*** snecklifter has joined #tripleo13:16
trozetEmilienM: oh i didnt realize that13:16
trozetEmilienM: well then, let's get all the puppet-tripleo side finished first ona ll branches13:16
jkilpatrmandre, never mind, deugging statement made it unhappy, still need to point it exactly at the python file though13:17
mandrejkilpatr: right you need the full path to the file13:18
trozetEmilienM: do you mind reviewing this one https://review.openstack.org/#/c/425934/ ?  It is missing release notes...13:18
*** remix_tj has quit IRC13:19
EmilienMtrozet: so do you plan to add it or not?13:19
trozetEmilienM: I'll add it now13:19
jrolldtantsur|brb: I agree it's important, but it's a WIP so I don't have all the info yet :)13:19
jrolljtomasek: are you looking for the nova or tripleo spec, or both?13:20
*** remix_tj has joined #tripleo13:20
jtomasekjroll: I am trying to understand what is changing to be able to identify impact on GUI13:20
jrolljtomasek: dtantsur|brb: this is the nova spec for the flavor changes: https://review.openstack.org/#/c/446570/ . and the upgrade path etc that will turn into a spec eventually: https://etherpad.openstack.org/p/nova-ironic-flavor-transition-plan13:21
*** ckyriakidou has joined #tripleo13:21
jtomasekjroll: thanks, I'll go through that13:21
*** Goneri has joined #tripleo13:21
*** dsavineau has quit IRC13:21
*** dsavineau has joined #tripleo13:22
jrolljtomasek: and to answer your question, it's a generic mechanism, so yes, it is approximately resources:$CUSTOM_RESOURCE_CLASS=113:22
jtomasekjroll, dtantsur|brb: I see, so to tie a node to a Role, node has resource_class: 'compute' and to tie a flavor to Role, flavor has resources: CUSTOM_compute=113:23
*** gbarros has joined #tripleo13:24
*** dsariel has joined #tripleo13:24
*** remixtj has joined #tripleo13:24
jrolljtomasek: sounds right, yes. we normalize the resource class so it would be "CUSTOM_COMPUTE", but that's just a nit :)13:24
*** remix_tj has quit IRC13:24
jrolldisclaimer: I know almost nothing about tripleo, so idk what a Role is :)13:25
EmilienMjroll: a role is a group of services we want to deploy on a node (physical or virtual)13:27
jrollEmilienM: yeah, I figured it was something like that13:28
EmilienMjroll: e.g. Controller can be called a role, where you have some services that control OpenStack APIs13:28
* jroll nods13:28
*** shardy_lunch is now known as shardy13:28
*** florianf has quit IRC13:29
cschwedeHi! Can someone please have a look at this patch: https://review.openstack.org/#/c/447480/13:32
*** remixtj is now known as remix_tj13:32
cschwedeIt fixes an automation issue when running Tempest verification tests. Also needs a backport to stable/ocata once merged13:33
cschwedeI see some failures in the centos scenario tests, not sure where they come from13:33
openstackgerritMerged openstack/tripleo-common master: Remove machine-id from image  https://review.openstack.org/44517413:34
openstackgerritMerged openstack/tripleo-heat-templates master: Fixes multiple issues with retry function in rhel-registration.  https://review.openstack.org/44504513:34
openstackgerritAdriano Petrich proposed openstack/tripleo-common master: add caching the GetParametersAction  https://review.openstack.org/44422013:34
EmilienMderekh: looking at https://bugs.launchpad.net/tripleo/+bug/1674236 - it might be the nova cells discover hosts again?13:35
openstackLaunchpad bug 1674236 in tripleo "CI: periodics jobs fail with "No valid host" error" [Critical,Triaged]13:35
EmilienMderekh: though we made it working for ocata, I would be surprised to see it broken now13:35
jaosoriorcschwede: you might wanna ask in #puppet-openstack since that's a commit for puppet-glance13:36
*** florianf has joined #tripleo13:36
*** gkadam has quit IRC13:36
*** snecklifter has quit IRC13:37
cschwedejaosorior: ah yes, makes sense - thx!13:37
derekhEmilienM: sshnaidm how long has the periodic job been failing?13:37
*** dprince has quit IRC13:37
EmilienMderekh: that's what I'm looking for now13:37
EmilienMderekh: it broke after March 16th13:38
*** dprince has joined #tripleo13:39
derekhok13:39
EmilienMI'm looking in nova commits13:40
EmilienMderekh: your conclusion in the bug report is pretty good, http://logs.openstack.org/periodic/periodic-tripleo-ci-centos-7-ovb-nonha/12d034e/logs/undercloud/var/log/nova/nova-conductor.txt.gz#_2017-03-19_07_21_12_70913:42
EmilienMowalsh: ^ if you have time to help in debug13:42
EmilienMowalsh: it sounds like nova isn't aware about nodes registered by ironic, since March 16th13:42
*** chem has joined #tripleo13:42
openstackgerritThomas Herve proposed openstack/tripleo-heat-templates master: Use httpd in Heat docker services  https://review.openstack.org/44750413:42
derekhEmilienM: owalsh /me wonders if this is relevant as it tinkers around with the ironic inventory https://review.openstack.org/#/c/441544/1113:43
*** morazi has joined #tripleo13:43
jkilpatrmandre, so if the script needs a full path why does all the documentation just use -i tripleo-ansible-inventory ?13:44
*** lblanchard has joined #tripleo13:44
openstackgerritThomas Herve proposed openstack/tripleo-heat-templates master: Use httpd in Zaqar docker service  https://review.openstack.org/44747613:44
*** rbrady-afk is now known as rbrady13:44
EmilienMderekh: nice catch13:45
openstackgerritTim Rozet proposed openstack/tripleo-heat-templates master: Enables OpenDaylight clustering in HA deployments  https://review.openstack.org/42593413:45
derekhEmilienM: its just a hunch, but worth looking at13:46
EmilienMderekh: I also thought about https://review.openstack.org/#/c/429274/ but really not sure13:46
EmilienMderekh: I pinged jaypipes13:48
derekhEmilienM: ack, afraid I'm not setup at the moment to test them but can later if needed13:48
*** jprovazn has joined #tripleo13:49
EmilienMderekh: you already helped, I'll continue from here. Thanks again13:49
derekhEmilienM: no prob13:49
*** rhallisey has quit IRC13:50
*** snecklifter has joined #tripleo13:52
sshnaidmderekh, EmilienM I have a environment with failed periodic job if you need to look at it live13:52
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: Add timestamps to quickstart logs  https://review.openstack.org/44607513:53
owalshEmilienM, derekh: looking...13:53
*** lucasagomes is now known as lucas-hungry13:53
derekhowalsh: cool, that env sshnaidm has might help13:54
EmilienMowalsh, derekh: on #openstack-nova, jay is also looking13:54
weshay_sicktrown, https://review.openstack.org/#/c/445213/13:54
*** rhallisey has joined #tripleo13:57
adarazssshnaidm: I think I fixed all the 2>&1 stuff in https://review.openstack.org/446075 :)13:57
*** [1]cdearborn has joined #tripleo13:57
sshnaidmadarazs, thanks, will re-review..13:57
*** japestinho has quit IRC14:00
*** panda is now known as panda|off14:01
*** dprince has quit IRC14:01
*** tvignaud has joined #tripleo14:03
*** dprince has joined #tripleo14:03
*** tbonds has joined #tripleo14:03
*** ealcaniz has joined #tripleo14:04
*** gbarros has quit IRC14:08
mandrejkilpatr: I think we wrote the doc at a time where ansible was still happy when passing a relative path to the inventory script14:08
mandrejkilpatr: ansible 2 was a complete rewrite14:08
EmilienMsshnaidm: if I submit a OOO-ci patch with Depends-On https://review.openstack.org/#/c/447512/ - it will build Nova from master, right?14:08
sshnaidmEmilienM, yes14:09
EmilienMsshnaidm: but other packages without promotion :-/14:09
EmilienMbut ok, let's try anyway14:09
EmilienMsshnaidm: do you have a safe way to test promotion job with Depends-On but without the risk of promoting if it pass?14:09
sshnaidmEmilienM, yeah, you can hack it - add "periodic-" to toci_job_type in tripleo-ci patch14:09
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406714:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged]14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423614:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]14:10
EmilienMsshnaidm: do you have a patch I can re-use?14:10
sshnaidmEmilienM, will create now14:10
EmilienMsshnaidm: thanks14:10
*** suuuper has quit IRC14:11
*** suuuper has joined #tripleo14:11
*** dprince has quit IRC14:12
*** udesale has joined #tripleo14:12
jkilpatrmandre, this is maintained right? Because I would like to use it even if I have to pick up some matience myself.14:13
jkilpatrjust not all of it.14:13
*** skramaja has quit IRC14:14
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test consistent repo  https://review.openstack.org/44751414:14
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test consistent repo  https://review.openstack.org/44751414:15
sshnaidmEmilienM, ^^14:15
EmilienMsshnaidm: Depends-On is missing, right?14:15
mandrejkilpatr: it's maintained14:15
EmilienMsshnaidm: I'll add it, thanks a lot14:15
sshnaidmEmilienM, added14:15
adarazsEmilienM: I took https://bugs.launchpad.net/tripleo/+bug/1674067 and will try to fix it up now.14:15
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)14:15
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: DONT REVIEW: test consistent repo  https://review.openstack.org/44751414:15
EmilienMsshnaidm: thanks :D14:16
EmilienMadarazs: thx.14:16
*** dmacpher has joined #tripleo14:16
mandrejkilpatr: just no one noticed the wrong path in the doc because I assume people will either have the UI run the validations for them or use the 'run-validations' wrapper14:16
openstackgerritTim Rozet proposed openstack/puppet-tripleo stable/ocata: Fixes issues with raising mysql file limit  https://review.openstack.org/44751514:16
*** dprince has joined #tripleo14:17
jkilpatrmandre, good, I'll just submit a patch to fix that real quick.14:17
sshnaidmEmilienM, I need to integrate into my status page searching for possible breaking commits..14:17
*** salmankhan has quit IRC14:17
mandrejkilpatr: awesome, thanks!14:18
sshnaidmwill try it after transition14:18
EmilienMsshnaidm: nice :D14:18
EmilienMsshnaidm: can you also add something in the TODO list for logging tasks: add log level filters support like devstack jobs14:18
EmilienMpeople always ask for it14:18
sshnaidmEmilienM, hmm.. in our bash scripts?14:20
sshnaidmEmilienM, it's done in /var/log usually where logging levels are supported (if they are)14:20
sshnaidmEmilienM, in bash scripts we just "echo" usually14:21
openstackgerritJustin Kilpatrick proposed openstack/tripleo-validations master: Fix docs to reflect Ansible 2.0 requirements  https://review.openstack.org/44751914:23
openstackgerritEmilien Macchi proposed openstack-infra/tripleo-ci master: DONT REVIEW: test consistent repo  https://review.openstack.org/44752214:27
*** dsariel has quit IRC14:29
*** artom has quit IRC14:29
EmilienMsshnaidm: ok14:29
thrashtrown: let me know if you have questions about https://bugs.launchpad.net/tripleo-quickstart/+bug/167435714:29
openstackLaunchpad bug 1674357 in tripleo-quickstart "devmode.sh not working because of missing script" [Undecided,New]14:29
*** artom has joined #tripleo14:29
trownthanks for the devmode bug thrash... looks like https://review.openstack.org/#/c/431764/ did not get the full CI run on it, and that is where the devmode job is14:29
thrashtrown: I can see why it would pass otherwise... $WORKSPACE is going to already be set up for most jobs.14:30
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add option to build delorean changes and include the repo  https://review.openstack.org/43400414:31
trownthrash: ya I think we just have to revert https://review.openstack.org/#/c/431764/ unfortunately, and make sure to run full CI on it when reproposing < rlandy14:32
openstackgerritPeng Liu proposed openstack/tripleo-heat-templates master: Add l2gw neutron service plugin support  https://review.openstack.org/44742914:34
rlandytrown: ok - we can revert  that change  - let me know if you are putting in the revert or I should14:35
trownrlandy: mind doing it? so I can +214:35
trownrlandy: should be able to just click revert button in gerrit14:35
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Revert "Update ci-script to enable TripleO gate dependencies"  https://review.openstack.org/44752614:36
rlandytrown: revert button clicked: https://review.openstack.org/#/c/447526/14:36
*** salmankhan has joined #tripleo14:36
rlandyhmmm ... I had this setup differently before - will have to redo it14:37
openstackgerritJohn Trowbridge proposed openstack/tripleo-quickstart master: Revert "Update ci-script to enable TripleO gate dependencies"  https://review.openstack.org/44752614:37
trownrlandy: just put the bug in the commit message of the revert ^14:38
trownrlandy: as in I just did that :P14:38
*** udesale has quit IRC14:39
*** isq has quit IRC14:40
openstackgerritPeng Liu proposed openstack/tripleo-heat-templates master: Add l2gw neutron service plugin support  https://review.openstack.org/44742914:40
rlandytrown: thanks14:41
openstackgerritAttila Darazs proposed openstack/tripleo-quickstart-extras master: build-test-packages: fix distgit branch naming  https://review.openstack.org/44753014:41
trownadarazs: sshnaidm could either of you look at https://review.openstack.org/447526 reverts a patch that broke devmode14:42
trownadarazs: sshnaidm we probably need to make devmode gate always trigger if full-deploy.sh is modified14:42
adarazstrown: sure, looking.14:43
*** dtantsur|brb is now known as dtantsur14:44
adarazsbtw please take a look at https://review.openstack.org/447530 that I just proposed, we need this to fix stable gates for newton & upwards.14:44
*** Goneri has quit IRC14:44
*** saneax is now known as saneax-_-|AFK14:46
trownadarazs: is that true? I think $release-rdo is for actual distgit ie spec files have changelog and what not, whereas rpm-$release is for DLRN14:46
adarazstrown: check the bug. according to Alan that's how it is from newton forward.14:46
adarazstrown: https://review.openstack.org/#/c/399578/1/scripts/tripleo.sh14:46
*** florianf has quit IRC14:47
*** yamahata has joined #tripleo14:47
trownadarazs: interesting... somehow totally missed that. thanks for fixing it14:47
*** florianf has joined #tripleo14:48
*** limao has quit IRC14:48
sshnaidmadarazs, does "source=" change too?14:50
adarazssshnaidm: that seems to be the same, according to the changes cited by apevec. either master or stable/something14:51
adarazsi.e. the source branch, which is the same as the gerrit branch.14:52
sshnaidmok14:52
openstackgerritEmilien Macchi proposed openstack/instack-undercloud master: Remove compute_manager, deprecated in Nova  https://review.openstack.org/44753114:52
*** limao has joined #tripleo14:55
*** mah has joined #tripleo14:55
*** lucas-hungry is now known as lucasagomes14:56
mahHi All, I have an issue related to the metadata when I restart the cluster nodes: the instances wait long time about 120 secs and didnot receive something related to metadata. knowing that it is working fine before the Restart. so how can I adjust this problem14:57
*** dtantsur is now known as dtantsur|mtg14:57
openstackgerritDougal Matthews proposed openstack/tripleo-common master: Run introspection 20 nodes at a time  https://review.openstack.org/43956814:58
openstackgerritCorey O'Brien proposed openstack/diskimage-builder master: Don't unmount missing paths  https://review.openstack.org/44662114:58
*** janki has quit IRC14:58
*** masco has joined #tripleo15:01
openstackgerritMatt Young proposed openstack/tripleo-quickstart master: Remove existing dstat log files  https://review.openstack.org/43714815:02
*** dsariel has joined #tripleo15:02
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406715:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423615:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]15:10
EmilienMFYI all: infra is experiencing mirror issue, that cause a bunch of tripleo jobs to fail (and not only tripleo)15:11
openstackgerritMartin André proposed openstack/tripleo-docs master: Fixups for containers documentation  https://review.openstack.org/44754115:11
*** anton has joined #tripleo15:11
EmilienMyou can do 'recheck' if needed - infra is working on it now (see #openstack-infra)15:11
mandreflaper87: ^^15:11
EmilienMit affects all branches (newton, ocata, master)15:12
bandiniEmilienM: ah thanks for the heads up!15:15
*** limao has quit IRC15:15
*** saneax-_-|AFK has quit IRC15:16
jpichI'm seeing some gate jobs in the queue for nearly 6h, is that common? What's the usual infra timeout?15:20
* flaper87 hugs mandre15:24
EmilienMjpich: can you give me an example? I'll check, thank15:24
EmilienMthanks*15:24
*** ramishra has quit IRC15:25
jpichEmilienM: I'm looking at http://status.openstack.org/zuul/ and https://review.openstack.org/#/c/442482/ on top of the queue for 5h50 :)15:25
*** gbarros has joined #tripleo15:26
jpichEmilienM: Sadly my network can't handle IPv6 so I can't take a peek at the job that seems stuck15:26
*** jkilpatr has quit IRC15:27
* EmilienM clicks15:27
EmilienMmhh, yeah looks like transient issue... Might be great to see at logs once it fails15:27
*** dsariel_ has joined #tripleo15:28
*** jkilpatr has joined #tripleo15:28
*** prateek has quit IRC15:28
jpichAlright, so nothing to do beyond waiting15:28
*** beekneemech is now known as bnemec15:32
*** chlong has joined #tripleo15:33
*** aufi has quit IRC15:34
jaosoriorare multinode jobs broken?15:35
d0ugaljtomasek, apetrich: reading the most recent comment on https://review.openstack.org/#/c/444220/ - maybe we need a dedicated "cache" container? then include the plan name in the key.15:36
jtomasekd0ugal: that could work, yes15:37
d0ugaljtomasek, apetrich: also, maybe we shouldn't invalidate the cache when the files in swift change but instead we should populate the cache then15:37
adarazsjaosorior: which ones? oooq ones have problem with newton and ocata branches for now. but otherwise they should work. apart from infra in general having problems.15:38
d0ugaljtomasek, apetrich: so it is always ready and waiting, at the moment it will be slow once every time it has been invalidated.15:38
jtomasekd0ugal: good point, it would be nice to figure out how to invalidate the cache in more automatic manner, then doing it explicitly in actions15:38
jaosorioradarazs: master seems to be failing a lot today15:38
openstackgerritDan Radez proposed openstack/tripleo-heat-templates master: Setting keystone region for congress  https://review.openstack.org/44755615:39
adarazsjaosorior: (04:11:33 PM) EmilienM: FYI all: infra is experiencing mirror issue, that cause a bunch of tripleo jobs to fail (and not only tripleo) -- that's all I know. :)15:39
apetrichjtomasek, d0ugal aye. I thought about a ttl for it but I didn't know how could we implement it easily in swift.15:39
jaosoriorah, right15:40
jaosorioradarazs: alright15:40
openstackgerritDan Radez proposed openstack/tripleo-heat-templates master: Setting keystone region for tacker  https://review.openstack.org/44755815:41
apetrichjtomasek, d0ugal the cache container and the plan in the key are great ideas I will change that, and add the extra places that invalidation is missing.But I have no idea on how to do an automatic invalidation15:42
jpichEmilienM: Ah, looks like it's finally off the queue indeed, I should have been more patient15:44
EmilienMjpich: passing?15:44
d0ugalapetrich: swift has a ttl thing - let me find an example...15:45
*** jaosorior has quit IRC15:45
apetrichd0ugal, oh cool15:45
jpichEmilienM: I think it failed, the next few patches have started running the jobs from scratch again (I'm following a patch lower down that change queue)15:46
d0ugalapetrich: Line 321 and 332/333 https://review.openstack.org/#/c/422789/11/tripleo_common/actions/plan.py15:46
*** adarazs is now known as adarazs_afk15:46
d0ugalapetrich: actual docs: https://docs.openstack.org/developer/swift/overview_expiring_objects.html15:47
apetrichd0ugal, cheers!15:47
*** Goneri has joined #tripleo15:48
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Add dracut-regenerate elements  https://review.openstack.org/44704715:50
*** links has joined #tripleo15:50
*** jprovazn has quit IRC15:53
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Always execute autorelabel  https://review.openstack.org/44707615:53
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: Modify ci-scripts to use full path to other scripts  https://review.openstack.org/44756315:54
*** links has quit IRC15:55
*** links has joined #tripleo15:55
*** jaosorior has joined #tripleo15:59
*** trown is now known as trown|outtypewww16:00
*** jlinkes has quit IRC16:01
*** rlandy is now known as rlandy|brb16:01
*** leanderthal has joined #tripleo16:03
*** rbrady is now known as rbrady-afk16:03
*** tbonds has quit IRC16:05
*** links has quit IRC16:05
*** paramite has quit IRC16:08
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406716:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423616:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)16:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]16:10
*** mburned is now known as mburned_out16:10
*** dtantsur|mtg is now known as dtantsur16:11
*** dmacpher is now known as dmacpher-afk16:14
*** ealcaniz has quit IRC16:14
openstackgerritGonéri Le Bouder proposed openstack/tripleo-quickstart master: doc: use curl instead of wget  https://review.openstack.org/44757616:15
*** tesseract has quit IRC16:15
*** jkilpatr has quit IRC16:16
*** saneax-_-|AFK has joined #tripleo16:17
*** arxcruz has quit IRC16:19
*** pcaruana has quit IRC16:27
*** jkilpatr has joined #tripleo16:30
*** yamahata has quit IRC16:31
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: WIP, create a README / TOC for oooq logs  https://review.openstack.org/44734716:33
openstackgerritTim Rozet proposed openstack/tripleo-puppet-elements stable/newton: Adds puppet-systemd  https://review.openstack.org/44326116:34
*** adarazs_afk is now known as adarazs16:44
*** mburned_out is now known as mburned16:45
*** jaosorior has quit IRC16:50
openstackgerritBen Nemec proposed openstack/instack master: Allow instack to work with pre-v2 versions of dib  https://review.openstack.org/44711616:50
openstackgerritBen Nemec proposed openstack/instack-undercloud stable/mitaka: Nothing to see here  https://review.openstack.org/44711716:50
*** leanderthal has quit IRC16:51
*** jaosorior has joined #tripleo16:52
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo master: Re-run ceilometer upgrade in step 5  https://review.openstack.org/44759916:56
chemwhois zaneb16:58
chemzaneb: just making sure it was you :)16:59
zaneblol16:59
*** zoli is now known as zoli|gone16:59
*** mburned is now known as mburned_out17:02
*** rlandy|brb is now known as rlandy17:02
openstackgerritPradeep Kilambi proposed openstack/puppet-tripleo master: Re-run ceilometer upgrade in step 5  https://review.openstack.org/44759917:03
*** fragatina has joined #tripleo17:05
*** mburned_out is now known as mburned17:06
*** fragatin_ has joined #tripleo17:06
*** aufi has joined #tripleo17:08
*** cylopez has quit IRC17:09
*** tbonds has joined #tripleo17:09
*** ooolpbot has joined #tripleo17:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION17:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406717:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423617:10
*** ooolpbot has quit IRC17:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)17:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]17:10
*** fragatina has quit IRC17:10
openstackgerritThomas Herve proposed openstack/tripleo-heat-templates master: Remove yaql call when building logging_groups  https://review.openstack.org/44760517:11
bogdandoHas the docker_config's environment an option to pass env-file for docker? I need an env file for etcd17:12
bogdandoflaper87, mandre: ^ ^17:13
openstackgerritHonza Pokorny proposed openstack/tripleo-specs master: Add network configuration spec  https://review.openstack.org/39638317:13
bogdandois that the proper file to check https://github.com/openstack/heat-agents/blob/8b7af869080dc6febfb479a7168660a1b0e6968e/heat-config-docker-cmd/install.d/hook-docker-cmd.py ?17:15
bogdandoshardy: ^ ^17:15
openstackgerritMerged openstack/tripleo-heat-templates master: Bind redis-sentinel to its network  https://review.openstack.org/44695617:15
openstackgerritMerged openstack/tripleo-heat-templates master: Don't try to run os-net-config from yum_update.sh  https://review.openstack.org/44490617:16
*** yamahata has joined #tripleo17:17
shardybogdando: yup that's the one17:17
shardylooks like it already supports it17:17
openstackgerritGonéri Le Bouder proposed openstack/tripleo-quickstart master: user: group creation require root privilege  https://review.openstack.org/44760917:19
*** isq has joined #tripleo17:21
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Apply setfiles on all mountpoints  https://review.openstack.org/44707617:22
*** nyechiel has quit IRC17:28
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Add dracut-regenerate elements  https://review.openstack.org/44704717:29
*** fragatin_ has quit IRC17:30
*** rbrady-afk is now known as rbrady17:31
*** rlandy is now known as rlandy|mtg17:31
*** jaosorior has quit IRC17:35
openstackgerritCorey O'Brien proposed openstack/diskimage-builder master: Ignore missing path in unmount_dir  https://review.openstack.org/44662117:39
sshnaidmbnemec, hi17:40
*** rbrady has quit IRC17:40
bnemecsshnaidm: o/17:40
*** chlong has quit IRC17:42
*** suuuper has quit IRC17:43
fultonjIs the only accptable way to backport a change to cherry-pick it?17:43
fultonjI'm trying to backport https://review.openstack.org/#/c/411987 into Newton.17:43
fultonjThe cherry-pick is not going easily, though I only need 3 lines. Should I keep pushing through this cherry-pick?17:43
*** leanderthal has joined #tripleo17:44
shardyfultonj: hi, the normal process is to cherry pick, what alternative did you have in mind?17:45
*** nyechiel has joined #tripleo17:46
fultonjshardy: i thought it might be easier to just directly change newton17:47
shardyfultonj: I think you need to temporarily abandon https://review.openstack.org/#/c/442969/ for https://review.openstack.org/#/c/411984/ to land17:48
shardyfultonj: I always start with a cherry-pick, even when manual fixup is required17:48
shardyfultonj: you could just make the changes manually I guess, but it's nice to use cherry-pick -x so you get the reference to what commit the changes came from, even when there are conflicts17:49
bnemecfultonj: What problems are you having with the cherry-pick?17:49
*** marrusl has quit IRC17:49
fultonjshardy: ok, i did the  cherry-pick -x17:49
bnemecEven if it conflicts it seems like that should be easy enough to resolve.17:50
bnemecSo there may be something else going on.17:50
*** florianf has quit IRC17:50
fultonji can do an edit of what the cherry pick -x produced17:50
fultonjand commit from there17:50
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-heat-templates stable/ocata: Updated from global requirements  https://review.openstack.org/44391817:51
shardyfultonj: yes that's fine, you're supposed to leave the Conflicts: note in the commit message, or otherwise comment so reviewers know to pay extra attention e.g it wasn't a clean cherry-pick17:51
fultonjshardy: ok, thanks17:52
*** lmiccini has quit IRC17:54
*** jpich has quit IRC17:54
*** dbecker has quit IRC17:54
*** aufi has quit IRC17:55
weshay_sickrlandy|mtg, https://review.openstack.org/#/c/442764/17:56
openstackgerritHonza Pokorny proposed openstack/tripleo-specs master: Add network configuration spec  https://review.openstack.org/39638317:58
openstackgerritBogdan Dobrelya proposed openstack/tripleo-heat-templates master: Containerize etcd service  https://review.openstack.org/44762717:59
*** jcoufal_ has joined #tripleo18:00
*** dtrainor has quit IRC18:00
bogdandoflaper87, mandre: PTAL https://review.openstack.org/#/c/447627/1 etcd wip (I have yet test it as I still have oooq issues with a test env)18:00
*** derekh has quit IRC18:02
*** nyechiel has quit IRC18:02
*** lucasagomes is now known as lucas-afk18:02
*** jcoufal has quit IRC18:03
openstackgerritBen Nemec proposed openstack/instack master: Allow instack to work with pre-v2 versions of dib  https://review.openstack.org/44711618:05
openstackgerritBen Nemec proposed openstack/instack-undercloud stable/mitaka: Nothing to see here  https://review.openstack.org/44711718:05
bnemecThird time's a charm?18:05
openstackgerritChandan Kumar proposed openstack-infra/tripleo-ci master: [TEST JOB] test tempest-12.2.0 for Mitaka  https://review.openstack.org/44686818:07
openstackgerritMerged openstack/tripleo-heat-templates master: Containerize panko api service  https://review.openstack.org/43913818:08
openstackgerritMerged openstack/tripleo-ui master: Remove '\n' linebreak from localised string  https://review.openstack.org/44723718:08
openstackgerritMerged openstack/tripleo-ui master: Add redux-form dependency  https://review.openstack.org/44552518:08
*** masco has quit IRC18:09
*** athomas has quit IRC18:09
openstackgerritMerged openstack/tripleo-quickstart-extras master: tempest: Fix junit results generation  https://review.openstack.org/44255818:10
openstackgerritMerged openstack/tripleo-quickstart-extras master: overcloud-deploy: unset -e flag in deploy script  https://review.openstack.org/44604218:10
*** ooolpbot has joined #tripleo18:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION18:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406718:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423618:10
*** ooolpbot has quit IRC18:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)18:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]18:10
openstackgerritMerged openstack/tripleo-quickstart master: update localhost instructions to indicate 127.0.0.2 is required  https://review.openstack.org/44521318:10
*** salmankhan has quit IRC18:10
*** dspano has joined #tripleo18:11
*** dtantsur is now known as dtantsur|afk18:12
sshnaidmmandre, hi18:15
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Populate logs exclude list with infra requirements  https://review.openstack.org/44637418:17
*** tosky has quit IRC18:18
sshnaidmEmilienM, seems like reverting nova patch helped18:20
EmilienMsshnaidm: indeed18:20
EmilienMsshnaidm: they are investigating18:20
sshnaidmEmilienM, great.. also would be nice to know how to detect such things, I didn't catch anything suspicious in logs18:21
EmilienMsshnaidm: in nova scheduler18:22
EmilienMof the undercloud18:22
EmilienMNo valid host found18:23
*** jprovazn has joined #tripleo18:23
*** fragatina has joined #tripleo18:23
*** tbonds has quit IRC18:25
*** morazi has quit IRC18:25
*** bogdando has quit IRC18:26
*** tbonds has joined #tripleo18:27
*** ckyriakidou has quit IRC18:31
*** akrivoka has quit IRC18:31
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates stable/newton: Configure /etc/hosts via os-collect-config script  https://review.openstack.org/44763818:32
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Apply setfiles on all mountpoints  https://review.openstack.org/44707618:39
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Apply setfiles on all mountpoints  https://review.openstack.org/44707618:42
*** mburned is now known as mburned_out18:44
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/ocata: etcd: secure EtcdInitialClusterToken parameter  https://review.openstack.org/44651718:48
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Allow to configure policy.json for OpenStack projects  https://review.openstack.org/44570018:49
*** rlandy|mtg is now known as rlandy18:49
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: scenario001/pingtest: enable Gnocchi resource again  https://review.openstack.org/44279518:50
*** mcornea has quit IRC18:52
*** mburned_out is now known as mburned18:55
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Parameterize OVB stack flavors  https://review.openstack.org/44516318:56
*** cdearborn has joined #tripleo19:03
*** [1]cdearborn has quit IRC19:05
openstackgerritMerged openstack/instack-undercloud stable/ocata: Use purged for firewalld  https://review.openstack.org/44367919:08
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Use separate log collection playbook for TripleO CI  https://review.openstack.org/44764819:08
openstackgerritSagi Shnaidman proposed openstack-infra/tripleo-ci master: Use separate log collection playbook for TripleO CI  https://review.openstack.org/44764819:10
*** ooolpbot has joined #tripleo19:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION19:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406719:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423619:10
*** ooolpbot has quit IRC19:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)19:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]19:10
*** thrash is now known as thrash|biab19:11
openstackgerritJames Slagle proposed openstack/tripleo-docs master: Additional Networking docs for deployed-server  https://review.openstack.org/44222219:18
openstackgerritMerged openstack/tripleo-quickstart master: Revert "Update ci-script to enable TripleO gate dependencies"  https://review.openstack.org/44752619:19
openstackgerritMerged openstack/instack-undercloud master: Run Zaqar with mod_wsgi  https://review.openstack.org/44248219:19
*** morazi has joined #tripleo19:21
openstackgerritwes hayutin proposed openstack-infra/tripleo-ci master: WIP: create a README_LOGS to document log files  https://review.openstack.org/44765319:28
*** leanderthal has quit IRC19:33
openstackgerritIan Main proposed openstack/tripleo-heat-templates master: Remove Kolla config copy in glance-api.  https://review.openstack.org/44396719:36
*** salmankhan has joined #tripleo19:37
*** [1]cdearborn has joined #tripleo19:37
*** cdearborn has quit IRC19:39
trozetEmilienM: added release notes and passed CI: https://review.openstack.org/#/c/425934/19:41
*** salmankhan has quit IRC19:41
openstackgerrityolanda.robla proposed openstack/diskimage-builder master: Apply setfiles on all mountpoints  https://review.openstack.org/44707619:41
EmilienMtrozet: cool19:44
EmilienMadarazs: any progress on https://bugs.launchpad.net/tripleo/+bug/1674067 ?19:50
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)19:50
EmilienMgate-tripleo-ci-centos-7-nonha-multinode-oooq jobs are failing on stable/newton currently19:50
*** leanderthal has joined #tripleo19:50
openstackgerritJohn Fulton proposed openstack/puppet-tripleo stable/newton: Include nova::compute::libvirt::qemu from the libvirt profile  https://review.openstack.org/44297019:50
EmilienMadarazs: mhh, I saw successful jobs, so I guess the bug is not consistent19:52
EmilienMor fixed already19:52
openstackgerritDougal Matthews proposed openstack/python-tripleoclient master: Don't create tempest-deployer-input.conf in tests  https://review.openstack.org/44766519:55
stevebakermorning19:57
NgI'm trying to figure out why https://review.openstack.org/#/c/442497/ isn't merging - as shardy suggested I abandoned the ocata backport of the change it depends on, and rechecked, but zuul still doesn't seem to want to open the gates19:59
Ngany suggestions? :)19:59
Ngoh, damn, one of the jobs was still running, and failed20:00
openstackgerritJulie Pichon proposed openstack/tripleo-ui stable/ocata: Remove '\n' linebreak from localised string  https://review.openstack.org/44767020:00
*** nyechiel has joined #tripleo20:00
*** leanderthal has quit IRC20:02
openstackgerritSteve Baker proposed openstack/tripleo-quickstart-extras master: Use upload command to upload container images  https://review.openstack.org/44625220:02
openstackgerritVincent S. Cojot proposed openstack/tripleo-heat-templates stable/ocata: Fixes multiple issues with retry function in rhel-registration.  https://review.openstack.org/44767120:03
*** dprince has quit IRC20:05
*** ooolpbot has joined #tripleo20:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION20:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406720:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423620:10
*** ooolpbot has quit IRC20:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)20:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]20:10
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart master: DO NOT MERGE: Test for OVB yum update hang  https://review.openstack.org/44602620:11
dspanoquit20:13
*** dspano has quit IRC20:13
*** dtrainor has joined #tripleo20:15
*** fultonj_ has joined #tripleo20:17
*** dparkes_ has joined #tripleo20:17
*** spredzy_ has joined #tripleo20:18
*** lucas-afk has quit IRC20:18
*** sai has quit IRC20:20
*** leanderthal has joined #tripleo20:20
*** rook has quit IRC20:20
*** bugzy_ has joined #tripleo20:21
*** rook has joined #tripleo20:21
*** lucasagomes has joined #tripleo20:21
*** rook is now known as Guest4502520:21
*** sai has joined #tripleo20:22
*** bandini_ has joined #tripleo20:24
*** sdake_ has joined #tripleo20:24
*** sdake_ has joined #tripleo20:24
*** radez` has joined #tripleo20:25
*** jcoufal has joined #tripleo20:26
*** Guest45025 is now known as rook-ugh20:27
*** jcoufal_ has quit IRC20:29
*** jkilpatr has quit IRC20:29
*** fultonj has quit IRC20:29
*** dparkes has quit IRC20:29
*** larstobi has quit IRC20:29
*** cwolferh has quit IRC20:29
*** hjensas has quit IRC20:29
*** spredzy has quit IRC20:29
*** sdake has quit IRC20:29
*** bandini has quit IRC20:29
*** alanmeadows has quit IRC20:29
*** bugzy has quit IRC20:29
*** marios has quit IRC20:29
*** radez has quit IRC20:29
*** hjensas has joined #tripleo20:30
openstackgerritIan Main proposed openstack/tripleo-heat-templates master: Remove kolla_config copy from keystone service.  https://review.openstack.org/44767620:31
*** marios has joined #tripleo20:31
*** dtrainor has quit IRC20:31
*** cwolferh has joined #tripleo20:32
*** shardy has quit IRC20:32
*** jkilpatr has joined #tripleo20:32
*** zoli|gone is now known as zoli20:33
*** dtrainor has joined #tripleo20:35
*** shardy has joined #tripleo20:39
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates stable/newton: WIP: Increase libvirt/qemu.conf max_files and max_processes  https://review.openstack.org/44768020:39
*** larstobi has joined #tripleo20:40
*** larstobi has quit IRC20:40
*** larstobi has joined #tripleo20:40
Slowerso where are the pep8 settings for checking the containerized deployments?20:42
SlowerI'm working on removing kolla_config's but pep8 thinks they need to exist20:42
*** leanderthal has quit IRC20:43
*** pmannidi has joined #tripleo20:43
rook-ughis there a special way to start os-collect-config?20:47
rook-ughzaneb: ^ ?20:47
*** ansmith has quit IRC20:49
*** owalsh has quit IRC20:50
*** owalsh has joined #tripleo20:50
*** morazi has quit IRC20:50
rook-ughhttps://gist.github.com/jtaleric/8061c9dedb292e9c64027836633f4ee2 is what i am seeing wrt to the service20:51
Slowerrook-ugh: what are you trying to do?20:51
Sloweriirc there's a bunch of config stuff that needs to be setup to make it work20:51
rook-ughwell the serivce isn't running20:51
Sloweros-collect-config?20:51
rook-ughSlower the node was rebooted20:51
rook-ughand os-collect-config isn't running20:51
rook-ughcorrec.t20:51
Slowerhmm20:51
rook-ughcorrect*20:51
rook-ughso, there is no error code, message or antyhing20:52
rook-ughjust this json dumped20:52
SlowerI forget how that works now :)20:52
*** kjw3 has joined #tripleo20:52
SlowerI know I've wrestled with that before though20:52
rook-ughSlower ha, story of my deployment thus far20:53
rook-ughbetween os-collect / os-refresh / 51-hosts20:53
Slowerrook-ugh: is there a service for it?20:53
rook-ughyup20:53
Slowercan you just start it?20:53
Sloweris it enabled?20:53
rook-ughyup20:54
openstackgerritJohn Fulton proposed openstack/tripleo-heat-templates stable/newton: WIP: Increase libvirt/qemu.conf max_files and max_processes  https://review.openstack.org/44768020:54
Slowerbut? :)20:54
rook-ughit typically just sits in a loop right looking for metadta20:54
rook-ughi mean it exits 020:54
rook-ughso it looks like it is fine20:54
rook-ughhttps://gist.github.com/jtaleric/2a4c2e367748ab8020ae44eb77377002 Slower20:55
rook-ughthat is what i see in journalk20:55
rook-ughjournal20:55
*** paramite has joined #tripleo20:56
Sloweris it possible it's just not getting data?20:57
Slowerhmm, would you need to delete the deployments to make it rerun them?20:57
Slower /var/lib/heat-config/deployed20:59
*** rhallisey has quit IRC21:00
*** iranzo has quit IRC21:01
*** bfournie has quit IRC21:03
*** nyechiel has quit IRC21:04
*** jayg is now known as jayg|g0n321:05
*** vpickard is now known as vpickard_21:05
openstackgerritIan Main proposed openstack/tripleo-heat-templates master: Change kolla_config from required to optional in pep8.  https://review.openstack.org/44769321:06
*** thrash|biab is now known as thrash21:07
openstackgerritIan Main proposed openstack/tripleo-heat-templates master: Remove kolla config from heat services.  https://review.openstack.org/44397421:08
*** ooolpbot has joined #tripleo21:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION21:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406721:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423621:10
*** ooolpbot has quit IRC21:10
openstackLaunchpad bug 1674067 in tripleo "stable/newton delorean fails to build tripleo-heat-templates" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)21:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]21:10
*** abishop has quit IRC21:12
*** Goneri has quit IRC21:12
*** apetrich has quit IRC21:14
*** lblanchard has quit IRC21:15
*** apetrich has joined #tripleo21:15
openstackgerritIan Main proposed openstack/tripleo-heat-templates master: Remove kolla_config copy from keystone service.  https://review.openstack.org/44767621:15
openstackgerritIan Main proposed openstack/tripleo-heat-templates master: Remove Kolla config copy in glance-api.  https://review.openstack.org/44396721:16
openstackgerritIan Main proposed openstack/tripleo-heat-templates master: Remove kolla start from ironic.  https://review.openstack.org/44562021:17
*** social has quit IRC21:24
rlandybnemec: I have been running test jobs on rdocloud. I was asked to test with neutron port security. We are deploying with network isolation. Which templates should be included in the env.yam file for port security and net-iso (multi-nic and public-bond)? for example: do I need baremetal-ports-public-bond-port-security.yaml and baremetal-ports-all-port-security.yaml and ?21:25
*** morazi has joined #tripleo21:26
*** social has joined #tripleo21:26
bnemecrlandy: For multi-nic you need the "all" template, for public-bond the "public-bond" one.21:27
bnemecrlandy: This is what I'm using locally for multi-nic: http://paste.openstack.org/show/603157/21:28
bnemec(and ignore the comment that says the entries for a single network - I changed the values from what was in the sample file)21:29
bnemec...entries *are* for...21:29
rook-ughSlower: so, /etc/os-collect-config.conf was empty21:30
rook-ughfun!21:30
openstackgerritIan Main proposed openstack/tripleo-heat-templates master: Remove kolla_config sections from nova services.  https://review.openstack.org/44770021:31
Slowerrook-ugh: huh21:31
rlandybnemec: ok - thanks - I'll follow the paste.21:31
rook-ughSlower: yup, so rebooting nodes causes this to happen21:33
rook-ughSlower so if you reboot nodes and want to scale thing sup21:33
rook-ughthings up*21:33
rook-ughyou will have a bad day21:33
bnemecrook-ugh: Really?  What version is this?21:35
rook-ughbnemec: newton21:35
rook-ughI had three nodes experience this.21:35
openstackgerritSteven Hardy proposed openstack/tripleo-heat-templates stable/newton: Configure /etc/hosts via os-collect-config script  https://review.openstack.org/44763821:35
*** mburned is now known as mburned_out21:35
rook-ugh /etc/os-collect-config.conf was totally blank after a reboot :/21:35
rook-ughbut running os-collect-config manually i saw the metadata-url21:36
rook-ughso I could rebuild the config file21:36
*** jprovazn has quit IRC21:36
*** shardy has quit IRC21:37
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/newton: Use --disable= in subscription-manager to avoid shell expansion.  https://review.openstack.org/43968321:37
*** bfournie has joined #tripleo21:39
bnemecrook-ugh: Hmm, I'm noticing "temp_url_expires=2147483587" in the metadata_url.  Is it possible these servers were up long enough that the tempurl expired and something broke?21:40
bnemecOr is this a pretty new deployment?21:40
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates stable/newton: Fixes multiple issues with retry function in rhel-registration.  https://review.openstack.org/44770621:40
rook-ughbnemec: nope it was a new deployment21:42
rook-ughthat hasn't timed out yet21:43
rook-ughbnemec: as i mentioned I used the metadata url that was still some how cached? and it worked21:43
rook-ughugh21:43
rook-ughovercloud-openstack-ComputeAllNodesDeployment-4hwvwrhulljh21:43
rook-ughthis scale deployment is going to fail21:43
rook-ughnot sure I am going to get 300 nodes :/21:43
bnemecYeah, I'm kind of grasping at straws here.  I don't know off the top of my head how that file would end up empty.21:44
rook-ughbnemec: understod.21:54
rook-ughbnemec I have run into all kinds of fun issues with OOO/scale21:54
rook-ughbnemec: details https://gist.github.com/jtaleric/f970a2e89d944915d8543429950423c6#51-hosts-failing-to-complete21:55
rook-ughfull details across all my issues are there :D21:55
rook-ughfull details of all my issues (not disclosed) :P21:55
*** paramite has quit IRC21:55
bnemecI wonder if we should set up a periodic job that reboots nodes and scales up.21:56
bnemecWe couldn't run it on every patch, but at least it would give us some test coverage of this sort of thing.21:56
bnemecRight now I doubt anyone upstream is testing those things at all.21:57
openstackgerritBen Kero proposed openstack-infra/tripleo-ci master: ovb: Remove bin/ and usr/ from working_dir tarball  https://review.openstack.org/44717421:57
openstackgerritBen Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct  https://review.openstack.org/44693421:59
openstackgerritBen Kero proposed openstack-infra/tripleo-ci master: Revert second repo-setup run for overcloud nodes  https://review.openstack.org/44714222:00
*** jkilpatr has quit IRC22:03
*** jmelvin has quit IRC22:03
openstackgerritPradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize Redis service  https://review.openstack.org/44215122:05
*** sdake_ is now known as sdake22:07
*** percevalbot has quit IRC22:07
*** lyarwood has quit IRC22:07
*** pkovar has quit IRC22:09
*** lyarwood has joined #tripleo22:09
*** ooolpbot has joined #tripleo22:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION22:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406722:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423622:10
*** ooolpbot has quit IRC22:10
openstackLaunchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)22:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]22:10
rook-ughbnemec: i would be there is no one doing that22:10
rook-ugh:/22:10
rook-ughi wish i didn't have to reboot bnemec22:10
rook-ughbnemec: but the nodes are pingable, but for some unknown reason become "connection reset by peer"22:12
rook-ughwhen trying to ssh to them22:12
rook-ughrebooting "fixes" it22:12
*** percevalbot has joined #tripleo22:12
mwhahaharook-ugh: bad iptables rules?22:13
bnemecWeird.22:14
* bnemec has to run though22:14
rook-ughmwhahaha: nope, they worked previously..22:14
rook-ughmy supecision is a issue with dhcp renew?22:14
*** [1]cdearborn has quit IRC22:16
openstackgerritBen Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct  https://review.openstack.org/44693422:18
*** alanmeadows has joined #tripleo22:19
*** jcoufal has quit IRC22:19
EmilienMbkero: can you look https://bugs.launchpad.net/tripleo/+bug/1674067 when you have time?22:20
openstackLaunchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)22:20
EmilienMbkero: also, what's the status of your cleanup from Friday?22:21
*** salmankhan has joined #tripleo22:22
bkeroEmilienM: The status is that there are patches posted, waiting for review.22:23
bkeroThe proposals in IRC today are just swapping around the Depends-On and adding some duplicate environment variables to an ansible playbook to prevent a circular dependency22:23
openstackgerritMerged openstack/tripleo-common master: overcloud_containers.yaml defaults  https://review.openstack.org/44732222:24
EmilienMbkero: can you paste the links here please?22:24
bkerosure22:24
bkeroEmilienM: https://review.openstack.org/#/c/447142/ and https://review.openstack.org/#/c/446934/22:25
EmilienMthanks22:25
EmilienMbkero: are they ready? you're still pushing on it22:25
bkeroEmilienM: They should be ready22:26
EmilienM(tm) :)22:26
EmilienMI'll review when jobs are green22:26
EmilienMbkero: can you also look https://bugs.launchpad.net/tripleo/+bug/1674067 please?22:26
openstackLaunchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)22:26
bkeroEmilienM: Yes22:26
rook-ughmwhahaha: if I find another node that acts up ill pull a console before just rebooting it :)22:27
mwhahaharook-ugh: arp cache would be my other question like if they stop arping and routes get lost22:27
bkeroEmilienM: I'll ping you when I get a notice that the jobs passed again. Should be in about an hour and a half.22:27
bkeroand I guess I should go trigger some stable branch jobs too22:28
EmilienMplease22:28
EmilienMbkero: ocata & newton22:28
openstackgerritBen Kero proposed openstack/tripleo-heat-templates stable/ocata: TEST CHANGE, IGNORE  https://review.openstack.org/44686522:29
rook-ughmwhahaha thinking about it, considering ssh failed to exchange keys, i think i should of had ssh client with some more verbose/debug output22:29
bkeroThere's ocata ^22:29
openstackgerritBen Kero proposed openstack/tripleo-heat-templates stable/newton: TESTING NEWTON TOCI NET CHANGES, IGNORE  https://review.openstack.org/44771422:30
bkeroThere's newton ^22:30
*** rook-ugh is now known as rook22:30
* bkero wonders how many hours of compute time he's using with every patchset22:31
rookexit22:31
*** rook is now known as Guest838322:31
openstackgerritBen Kero proposed openstack/tripleo-heat-templates stable/newton: TESTING NEWTON TOCI NET CHANGES, IGNORE  https://review.openstack.org/44771422:35
bkeroForgot the Depends-On in the newton tests ^ :)22:35
*** zoli is now known as zoli|gone22:36
EmilienMbkero: a lot, that's why we need to be careful22:39
*** paramite has joined #tripleo22:40
openstackgerritOpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements  https://review.openstack.org/44012422:41
*** jkilpatr has joined #tripleo22:41
bkeroHaving trouble reproducing: http://paste.openstack.org/show/603514/22:44
bkeroI need to go see what this six.moves business is about. Maybe a missing dep from dlrn?22:44
bkeroNope, six is mentioned in dlrn's requirements.txt, hmm22:45
*** Guest8383 is now known as rook-ugh22:47
bkeroLooks like 'virtualenv venv; venv/bin/pip install -U setuptools' is failing on my clean centos7 host :/22:49
*** cmyster has quit IRC23:00
*** cmyster has joined #tripleo23:00
*** alanmeadows has quit IRC23:00
*** alanmeadows has joined #tripleo23:00
bkeroLooks like it's a known issue since the python-virtualenv package in centos7 is so old. -_- Yak partially shaven.23:05
*** dsariel has quit IRC23:10
*** ooolpbot has joined #tripleo23:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION23:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167406723:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/167423623:10
*** ooolpbot has quit IRC23:10
openstackLaunchpad bug 1674067 in tripleo "newton: tripleo-quickstart jobs don't build packages from stable/newton" [Critical,Triaged] - Assigned to Attila Darazs (adarazs)23:10
openstackLaunchpad bug 1674236 in tripleo "CI / promotion: Nova isn't aware of the nodes that were registered with Ironic" [Critical,Triaged]23:10
*** dsariel_ has quit IRC23:10
*** pradk has quit IRC23:15
openstackgerritPradeep Kilambi proposed openstack/tripleo-common master: Add redis images to overcloud containers  https://review.openstack.org/44217923:16
bkeroOkay, reproducing now23:17
bkeroWell, fixed that issue. Attempting reproduction instructions :)23:18
*** bugzy_ is now known as bugzy23:19
bkeroEmilienM: I can't reproduce. I'm thinking that maybe the reviews mentioned in the bug fixed it. http://paste.openstack.org/show/603521/23:21
*** paramite has quit IRC23:22
bkeroLet me see if I can replicate it in quickstart23:23
*** rhallisey has joined #tripleo23:32
*** dsneddon has quit IRC23:38
bkeroYep, i see problem and can fix23:43
*** salmankhan has quit IRC23:56
*** thrash is now known as thrash|g0ne23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!