Tuesday, 2014-07-29

*** sdake has quit IRC00:01
*** jml has quit IRC00:03
*** eguz has quit IRC00:04
*** rainya has quit IRC00:09
*** jml has joined #tripleo00:10
*** jml has joined #tripleo00:10
TheJulialifeless: re: 105878, out of curiosity, were you thinking get_input or am I misinterpretting?00:12
*** edmund has quit IRC00:14
TheJuliatchaypo: cats always think that, sadly I have to free feed mine00:15
tchaypobecause you sometimes abandon them for days?00:16
tchaypo(well, abandonment is how they pretend to see it at least)00:16
TheJuliatchaypo: no, otherwise things begin sliding off flat surfaces... and eventually she will knock over monitors/block view of whatever one I'm looking at00:18
cineramamy colleagues are enjoying the new scratching post i acquired for our office :)00:19
lifelessTheJulia: when you rebase it00:20
lifelessTheJulia: you'll see tchaypos stuff for public binds00:20
lifelessTheJulia: I think it will be trivial00:20
lifelesstchaypo: a tad00:21
TheJuliaThat was going to be my next question :)00:22
*** rlandy has quit IRC00:22
* tchaypo is still struggling with this idea that anything he wrote could be useful00:23
StevenKcinerama: Haha, I thought you had bought a scratching post for Moffett Towers when I first read that00:25
*** eghobo has joined #tripleo00:25
*** michchap_ has quit IRC00:25
*** eghobo has quit IRC00:25
*** michchap has joined #tripleo00:25
cineramaStevenK: perhaps that is a good idea. might need to be bigger00:26
*** sdake has joined #tripleo00:29
*** sdake has quit IRC00:29
*** sdake has joined #tripleo00:29
tchaypoStevenK: so i now have myself a cronjob every 5 minutes to run apt-mirror and bandersnatch00:35
tchaypobut conditional on a file I can set that indicates I'm at home00:36
tchaypohrm. that's a crappy check, let me fix that00:36
*** eghobo has joined #tripleo00:36
*** julim has joined #tripleo00:38
*** julim has quit IRC00:38
*** julim has joined #tripleo00:40
SpamapStchaypo: I bet you could tie in google location based reminders into an API of some kind to do that. ;)00:49
tchaypopfft.00:50
tchaypoi can just use iwconfig to check to see if I'm connected to my home network...00:50
StevenKtchaypo: -10 for not using iw00:52
tchaypowhat's iw00:52
StevenKDescription: tool for configuring Linux wireless devices00:53
* tchaypo was asking rhetorically00:53
SpamapSreally you can just do 'arp -n ip.your.router.usually.has' and grep for the mac00:53
tchaypoobviously it has the same relationship to iwconfig that ip has to ipconfig00:53
lifelessSpamapS: arpcache makes that invalid :)00:54
SpamapS$ cat /proc/sys/net/ipv4/neigh/wlan0/gc_stale_time01:01
SpamapS6001:01
SpamapSI'll take it.01:01
lifelessvs iw dev wlan0 connection info01:02
lifelessmeh01:02
*** dsneddon is now known as dsneddon_gone01:04
lifelesswhy oh why doesn't smarttravel let you change tickets01:05
* tchaypo opens OWAfor the once a day check01:09
tchaypolifeless: that reminds me, i need to get in touch with CWT soon01:09
lifelessre?01:10
tchaypofiguring out how to handle the paris trip01:11
tchaypoif i wait till we get confirmation from HP, i think it's goign to make the parts i have to pay for more expensive, so i want to figure out how I can pay for those bits early..01:12
openstackgerritJames Polley proposed a change to openstack/tripleo-image-elements: Fix issues with provider networks and public ips  https://review.openstack.org/10344901:13
tchaypo^^ just a rebase01:13
*** eguz has joined #tripleo01:29
*** eghobo has quit IRC01:33
*** eguz has quit IRC01:35
*** eghobo has joined #tripleo01:36
*** Yanyanhu has joined #tripleo01:38
*** rwsu has quit IRC01:42
*** eghobo has quit IRC01:44
openstackgerritJulia Kreger proposed a change to openstack/tripleo-heat-templates: Change overcloud to use VIP for MySQL  https://review.openstack.org/10587801:44
* tchaypo runs bandersnatch01:45
* tchaypo starts seeing error 503: no healthy backends01:45
* tchaypo cries a little01:45
StevenKFrom bandersnatch?01:46
*** lblanchard has quit IRC01:47
TheJuliagoodnight01:47
*** nosnos has joined #tripleo01:48
tchaypoit' come back now01:51
tchayponight TheJulia01:53
cineramabye TheJulia01:54
* tchaypo obviously hadn't run devtest in a long time on this laptop01:55
StevenKtchaypo: Oh?01:55
StevenKHuh, is 4.9 going to be the last release of gcc 401:58
StevenKgreghaynes: If you're still working your os-cloud-config keystone branch needs a rebase02:00
greghaynesbwah02:00
StevenKFor example, if somebody adds a feature to Nova or a hypervisor driver to deliver clowns and rainbows whenever you boot a flavor with "extra_specs:clowns_and_rainbows" = "yes please", ...02:02
StevenKHAHAHA02:02
greghaynessomeone needs to codify that in a unit test02:04
tchaypoit had to download all the cloudimages02:09
tchaypothen percona stuff..02:10
*** yamahata has quit IRC02:16
*** sandhya has joined #tripleo02:26
*** rainya has joined #tripleo02:44
*** rainya has quit IRC02:54
*** rainya has joined #tripleo02:54
*** sandhya has quit IRC02:58
* tchaypo deletes bandersnatch mirror02:58
* tchaypo installs devpi02:58
* tchaypo does another run to seed devpi with packages02:58
*** untriaged-bot has joined #tripleo03:00
untriaged-botUntriaged bugs so far:03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784503:00
uvirtbotLaunchpad bug 1347845 in tripleo "Cannot set selinux file contexts when kernel does not have selinux support compiled in" [Undecided,New]03:00
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/134193603:00
uvirtbotLaunchpad bug 1341936 in os-collect-config "Require keystone v3 auth_url in heat collector" [Undecided,In progress]03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134751403:00
uvirtbotLaunchpad bug 1347514 in tripleo "Overcloud upgrade failing due to tftp issue on one of the overcloud nodes" [Undecided,New]03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784803:00
uvirtbotLaunchpad bug 1347848 in tripleo "/etc/hosts information about cloud hsots lost on reboot" [Undecided,In progress]03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134598003:00
uvirtbotLaunchpad bug 1345980 in tripleo "Heat not able to create swift server" [Undecided,New]03:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134401403:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134823403:00
uvirtbotLaunchpad bug 1344014 in tripleo "horizon source install fails with "/tmp/in_target.d/install.d/100-horizon: line 5: /opt/stack/venvs/horizon/bin/pip: No such file or directory"" [Undecided,Fix committed]03:00
*** untriaged-bot has quit IRC03:00
uvirtbotLaunchpad bug 1348234 in tripleo "In a multi-controller overcloud config, br-ex is assigned the same IP in all controller nodes" [Undecided,New]03:00
*** nosnos has quit IRC03:20
vinshtchaypo, ping.03:24
tchaypoPong03:25
vinshHI :)03:27
vinshWas wondering if you had any scratch templates that make use of the expanded keepalived stuff you added in 104368?03:27
vinshSave me the time of botching the mustache03:27
vinshImma add this to my project, test the change out.03:28
vinshKey part is different vrrp vlan id's... as we have cases where users are sharing racks and hitting each other.  So this is great.03:28
tchaypoYeah, check for a change I have open in TripleO-Heat-templates03:28
vinshOK, will do.03:28
vinshshould have looked there already! :)03:29
*** eghobo has joined #tripleo03:29
tchaypo Unless that was what you were looking at, in which case TripleO-image-elements is what you want03:29
vinshI have the t-i-e change... but Id on't see any open t-h-t changes in your name03:30
vinshMaybe its taken over by someone else?03:30
tchaypo101438 is the number I think03:32
tchaypoOh03:32
tchaypoOh03:32
tchaypoThat's the one that landed already03:32
vinsh\0_o/03:32
tchaypoI'm not sure if it does what you want though03:33
vinshno, I'm looking for an overcloud-source that makes uses of the new config options you added for keepalived...I can probably figure it out.. but was looking to save time and copy paste :)03:33
tchaypoI'm out at lunch at the moment so I'm having to rely on my iPhone email03:34
vinshwait! thats the one!03:34
vinshsorry! I missed a didget in the id03:34
*** mestery_ has joined #tripleo03:34
vinshCarry on good sir.03:34
tchaypoHaha03:34
tchaypoExcellent :)03:34
vinshYeah :)  I'll be testing this out.. will report back in the review with +/- for ya.03:35
*** sandhya has joined #tripleo03:35
*** mestery has quit IRC03:36
tchaypoThanks :)03:36
StevenKBleh, $USE_IRONIC is 0, but still devtest is using ironic :-(03:39
*** mestery_ has quit IRC03:46
lifelessStevenK: oh ?03:52
*** mestery has joined #tripleo03:56
*** mestery_ has joined #tripleo03:59
*** mestery has quit IRC04:02
*** mestery_ has quit IRC04:11
*** edmund has joined #tripleo04:16
*** nosnos has joined #tripleo04:16
*** tzumainn has quit IRC04:16
tchaypodid you export?04:20
tchaypodid you blow away your envs and recreate?04:20
tchaypoffs04:20
tchaypoffs04:20
tchaypo+ echo 'Error: Heat environment cache "/home/polleyj/.cache/tripleo/undercloud-env.json" not set to permissions of 0600.'04:20
tchaypothats how long its been since I've used this machine04:21
greghayneswow04:21
*** mestery has joined #tripleo04:21
*** sandhya has quit IRC04:23
StevenK% ironic node-list04:26
StevenKpublicURL endpoint for baremetal service not found04:26
StevenKAh ha, victory when I left a build while noming04:27
*** sseago has quit IRC04:36
*** eghobo has quit IRC04:42
*** eghobo has joined #tripleo04:42
*** eguz has joined #tripleo04:44
*** sandhya has joined #tripleo04:45
*** eghobo has quit IRC04:48
openstackgerritSteve Kowalik proposed a change to openstack/os-cloud-config: Filter out registered nodes with nova-bm  https://review.openstack.org/10967404:54
*** morganfainberg is now known as morganfainberg_Z04:59
*** edmund has quit IRC04:59
StevenKgreghaynes: os-cloud-config master still calls keystone-manage pki_setup, btw05:00
*** rdopieralski has joined #tripleo05:04
lifelessjogo: hey05:10
*** rainya has quit IRC05:10
lifelessjogo: you use reveal.js right? have you perhaps done a helion theme ?05:10
*** rushiagr has joined #tripleo05:13
*** dshulyak_ has joined #tripleo05:14
*** sseago has joined #tripleo05:16
jogolifeless: I have not done a helion theme unfortunately05:23
lifelessjogo: done a hp blue theme perhaps ?05:24
StevenKjogo: Hai! When did you get into Brisbane?05:24
jogolifeless: no, but its easy to do05:26
jogoStevenK: on same flight as mikal05:27
* StevenK thought mikal_ was in CBR05:27
*** sandhya has quit IRC05:28
jogoCBR?05:28
jogohe is at the nova mid cycle05:28
StevenKjogo: Canberra's airport code05:28
StevenKOh, right05:28
StevenKAnd straight from the mid cycle to Brisbane?05:29
jogoyup05:29
*** hewbrocca has quit IRC05:31
*** mrunge has joined #tripleo05:45
*** mrunge has quit IRC05:46
*** noslzzp has quit IRC05:49
*** mrunge has joined #tripleo05:56
*** victor_lowther has quit IRC06:02
*** jtomasek has joined #tripleo06:03
*** victor_lowther has joined #tripleo06:10
StevenK% register-nodes -n <(jq "[.nodes[0]]" $TE_DATAFILE)06:12
StevenK'Node' object has no attribute '__getitem__'06:12
StevenKOh, blah06:12
*** victor_lowther has quit IRC06:22
*** pcrews has quit IRC06:22
tchaypothe nova midcycle is in CBR?06:28
tchaypothat's... out of the way...06:28
StevenKtchaypo: It is not, mikal lives in Canberra06:28
*** victor_lowther has joined #tripleo06:29
tchayposure, it's not out of the way *for him*06:32
tchaypobut actually this gives me an idea06:32
tchaypolifeless: can our next mid-cycle be in christchurch?06:32
StevenKBwahaha06:34
StevenKtchaypo: If that happens, I think Sarah will attend :-P06:34
lifelesstchaypo: I would like that06:36
lifelesstchaypo: I'm not sure it makes sense from the broader community angle06:36
openstackgerritlifeless proposed a change to openstack/tripleo-heat-templates: Expose public_interface_tag support.  https://review.openstack.org/10345406:38
*** rushiagr has quit IRC06:39
* tchaypo pulls06:39
*** rushiagr has joined #tripleo06:40
*** jprovazn has joined #tripleo06:41
*** pblaho has joined #tripleo06:42
*** eguz has quit IRC06:51
openstackgerritSteve Kowalik proposed a change to openstack/os-cloud-config: Filter out registered nodes with nova-bm  https://review.openstack.org/10967406:52
openstackgerritSteve Kowalik proposed a change to openstack/os-cloud-config: Filter out registered nodes with ironic  https://review.openstack.org/11021806:54
*** dshulyak_ has quit IRC07:04
*** cwolfe__ has quit IRC07:06
*** cwolferh has joined #tripleo07:12
*** jcoufal-mobile has joined #tripleo07:18
*** jcoufal-mobile has quit IRC07:23
*** jml has quit IRC07:25
*** jml has joined #tripleo07:27
*** jistr has joined #tripleo07:30
openstackgerritAlexis Lee proposed a change to openstack/tripleo-image-elements: Kibana element - Logstash Web UI  https://review.openstack.org/11022107:34
*** ifarkas has joined #tripleo07:36
openstackgerritMatthew Macdonald-Wallace proposed a change to openstack/tripleo-image-elements: Kibana element - Logstash Web UI  https://review.openstack.org/11022107:39
*** jp_at_hp has joined #tripleo07:46
*** lsmola has joined #tripleo07:49
openstackgerritA change was merged to openstack/tuskar: Removed MySQL-python test requirement  https://review.openstack.org/11007507:51
Nglifeless: no, wed-fri are us/east days07:57
lifelessack07:57
lifelessNg: wondering when we shall continue our hp2 bringup07:57
*** jistr_ has joined #tripleo07:58
*** jistr_ has quit IRC07:59
Nglifeless: Are you EOD now? I don't appear to have the 9am call in my calendar, so if you're around a bit we could continue now?08:00
jp_at_hplifeless: Can I get your review on https://review.openstack.org/106443 please?  I was asking others to hold off on reviewing it so you could get to it, as I know you had reservations.08:00
Nglooks like the seed was created, but didn't boot :<08:00
lifelessNg: joining the call?08:01
lifelessNg: I can be around a bit08:01
lifelessjp_at_hp: thanks, shall do08:01
Nglifeless: huh, On my way!. It's not in my calendar08:01
Ngbut my phone is reminding me abou tit08:01
Nglifeless: https://bugs.launchpad.net/heat/+bug/130073408:05
uvirtbotLaunchpad bug 1300734 in heat "Stack abandon deletes stack domain project" [High,Triaged]08:05
StevenKBleh. EEM has all my expenses except for the last day :-/08:05
openstackgerritA change was merged to openstack/diskimage-builder: Adds RHEL common element  https://review.openstack.org/10467008:08
*** dtantsur|afk is now known as dtantsur08:09
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Bootstrap 3 and SCSS  https://review.openstack.org/11022908:09
*** killer_prince is now known as lazy_prince08:19
*** lazy_prince is now known as killer_prince08:22
*** andrearosa has quit IRC08:29
openstackgerritNicholas Randon proposed a change to openstack/tripleo-heat-templates: Add specific pass-through config for each role  https://review.openstack.org/10841508:30
*** xuhaiwei has quit IRC08:31
*** andreaf has joined #tripleo08:32
lifelessNg: so yea08:36
lifelessNg: derek put more refs in hp108:37
lifelessNg: where did we get to on hp2?08:37
*** jang has joined #tripleo08:37
Nglifeless: last thing I ran on friday was devtest_seed.sh. The seed booted, but doesn't seem to have become pingable. just about to poke at its console to figure out why08:37
lifelessNg: ok08:38
lifelessNg: I'll be around for a bit - little relaxing then expenses08:40
lifelessNg: so ping for input08:40
Nglifeless: good idea to stock up on relaxation before tackling EEM ;)08:40
lifelessenraging electronic machinery08:42
*** andrearosa has joined #tripleo08:46
StevenKMy EEM paperwork is not done, waiting for a few more transactions to load before Ng gets his favourite thing ever08:54
StevenKI'm hoping they'll turn up tomorrow so that I can submit the report before getting more at pycon08:56
*** gfidente has joined #tripleo08:56
lifelesscrap08:57
lifelessjust found out that for some reason I'm booked on a totally diffferent flight on thursday :/08:57
lifelessbizarrely I have a reminder from airnz for the flight i thought I was on!08:59
* StevenK stabs CBS09:01
*** untriaged-bot has joined #tripleo09:01
untriaged-botUntriaged bugs so far:09:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784509:01
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/134193609:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134751409:01
uvirtbotLaunchpad bug 1347845 in tripleo "Cannot set selinux file contexts when kernel does not have selinux support compiled in" [Undecided,New]09:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784809:01
uvirtbotLaunchpad bug 1341936 in os-collect-config "Require keystone v3 auth_url in heat collector" [Undecided,In progress]09:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134598009:01
uvirtbotLaunchpad bug 1347514 in tripleo "Overcloud upgrade failing due to tftp issue on one of the overcloud nodes" [Undecided,New]09:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134823409:01
uvirtbotLaunchpad bug 1347848 in tripleo "/etc/hosts information about cloud hsots lost on reboot" [Undecided,In progress]09:01
uvirtbotLaunchpad bug 1345980 in tripleo "Heat not able to create swift server" [Undecided,New]09:01
*** untriaged-bot has quit IRC09:01
uvirtbotLaunchpad bug 1348234 in tripleo "In a multi-controller overcloud config, br-ex is assigned the same IP in all controller nodes" [Undecided,New]09:01
StevenKThey filmed everything at comic con, but Austrlians can't view the videos09:01
StevenKlifeless: Different flight how?09:02
lifelessStevenK: virgin vs airnz, 15m earlier09:03
openstackgerritGiulio Fidente proposed a change to openstack/tripleo-heat-templates: set compute nodes deployment depend on controller config  https://review.openstack.org/11006909:03
openstackgerritGiulio Fidente proposed a change to openstack/tripleo-heat-templates: fix incorrect reference to rabbit_user_name  https://review.openstack.org/11023909:07
gfidentelifeless, hi there, wanted to add some details about ^^ ; I see you left a comment about it slowing down the deployment09:25
gfidenteI was discussing this with TheJulia yesterday09:26
gfidentethe problem we are trying to address is that rabbit and mysql clients get disconnected and reconnected until keepalive + haproxy + clusters are stable09:28
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of JQ expression  https://review.openstack.org/11024309:29
openstackgerritAlexis Lee proposed a change to openstack/tripleo-incubator: Promote HEAT_ENV over env vars  https://review.openstack.org/9513009:29
gfidentecurrently, without the patch, we end up with nova-compute logging an error where it says conductor is not available due to the rabbit cluster being shutting down and then up the nodes for cluster configuration09:29
gfidenteand we aren't using the VIP for mysql yet, it may be challenging as well09:30
gfidenteso I understand that would be slowing down the deployment of the compute nodes, but not that much ... we just need to wait for the controllers to be running before moving into the computes configuration09:31
gfidenteand I don't see many alternatives to that... if not trying to implement some checks on the clients09:32
*** derekh_ has joined #tripleo09:33
*** jang has quit IRC09:33
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Improve readability of JQ expression  https://review.openstack.org/11024309:34
*** rlandy has joined #tripleo09:37
lifelessgfidente: presumably this will happen when rabbit is bounced for reconfig as well09:48
lifelessgfidente: so it seems like its business as normal, no?09:48
openstackgerritAlexis Lee proposed a change to openstack/tripleo-incubator: Promote HEAT_ENV over env vars  https://review.openstack.org/9513009:51
*** akrivoka has joined #tripleo09:52
gfidentelifeless, I'm not sure because I haven't tested the reconfig scenario, I see your point though about a similar condition being caused by a reconfig of the controllers09:55
*** Yanyanhu has quit IRC09:56
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Bootstrap 3 and SCSS  https://review.openstack.org/11022909:58
gfidenteonly I don't see many alternatives; for the specific issue with nova I can check with someone @nova if this is a misbehaviour which could be managed differently by nova, but I suspect more issues related to the ordering of operations happening on external nodes could raise09:59
lifelessgfidente: anyhow, I really don't think we should slow down the deploy of machines; the software-deployment on top of the machines is the place to pause things09:59
*** ramishra has joined #tripleo09:59
lifelessgfidente: I don't understand the impact though, what actually *goes wrong*?09:59
gfidentenova-compute/nova-conductor -> vip -> rabbit10:00
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Test revert  https://review.openstack.org/11025010:00
gfidentedue to rabbit nodes being brought down and then up again for the purpose of configuring the cluster10:00
derekh_CI appears to be broken, testing revert ^^10:01
gfidentenova-compute/nova-conductor get disconnected and reconnected10:01
lifelessgfidente: yes, but thats normal10:01
jprovazngfidente: lifeless: a check in os-refresh-config script which would test if rabbitmq on server side is running and ready might be an alternative to dependency in heat template10:01
lifelessgfidente: what is the /problem/ that occurs?10:01
gfidenteyes only nova-compute at that point shows a log indicating that it started before nova-conductor and as such isn't capable of sending the status updates10:01
lifelessgfidente: thats fine, no ?10:02
gfidenteso nova-conductor reports the compute node as down10:02
lifelessgfidente: for how long?10:02
lifelessgfidente: also this is on deploy, so nova-conductor isn't necessarily up either ;)10:02
gfidentefor 2h, at which point the OS socket timeout enforces socket close, nova-compute reconnects and things get back to work10:02
lifelessgfidente: ah!, now we have the issue ;)10:02
gfidenteoverall is not too bad10:03
lifelessgfidente: missing keepalive setting on the client amqp socket then10:03
gfidentebut I couldn't think of many other things10:03
gfidentelifeless, we could 'enforce' haproxy to close the connection before10:04
gfidentebut then it would even when things are working and there isn't traffic in between the nodes10:04
lifelessoh hmm10:05
lifelessgfidente: if you have this replicated, worth checking to see if haproxy has a tcp keepalive set10:05
lifelessgfidente: we'll want that, because this can happen at other times than deployment10:05
gfidentelifeless, I can see that yes10:06
gfidentejprovazn, regarding a 'check' in the script, I was checking nova code it seems to do an rpc ping10:06
*** jcoufal-mobile has joined #tripleo10:07
*** jcoufal-mobile has quit IRC10:08
gfidentejprovazn, the ping is repeated and is supposed to recover when conductor/cluster are back up but it actually doesn't seem to work10:10
*** rakesh_hs has joined #tripleo10:12
jprovazngfidente: agree with your yesterday's idea that maybe we could do some other check which would test something what is set when nova-conductor comes up, e.g. "rabbitmqctl list_queues |grep conductor"10:13
gfidentejprovazn, got it, will try that10:13
gfidentethanks10:13
jprovazngfidente: unless the tcpkeepalive check on haproxy level will not work :)10:14
lifelessjp_at_hp: btw see 5ab09ed27c34c4fa003917baec3bf366a7ce1f26 in global requirements repo10:15
lifelessjp_at_hp: I'm thinking we're going to have a headache with the rootwrap 0 release10:15
gfidentejprovazn, so you'd try lowering the timeout on haproxy first and, should that fail, think about a check in the element?10:15
lifelessgfidente: no - get haproxy to use tcp keepalive10:16
lifelessgfidente: its whats meant to signal this to haproxy10:16
lifelessgfidente: the 2H thing is some multiple of MSL if I recall correctly. I am going to guess that the fedora firewall is interfering with getting a RST on unexpected traffic10:17
lifelessgfidente: the tcp keepalive will deal with that10:17
gfidentelifeless, the 2h timeout is from /proc/sys/net/ipv4/tcp_keepalive_time10:18
lifelessblink10:18
jprovazngfidente: lifeless: actually tcpka is already on in haproxy.cfg10:18
gfidentejprovazn, lifeless yes that is what makes -compute reconnect after 2h10:19
*** akrivoka has quit IRC10:19
*** derekh_ changes topic to "CI JOBS FAILING, BUG 1349774 | Using OpenStack to deploy OpenStack;meetings Tuesday 1900//0700 UTC in #openstack-meeting-alt"10:20
lifelessjp_at_hp: what keepalive timeout do we use in hp cloud's servers?10:21
*** marzif has joined #tripleo10:24
tteggellifeless: 4510:24
lifelessgfidente: ^10:25
lifelesstteggel: thanks10:25
gfidentelifeless, :)10:25
jp_at_hptteggel: we have different values for different server types though, don't we?  So that would specifically be a nova value, I expect on compute and control, etc?  Correct?10:26
tteggelcorrect that's a -compute value10:26
lifelessdo we need different values ? or is that accidental ?10:26
jp_at_hplifeless: I would have to ask our discrete tribal knowledge pools...10:29
tteggeljp_at_hp: haha too true. I'm looking at our chef recipes10:29
jprovazngfidente: as you pointed out, tcpka won't help because it's haproxy<->rabbitmq check only, we need client<-<rabbitmq check10:30
lifelessjprovazn: we'll want both I think10:30
lifelessjprovazn: client <->haproxy to deal with haproxy node failures10:30
jprovaznlifeless: yes, though tcpka doesn't solve this particular issue :(10:30
lifelessjprovazn: and haproxy<->rabbitmq to deal with the rabbit going10:30
lifelessjprovazn: 45 second outage would be tolerable, no?10:31
gfidenteI'm not sure the root issue is around the keepalive timeout though10:31
lifelessgfidente: I'm betting on firewall myself10:31
lifelesssince no machines are failing10:32
lifelessor being rebooted10:32
lifelessthe kernel /should/ be issuing RST on writes to a socket on a closed process10:32
lifelessor10:32
lifelessperhaps you have a stale process10:32
lifelessgfidente: I agree that the keepalive isn't the root cause - but having a good mitigation in place is a useful thing10:33
*** akuznetsov has quit IRC10:33
lifelessbecause other things might cause the same broad issue10:33
gfidentelifeless, yes I agree and I'm suspecting maybe rabbitmqctl stop_app is probably not actually closing the socket?10:33
jprovaznlifeless: 45 secs might be fine (requires some testing)10:33
lifelessyeah, we've had issues with rabbitmq restarts going badly10:33
lifelesscould be related10:33
gfidenteI'll try to gather more details and get back if anything useful pops up10:34
*** akuznetsov has joined #tripleo10:38
*** sandhya has joined #tripleo10:41
*** ccrouch has joined #tripleo10:46
StevenKjp_at_hp: CI doesn't exercise pull-tools, could you paste to https://review.openstack.org/109386 some output of the different scenarios?10:47
jp_at_hpStevenK: is that text sufficient for you?10:50
*** ccrouch has quit IRC10:52
StevenKjp_at_hp: Yes, I understand what it's supposed to do, I'm looking for actual output from pull-tools, not an explanation10:53
*** ccrouch has joined #tripleo10:54
openstackgerritRadomir Dopieralski proposed a change to openstack/tuskar-ui: Bootstrap 3 and SCSS  https://review.openstack.org/11022910:55
jp_at_hpI'm kinda wondering why, and as I typically run devtest.sh start to end, not the easiest :(10:55
openstackgerritChristian Berendt proposed a change to openstack/os-cloud-config: Bump hacking to 0.9.x series  https://review.openstack.org/10829810:55
StevenKjp_at_hp: Because CI doesn't test that portion of the script10:56
jp_at_hpso what is it that you want the output to show?  pre-change fail and post-change pass?10:56
Ngblah calls blah10:57
Nglifeless: so it seems that the seed came up on the default libvirt network and not the brbm one10:57
StevenKjp_at_hp: post-change full output with one or more of the branches not on master10:57
jp_at_hpStevenK: OK - I might get to that sometime later today.  Kinda buried in other things right now10:58
derekh_ok, temprevert horizon 92146772b677e9fce57cc11b4a4a1542a05c23b2 134977410:59
derekh_has passed both undercloud jobs,10:59
derekh_so we can either wait for overcloud to finish, or just merge it, its obviously a better place then everything failing10:59
derekh_thoughts ^^^^11:00
derekh_bug 134977411:00
uvirtbotLaunchpad bug 1349774 in tripleo "Error parsing _stylesheets.html" [Critical,Triaged] https://launchpad.net/bugs/134977411:00
*** rushiagr has quit IRC11:00
*** rushiagr_ has joined #tripleo11:00
*** ramishra has quit IRC11:00
Nghmm11:01
*** rushiagr_ has quit IRC11:02
*** rushiagr_ has joined #tripleo11:02
* derekh_ will update the commit message once we decide what todo11:02
*** mrunge has quit IRC11:04
*** pblaho has quit IRC11:06
*** rushiagr_ has quit IRC11:06
openstackgerritJon-Paul Sullivan (jp_at_hp) proposed a change to openstack/tripleo-image-elements: install syslinux-dev  https://review.openstack.org/11025911:10
*** rushiagr_ has joined #tripleo11:11
lifelessNg: its meant to come up on both11:12
lifelessNg: check os-collect-config log within the seed11:12
Nglifeless: yeah just noticed that from comparing to hp111:12
TheJuliagood morning11:12
*** rushiagr_ has quit IRC11:12
lifelessruh roh11:13
*** rushiagr_ has joined #tripleo11:13
*** rushiagr_ has quit IRC11:17
*** rushiagr_ has joined #tripleo11:21
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Revert Update Twitter Bootstrap to version 3  https://review.openstack.org/11025011:22
derekh_ok, all 3 jobs passed, merge this https://review.openstack.org/#/c/109412/1 then https://review.openstack.org/#/c/109412/1 to unblock CI11:23
derekh_jp_at_hp: lifeless Ng SpamapS GheRiver1 lxsli bnemec greghaynes slagle ^^^11:24
* Ng looks11:24
derekh_woops did I post the same link twice, the second one should be https://review.openstack.org/#/c/110250/211:24
*** rushiagr_ has quit IRC11:25
*** rushiagr_ has joined #tripleo11:27
*** sandhya has quit IRC11:31
lifelessTheJulia: at the pub on the first night of the sprint - cinerama, me, you; memory says there was a fourth ?11:31
*** GheRiver1 is now known as GheRivero11:32
* GheRivero looks11:32
slaglederekh_: the first didn't pass all the CI jobs. do we want to recheck it, or just go as-is?11:32
derekh_slagle: I think its good as is, the patch that depends on it passed them all, I probably should have recheck it while the other one was running11:33
jp_at_hpslagle: derekh_ for context that failed to pull pip.  So obviously related to nova stats ;-)11:34
*** morazi has joined #tripleo11:34
*** rushiagr_ has quit IRC11:34
jp_at_hpimho we have a nova bm pass and an undercloud pass, and an ubuntu pass anda f20 pass in the 2 that got through11:35
jp_at_hp(oh - and an ironic pass)  so a cross-section of all tests did pass successfully.11:35
*** pblaho has joined #tripleo11:36
openstackgerritA change was merged to openstack-infra/tripleo-ci: Remove nova revert "Add extensible resources"  https://review.openstack.org/10941211:37
*** rakesh_hs has quit IRC11:37
openstackgerritA change was merged to openstack-infra/tripleo-ci: Revert Update Twitter Bootstrap to version 3  https://review.openstack.org/11025011:37
slagleyea agreed11:37
derekh_revert also submitted to horizon https://review.openstack.org/#/c/110258/ (its a big patch ...)11:37
derekh_slagle: jp_at_hp StevenK thanks11:38
*** dsneddon_gone has quit IRC11:38
jp_at_hpI think it is definitely more of a thank you derekh_ ...11:39
derekh_:-)11:39
Ngcrap, you guys merged it before I got done writing an email11:41
NgI just flipped back to the review windows and saw "Twitter bootstrap" in a commit message and was sure I must have the wrong review links ;)11:42
*** dshulyak_ has joined #tripleo11:43
derekh_Ng: no prob11:50
*** derekh_ changes topic to "Using OpenStack to deploy OpenStack;meetings Tuesday 1900//0700 UTC in #openstack-meeting-alt"11:50
derekh_ok, CI should be passing again, jobs were failing for the last 13 hours or so11:50
* lifeless finally wraps up EEM... for now11:52
lifelessmarriott utterly failed at emailing me the receipt11:53
TheJulialifeless: I think it was greghaynes11:53
lifelessso I have to ring them11:53
lifelessTheJulia: thanks11:53
*** dshulyak_ has quit IRC11:55
StevenKlifeless: Marriott also failed at that11:55
StevenKBut I at least have a very badly printed receipt11:55
*** weshay has joined #tripleo11:56
lifelessStevenK: I thought I had one.11:58
lifelessStevenK: I canna find it11:58
openstackgerritJan Provaznik proposed a change to openstack/tripleo-heat-templates: Parametrizing the number of Swift replicas  https://review.openstack.org/8607611:58
TheJuliaWell on a plus side, its 8 am locally, you could call them and ask for another copy.12:00
lifelessTheJulia: I will, but not now :).12:00
lifelessright now, its 000012:01
lifelessso=>sleep12:01
TheJulia:)12:01
Nglifeless: IPs and vlan tag make it into init-neutron-ovs, but the interface/device variables all seem to be empty. Not sure if that's significant yet, still digging through the log12:02
lifelessNg: gl12:02
* lifeless sleeps12:02
Ngyeah, looks significant, lack of EXTERNAL_BRIDGE causes the later iptables commands to fail, exploding the o-r-c run12:03
*** hewbrocca has joined #tripleo12:03
*** dguerri`afk is now known as dguerri12:10
*** mrunge has joined #tripleo12:15
*** mrunge has quit IRC12:29
*** jistr has quit IRC12:29
openstackgerritLadislav Smola proposed a change to openstack/tuskar-ui: Aggregated statistics for roles  https://review.openstack.org/11027612:30
*** jistr has joined #tripleo12:33
*** dtantsur is now known as dtantsur|lunch12:33
*** jdob has joined #tripleo12:38
*** lblanchard has joined #tripleo12:48
*** nosnos has quit IRC12:51
*** dtantsur|lunch is now known as dtantsur12:53
*** sballe has joined #tripleo12:55
*** cwolferh has quit IRC12:56
*** noslzzp has joined #tripleo13:01
*** jistr has quit IRC13:01
*** jistr has joined #tripleo13:03
*** pblaho_ has joined #tripleo13:03
*** pblaho has quit IRC13:03
openstackgerritNicholas Randon proposed a change to openstack/tripleo-heat-templates: controller0AllNodesConfig to controller0AllNodes.  https://review.openstack.org/11028613:13
*** tzumainn has joined #tripleo13:13
*** ccrouch has quit IRC13:19
*** ccrouch has joined #tripleo13:20
*** dguerri is now known as dguerri`afk13:25
*** edmund has joined #tripleo13:29
*** dguerri`afk is now known as dguerri13:29
*** jprovazn has quit IRC13:29
*** akrivoka has joined #tripleo13:30
*** jprovazn has joined #tripleo13:31
*** pcrews has joined #tripleo13:52
openstackgerritMatthew Gilliard proposed a change to openstack/os-cloud-config: Allow capabilities to be set on ironic nodes  https://review.openstack.org/11004813:55
openstackgerritMatthew Gilliard proposed a change to openstack/os-cloud-config: Allow capabilities to be set on ironic nodes  https://review.openstack.org/11004813:58
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Allow user to keep virsh instances alive  https://review.openstack.org/10898914:03
*** pblaho_ has quit IRC14:09
*** ddieterly has joined #tripleo14:12
*** jistr has quit IRC14:12
*** jistr has joined #tripleo14:13
*** jdob has quit IRC14:15
*** jdob has joined #tripleo14:15
*** jdob has quit IRC14:15
*** jdob has joined #tripleo14:16
*** jdob has quit IRC14:16
*** jdob has joined #tripleo14:17
*** chuckC has joined #tripleo14:18
*** ccrouch1 has joined #tripleo14:23
openstackgerritNicholas Randon proposed a change to openstack/tripleo-incubator: Correct default options for help  https://review.openstack.org/11030114:25
*** ccrouch has quit IRC14:27
*** akrivoka has quit IRC14:31
*** rainya has joined #tripleo14:31
*** rainya has joined #tripleo14:32
*** dguerri is now known as dguerri`afk14:43
openstackgerritIvan Berezovskiy proposed a change to openstack/diskimage-builder: Create cache directory  https://review.openstack.org/11031014:45
*** dguerri`afk is now known as dguerri14:45
openstackgerritGerry Drudy proposed a change to openstack/tripleo-image-elements: Enable the swift container-reconciler  https://review.openstack.org/11031114:46
openstackgerritJay Dobies proposed a change to openstack/tuskar: Added plan parameter data model  https://review.openstack.org/10869714:50
*** akrivoka has joined #tripleo14:54
*** jprovazn has quit IRC14:58
*** untriaged-bot has joined #tripleo15:00
untriaged-botUntriaged bugs so far:15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784515:00
untriaged-bothttps://bugs.launchpad.net/os-collect-config/+bug/134193615:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134751415:00
uvirtbotLaunchpad bug 1347845 in tripleo "Cannot set selinux file contexts when kernel does not have selinux support compiled in" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134987215:00
uvirtbotLaunchpad bug 1341936 in os-collect-config "Require keystone v3 auth_url in heat collector" [Undecided,In progress]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784815:00
uvirtbotLaunchpad bug 1347514 in tripleo "Overcloud upgrade failing due to tftp issue on one of the overcloud nodes" [Undecided,New]15:00
uvirtbotLaunchpad bug 1349872 in tripleo "setup-clienttools doesn't honour proxy config" [Undecided,New]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134598015:00
uvirtbotLaunchpad bug 1347848 in tripleo "/etc/hosts information about cloud hsots lost on reboot" [Undecided,In progress]15:00
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134823415:00
uvirtbotLaunchpad bug 1345980 in tripleo "Heat not able to create swift server" [Undecided,New]15:00
*** untriaged-bot has quit IRC15:00
uvirtbotLaunchpad bug 1348234 in tripleo "In a multi-controller overcloud config, br-ex is assigned the same IP in all controller nodes" [Undecided,New]15:00
*** dguerri is now known as dguerri`afk15:01
*** morganfainberg_Z is now known as morganfainberg15:01
*** dguerri`afk is now known as dguerri15:01
*** rdopieralski has quit IRC15:03
greghaynesTheJulia: whatd I do?15:07
openstackgerritJay Dobies proposed a change to openstack/tuskar: Added plan parameter data model  https://review.openstack.org/10869715:07
*** rushiagr has joined #tripleo15:09
*** jistr has quit IRC15:11
jmoffittis there any way to turn on debugging or a verbose mode for when running the tuskar-dbsync? I'm still trying to resolve the following error: CRITICAL tuskar [-] OperationalError: (OperationalError) unable to open database file None None15:14
openstackgerritScott DAngelo proposed a change to openstack/tripleo-image-elements: Add explicit configuraton parameters for DB pool size  https://review.openstack.org/10965115:14
*** dshulyak_ has joined #tripleo15:25
derekh_ironic job now failing15:35
derekh_bug 134991315:35
uvirtbotLaunchpad bug 1349913 in tripleo "Error: Failed to execute command via SSH" [Critical,Triaged] https://launchpad.net/bugs/134991315:35
openstackgerritGiulio Fidente proposed a change to openstack/tripleo-heat-templates: Restore rabbit.host and add an haproxy rabbit listener  https://review.openstack.org/10864715:41
*** rwsu has joined #tripleo15:42
*** cwolferh has joined #tripleo15:43
*** ramishra has joined #tripleo15:43
*** ramishra has joined #tripleo15:44
*** ramishra has quit IRC15:44
*** ramishra has joined #tripleo15:44
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Use locally-cloned client tools  https://review.openstack.org/11034115:46
*** dguerri is now known as dguerri`afk15:47
cineramahi guys15:47
*** ramishra has quit IRC15:49
*** dguerri`afk is now known as dguerri15:49
*** ifarkas has quit IRC15:49
TheJuliagreghaynes: nothing :)15:50
TheJuliacinerama: good morning15:50
*** derekh_ changes topic to "IRONIC JOB FAILING BUG 1349913 | Using OpenStack to deploy OpenStack;meetings Tuesday 1900//0700 UTC in #openstack-meeting-alt"15:54
*** eghobo has joined #tripleo15:56
*** IvanBerezovskiy has left #tripleo15:57
*** eguz has joined #tripleo15:58
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Revert SSH virsh to use the new ManagementInterface  https://review.openstack.org/11035216:01
*** eghobo has quit IRC16:02
*** lucasagomes has joined #tripleo16:02
lxsliAny reason we don't use oac at build time to install static files?16:04
openstackgerritMatthew Gilliard proposed a change to openstack/tripleo-incubator: Use locally-cloned client tools  https://review.openstack.org/11034116:04
*** jang has joined #tripleo16:04
slaglelxsli: i think it just kind of lacked the finer grained control we need to install files. permissions, etc.16:05
slaglelxsli: the control file stuff you're working on would address that i suppose :)16:05
lxsliawesome might push a patch then16:05
slaglein general, though, i'm pro making things more declarative vs procedural bash scripts16:06
lxsliincreasing numbers of elements are including a files/ dir and installing everything from the install.d script16:06
slagleone thing to watch out for though is package-installs vs source-installs types16:06
ccrouch1gfidente: ping16:07
gfidentepong ccrouch116:07
greghayneslxsli: we do as much as we can16:18
*** andrearosa has quit IRC16:19
lxsligreghaynes: huh?16:19
greghaynesinstall static files at build time16:19
greghaynesvia oac16:19
lxslido you have any examples please?16:19
lxsliI was thinking of adding support for a static/ folder which an oac elemeent install.d script rsyncs into the chroot16:20
greghaynesoh, you want to bypass the o-a-c rsync in at run time?16:20
lxslior not rsyncs, runs oac on16:20
lxslinah just eliminate the manual install.d scripting required to install static files16:21
lxsliI'm spending some time telling people to do that instead of installing static files with oac and if I could just say "move the files to static/" it'd be an easier sell16:21
greghaynescheck out files in elements/*/os-apply-config16:21
lxsliI've read all those16:22
greghaynesI am confused of the exact use case apparently16:22
lxslipatch will be ready soon :)16:22
*** rakesh_hs has joined #tripleo16:31
openstackgerritstephen-mulcahy proposed a change to openstack/tripleo-image-elements: Add new multipath element  https://review.openstack.org/11036316:31
*** rakesh_hs2 has joined #tripleo16:32
openstackgerritAlexis Lee proposed a change to openstack/tripleo-image-elements: Install static element files with oac  https://review.openstack.org/11036616:34
*** rakesh_hs has quit IRC16:35
*** yamahata has joined #tripleo16:36
*** chuckC has quit IRC16:38
*** bashok has joined #tripleo16:39
openstackgerritstephen-mulcahy proposed a change to openstack/tripleo-image-elements: Add new multipath element  https://review.openstack.org/11036316:40
*** andrearosa has joined #tripleo16:40
*** ddieterly has quit IRC16:42
ekarlsogreghaynes: u about boss ? :)16:43
greghayneskinda - in sprint like yesterday so theres some serious interrupt storm happening16:44
greghaynesbut ask away16:44
*** ddieterly has joined #tripleo16:45
*** pblaho_ has joined #tripleo16:52
*** dsneddon has joined #tripleo16:54
*** marzif has quit IRC16:54
*** akuznetsov has quit IRC16:56
*** akuznetsov has joined #tripleo16:58
*** eguz has quit IRC16:59
*** rainya has quit IRC17:00
*** akrivoka has quit IRC17:03
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Revert SSH virsh to use the new ManagementInterface  https://review.openstack.org/11035217:06
derekh_that test revert ^^ failed the ironic job, I think for an unrelated problem, kicked it off again and gotta go17:07
*** rainya has joined #tripleo17:07
*** rainya has quit IRC17:08
*** eghobo has joined #tripleo17:08
openstackgerritstephen-mulcahy proposed a change to openstack/tripleo-image-elements: Add new multipath element  https://review.openstack.org/11036317:08
*** rainya has joined #tripleo17:09
*** derekh_ has quit IRC17:10
*** gcha has quit IRC17:21
*** gcha has joined #tripleo17:21
SpamapSslagle: we missed something in the occ-local spec. What if dir1 and dir2 both have foo.json?17:21
lifelessSpamapS: in terms of the cache path?17:28
slagleSpamapS: i figured we'd have the full file path in the content dict17:28
slagleoh, is the cache the issue?17:28
SpamapSspec does not say full file path17:32
SpamapSand we'd have to url encode that17:33
SpamapSwhich I think is the answer17:33
SpamapSBut that will be oogly17:33
SpamapSc'est la vie17:33
*** dtantsur is now known as dtantsur|afk17:34
*** sballe has quit IRC17:36
*** ddieterly has quit IRC17:39
lifelessSpamapS: I'm wondering17:39
*** ddieterly has joined #tripleo17:39
lifelessSpamapS: if we moved the merge into o-c-c, we could cache the merge result not the inputs17:39
lifelessSpamapS: and avoid triggering unless the merged metadata differs17:40
*** jdob has quit IRC17:40
*** jdob1 has joined #tripleo17:40
*** jdob1 is now known as jdob17:40
SpamapSlifeless: certainly. That follows the patterns other agents use I think.17:41
*** ddieterly has quit IRC17:44
*** cody-somerville has joined #tripleo17:45
*** rushiagr has quit IRC17:56
*** jdob has quit IRC17:59
*** jdob has joined #tripleo17:59
lifelessI'm trying to remember why we made oac do the merge18:12
*** aldavud has joined #tripleo18:21
*** sdake has quit IRC18:33
*** sdake has joined #tripleo18:33
*** panda has quit IRC18:39
*** jcoufal has joined #tripleo18:39
openstackgerritJonathan Brownell proposed a change to openstack/tripleo-heat-templates: Move cacert property out of "stunnel" and into a new "ssl" group  https://review.openstack.org/11012118:40
*** panda has joined #tripleo18:40
lifelessStevenK: https://www.marriott.com/marriott/contact.mi 'folio request'18:42
*** aldavud is now known as aldavud_away18:51
*** gcha has quit IRC18:55
*** jistr has joined #tripleo18:56
*** pblaho_ is now known as pblaho18:56
*** gcha has joined #tripleo19:01
jp_at_hpIs the meeting now?19:07
*** gfidente has quit IRC19:08
*** ddieterly has joined #tripleo19:08
openstackgerritAdam Gandelman proposed a change to openstack/tripleo-specs: Decouple OS services from boot-stack element  https://review.openstack.org/10688519:09
*** gcha has quit IRC19:11
lxslijp_at_hp: yes, waiting for lifeless though19:11
*** gcha has joined #tripleo19:12
lxsliwe started19:13
*** gcha has quit IRC19:13
*** gcha has joined #tripleo19:14
openstackgerritClint 'SpamapS' Byrum proposed a change to openstack/os-collect-config: Add a local data collector  https://review.openstack.org/10943119:15
*** gfidente has joined #tripleo19:21
lifelessI thought the meeting was in 12 hours?19:22
openstackgerritClint 'SpamapS' Byrum proposed a change to openstack/os-collect-config: Add a local data collector  https://review.openstack.org/10943119:23
*** rainya has quit IRC19:28
*** rainya has joined #tripleo19:29
*** derekh_ has joined #tripleo19:30
openstackgerritA change was merged to openstack/tripleo-image-elements: Add unit test to enforce unique file names.  https://review.openstack.org/9470519:33
TheJuliaMy calendar says now, last week would have been the meeting that was 12 hours later.19:34
*** gcha has quit IRC19:34
*** gcha has joined #tripleo19:35
lxsligreghaynes: https://review.openstack.org/#/c/97283, think your comment has been addressed19:40
*** rakesh_hs2 has quit IRC19:44
*** gcha has quit IRC19:45
*** gcha has joined #tripleo19:45
*** openstackgerrit has quit IRC19:48
*** aldavud_away has quit IRC19:54
tchaypogood morning20:00
*** mikal_ is now known as mikal20:01
*** jp_at_hp has quit IRC20:03
cineramahi tchaypo20:03
*** eguz has joined #tripleo20:04
stevebakermorning20:05
stevebakerThe HOT port change is going to need some kind of coordination/flag-day since it requires frequent rebasing, and every exiting tht review will need changing after it lands20:06
stevebakerhttps://review.openstack.org/#/c/105328/20:06
*** jistr has quit IRC20:06
stevebakerevery *existing*20:06
tchaypowhee20:07
*** jmoffitt_ has joined #tripleo20:07
*** eghobo has quit IRC20:07
greghaynesstevebaker: in the past we sent a mail saying "everyone rebase t-h-t patches off this" and then -2'd everything until they did it20:08
greghaynesdepending on how far along the HOT patch is20:08
ekarlsolooking at https://review.openstack.org/#/c/102198/20:09
ekarlsois there a good way to preserve the permissions on /etc/hosts there ?20:09
stevebakergreghaynes: I think its ready, it just needs reviews, and CI in a good enough shape to validate it20:10
*** jmoffitt has quit IRC20:11
greghaynesekarlso: oh, the issue is that your writing out a new file and then mv'ing20:11
ekarlsoyes20:11
ekarlsomore optimal way ? :)20:11
*** derekh_ has quit IRC20:12
greghaynesekarlso: potential way - cp -a and then truncate20:12
greghaynesrather than make a new file20:12
greghaynespeople might have a better way though20:13
* ekarlso is open for comments20:13
tchaypoI was going to suggest cat20:13
tchaypocat $TEMPLFILE > /etc/hosts20:13
greghaynesah, or that20:14
tchaypoas long as $TEMPFILE is correctly based on the previous contents of /etc/hosts you should be fine...20:14
ekarlsotchaypo: wouldn't u need a cat before the awk then ? :)20:14
tchaypowell, actually, I think it has all the same failure cases as you'd get from creating a new file and then using mv, but it preserves the inode20:14
ekarlsoor nvm20:15
ekarlsohmmms20:15
greghaynesstevebaker: how about once we get them ready to +A we send out the mail?20:15
tchaypono I don't think so - just replace "mv $temp "$file"" with "cat $temp > $file"20:16
greghaynesI see one failed CI and they just need reviews, like you say20:16
ekarlsotchaypo: i'll try that and rebuild my images :)20:16
* greghaynes tries to review them now20:16
stevebakergreghaynes: Once https://review.openstack.org/#/c/105327/6 lands I'll do another rebase, then we can push for reviews/testing20:17
*** Penick has joined #tripleo20:17
greghaynesyes, that one failed CI for some reason20:17
stevebakerrechecking20:18
greghayneshuh, unrelated but syslog on that fail has error: Could not load host key: /etc/ssh/ssh_host_dsa_key20:19
greghaynesI wonder if we have bad perms setup20:19
*** rlandy has quit IRC20:22
lifelessslagle: fuller reply on the spec, hope its coherent20:23
lifelessslagle: (since I get to claim jetlag, I'm going to :))20:23
lifelessslagle: btw I love your draft spec review page20:23
*** rainya has quit IRC20:30
*** rainya has joined #tripleo20:31
*** rainya has quit IRC20:33
*** rainya has joined #tripleo20:34
*** dshulyak_ has quit IRC20:38
*** dshulyak_ has joined #tripleo20:41
*** lblanchard has quit IRC20:41
*** dguerri is now known as _dguerri20:45
*** _dguerri is now known as dguerri`afk20:45
*** pblaho has quit IRC20:46
*** bashok_ has joined #tripleo20:50
*** ccrouch1 has quit IRC20:52
*** bashok has quit IRC20:53
*** bashok__ has joined #tripleo20:54
slaglelifeless: alright, cool, thanks. i'll check it out. i emailed the list as well20:54
slaglei'm about to get pulled to a family portrait session, but maybe if you're around tomorrow, we could cat about multi hypervisor20:55
slaglebeen hacking on it a bit and i've got the overcloud coming up in lxc, but it can't get metadata from the undercloud20:56
slaglei don't think the PREROUTING rule we've added in iptables to redirect 169.254.169.254 traffic is taking effect20:56
slagleperhaps b/c the lxc instance has a namespaced network interface...not sure.20:56
*** jdob has quit IRC20:57
*** bashok_ has quit IRC20:57
*** openstackgerrit has joined #tripleo20:58
*** untriaged-bot has joined #tripleo21:01
untriaged-botUntriaged bugs so far:21:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784521:01
uvirtbotLaunchpad bug 1347845 in tripleo "Cannot set selinux file contexts when kernel does not have selinux support compiled in" [Undecided,Confirmed]21:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134987221:01
uvirtbotLaunchpad bug 1349872 in tripleo "setup-clienttools doesn't honour proxy config" [Undecided,New]21:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134751421:01
uvirtbotLaunchpad bug 1347514 in tripleo "Overcloud upgrade failing due to tftp issue on one of the overcloud nodes" [Undecided,New]21:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134784821:01
uvirtbotLaunchpad bug 1347848 in tripleo "/etc/hosts information about cloud hsots lost on reboot" [Undecided,In progress]21:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134598021:01
uvirtbotLaunchpad bug 1345980 in tripleo "Heat not able to create swift server" [Undecided,New]21:01
untriaged-bothttps://bugs.launchpad.net/tripleo/+bug/134823421:01
uvirtbotLaunchpad bug 1348234 in tripleo "In a multi-controller overcloud config, br-ex is assigned the same IP in all controller nodes" [Undecided,Confirmed]21:01
*** untriaged-bot has quit IRC21:01
*** jang has quit IRC21:01
openstackgerritTzu-Mainn Chen proposed a change to openstack/tuskar-ui: Added mock tuskar driver  https://review.openstack.org/11046621:02
*** dshulyak_ has quit IRC21:03
*** eguz has quit IRC21:03
*** ccrouch has joined #tripleo21:03
*** jang has joined #tripleo21:05
*** weshay has quit IRC21:07
lifelessslagle: could be - lets do that!21:09
*** eghobo has joined #tripleo21:10
*** eghobo has quit IRC21:15
*** dguerri`afk is now known as _dguerri21:15
*** jtomasek has quit IRC21:15
*** eghobo has joined #tripleo21:19
*** _dguerri is now known as dguerri`afk21:31
*** stevebaker is now known as steveburt21:41
*** steveburt is now known as stevebaker21:41
*** gfidente has quit IRC21:44
*** jang has quit IRC21:52
*** jang1 has joined #tripleo21:52
*** chuckC has joined #tripleo22:00
*** jang1 has quit IRC22:05
*** BadCub02 has quit IRC22:14
*** rainya has quit IRC22:22
*** Penick has quit IRC22:28
*** rainya has joined #tripleo22:28
*** rainya has quit IRC22:30
*** edmund has quit IRC22:34
openstackgerritChris Krelle proposed a change to openstack/diskimage-builder: Add ironic-python-agent element  https://review.openstack.org/11048722:36
openstackgerritJonathan Brownell proposed a change to openstack/tripleo-image-elements: stunnel: Allow CA certificate to be specified  https://review.openstack.org/11007222:38
StevenKNobodyCam: Why is that element in dib, when the other ironic elements are in tripleo-image-elements?22:38
lifelesshysterical raisins22:39
openstackgerritEndre Karlson proposed a change to openstack/tripleo-image-elements: Hosts element: Use 'markers' for updating hosts  https://review.openstack.org/10219822:39
lifelessStevenK: deploy-ironic is in dib22:39
lifelessI'd support ipa being in tripleo-image-elements22:39
NobodyCamok that is just an initial draft commiming from hte meetup22:42
NobodyCamI am happy to move22:42
lifelessplease22:42
lifelessmm22:42
lifelessactually no dib is fine22:42
lifelessbecause its not tripleo specific22:42
NobodyCamit is a deploy ramdisk22:43
cody-somervilleHey. Where would be the latest TripleO presentation content be found?22:44
lifelesscody-somerville: I'm not sure; I can dig up the latest deck given to me, if you like22:45
cody-somervillelifeless: That would be great, thanks.22:45
openstackgerritEndre Karlson proposed a change to openstack/tripleo-image-elements: Hosts element: Use 'markers' for updating hosts  https://review.openstack.org/10219822:45
ekarlsogreghaynes: wanna look at ^22:46
*** ccrouch has quit IRC22:46
greghaynescody-somerville: tchaypo was working on one recently22:47
*** eghobo has quit IRC22:50
openstackgerritJonathan Brownell proposed a change to openstack/tripleo-image-elements: stunnel: Allow CA certificate to be specified  https://review.openstack.org/11007222:50
*** eghobo has joined #tripleo22:51
*** ddieterly has quit IRC22:52
*** jp_at_hp has joined #tripleo22:53
*** ddieterly has joined #tripleo22:53
*** ddieterly has quit IRC22:54
*** derekh_ has joined #tripleo22:54
derekh_the temp revert of the ironic patch passed CI https://review.openstack.org/#/c/110352/22:55
openstackgerritDerek Higgins proposed a change to openstack-infra/tripleo-ci: Allow new virsh commands from ironic  https://review.openstack.org/11049222:55
*** bashok__ has quit IRC22:55
derekh_and here ^^ is the fix for the test envs22:55
derekh_but I'm not going to be able to rebuild test envs now, thoughts ?22:56
derekh_brb22:56
*** jcoufal has quit IRC22:58
*** ccrouch has joined #tripleo23:00
lifelessderekh_: o/23:03
jp_at_hplifeless: derekh_ I was about to +A the ironic revert.  any reason I shouldn't?23:09
tchaypocody-somerville: what kind of presentation/audience are you thinking of?23:09
jp_at_hpand I definitely want to put together something that reads the tripleo-ci reverts and applies them to a devtest run...23:10
derekh_jp_at_hp: lifeless: hi, so basically I think that commit will allow the new virsh commands through but it needs to get into the TE's23:12
derekh_jp_at_hp: lifeless in the meantime the revert should unblock things23:12
lifelesscool23:12
jp_at_hpderekh_: yeah, I was talking about approving the revert, but fair play to get the te config patch up too23:13
* derekh_ got nothing done today he intended to do :-(23:13
derekh_also its probably only a matter of time before the horizon revert hits a conflict, its a it commit to revert23:14
lifelesswendar: catch you another time :)23:15
derekh_and I don't think anybody is looking into it https://review.openstack.org/#/c/110258/23:15
*** yamahata_ has quit IRC23:15
*** rainya has joined #tripleo23:20
*** rainya has quit IRC23:22
*** rainya has joined #tripleo23:22
*** rainya has quit IRC23:27
*** lucasagomes has quit IRC23:27
openstackgerritA change was merged to openstack-infra/tripleo-ci: Revert SSH virsh to use the new ManagementInterface  https://review.openstack.org/11035223:27
*** rainya has joined #tripleo23:27
*** dkehn_ has joined #tripleo23:28
jp_at_hpthat'll teach me to get distracted before hitting submit on a web page...23:28
vinshtchaypo, With keep alived... https://review.openstack.org/#/c/104368/13 for example.  Are we relying only on highest ip address to decide on master?23:29
*** derekh_ changes topic to "Using OpenStack to deploy OpenStack;meetings Tuesday 1900//0700 UTC in #openstack-meeting-alt"23:31
*** dkehnx has quit IRC23:31
derekh_jp_at_hp: I was wondering where you went to23:32
TheJuliavinsh: that is what VRRP does by default, highest address wins when the priorities are the same23:36
vinshYep, just checking.23:36
lifelessvinsh: note that we don't manually determine highest address23:37
lifelessvinsh: it doesn't need to map to any particular machine23:37
vinsh*shouldn't have to23:37
lifelessvinsh: so far we've had no need to do that:)23:38
vinshI know internally over here.. some are hoping its usually controller0 or the bootstrap host23:38
lifelesswhy?23:38
greghaynesbecause failing over while doing db migrations sucks23:38
*** cwolferh has quit IRC23:38
vinshI'm trying to take stock of the situation still.23:39
lifelessgreghaynes: ah23:39
lifelessgreghaynes: there's a thing to synchronise iptables state to make that seamless, wonder if there's a similar haproxy thing23:40
greghaynesyea, conntrackd IIRC23:40
lifelessgreghaynes: I guess we want it to not fail over if a node is up, at all23:40
lifelessgreghaynes: is that an option?23:40
greghaynesusing keepalived, no23:40
lifelessnuts23:40
greghaynesunless you are explicit about priorities beforehand23:41
lifelesswell23:41
lifelessthat doesn't prevent it, does it?23:41
greghaynesno, but out bootstrap host would let us do that - we could ensure that bootstrap host always gets vip if its up23:41
greghayneswhich would prevent us failing over while doing initialization23:41
lifelessif for instance keepalived was glitching23:41
lifelesson the bootstrap host23:42
lifelessit would still happen23:42
greghaynesyeeep23:42
vinshya.. they all get MASTER and 100 or whatever23:42
vinshso IP is only tie breaker.23:42
vinshBut that should be fine.. and probably isn't the problem to solve.23:43
greghaynesmaybe we do db sync against localhost but tell applications to use VIP?23:43
lifelessas a compromise position that seems reasonable23:43
lifelessif the db on the bootstrap host is awol we'd fail23:43
*** edmund has joined #tripleo23:43
greghaynesyep23:43
*** cwolferh has joined #tripleo23:43
lifelessbut it would be more visible and easier to debug I suspect23:43
lifelessslagle: point me at your patch sets?23:44
*** derekh_ has quit IRC23:46
TheJuliagreghaynes: we briefly talked about that last week, and as I understood it then, that would be secondary config files that the db sync stuff would be pointed to which contain localhost.  Is that right?23:50
* greghaynes waves hands furiously23:51
greghaynesactual implementation wise... hadnt thought totally through. Either that or sed23:51
TheJuliaor an os-collect-config local datasource at some point23:52
TheJulia:)23:52
* TheJulia is having deja vu23:52
greghaynesyeaa, that could be an optoin23:53
*** dkehn_ is now known as dkehnx23:53
openstackgerritJames Polley proposed a change to openstack/tripleo-incubator: Re-iterated the switch from baremetal to Ironic  https://review.openstack.org/9329923:53
openstackgerritJim Rollenhagen proposed a change to openstack/diskimage-builder: Add ironic-python-agent element  https://review.openstack.org/11048723:58
*** edmund has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!