Wednesday, 2015-08-19

*** alop has quit IRC00:02
*** shoutm has quit IRC00:36
*** shoutm has joined #openstack-ansible00:37
*** woodard has joined #openstack-ansible00:41
*** woodard has quit IRC00:41
*** woodard has joined #openstack-ansible00:42
*** woodard has quit IRC00:55
*** darrenc is now known as darrenc_afk01:07
*** woodard has joined #openstack-ansible01:08
*** woodard has quit IRC01:12
*** darrenc_afk is now known as darrenc01:21
*** britthouser has joined #openstack-ansible01:48
*** britthou_ has joined #openstack-ansible01:50
*** britthouser has quit IRC01:53
*** mrodden has quit IRC03:27
*** raginbajin has quit IRC03:27
*** raginbajin has joined #openstack-ansible03:29
*** logan2 has quit IRC03:31
*** mrodden has joined #openstack-ansible03:32
*** shoutm has quit IRC03:45
*** shoutm has joined #openstack-ansible03:56
*** JRobinson__ is now known as JRobinson__afk03:59
*** tlian2 has quit IRC04:05
*** raginbajin has quit IRC04:10
*** raginbajin has joined #openstack-ansible04:28
*** bgmccollum has quit IRC04:28
*** sdake has quit IRC04:29
*** bgmccollum has joined #openstack-ansible04:33
*** JRobinson__afk has quit IRC04:33
*** JRobinson__afk has joined #openstack-ansible04:42
*** JRobinson__afk is now known as JRobinson__04:43
*** ashishb has joined #openstack-ansible05:20
*** shausy has joined #openstack-ansible05:23
*** shausy has quit IRC05:36
*** shausy has joined #openstack-ansible05:36
*** britthou_ has quit IRC06:21
*** britthouser has joined #openstack-ansible06:22
*** jlvillal has quit IRC06:58
*** jlvillal has joined #openstack-ansible06:58
*** fawadkhaliq has joined #openstack-ansible07:23
*** JRobinson__ has quit IRC07:26
*** gparaskevas has joined #openstack-ansible07:35
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Updated juno for new dev work - 15 Aug 2015  https://review.openstack.org/21344007:38
odyssey4mehughsaunders ^ that is updated based on what successfully built - the updated sha for ceilometerclient was causing build issues - we don't need to specify a sha as it gets pulled in through dependencies anyway, and the requirements from heat include the min/max versions to make the go07:39
hughsaundersodyssey4me: ok07:40
odyssey4meandymccr are you going to look into the swift logging issue this morning?07:42
odyssey4meI'll take a poke at https://bugs.launchpad.net/openstack-ansible/+bug/148613707:42
openstackLaunchpad bug 1486137 in openstack-ansible "cinder-volume log missing" [Medium,New]07:42
odyssey4meand by swift log issue I mean https://bugs.launchpad.net/openstack-ansible/+bug/148613307:42
openstackLaunchpad bug 1486133 in openstack-ansible "zero sized swift logs" [Low,New]07:42
andymccrumm yeh i can try take a look07:43
andymccron my current install (3 volumes containers) only 1 has logs07:44
odyssey4meandymccr cinder logs, or swift logs?07:45
andymccrcinder07:45
andymccri dont have a swift install up (im doing ceph lately)07:45
odyssey4meah, wel if you're looking into that then I'll hold back07:46
odyssey4meI'll dig into the swift logs then :)07:46
andymccrok fair trade07:46
andymccrive got it installed already so might be easiest. swift i'll have to spin up a new one :D07:46
odyssey4meright - I'll assign the cinder log one to you and the swift log one to me07:47
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Keystone SSL cert/key distribution and configuration  https://review.openstack.org/19447407:51
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Enable log_slave_updates  https://review.openstack.org/21346707:51
odyssey4meinterestingly, it's only on hpcloud that we end up reaching our build timeout and that we have master branch build issues07:53
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Removes trailing whitespace for bashate  https://review.openstack.org/20766307:53
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Add support for additional nova.conf options  https://review.openstack.org/21049207:53
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Set iptables-persistent install execution to append to log  https://review.openstack.org/21417207:54
hughsaundersgit-harry: how wip is this? Should I review it? https://review.openstack.org/#/c/212079/07:56
git-harryhughsaunders: I think the role is pretty much there. There is a dependent review that is failing, I haven't had time to look at why but I suspect it's most likely due to the lack of discrimination in applying conf options as opposed to an issue with the role. So assuming that is a simple fix the role review shouldn't change.07:59
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Enable/disable Swift/OpenStack deployment properly  https://review.openstack.org/21421307:59
git-harryhughsaunders: in other words, yes please review it.08:00
hughsaundersok :)08:00
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Enable/disable Swift/OpenStack deployment properly  https://review.openstack.org/21421308:04
odyssey4meyeah, the build fails there are very likely to be the common build fails we're seeing from any hpcloud nodes in nodepool08:06
hughsaundersgit-harry: swift_conf__swift__hyphen__hash__swift_hash_path_prefix = "changeme" <-- thats hilarious, but I can see why its necessary08:07
*** gparaskevas_ has joined #openstack-ansible08:08
git-harryhughsaunders: yeah, it looks a bit crazy on first sight but it is required and everything lines up nicely in a yaml file08:09
*** gparaskevas has quit IRC08:09
*** markvoelker has quit IRC08:22
odyssey4memattt would you mind taking a quick look through the kilo backports in the review queue?08:28
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Enable/disable Swift/OpenStack deployment properly  https://review.openstack.org/21421308:31
matttodyssey4me: sure08:33
odyssey4methanks08:35
*** fawadkhaliq has quit IRC08:36
*** shoutm has quit IRC08:40
*** markvoelker has joined #openstack-ansible09:23
*** fawadkhaliq has joined #openstack-ansible09:28
*** markvoelker has quit IRC09:28
odyssey4meFYI all:  https://review.openstack.org/21453109:37
*** jaypipes has quit IRC09:46
*** cloudnull_zzz has quit IRC10:05
*** dolphm has quit IRC10:05
*** bogeyon18 has quit IRC10:06
*** sigmavirus24_awa has quit IRC10:06
*** eglute has quit IRC10:06
*** miguelgrinberg has quit IRC10:07
*** d34dh0r53 has quit IRC10:07
*** palendae has quit IRC10:07
*** miguelgrinberg has joined #openstack-ansible10:08
*** eglute has joined #openstack-ansible10:08
*** bogeyon18 has joined #openstack-ansible10:08
*** sigmavirus24_awa has joined #openstack-ansible10:08
*** palendae has joined #openstack-ansible10:08
*** d34dh0r53 has joined #openstack-ansible10:08
*** cloudnull has joined #openstack-ansible10:08
*** dolphm has joined #openstack-ansible10:09
*** Guest62465 has quit IRC10:12
*** spotz_zzz has quit IRC10:12
*** blewis has joined #openstack-ansible10:14
*** spotz_zzz has joined #openstack-ansible10:14
*** blewis is now known as Guest777010:14
andymccrodyssey4me: i did a test install. cinder-volumes log isnt empty10:14
odyssey4meandymccr odd10:15
*** shoutm has joined #openstack-ansible10:18
*** javeriak has joined #openstack-ansible10:40
*** fawadkhaliq has quit IRC10:53
openstackgerritMatt Thompson proposed stackforge/os-ansible-deployment: Trigger restart after adding user to cephkeys  https://review.openstack.org/21455711:00
*** shoutm has quit IRC11:01
*** britthouser has quit IRC11:10
*** markvoelker has joined #openstack-ansible11:24
*** shoutm has joined #openstack-ansible11:26
*** markvoelker has quit IRC11:28
*** javeriak has quit IRC11:31
*** fawadkhaliq has joined #openstack-ansible11:35
*** javeriak has joined #openstack-ansible11:37
*** javeriak has quit IRC11:38
evrardjpgood afternoon everyone11:46
evrardjpor good morning, depending on your timezone11:46
*** jaypipes has joined #openstack-ansible11:53
openstackgerritMerged stackforge/os-ansible-deployment: Replace ADFS example DNS name with something appropriate  https://review.openstack.org/21401211:55
openstackgerritMerged stackforge/os-ansible-deployment: Adds a pep8 target to tox.ini  https://review.openstack.org/21401311:55
openstackgerritMerged stackforge/os-ansible-deployment: Enable log_slave_updates  https://review.openstack.org/21346711:55
openstackgerritMerged stackforge/os-ansible-deployment: Update rabbitmq version deployed to v3.5.4  https://review.openstack.org/21051611:56
matttevrardjp: did you figure out the nova issue yesterday?11:57
evrardjpyup11:57
evrardjprouting issue11:57
evrardjpnot really clear in the logs11:57
evrardjpeven with debug ;)11:57
openstackgerritMerged stackforge/os-ansible-deployment: Keystone SSL cert/key distribution and configuration  https://review.openstack.org/19447411:57
matttrouting issue between nova-compute and mon ?11:57
odyssey4meandymccr ok, two builds on rax - both have populated swift logs and cinder-volume logs11:58
evrardjpnope, between nova-compute and backend node11:58
odyssey4meso we're back to comparing builds in openstack-infra instead11:58
evrardjpbut it doesn't change a thing11:58
evrardjpit's fixed now11:58
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Update rabbitmq version deployed to v3.5.4  https://review.openstack.org/21457411:59
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Keystone SSL cert/key distribution and configuration  https://review.openstack.org/21457511:59
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Enable log_slave_updates  https://review.openstack.org/21457611:59
openstackgerritMerged stackforge/os-ansible-deployment: Correct binding logic in haproxy configuration  https://review.openstack.org/21432812:00
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Removes trailing whitespace for bashate  https://review.openstack.org/20766312:01
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Set iptables-persistent install execution to append to log  https://review.openstack.org/21417212:03
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Update the documented ceph user variables  https://review.openstack.org/20913012:04
*** b3rnard0 is now known as b3rnard0_away12:05
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Update documentation for multiple VLAN ranges  https://review.openstack.org/20959812:06
*** woodard has joined #openstack-ansible12:11
openstackgerritMerged stackforge/os-ansible-deployment: Remove unused variables in os_swift role  https://review.openstack.org/21432612:11
*** KLevenstein has joined #openstack-ansible12:12
evrardjpI'll handle this:12:15
evrardjphttps://bugs.launchpad.net/openstack-ansible/+bug/148425612:15
openstackLaunchpad bug 1484256 in openstack-ansible trunk "Apache servers reporting version in response header" [Low,Confirmed]12:15
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Disable using lvm-backed cinder on second partition for AIO's  https://review.openstack.org/21458212:16
odyssey4meevrardjp no need to announce it - just assign the bug to yourself. :)12:17
odyssey4methanks!12:17
evrardjpjust ensuring nobody took care of it12:18
odyssey4mefyi - as we still haven't got to the bottom of the issue in master, I've proposed https://review.openstack.org/214582 to work around it for now to unblock master merges12:18
odyssey4memattt andymccr hughsaunders ^12:18
*** woodard has quit IRC12:20
*** benwh4 has joined #openstack-ansible12:21
benwh4hello12:21
matttodyssey4me: is it always b4 or all hp builds ?12:21
odyssey4memattt all hp builds12:22
odyssey4mewe've been failing builds on all hp nodes since last week12:22
hughsaundersahh only hp. I was wondering why that code would suddenly be an issue as it hasn't changed since feb12:22
matttodyssey4me: shall we spin up an HP account and try it?12:22
matttthey give $300 credit so it shouldn't cost us anything12:23
odyssey4memattt afaik cloudnull has one, so I'd like to unblock master for now - then put in a review to revert and recheck it to gather data while also testing in hpcloud through cloudnull's acct12:23
matttodyssey4me: yeah i'm fine with that, i'll review once it gates12:23
odyssey4meyup, agreed12:24
*** ashishb has quit IRC12:24
odyssey4meI'd like to see it gate on an hpcloud node.12:24
*** woodard has joined #openstack-ansible12:24
mattti think we better recheck a few times before we mege12:24
mattt*merge12:24
matttto be sure12:24
odyssey4memattt just one working build on an hpcloud node is enough to confirm - we've had nothing work on them for over a week12:25
odyssey4meit's a miracle we've merged anything into master12:25
mattthehe12:25
mattti suppose yeah12:25
odyssey4mebenwh4 o/12:25
mgariepygood morning everyone12:26
odyssey4meo/ mgariepy12:26
*** britthouser has joined #openstack-ansible12:28
odyssey4memgariepy how's your testing going?12:28
*** KLevenstein has quit IRC12:29
benwh4quick question when using the script inventory-manage.py --remove does it remove the container or just in the inventory ?12:30
odyssey4mebenwh4 just the inventory item12:30
benwh4so what is the way to remove the container ?12:31
odyssey4meto remove the container, just execute: lxc-stop <container-name>; lxc-destroy <container-name>12:31
benwh4ok thx12:31
mgariepyodyssey4me, slowly.12:32
mgariepyhaha12:32
mgariepyi found some ssds, i'll put them in the server, it should help speeding up things a bit.12:33
*** logan2 has joined #openstack-ansible12:39
*** markvoelker has joined #openstack-ansible12:40
*** ashishb has joined #openstack-ansible12:40
*** fawadkhaliq has quit IRC12:46
mgariepy does all openstack project use hpcloud to build-test stuff ?12:52
hughsaundersmgariepy: the infra team get instances from HP, Rackspace and possibly others to run tests on12:55
*** tlian has joined #openstack-ansible12:57
mgariepyok12:58
evrardjpmgariepy: nice to have ssds, I'm stuck with old 5400RPMs for my testing13:01
evrardjpI think when my production openstack will be there, I'll use the openstack for my testing purposes13:02
mgariepyi have a small cloud 36 compute nodes but i want to use physical hardware to test osad so I can install this on the production nodes.13:03
evrardjpthat's what I avoid, you'll never know :p13:04
mgariepymy current cloud is on mirantis fuel.13:05
mgariepythe installer is nice13:06
mgariepybut that's pretty much it.13:06
evrardjpI went through that also ;)13:07
evrardjpyou've made the right move, I think13:08
evrardjp(for my experience and for my usage, at least, it has been the right move)13:08
mgariepyat least with OSAD playbook/roles are easy to understand, and also easy to push patch13:10
mgariepyunless hpcloud fail all build haha13:10
evrardjp:D13:12
evrardjpit's a sensitive topic I think13:12
evrardjpI'd not joke about it ;)13:12
mgariepyhow long have you been playing with openstack ?13:12
*** b3rnard0_away is now known as b3rnard013:14
odyssey4memgariepy :) not so sensitive that I can't see the humour in it :p13:15
openstackgerritJean-Philippe Evrard proposed stackforge/os-ansible-deployment: Apache servers not reporting version in response header anymore  https://review.openstack.org/21460613:15
evrardjpdamn it won't pass build time!13:15
evrardjp;)13:16
mgariepy ssd installation time for my (old crappy server without hotswapable bay :( )13:17
odyssey4meevrardjp nice patch, thanks :)13:20
odyssey4meone thing though - the bug is naughty, so I think that perhaps we should split it into two bugs - it indicates that the server version and the server name is exposed13:21
odyssey4meyour patch covers the server version, but not the name which requires setting 'ServerSignature Off' - do you want to add that into the existing patch, or perhaps a follow on patch?13:21
evrardjpI thought to create a patch with other security hardening13:22
evrardjplike disabling stats13:22
evrardjpstats module13:22
evrardjpbut I'll amend my current commit with ServerSignature13:22
evrardjpI've tested in kilo branch, but not in master because master doesn't build right now here13:23
evrardjp(where I am testing)13:23
evrardjpI don't know how to backport it in kilo through the interface, though13:23
evrardjp(gerrit I meant)13:24
odyssey4meevrardjp good plan - when you see something that needs to be done and you think to yourself 'I should do that' or 'I wish someone would do that'... then please log a bug - that way you can pick it up when you have a gap, but someone else might do so before you13:24
evrardjpIt's just a question of timing, I just saw it at the same time as I was discovering this13:24
evrardjpI meant, a few minutes ago13:25
odyssey4meevrardjp no worries, let me show you through another patch when one merges that needs backporting13:25
evrardjpit's not really a bug13:25
evrardjpIt's a security enhancement I'd say13:25
evrardjpok13:25
odyssey4meevrardjp that sort of thing can go into a 'wishlist' bug, which is like a small enhancement that doesn't warrant a blueprint13:25
evrardjpok13:25
evrardjpif feature requests are considered as bugs, then bug it is!13:26
odyssey4mehaha :)13:26
gparaskevas_Hello again!13:30
odyssey4megparaskevas_ hello!13:31
gparaskevas_odyssey4me: just saw your tag and i cant tell you how much i like fruits! I had problem with the git review if you rememeber13:31
gparaskevas_odyssey4me: i am trying to recommit now13:32
odyssey4megparaskevas_ yup, did you manage to get git working?13:32
*** prad_ has joined #openstack-ansible13:32
gparaskevas_odyssey4me: i will try now13:32
odyssey4meawesome :)13:32
openstackgerritJean-Philippe Evrard proposed stackforge/os-ansible-deployment: Apache servers not reporting version in response header anymore  https://review.openstack.org/21460613:32
*** mgariepy has quit IRC13:37
*** mgariepy has joined #openstack-ansible13:37
odyssey4meevrardjp fyi, we don't backport a patch until the master patch has merged - the reason is that prior to the master merge the patch may go through several change sets as people test it, comment on it and improve it13:39
evrardjpI remember that discussion and decision13:39
odyssey4mewell, generally we don't - there are exceptions when necessary, but we try to keep those as exceptions :)13:39
gparaskevas_odyssey4me: i will redo the changes as they are minor from a clean source! Should i clone master then do the changes  and then git review?13:40
evrardjpI don't see an issue with it13:40
odyssey4megparaskevas_ yes, that's the best way - ideally clone master, git checkout -b bug/<bugnumber>; do the edits then commit and review13:41
odyssey4mehughsaunders with regards to https://review.openstack.org/209568 you're suggesting that tempest default to testing Ceilometer unless it's changed?13:42
gparaskevas_odyssey4me: awesome13:44
*** woodard has quit IRC13:47
evrardjpodyssey4me: is it possible to add this link on the launchpad? https://wiki.openstack.org/wiki/Meetings/openstack-ansible13:50
evrardjpit's really hard to fine for ppl not on irc13:50
evrardjpnot always*13:50
odyssey4meevrardjp good idea - I'll organise that when the people who can are online13:51
evrardjpyou mean cloudnull, right? ;)13:51
odyssey4meevrardjp yeah, he's one of the people who can - I don't seem to have the rights13:53
odyssey4meoh great, linuxbridge testing is finally going voting in the upstream gate https://review.openstack.org/205674 \o/13:53
odyssey4meApsu rackertom ^13:53
*** prad_ is now known as pradk13:56
*** sdake has joined #openstack-ansible13:56
odyssey4mecloudnull has just gotten home from holiday - I doubt he'll be in today... even if he is, he has a lot of catching up to do13:57
*** sdake_ has joined #openstack-ansible13:58
*** woodard has joined #openstack-ansible13:58
openstackgerritJean-Philippe Evrard proposed stackforge/os-ansible-deployment: Apache servers will not reporting version anymore  https://review.openstack.org/21460613:59
evrardjpwhat's this voting odyssey4me?13:59
evrardjphow long was cloudnull gone? Like 2 weeks? It's not that much in terms of days, but it's a lot in terms of work done here :p14:00
*** javeriak has joined #openstack-ansible14:01
*** sdake has quit IRC14:01
odyssey4meevrardjp https://review.openstack.org/205674 is the upstream openstack-infra gate check change to make all neutron code be tested using linuxbridge... linuxbridge is what we use in openstack-ansible, so it's important to us14:02
evrardjpyeah I know that's what we are using! ;)14:03
odyssey4meup until now there has not been gating for linuxbridge, so there have been functions of neutron which have not worked for us14:03
evrardjpThere was indeed a big focus lately on ovs14:03
evrardjpneutron team was almost speaking only of that in the last summit14:03
evrardjpgood to know there is some kind of voting on this14:04
odyssey4meyeah, ovs makes sense to people who can afford the time to get to know networking really well - but most of us can't deep dive into networking enough to learn that whole new discipline14:04
odyssey4melinuxbridge is networking that almost every systems engineer understands, and it's been around for ages - so it's very stable14:05
*** javeriak has quit IRC14:05
odyssey4meovs had the bad habit of causing kernel panics when I was operating an OpenStack environment with Neutron/OVS14:05
odyssey4meevrardjp the entire openstack does all its voting and governance in the open, it's quite amazing14:06
*** Mudpuppy has joined #openstack-ansible14:06
*** Mudpuppy has quit IRC14:07
*** Mudpuppy has joined #openstack-ansible14:08
gparaskevas_odyssey4me: what shoud i include in the commit message for backporting to kilo?14:09
*** shoutm has quit IRC14:10
odyssey4megparaskevas_ backports aren't handled until master has committed, and there's no need to include anything in the commit message related to it14:10
matttis there any point rechecking reviews until we have a stable hp build?14:10
*** yaya has joined #openstack-ansible14:11
*** sdake_ is now known as sdake14:13
*** spotz_zzz is now known as spotz14:13
*** gparaskevas_ has quit IRC14:13
*** gparaskevas has joined #openstack-ansible14:15
openstackgerritMerged stackforge/os-ansible-deployment: Update rabbitmq version deployed to v3.5.4  https://review.openstack.org/21457414:16
odyssey4memattt nope, it actually makes it harder for me to get the unblocker in quickly as I have to wait for more nodes to become available14:16
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Enable log_slave_updates  https://review.openstack.org/21457614:17
palendaeodyssey4me: Out of curiosity, have you bumped the cinder SHA?14:17
palendaeMaybe they have a bug that's surfacing between the 2 AIO setups?14:18
*** Mudpuppy has quit IRC14:18
odyssey4mepalendae we were failing with liberty-2's sha's, so we bumped to 15 Aug: https://review.openstack.org/21291914:18
*** Mudpuppy has joined #openstack-ansible14:18
palendaeAh14:19
odyssey4meright through both periods the failure rate has remained the same14:19
odyssey4meso I'm resorting to https://review.openstack.org/21458214:19
odyssey4mehopefully it'll work, when I get a single proper cloud vm for it to execute all the way on14:19
gparaskevasodyssey4me: can you take a look at my erros? -> http://paste.openstack.org/show/421883/14:20
odyssey4megparaskevas what version of review is that? as I recall you needed a higher version to support https14:21
*** sigmavirus24_awa is now known as sigmavirus2414:22
odyssey4mebgmccollum I think you mentioned the version needed?14:22
*** jwagner_away is now known as jwagner14:23
evrardjpodyssey4me: for the haproxy rewrite, I'm not against being fully independant of openstack deployment for haproxy role, but at some point I have to put some variables that depend on the environement14:25
evrardjpshould I put that under user_variables or in the default of the role?14:25
evrardjplike atm, I have haproxy_bindable_test_address: "{{ external_lb_vip_address }}" in user_variables, because this way I can abstract the external_lb_vip_address of the haproxy role14:26
gparaskevasodyssey4me: git-review-1.24-5.fc22.noarch14:26
odyssey4megparaskevas I don't think that's a high enough version to support https :/14:27
gparaskevasshould i change to ssh?14:27
odyssey4meevrardjp perhaps use http://docs.ansible.com/ansible/playbooks_variables.html#variable-file-separation vars files ?14:28
odyssey4megparaskevas if you can, then that'll work assuming your proxy allows access via ssh to :29418 which is where you got stuck last time14:28
gparaskevasok that may a problem thoug because our proxy engineer is not here so i will give it a try from home!14:29
odyssey4meevrardjp otherwise set a default in the role, then an override can be used in user_variables for the environment14:29
gparaskevasodyssey4me: i hope i submit today! thanks again14:30
odyssey4megparaskevas looking forward to it!14:31
odyssey4memattt odd, https://review.openstack.org/214582 didn't work in hpcloud - so the issue appears to be unrelated to that too14:34
odyssey4meperhaps it does come down to packages installed or something like that14:35
matttodyssey4me: i'm going to try an instance on HP14:36
openstackgerritSerge van Ginderachter proposed stackforge/os-ansible-deployment: Allow Horizon setup with external SSL termination  https://review.openstack.org/21464714:37
matttor not, says i'll get a verificationc all within the next day :P14:40
mattt*call14:40
*** sdake_ has joined #openstack-ansible14:41
*** jwagner is now known as jwagner_away14:43
*** sdake has quit IRC14:44
svgHey all, I still have a first version of my ceph patch in the juno tree: https://review.openstack.org/#/c/181022/ But I won't be able to work on that any more. Given the majo differences in OSAD between juno and kilo, it would remain a totally different patch than the one in kilo and master.14:44
svgAnyone opposed to me abonding that?14:44
*** benwh4 has quit IRC14:46
odyssey4mesvg I think you can safely abandon it - I don't think we're going to do a feature release to juno for just that reason...14:46
andymccrsvg: makes sense to me to abandon it for juno14:47
odyssey4mebackports to juno are hard due to the big differences14:47
andymccrplus id rather we encourage everybody to use kilo++14:47
odyssey4me++14:47
openstackgerritSerge van Ginderachter proposed stackforge/os-ansible-deployment: Enable Neutron lbaas-haproxy  https://review.openstack.org/20851914:49
svgexactly my thoughts14:49
*** woodard has quit IRC14:51
*** woodard has joined #openstack-ansible14:55
*** yapeng has joined #openstack-ansible14:58
*** gparaskevas has quit IRC14:59
*** yapeng has left #openstack-ansible14:59
*** javeriak has joined #openstack-ansible15:00
*** logan2 has quit IRC15:01
evrardjpsvg: for your commit for ssl termination, it's terminating in horizon instead of your reverse_proxy/load balancer, right?15:03
*** jwagner_away is now known as jwagner15:04
svgevrardjp: not sure I understand15:04
svgApache does SSL now. My patch allows to disable that en setup things to do ssl on rp/lb15:04
svgssl termination that is15:04
sigmavirus24odyssey4me: andymccr ping15:04
odyssey4mesigmavirus24 pong15:04
andymccrhello15:04
sigmavirus24so do we have any further info about the cinder-volume log hijinks?15:05
sigmavirus24I think the last I checked, we marked our bug as incomplete15:05
*** logan2 has joined #openstack-ansible15:05
andymccrsigmavirus24: we've both tested it and it works fine - i cant recreate that issue at all15:05
sigmavirus24I can't either15:05
sigmavirus24But it's something reproducible on hpcloud it seems15:06
odyssey4mesigmavirus24 yeah :/15:06
sigmavirus24And it seems to only have started after our last SHA update, yes?15:06
odyssey4meto I've confirmed with -infra that hpcloud is known to be slower15:06
odyssey4meI expect that perhaps we have higher contention there15:06
odyssey4mesigmavirus24 nope, the liberty-2 sha update was failing all the time too15:06
*** jwagner is now known as jwagner_away15:06
sigmavirus24on this same issue?15:06
sigmavirus24So is this just a "hpcloud is always the problem" thing then?15:07
sigmavirus24And "recheck until we get a RAX server"?15:07
mattti'm buildig on an hp instance now15:08
matttif anyone wants to have a peek15:08
*** woodard has quit IRC15:08
andymccrthe cinder-volumes logs dont cause gate failures do they?15:08
sigmavirus24andymccr: no but cinder-volume being fubar'd does15:09
andymccrsigmavirus24: sure but then thats a separate issue no?15:09
sigmavirus24and no logs means we can't figure out why it's fubar'd15:09
sigmavirus24sure15:09
sigmavirus24I just can't help but wonder if they're related in some sense15:09
andymccri mean if cinder-volumes is broken on hpcloud installs, then its not that surprising the logs are also broken15:09
sigmavirus24mattt: can you throw my keys on there when it's done?15:09
sigmavirus24andymccr: right15:09
*** woodard has joined #openstack-ansible15:09
sigmavirus24but the thing is that the ansible logs don't indicate that cinder-volumes is broken itself15:09
sigmavirus24the service starts/restarts just fine15:09
sigmavirus24the problems we see is during the volume basic tempest tests15:10
andymccrok but its failing some tempest tests im assuming?15:10
sigmavirus24bingo15:10
andymccrsure so its obviously an issue that needs to be fixed but i think digging into why the logging isnt working is probably not the best approach.15:10
andymccrif i were a betting man id suggest that fixing cinder-volumes on hpcloud will also fix the logging :)15:11
sigmavirus24Right that's why I'm interested in the bug too :D15:11
odyssey4meok, we have some options15:13
odyssey4methanks to -infra15:13
odyssey4mewe should probably try them all, because any one of them could give us a better clue15:14
odyssey4meso rax cloud images for devstack-trusty are based on a snapshot, whereas hpcloud's are based on disk-image-builder - so the images are definitely different15:14
odyssey4meone suggestion is to switch the image we use to the 'ubuntu-trusty' image which is built using d-i-b for both providers, so it's consistent - I think it's worth submitting a patch to add a check on those images as a non-voting check to see how it goes15:15
matttwhy are we using a devstack image in the first place?  didn't realise we had a choice15:16
odyssey4mealso, hpcloud has lower performance and it's likely that the resource constraints we're putting the instance under is too much for an instance which is contending on its host or on crappy hardware - whatever - so the option there is to try lowering our resource usage15:17
sigmavirus24mattt: we don't quite have a choice15:17
odyssey4memattt we originally used the 'bare-trusty' image, which we were advised was not very bare and that the devstack image was a better choice15:17
odyssey4mesigmavirus24 we do, but from a catalog of existing images15:17
odyssey4meso it's worth putting in a patch which kills some of the extra containers to see how that plays out - if it does, then we know that it's a resource issue and we can deal with it15:18
matttyeah i'm not sure we need 2x keystone, 2x repo, etc.15:19
mattti'm guessing ceilometer/mongo aren't helping either15:19
sigmavirus24yeah, separate AIO jobs would be ... nice15:19
odyssey4meanother thing we can do is optimise how we use -infra to speed up the builds, so we don't get as many timeouts... this means using their git cache, pip cache, apt cache and cirros cached image15:19
palendae^15:19
sigmavirus24I think we're pushing too much into one15:19
palendaeodyssey4me: How does the pip cache interact with our repo builder?15:19
palendaeI'm all for those things, just wanna know15:20
evrardjpodyssey4me: the last one seems dangerous15:20
evrardjpcaching is gonna cause pain15:20
evrardjpif you can't manage it yourself15:20
evrardjpno?15:20
odyssey4meagreed, but we have yet to have the time to step back and re-look at our gating strategy and figure out a betterway, then do it15:20
palendae^15:21
evrardjpthe same image for both provider seems good15:21
odyssey4meevrardjp the caches are used for all openstack gate checks, so when something goes wrong hordes of people look at the problem15:21
evrardjpok I understand the value now15:21
*** ashishb has quit IRC15:22
odyssey4mepalendae to answer your question - I would guess that the repo container would need to bind mount the cache directory on the host, then use that cache as part of its downloads15:22
evrardjpnot the topic, but has anyone tried to run haproxy in a container?15:22
*** daneyon has joined #openstack-ansible15:22
odyssey4meessentially it just cuts down the reaching out we need to do, so it speeds up the build15:22
evrardjpI was wondering why it wasn't defined with the env.d as the others15:22
evrardjplike*15:22
odyssey4mewe're not going to be able to do everything, but it is something we should try to do15:23
evrardjpspeed up the building process is good15:23
evrardjpreducing test coverage isn't15:23
mgariepyis the wresp problem normal in : http://logs.openstack.org/70/213170/4/check/gate-os-ansible-deployment-dsvm-commit/a93720c/logs/aio1_glance_container-f7ef29f9/glance-registry.log @2015-08-18 05:20:34.26415:23
evrardjpso we need to pay attention15:23
odyssey4meevrardjp haproxy in the project has largely just been a test/dev tool, so very little focus is placed on it15:23
evrardjpk15:23
*** woodard has quit IRC15:24
odyssey4memgariepy normal? no idea - you'd have to inspect across several job results to see15:25
mgariepyand does someone have a script to download all the build log for a failed build?15:25
odyssey4menot to my knowledge15:25
odyssey4meit would be useful though15:25
mgariepyjust to use grep on the logs.. lol15:25
*** daneyon_ has quit IRC15:26
odyssey4memattt the reason for 2 x keystone is to confirm the code paths related to pushing keys around and other things that only happen when there's more than one container; the same for horizon afaik, for the repo it's about the repo sync, etc15:27
*** woodard has joined #openstack-ansible15:27
odyssey4mebut that said, I think that we can do gate checks differently15:27
odyssey4mean AIO can be a standard build with bells and whistles, the gate checks can customise the build to be leaner and more focused on testing specific things15:28
*** phalmos has joined #openstack-ansible15:31
odyssey4mehughsaunders you around?15:31
hughsaundersodyssey4me: only just15:31
odyssey4mehughsaunders you added https://github.com/stackforge/os-ansible-deployment/blob/master/scripts/bootstrap-aio.sh#L277-L28715:31
odyssey4mehow does it work?15:31
odyssey4meah, never mind - the commit message tells me :)15:33
hughsaunders:)15:34
*** britthouser has quit IRC15:35
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Reduce AIO affinities  https://review.openstack.org/21467615:41
odyssey4mesigmavirus24 mattt andymccr palendae let's see how that goes as a stop-gap15:42
matttstill think it's unrelated15:42
odyssey4meas part of the liberty preparation we'll have to ensure that we change the gate checks to provide more coverage and use multiple nodes15:42
matttat least regarding cinder15:42
odyssey4meright, next thing to try is going to be to normalise the image15:42
matttbut that may be a necessary change nayway15:42
odyssey4memattt I am inclined to suspect that there's a package either missing, or is older/newer that's causing an issue15:43
odyssey4mesomething already in the image15:43
matttpossible15:44
matttas an aside, should we not be doing DEBIAN_FRONTEND=noninteractive in all our gate/bootstrap scripts?15:44
matttwhen i ran gate-check-commit.sh on this HP node it popped up a dialog after installing a kernel15:45
odyssey4meprobably, although we haven't had packages ask for interaction15:45
palendaeI think we have15:45
odyssey4mewe should probably just export that up front15:45
palendaewell, I have15:45
palendaeWhen I apt-get upgrade manually15:45
palendaebut that's on RAX public cloud specifically15:45
palendaeNot sure if the infra images are the same15:45
*** sdake_ is now known as sdake15:49
*** jwagner_away is now known as jwagner15:50
*** yapeng has joined #openstack-ansible15:55
*** woodard_ has joined #openstack-ansible15:58
*** yapeng has quit IRC15:59
*** phalmos has quit IRC16:00
*** woodard has quit IRC16:02
*** woodard_ has quit IRC16:02
*** daneyon_ has joined #openstack-ansible16:05
*** Mudpuppy_ has joined #openstack-ansible16:07
*** fawadkhaliq has joined #openstack-ansible16:08
*** daneyon has quit IRC16:08
*** yaya_ has joined #openstack-ansible16:08
*** Mudpuppy_ has quit IRC16:09
*** yaya_ has quit IRC16:09
*** jwagner is now known as jwagner_away16:09
*** Mudpuppy_ has joined #openstack-ansible16:10
*** Mudpuppy has quit IRC16:10
*** yaya_ has joined #openstack-ansible16:11
*** yaya has quit IRC16:11
*** yaya_ is now known as yaya16:11
*** alextricity has quit IRC16:12
*** Mudpuppy has joined #openstack-ansible16:12
*** yaya_ has joined #openstack-ansible16:12
*** spotz is now known as spotz_zzz16:12
*** alextricity has joined #openstack-ansible16:13
*** Mudpuppy_ has quit IRC16:15
*** yaya has quit IRC16:15
*** yaya_ is now known as yaya16:15
*** jwagner_away is now known as jwagner16:15
*** k_stev has joined #openstack-ansible16:19
*** Mudpuppy has quit IRC16:26
*** phalmos has joined #openstack-ansible16:32
*** phalmos has quit IRC16:32
*** phalmos has joined #openstack-ansible16:33
odyssey4mepalendae mattt sigmavirus24 ok, so https://review.openstack.org/214700 is in to add a non-voting check that uses the same image across both clouds16:34
palendaeodyssey4me: So you're using Ubuntu sans any devstack stuff?16:35
palendae(which we should probably be doing anyway)16:35
palendaeIf I understand the difference based on the names of the nodes16:36
odyssey4mepalendae we had less choices before16:36
palendaeAh16:36
palendaeSo this is new?16:36
palendaethe ubuntu-trusty node, that is16:36
odyssey4meif I understand it correctly, we could actually set our own image specs now that they're using disk image builder - this is still experimental afaik16:36
odyssey4meyep, looks like it - right now it's only used in experimental jobs16:36
palendaeOk16:37
*** javeriak has quit IRC16:56
*** yaya has quit IRC16:56
*** javeriak has joined #openstack-ansible16:57
*** britthouser has joined #openstack-ansible17:00
*** jwagner is now known as jwagner_away17:06
openstackgerritMerged stackforge/os-ansible-deployment: Enable log_slave_updates  https://review.openstack.org/21457617:06
*** mpmsimo has joined #openstack-ansible17:07
*** phalmos has quit IRC17:07
odyssey4mehmm, I think that means it's tagging time17:09
odyssey4memattt how's that manual test in hpcloud running?17:10
*** mpmsimo has quit IRC17:10
*** sdake_ has joined #openstack-ansible17:17
*** b3rnard0 is now known as b3rnard0_away17:17
*** phalmos has joined #openstack-ansible17:19
*** shausy has quit IRC17:19
*** shausy has joined #openstack-ansible17:20
*** fawadk has joined #openstack-ansible17:20
*** sdake has quit IRC17:20
*** fawadkhaliq has quit IRC17:21
matttodyssey4me: still trucking17:23
*** mpmsimo has joined #openstack-ansible17:25
odyssey4mewell, reducing the affinities failed17:26
odyssey4melet me turn off ceilometer too17:26
*** britthouser has quit IRC17:27
*** britthouser has joined #openstack-ansible17:28
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Reduce AIO affinities and disable Ceilometer deployment  https://review.openstack.org/21467617:29
svgmattt: Space trucking? \m/17:32
*** KLevenstein has joined #openstack-ansible17:34
*** KLevenstein has quit IRC17:36
*** sdake has joined #openstack-ansible17:37
matttodyssey4me: it's running tempest atm, but from what i can tell cinder is up and running17:38
matttso looks like this hasn't uncovered anything17:38
matttanyway, off shopping, back in a bit17:38
odyssey4methanks mattt17:38
*** sdake_ has quit IRC17:40
*** Mudpuppy has joined #openstack-ansible17:42
*** k_stev has quit IRC17:43
*** woodard has joined #openstack-ansible17:46
*** sdake has quit IRC17:47
*** sdake has joined #openstack-ansible17:51
*** woodard has quit IRC17:54
*** woodard has joined #openstack-ansible17:54
*** mpmsimo has quit IRC17:58
*** javeriak_ has joined #openstack-ansible17:59
sigmavirus24mattt: It got the same message18:00
sigmavirus24odyssey4me: I'm investigating18:00
odyssey4meta18:00
odyssey4mefyi - 11.1.2 is tagged18:01
sigmavirus24or18:01
sigmavirus24it looks like18:01
sigmavirus24eh this is server basic but it's a similar message to what was being put in the logs for cinder-scheduler18:02
*** javeriak has quit IRC18:02
odyssey4mean error in the log, or the logs show stuff that looks fine?18:03
*** k_stev has joined #openstack-ansible18:03
sigmavirus24the tempest message says No Valid Host 50018:06
odyssey4meah yes - the scheduler didn't find a suitable host18:06
odyssey4meand cinder-volume is running?18:07
odyssey4me(on the host)18:07
sigmavirus24yes it is18:08
sigmavirus24and the log is there18:08
odyssey4mecan you manually create a new volume?18:09
sigmavirus24odyssey4me: lemme try18:09
sigmavirus24seems like I can18:12
* sigmavirus24 tried a 1 GB volume18:12
*** shausy has quit IRC18:16
sigmavirus24odyssey4me: so I have several volumes going18:19
sigmavirus24141 GB in total18:19
odyssey4mehmm, then wtf18:19
odyssey4meI think it's likely some sort of race/resource issue then.18:19
odyssey4meeben with tempest going in serial.18:19
odyssey4me*even18:19
odyssey4meso I'm thinking of patching the gate to run with only one worker for all services18:19
sigmavirus24odyssey4me: so this build took forever18:25
sigmavirus24nah nevermind18:25
sigmavirus24cinder's unmount/mount should happen earlier18:25
*** jwagner_away is now known as jwagner18:26
*** phalmos has quit IRC18:26
*** bapalm has quit IRC18:31
sigmavirus24yeah I'm at a loss for why this is happening18:32
*** bapalm has joined #openstack-ansible18:32
openstackgerritTom Cameron proposed stackforge/os-ansible-deployment: Add new roles to create a router container and custom routes  https://review.openstack.org/20368318:33
*** spotz_zzz is now known as spotz18:37
*** mpmsimo has joined #openstack-ansible18:41
*** cfarquhar has joined #openstack-ansible18:47
*** cfarquhar has quit IRC18:47
*** cfarquhar has joined #openstack-ansible18:47
*** javeriak_ has quit IRC18:47
*** spotz is now known as spotz_zzz18:48
odyssey4mesigmavirus24 yeah, I think we need to decide what to do about it in the meeting tomorrow18:49
*** spotz_zzz is now known as spotz18:49
*** jwagner is now known as jwagner_away18:54
*** b3rnard0_away is now known as b3rnard019:03
*** fawadk has quit IRC19:04
*** jwagner_away is now known as jwagner19:08
*** jwagner is now known as jwagner_away19:09
*** mnestheu1 has joined #openstack-ansible19:10
*** mnestheu1 is now known as scarlisle19:10
*** yaya has joined #openstack-ansible19:10
*** JRobinson__ has joined #openstack-ansible19:10
scarlisleHowdy19:11
scarlisleHopefully a quick question for you gentlemen.  I need to modify the spice console endpoint and I can see a couple of different ways of making the change, but I'm not sure what the *right* way is.  I'm specifically referring to setting either "nova_html5proxy_base_url" in user_variables.yml or setting "nova_console_endpoint" in user_variables.yml19:13
odyssey4mescarlisle not sure, personally - let me take a peek19:14
scarlislekk19:17
*** mpmsimo has quit IRC19:18
*** jwagner_away is now known as jwagner19:42
*** alejandrito has joined #openstack-ansible19:59
*** alejandrito_ has joined #openstack-ansible20:00
*** alejandrito has quit IRC20:00
*** JRobinson__ has quit IRC20:02
*** jwagner is now known as jwagner_away20:04
*** yaya has quit IRC20:04
*** jwagner_away is now known as jwagner20:12
*** alejandrito_ has quit IRC20:18
*** mpmsimo has joined #openstack-ansible20:33
*** yaya has joined #openstack-ansible20:36
*** jwagner is now known as jwagner_away20:48
*** mpmsimo has quit IRC20:59
*** jmckind has joined #openstack-ansible21:01
*** mpmsimo has joined #openstack-ansible21:05
*** KLevenstein has joined #openstack-ansible21:10
svgodyssey4me: scarlisle would like to know that as well, as I remain confused with the different endpoints config21:20
*** mpmsimo has quit IRC21:20
*** yaya has quit IRC21:20
*** Mudpuppy has quit IRC21:21
*** mpmsimo has joined #openstack-ansible21:22
*** coolj has joined #openstack-ansible21:23
*** woodard has quit IRC21:31
*** mpmsimo has quit IRC21:33
*** b8se11 has joined #openstack-ansible22:12
*** pradk has quit IRC22:23
*** KLevenstein has quit IRC22:23
*** scarlisle has quit IRC22:35
*** JRobinson__ has joined #openstack-ansible22:38
*** mpmsimo has joined #openstack-ansible22:39
*** mpmsimo has quit IRC22:40
*** britthouser has quit IRC22:56
*** spotz is now known as spotz_zzz23:04
*** JRobinson__ is now known as JRobinson__afk23:05
*** mpmsimo1 has joined #openstack-ansible23:10
*** tlian2 has joined #openstack-ansible23:18
*** tlian has quit IRC23:20
*** JRobinson__afk is now known as JRobinson__\23:21
*** JRobinson__\ is now known as JRobinson__23:21
*** shoutm has joined #openstack-ansible23:37
*** mpmsimo1 has quit IRC23:40

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!