Monday, 2015-03-30

*** mahito has joined #openstack-ansible00:00
*** Mudpuppy has joined #openstack-ansible00:21
*** sdake has joined #openstack-ansible00:23
*** sdake has quit IRC00:26
*** sdake has joined #openstack-ansible00:26
*** daneyon has joined #openstack-ansible00:47
*** daneyon_ has joined #openstack-ansible01:08
*** daneyon has quit IRC01:08
openstackgerritMerged stackforge/os-ansible-deployment: Move SYMLINK_DIR declaration to bootstrap-aio.sh  https://review.openstack.org/16731601:20
*** stevemar has quit IRC01:45
*** stevemar has joined #openstack-ansible01:46
*** sdake has quit IRC01:51
*** gregdek_ has quit IRC03:19
*** sandywalsh has quit IRC03:28
*** sandywalsh has joined #openstack-ansible03:37
*** sdake has joined #openstack-ansible03:49
*** sdake has quit IRC04:49
*** mrodden has quit IRC05:17
*** mrodden has joined #openstack-ansible05:31
*** vmtrooper has joined #openstack-ansible05:50
*** Mudpuppy has quit IRC06:05
*** vmtrooper has quit IRC06:10
*** vmtrooper has joined #openstack-ansible06:11
*** vmtrooper has quit IRC06:15
*** markvoelker has joined #openstack-ansible06:22
*** stevemar has quit IRC06:27
*** markvoelker has quit IRC06:27
*** stevemar has joined #openstack-ansible06:27
*** stevemar has quit IRC06:34
*** stevemar has joined #openstack-ansible06:34
*** Mudpuppy has joined #openstack-ansible06:36
*** Mudpuppy has quit IRC06:40
*** stevemar has quit IRC07:19
*** markvoelker has joined #openstack-ansible07:23
*** markvoelker has quit IRC07:27
svgI have no direct internet access on my target hosts. I already mirrored apt repo's on out internal server, but the scripts still need to access a bunch of external servers. Is there a config option to use a proxy host for everything?07:37
odyssey4mesvg not within os-ansible-deployment, no, but you should easily enough be able to use ansible to deploy proxy configuration for apt/wget/etc to all the hosts by deploying the required configs07:45
svggood point07:49
*** vmtrooper has joined #openstack-ansible07:59
*** vmtrooper has quit IRC08:04
svghmz08:08
svgdpkg: regarding .../mariadb-client-core-5.5_5.5.41+maria-1~trusty_amd64.deb containing mariadb-client-core-5.5:08:08
svg mariadb-client-5.5 conflicts with mysql-client-core-5.508:08
svg  mariadb-client-core-5.5 provides mysql-client-core-5.5 and is to be installed.08:08
svgthat looks like sth is messed up in that mariadb repo08:08
odyssey4mesvg it's more likely that the server installed the trusty package before the mariadb repo package - but it is possible that the mariadb repo has broken something again08:23
openstackgerritgit-harry proposed stackforge/os-ansible-deployment: Updated repository for minimum viable kilo install  https://review.openstack.org/16698608:24
svgodyssey4me: no, it had the mariadb version installed08:57
svgproblem is obvious here:08:57
svghttp://paste.ubuntu.com/10705761/08:57
svgthe mariadb version both provides and conflicts with mysql-client-5.5 ...08:58
openstackgerritMerged stackforge/os-ansible-deployment: Remove support for ~/.openstack_deploy/  https://review.openstack.org/16701009:22
*** markvoelker has joined #openstack-ansible09:24
*** mahito_ has joined #openstack-ansible09:25
*** mahito has quit IRC09:28
*** markvoelker has quit IRC09:29
*** vmtrooper has joined #openstack-ansible09:48
*** vmtrooper has quit IRC09:53
openstackgerritHugh Saunders proposed stackforge/os-ansible-deployment-specs: Manage resolv.conf  https://review.openstack.org/16807409:54
*** mahito_ has quit IRC10:04
*** markvoelker has joined #openstack-ansible10:25
*** markvoelker has quit IRC10:30
openstackgerritMerged stackforge/os-ansible-deployment: Fix dedicated replication network logic  https://review.openstack.org/16717710:42
*** jaypipes has joined #openstack-ansible11:17
*** markvoelker has joined #openstack-ansible11:26
*** markvoelker has quit IRC11:31
openstackgerritgit-harry proposed stackforge/os-ansible-deployment: Update cinder services to kilo  https://review.openstack.org/16806611:33
*** vmtrooper has joined #openstack-ansible11:37
*** nosleep77 has joined #openstack-ansible11:38
*** vmtrooper has quit IRC11:42
*** galstrom_zzz is now known as galstrom12:00
*** markvoelker has joined #openstack-ansible12:10
svgIs there a mailing list for the rpc project?12:11
odyssey4mesvg openstack-dev/openstack-operators would be the appropriate mailing lists to use12:15
svgok thx12:15
svgI'm in the proces of playbooks/infrastructure/infrastructure-setup.yml (v10) - I notice that it fails each time it waits on the container's networking to come up, it then goes on after a playbook re-run - sounds like some timeout being too short?12:19
odyssey4mesvg it could be, although we run the same process in the single node and multi node gate checks... so there might be something peculiar to your environment/config that's causing it to take longer to come up12:21
svgno idea, I'm testing on very recent machines (blades with 256GB Ram, 40 cores, ..)12:22
odyssey4meit'll possibly be something related to how the host networking is setup - maybe something to do with bonding or something else that would cause physical/virtual interface startup delays12:24
odyssey4meanyway, considering that it's easily worked around, I expect that it's not a crucial issue which is holding you back at all12:25
openstackgerritHugh Saunders proposed stackforge/os-ansible-deployment-specs: Manage resolv.conf  https://review.openstack.org/16807412:33
svgno, certainly not, just mentioning12:34
matttsvg: maybe you need to turn the turbo button off12:36
mattt:)12:37
* svg looks12:37
matttsvg: that sounds like a pretty sweet rig :)12:37
*** sandywalsh has quit IRC12:37
svgthose are thoe hosts meant to be used for kvm/compute12:37
svglxc-attach keeps yelling "$: command not found", for whatever container I try12:38
*** sandywalsh has joined #openstack-ansible12:40
matttsvg: i don't have an install in front of me right now but it sounds like you're using a non-root user12:41
svga non-root user when issuing that command?12:42
matttsvg: yeah, did you deploy this all using root?12:43
svglocaly a non privileged user on the ansible side, but logging in with root on the target hosts, yes12:44
svgtrying the lxc-attach as root12:44
matttsvg: lxc-attach is in /usr/bin, so your user should see it irrespective of which user you are12:48
svgouch, nm, that was a PEBCAK, sorry guy12:50
svgguys12:50
matttsvg: don't worry, my day is usually filled w/ those12:51
svgdamn, rabbitmq site is slow as hell now13:01
matttsvg: https://review.openstack.org/#/c/167723/13:02
matttwe ran into that issue last week13:02
svgaha13:05
*** galstrom is now known as galstrom_zzz13:12
*** sdake has joined #openstack-ansible13:25
*** vmtrooper has joined #openstack-ansible13:26
*** sdake_ has joined #openstack-ansible13:28
*** vmtrooper has quit IRC13:31
*** sdake has quit IRC13:31
*** britthouser has joined #openstack-ansible13:39
openstackgerritMerged stackforge/os-ansible-deployment: Allow swift stand alone to be installed for Juno  https://review.openstack.org/16717013:40
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Implement tunable keystone configuration  https://review.openstack.org/16568313:41
*** britthouser has quit IRC13:46
*** britthouser has joined #openstack-ansible13:47
*** britthouser has quit IRC13:49
*** sdake has joined #openstack-ansible13:49
*** britthouser has joined #openstack-ansible13:50
*** stevemar has joined #openstack-ansible13:52
*** sdake_ has quit IRC13:53
*** KLevenstein has joined #openstack-ansible13:55
*** ccrouch has left #openstack-ansible13:55
*** sigmavirus24_awa is now known as sigmavirus2414:09
sigmavirus24oh14:17
* sigmavirus24 wonders if the GitHub DDoS is affecting RabbitMQ14:17
* sigmavirus24 checks who Rabbit uses for a CDN14:17
*** hj-hp has joined #openstack-ansible14:30
*** galstrom_zzz is now known as galstrom14:33
*** hj-hp has quit IRC14:33
*** Mudpuppy has joined #openstack-ansible14:34
*** hj-hp has joined #openstack-ansible14:38
*** ccrouch has joined #openstack-ansible14:49
svghm, so it seems my containers restart very quickyly (as in immediately), I can ssh into them practiclly as soon as the restart container task finishes, but the ansible wait_for task doesn't seem to work here, until I restart the playbook14:52
*** jwagner_away is now known as jwagner14:55
*** alextricity has quit IRC15:09
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Implement tunable keystone configuration  https://review.openstack.org/16568315:10
*** britthouser has quit IRC15:11
*** britthouser has joined #openstack-ansible15:12
*** vmtrooper has joined #openstack-ansible15:15
*** vmtrooper has quit IRC15:20
*** sdake_ has joined #openstack-ansible15:26
*** sdake has quit IRC15:30
*** alextricity has joined #openstack-ansible15:36
openstackgerritMerged stackforge/os-ansible-deployment: Remove logstash multiline filter and change default number of workers  https://review.openstack.org/16554615:46
*** galstrom is now known as galstrom_zzz15:48
cloudnullpalendae you're now core. congrats and sorry :)15:50
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment-specs: Tunable OpenStack Configuration Specification  https://review.openstack.org/16897615:51
* palendae failed at not being seen15:51
d34dh0r53congrats palendae15:53
matttpalendae: congrats!15:56
cloudnullPSA: if you had a change that was dependent on https://review.openstack.org/#/c/166986/ (Minimal Kilo Patch) post git-harry 's changes you will need to rebase your dependent patch.15:57
* mattt needs to read openstack-dev more often :(15:57
palendaeThanks15:57
matttcloudnull: what are we doing regarding osprofiler?15:57
matttare we leaving it in api-paste.ini and adding toggles to allow it to be turned on/off?15:58
cloudnullnothing at this point. the paste configs from upstream we're brought in, wholesale, from upstream .15:58
cloudnulli'd say that we make it configurable.15:58
cloudnulland leave the default paste api untouched wherever possible.15:59
Sam-I-Amdoes osprofiler do anything without additional config in the service config file?16:00
*** daneyon has joined #openstack-ansible16:07
stevelleSam-I-Am: are you asking for documented behavior of osprofiler or actual behavior?16:07
stevelledocumented behavior is set up to be lazy-on16:08
stevelleso it should be mostly harmless16:08
sigmavirus24why are we still talking about osprofiler?16:08
Sam-I-Amstevelle: i'm trying to make sure glance in the upstream install guide doesnt blast useless crap into rabbitmq16:09
*** daneyon_ has quit IRC16:09
*** rrrobbb has joined #openstack-ansible16:10
*** rromans has quit IRC16:10
stevelleSam-I-Am: documented behavior says it wont16:10
*** rromans has joined #openstack-ansible16:11
*** rromans has quit IRC16:19
*** rromans has joined #openstack-ansible16:19
sigmavirus24Sam-I-Am: osprofiler should be off by default in Glance16:26
sigmavirus24Sam-I-Am: if it isn't, then you should file a bug so I can fix it16:26
matttcloudnull: cool i'll do that then16:26
matttsigmavirus24: did you write osprofiler16:27
sigmavirus24ಠ_ಠ16:27
Sam-I-Amsigmavirus24: so, even if its in -paste.ini...16:34
sigmavirus24Sam-I-Am: we only load one profile from paste.ini (flavor = )16:34
sigmavirus24Oh I see why you're asking: https://github.com/openstack/glance/blob/master/etc/glance-api-paste.ini#L1516:35
Sam-I-Amsigmavirus24: yes16:35
Sam-I-Amubuntu picked up that file16:35
sigmavirus24https://github.com/openstack/glance/blob/master/etc/glance-api.conf#L45916:35
sigmavirus24False is the default iirc16:35
sigmavirus24so while the middleware is there, it'll be non-operative iirc16:35
Sam-I-Amsigmavirus24: ok, that works16:35
* sigmavirus24 double checks to be paranoid16:36
Sam-I-Amlooks to be disabled in ubuntu too16:36
sigmavirus24Sam-I-Am: https://github.com/openstack/glance/blob/92180691248db1797591357e773166c0518bc0e0/glance/common/wsgi.py#L104..L10916:36
sigmavirus24False is the default16:37
Sam-I-Amwoo16:38
Sam-I-Amaight, back to concentrating on the install guide16:38
*** jlvillal_ has joined #openstack-ansible16:40
*** jlvillal_ has left #openstack-ansible16:41
*** KLevenstein has quit IRC16:44
*** galstrom_zzz is now known as galstrom16:48
*** jaypipes has quit IRC16:49
openstackgerritMerged stackforge/os-ansible-deployment: Change heat_metadata_server_url to external API.  https://review.openstack.org/16478516:54
*** rrrobbb has quit IRC16:55
*** rrrobbb has joined #openstack-ansible16:56
*** vmtrooper has joined #openstack-ansible17:03
*** KLevenstein has joined #openstack-ansible17:07
*** vmtrooper has quit IRC17:08
*** daneyon_ has joined #openstack-ansible17:10
*** daneyon has quit IRC17:10
alextricityNot too familiar with the recent openstack_environement changes. I'm looking through the openstack_user_config.yml and I see repo-infra_hosts. What is this exactly?17:15
alextricityDoes the frozen repo get copied over to these hosts, then maybe the playbooks use these repos to install pkgs and such?17:16
*** daneyon has joined #openstack-ansible17:20
*** daneyon_ has quit IRC17:23
*** jmccrory has joined #openstack-ansible17:40
*** jaypipes has joined #openstack-ansible17:42
cloudnullalextricity the repo bits are for the python source code. This allows you to maintain the python source code, built as python wheels, within your environment.17:56
alextricitycool! thanks cloudnull17:57
palendaealextricity: The repo is powered by https://github.com/cloudnull/yaprt. It looks though python requirements and builds a graph of dependency versions, then builds them and puts them in the repo18:02
*** Mudpuppy_ has joined #openstack-ansible18:15
*** Mudpuppy has quit IRC18:15
alextricityOh cool. Also, what is the best way to destroy my containers and rebuild my inventory now?18:16
cloudnullyou want to build all the things again ?18:17
alextricityyeah18:18
cloudnull./scripts/teardown.sh18:18
alextricityI forget to put is_metal under "properties" in the environment.yml file :{18:18
*** Mudpuppy_ is now known as Mudpuppy18:20
*** Mudpuppy has quit IRC18:27
*** daneyon_ has joined #openstack-ansible18:27
*** Mudpuppy has joined #openstack-ansible18:28
*** daneyon has quit IRC18:28
alextricityDang...the teardown script ran ansible, then ansible failed parsing the JSON, proceeding to delete my inventory file.   Doesn't look like it creates a backup either.18:29
alextricitySo none of the containers got deleted. Maybe it would be a good idea to have the script fail if that happens? Does this look familiar to anyone: http://pastebin.com/RdeT0mw018:30
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated nova logic to correct issues with addon devices  https://review.openstack.org/16657818:30
palendaealextricity: Teardown definitely nukes everything18:31
openstackgerritDavid Wilde proposed stackforge/os-ansible-deployment-specs: Initial commit of nova-kilofication  https://review.openstack.org/16904118:34
*** KLevenstein_ has joined #openstack-ansible18:38
*** KLevenstein has quit IRC18:41
*** KLevenstein_ is now known as KLevenstein18:41
*** vmtrooper has joined #openstack-ansible18:52
*** vmtrooper has quit IRC18:58
*** jwagner is now known as jwagner_away19:00
*** jwagner_away is now known as jwagner19:01
*** jwagner is now known as jwagner_lunch19:02
openstackgerritMiguel Grinberg proposed stackforge/os-ansible-deployment: WIP Horizon config changes for kilo  https://review.openstack.org/16746019:09
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Improve Galera running, stop, bootstrap behavior  https://review.openstack.org/16904919:12
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Improve Galera running, stop, bootstrap behavior  https://review.openstack.org/16904919:16
*** jwagner_lunch is now known as jwagner19:45
*** Mudpuppy has quit IRC19:56
*** rrrobbb has quit IRC20:05
*** Mudpuppy has joined #openstack-ansible20:06
*** Mudpuppy has quit IRC20:06
*** Mudpuppy has joined #openstack-ansible20:06
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Ensure LXC bridge exists, is up, and has rules  https://review.openstack.org/16906920:14
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Keystone config changes for Kilo  https://review.openstack.org/16747520:16
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Improve Galera running, stop, bootstrap behavior  https://review.openstack.org/16904920:17
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Ensure LXC bridge exists, is up, and has rules  https://review.openstack.org/16906920:17
sigmavirus24stevelle: cloudnull so we're splitting out the minimal kilo patch?20:35
stevelleI'm attempting to smooth the ramp with the bits that can be pulled out20:36
cloudnullsigmavirus24 yup.20:37
cloudnullstevelle for prez!20:37
sigmavirus24Send pez to stevelle ? Okay!20:38
cloudnullthe yak needed more shaving.20:38
*** vmtrooper has joined #openstack-ansible20:41
*** vmtrooper has quit IRC20:47
*** KLevenstein has quit IRC20:50
*** jmccrory has quit IRC20:56
*** jmccrory has joined #openstack-ansible21:00
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Improve Galera running, stop, bootstrap behavior  https://review.openstack.org/16904921:04
*** jmccrory1 has joined #openstack-ansible21:04
*** jmccrory1 has quit IRC21:04
*** jmccrory has quit IRC21:07
*** sdake has joined #openstack-ansible21:18
*** nosleep77 has quit IRC21:20
*** sdake_ has quit IRC21:21
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Improve Galera running, stop, bootstrap behavior  https://review.openstack.org/16904921:26
palendaestevelle: Did you figure out why it was failing the tests?21:31
stevellepalendae: no21:31
stevellepalendae: have you been able to do any more for the scenario tests?21:35
palendaestevelle: I haven't been able to get a case where it's not crashing on network stuff21:36
palendaeApsu: Have you been able to narrow down the kernel issue? Is it a linux bridge thing?21:36
stevellecloudnull: was mentioning some of the API calls returning 200 instead of expected 202, would be cool to isolate that from the "intermittent" issues21:37
palendaeI'm seeing the time outs, not 200/20221:37
Apsupalendae: Actually I believe the kernel is returning from the netlink/ioctl/syscalls involved in link creation/bridge enslaving before the process is complete.21:37
palendaeWhich is weird, cause I'm turning off network scenarios21:37
ApsuI wrote a fuzzing script to slam it with bridge/port operations and test the state after every single step, with adjustable delays21:38
ApsuAnd I'm finding that all kinds of points in time, ip link will return 0 and yet the thing you just did is *not* in effect immediately afterwards.21:38
ApsuThere's delays that vary in timing.21:38
palendaeDoes 'the thing you just did' finish?21:39
palendaeLike, does it become true, just outside of that window between when you called it and you checked?21:39
ApsuYes.21:40
ApsuIt does.21:40
ApsuWhich is slightly different than not ending up in a bridge. But21:40
ApsuI just finished the script and I'm testing A) dummy port types instead of veth pairs and B) without network namespaces (but still in the agents container)21:40
ApsuSo my next step is to test veth pairs hooking into containerized netns's21:41
*** jwagner is now known as jwagner_away21:41
ApsuInterestingly enough... when you don't create a lot of interfaces to add at once, it doesn't seem to fail, even running in parallel at fastest speed21:42
ApsuBut if you make a lot of interfaces to add at once, invariably there's some kind of delay in there21:42
ApsuAdding port 'br-5.port-3' to bridge 'br-5'21:43
Apsu-- Checking port 'br-5.port-3' master is bridge 'br-5'21:43
Apsu-- Port 'br-5.port-3' not in bridge 'br-5'! Exiting21:43
Apsu^ one of three or four "doesn't exist right after successfully doing the thing" errors occurring without delays inserted.21:44
ApsuSo... I almost think we might need some non-busy-wait loops in neutron to verify certain actions.21:44
palendaeSo bringing that up to the tempest level, the workers (and probably even just the tests when run in serial) are creating stuff faster than the kernel is registering21:44
ApsuEssentially yeah, because neutron's LBA is entirely convinced that if ip/brctl return 0, what you told them to do is FULLY done.21:45
ApsuAnd it's not even close.21:45
*** KLevenstein has joined #openstack-ansible21:52
stevelleso neutron is a bit naive, expecting synchronous execution.21:53
*** daneyon has joined #openstack-ansible21:58
*** daneyon_ has quit IRC21:59
stevellepalendae: made it over 30m w/o failure on the patch this time, so it will probably pass.  the following patch just needs rebase clicked after that, afaik.22:00
*** Mudpuppy has quit IRC22:04
*** KLevenstein has quit IRC22:25
*** vmtrooper has joined #openstack-ansible22:30
*** vmtrooper has quit IRC22:35
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Ensure LXC bridge exists, is up, and has rules  https://review.openstack.org/16906922:36
*** galstrom is now known as galstrom_zzz22:54
*** markvoelker has quit IRC22:57
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated repository for minimum viable kilo install  https://review.openstack.org/16698623:09
cloudnullstevelle this is the commit that deps on your two changes thus far. ^23:10
cloudnullthanks for doing all of that btw. :)23:10
palendaestevelle, cloudnull: Left a comment on https://review.openstack.org/#/c/169069/ just so I understand what the new lxc bridge stuff is doing23:15
*** stevemar has quit IRC23:18
*** logan2 has quit IRC23:25
*** mattoliverau has quit IRC23:29
stevellepalendae: will bang on it again23:29
*** mattoliverau has joined #openstack-ansible23:29
*** logan2 has joined #openstack-ansible23:33
*** bluebox has joined #openstack-ansible23:37
blueboxhey guys, there seems to be only a /kilo under the specs repo tree. Where would it be appropiate to push specs generally for master23:38
*** hj-hp has quit IRC23:39
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Ensure LXC bridge exists, is up, and has rules  https://review.openstack.org/16906923:41
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Ensure LXC bridge exists, is up, and has rules  https://review.openstack.org/16906923:43
*** bluebox is now known as javeriak23:44
openstackgerritMerged stackforge/os-ansible-deployment: Improve Galera running, stop, bootstrap behavior  https://review.openstack.org/16904923:45
javeriak  /msg NickServ VERIFY REGISTER javeriak kcemobngxnln23:45
*** markvoelker has joined #openstack-ansible23:53
*** mahito has joined #openstack-ansible23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!