*** sacharya has quit IRC | 00:00 | |
*** gouthamr has quit IRC | 00:09 | |
*** galstrom is now known as galstrom_zzz | 00:14 | |
*** leifmadsen has joined #openstack-ansible | 00:15 | |
*** cemmason has joined #openstack-ansible | 00:16 | |
*** openstackgerrit has quit IRC | 00:16 | |
*** openstackgerrit has joined #openstack-ansible | 00:17 | |
*** sacharya has joined #openstack-ansible | 00:21 | |
*** mss_ has quit IRC | 00:22 | |
*** sacharya has quit IRC | 00:23 | |
*** niknakpaddywak has joined #openstack-ansible | 00:24 | |
*** cemmason has quit IRC | 00:33 | |
*** rebase has quit IRC | 00:39 | |
*** sdake_ has joined #openstack-ansible | 00:45 | |
*** sdake has quit IRC | 00:45 | |
openstackgerrit | Matthew Oliver proposed openstack/openstack-ansible: Add statsd configuration for swift https://review.openstack.org/246182 | 00:57 |
---|---|---|
*** cemmason has joined #openstack-ansible | 01:01 | |
*** elo has joined #openstack-ansible | 01:03 | |
*** eil397 has quit IRC | 01:06 | |
*** sdake_ is now known as sdake | 01:12 | |
*** galstrom_zzz is now known as galstrom | 01:15 | |
*** jhesketh has joined #openstack-ansible | 01:15 | |
*** rebase has joined #openstack-ansible | 01:15 | |
*** jhesketh has quit IRC | 01:16 | |
*** skamithi has joined #openstack-ansible | 01:16 | |
*** skamithi has quit IRC | 01:17 | |
*** skamithi14 has joined #openstack-ansible | 01:21 | |
*** skamithi13 has quit IRC | 01:24 | |
*** galstrom is now known as galstrom_zzz | 01:26 | |
*** cemmason has quit IRC | 01:42 | |
*** baker has joined #openstack-ansible | 01:43 | |
*** baker has quit IRC | 01:44 | |
cloudnull | evenings | 01:45 |
*** baker has joined #openstack-ansible | 01:46 | |
*** baker has quit IRC | 01:48 | |
*** mss has joined #openstack-ansible | 01:52 | |
*** jhesketh has joined #openstack-ansible | 01:58 | |
*** jhesketh has quit IRC | 02:03 | |
*** Grey_A has joined #openstack-ansible | 02:03 | |
*** niknakpaddywak has quit IRC | 02:05 | |
*** jhesketh has joined #openstack-ansible | 02:06 | |
*** skamithi14 has quit IRC | 02:11 | |
*** skamithi13 has joined #openstack-ansible | 02:11 | |
*** Grey_A has left #openstack-ansible | 02:17 | |
*** niknakpaddywak has joined #openstack-ansible | 02:19 | |
bgmccollum | is this thing on? | 02:25 |
cloudnull | nope | 02:26 |
*** gouthamr has joined #openstack-ansible | 02:34 | |
*** neillc_away is now known as neillc | 02:36 | |
*** sacharya has joined #openstack-ansible | 02:53 | |
*** eil397 has joined #openstack-ansible | 02:56 | |
*** rebase has quit IRC | 02:56 | |
*** jthorne has joined #openstack-ansible | 02:57 | |
bgmccollum | hah | 02:57 |
bgmccollum | working on clouds.yml config for openstack client | 02:58 |
*** eil397 has quit IRC | 02:59 | |
*** eil397 has joined #openstack-ansible | 02:59 | |
openstackgerrit | Byron McCollum proposed openstack/openstack-ansible: Implement support for OpenStack client clouds.yml configuration file https://review.openstack.org/248986 | 03:14 |
openstackgerrit | Byron McCollum proposed openstack/openstack-ansible: Implement OpenStack client clouds.yml configuration file https://review.openstack.org/248986 | 03:15 |
*** eil397 has quit IRC | 03:18 | |
*** pcaruana has quit IRC | 03:27 | |
*** skamithi13 has quit IRC | 03:48 | |
*** alkari has joined #openstack-ansible | 04:09 | |
*** alkari has quit IRC | 04:13 | |
openstackgerrit | Merged openstack/openstack-ansible: IRR - Implemented for setup-hosts https://review.openstack.org/246392 | 04:13 |
*** sacharya has quit IRC | 04:18 | |
openstackgerrit | Neill Cox proposed openstack/openstack-ansible: Stop configure_diskspace from using small partitions https://review.openstack.org/216880 | 04:23 |
cloudnull | YAY the first IRR patch landed :) | 04:32 |
cloudnull | now i need https://review.openstack.org/#/c/241483/ to continue down the IRR path | 04:33 |
*** sacharya has joined #openstack-ansible | 04:40 | |
*** alkari has joined #openstack-ansible | 04:45 | |
openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Functional backport for the new repo-build process https://review.openstack.org/244215 | 04:48 |
*** jmccrory has quit IRC | 04:58 | |
*** jmccrory has joined #openstack-ansible | 05:00 | |
*** sdake_ has joined #openstack-ansible | 05:09 | |
*** gouthamr has left #openstack-ansible | 05:10 | |
*** sdake has quit IRC | 05:10 | |
*** ashishb has joined #openstack-ansible | 05:14 | |
*** sdake_ has quit IRC | 05:26 | |
*** shausy has joined #openstack-ansible | 05:41 | |
*** javeriak has joined #openstack-ansible | 05:43 | |
*** pcaruana has joined #openstack-ansible | 05:49 | |
*** mss has quit IRC | 05:54 | |
*** javeriak has quit IRC | 05:57 | |
*** mss has joined #openstack-ansible | 06:09 | |
*** gouthamr has joined #openstack-ansible | 06:10 | |
openstackgerrit | Neill Cox proposed openstack/openstack-ansible: Stop configure_diskspace from using small partitions https://review.openstack.org/216880 | 06:13 |
*** javeriak has joined #openstack-ansible | 06:21 | |
*** javeriak has quit IRC | 06:23 | |
*** javeriak has joined #openstack-ansible | 06:23 | |
*** alkari has quit IRC | 06:25 | |
*** pcaruana has quit IRC | 06:28 | |
*** javeriak has quit IRC | 06:29 | |
*** javeriak_ has joined #openstack-ansible | 06:29 | |
*** javeriak has joined #openstack-ansible | 06:32 | |
*** javeriak_ has quit IRC | 06:34 | |
*** phiche has joined #openstack-ansible | 06:43 | |
*** phiche has quit IRC | 06:46 | |
*** phiche has joined #openstack-ansible | 06:51 | |
*** phiche has quit IRC | 06:56 | |
*** henrymy has joined #openstack-ansible | 07:01 | |
*** henrymy has quit IRC | 07:02 | |
*** phiche has joined #openstack-ansible | 07:02 | |
*** phiche has quit IRC | 07:04 | |
*** phiche has joined #openstack-ansible | 07:05 | |
*** phiche has quit IRC | 07:06 | |
*** phiche has joined #openstack-ansible | 07:17 | |
*** sacharya has quit IRC | 07:27 | |
*** mss has quit IRC | 07:35 | |
evrardjp | hello everyone | 07:54 |
evrardjp | bgmccollum: you're there? | 07:54 |
*** javeriak has quit IRC | 07:55 | |
*** sacharya has joined #openstack-ansible | 07:56 | |
*** sacharya has quit IRC | 08:03 | |
*** mpavone has joined #openstack-ansible | 08:16 | |
*** mss has joined #openstack-ansible | 08:31 | |
*** subscope has joined #openstack-ansible | 08:36 | |
*** sdake has joined #openstack-ansible | 08:45 | |
*** sdake has quit IRC | 08:48 | |
*** subscope has quit IRC | 08:48 | |
*** cemmason has joined #openstack-ansible | 08:48 | |
*** sdake has joined #openstack-ansible | 08:48 | |
*** subscope has joined #openstack-ansible | 08:52 | |
*** tiagogomes has joined #openstack-ansible | 08:53 | |
*** sdake has quit IRC | 09:01 | |
*** subscope has quit IRC | 09:09 | |
*** subscope has joined #openstack-ansible | 09:10 | |
*** mattoliverau has quit IRC | 09:10 | |
*** matt6434 has joined #openstack-ansible | 09:11 | |
*** javeriak has joined #openstack-ansible | 09:16 | |
*** openstackgerrit has quit IRC | 09:16 | |
*** openstackgerrit has joined #openstack-ansible | 09:17 | |
*** slotti has joined #openstack-ansible | 09:19 | |
*** javeriak has quit IRC | 09:27 | |
*** javeriak has joined #openstack-ansible | 09:28 | |
*** javeriak_ has joined #openstack-ansible | 09:28 | |
*** javeriak has quit IRC | 09:32 | |
*** ashishb has quit IRC | 09:35 | |
*** subscope has quit IRC | 09:37 | |
*** ashishb has joined #openstack-ansible | 09:41 | |
*** adac has joined #openstack-ansible | 10:02 | |
adac | Did someone of you manage to install AIO build? | 10:03 |
adac | with this script: curl https://raw.githubusercontent.com/openstack/openstack-ansible/master/scripts/run-aio-build.sh | sudo bash | 10:03 |
adac | I'm running into trouble with the connection to mongodb | 10:04 |
adac | restarting the script then results in an error that the user on mongodb already exists | 10:04 |
*** sacharya has joined #openstack-ansible | 10:08 | |
adac | ok I can clean the messs up with the teardown.sh script and then restart it again | 10:09 |
adac | however still I have trouble to connect to mongodb it seems | 10:09 |
*** gparaskevas has joined #openstack-ansible | 10:12 | |
*** sacharya has quit IRC | 10:12 | |
*** javeriak_ has quit IRC | 10:25 | |
*** javeriak has joined #openstack-ansible | 10:26 | |
*** prometheanfire has quit IRC | 10:27 | |
*** prometheanfire has joined #openstack-ansible | 10:29 | |
*** subscope has joined #openstack-ansible | 10:30 | |
*** javeriak_ has joined #openstack-ansible | 10:34 | |
*** javeriak has quit IRC | 10:37 | |
*** subscope has quit IRC | 10:41 | |
odyssey4me | adac we use that process for gate testing, so yes it works | 10:43 |
odyssey4me | adac I would recommend doing it step by step instead as per http://docs.openstack.org/developer/openstack-ansible/developer-docs/quickstart-aio.html#running-a-customized-aio-build | 10:44 |
odyssey4me | if you disable the ceilometer deployment then you will bypass mongo issues | 10:44 |
adac | odyssey4me, this is probably a good idea, since I have no plan what "ceilometer" does ;-) | 10:47 |
odyssey4me | adac I hope you're starting with a minimal Ubuntu install base - otherwise you may hit other trouble | 10:47 |
adac | odyssey4me, I try the teardown script, that worked just fine the first time at least | 10:48 |
adac | now it is running since ages the scond time | 10:48 |
adac | I think I have to reinstall ubuntu then... | 10:48 |
*** javeriak_ has quit IRC | 10:53 | |
*** matt6434 has quit IRC | 10:54 | |
*** javeriak has joined #openstack-ansible | 10:54 | |
*** matt6434 has joined #openstack-ansible | 10:55 | |
*** javeriak_ has joined #openstack-ansible | 10:57 | |
kmARC | odyssey4me: No matching distribution found for pycrypto | 10:59 |
kmARC | I start feeling I'm running into every possible bugs :) | 10:59 |
odyssey4me | kmARC hmm, that's odd - at what stage are you seeing this? | 11:01 |
*** javeriak has quit IRC | 11:02 | |
odyssey4me | hughsaunders this is the review I mentioned earlier: https://review.openstack.org/241483 | 11:02 |
hughsaunders | odyssey4me: cool, will have a look when I'm done testing mancdaz's mass pr | 11:03 |
kmARC | setup-infrastructure.yml, galera_container | 11:03 |
kmARC | brb | 11:04 |
odyssey4me | kmARC alright, did the repo-build run successfully? | 11:04 |
*** phiche has quit IRC | 11:04 | |
odyssey4me | hughsaunders thanks | 11:04 |
odyssey4me | adac yeah the teardown script is ok, but not foolproof | 11:05 |
*** javeriak_ has quit IRC | 11:10 | |
evrardjp | hello everyone | 11:15 |
*** tiagogomes has quit IRC | 11:18 | |
*** tiagogomes_ has joined #openstack-ansible | 11:18 | |
*** toddnni has quit IRC | 11:19 | |
*** toddnni has joined #openstack-ansible | 11:22 | |
odyssey4me | o/ evrardjp | 11:25 |
odyssey4me | our gate check updates in -infra have completed but the puppet jobs haven't quite finished for some reason... busy following that up | 11:25 |
evrardjp | I had nice scripts to apply all these commits to all the repos, now I realize that I destroyed the irr | 11:26 |
evrardjp | cause I thought it was going to merge directly... | 11:26 |
evrardjp | the 2 line change on the .gitignore will cause me pain :p | 11:26 |
odyssey4me | evrardjp yeah, we should have fixed that a while ago - sorry | 11:26 |
evrardjp | the problem is that .gitreview is not there, so when you redownload, you can't git review stuff done :p | 11:27 |
*** phiche has joined #openstack-ansible | 11:34 | |
odyssey4me | evrardjp if you git review -d for each review, then you'll be able to edit and amend the commit | 11:34 |
evrardjp | true, if there is .gitreview to have git review work :) | 11:35 |
evrardjp | else the git review -s won't work :p | 11:35 |
odyssey4me | evrardjp ah yes - I just added a .gitreview before doing git review -s | 11:36 |
evrardjp | that's what in my commit :p | 11:36 |
evrardjp | have the git dotfiles :) | 11:37 |
evrardjp | but you're making me doubt | 11:37 |
kmARC | odyssey4me: yes, the repo containers look fine to me | 11:37 |
evrardjp | yeah there is a .gitreview in the trees | 11:37 |
odyssey4me | kmARC ok, but the pip wheel it's trying to install should be in the repo containers | 11:38 |
kmARC | makes sense | 11:39 |
odyssey4me | kmARC go to /opt/openstack-ansible/playbooks | 11:39 |
odyssey4me | then execute: ansible repo_all -m shell -a 'ls -al /var/www/repo/os-releases/' | 11:39 |
odyssey4me | there should be one directory there - likely 12.0.1 ? | 11:39 |
kmARC | yes | 11:40 |
odyssey4me | kmARC ok, so execute: ansible repo_all -m shell -a 'ls -al /var/www/repo/os-releases/12.0.1/pycrypto*' | 11:40 |
kmARC | wow, lot of red stuff :) | 11:41 |
kmARC | /bin/sh: 2: /var/www/repo/os-releases/12.0.1/pycrypto-2.6.1-cp27-none-linux_x86_64.whl: Permission denie | 11:41 |
odyssey4me | kmARC oh? can you pastebin the output somewhere? | 11:41 |
odyssey4me | hmm, that's odd | 11:42 |
kmARC | ok, sorry it was a line break problem | 11:42 |
kmARC | now pasted again, looks fine | 11:42 |
odyssey4me | kmARC is there more than one container? | 11:42 |
kmARC | 3 | 11:43 |
odyssey4me | ok, did all containers give a good response? | 11:43 |
kmARC | yes | 11:43 |
odyssey4me | alright, now let's try something else | 11:44 |
odyssey4me | can you pastebin the results of this somewhere? ansible galera_all -m shell -a 'cat ~/.pip/pip.conf' | 11:44 |
kmARC | OK here are the problems http://paste.openstack.org/show/IDFAXZMTvjoWq9pdUe70/ | 11:45 |
evrardjp | in advance: sorry for the spam, and I hope this script works. | 11:47 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-apt_package_pinning: New git dotfiles for independant repository https://review.openstack.org/248632 | 11:47 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-lxc_container_create: New git dotfiles for independant repository https://review.openstack.org/248634 | 11:47 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-lxc_hosts: New git dotfiles for independant repository https://review.openstack.org/248635 | 11:47 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-openstack_hosts: New git dotfiles for independant repository https://review.openstack.org/248636 | 11:47 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-pip_install: New git dotfiles for independant repository https://review.openstack.org/248637 | 11:47 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-py_from_git: New git dotfiles for independant repository https://review.openstack.org/248638 | 11:47 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-rsyslog_client: New git dotfiles for independant repository https://review.openstack.org/248639 | 11:47 |
kmARC | odyssey4me: should I destroy the galera_all containers and redeploy them? | 11:48 |
odyssey4me | kmARC ok - that's probably due to a failure in executing the playbooks on the first container... try this now: ansible galera_all -m shell -a 'curl http://192.168.0.1:8181/os-releases/12.0.1/ | grep pycrypto' | 11:48 |
kmARC | hm | 11:48 |
kmARC | neither of the containers can connect at all | 11:48 |
odyssey4me | kmARC none of them? | 11:49 |
kmARC | oh wait | 11:49 |
kmARC | this is a wrong IP | 11:49 |
odyssey4me | ah | 11:49 |
kmARC | or not? | 11:49 |
odyssey4me | well, that depends on the config you set out | 11:49 |
kmARC | this is the IP of the first infra box | 11:49 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-lxc_hosts: Removed undesired .DS_Store file https://review.openstack.org/248643 | 11:49 |
kmARC | (and the LB) | 11:50 |
odyssey4me | kmARC yes, it should be the lb | 11:50 |
kmARC | looks like I cannot telnet either | 11:50 |
odyssey4me | the lb address you set out as 'internal_lb_vip_address' in /etc/openstack_deploy/openstack_user_config.yml | 11:50 |
odyssey4me | right, so on your lb host confirm whether haproxy is running? | 11:51 |
odyssey4me | ie: service haproxy status | 11:51 |
kmARC | it's not. Strange | 11:51 |
odyssey4me | aha, try and restart it | 11:51 |
odyssey4me | if it has a config error, it should tell you which one is a problem | 11:52 |
kmARC | it's not even installed | 11:52 |
kmARC | how is that possible | 11:52 |
odyssey4me | ah | 11:53 |
odyssey4me | that is normal if you didn't run it manually | 11:53 |
odyssey4me | the sequence for running the playbooks would be: | 11:53 |
odyssey4me | 1 - setup-hosts | 11:53 |
odyssey4me | 2 - haproxy-install | 11:53 |
odyssey4me | 3 - setup-infrastructure | 11:53 |
odyssey4me | 4 - setup-openstack | 11:53 |
kmARC | yepp, this is the sequence I usually do | 11:54 |
odyssey4me | haproxy is not included as a default in the meta playbooks because some deployers use an F5 load balancer instead | 11:54 |
kmARC | what can be is that in the multiple try-teardown cycles I missed this one | 11:54 |
odyssey4me | yeah, entirely possible | 11:54 |
evrardjp | pabelanger: are you there? | 11:54 |
odyssey4me | run the haproxy-install playbook and verify that it actually installed it | 11:55 |
kmARC | odyssey4me: thanks, hope that this was the problem. Let me check back later now I need to run | 11:55 |
odyssey4me | evrardjp unlikely - pabelanger will only show up in our afternoon, normally | 11:55 |
odyssey4me | hmm, the functional jobs are still showing as 'NOT_REGISTERED' | 11:56 |
odyssey4me | infra is looking into that | 11:57 |
*** sdake has joined #openstack-ansible | 11:58 | |
*** subscope has joined #openstack-ansible | 12:00 | |
*** sdake has quit IRC | 12:03 | |
evrardjp | that was my question :) | 12:05 |
*** mpavone has quit IRC | 12:07 | |
*** javeriak has joined #openstack-ansible | 12:09 | |
*** sdake has joined #openstack-ansible | 12:09 | |
adac | odyssey4me, what is missing in my eyes in the start script for the AIO is a check if an openssh-server is installed | 12:12 |
odyssey4me | adac well, I guess that could be a bad assumption... but we are assuming that you have an sshd installed, otherwise you wouldn't be able to get to your server ;) | 12:13 |
adac | odyssey4me, I'm testing it on my laptop currently :-) | 12:13 |
adac | directly! | 12:13 |
*** subscope has quit IRC | 12:13 | |
*** subscope has joined #openstack-ansible | 12:14 | |
odyssey4me | adac bad idea | 12:15 |
*** sacharya has joined #openstack-ansible | 12:15 | |
odyssey4me | you need to make use of something that you can trash and rebuild - best to use virtualbox or some other vm mechanism | 12:16 |
odyssey4me | otherwise you'll end up with a pretty dirty machine | 12:16 |
evrardjp | maybe his laptop is trash/rebuild? ;) | 12:16 |
odyssey4me | lol | 12:16 |
adac | odyssey4me, yeah I see. But actually It is an old notbook which I'm not using anymore ;) | 12:16 |
evrardjp | ahah! touché! | 12:17 |
adac | actually it is trash :D | 12:17 |
adac | haha | 12:17 |
*** ChanServ sets mode: +o odyssey4me | 12:17 | |
evrardjp | no issues with hardware requirements? | 12:17 |
evrardjp | laptops are not usually powerhouses | 12:17 |
adac | evrardjp, well I just wanted to test the installation part | 12:18 |
adac | not really run it there | 12:18 |
adac | but yeah the memory is to weak in any case | 12:18 |
*** ChanServ sets mode: +o odyssey4me | 12:18 | |
adac | but I run into trouble with mogodb as I told odyssey4me before | 12:18 |
evrardjp | yup it could cause some lxc containers to crash (that happened to me) | 12:18 |
adac | evrardjp, ok that fully explains it | 12:19 |
adac | evrardjp, odyssey4me how much disk space do I need for testing the AIO? | 12:19 |
evrardjp | if your system is OOM, he eventually needs to get ram | 12:19 |
adac | 80 it seems is that correct? | 12:19 |
evrardjp | depends on your vg | 12:19 |
evrardjp | 80 is a bare minimum | 12:20 |
*** mpavone has joined #openstack-ansible | 12:20 | |
adac | that also makes it difficult to test it via virtualbox if it is really 80 | 12:20 |
*** sacharya has quit IRC | 12:20 | |
evrardjp | maybe we should work on having CoW of lxc systems with btrfs/overlayfs | 12:21 |
adac | evrardjp, is 16GB Ram rally neede or can i also go with les? | 12:21 |
adac | *less | 12:21 |
adac | mean just for testing the installation | 12:21 |
evrardjp | on an AIO? | 12:21 |
adac | exactly | 12:21 |
evrardjp | that's tough question, maybe ask odyssey4me, he knows more in AIO than me :p | 12:21 |
evrardjp | 4GB is definitely not enough | 12:22 |
*** ChanServ sets mode: +o hughsaunders | 12:22 | |
evrardjp | 8GB... I don't know | 12:22 |
odyssey4me | adac it's covered in http://docs.openstack.org/developer/openstack-ansible/developer-docs/quickstart-aio.html#quick-start | 12:22 |
odyssey4me | although you can do it with 8GB RAM, but it mmight behave a little poorly | 12:23 |
adac | odyssey4me, yeah it says 16 but I do not have that many ond wondering if the installation will crash when I have about 8 only | 12:23 |
adac | but I see it should be fine but slow :) | 12:23 |
evrardjp | maybe reduce your requirements, by removing swift and ceilometer? | 12:23 |
odyssey4me | adac as long as you're not using an lvm vg you'll get away with around 65GB of free space on the disk | 12:23 |
adac | evrardjp, yes odyssey4me suggested me that. Should try it out | 12:24 |
odyssey4me | yeah, if you cut down requirements it'll help | 12:24 |
adac | odyssey4me, ok thanks | 12:24 |
*** ChanServ sets mode: +o odyssey4me | 12:24 | |
evrardjp | you need to cut in either scope, budget or time, that's the triforce! | 12:24 |
odyssey4me | adac this patch will help with that: https://review.openstack.org/248887 | 12:24 |
odyssey4me | adac I'm doing some work at the moment to implement more focused use-cases which will require less too... but I don't have patches up for review yet | 12:25 |
adac | odyssey4me, evrardjp The admin just messaged me. He has 8 gigs of ram for me in a present box :-) So I think I will give virtualbox a try again now. with 8GB of ram and about 80 gigs of space :) | 12:27 |
evrardjp | haha :D | 12:28 |
adac | guys I'm back soon! With new horse powers and surely some more questions! :D | 12:29 |
adac | thank you! | 12:29 |
*** adac has quit IRC | 12:30 | |
*** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible" | 12:31 | |
*** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://goo.gl/tTmdgs" | 12:32 | |
*** alejandrito has joined #openstack-ansible | 12:34 | |
*** ashishb has quit IRC | 12:38 | |
odyssey4me | evrardjp fyi, I think I know what the issue is with the gate jobs - but I need to chat to pabelanger to figure out a suitable solution | 12:39 |
evrardjp | good | 12:39 |
evrardjp | I'm thinking to the best way to approach policy files distribution during that time, don't worry, I have other stuff to do :p | 12:39 |
*** javeriak has quit IRC | 12:41 | |
*** subscope has quit IRC | 12:44 | |
*** subscope has joined #openstack-ansible | 12:46 | |
*** javeriak has joined #openstack-ansible | 12:49 | |
*** mpavone has quit IRC | 13:00 | |
*** gouthamr has left #openstack-ansible | 13:00 | |
*** jhesketh has quit IRC | 13:06 | |
*** jhesketh has joined #openstack-ansible | 13:07 | |
*** mpavone has joined #openstack-ansible | 13:12 | |
*** javeriak has quit IRC | 13:13 | |
*** javeriak has joined #openstack-ansible | 13:14 | |
*** dolpher has quit IRC | 13:16 | |
*** dolpher has joined #openstack-ansible | 13:17 | |
*** adac has joined #openstack-ansible | 13:21 | |
adac | arrrg. He has only time tomorrow to upgrade my RAM memory | 13:22 |
*** shausy has quit IRC | 13:23 | |
openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: AIO bootstrap in Ansible https://review.openstack.org/239525 | 13:25 |
bgmccollum | odyssey4me: is there a way move that review to target master? or just abandon and open a new review? | 13:28 |
odyssey4me | bgmccollum cherry-pick it to master, I guess :) | 13:29 |
bgmccollum | ok, ill push the fixes, then cheery pick to master | 13:29 |
*** josh6627 has joined #openstack-ansible | 13:29 | |
*** josh6627 has quit IRC | 13:29 | |
*** jhesketh has quit IRC | 13:30 | |
openstackgerrit | Byron McCollum proposed openstack/openstack-ansible: Implement OpenStack client clouds.yml configuration file https://review.openstack.org/249209 | 13:33 |
*** markvoelker has quit IRC | 13:35 | |
bgmccollum | odyssey4me: went the opposite route. abandoned liberty review, opened new review targeting master, and manually cherry-picked + fixes. will make things cleaner in the future. | 13:40 |
*** markvoelker has joined #openstack-ansible | 13:53 | |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-specs: Added spec for policy-files-distribution https://review.openstack.org/249227 | 13:59 |
*** subscope has quit IRC | 14:00 | |
*** woodard has joined #openstack-ansible | 14:01 | |
*** woodard has quit IRC | 14:01 | |
*** woodard has joined #openstack-ansible | 14:02 | |
*** jhesketh has joined #openstack-ansible | 14:02 | |
*** karimb has joined #openstack-ansible | 14:03 | |
*** jhesketh has quit IRC | 14:03 | |
*** jhesketh has joined #openstack-ansible | 14:04 | |
*** subscope has joined #openstack-ansible | 14:04 | |
mhayden | odyssey4me: thanks for the aio heads up | 14:09 |
mhayden | i'll hopefully adjust that later today | 14:09 |
mhayden | any more thoughts on the switch to paramiko for the gate | 14:09 |
*** subscope has quit IRC | 14:10 | |
*** javeriak has quit IRC | 14:10 | |
*** subscope has joined #openstack-ansible | 14:11 | |
*** subscope has quit IRC | 14:13 | |
*** subscope has joined #openstack-ansible | 14:13 | |
*** subscope has quit IRC | 14:16 | |
*** subscope has joined #openstack-ansible | 14:17 | |
*** sacharya has joined #openstack-ansible | 14:45 | |
*** greg_a has joined #openstack-ansible | 14:47 | |
*** sacharya has quit IRC | 14:49 | |
*** mgoddard_ has joined #openstack-ansible | 14:59 | |
lbragstad | cloudnull miguelgrinberg fyi - https://review.openstack.org/#/c/244853/ | 15:02 |
*** mgoddard has quit IRC | 15:03 | |
cloudnull | morning | 15:05 |
*** spotz_zzz is now known as spotz | 15:05 | |
odyssey4me | mhayden so there are two approaches - disable ssh pipelining https://review.openstack.org/248739 or switch to paramico https://review.openstack.org/248361 | 15:07 |
odyssey4me | both do seem to work | 15:07 |
cloudnull | any cores about that can do a review on https://review.openstack.org/#/c/241483/ -- i need that to continue the irr work | 15:09 |
odyssey4me | cloudnull hughsaunders will try to review that today after some other work he's busy with | 15:10 |
odyssey4me | maybe mattt could too - not sure? | 15:10 |
cloudnull | kk. | 15:10 |
cloudnull | whats the state of the irr gate? did we disable the functional tests? | 15:11 |
mattt | cloudnull: i can have a look | 15:11 |
*** jwagner_away is now known as jwagner | 15:13 | |
openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Update Kilo SHA's - 20.11.2015 https://review.openstack.org/245968 | 15:14 |
cloudnull | ^ re-updated the kilo shas to pull in the backport lbragstad worked on for fernet | 15:15 |
odyssey4me | cloudnull no, the functional tests are broken - busy working with infra to figure out why | 15:15 |
odyssey4me | cloudnull does that kilo update need an updated date? | 15:16 |
openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Update Kilo SHA's - 24.11.2015 https://review.openstack.org/245968 | 15:17 |
cloudnull | sure | 15:17 |
odyssey4me | cloudnull and perhaps updates to playbooks/defaults/repo_packages/openstack_other.yml too? | 15:17 |
cloudnull | odyssey4me: nothing to update | 15:17 |
*** markvoelker has quit IRC | 15:18 | |
cloudnull | i I could move the date forward, but its all the same bits at this point | 15:18 |
*** markvoelker has joined #openstack-ansible | 15:18 | |
cloudnull | odyssey4me: are the functional tests busted because of something in the role or is it an infra thing? | 15:19 |
spotz | Let me know if there's anything small to do or reviews needed, if today is like yesterday should have some free time | 15:19 |
odyssey4me | spotz we always have reviews to do :) see the channel topic for a dashboard | 15:21 |
odyssey4me | cloudnull it's an issue somewhere between infra's puppet jobs failing a bit, and likely also some additional job configuration required - pabelanger is on it :) | 15:22 |
spotz | Okie, I'll hit that up. Its help you guys or go look for soemthing over in barbican:) | 15:22 |
cloudnull | odyssey4me: sweet | 15:22 |
cloudnull | pabelanger: if i can help in any way please let me know | 15:22 |
cloudnull | spotz: so you have to choose the lesser of evils i see | 15:23 |
cloudnull | :p | 15:23 |
pabelanger | odyssey4me: cloudnull: Ya, working in -infra | 15:23 |
pabelanger | but I don't think the issue was the patchset | 15:23 |
pabelanger | JJB just failing for another reason | 15:23 |
odyssey4me | spotz well, a new barbican role for OSA could be nice to see :) | 15:23 |
spotz | heheh well if there's nothing there I just make my own bug and do localization:) | 15:24 |
*** subscope has quit IRC | 15:26 | |
cloudnull | ++ barbican in OSA would be cool for sure | 15:35 |
*** subscope has joined #openstack-ansible | 15:36 | |
*** kmARC_2 has joined #openstack-ansible | 15:36 | |
*** kmARC has quit IRC | 15:37 | |
mhayden | odyssey4me: i'm fine with either so long as it makes ssh more reliable in the gate :P | 15:38 |
*** kmARC has joined #openstack-ansible | 15:38 | |
odyssey4me | mhayden , cloudnull , evrardjp any thoughts on disabling ssh pipelining versus switching to paramico ? | 15:38 |
openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Use Ansible AIO playbook for run-aio-build.sh https://review.openstack.org/244708 | 15:38 |
odyssey4me | ssh pipeline disable: https://review.openstack.org/248739 | 15:38 |
evrardjp | I had more chance in the past with paramiko | 15:38 |
openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Updating AIO docs for Ansible playbook https://review.openstack.org/244720 | 15:38 |
odyssey4me | switch to paramico: https://review.openstack.org/248361 | 15:38 |
evrardjp | disabling pipelining is slow, paramiko too | 15:39 |
evrardjp | I found paramiko better overall, but's maybe just an impression | 15:40 |
mhayden | i've used paramiko for other projects and it's always been nice | 15:40 |
*** subscope has quit IRC | 15:40 | |
mhayden | and i believe it respects ~/.ssh/config as well as keys | 15:40 |
mhayden | (as the ssh client does) | 15:40 |
odyssey4me | bgmccollum any thoughts from you, considering that you did some digging into ssh issues before? | 15:41 |
mhayden | we could try flipping pipelining off since it's a smaller change | 15:41 |
mhayden | but i think controlmaster might be causing problems, too | 15:41 |
mhayden | i could be wrong | 15:41 |
mhayden | ansible does some crazy stuff around sending data over ssh | 15:41 |
evrardjp | mhayden: IIRC, controlmaster and pipelining are connected together | 15:42 |
evrardjp | I don't remember | 15:42 |
mhayden | oh, okay | 15:42 |
mhayden | perhaps we shouldn't specify ssh command options at all and turn pipelining off | 15:42 |
evrardjp | I even tried to specifically define controlmaster path etc... | 15:43 |
cloudnull | mhayden: on your ML post the code you linked back to was for ansible 2.0, have you had a chance to look into the 1.9 bits to see if we can fix the issue? | 15:43 |
odyssey4me | well, something recent changed which is causing this - I suspect it's something in base ubuntu itself... perhaps a default sshd config change, or a change in sshd or ssh itself? | 15:43 |
mhayden | cloudnull: not yet -- didn't 2.0 go stable ? | 15:43 |
mhayden | i haven't had a chance to look at it yet | 15:44 |
cloudnull | no | 15:44 |
evrardjp | not stable yet | 15:44 |
cloudnull | its in rc1 i think | 15:44 |
cloudnull | https://github.com/ansible/ansible/blob/v1.9.4-1/lib/ansible/runner/connection_plugins/ssh.py | 15:44 |
mhayden | oh, nvm | 15:44 |
cloudnull | ^ thats the connection plugin we've been using | 15:44 |
mhayden | ah, it's in _communicate() in 1.9 | 15:44 |
mhayden | so it looks like it can't write data to stdin when that happens | 15:45 |
mhayden | which is still really similar to what i saw in the 2.0 code | 15:45 |
odyssey4me | another thing to consider is that it seems that the -nv job succeeds more often than the dsvm job... so perhaps it's a base set of packages or some sort of config difference... | 15:45 |
odyssey4me | we could switch wholesale to the base image used in the -nv job | 15:45 |
mhayden | weird | 15:46 |
odyssey4me | but that doesn't work for juno, so I'd have to setup different jobs based on branch | 15:46 |
mhayden | odyssey4me: is -nv just plain vanilla ubuntu 14.04? | 15:46 |
evrardjp | for pipelining, IIRC you could disable it only for the run by setting an env variable | 15:46 |
mhayden | evrardjp: i thought you put pipelining=False in ansible.cfg? | 15:46 |
evrardjp | you could | 15:46 |
evrardjp | I think you can also have ANSIBLE_SSH_PIPELINING=0 | 15:47 |
odyssey4me | mhayden -nv is a minimal image built by disk-image-builder and deployed across all cloud providers for infra... the dsvm job is d-i-b based in hpcloud and snapshot based in rax cloud | 15:47 |
evrardjp | to avoid editing config files | 15:47 |
mhayden | interesting, odyssey4me | 15:47 |
evrardjp | if it helps | 15:47 |
mhayden | i like the nv idea then | 15:47 |
mhayden | :) | 15:47 |
mhayden | also nv has almost zero ssh issues | 15:47 |
odyssey4me | evrardjp so my concern here is that this issue will hit production environments | 15:47 |
odyssey4me | the gate is our canary | 15:47 |
mhayden | the ansible aio bootstrap would always make it much further in the nv | 15:47 |
*** javeriak has joined #openstack-ansible | 15:48 | |
mhayden | good point, odyssey4me | 15:49 |
mhayden | i never get those ssh erorrs when i do a build on a plain ubuntu 14.04 rax image | 15:49 |
mhayden | okay, i'm going to tear myself away from the computer for a bit | 15:50 |
mhayden | :) | 15:50 |
evrardjp | odyssey4me: that's why I mentionned it's possible to use ENV variables | 15:50 |
evrardjp | stick to safe defaults for everyone | 15:50 |
evrardjp | and tweak for gate checks | 15:51 |
odyssey4me | ok, I'm going to work with pabelanger to switch the integration tests to use the 'ubuntu-trusty' image, as we have in https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L6777 | 15:52 |
evrardjp | don't forget that ppl can also tweak their ansible.cfg for production (while not recommended, because it's in repo code) | 15:52 |
odyssey4me | I suspect that the switch over, and some cruft removal, will also resolve issues for the role functional tests | 15:52 |
evrardjp | (but this could also move because we are using a bootstrap ansible that could copy this file...) | 15:53 |
evrardjp | ok | 15:53 |
evrardjp | nice | 15:54 |
pabelanger | nice | 15:54 |
pabelanger | odyssey4me: what does that job do today? | 15:54 |
odyssey4me | pabelanger is there a reason to prefer using the devstack-ubuntu image over using the ubuntu-trusty image for role functional tests? | 15:55 |
*** phiche has quit IRC | 15:55 | |
odyssey4me | pabelanger gate-openstack-ansible-commit-nv and gate-openstack-ansible-dsvm-commit are exactly the same in execution, but use different images - the nv job uses ubuntu-trusty and the dsvm job uses devstack-trusty | 15:56 |
odyssey4me | ubuntu-trusty is a newer d-i-b based image in all cloud providers, whereas devstack-trusty is prepared differently in each provider | 15:57 |
odyssey4me | the images look different | 15:57 |
*** subscope has joined #openstack-ansible | 15:57 | |
pabelanger | odyssey4me: eventually all trusty images (minus devstack-) will be removed. So, using devstack-* nodes is for future proofing | 15:57 |
pabelanger | -infra will basically have a single image for trusty, centos7, fedora-21, etc | 15:58 |
pabelanger | they are a little different | 15:58 |
pabelanger | difference is being build using diskimage-builder and native cloud imagtes | 15:59 |
pabelanger | images* | 15:59 |
odyssey4me | pabelanger odd, I was told otherwise - I heard that ubuntu-trusty was the way forward | 15:59 |
*** subscope has quit IRC | 15:59 | |
mattt | cloudnull: going to have to test that one as i'm not familiar enough with the python code to simply review | 15:59 |
pabelanger | odyssey4me: Ya, it is confusing. You might be right, either way, they are condensing to a single image. I should confirm again to see what it is | 16:00 |
odyssey4me | pabelanger condensing to a d-i-b based image that's consistent between clouds helps... you're always testing on the same thing | 16:01 |
pabelanger | odyssey4me: right. Last I looked not may were using ubuntu-trusty, should check again and see how is using it | 16:02 |
pabelanger | downside, there is no centos-7 dib yet | 16:02 |
pabelanger | should try and get it going | 16:02 |
pabelanger | odyssey4me: haha, looks like you are the only team that is currently using it :) | 16:03 |
odyssey4me | pabelanger yeah, afaik the new images were still being 'beta' tested - we dived in to try them out, hence the -nv job :) | 16:03 |
pabelanger | everything else is experimental | 16:03 |
*** kmARC_2 has quit IRC | 16:03 | |
odyssey4me | as you know, no-one will use it until they're forced tp | 16:03 |
odyssey4me | *to | 16:03 |
pabelanger | right | 16:03 |
odyssey4me | everyone ready for a quick triage meeting? cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, dolphm, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak | 16:04 |
Sam-I-Am | yo | 16:04 |
pabelanger | so, that's why I defaulted to devstack-* nodes,dibs were not ready | 16:04 |
d34dh0r53 | \o/ | 16:04 |
*** Bjoern_ has joined #openstack-ansible | 16:04 | |
mattt | \o | 16:04 |
odyssey4me | any volunteers to lead the triage? | 16:05 |
cloudnull | o/ present | 16:05 |
odyssey4me | pabelanger I'd like to chat a bit later around how better to implement our jobs if you can spare the time? | 16:05 |
odyssey4me | in an hour or so | 16:05 |
Sam-I-Am | odyssey4me: looks like you're it :) | 16:07 |
odyssey4me | alright, let's do it: https://bugs.launchpad.net/openstack-ansible/+bug/1506285 | 16:07 |
openstack | Launchpad bug 1506285 in python-openstackclient "11.2.1 : openstack client with V3 auth causes usability issues" [Undecided,New] | 16:07 |
openstackgerrit | Merged openstack/openstack-ansible: Remove duplicated Aodh host var https://review.openstack.org/248696 | 16:07 |
pabelanger | odyssey4me: Sounds good. Should be back from walking daughter to school | 16:08 |
Sam-I-Am | this bug... did something change in the openstack client? | 16:09 |
odyssey4me | ok, so this bug was a mess | 16:10 |
Sam-I-Am | it is | 16:10 |
odyssey4me | it has not been particularly clear what the issue is | 16:10 |
odyssey4me | but this was registered, as per the last comment: https://bugs.launchpad.net/openstack-ansible/+bug/1518351 | 16:11 |
openstack | Launchpad bug 1518351 in openstack-ansible trunk "Playbooks stop when enabling keystone ldap backend " [Medium,In progress] - Assigned to Kevin Carter (kevin-carter) | 16:11 |
Bjoern_ | the issue is clearly stated | 16:11 |
*** Bjoern_ is now known as BjoernT | 16:11 | |
odyssey4me | so that has a patch in play which resolves the playbook issue, I believe - I've not yet tested it | 16:11 |
BjoernT | the openstack client did not change, the issue getting introduces when enabling multi domain in keystone | 16:11 |
jthorne | i thought the issue was pretty clear, with ldap enabled, you have to run "openstack user list --domain default" instead of "openstack user list". if you run "openstack user list" the user gets a 401. | 16:12 |
BjoernT | yeah among other thing like playbooks won | 16:12 |
BjoernT | 't work | 16:12 |
BjoernT | but I moved that into a different bug | 16:12 |
BjoernT | 1518351 | 16:12 |
odyssey4me | agreed, BjoernT - as stated above :) | 16:12 |
odyssey4me | jthorne can you please put that into https://bugs.launchpad.net/openstack-ansible/+bug/1506285 so that the keystone team can respond, as that is the exact info they need | 16:13 |
openstack | Launchpad bug 1506285 in python-openstackclient "11.2.1 : openstack client with V3 auth causes usability issues" [Undecided,New] | 16:13 |
cloudnull | jthorne: BjoernT: https://review.openstack.org/#/c/248900/ | 16:13 |
cloudnull | sorry in two meetings | 16:14 |
BjoernT | odyssey4me everything is in the bug | 16:14 |
odyssey4me | I suspect this is not exclusive to an ldap back-end, this is probably a more general multi-domain issue | 16:14 |
BjoernT | see #12 | 16:14 |
jthorne | cloudnull: oh my god i just clicked "open side-by-side all" in your review | 16:14 |
BjoernT | yes correct multi domain in general | 16:14 |
jthorne | so i submitted this which is similar to cloudnull's https://bugs.launchpad.net/openstack-ansible/+bug/1519174 | 16:15 |
openstack | Launchpad bug 1519174 in openstack-ansible "Set OpenStack service usernames and passwords to reference their own variables and work with LDAP Keystone Identity Backend" [Undecided,New] | 16:15 |
*** gparaskevas has quit IRC | 16:16 | |
odyssey4me | BjoernT so are you asking openstackclient to be modified to assume the default domain if no domain is given? | 16:16 |
*** mpavone has quit IRC | 16:17 | |
BjoernT | No all i'm asking is to see if we can do something about it since someone Mirantis claimed it might possibly work with a policy change | 16:17 |
BjoernT | Again, please READ #12 | 16:17 |
odyssey4me | right, I'm reading #12 and there is nothing actionable for openstack-ansible there as far as I can see | 16:18 |
BjoernT | yes when we decide to roll out a policy change | 16:18 |
BjoernT | Please assign it to Dolph and go on | 16:18 |
odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1517304 | 16:20 |
openstack | Launchpad bug 1517304 in openstack-ansible "os-swift is missing configuration for the object-reconstructor" [Undecided,New] | 16:20 |
odyssey4me | andymccr any thoughts on whether you'd consider this a bug or a wishlist item? | 16:21 |
andymccr | odyssey4me: id say wishlist | 16:22 |
odyssey4me | wishlist, alright | 16:23 |
*** mgoddard_ has quit IRC | 16:24 | |
odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1518162 | 16:24 |
openstack | Launchpad bug 1518162 in openstack-ansible "Set openrc_os_username to keystone_admin_user_name variable" [Undecided,New] | 16:24 |
jthorne | odyssey4me: ignore that | 16:24 |
jthorne | odyssey4me: i merged that into another bug report | 16:24 |
*** mgoddard_ has joined #openstack-ansible | 16:24 | |
evrardjp | k | 16:24 |
jthorne | i merged that into this one: https://bugs.launchpad.net/openstack-ansible/+bug/1519174 | 16:25 |
openstack | Launchpad bug 1519174 in openstack-ansible "Set OpenStack service usernames and passwords to reference their own variables and work with LDAP Keystone Identity Backend" [Undecided,New] | 16:25 |
jthorne | which may overlap with cloudnull's submission | 16:25 |
*** rromans is now known as rromans_afk | 16:26 | |
odyssey4me | cloudnull ping? | 16:28 |
cloudnull | yes? | 16:29 |
cloudnull | sorry totally split brained | 16:29 |
odyssey4me | cloudnull have you looked at the patch in https://bugs.launchpad.net/openstack-ansible/+bug/1519174 and how does it relate to https://review.openstack.org/248900 ? | 16:29 |
openstack | Launchpad bug 1519174 in openstack-ansible "Set OpenStack service usernames and passwords to reference their own variables and work with LDAP Keystone Identity Backend" [Undecided,New] | 16:29 |
*** rebase has joined #openstack-ansible | 16:30 | |
cloudnull | jthorne: this was already happening | 16:30 |
cloudnull | thats why i didnt add it to the patch to add the ldap bool | 16:31 |
cloudnull | IR https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_ceilometer/tasks/ceilometer_service_setup.yml | 16:31 |
cloudnull | calls https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_ceilometer/tasks/ceilometer_service_add.yml | 16:31 |
jthorne | oh i see | 16:31 |
cloudnull | so the service usernames and passwords are consistent | 16:32 |
jthorne | i gotcha | 16:32 |
jthorne | the other piece in my bug report is the "when: keystone_ldap not set" which has the same outcome as your patch | 16:34 |
cloudnull | this https://review.openstack.org/#/c/248900 and https://review.openstack.org/#/c/238509/ is all we should need to fix the ldap deployment as I can see so far | 16:34 |
*** rebase has quit IRC | 16:34 | |
cloudnull | yea I made it configurable per role and globally controlled in the group_vars | 16:34 |
jthorne | gotcha. | 16:34 |
odyssey4me | ok cloudnull shall we add this bug in the commit messages of those reviews, or shall we mark the bug as a dup? | 16:36 |
cloudnull | no their different issues but are required for our ldap configs | 16:36 |
cloudnull | their in the same thread, but solve different problems for sure. | 16:37 |
odyssey4me | ok, so I'll tweak the commit message of the reviews to include this one | 16:37 |
cloudnull | if you think its needed | 16:38 |
odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1518485 | 16:39 |
openstack | Launchpad bug 1518485 in openstack-ansible " lxcbr0 disappeared run-playbooks.sh " [Undecided,New] - Assigned to Anton Haldin (anton-haldin) | 16:39 |
evrardjp | disappeared :) | 16:39 |
evrardjp | not enough info, we don't even know the lxc installed version | 16:40 |
odyssey4me | yeah, the only way I can see this happening is if lxc was removed using a teardown or something | 16:40 |
evrardjp | discard my remark | 16:41 |
evrardjp | mmm this line should still happen: https://github.com/openstack/openstack-ansible/blob/master/scripts/run-playbooks.sh#L56 | 16:41 |
evrardjp | but it's definitely not enough info | 16:41 |
odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1518492 | 16:43 |
openstack | Launchpad bug 1518492 in openstack-ansible "Flawed volume delete behavior in teardown script" [Undecided,New] | 16:43 |
odyssey4me | this is a scope issue again - in my view the teardown should only remove things implemented by the playbooks... it should not destroy things prepared on the host by the bootstrap (assuming the bootstrap-aio script was used). | 16:44 |
odyssey4me | any volunteers to add a warning, or to shift the scope? | 16:45 |
evrardjp | I'll take it | 16:46 |
odyssey4me | thanks evrardjp | 16:46 |
evrardjp | np | 16:46 |
odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1518845 | 16:47 |
openstack | Launchpad bug 1518845 in openstack-ansible "cinder doesn't set a default volume type" [Undecided,New] | 16:47 |
odyssey4me | as I recall, Richard is planning to fix this himself | 16:48 |
odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1519053 | 16:49 |
openstack | Launchpad bug 1519053 in openstack-ansible "container backend not set correctly with Ansible v2" [Undecided,New] - Assigned to Jimmy McCrory (jimmy-mccrory) | 16:49 |
evrardjp | low because ansible2 | 16:49 |
evrardjp | but important for us to follow | 16:50 |
odyssey4me | yeah, jmccrory has self allocated it | 16:50 |
odyssey4me | I'm sure there'll be plenty more issue found. | 16:50 |
evrardjp | :D | 16:50 |
odyssey4me | next: https://bugs.launchpad.net/openstack-ansible/+bug/1519105 | 16:51 |
openstack | Launchpad bug 1519105 in openstack-ansible "Glance has wrong rpc_backend configured when ceilometer is enabled" [Undecided,New] - Assigned to Miguel Alejandro Cantu (miguel-cantu) | 16:51 |
*** ashishb has joined #openstack-ansible | 16:51 | |
odyssey4me | it looks like alextricity is on it already | 16:52 |
alextricity | yup | 16:52 |
odyssey4me | importance? | 16:52 |
alextricity | I've uncovered more problems though | 16:52 |
alextricity | medium | 16:52 |
*** javeriak has quit IRC | 16:53 | |
odyssey4me | alextricity more issues? | 16:53 |
alextricity | Yes. Serious ones too. The ceilometer-agent-central service never comes up, even though the playbooks successfully finish | 16:54 |
mattt | odyssey4me: regarding https://bugs.launchpad.net/openstack-ansible/+bug/1518845, you can set a default, but we don't automatically pick one for the deployment | 16:54 |
openstack | Launchpad bug 1518845 in openstack-ansible "cinder doesn't set a default volume type" [Medium,Confirmed] - Assigned to Richard Jones (r1chardj0n3s) | 16:54 |
alextricity | Seemingly it looks like the ceilometer service is running, however. | 16:54 |
alextricity | And upstart gives me *no info* as to why it doesn't come up | 16:54 |
alextricity | Whenever I run 'start ceilometer-agent-central', the services comes up, but then dies | 16:55 |
alextricity | And no log file is generated | 16:55 |
alextricity | I'll make a bug once I figure out the issue | 16:55 |
odyssey4me | alextricity ugh, I see that on one of my recent builds | 16:56 |
odyssey4me | let's rather register a bug and assign it to you | 16:56 |
alextricity | Okay. I'll do that | 16:57 |
odyssey4me | I'm doing some gate work and may be able to assist you there | 16:57 |
alextricity | Thanks bud :) | 16:57 |
odyssey4me | alextricity ping me the bug ref when you've registsred it please | 16:57 |
alextricity | Sure will | 16:57 |
odyssey4me | alright, that's the new bug list - any others? | 16:57 |
*** javeriak has joined #openstack-ansible | 16:57 | |
odyssey4me | mattt you wanted to go back to https://bugs.launchpad.net/openstack-ansible/+bug/1518845 ? | 16:58 |
openstack | Launchpad bug 1518845 in openstack-ansible "cinder doesn't set a default volume type" [Medium,Confirmed] - Assigned to Richard Jones (r1chardj0n3s) | 16:58 |
mattt | odyssey4me: yeah, just pointing out that it is possible to set that | 16:58 |
mattt | odyssey4me: as i htink richard is suggesting we do that in our role | 16:58 |
odyssey4me | mattt yeah, as I recall he wanted to fix it to get familiar with the code | 16:59 |
mattt | odyssey4me: oh! ok great | 16:59 |
odyssey4me | perhaps you could add some triage info to the bug report to help him along the way? :) | 16:59 |
mattt | sure | 16:59 |
*** javeriak_ has joined #openstack-ansible | 17:00 | |
*** javeriak has quit IRC | 17:00 | |
evrardjp | done? | 17:00 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible: Improve message of the teardown.sh https://review.openstack.org/249325 | 17:00 |
*** mgoddard_ has quit IRC | 17:00 | |
*** slotti has quit IRC | 17:01 | |
*** mgoddard has joined #openstack-ansible | 17:01 | |
odyssey4me | alright everyone, thank you all for your time! | 17:01 |
spotz | evrardjp question on that patch you just submitted | 17:02 |
evrardjp | mmm yes? | 17:02 |
spotz | Are you talking about a single lxc or multiple when you're talking about the lvm group? | 17:03 |
evrardjp | lxc volume group | 17:03 |
spotz | Just trying to figure out if the ' is in the right place:) | 17:04 |
evrardjp | yeah english is not my mother language | 17:04 |
spotz | I think you want lxcs' if you're talking in the plural | 17:04 |
evrardjp | let me rephrase that | 17:04 |
spotz | k | 17:04 |
openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible: Improve message of the teardown.sh https://review.openstack.org/249325 | 17:06 |
evrardjp | avoided the 's ;) | 17:06 |
*** sdake_ has joined #openstack-ansible | 17:07 | |
evrardjp | spotz: it's one vg, so it should be lxcs' | 17:07 |
*** javeriak_ has quit IRC | 17:07 | |
evrardjp | but I think it's more clear now | 17:07 |
*** sdake has quit IRC | 17:07 | |
evrardjp | clearer | 17:07 |
*** javeriak has joined #openstack-ansible | 17:09 | |
spotz | yeah but if you're talking about multiple lxcs it's lxcs' vs lxc's. Hence the question:) | 17:09 |
spotz | sweet perfect:) | 17:10 |
spotz | grammar just sucks, I personally love run on sentances | 17:11 |
*** javeriak_ has joined #openstack-ansible | 17:13 | |
*** javeriak has quit IRC | 17:13 | |
*** adac has quit IRC | 17:15 | |
*** eil397 has joined #openstack-ansible | 17:15 | |
*** alkari has joined #openstack-ansible | 17:16 | |
*** sdake_ has quit IRC | 17:17 | |
*** javeriak_ has quit IRC | 17:19 | |
*** javeriak has joined #openstack-ansible | 17:23 | |
*** bangfrog has joined #openstack-ansible | 17:45 | |
openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Fixes playbook runtime issues with ldap https://review.openstack.org/248900 | 17:48 |
*** javeriak_ has joined #openstack-ansible | 17:48 | |
openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Keystone domain fix https://review.openstack.org/238509 | 17:49 |
*** karimb has quit IRC | 17:49 | |
openstackgerrit | Jesse Pretorius proposed openstack/openstack-ansible: Keystone domain fix https://review.openstack.org/238509 | 17:50 |
bgmccollum | odyssey4me: most of the SSH issues i saw were when ansible forks were > sshd max sessions / max startups... | 17:50 |
*** javeriak has quit IRC | 17:51 | |
bgmccollum | been meaning to work on figuring out the best forks values based on sshd config, but it requires fetching the lesser value across all the infra nodes. | 17:54 |
odyssey4me | bgmccollum yeah, this time it's different - it seems to relate to pipelining | 17:54 |
*** sacharya has joined #openstack-ansible | 18:00 | |
cloudnull | odyssey4me: this is the package diff between dsvm and nv http://paste.openstack.org/show/479906/ | 18:02 |
*** sacharya has quit IRC | 18:04 | |
*** elgertam has joined #openstack-ansible | 18:06 | |
bgmccollum | alextricity: when ive seen services start, immediately die, and have no logs, its been a permission issue on the logging path...might be something to check | 18:09 |
cloudnull | odyssey4me: i think we may have broken pipelining | 18:10 |
cloudnull | i found this | 18:11 |
cloudnull | https://github.com/ansible/ansible/pull/9246 | 18:11 |
cloudnull | which was a closed issue by dehan | 18:11 |
cloudnull | but documented that if you have ssh_args pipelining may not work | 18:11 |
cloudnull | and we have https://github.com/openstack/openstack-ansible/blob/master/playbooks/ansible.cfg#L25 | 18:11 |
openstackgerrit | Kevin Carter proposed openstack/openstack-ansible: Remove ssh args to ensure pipelining is working https://review.openstack.org/249348 | 18:15 |
cloudnull | ^ testing the theory. | 18:16 |
*** elgertam1 has joined #openstack-ansible | 18:20 | |
bgmccollum | cloudnull: did you see this comment? -- https://github.com/ansible/ansible/issues/7251#issuecomment-57888223 | 18:21 |
cloudnull | yes | 18:22 |
*** elgertam has quit IRC | 18:22 | |
cloudnull | i've been diving into the connection plugin and found https://github.com/ansible/ansible/blob/v1.9.4-1/lib/ansible/runner/connection_plugins/ssh.py#L68-L72 | 18:22 |
cloudnull | which then lead me to good that found that issue | 18:23 |
bgmccollum | cloudnull : i see ControlMaster and ControlPersist defined, but not ControlPath | 18:23 |
bgmccollum | in the ansible.cfg ssh_args | 18:23 |
cloudnull | yes, it was not being used | 18:23 |
cloudnull | but with that line removed it would be | 18:23 |
cloudnull | this little bit may be biting us https://github.com/ansible/ansible/blob/v1.9.4-1/lib/ansible/runner/connection_plugins/ssh.py#L65-L68 | 18:24 |
*** elgertam1 has quit IRC | 18:24 | |
cloudnull | i guess we could simply add the ControlPath but im questioning what benifit we get from the other bits | 18:25 |
odyssey4me | cloudnull yeah, it seems that we're pretty much using the defaults anyway | 18:28 |
bgmccollum | were all these ssh options added when trying to diagnose the other ssh failure from the past? | 18:29 |
odyssey4me | cloudnull fyi the other bits were added to combat the forks issue before we knew it related to forks | 18:29 |
odyssey4me | bgmccollum ++ | 18:29 |
bgmccollum | bingo | 18:29 |
odyssey4me | so let's rip them out! | 18:29 |
bgmccollum | nuke it | 18:29 |
cloudnull | so we'll see if the test works | 18:29 |
cloudnull | i was digging through the ansible code before I found that nonsense. . . While I love ansible I HATE their coding style (or lack there of)... | 18:30 |
odyssey4me | the only other issue plaguing the gate is failed tempest runs, which I expect should be resolved by https://review.openstack.org/248887 for now while I work on further tweaks to split the gate up | 18:30 |
odyssey4me | pabelanger you mean like this? https://review.openstack.org/248774 | 18:36 |
pabelanger | odyssey4me: Ya, I suspect that is the issue. I currently use sudo: yes in my roles, but need to convert to newer syntax | 18:37 |
pabelanger | I should test that too | 18:37 |
odyssey4me | pabelanger that just failed too :/ | 18:37 |
pabelanger | odyssey4me: how did you run your roles before, under root directly or via sudo? | 18:43 |
cloudnull | pabelanger: in the new roles they were being executed directly as root | 18:48 |
pabelanger | ya | 18:48 |
pabelanger | so that's the difference | 18:48 |
cloudnull | tox -e ... | 18:48 |
pabelanger | we are using non-root for tox -e | 18:48 |
pabelanger | so: https://github.com/openstack/openstack-ansible-apt_package_pinning/blob/master/tasks/main.yml#L16 needs sudo: yes | 18:49 |
pabelanger | or what ever the new syntax is | 18:49 |
cloudnull | so we can, in our inventory add a sudo option to escalate | 18:49 |
pabelanger | ya | 18:49 |
cloudnull | ha great minds | 18:49 |
pabelanger | don't know which why is better honestly :) Should read up on it and see what people default too | 18:49 |
pabelanger | sudo: yes was pretty easy to do | 18:49 |
cloudnull | we should be able to do that in inventory without effecting the role | 18:49 |
cloudnull | https://github.com/openstack/openstack-ansible-apt_package_pinning/blob/master/tests/inventory | 18:50 |
cloudnull | ^ adding it to there should be all we need to do | 18:50 |
cloudnull | IE what evrardjp did here https://review.openstack.org/#/c/248774/1/tests/inventory,cm | 18:50 |
pabelanger | ya, so this should work: https://review.openstack.org/#/c/248774/ | 18:51 |
cloudnull | do we have a place to test without impacting the jobs ? | 18:52 |
pabelanger | not sure I follow | 18:53 |
*** eil397 has quit IRC | 18:53 | |
cloudnull | for the gate | 18:53 |
cloudnull | i mean i can spin up an env add a user with sudo no pass and test | 18:54 |
cloudnull | but i didnt know if we had an instance to beat on while debuging the gate | 18:54 |
*** sacharya has joined #openstack-ansible | 18:56 | |
pabelanger | Ah | 19:05 |
pabelanger | nothing yet | 19:05 |
pabelanger | for me, I have a simple playbook to fire up a node on my local cloud | 19:05 |
evrardjp | cloudnull odyssey4me bgmccollum, would you mind explaining your ansible findings, I'm not sure to have understood | 19:07 |
evrardjp | at some point we added ssh_args, and it broke pipelining, that's what you mean? | 19:08 |
*** skamithi13 has joined #openstack-ansible | 19:09 | |
*** alejandrito has quit IRC | 19:10 | |
cloudnull | evrardjp: o/ | 19:11 |
cloudnull | so yes a long while back we had added ssh_args to the system | 19:11 |
bgmccollum | well technically, it may have broken Control* settings... | 19:11 |
evrardjp | I went through that | 19:11 |
evrardjp | didn't I speak about it on this chan? | 19:11 |
cloudnull | idk. and if you did i missed it ... :( | 19:12 |
evrardjp | anyway, pipelining is the foe of reliability | 19:12 |
evrardjp | let me find an article for you | 19:13 |
evrardjp | I can't find anything better than that for the moment: http://www.anchor.com.au/blog/2010/02/ssh-controlmaster-the-good-the-bad-the-ugly/ | 19:17 |
evrardjp | maybe I'm completely out of scope | 19:18 |
evrardjp | I should read how pipelining is implemented | 19:18 |
*** vCloudernBeer has joined #openstack-ansible | 19:21 | |
bgmccollum | evrardjp: my understanding is that pipelining is an ansible thing...where as controlmaster is an ssh thing | 19:26 |
evrardjp | isn't pipelining a fancy name to put controlmaster controlpath ... arguments for connections? | 19:27 |
bgmccollum | evrardjp: https://github.com/ansible/ansible/issues/7251#issuecomment-57888223 | 19:27 |
evrardjp | ok my bad | 19:29 |
bgmccollum | no worries...its all very confusing | 19:29 |
evrardjp | so, wouldn't it be better for the future, to remove our specific ansible.cfg configuration changes (ssh_args and pipelining True), to be with safe defaults, and enable pipelining for the gate checks? | 19:30 |
evrardjp | or for AIOs for example | 19:30 |
bgmccollum | evrardjp: cloudnull has a patch up with just the ssh_args removal. i think after a couple rechecks to confirm no ssh failures. if there are still failure, we might look at disabling pipelining, or both... | 19:31 |
odyssey4me | evrardjp you'll notice that the sudo method din't work in https://review.openstack.org/248774 - are you sure that's done right? | 19:33 |
odyssey4me | cloudnull pabelanger ^ | 19:33 |
evrardjp | I'm not sure | 19:35 |
*** javeriak has joined #openstack-ansible | 19:35 | |
evrardjp | I followed the docs | 19:35 |
*** ashishb has quit IRC | 19:36 | |
*** javeriak_ has quit IRC | 19:36 | |
odyssey4me | evrardjp are you sure that become_ is not an ansible 2 thing? | 19:37 |
palendae | Pretty sure become is a 1.9+ thing | 19:38 |
evrardjp | I'll check on my host | 19:38 |
odyssey4me | ok, I'm out for the night - enough is enough | 19:39 |
odyssey4me | cloudnull FYI it seems that the upsteam LXC bug related to https://review.openstack.org/196702 has been fixed, so I've restored the patch - time to give that a try again | 19:40 |
evrardjp | I'll check tomorrow, sorry for that | 19:42 |
cloudnull | odyssey4me: we'll likely need to make sure we have a newer kernel too | 19:44 |
bgmccollum | anyone have some examples of recent tempest failures in the gate? | 19:44 |
odyssey4me | cloudnull perhaps just make it run only if the kernel is newer than x | 19:44 |
odyssey4me | in my cloud server spun up a day or two ago, it was present | 19:45 |
odyssey4me | bgmccollum several, most of the -v's are related to either tempest or ssh | 19:45 |
odyssey4me | bgmccollum that's why I added https://review.openstack.org/248887 to reduce resource usage | 19:46 |
bgmccollum | odyssey4mel: tempest failures specifically...i know there is an issue on RAX pub cloud where libvirt cant identify the CPU (based of a know set of cpu flags)...so nova compute doesn't start...vms won't boot...just wanted to see if any of those failures were related to that...i have a modified libvirt cpu_map.xml that allows libvirt to identify the cpu make / model and fully startup... | 19:48 |
bgmccollum | odyssey4me: ^ | 19:48 |
bgmccollum | not all hosts exhibit the problem though... | 19:49 |
odyssey4me | oh, that's interesting | 19:49 |
*** sdake has joined #openstack-ansible | 19:49 | |
odyssey4me | so we do have the occasional scheduler failures due to no host found | 19:49 |
bgmccollum | thats probably it | 19:50 |
odyssey4me | if you think those are related to libvirt, then perhaps that's worth adding for gate checks | 19:50 |
*** adac has joined #openstack-ansible | 19:51 | |
odyssey4me | bgmccollum can you push up a WIP review which I can work from | 19:51 |
odyssey4me | I'll rearrange where it should go and how it can be applied - we can give that a go and see. | 19:51 |
*** sdake_ has joined #openstack-ansible | 19:52 | |
bgmccollum | odyssey4me: this is what needs to be added to the cpu_map.xml -- https://github.com/cloud-training/rpc-heat-ansible/blob/master/playbooks/files/usr/share/libvirt/cpu_map.xml#L337-L381 | 19:52 |
bgmccollum | then you just bounce libvirt and nova-compute | 19:52 |
bgmccollum | https://github.com/cloud-training/rpc-heat-ansible/blob/master/playbooks/rpc-11.0-playbook.yml#L481-L496 | 19:52 |
pabelanger | odyssey4me: worked for me: https://review.openstack.org/#/c/249376/2 | 19:53 |
odyssey4me | cool, thanks - I'll factor that in | 19:53 |
bgmccollum | odyssey4me: ill cook up something | 19:53 |
odyssey4me | pabelanger thanks - I'll have to look into it tomorrow | 19:54 |
odyssey4me | thanks for confirming though | 19:54 |
odyssey4me | and bgmccollum thanks for the heads up - that's super useful! | 19:54 |
*** sdake has quit IRC | 19:54 | |
*** jwagner is now known as jwagner_away | 19:55 | |
bgmccollum | np | 19:55 |
odyssey4me | now I really am outta here | 19:55 |
bgmccollum | go | 19:55 |
vCloudernBeer | meeting done? | 19:58 |
*** BjoernT is now known as Bjoern_zZzZzZzZ | 20:03 | |
*** ashishb has joined #openstack-ansible | 20:04 | |
cloudnull | vCloudernBeer: yup, we meet at 1600 utc for bug triage | 20:05 |
*** sacharya has quit IRC | 20:06 | |
vCloudernBeer | <cloudnull> thanks. | 20:06 |
bgmccollum | cloudnull: mind if i throw a couple rechecks at the ssh_args patch? see if we can get it to tip over... | 20:06 |
cloudnull | sorry | 20:06 |
cloudnull | sure by all means | 20:06 |
bgmccollum | done | 20:06 |
bgmccollum | ill keep an eye on it | 20:06 |
*** sdake_ has quit IRC | 20:15 | |
*** Bjoern_zZzZzZzZ is now known as BjoernT | 20:15 | |
*** sdake has joined #openstack-ansible | 20:15 | |
*** greg_a has quit IRC | 20:16 | |
*** skamithi13 has quit IRC | 20:17 | |
*** javeriak has quit IRC | 20:22 | |
*** vCloudernBeer has quit IRC | 20:22 | |
*** jwagner_away is now known as jwagner | 20:32 | |
*** ashishb has quit IRC | 20:34 | |
*** sacharya has joined #openstack-ansible | 20:44 | |
*** eil397 has joined #openstack-ansible | 20:47 | |
*** greg_a has joined #openstack-ansible | 20:50 | |
*** matt6434 is now known as mattoliverau | 20:52 | |
*** b3rnard0 is now known as b3rnard0_away | 20:53 | |
*** sdake has quit IRC | 20:59 | |
*** sdake has joined #openstack-ansible | 20:59 | |
*** phiche has joined #openstack-ansible | 21:01 | |
*** karimb has joined #openstack-ansible | 21:11 | |
*** adac has quit IRC | 21:16 | |
*** adac has joined #openstack-ansible | 21:30 | |
*** sacharya has quit IRC | 21:41 | |
*** sacharya has joined #openstack-ansible | 21:43 | |
*** elo has quit IRC | 21:46 | |
*** jaypipes has quit IRC | 21:47 | |
*** sacharya has quit IRC | 21:47 | |
*** alkari has quit IRC | 21:54 | |
*** elo has joined #openstack-ansible | 21:54 | |
evrardjp | FYI I did a quick check of the variables in the inventory, it works on my vagrant machine | 21:56 |
evrardjp | but it doesn't seem to work with current tox configuration | 21:57 |
evrardjp | or playbook | 21:57 |
evrardjp | or whatever | 21:57 |
evrardjp | I'll check further tomorrow | 21:57 |
evrardjp | the ansible_become_ variables* | 21:58 |
evrardjp | oh another quick check seem that setting a remote_tmp folder in ansible.cfg makes the ansible-playbook run by tox working on my local box | 22:01 |
*** adac has quit IRC | 22:15 | |
*** zaina has joined #openstack-ansible | 22:19 | |
*** jthorne has quit IRC | 22:30 | |
*** zaina has quit IRC | 22:43 | |
*** zaina has joined #openstack-ansible | 22:45 | |
*** b3rnard0_away is now known as b3rnard0 | 22:56 | |
*** sacharya has joined #openstack-ansible | 22:57 | |
*** phiche has quit IRC | 23:04 | |
*** spotz is now known as spotz_zzz | 23:06 | |
*** tlian has joined #openstack-ansible | 23:26 | |
*** tlian has quit IRC | 23:44 | |
*** woodard_ has joined #openstack-ansible | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!