| *** weezS has quit IRC | 00:02 | |
| *** weezS has joined #openstack-ansible | 00:05 | |
| *** gouthamr_ has quit IRC | 00:07 | |
| *** gouthamr has joined #openstack-ansible | 00:11 | |
| *** weezS has quit IRC | 00:14 | |
| *** weezS has joined #openstack-ansible | 00:16 | |
| *** weezS has quit IRC | 00:18 | |
| *** pbandark has quit IRC | 00:30 | |
| *** markvoelker has quit IRC | 00:39 | |
| *** dxiri has quit IRC | 00:51 | |
| *** chyka has quit IRC | 00:55 | |
| *** chyka has joined #openstack-ansible | 00:55 | |
| *** chyka has quit IRC | 01:02 | |
| *** gouthamr has quit IRC | 01:04 | |
| *** savvas has quit IRC | 01:16 | |
| *** savvas has joined #openstack-ansible | 01:16 | |
| *** savvas has quit IRC | 01:21 | |
| *** savvas has joined #openstack-ansible | 01:23 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_container_create stable/pike: Updated from global requirements https://review.openstack.org/521214 | 01:34 |
|---|---|---|
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia stable/pike: Remove ignore error and unnecessary when https://review.openstack.org/520101 | 01:35 |
| *** markvoelker has joined #openstack-ansible | 01:40 | |
| *** weezS has joined #openstack-ansible | 01:42 | |
| gun1x | can i get ceph backup images to be stored on HDD and cinder rbd volumes to be stored on ssd ? | 01:43 |
| *** weezS has quit IRC | 01:44 | |
| *** mwynne has quit IRC | 01:52 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Tune-up the galera role for efficiency https://review.openstack.org/466827 | 01:57 |
| *** lihi has quit IRC | 01:57 | |
| cloudnull | gun1x: I want to say yes. however I 'm really no ceph expert. | 01:58 |
| *** lihi has joined #openstack-ansible | 01:58 | |
| *** oanson has quit IRC | 01:59 | |
| *** oanson has joined #openstack-ansible | 01:59 | |
| *** askb has quit IRC | 02:00 | |
| *** galstrom_zzz is now known as galstrom | 02:06 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Tune-up the galera role for efficiency https://review.openstack.org/466827 | 02:16 |
| *** galstrom is now known as galstrom_zzz | 02:21 | |
| *** germs has quit IRC | 02:34 | |
| *** germs has joined #openstack-ansible | 02:34 | |
| *** germs1 has joined #openstack-ansible | 02:37 | |
| *** germs has quit IRC | 02:39 | |
| *** aludwar has quit IRC | 03:50 | |
| *** gkadam has joined #openstack-ansible | 04:07 | |
| SamYaple | gun1x: yep | 04:11 |
| SamYaple | gun1x: really easy todo on lumionus | 04:11 |
| SamYaple | gun1x: not as easy on jewel and before, but completely possible | 04:11 |
| *** gkadam has quit IRC | 04:21 | |
| *** savvas has quit IRC | 04:30 | |
| *** savvas has joined #openstack-ansible | 04:31 | |
| *** m|y|k has joined #openstack-ansible | 04:46 | |
| *** m|y|k has quit IRC | 04:50 | |
| *** m|y|k has joined #openstack-ansible | 04:51 | |
| *** aludwar has joined #openstack-ansible | 04:54 | |
| *** germs1 has quit IRC | 05:19 | |
| *** lbragstad has joined #openstack-ansible | 05:29 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Tune-up the galera role for efficiency https://review.openstack.org/466827 | 05:32 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Improve the speed of execution for Galera https://review.openstack.org/521229 | 05:33 |
| *** savvas has quit IRC | 05:39 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Enable support for hardware that brings the network up slowly https://review.openstack.org/520186 | 05:41 |
| *** hachi_ has joined #openstack-ansible | 05:56 | |
| *** DanyC has joined #openstack-ansible | 06:43 | |
| *** lbragstad has quit IRC | 07:00 | |
| *** chyka has joined #openstack-ansible | 07:13 | |
| *** chyka has quit IRC | 07:17 | |
| mcarden | . | 07:19 |
| *** woodard has joined #openstack-ansible | 07:19 | |
| *** m|y|k has quit IRC | 07:29 | |
| *** m|y|k has joined #openstack-ansible | 07:44 | |
| *** jiaopengju has joined #openstack-ansible | 07:50 | |
| *** jiaopengju has quit IRC | 07:51 | |
| *** DanyC has quit IRC | 07:52 | |
| *** sxc731 has joined #openstack-ansible | 07:56 | |
| *** gkadam has joined #openstack-ansible | 08:22 | |
| *** jwitko has quit IRC | 08:49 | |
| *** chyka has joined #openstack-ansible | 09:02 | |
| *** chyka has quit IRC | 09:06 | |
| *** bhujay_ has joined #openstack-ansible | 09:08 | |
| *** jbadiapa has quit IRC | 09:16 | |
| *** SmearedBeard has joined #openstack-ansible | 10:04 | |
| *** mardim has quit IRC | 10:06 | |
| *** ivve has joined #openstack-ansible | 10:09 | |
| gun1x | SamYaple: is that possible with OSA also? | 10:10 |
| *** woodard has quit IRC | 10:19 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/pike: Force SUSE repo refresh and switch default mirror. https://review.openstack.org/521247 | 10:21 |
| odyssey4me | gun1x OSA uses ceph-ansible to get ceph deployed, and doesn't restrict the architecture in any way - so yes, you can do whatever ceph can do using the deployment tooling OSA provides | 10:31 |
| odyssey4me | you'll still have to figure out how to do it with ceph yourself - we don't have any shortcut formula | 10:32 |
| *** chyka has joined #openstack-ansible | 11:00 | |
| *** chyka has quit IRC | 11:05 | |
| *** sxc731 has quit IRC | 11:05 | |
| *** aluria has quit IRC | 11:16 | |
| *** woodard has joined #openstack-ansible | 11:26 | |
| *** woodard has quit IRC | 11:31 | |
| *** m|y|k has quit IRC | 11:37 | |
| *** bhujay_ has quit IRC | 11:38 | |
| gun1x | odyssey4me: no need, i plan to deploy ceph with osa. i will read the docu and try to figure it out. it's good to have object storage on hdd and block storage on ssd because object storage is usualy backup and cinder is required for compute | 11:38 |
| odyssey4me | gun1x sure, but with ceph you don't have to make a hard and fast choice... you can use ssd for caching transactions to hdd's too - and you can change how you configure your hardware over time too... it gives room to experiment | 11:41 |
| odyssey4me | but yes, in the beginning it's usually better not to overthink it - keep it simple, put good monitoring in to observe how it behaves, then experiment with variations until you find the sweet spot of cost vs benefit | 11:42 |
| odyssey4me | you can also implement multiple 'types' for your block storage if you want - so people can choose slow/medium/fast based on their own needs | 11:43 |
| odyssey4me | those can be backed by different hardware profiles | 11:43 |
| gun1x | odyssey4me: i started to use prometheus for monitoring. i use it at home, and i added a node_exporter on each vm template, so i deploy all vms with exprter ... and i got a cool dashboard for desktop and another one for VMs | 11:43 |
| gun1x | odyssey4me: prometheus has a ceph exporter but i didnt have time to test it. it should provide the info required to redesign if something is bad | 11:44 |
| *** chyka has joined #openstack-ansible | 11:53 | |
| odyssey4me | awesome, sounds good | 11:53 |
| *** SmearedBeard has quit IRC | 11:57 | |
| *** chyka has quit IRC | 11:57 | |
| *** m|y|k has joined #openstack-ansible | 12:13 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Allow deployment without glance-registry https://review.openstack.org/521251 | 12:20 |
| *** m|y|k has quit IRC | 12:22 | |
| *** m|y|k has joined #openstack-ansible | 12:22 | |
| *** m|y|k has quit IRC | 12:24 | |
| *** m|y|k has joined #openstack-ansible | 12:24 | |
| *** woodard has joined #openstack-ansible | 12:25 | |
| *** m|y|k has quit IRC | 12:28 | |
| *** woodard has quit IRC | 12:29 | |
| *** m|y|k has joined #openstack-ansible | 12:29 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Allow deployment without glance-registry https://review.openstack.org/521251 | 12:30 |
| *** lenserpgo has quit IRC | 12:30 | |
| *** SmearedBeard has joined #openstack-ansible | 12:32 | |
| odyssey4me | FYI to anyone coming online, it appears that all releasenotes jobs across openstack are failing - if you see that on your test results, please do not bother rechecking until they get it resolved | 12:50 |
| gun1x | odyssey4me: what do you use for deploying to physical servers? so that you have the compute with all 4 bridges? | 12:53 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Retry apt cache update https://review.openstack.org/521252 | 12:56 |
| *** chyka has joined #openstack-ansible | 12:56 | |
| odyssey4me | gun1x some use cobbler, some use a simple pxe boot system with a preseed, some use other systems | 12:57 |
| odyssey4me | the https://github.com/openstack/openstack-ansible-ops/tree/master/multi-node-aio tooling actually implements a pxe-boot/preseed combination which does that all and may be useful to reference | 12:57 |
| gun1x | odyssey4me: i find it easiert to pxe boot an ubuntu with DHCP and run a script on it that bootstraps the server for OSA... | 12:58 |
| odyssey4me | there it is - it looks like it templates out a net config onto the web server here: https://github.com/openstack/openstack-ansible-ops/blob/60b8cd7e18d69fcc0dbeece8952a7bab0bf4a6fe/multi-node-aio/playbooks/deploy-pxe.yml#L220-L227 | 13:00 |
| *** chyka has quit IRC | 13:00 | |
| odyssey4me | then it fetches it in a post install script here: https://github.com/openstack/openstack-ansible-ops/blob/ce29ea23d10b5467ff314adfb09ee618e09fef57/multi-node-aio/playbooks/pxe/debian/vm-post-install-script.sh.j2#L14 | 13:00 |
| gun1x | odyssey4me: oh, ansible can do for statement? cool! | 13:05 |
| gun1x | odyssey4me: yea, that is exactly what i wanted to do. deploy via pxe and have post install script | 13:06 |
| *** hachi_ has quit IRC | 13:08 | |
| *** dave-mccowan has joined #openstack-ansible | 13:09 | |
| *** hachi_ has joined #openstack-ansible | 13:09 | |
| *** dave-mccowan has quit IRC | 13:11 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Allow deployment without glance-registry https://review.openstack.org/521251 | 13:11 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Allow deployment without glance-registry https://review.openstack.org/521251 | 13:12 |
| *** woodard has joined #openstack-ansible | 13:27 | |
| *** woodard has quit IRC | 13:32 | |
| SmearedBeard | hey, Im including barbican with my deployment but hitting an issue with missing files, http://paste.openstack.org/show/626697/ | 13:44 |
| SmearedBeard | it looks the same as https://review.openstack.org/#/c/493739/ but this was closed with saying a fix was going in another direction | 13:44 |
| SmearedBeard | I pulled from stable/pike | 13:44 |
| odyssey4me | no, those two things have nothing to do with each other | 13:44 |
| odyssey4me | which OS are you deploying on to? | 13:44 |
| odyssey4me | at this time there's only support for ubuntu-xenial in the role: https://github.com/openstack/openstack-ansible-os_barbican/tree/master/vars | 13:45 |
| *** m|y|k has quit IRC | 13:45 | |
| *** m|y|k has joined #openstack-ansible | 13:46 | |
| odyssey4me | that is very possibly not hard to fix - the start would be to add an appropriate vars file for redhat and figure out the equivalent package list | 13:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Allow deployment without glance-registry https://review.openstack.org/521251 | 14:01 |
| *** cshen has joined #openstack-ansible | 14:12 | |
| *** cshen has quit IRC | 14:13 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Retry apt cache update https://review.openstack.org/521257 | 14:17 |
| SmearedBeard | odyssey4me so just cp debian.yml centos.yml and fix the package names? | 14:18 |
| odyssey4me | SmearedBeard that'd be a good start | 14:18 |
| odyssey4me | then run it and see what falls out next | 14:18 |
| odyssey4me | or you might get lucky and it'll just work (tm) ;) | 14:18 |
| SmearedBeard | odyssey4me sounds easy, and something I could definitely send upstream when done | 14:18 |
| odyssey4me | if you can push your changes up then we can help you figure out where things break | 14:19 |
| odyssey4me | FYI, the right file name would be 'redhat.yml' because it'll cover both RHEL and CentOS | 14:20 |
| SmearedBeard | odyssey4me cool.. how come postgresql-libs is needed? I dont recall seeing postgres is using somewhere | 14:23 |
| odyssey4me | SmearedBeard I can't recall - it may not be necessary, but it might also be a dependency for some sort of build thing | 14:24 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Allow deployment without glance-registry https://review.openstack.org/521251 | 14:25 |
| gun1x | odyssey4me: wheny you have 4 network interfaces on the compute/controller, how do you get ansible to bootstrap the bridges required for OSA? :D | 14:51 |
| gun1x | odyssey4me: i mean, how do you know from ansible which nic does what :D :D | 14:51 |
| SmearedBeard | odyssey4me I dont wanna jynx it but looking good so far | 14:57 |
| odyssey4me | gun1x that's exactly why we don't have something to automate the process - it's too specific to the environment, the hardware, etc | 15:01 |
| odyssey4me | it's especially hard if your hardware is not homogenous | 15:02 |
| odyssey4me | SmearedBeard awesome - may as well push a patch up while you wait ;) | 15:02 |
| *** sxc731 has joined #openstack-ansible | 15:02 | |
| gun1x | odyssey4me: this is annoying because i just want to do another osa deploy for testing and i have no ideea if i should waste half an hour to bootstrap with bash scripts or do some ansible to learn and get something done for long term configs | 15:05 |
| *** sxc731 has quit IRC | 15:08 | |
| SmearedBeard | odyssey4me might as well look at suse, as long as aws has that image for me to find packages | 15:09 |
| odyssey4me | gun1x do whatever you're most comfortable with right now, then later when the pressure if off you can iterate with something different/new | 15:09 |
| SmearedBeard | what version of suse does osad support? | 15:09 |
| odyssey4me | SmearedBeard you can just use vagrant - every repo has avagrantfile setup with the tests and all that | 15:10 |
| * SmearedBeard looking now | 15:10 | |
| odyssey4me | but for suse it's leap 423 IIRC | 15:10 |
| odyssey4me | but, for example, if you have vagrant and virtualbox - just do: vagrant up opensuse423 | 15:11 |
| odyssey4me | that will execute exactly the same tests as the stuff we use in openstack infra | 15:11 |
| odyssey4me | https://github.com/openstack/openstack-ansible-os_barbican/blob/master/Vagrantfile#L39 | 15:12 |
| *** m|y|k has quit IRC | 15:20 | |
| *** woodard has joined #openstack-ansible | 15:20 | |
| *** m|y|k has joined #openstack-ansible | 15:20 | |
| SmearedBeard | odyssey4me cool, I vaguely aware on how vagrant works, but had the change to actually use, so this will be good... my play failed but ill fix it and send after some testing | 15:21 |
| odyssey4me | SmearedBeard I suggest that regardless of whether it works or not, push up a patch before you finish up for the day. It's a good practise for a number of reasons - 1 is that it signals to others that you're working on it, so no-one else ends up doing the same thing. 2 is that someone else may be able to help figure out the things you haven't figure out yet and will give you feedback in the review. :) | 15:23 |
| odyssey4me | 3 is that if you never get around to completing it, it gives the opportunity for someone else to pick up where you lfet off | 15:24 |
| odyssey4me | *left | 15:24 |
| *** woodard has quit IRC | 15:24 | |
| *** sxc731 has joined #openstack-ansible | 15:28 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Use correct name for networking-odl git group https://review.openstack.org/521260 | 15:31 |
| *** m|y|k_ has joined #openstack-ansible | 15:35 | |
| *** savvas has joined #openstack-ansible | 15:36 | |
| *** m|y|k has quit IRC | 15:39 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-galera_server master: Tune-up the galera role for efficiency https://review.openstack.org/466827 | 15:39 |
| *** sxc731 has quit IRC | 15:55 | |
| savvas | http://paste.openstack.org/show/626708/ anyone else getting errors with building RabbitMQ servers on stable/pike checkout? | 16:02 |
| savvas | I don't have it on 16.0.2 release but on this one it just won't build, have tried starting fresh twice now | 16:02 |
| *** exodusftw has quit IRC | 16:02 | |
| odyssey4me | savvas I've not seen that | 16:02 |
| savvas | crashes on various points, the rabbitmq cookie check, the above paste | 16:02 |
| savvas | I've tried destroying and recreating the containers to let it rerun the playbook as well, but no luc | 16:03 |
| savvas | k | 16:03 |
| odyssey4me | cloudnull if you don't mind, I think it'd be good to see https://review.openstack.org/#/c/516002/26/group_vars/haproxy_all/haproxy.yml broken out into its own patch - I'm hitting issues with haproxy failing when building without some things in the inventory, and that solves it nicely | 16:03 |
| savvas | any suggestions on how to troubleshoot odyssey4me ? I've tried running the lxc destroy/create, I've tried running the rabbitmq-install by itself and I've tried destroying the rabbitmq containers and letting setup-hosts recreate them followed by running setup-infra again | 16:05 |
| odyssey4me | savvas not sure - take a look at what's going on in the containers I guess and try to figure out how to fix it | 16:06 |
| odyssey4me | I'm doing some other work and can't afford the distraction to dig into the logs in more detail. | 16:07 |
| savvas | ye did that, last error says process ain't running but when I enter the container it does seem to be | 16:07 |
| savvas | no worries, was just trying my luck here ;> | 16:07 |
| odyssey4me | hmm, perhaps it just needs a delay/retry loop or something to give it a bit more time to start? | 16:07 |
| savvas | perhaps, although I figured if I rerun the playbook, it should recognize that it is running hte next time. | 16:09 |
| savvas | I haven't seen the pid change | 16:09 |
| *** m|y|k_ is now known as m|y|k{afk} | 16:18 | |
| *** woodard has joined #openstack-ansible | 16:21 | |
| *** woodard_ has joined #openstack-ansible | 16:25 | |
| *** woodard has quit IRC | 16:26 | |
| *** woodard_ has quit IRC | 16:30 | |
| SmearedBeard | first time using Gerrit and it is.... different | 16:37 |
| savvas | http://paste.openstack.org/show/626711/ this is what I get when starting fresh on the rabbitmq playbook, it starts with the cookie issue which eventually will lead to http://paste.openstack.org/show/626708/ when rerunning it at a later stage | 16:39 |
| savvas | I'll format the nodes one more time, start fresh, see if I've overlooked anything but I don't think so | 16:40 |
| *** sxc731 has joined #openstack-ansible | 16:42 | |
| odyssey4me | SmearedBeard it is different, but once you get used to having the history of a patch in patch sets, it's hard to go back to github and other tools which don't have that | 16:56 |
| SmearedBeard | we'll see :) just running some tests now | 16:57 |
| *** viktor_ has joined #openstack-ansible | 16:58 | |
| *** viktor_ is now known as koltcov | 16:58 | |
| *** weezS has joined #openstack-ansible | 17:20 | |
| *** woodard has joined #openstack-ansible | 17:23 | |
| gun1x | logan-: where you the guy doing osa + ceph ? | 17:23 |
| gun1x | like all dya ? | 17:23 |
| gun1x | *day | 17:23 |
| bndzor | checked url history matching logan, found https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-prod-ceph.html | 17:24 |
| *** weezS has quit IRC | 17:24 | |
| gun1x | bndzor: you might want to check https://logan.protiumit.com/2016/08/02/openstack-ansible-ceph.html | 17:25 |
| gun1x | this is 2016, but maybe good info. its from his blog | 17:25 |
| bndzor | Awsome! Thanks | 17:25 |
| bndzor | I have about 40 machines i want to try running | 17:25 |
| gun1x | bndzor: for tests you should go with cirros | 17:26 |
| gun1x | bndzor: its a special distro for opesntack tests, really lightweight | 17:26 |
| bndzor | Alright | 17:26 |
| bndzor | is there anyway to make ansible to use more threads somehow, to speed it up? It feels like its really doing one thing at a time | 17:26 |
| *** woodard has quit IRC | 17:28 | |
| logan- | o/ | 17:28 |
| gun1x | bndzor: if you mean getting ansible to run playbooks faster, no, sorry. | 17:28 |
| logan- | the blog info is fairly outdated now that the ceph-ansible integration is done in OSA | 17:28 |
| logan- | https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-prod-ceph.html is the best doc to look at currently | 17:29 |
| bndzor | Yep, foudn that cheers logan- | 17:29 |
| logan- | be aware that only stable/pike and master currently support this config | 17:29 |
| logan- | 16.0.2 (latest pike tag) does not have a necessary commit to support that config | 17:29 |
| bndzor | Yeah thats what im going to setup | 17:29 |
| bndzor | oh, which one should i go for ? | 17:29 |
| odyssey4me | bndzor gun1x you can up the number of threads ansible uses, and you can customises the serialisation if it's a test environment or new deployment | 17:30 |
| logan- | stable/pike (which will be 16.0.3 at some point) | 17:30 |
| logan- | so instead of 'git checkout 16.0.2', you need to 'git checkout stable/pike' | 17:30 |
| odyssey4me | the playbooks are delierately serialised to minimise disruption when doing changes on an existing environment | 17:30 |
| gun1x | logan-: so atm we need to get pike upstream (unstable) to get that to work? | 17:30 |
| bndzor | gotcha, thanks | 17:30 |
| logan- | correct until the next pike tag (16.0.3) is cut, there will not be a tag which includes https://github.com/openstack/openstack-ansible/commit/9ade47f72f618b64dd4401572d6fb503f7c24864 | 17:31 |
| gun1x | oh shit: storage-infra_hosts: *infrastructure_hosts | 17:31 |
| gun1x | AT LAST | 17:31 |
| gun1x | no more code repeating | 17:31 |
| gun1x | NICE!! | 17:32 |
| odyssey4me | haha, yeah using yaml tags can be quite useful | 17:32 |
| bndzor | im doing a test of the b production setup, once im done il try out the ceph | 17:32 |
| gun1x | odyssey4me: yea man ... some parts of osa looks like you get payed per lines of code, no offence :D | 17:32 |
| bndzor | ceph-osd_hosts, are thoose the ones that contain the storage ? | 17:32 |
| *** sxc731 has quit IRC | 17:33 | |
| logan- | yes bndzor | 17:33 |
| bndzor | roger | 17:33 |
| odyssey4me | gun1x nothing stopped you from customising the groups to include a bunch of children and thus make your openstack_user_config smaller... but then you have to validate it on every upgrade | 17:33 |
| bndzor | used_ips, are thoose ranges defined for dhcp ? | 17:33 |
| odyssey4me | and we're targeting changing that whole system to a much simpler inventory this cycle... hopefully it gets done in time | 17:34 |
| bndzor | and, i see 3 cidr_networks and 4 used_ips in the ceph example | 17:34 |
| bndzor | but in thoose 4, 4 cidrs are used | 17:34 |
| odyssey4me | bndzor they're blocks to reserve IP's in the cidr's that get allocated to containers... the reservations are to prevent an ip being allocated that you're using for something else - like a router, switch, storage device or something | 17:35 |
| bndzor | Ah ok, makes sense | 17:35 |
| gun1x | odyssey4me: i am really bad with ansible. i should learn it inside out. | 17:35 |
| odyssey4me | https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.example#L73-L95 | 17:35 |
| odyssey4me | gun1x we all start somewhere - learning is continuous | 17:36 |
| bndzor | mm, i come from a xen and puppet environment so i have lots of things to read up on | 17:36 |
| gun1x | odyssey4me: i would learn all my life, than die ... but i will learn all my life, then die. :D | 17:37 |
| gun1x | than/then joke. nvm. sorry if that was too nerd | 17:38 |
| bndzor | One thing im a bit curious about is, lets say that i do a config with 10 compute nods, and i want to add in 10 more. How would it be done, append them to the config and just run the playbooks ? | 17:38 |
| odyssey4me | bndzor yep | 17:38 |
| bndzor | nice! | 17:38 |
| bndzor | but damn the ansible process is painfully slow :p | 17:39 |
| gun1x | bndzor: it gets the job dne without failing | 17:39 |
| bndzor | I did my own bash scripted installation, from clean machines to a full system running (without ceph tho) in less than 5 minutes, but it did not feel professional | 17:39 |
| gun1x | bndzor: you want to join a boat ... having proper deployment tools mean a lot. manually configuring stuff is bad. anyway osa is a good boat to join | 17:41 |
| odyssey4me | bndzor yeah, I hear you - we're doing some evolutions to speed things up, but of course we have plenty of existing environments and odd edge cases so we would rather be slow and careful | 17:41 |
| bndzor | odyssey4me: yeah, i guess its more flexible | 17:42 |
| odyssey4me | bndzor it's a little tricky to do, but it is possible to not use containers and things deploy very quickly... but then you have to deal with losing the ability to just trash a container if it's misbehaving and rebuild it | 17:43 |
| *** germs has joined #openstack-ansible | 17:43 | |
| bndzor | ah | 17:44 |
| bndzor | ah im gonna cancel this deployment and try out the ceph one | 17:46 |
| bndzor | Do i need to configure the drives somehow on the machines ? ( i have 3 machines, the raid is on /dev/sdb) | 17:46 |
| bndzor | and question is, higher replication in ceph, or raid0 on hosts ? | 17:47 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Disable glance-registry and the v1 API by default https://review.openstack.org/521264 | 17:47 |
| odyssey4me | that'll have to be for logan- to respond :) | 17:48 |
| bndzor | :) | 17:49 |
| logan- | bndzor: couple things -- first I would set osd_objectstore: bluestore in user_variables with your other ceph variables. bluestore is the new default backend in Luminous, but ceph-ansible is still defaulting to filestore. if you're building a new cluster now you should probably use bluestore unless you have a good reason not to. | 17:51 |
| logan- | i wouldn't raid the drives unless you really have a good reason. ceph will distribute the replicas on its own according to the placement policies you set and the typical recommendation is to just feed it raw block devices | 17:51 |
| logan- | if they're blank (no partition table etc) you're good to go, ceph-ansible will use 'ceph-disk prepare' to partition and configure them as osds | 17:52 |
| logan- | if they have old data, you can use 'ceph-disk zap /dev/sdX' to wipe them before you start running playbooks | 17:53 |
| bndzor | Checking the storage nodes now. thanks | 17:55 |
| bndzor | (im reinstalling all with maas) | 17:55 |
| bndzor | il just remove the drives from mounting on /storage and format them | 17:55 |
| gun1x | logan-: how is bluestore different than other backends ? | 17:56 |
| logan- | gun1x: nice writeup here on it: http://ceph.com/community/new-luminous-bluestore/ | 17:57 |
| logan- | the same hardware performance gains over filestore are 1.5-2x in many situations | 17:58 |
| gun1x | logan-: i think they got this idea from scaleio | 17:58 |
| gun1x | is it ok to use luminous since its not LTS? or is it? i mean, don't you have to update next year to new version? or is this easy on ceph ? | 18:02 |
| logan- | luminous is the current ceph LTS release | 18:02 |
| bndzor | so reading the doc, the table that shows IP assignments, i assume i should set it up on the hosts, correct ? Or is mgmt enough and ansible does the rest ? | 18:02 |
| gun1x | ach ok | 18:02 |
| logan- | bndzor: you'll need to configure the bridges on the hosts | 18:03 |
| bndzor | yep | 18:03 |
| bndzor | but do i assign the ips to them ? | 18:03 |
| logan- | yes | 18:03 |
| bndzor | alright | 18:03 |
| logan- | well, for management yes. i get your question now.. I don't think the hosts will need an IP on the other bridges necessarily | 18:04 |
| *** SmearedBeard has quit IRC | 18:04 | |
| bndzor | Yeah, thats a bit unclear :) But i can assign that no issues | 18:04 |
| logan- | gun1x: ceph upgrades have been pretty painless in my experience, but yeah I wouldn't recommend running the non-LTS releases in prod as they tend to be testing out features in development for the next LTS. | 18:06 |
| bndzor | is rsyslog server required, or can run it on infra1 for example ? | 18:08 |
| logan- | it is not required | 18:08 |
| bndzor | (its just for testing right now, so i dont need / require logging) | 18:08 |
| logan- | but yes you can place the container on one of your infrastructure hosts | 18:08 |
| bndzor | ah ok, thats nice. il try that then | 18:09 |
| odyssey4me | part of why the host group allocation is a bit verbose is to give the flexiblity to scale out as wide as you want, but also scale in as much as you want | 18:13 |
| *** koltcov has quit IRC | 18:14 | |
| bndzor | I like it, it also looks cleaner | 18:14 |
| savvas | bndzor: also make sure you define your drives in user_variables.yml or per host in openstack_user_config.yml if you use osd_scenario collocated | 18:15 |
| logan- | yup ^... or use something like this: https://github.com/ceph/ceph-ansible/blob/master/group_vars/osds.yml.sample#L84-L92 | 18:16 |
| bndzor | how do you mean savvas | 18:17 |
| savvas | your playbook will fail if you don't define your osd devices | 18:17 |
| logan- | you'll need to define which disks to provision as osds: https://github.com/ceph/ceph-ansible/blob/master/group_vars/osds.yml.sample#L72-L79 | 18:17 |
| savvas | or enable auto discovery | 18:17 |
| bndzor | Il just go for osd_auto_discovery: true | 18:18 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Disable glance-registry and the v1 API by default https://review.openstack.org/521264 | 18:20 |
| bndzor | Anything else i should think of? If not, im ready to start the deployment | 18:21 |
| *** woodard has joined #openstack-ansible | 18:21 | |
| savvas | bndzor: I am yet to run a successful setup on stable/pike checkout | 18:22 |
| savvas | so I am curious to see if you get any further | 18:23 |
| bndzor | What issues do you have | 18:23 |
| bndzor | ? | 18:23 |
| savvas | My rabbitmq containers won't setup | 18:23 |
| savvas | just started fresh on everything again, ended up redoing my ansible node as well as I had checked out different versions alrdy, wanted to be sure my files are clean | 18:24 |
| savvas | I also noticed certain processes will fail on setup-infrastructure if I don't run the haproxy playbook first | 18:24 |
| bndzor | i just rm -rf /opt /openstack /etc/openstack* and /usr/local/lib/* and /usr/local/bin/* :p | 18:24 |
| savvas | you should remove your /etc/ansible/roles as well from what I've been told | 18:25 |
| odyssey4me | savvas that's odd, because haproxy's setup is included in setup-infrastructure: https://github.com/openstack/openstack-ansible/blob/stable/pike/playbooks/setup-infrastructure.yml#L18 | 18:25 |
| bndzor | savvas: thanks for the heads up, will do | 18:25 |
| savvas | ye I noticed, but I've had it on 2 different playbooks already that my galera won't setup without running haproxy first | 18:26 |
| odyssey4me | yes, but galera is setup after that | 18:26 |
| *** woodard has quit IRC | 18:26 | |
| odyssey4me | if you're re-using hosts then it makes sense | 18:26 |
| savvas | yes I am | 18:26 |
| savvas | I run a 3 node all-in-one setup | 18:26 |
| savvas | infra, compute, storage | 18:26 |
| odyssey4me | because when you wipe the containers, you also wipe the apt cache service that the hosts are configured to use | 18:26 |
| savvas | mm could that expalin for the errors I am getting with rabbitmq as well odyssey4me? | 18:28 |
| odyssey4me | unlikely, as that only affects the hosts - and rabbit is in a container | 18:29 |
| savvas | I've also noticed various errors on getting pip files | 18:29 |
| savvas | where it only succeeds over the fallback url | 18:29 |
| odyssey4me | also IIRC your issue related to the cluster init failing, wasn't it? that sounds very specifically like r arabbit issue after it's been installed | 18:29 |
| savvas | I think that may have something to do with me using the same internal/external ip for haproxy | 18:29 |
| odyssey4me | well, that won't work unless you disable https on the public endpoints | 18:29 |
| savvas | I did that | 18:30 |
| odyssey4me | http and https cannot saher the same ip:port | 18:30 |
| savvas | http://paste.openstack.org/show/626717/ | 18:30 |
| odyssey4me | ok, but the pip things will fail if the repo isn't there and youve had the pip things configured before | 18:30 |
| savvas | ## Disable SSL due to intenral and external lb IP being the same | 18:30 |
| savvas | openstack_service_publicuri_proto: http | 18:30 |
| savvas | openstack_external_ssl: false | 18:30 |
| savvas | haproxy_ssl: false | 18:30 |
| odyssey4me | so yes, if you're not cleaning up properly, and wiping and rebuilding containers, then you're going to hit plenty of odd issues | 18:30 |
| savvas | I added this to my user_variables.yml | 18:30 |
| odyssey4me | which is why we keep recommending that test labs are rebuilt rather than cleaned up | 18:31 |
| savvas | I've rebuilt my ubuntu nodes every time | 18:31 |
| savvas | just the ansible node I've been rm -rfing the directories | 18:31 |
| odyssey4me | ok, then I'm not understanding what you're seeing - sounds more like you're going through a proxy or something? | 18:32 |
| savvas | http://paste.openstack.org/show/626718/ | 18:32 |
| savvas | this is where setup-infra crashes on now | 18:32 |
| savvas | when using 3 newly installed bare metal nodes for deployment | 18:32 |
| savvas | setup-hosts completes without errors, but can't get through setup-infra | 18:33 |
| odyssey4me | you're getting a 503 from haproxy for some reason: http://107.178.62.254:8181/os-releases/16.0.3/ubuntu-16.04-x86_64/get-pip.py | 18:33 |
| odyssey4me | why is that | 18:33 |
| savvas | ye that ain't right | 18:34 |
| savvas | let me due my diligence, sec | 18:34 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Disable glance-registry and the v1 API by default https://review.openstack.org/521264 | 18:34 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Disable glance-registry and the v1 API by default https://review.openstack.org/521264 | 18:38 |
| bndzor | finally deploying.. perhaps should go and make some tacos meanwhile | 18:38 |
| bndzor | 8) | 18:39 |
| savvas | lol odyssey4me alright this one's on me, when restoring I copy pasted the original container cidr | 18:39 |
| odyssey4me | haha, doit | 18:39 |
| savvas | will containers reinitialize networking if I rerun setup-hosts.yml with a changed container cidr? | 18:40 |
| *** woodard has joined #openstack-ansible | 18:40 | |
| odyssey4me | I'm not really familiar enough with what goes on there to answer. | 18:42 |
| savvas | may as well start over to be sure | 18:42 |
| savvas | I am thinking I may have had this error in there while I was going through the rabbitmq errors | 18:43 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Implement testing using tempest https://review.openstack.org/521268 | 18:43 |
| savvas | somewhere in between what I troubleshooted on the ceph-osd issue and after I ended up copy/pasting a few config lines from docs.oa to avoid any typos, it may have snuck in there | 18:44 |
| odyssey4me | networking is the most common problem area for new starters, unfortunately | 18:44 |
| odyssey4me | most openstack deployments use a very simple network setup, and suffer for it later | 18:44 |
| savvas | it is a good way to get familiar with it though | 18:44 |
| odyssey4me | we do tend to force thinking through it properly and gaining a fair understanding of networking up front | 18:45 |
| savvas | ye I can see that | 18:45 |
| *** m|y|k{afk} has quit IRC | 18:45 | |
| savvas | I just started with openstack ansible but I love the concept | 18:45 |
| savvas | especially the idea behind use of the LXC containers | 18:45 |
| savvas | it actually enables you to build fault-tolerant HA clusters with very few nodes | 18:46 |
| odyssey4me | yup, you can actually do it on a single node if you want, although that kinda defeats the purpose of being fault tolerant | 18:47 |
| savvas | exactly | 18:47 |
| savvas | ye I had a few runs on virtual box vms on my workstation at first | 18:47 |
| savvas | but virtualbox doesn't allow mac flooding so that presented a whole new challenge by itself | 18:47 |
| savvas | decided yesterday to bring up a few servers to actually try and get it working properly the wayit should be | 18:48 |
| odyssey4me | back in kilo we actually used to gate our integrated build with a setup which had multiple clusters on the same host, but we can't do that these days - all the services are too resource hungry for the little 8 vCPU/8GB RAM instance we use for testing | 18:48 |
| savvas | yes I noticed that as well | 18:48 |
| savvas | my playbooks would crash randomly with weird errors which turned out to be a resource problem | 18:49 |
| savvas | I couldn't get it running on a 8GB VM, think my first working infra was on a 10G VM | 18:49 |
| odyssey4me | by mac flooding you mean something like mac spoofing? ie the vm can't communicate to other vm's using anything other than its assigned mac from a virtualbox standpoint? | 18:49 |
| savvas | well the containers could communicate with the host | 18:49 |
| savvas | and the host could communicate with the other hosts in the cluster (the other virtual box vms) | 18:50 |
| savvas | but the LXC containers couldn't communicate to the other hosts | 18:50 |
| odyssey4me | yeah, I think it is possible with virtualbox - I've been meaning to spend some quality time working out some nice vagrantfile implementations to demonstrate OSA in various configurations | 18:50 |
| odyssey4me | but, well, time | 18:50 |
| savvas | ye I hear you | 18:51 |
| savvas | vagrant could be great for it | 18:51 |
| savvas | but from what I've read there will be no way to get it properly working on virtualbox though | 18:51 |
| odyssey4me | I find myself more inclined to develop a new feature, and hope that someone else gets the inclination to do it and wants to do it more than I do. | 18:51 |
| odyssey4me | heh, that sounds like a challenge ;) | 18:51 |
| savvas | well if you manage to do it, I'd surely be interested to figure out how you did it ;p | 18:52 |
| savvas | I've got the br-mgmt on the 3 nodes configured in a natnetwork with promiscuous mode allow all on all | 18:52 |
| savvas | I've tried bridging too | 18:53 |
| *** koltcov has joined #openstack-ansible | 18:54 | |
| odyssey4me | I may try it out some time, but it's unlikely to be soon. | 18:56 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Disable glance-registry and the v1 API by default https://review.openstack.org/521264 | 19:00 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Implement testing using tempest https://review.openstack.org/521268 | 19:00 |
| savvas | I'll give it another go as well when I get a chance | 19:00 |
| bndzor | hmm, it failed during the Ensure that the LXC cache has been prepared | 19:16 |
| bndzor | took too long i guess | 19:16 |
| openstackgerrit | michael rice proposed openstack/openstack-ansible-plugins master: fixes errors caused when using keystone federation https://review.openstack.org/521283 | 19:17 |
| bndzor | Any idea how i can increase the amount of retries ? | 19:20 |
| odyssey4me | bndzor I think it's hard-coded right now - I've been meaning to push a patch to make it customisable. | 19:21 |
| odyssey4me | default is 10 mins: https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/tasks/lxc_cache_preparation.yml#L111-L113 | 19:21 |
| bndzor | where do i edit that ? | 19:21 |
| odyssey4me | that should bein /etc/ansible/roles/lxc_hosts/tasks/... | 19:22 |
| *** hachi_ has quit IRC | 19:23 | |
| *** hachi_ has joined #openstack-ansible | 19:24 | |
| bndzor | thanks odyssey4me, gave it 600 times to try :p | 19:25 |
| bndzor | is it ok to just rerun setup-hosts.yml ? | 19:26 |
| bndzor | or do i need to cleanup somehow | 19:26 |
| errr | odyssey4me: I got the first one of the patches submitted for the keystone federation issues. There is still one more I think but Im having issues getting my current AIO to build so I can test it | 19:26 |
| odyssey4me | bndzor no cleanup, just re-run | 19:28 |
| odyssey4me | errr okie dokey, thanks! | 19:28 |
| bndzor | awsome | 19:29 |
| errr | odyssey4me: if there is anything I need to do please let me know, the customer install is coming up in the next 4 days or so so I need to try to get that into newton as soon as possible | 19:29 |
| *** dave-mccowan has joined #openstack-ansible | 19:31 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts master: Make the cache prep timeout configurable https://review.openstack.org/521313 | 19:33 |
| odyssey4me | errr that totally makes sense | 19:34 |
| odyssey4me | errr could you tweak the subject line to be a little more targeted - suggestion in review | 19:36 |
| errr | sure thing, thanks | 19:36 |
| odyssey4me | bndzor you may find https://review.openstack.org/521313 handy | 19:36 |
| openstackgerrit | michael rice proposed openstack/openstack-ansible-plugins master: Add missing domain parameter to _get_group_roles https://review.openstack.org/521283 | 19:37 |
| errr | done :) | 19:38 |
| odyssey4me | thanks, I see a lint error is showing up but that's unrelated | 19:38 |
| odyssey4me | I'll push a patch to resolve that tomorrow morning. Time for me to take a break tonight. | 19:39 |
| odyssey4me | Cheers all - have a great w/end! | 19:39 |
| bndzor | Cool, thanks! | 19:43 |
| bndzor | I just modified tasks/lxc_cache_preparation.yml for now | 19:43 |
| bndzor | THanks for the help odyssey4me, have a good evening | 19:43 |
| *** dave-mccowan has quit IRC | 19:43 | |
| bndzor | Hmm, so it completed, but for some reason it does nto resume.. its just keeping retrying | 20:03 |
| *** SmearedBeard has joined #openstack-ansible | 20:16 | |
| *** Smeared_Beard has joined #openstack-ansible | 20:28 | |
| *** SmearedBeard has quit IRC | 20:30 | |
| errr | Im having an issue on stable/pike building an AIO. The galera container keps failing trying to get pip and pip packages. | 20:33 |
| errr | I checked and there is an entry in haproxy pointing to the repo container, but when I check ss or netstat nothing is bound to port 8181 and if I try to restart haproxy it never restarts, and I cant stop it either.. | 20:35 |
| *** hachi_ has quit IRC | 20:43 | |
| *** koltcov has quit IRC | 21:03 | |
| *** Smeared_Beard has quit IRC | 21:07 | |
| bndzor | hmm | 21:40 |
| bndzor | http://paste.openstack.org/show/626720/ | 21:40 |
| bndzor | Any ideas ? | 21:40 |
| savvas | @odyssey4me fresh attempt with reinstalled bare metal crashes again on galera | 21:40 |
| savvas | http://paste.openstack.org/show/626721/ | 21:40 |
| savvas | ignore that | 21:43 |
| savvas | bndzor: is your partition full? | 21:45 |
| savvas | errr: what do you get when you "curl http://INTERNAL_LB_IP:8181 | 21:45 |
| bndzor | nothing is full | 21:47 |
| bndzor | but i rerun the playbook because of some issues i had | 21:47 |
| errr | savvas: connection refused, but I just destroyed the server and am trying again on another server. | 21:47 |
| bndzor | just ran a lxc-containers-destroy and giving it a try again to see if this fixes the issue | 21:48 |
| savvas | errr: check your haproxy.log to see if it establishes a connection with your containers, if not check the keepalived conf to see if your IPs are correct there and if so check that your container have connectivity | 21:52 |
| savvas | 8181 is for the repo container | 21:52 |
| savvas | so also check if /var/www/repo has files in it and that nginx is running | 21:53 |
| errr | savvas: it was like haproxy wasnt running properly, it was never bound to port 8181 on the main deploy host so it was never passing the connection to the container | 21:53 |
| errr | then it wouldnt restart or reload or even stop | 21:53 |
| errr | systemctl stop haproxy.service wouldnt stop it | 21:54 |
| bndzor | nope still happening | 21:54 |
| bndzor | gdamnit | 21:54 |
| savvas | ok ye probably best to restart the playbook or start fresh, I had an issue before but mine had to do with my keepalived settings | 21:55 |
| *** SmearedBeard has joined #openstack-ansible | 22:05 | |
| *** gouthamr has joined #openstack-ansible | 22:08 | |
| savvas | odyssey4me: does this ring any bell for you? The rabbitmq install fails, same errors as before. I notice that my physical nodes are able to acces the LB ip and also grab get-pip.py from it | 22:14 |
| savvas | however the containers are not able to, they can ping the LB ip but not grab any files from the repo url, connection refused | 22:14 |
| savvas | if I check my haproxy log | 22:16 |
| savvas | http://paste.openstack.org/show/626723/ | 22:17 |
| savvas | to me it sounds like I may have messed up my bridge settings somehow but I can't find my mistake | 22:21 |
| savvas | my galera containers can reach the repo over 8181 as well as the memcached containers | 22:25 |
| savvas | NONE of the others do, so my guess is that somewhere during setup of the playbook something is enabled on the containers that simply hasn't been done on the rabbitmq and any of the other containers which come after that | 22:26 |
| bndzor | rebooted all nodes, and seems like problem solved it self | 22:31 |
| *** gouthamr has quit IRC | 22:33 | |
| gun1x | odyssey4me: how do you up the number of ansible threads ? | 22:38 |
| *** askb has joined #openstack-ansible | 22:40 | |
| savvas | forget it thought I saw ap attern there but completely random. Destroying and re-creating all the rabbitmq hosts gets me 2 who can reach the LB IP, 1 that can't. | 22:42 |
| *** ajmaidak has quit IRC | 22:50 | |
| *** ajmaidak has joined #openstack-ansible | 22:51 | |
| *** dave-mccowan has joined #openstack-ansible | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!