16:00:25 <ajpower> #startmeeting kolla 16:00:28 <openstack> Meeting started Wed Aug 17 16:00:25 2016 UTC and is due to finish in 60 minutes. The chair is ajpower. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:32 <openstack> The meeting name has been set to 'kolla' 16:00:49 <rhallisey> ajafo, wrong location 16:01:05 <rhallisey> ajpower, , #openstack-meeting-4 16:01:12 <rhallisey> ajafo, sorry wrong person 16:04:33 <ajpower> #openstack-meeting-4 16:05:49 <openstackgerrit> Merged openstack/kolla: Replace horizon default config with custom config https://review.openstack.org/306928 16:07:31 <rhallisey> ajafo, `/join #openstack-meeting-4` 16:07:36 <rhallisey> ajpower, ^ 16:07:40 <rhallisey> sorry ajafo 16:07:44 <rhallisey> :/ 16:08:32 <ajpower> join #openstack-meeting-4 16:08:58 <rhallisey> ajafo, add a / 16:17:13 <kbaegis> So what's the procedure/strategy in kolla to upgrade a single container? 16:33:11 <kbaegis> Is there an individual runbook I can deploy to move horizon between versions, for example? 16:33:34 <rhallisey> kbaegis, you can run play specific for a service with tags 16:33:54 <rhallisey> so you can run the upgrade play with the horizon tags and that should get what you want 16:35:00 <kbaegis> so the command would be kolla-ansible upgrade horizon? 16:35:16 <kbaegis> or would I alter globals and select a horizon version? 16:36:55 <rhallisey> kbaegis, kolla-ansible upgrade --tags horizon 16:37:06 <kbaegis> Oh, nice :) 16:47:04 <kbaegis> Latest issue: http://hastebin.com/pijeqimofi.vbs 16:47:24 <kbaegis> Anyone played around with non-cooperative iptables implementations and willing to help? 16:48:11 <kbaegis> Tap device is created successfully, bridge too 16:48:22 <kbaegis> It's just these dang iptables rules :) 16:50:30 <kbaegis> Running the commands from the log against iptables manually I get errors like 16:50:34 <kbaegis> iptables: No chain/target/match by that name. 16:50:42 <kbaegis> or iptables: Index of insertion too big 16:53:26 <kbaegis> Huh. Maybe it needs CONFIG_INET_DIAG_DESTROY 16:53:29 <inc0> coolsvap, so ad rally 16:53:47 * britthouser listens in 16:53:49 <inc0> it should work now, we just need to change deployments 16:53:53 <duonghq> hey sdake_, the bug, do we have any replacement for deprecated tag option? 16:53:55 <inc0> cuz access creds are invalid 16:54:03 <ntpttr___> hey inc0 I'm running into an issue with my cinder/ceph deployment in kolla, do you have a moment? 16:54:08 <inc0> sure 16:54:13 <inc0> what's up Nate? 16:54:30 <sdake_> duonghq i dont have a clear picture of the issue yet duonghq 16:54:31 <ntpttr___> inc0: so everything is starting up fine and I can create and attach volumes and all, but the backup service isn't working 16:54:37 <sdake_> duonghq - but there are docs in teh queue addressing it 16:54:39 <sdake_> they are just wrong 16:54:49 <ntpttr___> inc0: basically whenever I try to backup a volume, it hangs in the 'creating' phase forever 16:54:52 <sdake_> duonghq so the solution first is to document a workaround 16:54:56 <coolsvap> inc0: hi 16:54:57 <inc0> and no errors? 16:54:57 <sdake_> duonghq then to fix the pbr issue 16:54:59 <ntpttr___> and it seems like privsep just isn't starting up 16:55:13 <coolsvap> inc0: where are we currently wrt rally 16:55:19 <ntpttr___> in the logs, here's mine: http://paste.openstack.org/show/558606/ and here's one where it works: http://paste.openstack.org/show/558537/ 16:55:33 <ntpttr___> the privsep daemon isn't starting, but there's no errors 16:55:54 <duonghq> sdake_: I look through the code and cannot find any replacement for deprecated option, I don't think its normal 16:56:02 <inc0> coolsvap, can you fix deployments in rally? our creds are wrong 16:56:03 <duonghq> sdake_ fix me if I wrong :) 16:56:27 <inc0> ntpttr___, it might be a bug 16:56:43 <coolsvap> inc0: sure let me connect to vpn 16:56:44 <inc0> I haven't seen this before 16:56:54 <ntpttr___> inc0: it looks like that might be the case, this is on a clean install with no errors or anything 16:57:12 <ntpttr___> has anyone here deployed cinder w/ ceph and then successfully used the backup service before? 16:57:45 <inc0> our tests on osic cluster should show that 16:58:57 <duonghq> sdake_: I think it should be a option in kolla-build.conf for tagging? 16:59:19 <sdake_> i dont think --tag should be a deprecated option at all 16:59:20 <ntpttr___> inc0: do you think I should go ahead and file a bug against kolla? 16:59:21 <sdake_> i dont know why it is 17:00:14 <coolsvap> inc0: we can update the credentials in osic.json file on the node 17:00:14 <inc0> ntpttr___, yeah, file it 17:00:16 <duonghq> sdake_: somebody marked it deprecated? 17:01:03 <inc0> coolsvap, might doing that? 17:01:07 <coolsvap> inc0: i am updating it from /etc/kolla/admin-openrc.sh 17:01:07 <inc0> I dunno where it is 17:01:08 <coolsvap> yes 17:01:23 <inc0> tmux a plz 17:01:28 <inc0> I'll watch it 17:01:35 <coolsvap> yeah sorry 17:01:38 <coolsvap> just a min 17:01:42 <inc0> np:) 17:03:53 <ntpttr___> okay got the bug report up https://bugs.launchpad.net/kolla/+bug/1614180 17:03:53 <openstack> Launchpad bug 1614180 in kolla "Backup w/ ceph hangs when creating a backup" [Undecided,New] 17:09:18 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP- openvswitch ip https://review.openstack.org/356114 17:10:21 <ntpttr___> inc0: do we need to be installing oslo.privsep somewhere? I don't see it anywhere in the kolla repo, or in this file where I see a lot of other oslo libraries https://github.com/openstack/kolla/blob/5f4ef0d84bf1d1f5f72f245a0dc9e95d25e0b92d/docker/openstack-base/Dockerfile.j2 17:10:37 <ntpttr___> in devstack they're installing it https://github.com/openstack-dev/devstack/blob/06f3639a70dc5884107a4045bef5a9de1fb725a5/lib/oslo 17:11:17 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP- openvswitch ip https://review.openstack.org/356114 17:19:49 <Pavo> evening everyone 17:20:35 <openstackgerrit> Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update cirros image visibility in init-runonce https://review.openstack.org/353933 17:26:08 <kbaegis> gah. Linuxbridging broken still on gentoo 17:27:01 <coolsvap> pbourke: do yo uwhave the kolla-rally-tests.sh script open in your terminal? 17:31:14 <kfox1111> sbezverk: rhallisey: so, for the glance bootstrap one. we postponed talking about dealing with the init-container templating thing for dealing with the json as we didn't have access to jq. that patch just merged though. 17:32:13 <sbezverk> kfox1111: I would suggest to merge what I have now as I tested it multiple times and it is working 17:32:17 <kfox1111> can we reconsider this: https://review.openstack.org/#/c/353740/4..6/services/neutron/neutron-bootstrap-job.yml.j2 17:32:17 <patchbot> kfox1111: patch 353740 - kolla-kubernetes - Neutron bootstrap to create service/project/role 17:32:27 <Pavo> kbaegis was it you asking about deploying only 1 of the images with kolla-ansible yesterday? 17:32:44 <rhallisey> kfox1111, I mean we can iterate on it 17:32:44 <sbezverk> if you feel like modifying it, please submit PS and we can have discussion 17:32:57 <kfox1111> k. 17:33:20 <sbezverk> it is just so many things are hanging because of this 17:34:10 <kfox1111> sure. but I just want to avoid having to do more when we add more bootstrapping to nova and friends too. 17:34:34 <sbezverk> kfox1111: I do not think we add more than just nova 17:34:39 <sbezverk> any time soon 17:34:48 <kfox1111> we can merge these 2 as is. 17:34:49 <kfox1111> cinder? 17:34:53 <kfox1111> heat? 17:34:58 <sbezverk> so if you propose PS after, I will help you to apply on other components 17:35:09 <kfox1111> k 17:35:51 <sbezverk> If I undersatand the primary goal is get VM 17:36:17 <kfox1111> ah. we can claim vm without cinder/heat, true. 17:36:46 <sbezverk> kfox1111 yep, I think it make sense 17:37:12 <kfox1111> one more thing and I think the glance one's good then. 17:37:22 <kfox1111> do we actually need the pvc in the bootstrap? 17:38:02 <kfox1111> I don't really see where its used, and it shouldn't be there if using the ceph backend. 17:38:09 <kfox1111> I think we can probably just drop it all together. 17:39:31 <sbezverk> kfox1111: I will need to retest it 17:40:22 <sdake_> pbourke you still around? 17:40:41 <sdake_> i dont blame ya 17:40:49 <sdake_> sweet dreams :) 17:42:58 <kfox1111> wait.. 17:43:03 <kfox1111> its used here: 17:43:26 <kfox1111> https://review.openstack.org/#/c/346215/11/services/glance/glance-bootstrap-job.yml.j2 line 76 17:43:27 <patchbot> kfox1111: patch 346215 - kolla-kubernetes - Glance bootstrap to create service/project/role 17:43:38 <kfox1111> whats the api container being used for? 17:44:32 <sbezverk> kfox1111: it looks like it is used during bootstrap process. 17:44:43 <kfox1111> where? 17:44:59 <kfox1111> I see database creation, and keystone registration. but not much else? 17:46:53 <sbezverk> kfox1111: it does this sudo chown -R glance: /var/lib/glance/ 17:47:27 <kfox1111> hmm.... 17:47:42 <kfox1111> oh. cause the KOLLA_BOOTSTRAP. 17:47:47 <sbezverk> kfox1111: if we do not have it mounted, it will be done against virtual directory 17:47:59 <sbezverk> kfox1111: right 17:48:13 <kfox1111> k. 17:48:17 <sbezverk> kfox1111: it is not much but please do not ask me to move it somewhere else 17:48:30 <kfox1111> so then I think we just need to do something like: https://review.openstack.org/#/c/354895/4/services/glance/glance-api-pod.yml.j2 17:48:30 <patchbot> kfox1111: patch 354895 - kolla-kubernetes - WIP - Deployment, Safe Shutdown, & Scaling for gla... 17:48:47 <kfox1111> with glance_backend_ceph 17:49:08 <kfox1111> around the volume and volumeMount. 17:49:35 <kbaegis> Pavo Yes :) 17:49:37 <sbezverk> kfox1111: can you add it on top after it is merged? 17:49:57 <sbezverk> I will not be able to test it 17:50:03 <sbezverk> I just do not have means 17:50:07 <kfox1111> k. 17:50:19 <Pavo> you can use the --tags argument to only deploy whatever containers you want 17:50:43 <kfox1111> +2 17:51:10 <sbezverk> kfox1111: thank you, as soon as it is merged please submit your patch 17:51:20 <kfox1111> I guess I can just rebase and fix in https://review.openstack.org/#/c/354895/4/services/glance/glance-api-pod.yml.j2 once merged. 17:51:20 <patchbot> kfox1111: patch 354895 - kolla-kubernetes - WIP - Deployment, Safe Shutdown, & Scaling for gla... 17:51:40 <kbaegis> Pavo Or so I've been told :) 17:51:56 <kbaegis> It's a cool feature 17:52:56 <Pavo> example kolla-ansible deploy -i ansible/inventory/multinode --tags horizon, will only redeploy horizon on the node thats in your inventory list 17:55:20 <sbezverk> kfox1111: rhallisey: we need to decide what to do with secret generator. All these bootstraps are consuming secrets. So it would make sense to include it now just for a timebeing as a separate tool.. 17:55:52 <rhallisey> sbezverk, I think we might be putting the cart ahead of the horse a little 17:56:07 <kfox1111> I'm ok merging the secret generator as is for now. 17:56:09 <rhallisey> we need to focus on a few things, accomplish them, and tackle new ones 17:56:22 <rhallisey> we have a bunch of efforts happening and it's hard to track 17:56:27 <sbezverk> kfox1111: did not want to ack without it ;-) 17:56:41 <kfox1111> sbezverk: makes sense. 17:56:51 <rhallisey> 1) lets get the core services working 17:56:57 <rhallisey> 2) CLI/ansible 17:57:14 <rhallisey> so hold on to this script for noew 17:57:32 <rhallisey> the secrets bit let's handle after we get the services working 17:57:39 <inc0> coolsvap, O 17:57:41 <inc0> I'm backl 17:57:48 <rhallisey> kfox1111, sbezverk is that ok? 17:58:01 <rhallisey> we need to focus on the big tasks first 17:58:05 <sbezverk> then we need to revert bootstrap changes 17:58:12 <kfox1111> rhallisey: the secrets are already staring to land. we just need a way to get them into k8s now. 17:58:21 <coolsvap> inc0: i am logging off 17:58:23 <kfox1111> the script is less then ideal, but pretty simple and works. 17:58:32 <rhallisey> kfox1111, I'm saying let's not block on that 17:58:38 <kfox1111> we can just merge it and then refactor it later when we have more time. 17:58:39 <rhallisey> service can land 17:58:42 <rhallisey> then we can do that 17:58:42 <inc0> coolsvap, did you move anything forward? 17:58:46 <coolsvap> i validate the scenarios, some amount of work is needed 17:58:53 <kfox1111> ah. 17:58:53 <coolsvap> validated the scenarios 17:58:58 <kfox1111> ok. that works too. 17:59:10 <kfox1111> we can review the generator independendly then. 17:59:23 <rhallisey> kfox1111, ya let's tackle these problems in smaller bits 17:59:27 <kfox1111> its just weird to leave trunk unusable. without other patches. 17:59:36 <kfox1111> though we have the same issue with kolla. 17:59:40 <rhallisey> kfox1111, it 17:59:51 <rhallisey> it's the nature of master in a new project to be a little unstbale 18:00:10 <kfox1111> yup. I get that. 18:00:17 <rhallisey> cool cool 18:00:20 <kfox1111> kk. 18:00:44 <sbezverk> kfox1111: rhallisey: we need to revert back to old bootstraps if you do not want to use the script.. 18:01:15 <rhallisey> sbezverk, is there a way we can have it for one service? 18:01:20 <rhallisey> glance has it right? 18:01:22 <kfox1111> sbezverk: he's saying we should just note that you need the extra script from review xxx to work. 18:01:33 <sbezverk> glance, neutron and keystone 18:01:35 <rhallisey> sbezverk, we can doc it 18:01:53 <kfox1111> we're already doing that with kolla too. 18:02:04 <openstackgerrit> Merged openstack/kolla-kubernetes: Glance bootstrap to create service/project/role https://review.openstack.org/346215 18:02:04 <rhallisey> we're getting close to landing the core services so we can return to this soon 18:02:06 <sbezverk> rhallisey eitherway works for me as lons as we move forward :-) 18:02:32 <rhallisey> kfox1111, yes true, it will be growing pains for now 18:02:38 <kfox1111> rhallisey: thoughts on https://review.openstack.org/#/c/356114 18:02:38 <patchbot> kfox1111: patch 356114 - kolla-kubernetes - WIP- openvswitch ip 18:02:55 <rhallisey> we won't really get a sense of stability until we have all the core services in place with our CLI/workflow done 18:03:19 <kfox1111> I think we can probably pull the WIP off of it, but I haven't really been able to test it yet. 18:03:55 <rhallisey> brb guys 18:05:11 <kfox1111> yeah, I was alittle afraid of that... all the json stuff in the kolla-config all conflict with each other. 18:05:20 <kfox1111> so neutron/keystone bootstrap secret reviews now have merge issues. 18:12:11 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Neutron bootstrap to create service/project/role https://review.openstack.org/353740 18:13:08 <sbezverk> kfox1111: resolved it 18:13:43 <kfox1111> +2'ed it. :) 18:19:49 <kfox1111> rhallisey: sbezverk: we should probably talk more about https://review.openstack.org/#/c/354895 too. 18:20:14 <patchbot> kfox1111: patch 354895 - kolla-kubernetes - WIP - Deployment, Safe Shutdown, & Scaling for gla... 18:20:48 <kfox1111> I think we will need to set the api address to 127.0.0.1 for it to work properly, but then all services will need similar patches as there is only one var for api ip right now. 18:21:17 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP - Deployment, Safe Shutdown, & Scaling for glance-api https://review.openstack.org/354895 18:23:46 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP - Deployment, Safe Shutdown, & Scaling for glance-api https://review.openstack.org/354895 18:29:32 <sbezverk> kfox1111: have you tested this patch with other services communicating between each other? 18:30:11 <sbezverk> cause this approach chnages also interprocess communication if I undersatnd correctly the idea 18:31:32 <kfox1111> I have not yet, as I need a solution to configuring it to listen on 127.0.0.1 instead of 0.0.0.0 18:31:54 <kfox1111> I didn't think glance did rpc. does it? 18:32:43 <kfox1111> I think for now, so I don't break everything to test, I can use an init container and override the config from 0.0.0.0 to 127.0.0.1. 18:32:49 <kfox1111> but I need the crudini patch to land in kolla. 18:33:36 <kfox1111> then we should be able to apply the same pattern to the other apis. 18:33:49 <kfox1111> we probably want to split the rpc services from the api services anyway. 18:34:06 <kfox1111> mostly I think thats already done, except maybe neutron-server I think does both. 18:34:58 <kfox1111> the neutron-server api's should be deployments and the neutron-server rpc's should be petsets I think. 18:35:17 <kfox1111> thought neturon did say its safe to restart a neutron-server these days. 18:35:39 <kfox1111> so the rpc stuff should in theory be safe to do mixed up in the api server with a deployment if that actually is true. 18:52:02 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP - Deployment, Safe Shutdown, & Scaling for glance-api https://review.openstack.org/354895 18:52:24 <kfox1111> k. once the crudini kolla patch lands, I think that one should probably work. needs a bit of testing once landed. 18:52:57 <kfox1111> we should then be able to apply the pattern to the other api services. 19:08:27 <sdake_> 30 mintes afk 19:08:27 <sdake_> bbiaf 19:14:32 <kbaegis> So I'm getting timeouts 19:14:43 <kbaegis> between ovsdb and openvswitch 19:15:28 <openstackgerrit> Leon Zachery proposed openstack/kolla: Simplify install process for faster quickstart execution https://review.openstack.org/356181 19:24:47 <kbaegis> http://hastebin.com/xiyenufera.vbs 19:25:27 <kbaegis> Linux bridging isn't working for me because of the iptables implementation, ovs is having timeouts between ovsdb and ovs-vswitchd 19:25:52 <kbaegis> So I can create volumes, upload images, spawn instances, but the network doesn't work for the L2 agents available 19:26:37 <kbaegis> All devices are getting created just fine (taps and bridges). 19:26:53 <kbaegis> OVS isn't up for more than 10 seconds due to not being able to reach ovsdb 19:29:03 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP - Deployment, Safe Shutdown, & Scaling for glance-api https://review.openstack.org/354895 19:30:48 <sdake_> kbaegis my bet is rtnetlink 19:31:12 <sdake_> but it could be the iptables integration with gentoo - which there probabyl sin't any 19:32:20 <kbaegis> sdake_ I need to become an authority on rtnetlink then. :) 19:32:41 <sdake_> i think my second guess might yield more results 19:32:46 <sdake_> but not sure 19:32:55 <kbaegis> sdake_ Like I said, it's able to create functioning tap devices 19:32:57 <sdake_> they are both speculation 19:33:09 <kbaegis> And they'll be plugged into the bridge 19:33:12 <sdake_> i missed that part 19:33:41 <sdake_> probably some iptables issue 19:33:48 <sdake_> do you hae iptables enabled by default in gentoo? 19:33:52 <kbaegis> It fails on the secgroup rules, yes 19:33:56 <kbaegis> Here's the traceback http://hastebin.com/pijeqimofi.vbs 19:34:14 <kbaegis> And yes, iptables is fully implemented in the kernel (all opts), and installed with all use-flags 19:34:29 <kbaegis> I rebuilt/repushed centos from source after 19:34:46 <kbaegis> I can validate that this is the only remaining issue for a base deployment on gentoo 19:35:03 <kbaegis> So if I can crack this, I don't have to use fuel-cpp on k8s 19:35:11 <kbaegis> Which is my personal preference :) 19:36:21 <sdake_> not sure - it could be either rtnetlink or iptables implementation in neutron 19:36:31 <sdake_> it is debuggable 19:36:36 <sdake_> since you can get a backtrace 19:36:40 <sdake_> protip print ftw :) 19:37:10 <sdake_> before doing that though, did you have iptables enabled by default in gentoo? 19:37:14 <sdake_> not built into the kernel 19:37:19 <sdake_> but enabled in the sysv scripts 19:37:27 <sdake_> or systemd or whatever one your using 19:38:44 <kbaegis> Hmm 19:39:14 <kbaegis> yikes. https://wiki.gentoo.org/ is down 19:39:48 <sdake_> wfm 19:41:26 <kbaegis> Huh. Did /etc/init.d/iptables restart and now I'm getting a new error- modprobe: FATAL: Module openvswitch not found 19:41:35 <sdake_> ya 19:41:36 <kbaegis> Is ovs required as a module, not as a builtin? 19:41:38 <sdake_> dont do that 19:41:47 <sdake_> not sure on ovs being a module 19:41:57 <sdake_> iptables should be disabled 19:42:01 <sdake_> not in an enabled state 19:42:09 <sdake_> i wanted to know if you had it enabled previously 19:42:27 <sdake_> chkconfig status iptables ought to do it 19:43:06 <kbaegis> I hadn't 19:43:47 <sdake_> so it wasn't enabled? 19:43:50 <kbaegis> I just saved the default rules and 19:43:57 <sdake_> ok 19:43:59 <kbaegis> It wasn't setup by sysvinit 19:44:01 <sdake_> lets start fresh 19:44:04 <sdake_> reboot your boxes 19:44:10 <sdake_> cleanup first 19:44:12 <kbaegis> :'( okay 19:44:15 <openstackgerrit> Leon Zachery proposed openstack/kolla: Simplify install process for faster quickstart execution https://review.openstack.org/356181 19:44:19 <sdake_> but dont delete your image 19:44:19 <sdake_> s 19:44:29 <sdake_> all we are going to do is iptables --flush 19:44:32 <sdake_> prior to deploy 19:44:38 <kbaegis> Okay 19:44:46 <sdake_> on the reboot 19:44:57 <sdake_> so whtaever you ahve eto do to make that happen, i'm sure you know by now :) 19:45:10 <sdake_> do on all nodes 19:45:17 <kbaegis> kolla-ansible destroy runs all the cleanup scripts, correct? 19:45:30 <kbaegis> So just do that, reboot, iptables —flush, deploy? 19:45:45 <sdake_> remind me - your aio atm? 19:46:07 <sdake_> if multinode - need to specify an inventory file 19:46:10 <kbaegis> I don't know what aio stands for :) 19:46:14 <kbaegis> Oh, all-in-one 19:46:15 <sdake_> aio = all in one 19:46:24 <kbaegis> Yes. That's step 1 19:46:37 <kbaegis> Then I'm going to spin the controllers up on 2 additional nodes (primarily into swap) 19:46:43 <sdake_> if your aio then destroy will do the job 19:46:50 <kbaegis> Since I only have 4gigs ram on those two 19:47:28 <sdake_> it may not be iptables --flush as well 19:47:39 <sdake_> i have a meeting in 10 mins to prep for, so i don't have time to read the manual page 19:47:46 <kbaegis> okay 19:47:57 <sdake_> ill be back in an hourish 19:48:00 <kbaegis> good prepping, ty for all the help 19:48:13 <sdake_> ya 19:48:27 <sdake_> if you could write down what you did to make it all work that might come in handy 19:48:35 <kbaegis> I will. 19:48:35 <sdake_> for others that follow in your path 19:48:48 <sdake_> etherpad is good 19:49:01 <sdake_> or an email to the mailing list 19:49:05 <kbaegis> There's a lot of dependencies for those compiling everything from source. Assumptions you get to make w/ distros :) 19:49:24 <kbaegis> I think this will help the Kolla community as well. When Ubuntu decides to change something in the kernel, for instance. 19:49:27 <sdake_> quetion, did the buildl keepalived binary work for you? 19:49:39 <kbaegis> I have it disabled at present 19:49:44 <sdake_> i see 19:49:48 <sdake_> so two things to solve not 1 :) 19:49:50 <kbaegis> I could workaround earlier (centos-bin) 19:49:55 <sdake_> you will need keepalived for step 2 19:49:56 <kbaegis> by manually assigning the vip 19:50:07 <kbaegis> But that's a hack against your checking scripts 20:00:37 <kbaegis> It was iptables —flush 20:01:03 <kbaegis> Redeploying, fingers crossed :) 20:12:26 <Pavo> anyone know what this error means? https://paste.pound-python.org/show/tXwye4n3DaYxKl95Rv20/ 20:13:59 <kbaegis> Pavo I'm guessing it's a pip problem. Not able to find files 20:14:07 <kbaegis> Did you do pip2 install -U git/ 20:14:29 <kbaegis> and pip2 install -U ansible 20:14:33 <Pavo> no, the docs only say pip install -U docker-py 20:14:47 <kbaegis> It's in there 20:14:51 <Pavo> installed ansible using yum 20:15:53 <kbaegis> Could work :) 20:15:55 <kbaegis> idk 20:20:33 <kbaegis> Well, I have an ovsdb-server.log now at least 20:22:40 <kbaegis> Evidently it requires ovs as a module 20:23:00 <kbaegis> Breaks on compiling it in. You guys may want to fix that, idk 20:24:18 <kbaegis> Easy enough for me to change; I don't personally care for managing kernel modules 20:26:08 <kbaegis> More concerningly, this may actually tie the host and guest kernel versions together by necessity. idk 20:30:11 <Pavo> this error doesn't make sense 20:35:32 <sdake_> kbaegis - that is the openvswitch project's domain 20:35:40 <sdake_> kbaegis we have no input into their development 20:35:49 <sdake_> they are pretty much not directly part of openstack 20:35:57 <sdake_> i mean i could dsend a message to them 20:35:58 <sdake_> but i'm not sure it would do much :) 20:36:27 <sdake_> kbaegis we load kernel modules from the host - iirc 20:37:27 <Pavo> sdake mind taking a look at my error 20:37:34 <Pavo> https://paste.pound-python.org/show/tXwye4n3DaYxKl95Rv20/ 20:38:43 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla: Start using orchestration_engine variable https://review.openstack.org/356538 20:58:00 <openstackgerrit> Merged openstack/kolla-kubernetes: Neutron bootstrap to create service/project/role https://review.openstack.org/353740 20:58:12 <kbaegis> recompiled;rebooted;checking to see if ovs will start without redeploy now 21:03:03 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding NOTE for Kubernetes Secret Generator https://review.openstack.org/356719 21:06:09 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding NOTE for Kubernetes Secret Generator https://review.openstack.org/356719 21:08:14 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding NOTE for Kubernetes Secret Generator https://review.openstack.org/356719 21:19:27 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Modifying Keystone bootstrap to use secrets https://review.openstack.org/356098 21:25:27 <sdake_> pavo 21:25:27 <sdake_> i'm beat 21:25:33 <sdake_> been up since 4am 21:25:40 <sdake_> need power nap - can it wait 21:26:57 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding NOTE for Kubernetes Secret Generator https://review.openstack.org/356719 21:28:46 <Pavo> sure 21:29:14 <kfox1111> weird... they make symlinks for configmaps... 22:14:24 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance-api https://review.openstack.org/354895 22:15:01 <kfox1111> sbezverk: I think that one's ready for review now. 22:16:44 <kbaegis1> sdake_ do you know where the openvswitch module is expected to be found? 22:20:00 <kfox1111> sbezverk: with that review, and a ceph backed glance, you should be able to do a no downtime minor version rolling upgrade. 22:51:13 <kbaegis1> lol 22:51:17 <kbaegis1> "modprobe: ERROR: could not insert 'openvswitch': Exec format error" 22:51:19 <kbaegis1> Great 22:52:50 <kbaegis1> Falling back to linuxbridging, trying the iptables —flush option 23:27:31 <kbaegis1> Sadly, no. iptables —flush didn't work :'( 23:28:14 <kbaegis> Oh well 23:33:34 <kbaegis> is there a way to manually insert those rules? 23:33:39 <kbaegis> To see rejections/conflicts? 23:34:53 <kbaegis> Alright, I'm going to cave and install centos 23:34:54 <kbaegis> :/ 23:35:06 <kbaegis> Not happy about it, but it'll to in a pinch 23:35:07 <kbaegis> :) 23:35:23 <kbaegis> And I'd rather have a functioning lab than one done right :) 23:45:54 <kbaegis> Wish I knew more than that iptables.manager generically fails on COMMIT 23:46:03 <kbaegis> I feel like that's solvable 00:28:47 <openstackgerrit> Jeffrey Zhang proposed openstack/kolla: Bump the ansible version to 2.1.1.0 in kolla-toolbox https://review.openstack.org/356311 00:36:55 <sdake> pavo back alive 00:37:53 <duonghq> morning sdake 00:38:19 <sdake> duonghq afternon actuallyy 00:38:23 <sdake> powernap + powerdinner 00:38:25 <sdake> ready to rck 00:38:28 <sdake> any9one need any help 00:39:08 <duonghq> what is current temperature sdake? 00:39:19 <sdake> hot 00:39:36 <duonghq> dose the winter is better? 00:39:39 <sdake> 5pm 00:39:49 <sdake> winter is much better 00:39:50 <sdake> prorbably 105-1110 atm 00:40:20 <sdake> 110 that is 00:41:04 <duonghq> good weather, huh? does it rain? 00:41:39 <duonghq> I mean winter 00:42:41 <sdake> during summer yes 00:42:44 <sdake> during winter no 00:42:50 <sdake> during summer we have monsoons 00:43:07 <duonghq> monsoons festival? 00:43:20 <sdake> monsoons are like a big rain storm 00:43:32 <duonghq> ah 00:43:37 <sdake> which cools off the heat 00:43:49 <sdake> peopel in arizona are angry during summer time 00:43:51 <sdake> and happy during winter 00:43:54 <sdake> its kind of wierd 00:44:00 <sdake> i think its opposite to the rest of the us ;) 00:44:10 <duonghq> there is storm come here in about 1-2 days :(, long time no storm go here 00:44:12 <sdake> angry might be the wrong word 00:44:19 <sdake> displeased might be better 00:44:32 <duonghq> I like the winter, too 00:44:54 <sdake> i've lost 10 punds in the lat 1 month because of the heat affecting my appetite 00:44:59 <sdake> continuous problem in ariona 00:45:25 <sdake> arizona isn't problem free 00:45:32 <sdake> but all the datacentres in the us are here 00:45:36 <sdake> becuase nothing ever goes wrong 00:45:41 <sdake> except its hotter then hell ;) 00:45:50 <duonghq> yay 00:46:07 <sdake> no tornadoes no earthquakes no nothing 00:46:29 <duonghq> just hot? 00:46:39 <sdake> yup just hot 00:46:49 <sdake> and 280% murder rate above avereage 00:46:52 <sdake> although i don't live in that part of town 00:47:19 <sdake> town/city 00:47:41 <sdake> murder rate increases in summer time - presumably because of resource constraints 00:47:56 <duonghq> bad "habbit" 00:48:54 <sdake> i feel safe all the time 00:48:57 <sdake> i dont even lock my doors 00:49:19 <sdake> others - in other parts of town - have 5 deadbolts on their door 00:49:32 <duonghq> so good for you 00:49:39 <sdake> location location location 00:49:50 <wirehead_> Most of the year, it’s the valley of the sun. Part of the year, it’s the surface of the sun. 00:50:07 <sdake> wirehead_ wow - never picked up on that 00:50:18 <duonghq> the chant spirit from list meeting still around, 00:50:29 <sdake> wirehead_ surpried- and i'm 42 years old :) 00:50:29 <wirehead_> One of my grand-bosses at Rackspace lived in Phoenix and said that. 00:50:55 <sdake> he must have been grand 00:51:00 <sdake> and more experienced then I 00:51:36 <sdake> wirehead_ do you use jabber 00:51:48 <wirehead_> I think I’m supposed to, but I don't. 00:51:49 <sdake> i see your not on 00:51:57 <sdake> would you mind indulging me for a moment 00:54:13 <sdake> battery at 2% brb if power dies out 00:55:27 <wirehead_> K, well the Jabber client is forcing me to download a 65 meg update. But I think I’m on. 00:56:38 <sbezverk> wirehead_ I see some issue with neutron boostrap, wondering if you see something similar 00:57:57 <sbezverk> wirehead_ when it tries to run db upgrade on neutron db and is it was already upgraded, I see continuous restart of bootstrap process 00:58:16 <wirehead_> hmmmm. 00:58:42 <sbezverk> wirehead_ wondering if there is way to check if db was already upgraded and then ignore upgrade 00:59:20 <Pavo> sdake cool, I am setting up nodes agan 00:59:31 <wirehead_> That seems like a bug in the update process. 00:59:33 <Pavo> lol like for the 6th time 01:00:04 <duonghq> sdake: how about the SAP case? 01:00:37 <sbezverk> wirehead_ most likely, they should handle it more gracefully 01:01:01 <sbezverk> will open a bug tomorrow against neutron 01:01:08 <Pavo> so whats the most stable release 2.0.3? 01:03:00 <duonghq> sdake: how about the SAP case? 01:03:27 <sdake> wirehead_ around on jabber? 01:03:28 <sdake> i lost power 01:03:29 <sdake> duonghq huh? 01:03:56 <duonghq> SAP use our work without license include 01:04:25 <sdake> duonghq someone told me that had been fixed 01:04:28 <wirehead_> sbezverk: When I test, I usually nuke everything and start from scratch. 01:04:30 <sdake> have a link to the offending repo? 01:15:30 <duonghq> sdake: still there? 01:17:37 <kbaegis> sdake I still need help :) 01:26:24 <kbaegis> I guess another way to go would be to build my own l2 agent 01:26:45 <duonghq> kbaegis: from scratch? 01:29:19 <kbaegis> No alternative :) 01:29:37 <kbaegis> duonghq so I guess so. Centos works for EVERYTHING but ovs or linux bridging 01:29:47 <kbaegis> there's vikraman/gentoo 01:29:55 <duonghq> how about the ubuntu ones? 01:35:56 <kbaegis> Might do. I didn't test that. 01:38:16 <kbaegis> I'll hardcode and reset. I'm pretty sure it's a kernel version thing though 01:38:54 <kbaegis> who's responsible for the centos one ooc? Could give me pointers? 01:42:38 <sbezverk> sdake ping 01:43:10 <duonghq> sbezverk: he is go out 01:43:31 <sbezverk> duonghq thank you will try to catch him later 01:45:47 <duonghq> (try stop openstack meeting bot from logging from last meeting, insane bug) 01:45:50 <duonghq> #endmeeting