16:00:25 <ajpower> #startmeeting kolla
16:00:28 <openstack> Meeting started Wed Aug 17 16:00:25 2016 UTC and is due to finish in 60 minutes.  The chair is ajpower. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:30 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:32 <openstack> The meeting name has been set to 'kolla'
16:00:49 <rhallisey> ajafo, wrong location
16:01:05 <rhallisey> ajpower, , #openstack-meeting-4
16:01:12 <rhallisey> ajafo, sorry wrong person
16:04:33 <ajpower> #openstack-meeting-4
16:05:49 <openstackgerrit> Merged openstack/kolla: Replace horizon default config with custom config  https://review.openstack.org/306928
16:07:31 <rhallisey> ajafo, `/join #openstack-meeting-4`
16:07:36 <rhallisey> ajpower, ^
16:07:40 <rhallisey> sorry ajafo
16:07:44 <rhallisey> :/
16:08:32 <ajpower> join #openstack-meeting-4
16:08:58 <rhallisey> ajafo, add a /
16:17:13 <kbaegis> So what's the procedure/strategy in kolla to upgrade a single container?
16:33:11 <kbaegis> Is there an individual runbook I can deploy to move horizon between versions, for example?
16:33:34 <rhallisey> kbaegis, you can run play specific for a service with tags
16:33:54 <rhallisey> so you can run the upgrade play with the horizon tags and that should get what you want
16:35:00 <kbaegis> so the command would be kolla-ansible upgrade horizon?
16:35:16 <kbaegis> or would I alter globals and select a horizon version?
16:36:55 <rhallisey> kbaegis, kolla-ansible upgrade --tags horizon
16:37:06 <kbaegis> Oh, nice :)
16:47:04 <kbaegis> Latest issue: http://hastebin.com/pijeqimofi.vbs
16:47:24 <kbaegis> Anyone played around with non-cooperative iptables implementations and willing to help?
16:48:11 <kbaegis> Tap device is created successfully, bridge too
16:48:22 <kbaegis> It's just these dang iptables rules :)
16:50:30 <kbaegis> Running the commands from the log against iptables manually I get errors like
16:50:34 <kbaegis> iptables: No chain/target/match by that name.
16:50:42 <kbaegis> or iptables: Index of insertion too big
16:53:26 <kbaegis> Huh.  Maybe it needs CONFIG_INET_DIAG_DESTROY
16:53:29 <inc0> coolsvap, so ad rally
16:53:47 * britthouser listens in
16:53:49 <inc0> it should work now, we just need to change deployments
16:53:53 <duonghq> hey sdake_, the bug, do we have any replacement for deprecated tag option?
16:53:55 <inc0> cuz access creds are invalid
16:54:03 <ntpttr___> hey inc0 I'm running into an issue with my cinder/ceph deployment in kolla, do you have a moment?
16:54:08 <inc0> sure
16:54:13 <inc0> what's up Nate?
16:54:30 <sdake_> duonghq i dont have a clear picture of the issue yet duonghq
16:54:31 <ntpttr___> inc0: so everything is starting up fine and I can create and attach volumes and all, but the backup service isn't working
16:54:37 <sdake_> duonghq - but there are docs in teh queue addressing it
16:54:39 <sdake_> they are just wrong
16:54:49 <ntpttr___> inc0: basically whenever I try to backup a volume, it hangs in the 'creating' phase forever
16:54:52 <sdake_> duonghq so the solution first is to document a workaround
16:54:56 <coolsvap> inc0: hi
16:54:57 <inc0> and no errors?
16:54:57 <sdake_> duonghq then to fix the pbr issue
16:54:59 <ntpttr___> and it seems like privsep just isn't starting up
16:55:13 <coolsvap> inc0: where are we currently wrt rally
16:55:19 <ntpttr___> in the logs, here's mine: http://paste.openstack.org/show/558606/ and here's one where it works: http://paste.openstack.org/show/558537/
16:55:33 <ntpttr___> the privsep daemon isn't starting, but there's no errors
16:55:54 <duonghq> sdake_: I look through the code and cannot find any replacement for deprecated option, I don't think its normal
16:56:02 <inc0> coolsvap, can you fix deployments in rally? our creds are wrong
16:56:03 <duonghq> sdake_ fix me if I wrong :)
16:56:27 <inc0> ntpttr___, it might be a bug
16:56:43 <coolsvap> inc0: sure let me connect to vpn
16:56:44 <inc0> I haven't seen this before
16:56:54 <ntpttr___> inc0: it looks like that might be the case, this is on a clean install with no errors or anything
16:57:12 <ntpttr___> has anyone here deployed cinder w/ ceph and then successfully used the backup service before?
16:57:45 <inc0> our tests on osic cluster should show that
16:58:57 <duonghq> sdake_: I think it should be a option in kolla-build.conf for tagging?
16:59:19 <sdake_> i dont think --tag should be a deprecated option at all
16:59:20 <ntpttr___> inc0: do you think I should go ahead and file a bug against kolla?
16:59:21 <sdake_> i dont know why it is
17:00:14 <coolsvap> inc0: we can update the credentials in osic.json file on the node
17:00:14 <inc0> ntpttr___, yeah, file it
17:00:16 <duonghq> sdake_: somebody marked it deprecated?
17:01:03 <inc0> coolsvap, might doing that?
17:01:07 <coolsvap> inc0: i am updating it from /etc/kolla/admin-openrc.sh
17:01:07 <inc0> I dunno where it is
17:01:08 <coolsvap> yes
17:01:23 <inc0> tmux a plz
17:01:28 <inc0> I'll watch it
17:01:35 <coolsvap> yeah sorry
17:01:38 <coolsvap> just a min
17:01:42 <inc0> np:)
17:03:53 <ntpttr___> okay got the bug report up https://bugs.launchpad.net/kolla/+bug/1614180
17:03:53 <openstack> Launchpad bug 1614180 in kolla "Backup w/ ceph hangs when creating a backup" [Undecided,New]
17:09:18 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP- openvswitch ip  https://review.openstack.org/356114
17:10:21 <ntpttr___> inc0: do we need to be installing oslo.privsep somewhere? I don't see it anywhere in the kolla repo, or in this file where I see a lot of other oslo libraries https://github.com/openstack/kolla/blob/5f4ef0d84bf1d1f5f72f245a0dc9e95d25e0b92d/docker/openstack-base/Dockerfile.j2
17:10:37 <ntpttr___> in devstack they're installing it https://github.com/openstack-dev/devstack/blob/06f3639a70dc5884107a4045bef5a9de1fb725a5/lib/oslo
17:11:17 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP- openvswitch ip  https://review.openstack.org/356114
17:19:49 <Pavo> evening everyone
17:20:35 <openstackgerrit> Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update cirros image visibility in init-runonce  https://review.openstack.org/353933
17:26:08 <kbaegis> gah.  Linuxbridging broken still on gentoo
17:27:01 <coolsvap> pbourke: do yo uwhave the kolla-rally-tests.sh script open in your terminal?
17:31:14 <kfox1111> sbezverk: rhallisey: so, for the glance bootstrap one. we postponed talking about dealing with the init-container templating thing for dealing with the json as we didn't have access to jq. that patch just merged though.
17:32:13 <sbezverk> kfox1111: I would suggest to merge what I have now as I tested it multiple times and it is working
17:32:17 <kfox1111> can we reconsider this: https://review.openstack.org/#/c/353740/4..6/services/neutron/neutron-bootstrap-job.yml.j2
17:32:17 <patchbot> kfox1111: patch 353740 - kolla-kubernetes - Neutron bootstrap to create service/project/role
17:32:27 <Pavo> kbaegis was it you asking about deploying only 1 of the images with kolla-ansible yesterday?
17:32:44 <rhallisey> kfox1111, I mean we can iterate on it
17:32:44 <sbezverk> if you feel like modifying it, please submit PS and we can have discussion
17:32:57 <kfox1111> k.
17:33:20 <sbezverk> it is just so many things are hanging because of this
17:34:10 <kfox1111> sure. but I just want to avoid having to do more when we add more bootstrapping to nova and friends too.
17:34:34 <sbezverk> kfox1111: I do not think we add more than just nova
17:34:39 <sbezverk> any time soon
17:34:48 <kfox1111> we can merge these 2 as is.
17:34:49 <kfox1111> cinder?
17:34:53 <kfox1111> heat?
17:34:58 <sbezverk> so if you propose PS after, I will help you to apply on other components
17:35:09 <kfox1111> k
17:35:51 <sbezverk> If I undersatand the primary goal is get VM
17:36:17 <kfox1111> ah. we can claim vm without cinder/heat, true.
17:36:46 <sbezverk> kfox1111 yep, I think it make sense
17:37:12 <kfox1111> one more thing and I think the glance one's good then.
17:37:22 <kfox1111> do we actually need the pvc in the bootstrap?
17:38:02 <kfox1111> I don't really see where its used, and it shouldn't be there if using the ceph backend.
17:38:09 <kfox1111> I think we can probably just drop it all together.
17:39:31 <sbezverk> kfox1111: I will need to retest it
17:40:22 <sdake_> pbourke you still around?
17:40:41 <sdake_> i dont blame ya
17:40:49 <sdake_> sweet dreams :)
17:42:58 <kfox1111> wait..
17:43:03 <kfox1111> its used here:
17:43:26 <kfox1111> https://review.openstack.org/#/c/346215/11/services/glance/glance-bootstrap-job.yml.j2 line 76
17:43:27 <patchbot> kfox1111: patch 346215 - kolla-kubernetes - Glance bootstrap to create service/project/role
17:43:38 <kfox1111> whats the api container being used for?
17:44:32 <sbezverk> kfox1111: it looks like it is used during bootstrap process.
17:44:43 <kfox1111> where?
17:44:59 <kfox1111> I see database creation, and keystone registration. but not much else?
17:46:53 <sbezverk> kfox1111: it does this     sudo chown -R glance: /var/lib/glance/
17:47:27 <kfox1111> hmm....
17:47:42 <kfox1111> oh. cause the KOLLA_BOOTSTRAP.
17:47:47 <sbezverk> kfox1111: if we do not have it mounted, it will be done against virtual directory
17:47:59 <sbezverk> kfox1111: right
17:48:13 <kfox1111> k.
17:48:17 <sbezverk> kfox1111: it is not much but please do not ask me to move it somewhere else
17:48:30 <kfox1111> so then I think we just need to do something like: https://review.openstack.org/#/c/354895/4/services/glance/glance-api-pod.yml.j2
17:48:30 <patchbot> kfox1111: patch 354895 - kolla-kubernetes - WIP - Deployment, Safe Shutdown, & Scaling for gla...
17:48:47 <kfox1111> with glance_backend_ceph
17:49:08 <kfox1111> around the volume and volumeMount.
17:49:35 <kbaegis> Pavo Yes :)
17:49:37 <sbezverk> kfox1111: can you add it on top after it is merged?
17:49:57 <sbezverk> I will not be able to test it
17:50:03 <sbezverk> I just do not have means
17:50:07 <kfox1111> k.
17:50:19 <Pavo> you can use the --tags argument to only deploy whatever containers you want
17:50:43 <kfox1111> +2
17:51:10 <sbezverk> kfox1111: thank you, as soon as it is merged please submit your patch
17:51:20 <kfox1111> I guess I can just rebase and fix in https://review.openstack.org/#/c/354895/4/services/glance/glance-api-pod.yml.j2 once merged.
17:51:20 <patchbot> kfox1111: patch 354895 - kolla-kubernetes - WIP - Deployment, Safe Shutdown, & Scaling for gla...
17:51:40 <kbaegis> Pavo Or so I've been told :)
17:51:56 <kbaegis> It's a cool feature
17:52:56 <Pavo> example kolla-ansible deploy -i ansible/inventory/multinode --tags horizon, will only redeploy horizon on the node thats in your inventory list
17:55:20 <sbezverk> kfox1111: rhallisey: we need to decide what to do with secret generator. All these bootstraps are consuming secrets. So it would make sense to include it now just for a timebeing as a separate tool..
17:55:52 <rhallisey> sbezverk, I think we might  be putting the cart ahead of the horse a little
17:56:07 <kfox1111> I'm ok merging the secret generator as is for now.
17:56:09 <rhallisey> we need to focus on a few things, accomplish them, and tackle new ones
17:56:22 <rhallisey> we have a bunch of efforts happening and it's hard to track
17:56:27 <sbezverk> kfox1111: did not want to ack without it ;-)
17:56:41 <kfox1111> sbezverk: makes sense.
17:56:51 <rhallisey> 1) lets get the core services working
17:56:57 <rhallisey> 2) CLI/ansible
17:57:14 <rhallisey> so hold on to this script for noew
17:57:32 <rhallisey> the secrets bit let's handle after we get the services working
17:57:39 <inc0> coolsvap, O
17:57:41 <inc0> I'm backl
17:57:48 <rhallisey> kfox1111, sbezverk is that ok?
17:58:01 <rhallisey> we need to focus on the big tasks first
17:58:05 <sbezverk> then we need to revert bootstrap changes
17:58:12 <kfox1111> rhallisey: the secrets are already staring to land. we just need a way to get them into k8s now.
17:58:21 <coolsvap> inc0: i am logging off
17:58:23 <kfox1111> the script is less then ideal, but pretty simple and works.
17:58:32 <rhallisey> kfox1111, I'm saying let's not block on that
17:58:38 <kfox1111> we can just merge it and then refactor it later when we have more time.
17:58:39 <rhallisey> service can land
17:58:42 <rhallisey> then we can do that
17:58:42 <inc0> coolsvap, did you move anything forward?
17:58:46 <coolsvap> i validate the scenarios, some amount of work is needed
17:58:53 <kfox1111> ah.
17:58:53 <coolsvap> validated the scenarios
17:58:58 <kfox1111> ok. that works too.
17:59:10 <kfox1111> we can review the generator independendly then.
17:59:23 <rhallisey> kfox1111, ya let's tackle these problems in smaller bits
17:59:27 <kfox1111> its just weird to leave trunk unusable. without other patches.
17:59:36 <kfox1111> though we have the same issue with kolla.
17:59:40 <rhallisey> kfox1111, it
17:59:51 <rhallisey> it's the nature of master in a new project to be a little unstbale
18:00:10 <kfox1111> yup. I get that.
18:00:17 <rhallisey> cool cool
18:00:20 <kfox1111> kk.
18:00:44 <sbezverk> kfox1111: rhallisey: we need to revert back to old bootstraps if you do not want to use the script..
18:01:15 <rhallisey> sbezverk, is there a way we can have it for one service?
18:01:20 <rhallisey> glance has it right?
18:01:22 <kfox1111> sbezverk: he's saying we should just note that you need the extra script from review xxx to work.
18:01:33 <sbezverk> glance, neutron and keystone
18:01:35 <rhallisey> sbezverk, we can doc it
18:01:53 <kfox1111> we're already doing that with kolla too.
18:02:04 <openstackgerrit> Merged openstack/kolla-kubernetes: Glance bootstrap to create service/project/role  https://review.openstack.org/346215
18:02:04 <rhallisey> we're getting close to landing the core services so we can return to this soon
18:02:06 <sbezverk> rhallisey eitherway works for me as lons as we move forward :-)
18:02:32 <rhallisey> kfox1111, yes true, it will be growing pains for now
18:02:38 <kfox1111> rhallisey: thoughts on https://review.openstack.org/#/c/356114
18:02:38 <patchbot> kfox1111: patch 356114 - kolla-kubernetes - WIP- openvswitch ip
18:02:55 <rhallisey> we won't really get a sense of stability until we have all the core services in place with our CLI/workflow done
18:03:19 <kfox1111> I think we can probably pull the WIP off of it, but I haven't really been able to test it yet.
18:03:55 <rhallisey> brb guys
18:05:11 <kfox1111> yeah, I was alittle afraid of that... all the json stuff in the kolla-config all conflict with each other.
18:05:20 <kfox1111> so neutron/keystone bootstrap secret reviews now have merge issues.
18:12:11 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Neutron bootstrap to create service/project/role  https://review.openstack.org/353740
18:13:08 <sbezverk> kfox1111: resolved it
18:13:43 <kfox1111> +2'ed it. :)
18:19:49 <kfox1111> rhallisey: sbezverk: we should probably talk more about https://review.openstack.org/#/c/354895 too.
18:20:14 <patchbot> kfox1111: patch 354895 - kolla-kubernetes - WIP - Deployment, Safe Shutdown, & Scaling for gla...
18:20:48 <kfox1111> I think we will need to set the api address to 127.0.0.1 for it to work properly, but then all services will need similar patches as there is only one var for api ip right now.
18:21:17 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP - Deployment, Safe Shutdown, & Scaling for glance-api  https://review.openstack.org/354895
18:23:46 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP - Deployment, Safe Shutdown, & Scaling for glance-api  https://review.openstack.org/354895
18:29:32 <sbezverk> kfox1111: have you tested this patch with other services communicating between each other?
18:30:11 <sbezverk> cause this approach chnages also interprocess communication if I undersatnd correctly the idea
18:31:32 <kfox1111> I have not yet, as I need a solution to configuring it to listen on 127.0.0.1 instead of 0.0.0.0
18:31:54 <kfox1111> I didn't think glance did rpc. does it?
18:32:43 <kfox1111> I think for now, so I don't break everything to test, I can use an init container and override the config from 0.0.0.0 to 127.0.0.1.
18:32:49 <kfox1111> but I need the crudini patch to land in kolla.
18:33:36 <kfox1111> then we should be able to apply the same pattern to the other apis.
18:33:49 <kfox1111> we probably want to split the rpc services from the api services anyway.
18:34:06 <kfox1111> mostly I think thats already done, except maybe neutron-server I think does both.
18:34:58 <kfox1111> the neutron-server api's should be deployments and the neutron-server rpc's should be petsets I think.
18:35:17 <kfox1111> thought neturon did say its safe to restart a neutron-server these days.
18:35:39 <kfox1111> so the rpc stuff should in theory be safe to do mixed up in the api server with a deployment if that actually is true.
18:52:02 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP - Deployment, Safe Shutdown, & Scaling for glance-api  https://review.openstack.org/354895
18:52:24 <kfox1111> k. once the crudini kolla patch lands, I think that one should probably work. needs a bit of testing once landed.
18:52:57 <kfox1111> we should then be able to apply the pattern to the other api services.
19:08:27 <sdake_> 30 mintes afk
19:08:27 <sdake_> bbiaf
19:14:32 <kbaegis> So I'm getting timeouts
19:14:43 <kbaegis> between ovsdb and openvswitch
19:15:28 <openstackgerrit> Leon Zachery proposed openstack/kolla: Simplify install process for faster quickstart execution  https://review.openstack.org/356181
19:24:47 <kbaegis> http://hastebin.com/xiyenufera.vbs
19:25:27 <kbaegis> Linux bridging isn't working for me because of the iptables implementation, ovs is having timeouts between ovsdb and ovs-vswitchd
19:25:52 <kbaegis> So I can create volumes, upload images, spawn instances, but the network doesn't work for the L2 agents available
19:26:37 <kbaegis> All devices are getting created just fine (taps and bridges).
19:26:53 <kbaegis> OVS isn't up for more than 10 seconds due to not being able to reach ovsdb
19:29:03 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: WIP - Deployment, Safe Shutdown, & Scaling for glance-api  https://review.openstack.org/354895
19:30:48 <sdake_> kbaegis my bet is rtnetlink
19:31:12 <sdake_> but it could be the iptables integration with gentoo - which there probabyl sin't any
19:32:20 <kbaegis> sdake_ I need to become an authority on rtnetlink then.  :)
19:32:41 <sdake_> i think my second guess might yield more results
19:32:46 <sdake_> but not sure
19:32:55 <kbaegis> sdake_ Like I said, it's able to create functioning tap devices
19:32:57 <sdake_> they are both speculation
19:33:09 <kbaegis> And they'll be plugged into the bridge
19:33:12 <sdake_> i missed that part
19:33:41 <sdake_> probably some iptables issue
19:33:48 <sdake_> do you hae iptables enabled by default in gentoo?
19:33:52 <kbaegis> It fails on the secgroup rules, yes
19:33:56 <kbaegis> Here's the traceback http://hastebin.com/pijeqimofi.vbs
19:34:14 <kbaegis> And yes, iptables is fully implemented in the kernel (all opts), and installed with all use-flags
19:34:29 <kbaegis> I rebuilt/repushed centos from source after
19:34:46 <kbaegis> I can validate that this is the only remaining issue for a base deployment on gentoo
19:35:03 <kbaegis> So if I can crack this, I don't have to use fuel-cpp on k8s
19:35:11 <kbaegis> Which is my personal preference :)
19:36:21 <sdake_> not sure - it could be either rtnetlink or iptables implementation in neutron
19:36:31 <sdake_> it is debuggable
19:36:36 <sdake_> since you can get a backtrace
19:36:40 <sdake_> protip print ftw :)
19:37:10 <sdake_> before doing that though, did you have iptables enabled by default in gentoo?
19:37:14 <sdake_> not built into the kernel
19:37:19 <sdake_> but enabled in the sysv scripts
19:37:27 <sdake_> or systemd or whatever one your using
19:38:44 <kbaegis> Hmm
19:39:14 <kbaegis> yikes. https://wiki.gentoo.org/ is down
19:39:48 <sdake_> wfm
19:41:26 <kbaegis> Huh. Did /etc/init.d/iptables restart and now I'm getting a new error- modprobe: FATAL: Module openvswitch not found
19:41:35 <sdake_> ya
19:41:36 <kbaegis> Is ovs required as a module, not as a builtin?
19:41:38 <sdake_> dont do that
19:41:47 <sdake_> not sure on ovs being a module
19:41:57 <sdake_> iptables should be disabled
19:42:01 <sdake_> not in an enabled state
19:42:09 <sdake_> i wanted to know if you had it enabled previously
19:42:27 <sdake_> chkconfig status iptables ought to do it
19:43:06 <kbaegis> I hadn't
19:43:47 <sdake_> so it wasn't enabled?
19:43:50 <kbaegis> I just saved the default rules and
19:43:57 <sdake_> ok
19:43:59 <kbaegis> It wasn't setup by sysvinit
19:44:01 <sdake_> lets start fresh
19:44:04 <sdake_> reboot your boxes
19:44:10 <sdake_> cleanup first
19:44:12 <kbaegis> :'( okay
19:44:15 <openstackgerrit> Leon Zachery proposed openstack/kolla: Simplify install process for faster quickstart execution  https://review.openstack.org/356181
19:44:19 <sdake_> but dont delete your image
19:44:19 <sdake_> s
19:44:29 <sdake_> all we are going to do is iptables --flush
19:44:32 <sdake_> prior to deploy
19:44:38 <kbaegis> Okay
19:44:46 <sdake_> on the reboot
19:44:57 <sdake_> so whtaever you ahve eto do to make that happen, i'm sure you know by now :)
19:45:10 <sdake_> do on all nodes
19:45:17 <kbaegis> kolla-ansible destroy runs all the cleanup scripts, correct?
19:45:30 <kbaegis> So just do that, reboot, iptables —flush, deploy?
19:45:45 <sdake_> remind me - your aio atm?
19:46:07 <sdake_> if multinode - need to specify an inventory file
19:46:10 <kbaegis> I don't know what aio stands for :)
19:46:14 <kbaegis> Oh, all-in-one
19:46:15 <sdake_> aio = all in one
19:46:24 <kbaegis> Yes.  That's step 1
19:46:37 <kbaegis> Then I'm going to spin the controllers up on 2 additional nodes (primarily into swap)
19:46:43 <sdake_> if your aio then destroy will do the job
19:46:50 <kbaegis> Since I only have 4gigs ram on those two
19:47:28 <sdake_> it may not be iptables --flush as well
19:47:39 <sdake_> i have a meeting in 10 mins to prep for, so i don't have time to read the manual page
19:47:46 <kbaegis> okay
19:47:57 <sdake_> ill be back in an hourish
19:48:00 <kbaegis> good prepping, ty for all the help
19:48:13 <sdake_> ya
19:48:27 <sdake_> if you could write down what you did to make it all work that might come in handy
19:48:35 <kbaegis> I will.
19:48:35 <sdake_> for others that follow in your path
19:48:48 <sdake_> etherpad is good
19:49:01 <sdake_> or an email to the mailing list
19:49:05 <kbaegis> There's a lot of dependencies for those compiling everything from source.  Assumptions you get to make w/ distros :)
19:49:24 <kbaegis> I think this will help the Kolla community as well.  When Ubuntu decides to change something in the kernel, for instance.
19:49:27 <sdake_> quetion, did the buildl keepalived binary work for you?
19:49:39 <kbaegis> I have it disabled at present
19:49:44 <sdake_> i see
19:49:48 <sdake_> so two things to solve not 1 :)
19:49:50 <kbaegis> I could workaround earlier (centos-bin)
19:49:55 <sdake_> you will need keepalived for step 2
19:49:56 <kbaegis> by manually assigning the vip
19:50:07 <kbaegis> But that's a hack against your checking scripts
20:00:37 <kbaegis> It was iptables —flush
20:01:03 <kbaegis> Redeploying, fingers crossed :)
20:12:26 <Pavo> anyone know what this error means? https://paste.pound-python.org/show/tXwye4n3DaYxKl95Rv20/
20:13:59 <kbaegis> Pavo I'm guessing it's a pip problem.  Not able to find files
20:14:07 <kbaegis> Did you do pip2 install -U git/
20:14:29 <kbaegis> and pip2 install -U ansible
20:14:33 <Pavo> no, the docs only say pip install -U docker-py
20:14:47 <kbaegis> It's in there
20:14:51 <Pavo> installed ansible using yum
20:15:53 <kbaegis> Could work :)
20:15:55 <kbaegis> idk
20:20:33 <kbaegis> Well, I have an ovsdb-server.log now at least
20:22:40 <kbaegis> Evidently it requires ovs as a module
20:23:00 <kbaegis> Breaks on compiling it in.  You guys may want to fix that, idk
20:24:18 <kbaegis> Easy enough for me to change; I don't personally care for managing kernel modules
20:26:08 <kbaegis> More concerningly, this may actually tie the host and guest kernel versions together by necessity.  idk
20:30:11 <Pavo> this error doesn't make sense
20:35:32 <sdake_> kbaegis - that is the openvswitch project's domain
20:35:40 <sdake_> kbaegis we have no input into their development
20:35:49 <sdake_> they are pretty much not directly part of openstack
20:35:57 <sdake_> i mean i could dsend a message to them
20:35:58 <sdake_> but i'm not sure it would do much :)
20:36:27 <sdake_> kbaegis we load kernel modules from the host - iirc
20:37:27 <Pavo> sdake mind taking a look at my error
20:37:34 <Pavo> https://paste.pound-python.org/show/tXwye4n3DaYxKl95Rv20/
20:38:43 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla: Start using orchestration_engine variable  https://review.openstack.org/356538
20:58:00 <openstackgerrit> Merged openstack/kolla-kubernetes: Neutron bootstrap to create service/project/role  https://review.openstack.org/353740
20:58:12 <kbaegis> recompiled;rebooted;checking to see if ovs will start without redeploy now
21:03:03 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding NOTE for Kubernetes Secret Generator  https://review.openstack.org/356719
21:06:09 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding NOTE for Kubernetes Secret Generator  https://review.openstack.org/356719
21:08:14 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding NOTE for Kubernetes Secret Generator  https://review.openstack.org/356719
21:19:27 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Modifying Keystone bootstrap to use secrets  https://review.openstack.org/356098
21:25:27 <sdake_> pavo
21:25:27 <sdake_> i'm beat
21:25:33 <sdake_> been up since 4am
21:25:40 <sdake_> need power nap - can it wait
21:26:57 <openstackgerrit> Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding NOTE for Kubernetes Secret Generator  https://review.openstack.org/356719
21:28:46 <Pavo> sure
21:29:14 <kfox1111> weird... they make symlinks for configmaps...
22:14:24 <openstackgerrit> Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance-api  https://review.openstack.org/354895
22:15:01 <kfox1111> sbezverk: I think that one's ready for review now.
22:16:44 <kbaegis1> sdake_ do you know where the openvswitch module is expected to be found?
22:20:00 <kfox1111> sbezverk: with that review, and a ceph backed glance, you should be able to do a no downtime minor version rolling upgrade.
22:51:13 <kbaegis1> lol
22:51:17 <kbaegis1> "modprobe: ERROR: could not insert 'openvswitch': Exec format error"
22:51:19 <kbaegis1> Great
22:52:50 <kbaegis1> Falling back to linuxbridging, trying the iptables —flush option
23:27:31 <kbaegis1> Sadly, no.  iptables —flush didn't work :'(
23:28:14 <kbaegis> Oh well
23:33:34 <kbaegis> is there a way to manually insert those rules?
23:33:39 <kbaegis> To see rejections/conflicts?
23:34:53 <kbaegis> Alright, I'm going to cave and install centos
23:34:54 <kbaegis> :/
23:35:06 <kbaegis> Not happy about it, but it'll to in a pinch
23:35:07 <kbaegis> :)
23:35:23 <kbaegis> And I'd rather have a functioning lab than one done right :)
23:45:54 <kbaegis> Wish I knew more than that iptables.manager generically fails on COMMIT
23:46:03 <kbaegis> I feel like that's solvable
00:28:47 <openstackgerrit> Jeffrey Zhang proposed openstack/kolla: Bump the ansible version to 2.1.1.0 in kolla-toolbox  https://review.openstack.org/356311
00:36:55 <sdake> pavo back alive
00:37:53 <duonghq> morning sdake
00:38:19 <sdake> duonghq afternon actuallyy
00:38:23 <sdake> powernap + powerdinner
00:38:25 <sdake> ready to rck
00:38:28 <sdake> any9one need any help
00:39:08 <duonghq> what is current temperature sdake?
00:39:19 <sdake> hot
00:39:36 <duonghq> dose the winter is better?
00:39:39 <sdake> 5pm
00:39:49 <sdake> winter is much better
00:39:50 <sdake> prorbably 105-1110 atm
00:40:20 <sdake> 110 that is
00:41:04 <duonghq> good weather, huh? does it rain?
00:41:39 <duonghq> I mean winter
00:42:41 <sdake> during summer yes
00:42:44 <sdake> during winter no
00:42:50 <sdake> during summer we have monsoons
00:43:07 <duonghq> monsoons festival?
00:43:20 <sdake> monsoons are like a big rain storm
00:43:32 <duonghq> ah
00:43:37 <sdake> which cools off the heat
00:43:49 <sdake> peopel in arizona are angry during summer time
00:43:51 <sdake> and happy during winter
00:43:54 <sdake> its kind of wierd
00:44:00 <sdake> i think its opposite to the rest of the us ;)
00:44:10 <duonghq> there is storm come here in about 1-2 days :(, long time no storm go here
00:44:12 <sdake> angry might be the wrong word
00:44:19 <sdake> displeased might be better
00:44:32 <duonghq> I like the winter, too
00:44:54 <sdake> i've lost 10 punds in the lat 1 month because of the heat affecting my appetite
00:44:59 <sdake> continuous problem in ariona
00:45:25 <sdake> arizona isn't problem free
00:45:32 <sdake> but all the datacentres in the us are here
00:45:36 <sdake> becuase nothing ever goes wrong
00:45:41 <sdake> except its hotter then hell ;)
00:45:50 <duonghq> yay
00:46:07 <sdake> no tornadoes no earthquakes no nothing
00:46:29 <duonghq> just hot?
00:46:39 <sdake> yup just hot
00:46:49 <sdake> and 280% murder rate above avereage
00:46:52 <sdake> although i don't live in that part of town
00:47:19 <sdake> town/city
00:47:41 <sdake> murder rate increases in summer time - presumably because of resource constraints
00:47:56 <duonghq> bad "habbit"
00:48:54 <sdake> i feel safe all the time
00:48:57 <sdake> i dont even lock my doors
00:49:19 <sdake> others - in other parts of town - have 5 deadbolts on their door
00:49:32 <duonghq> so good for you
00:49:39 <sdake> location location location
00:49:50 <wirehead_> Most of the year, it’s the valley of the sun.  Part of the year, it’s the surface of the sun.
00:50:07 <sdake> wirehead_ wow - never picked up on that
00:50:18 <duonghq> the chant spirit from list meeting still around,
00:50:29 <sdake> wirehead_ surpried- and i'm 42 years old :)
00:50:29 <wirehead_> One of my grand-bosses at Rackspace lived in Phoenix and said that.
00:50:55 <sdake> he must have been grand
00:51:00 <sdake> and more experienced then I
00:51:36 <sdake> wirehead_ do you use jabber
00:51:48 <wirehead_> I think I’m supposed to, but I don't.
00:51:49 <sdake> i see your not on
00:51:57 <sdake> would you mind indulging me for a moment
00:54:13 <sdake> battery at 2% brb if power dies out
00:55:27 <wirehead_> K, well the Jabber client is forcing me to download a 65 meg update.  But I think I’m on.
00:56:38 <sbezverk> wirehead_ I see some issue with neutron boostrap, wondering if you see something similar
00:57:57 <sbezverk> wirehead_ when it tries to run db upgrade on neutron db and is it was already upgraded, I see continuous restart of bootstrap process
00:58:16 <wirehead_> hmmmm.
00:58:42 <sbezverk> wirehead_ wondering if there is way to check if db was already upgraded and then ignore upgrade
00:59:20 <Pavo> sdake cool, I am setting up nodes agan
00:59:31 <wirehead_> That seems like a bug in the update process.
00:59:33 <Pavo> lol like for the 6th time
01:00:04 <duonghq> sdake: how about the SAP case?
01:00:37 <sbezverk> wirehead_ most likely, they should handle it more gracefully
01:01:01 <sbezverk> will open a bug tomorrow against neutron
01:01:08 <Pavo> so whats the most stable release 2.0.3?
01:03:00 <duonghq> sdake: how about the SAP case?
01:03:27 <sdake> wirehead_ around on jabber?
01:03:28 <sdake> i lost power
01:03:29 <sdake> duonghq huh?
01:03:56 <duonghq> SAP use our work without license include
01:04:25 <sdake> duonghq someone told me that had been fixed
01:04:28 <wirehead_> sbezverk: When I test, I usually nuke everything and start from scratch.
01:04:30 <sdake> have a link to the offending repo?
01:15:30 <duonghq> sdake: still there?
01:17:37 <kbaegis> sdake I still need help :)
01:26:24 <kbaegis> I guess another way to go would be to build my own l2 agent
01:26:45 <duonghq> kbaegis: from scratch?
01:29:19 <kbaegis> No alternative :)
01:29:37 <kbaegis> duonghq so I guess so.  Centos works for EVERYTHING but ovs or linux bridging
01:29:47 <kbaegis> there's vikraman/gentoo
01:29:55 <duonghq> how about the ubuntu ones?
01:35:56 <kbaegis> Might do.  I didn't test that.
01:38:16 <kbaegis> I'll hardcode and reset.  I'm pretty sure it's a kernel version thing though
01:38:54 <kbaegis> who's responsible for the centos one ooc?  Could give me pointers?
01:42:38 <sbezverk> sdake ping
01:43:10 <duonghq> sbezverk: he is go out
01:43:31 <sbezverk> duonghq thank you will try to catch him later
01:45:47 <duonghq> (try stop openstack meeting bot from logging from last meeting, insane bug)
01:45:50 <duonghq> #endmeeting