16:35:46 #startmeeting kolla 16:35:47 Meeting started Wed Aug 24 16:35:46 2016 UTC and is due to finish in 60 minutes. The chair is zhubingbing. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:35:48 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:35:51 The meeting name has been set to 'kolla' 16:36:08 zhubingbing, meeting is in #openstack-meeting-4 16:36:57 rhallisey: it doesn't appear to be ? 16:59:51 zhubingbing wrong room ... :) 17:00:06 zhubingbing can you please end meeting here 17:00:29 so bad, when somebody start meeting out of room, the bot will crazy 17:00:43 somebody should backup meeting log 17:00:48 sdake: can you do that 17:01:00 duonghq that is infra's job 17:01:02 maybe we should make the backup first... :) 17:01:03 it is recorded 17:01:05 like irc is 17:01:21 but when somebody end meeting here, we'll lose the log 17:01:39 duonghq meetings are timestamped 17:01:42 #endmeeting 17:01:46 why is it possible to start the kolla meeting here? should this be possible? 17:01:46 * sdake groans 17:01:54 berendt yes its possible 17:02:07 but it is not possible to end it :) 17:02:15 berendt: this is what happen last weeks 17:02:30 berendt: you can see last week log 17:02:51 zhubingbing, please do #endmeeting kolla 17:03:03 inc0: anybody can endmeeting kolla 17:03:12 ok 17:03:28 quite disappoint about the meeting bot 17:03:31 duonghq sdake tried it but no end messages appeared? 17:03:53 onlly the chairs can control the meeting ot 17:04:04 anyway dont sweat it 17:04:15 lets focus on reviews and osic plz :) 17:04:30 pbourke how do you run tempest 17:04:38 sdake: that im not sure of 17:04:39 sdake, berendt, http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-08-17-16.00.txt 17:04:52 I am out for today, will spend some time on reviews tomorrow 17:05:10 sdake: best to check with coolsvap when he's on 17:05:18 heading home now be back on tomorrow 17:05:27 berendt: just last week irc meeting log, somewhat funny 17:05:39 Hui Kang proposed openstack/kolla: Add etcd container https://review.openstack.org/355156 17:06:12 logs from today still there http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-08-24-16.00.log.txt 17:06:17 sdake: iirc: the baremetal role get in master already 17:06:27 inc0: could you have another look at https://review.openstack.org/#/c/337594/ ? 17:06:31 egozales90: good news, still cannot understand 17:06:32 inc0: I dont like the idea of pause 17:06:39 inc0: would like to see it merge for n3 17:07:00 inc0: keep in mind this is a very real scenario that can cause data loss 17:07:03 sdake: ^ 17:07:11 pbourke can you mvoe the iwnodws around in the osic cluster 17:07:14 so they appear properly 17:07:58 egonzalez90: https://review.openstack.org/#/c/359596/ -> gate is already green, would you mind re-review it. 17:08:03 sdake: i cant handle all the splits either :) 17:08:04 pbourke, I know, however having this as a config option will mean people who actually wants to redeploy will have a hard time 17:08:10 and it won't be easy to figure out 17:08:30 I'd be ok with Jeffrey's suggestion of a mixture - you can configure behaviour with config 17:08:39 but unless config is set, pasue will kick in 17:08:47 where did the splits go 17:09:08 like ceph_osd_behaviour: "keep" or "wipe" 17:09:36 and unless it's one of these it will pause "do you really want to wipe you drives? to automate set ceph_osd_behaviour" 17:09:39 sdake: i thought you asked me clean them up 17:09:45 no 17:10:00 i wanted them fixed not deleted :( 17:10:12 for me fixed == deleted :p 17:10:25 sure you can create them again the way you want them 17:10:47 inc0: ok thanks for that, I'll take a look at doing it that way 17:11:08 I almost finish the bp ansible-specific-task-become, hope that somebody review my works when you have free time 17:11:46 pbourke there all fixed 17:12:54 duonghq sent a recheck. Gates fix was merged today, all should be green after recheck 17:13:58 but ubuntu xenial tox :( 17:14:12 egonzalez90: I have some doubts, but let see 17:18:19 hi sdake, I just sent you an email regarding some question, can you take a look 17:20:47 egonzalez90: I need sleep now, see you tomorrow 17:22:18 wally__ i don't provide support from my inbox sorry 17:22:36 the community provides limited support to get you going via irc however 17:22:37 ok my quesiton is 17:23:13 i just re-do again, and I ran the kolla-ansible prechecks 17:23:23 i got this error 17:23:51 http://pastebin.com/jD3BJXM4 17:24:05 can you please take a look 17:24:25 i have configured my network as you said 17:24:31 wrong version of ansible 17:24:58 root@controller:/etc/kolla# ansible --version ansible 1.9.4 configured module search path = /usr/share/ansible 17:25:07 the same as yeterday 17:25:32 TASK: [prechecks | Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes] *** fatal: [localhost] => Failed to template {% if '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3]) %} True {% else %} False {% endif %}: an unexpected type error occurred. Error was argument of type 'NoneType' is not iterable 17:25:49 not sure then 17:25:51 this error, is ansible version peoblem? 17:25:53 too busy to help 17:26:05 t-6 days to milestone 3 and end of osic gear 17:26:21 so you will either have to wait or find other help in channel o fwhich there is plenty :) 17:26:41 ok, thanks a lot, 17:27:32 anyone knows what the error is ? 17:27:57 im looking now 17:28:23 can you post your globals.yml 17:28:32 ok one sec 17:29:23 http://pastebin.com/BUuAixZv 17:30:15 what are the IPs for eth0 and eth1 17:30:25 i post my ifconfig here 17:30:26 http://pastebin.com/qbcmBE3e 17:30:41 there is no ip for eth1, i did not set ip 17:30:57 so you need two IPs, one for mgmt and one for 'externa' 17:31:01 so two adapters need to be configured 17:31:14 ok, i see 17:31:15 and the VIP belongs on the external (i think) 17:31:20 so set an IP on both adapters 17:31:35 so now I have eth0, which is 10.145.213.51. that is management? 17:31:47 based on what you have in globals 17:32:01 line 61 and 74 17:32:21 when i set it up, it was all virtual so, same vlan 17:32:28 but, im certain this is whats going on :) 17:32:40 ok, let me try 17:32:49 eth1 is def your external, and eth0 is mgmt, 17:33:02 just not sure where the vip actually belongs 17:33:28 jrich523: iirc the external interface should be no ip is set, neutron will do its work 17:33:56 hmm wel lthen he needs to change the interface i think 17:34:18 the Internet is laggy here, cannot look at his link 17:34:45 but if it's kolla_internal_vip, so it belongs to network_interface 17:34:46 thats some serious lag, but pastbin is horribly covered in ads 17:34:55 yup, which in his case is eth0 17:35:13 well, then need 2 ips or 1 ip set? 17:35:18 also way different 17:35:24 confused 17:35:34 wally_ you set up all-in-one or multinode? 17:35:35 his IP on eth0 is :10.145.213.51 and the VIP is 10.10.10.254 17:35:39 all in one 17:35:57 i am going to setup eth1: 10.10.10.51 17:36:12 so you can safely disable haproxy (by add enable_haproxy: "no") 17:36:13 according to duonghq you dont need to, so, you're probably ok there i think 17:36:28 and set the VIP to your real ip of network_interface 17:36:44 wait, what? 17:37:02 just for test, because of if you enable haproxy, the VIP must different than the NIC IP 17:37:06 the VIP should be its own IP, on top of Network_interface's (eth0) IP 17:37:07 this is the ip 10.145.213.51 , I ssh into the server, 17:37:12 ahhhh 17:37:21 well unless you've got a crazy netmask 17:37:31 the VIP and eth0 ip are WAY different 17:37:43 auto eth0 iface eth0 inet static address 10.145.213.51 netmask 255.255.255.0 dns-nameservers 8.8.8.8 gateway 10.145.213.1 17:37:50 this is my interfaces file 17:37:56 jrich523: if you disable haproxy, so VIP == network_interface 17:38:09 yeah i suppose thats fine for what he's doing 17:38:15 not too bad setting for 1st run 17:38:17 so what is the global file 17:38:22 you still have to provide the VIP in globals tho right, just happens to be the same IP? 17:38:32 globals.yml that you posted for me 17:38:32 right 17:38:51 but you said what do i need to edit in the global.yaml 17:39:23 two things, but im looking for the first :) 17:39:30 wally__: your 2 IP is in different subnet, but the error you post is not relate the that problem (iirc) 17:39:39 *the -> to 17:40:11 i only setup one 1p now, which is 10.145.213.51. and I use this ip to ssh into the server 17:40:19 I guess that the problem is in " ipaddr(ip_addr_output.stdout.split()[3])" part 17:40:27 no idea 17:40:44 ok folks working on osic - i've got it down to 8 primary scenarios 17:40:44 wally_ yeah,... I think it is fine 17:41:04 how is the network diagram setup in kolla, I am confused 17:41:14 no no 17:41:25 so based on the test name: Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes 17:41:39 it's too much for today, need some sleep, sorry wally_, jrich523 17:41:40 his eth0 ip is a different subnet from the VIP 17:41:45 bye 17:41:51 make the VIP something on the same subnet as eth0 17:41:52 but I got the error when do the pre-check, 17:41:56 right 17:42:06 its checking to see if the VIP are on the same network 17:42:07 ok let me see 17:42:08 which, they are not 17:42:24 also, it oddly assumes a /24 network, kinda odd 17:42:39 change the VIP so that its in the same subnet as eth0 17:43:04 ok let me try 17:43:19 eth0: 10.145.213.51 / VIP 10.10.10.254 - different subnets 17:43:32 make the VIP something like 10.145.213.52 or whatever is free 17:44:11 thx 17:51:38 jrich523 ya /24 is a bad assumption - ip interface shoudl be read 17:51:56 still early, the assumption is fine for now :) 17:52:13 but yeah, at it to that long list of things to do :) 18:11:11 inc0 - where is the tempest .testr. file 18:14:14 no clue sdake - I don't know tempest at all 18:14:23 * sdake groans 18:14:37 find is your hope 18:14:37 coolsvap is the one who got that setup originallly 18:14:48 i already used find 18:14:51 $find -n .testr 18:15:17 but I think rally runs tempest 18:15:27 tempest is kinda embedded in rally somehow 18:15:29 not sure how tho 18:19:23 inc0 i find that highly unbelievable 18:20:18 hrm 18:20:23 i guess it does integrate with it 18:22:44 sdake, look at tmux 18:23:03 ya who do you think is typign that in 18:23:19 well fuck. 18:23:30 oh i see what you mean 18:23:31 say 18:23:36 did the last run of stuff 18:23:39 have a time on it? 18:23:42 can you scrol up? 18:24:14 it;s in rally db 18:24:22 6 hours 18:24:30 Merged openstack/kolla-kubernetes: Introduces new approach in starting OVS in Kube https://review.openstack.org/358895 18:24:34 what is in rally db 18:24:43 results of all ran tests 18:24:46 yes 18:24:54 now we need tempest in the mix ;) 18:25:05 and work can actually begin 18:25:27 this is how you run tempest 18:25:32 but something is bad 18:26:32 Merged openstack/kolla-kubernetes: neutron openvswitch agent ip https://review.openstack.org/356114 18:27:05 deps busted 18:27:10 sudo pip -U pip 18:27:24 yeah....syntax error \o/ 18:27:34 sudo pip install -U pip 18:28:02 hi,I just done the deloyment in AIO, it worked, I can login to the horizon, 18:28:14 awesome, way to go wally__ 18:28:19 wally__, do one woot for kolla:) 18:28:42 I am just wondering how can I change the admin login password. it is very long, I changed the .openrs.sh file , but not work 18:28:59 wally__, /etc/kolla/passwords.yml 18:29:10 but once it's set you need to change it in horizon 18:29:14 or keystone 18:29:14 i run this kolla-ansible post-deploy, and get the password 18:29:25 but basically passwords.yml is config file for well...passwords 18:29:42 i just want to change admin login passsword 18:29:52 wally__, log in to horizon with the long one 18:30:06 :( 18:30:07 top right corner afair, you can change password there 18:30:20 but you need to login first 18:30:22 you mean insided horizon 18:30:39 yeah 18:30:57 Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 18:31:20 ok worked 18:31:21 thanks 18:31:25 :) 18:31:26 np 18:31:32 soo...how about this woot for kolla?;) 18:31:52 j/k don't woot unless you really believe in the woot 18:32:01 nice, fast 18:32:26 question, after i reboot the server, all the docker image will start auto? 18:32:44 yeah 18:32:45 inc0 is this how you were doing it previously with reporting? 18:34:17 I just dumped all into files 18:34:27 but don't quote me on that, I'm newbie in rally 18:35:16 someone ran tmepest reports 18:35:38 to me it seems deps are broken 18:35:46 there was a freaking syntax error there 18:37:14 build/temp.linux-x86_64-2.7/check_libyaml.c:2:18: fatal error: yaml.h: No such file or directory^M 18:37:20 can you make that file appear please 18:37:30 probaby PyYaml-dev or something 18:45:34 Kevin Fox proposed openstack/kolla-kubernetes: l3/metadata DaemonSets + DVR, & labels https://review.openstack.org/357557 18:54:41 question, I create a vm, but the vm canot go outside, and I cannot ping and ssh into the vm, how is the network for the vm? 18:54:46 damn thought you wer ein a venv inc0 18:54:47 sorry 18:54:58 let's try now 18:55:05 yup 18:55:14 source creds first 18:55:25 nah 18:55:28 rally has it covered 18:55:30 sweet it works 18:55:47 but we'll need to hack around it every time we do new deployment 18:55:49 which sux 18:56:00 bummer its not easy to install 18:56:01 we can cp -R this working venv 18:56:06 to new env 18:56:12 hacky but should work 18:56:13 ya i'll create a taraall of root 18:56:26 just this venv 18:56:33 i already baked up /root 18:56:36 and extract it to .venv in dir created 18:56:37 but will do again 18:56:47 we're running out of space btw;) 18:57:00 not surprising 18:57:03 also it's you who look at this terminal on a poststamp? 18:57:03 what do we have left 18:57:11 it's like 1/6 of my screen 18:57:22 ya laptop - busted screen 18:57:30 you know the story 18:57:35 yeah 18:57:41 "dont stand left of me" 18:58:05 haha 18:58:13 say do you know how we mae rally get a new database 18:58:29 yeah it's just sqlite somewhere 18:58:43 i'd like to remove the db and start fresh 18:58:51 but waotor this tempest run 18:58:58 i need to see how it is reported 19:00:04 inc0 i see 19:00:05 thanks 19:00:22 is the temepst job still running? 19:02:13 Kevin Fox proposed openstack/kolla-kubernetes: openvswitch service hookup/rename https://review.openstack.org/360051 19:02:55 it is. 19:03:47 inc0 back killing me - going to lay dow for an hour or two - and then i'll be ready to go with a full run 19:03:59 kk 19:04:00 i'll take care of running and getting output for scenario2 19:04:04 we need then scenario 3 setup 19:04:10 I need to get back to patches 19:04:11 scenario runtime is probably 4-8 hours 19:04:15 right 19:04:16 let me know when you're back up 19:04:35 i struggle with setup because I don't know the network environment 19:04:52 perhaps you can show me how you go thorugh that so I can just get this done 19:04:55 everyone struggles with setups, that's why kolla exist 19:04:58 we have all the pieces in place 19:05:08 well I struggle because I lack network informatio ;-) 19:05:33 anyway bbiaf 19:05:43 need tha t tempest to finish before i can procee 19:09:49 i created a vm . but the VM canot reach outside, how is the network setup for a VM? 19:34:05 Kevin Fox proposed openstack/kolla-kubernetes: openvswitch service hookup/rename https://review.openstack.org/360051 19:34:38 Hi Folks, has anyone tried setting up docker on ubuntu 14.04? When I am trying to install I am getting hell of dependencies which I never saw earlier and unable to find a way around 19:34:52 *recently 19:36:34 adnavare: can you create a paste with the output of sudo apt-mark showhold as well as sudo apt-mark showmanual 19:36:53 wondering if there are any packages on the hold list that are the sam ebeing complained about 19:38:27 mark-casey: showhold did not returned anything 19:38:37 showmanual output is here http://pastebin.com/n0zgZSQP 19:41:01 mark-casey: yesterday i got to know docker has a new release that only supports ubuntu xenial, but i am wondering there has to be a way for trusty 19:43:27 re: Xenial they've stated that publicly or that was a theory? 19:56:56 Kevin Fox proposed openstack/kolla-kubernetes: l3/metadata DaemonSets + DVR, & labels https://review.openstack.org/357557 19:58:46 Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 19:59:29 Merged openstack/kolla-kubernetes: openvswitch service hookup/rename https://review.openstack.org/360051 20:04:42 mark-casey: I am not sure, I got to know from sdake 20:05:25 mark-casey: can we actually try to get docker 1.10 specifically? So that I don't need these dependencies 20:07:01 mark-casey: i have this in my sources.list.d/docker.list "deb https://apt.dockerproject.org/repo ubuntu-trusty main" 20:07:27 and then when i try to do apt-get install docker-engine it is throwing all these dependencies 20:19:04 hi guys, after I created VM, but the VM cannot go to internet, how is the network setup? 20:19:20 i cannot ping and ssh into the VM 20:25:33 wally__ does your secuity groups in openstack have SSH and ICMP ingress added? 20:26:16 and for the instance to have internet access you have to have an external network inside openstack configured with DNS to resolve 20:30:01 yeah, i have ssh an dcmp ingress added 20:31:07 I setup all in one, and my sever has internet access, 20:31:27 how do u mean by external network inside openstack configured 20:36:20 Michal Jastrzebski (inc0) proposed openstack/kolla: Customization for base https://review.openstack.org/358042 20:38:33 adnavare: IIRC the errors you were getting don't mean it doesn't have the dependencies, they mean it is holding on to older versions of those dependencies for an unknown reason. If you want to see which versions are currently available you could 'sudo apt-cache show docker-engine | grep Version' 20:43:58 adnavare: if 1.10 is listed then installing it may be as simple as sudo apt-get install docker-engine=1.10.0-0~trusty 20:44:19 or there are also a few point revisions like 1.10.3 20:59:34 jrich523, just finished meddling with headphones, it was easy indeed 21:14:03 mark-casey: Okay yes install docker-engine=1.10.0-0~trusty did solved the problem. Somehow earlier it did not worked. 21:14:20 mark-casey: may be i tried something wrong the last time. 21:39:24 would someone mind reviewing this when they have some spare time? https://review.openstack.org/#/c/358305/ 22:06:34 Hello, I am planning to work on https://bugs.launchpad.net/kolla/+bug/1616387 - and am wondering if there is any corner cases that I should be aware of. Besides removing all references to fedora. 22:06:35 Launchpad bug 1616387 in kolla "Remove base_distro type fedora from Dockerfile templates" [Undecided,New] 22:19:21 inc0 around? 22:19:29 yeah 22:19:38 can you make rally verify install work again 22:19:41 i am not ure what you did 22:20:00 i hae the virtual env 22:20:02 but i'm not sure where it goes 22:20:07 its for-deployment-id-buildenv 22:20:35 here 22:35:53 inc0 after i recreated the databasea 22:35:57 and played with a few things 22:36:02 now rally isn'tworking at all 22:36:13 inc0 almost got it entirey scripted on the gather data front 22:38:11 ehh I don't knwo rally:S 22:39:23 there was another script here right? 22:39:32 ya i renamed it tot his one 22:39:38 and going ot make a new one ot run everything 22:39:52 also temest works but fais 22:40:50 Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 22:41:15 inc0 i'm goin to go get lunch or breakfasat or whatever 22:41:32 inc0 bbi20-30 mins - please see if yo u can get that working 22:41:38 let's hoep 22:44:51 inc0: glad to hear it went smoothly :) 22:45:19 yeah thanks for pointers 22:50:38 sdake, works, somebody messed up admin openrc. Also moved deployment creation out of forloop. 22:50:53 glad to help inc0 :) 23:06:26 sdake: ping 23:08:04 inc0 thanks so ready to go then on the test run? 23:08:21 1 conc and 1 times is runnihng now 23:08:30 feel free to trash db again and redo whole thing 23:09:09 i always use /etc/kolla/admin-openrc 23:09:12 was that file modified? 23:09:31 it was invalid 23:09:48 nova list worked with those creds ;-) 23:10:19 yeah 23:10:24 but not everything 23:10:37 OS_PROJECT_DOMAIN_ID=default 23:10:42 I changed it to DOMAIN_NAME 23:21:14 ok 3 tests are running 23:21:27 in serial order 23:21:36 then we need to run tempest manually 23:21:46 because the install is manual 23:22:10 inc0 re https://etherpad.openstack.org/p/kolla-N-midcycle-osic 23:23:06 inc0 how many storage/compute/control nodes does scenario 2 have? 23:23:13 and what networing is in use? 23:23:44 just pick a numbe 23:23:45 r 23:23:51 inc0 i want to consolidate some scnearios 23:23:55 how many do we have now 23:24:03 and as networking goes, it should be ok as it is 23:24:16 inc0 one of our scenarios is linuxbridge another is ovs 23:28:20 inc0 what is the current inventory file in use? 23:28:33 ~/inventory-ips-only afaik 23:28:38 brb gonna eat sth 23:30:43 cool 3 control 100 compute 20 storage 23:30:46 can consolidate 23:39:22 we basicalaly have 6 data captures in total per distro now 23:39:24 much mor etidy 23:39:37 scenario 2 is capturing now 23:39:54 once we run tempest need to deploy scenario 3 23:48:12 hi guys, Is there a network diagram for kolla deployment, my vm still cannot reach outside, I dot know how does a VM can reach outside? 23:48:40 inc0 2016-08-24 18:48:22.205 12258 INFO rally.task.runner [-] Task 477b86f1-3e08-4a15-a692-a5fd55ce1eb7 | ITER: 111 END: Error IpAddressGenerationFailureClient: No more IP addresses available on network b73d8375-e54b-46cf-8070-7d5c7966444c. 23:49:05 lol 23:49:15 we should redeploy whole thing 23:49:35 i think that seems reasonable 23:49:39 how long will that take you :) 23:50:05 dont delete files in /etc dir 23:50:10 they contain custom quota config 23:50:49 sdake, should be about 20min;) 23:50:55 wanna do the honors? 23:51:00 including ceph? 23:51:06 nah - not sure i woudl get ceph right 23:51:08 ahh...with ceph 23:51:10 then 21min 23:51:16 i see some special playbooks for dealign with the disk labeling 23:51:17 I can watch 23:51:25 how about i wtch you drive 23:51:54 kk 23:52:04 I'm killing this thing 23:52:50 what about disk labeling? 23:52:57 oh aftre destroy 23:54:12 cmdfailed? 23:54:23 jeffrey had soem special playbook to deal with the disk partitions iirc 23:59:33 is 2.0.3 tag associated with kolla stable/mitika branch? 00:00:05 aNupoisc yes - 2.0.3 is unreleased s of yet 00:00:07 or is the latest tag is 2.0.2 with stable/mitika 00:00:16 the latest release is 2.0.2 00:00:35 you cn check the docs - they receently changed on install method depending on whether your doing an eval or dev 00:00:47 so if i clone kolla stable/mitika what tag would I get? 00:00:53 2.0.3 00:01:06 but that wont work correctly without hacking 00:01:27 read quickstart - it covers this in detail 00:02:25 inc0 init-runonce-osic does the network setup and image loading? 00:02:32 yeah 00:02:40 anything needed after that ? 00:02:50 should be enough 00:03:48 sdake: ahh that's why. I am getting an error "Tag 2.0.3 not found in repository docker.io/kolla/ubuntu-source-heka" so wondering did I cloned something wrong 00:04:05 i think next ste pfor somoene thta understands the caching layer in ceph is to create a labeling playbook for that 00:04:09 sdake: thanks, I will clone with tag 2.0.2 00:04:10 aNupoisc did you setup your docker registry in globals.yml? 00:04:19 aNupoisc you cn still work from git if you want 00:04:35 aNupoisc to do that, set openstack_release: "2.0.2" in globals.yml 00:05:38 inc0: my docker registry is default #docker_registry: "172.16.0.10:4000" 00:05:54 sdake: Okay let me try with that approach 00:06:26 aNupoisc, this is commented;) 00:06:46 ya he wants to pull from the hub 00:07:29 Yes. 00:07:41 do i need to enable this? inc0 00:07:50 not if pulling from hub 00:07:53 only if you're running local registry 00:08:21 i am running on all-in-one environment and i read no need of local registry 00:08:36 even multinode you cana pull from hub 00:08:39 its just slower 00:08:54 sdake: inc0: okay so for my case what I am doing is correct 00:08:59 yup 00:09:07 i just need to get kolla of tag 2.0.2 00:09:25 aNupoisc 2.0.3 is compatible with 2.0.2 images 00:09:35 our z streams are bug fix only 00:10:44 sdake: But it is throwing an error "Tag 2.0.3 not found in repository docker.io/kolla/ubuntu-source-heka" when i try to deploy 00:12:53 sdake: I am trying to deploy ceph with kolla 00:13:19 so not sure openstack_release=2.0.2 or not? 00:13:21 ceph aio requires specil config 00:13:35 read thee ceph guide 00:14:49 sdake: yes i am following https://github.com/openstack/kolla/blob/master/doc/ceph-guide.rst#managing-ceph but I am at deploy step 00:14:59 and it is failing at heka bootstrap container 00:15:13 sdake: till now I have installed docker and run prechecks 00:15:20 in globals.yml 00:15:23 do you have the line 00:15:35 openstack_rlelease: "2.0.2" 00:15:42 sdake: nope 00:15:55 pretty sure i mentioned that as a workaround at timestamp 17:04:31 ;) 00:16:19 yeah, you mentioned 00:17:09 sdake: but I was not sure to use openstack_release as I don't want all opestack services as i am deploying ceph 00:17:14 so confirmed 00:17:37 you need to use that flag - it wont enable extra services 00:17:43 sdake: so are you saying if i put the option and run deploy then it would work 00:17:46 sdake: okay 00:17:47 unless you use the enable_service: true or whatever it is 00:17:55 aNupoisc yup thats right 00:18:05 sdake: no i am enabling just the service that i want. :) 00:18:26 sdake: that's quicker way then cloning again with tag 2.0.2, cool 00:20:33 Li Yingjun proposed openstack/kolla: Add option to support dhcp/l3 agent ha https://review.openstack.org/352180 00:27:17 sdake: yeah that worked 00:27:21 :) 00:28:08 sdake: so that means even if i have 2.0.3 it uses images of 2.0.2 00:28:14 is that correct 00:28:18 yup 00:28:41 sdake: but why does not use 2.0.3 images 00:28:52 see pbr note in quickstart 00:29:05 there are no 2.0.3 images in the docker hub 00:29:06 because it is unreleased software 00:30:12 sdake: hmm. interesting 00:30:48 sdake: great to know 00:34:07 inc0 not runonce 00:34:11 it has to be in the parent script 00:34:27 no, as we create deployment here 00:34:34 also we need to hack around this deployment crap 00:34:35 and tempest shoudl be installed 00:34:37 you can create a deployment in the parent too 00:36:40 we need that line there 00:36:46 i want a separate deployment per rally scenario 00:36:50 for easier data collection 00:40:05 source dir i swrong 00:40:55 inc0 cp -ar 00:41:20 probably want a rally verify install in there 00:41:25 to make sure rally knows tempest is installed 00:41:57 lets clean db 00:42:04 yeah good idea 00:42:05 and give it a spin 00:42:20 btw we could just dump sqlite after each iteration 00:42:26 so we'll have all the rally data 00:42:41 rather have the json 00:42:54 there is a capture script in there too 00:42:56 which needs work 00:42:59 verification data is somewhere else 00:44:05 looks like that worked 00:44:11 yeah 00:44:12 sdake: OSIC worked? 00:44:34 duonghq we hae almost all data capture ully automated 00:44:42 that is what i was referring to 00:44:55 sdake, I'd just copy sqlite and dump stuff later 00:45:00 with htmls and shit 00:45:10 dangerous 00:45:13 i'd rather have json 00:45:19 we will convert json to rst 00:45:19 archiving whole sqlite as a precaucion 00:45:28 let's archive all of it 00:45:35 ok archive for backup wfm 00:45:39 sdake: it can run automatically or still somebody must kick it off 00:45:53 duonghq well yes someone needs to type in one command... 00:46:05 after getting the escenario we want 00:46:09 which is a little morecomplex ;) 00:47:48 need another grep in those tasks 00:48:18 roger 00:48:51 so tired 00:49:23 it is your shift atm? 00:51:26 sdake, just dump all of it 00:51:30 tomorrow we'll make it better 00:51:35 that isn't good enough 00:51:42 oh you mean dump everything 00:51:43 dumping all of them? 00:51:48 yeah 00:51:52 i expect to be on scenario 3 by today 00:52:05 hmm, gate still is unstable, sad 00:52:05 we dont have time to wait around until tomorrow 00:52:07 someone else will have to bump up the number 00:52:42 there are two operations 00:52:45 one to capture html 00:52:48 one to capture json 00:55:50 inc0 pretty sure the line includes the name which is 128x24, 128x48, or 128x96 00:56:05 the problem with that copying the lines 3 times is that the output to wrong dir 00:56:17 i guess we could hardcode the output dir too ;) 00:56:29 i need to run to circklee k to grab a soda, bbi10 00:57:09 Need ; before do, @inc 00:57:20 Need ; before do, inc0 00:57:22 yeah 00:57:29 hmm, circle-K store become more and more wide spread here 01:01:49 sdake that will be ok 01:01:49 ? 01:03:13 anybody know where is coolsvap? he doesn't active in IRC for awhile? 01:03:26 I haven't seen him for couple days 01:11:15 I hate bash 01:11:46 irc0: what's wrong 01:20:11 I give up. 01:20:58 inc0: can I help you? 01:21:11 nah, I'll take a look tomorrow 01:21:17 cya guys 01:21:23 thanks duonghq tho 01:21:34 cya inc0 01:37:08 inc0 01:55:38 sdake_ inc0 take a rest atm 02:02:23 - - 02:10:05 nearly all gate is green, phew... (except bandit) 02:18:33 ping hui Kang 03:55:00 long time no sê coolsvap_ 03:55:04 long time no see 03:55:24 duonghq: approx 16hrs i suppose :) 03:56:19 not feeling well from yesterday 03:56:42 are you better now? 04:04:07 kinda 04:08:37 anybody got (\"Container command \\'kolla_start\\' not found or does not exist.\" randomly but frequently? 04:20:31 Pavo: are you there? 04:27:59 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 04:28:02 coolsvap_: would you mind review my ps: https://review.openstack.org/#/c/359596/ 04:33:44 duonghq: will do 04:42:42 thank coolsvap_ 05:25:40 Tin Lam proposed openstack/kolla: Remove base_distro type 'fedora' https://review.openstack.org/360249 06:21:39 thank egonzalez90 07:45:44 Hi All, I have deployed openstack multinode set up using openstack-kolla. When I enable l3 HA and create a router and assign subnet router I am observing the below error 07:45:47 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 99; Stdin: ; Stdout: ; Stderr: /var/lib/kolla/venv/bin/neutron-rootwrap: Unauthorized command: ip netns exec qrouter-90a719b6-c447-4dc7-a1c5-e21e16d9f03f keepalived -P -f /var/lib/neutron/ha_confs/90a719b6-c447-4dc7-a1c5-e21e16d9f03f/keepalived.conf -p /var/lib/neutron/ha_confs/90a719b6-c447-4dc7-a1c5-e21e16d9f03f.pid -r /var/lib/neutron/ha_confs/90a719b6-c447-4dc7-a1c5-e21e1 07:46:04 Does any one come across this error 08:08:04 Jeffrey Zhang proposed openstack/kolla: Move to transport_url for rabbitmq configuration https://review.openstack.org/356449 08:54:34 Do we have a documentation of our gate setup? 08:56:11 good morning 09:04:06 Paul Bourke proposed openstack/kolla: Set 'balance source' for Horizon in haproxy https://review.openstack.org/356514 09:09:46 Paul Bourke proposed openstack/kolla: Import the Murano core libary as part of setup https://review.openstack.org/360002 09:32:02 Hey guys 09:32:13 does kolla at somew point run neutron-ovs-cleanup ? 09:32:34 I am running into problems after reboot with my dhcp, l3 and metadata agent 09:32:40 are you aware of this ? : https://bugs.launchpad.net/cloud-archive/+bug/1249708 09:32:40 Launchpad bug 1249708 in neutron (Ubuntu) "Missing neutron-ovs-cleanup service" [High,Fix released] - Assigned to James Page (james-page) 09:35:29 this script should run, when neutron_server starts 09:53:00 Prithiv proposed openstack/kolla: Dockerfiles for collectd Container https://review.openstack.org/358632 09:53:02 Merged openstack/kolla: Updated from global requirements https://review.openstack.org/360234 09:55:36 Vladislav Belogrudov proposed openstack/kolla: Add option to support dhcp/l3 agent ha https://review.openstack.org/352180 10:01:37 Prithiv proposed openstack/kolla: Dockerfiles for collectd Container https://review.openstack.org/358632 10:08:37 Shaun Smekel proposed openstack/kolla: Add full support for fernet https://review.openstack.org/349366 10:09:22 Vladislav Belogrudov proposed openstack/kolla: Allow Neutron to be run in DVR mode https://review.openstack.org/356544 10:13:22 Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 10:16:38 Shaun Smekel proposed openstack/kolla: Customizations for RabbitMQ https://review.openstack.org/354409 10:19:19 Shaun Smekel proposed openstack/kolla: Customizations for MariaDB https://review.openstack.org/354422 10:19:43 Shaun Smekel proposed openstack/kolla: Customizations for MariaDB https://review.openstack.org/354422 10:22:24 Shaun Smekel proposed openstack/kolla: Customizations for Ironic https://review.openstack.org/354427 10:26:16 Shaun Smekel proposed openstack/kolla: Fix incorrectly implemented footer blocks https://review.openstack.org/358193 11:10:27 @Sarya: looks like a rootwrap problem, have you seen this ? :https://bugs.launchpad.net/kolla/+bug/1599489 ? Other people seem to report issues when deploying from images built from sources 11:10:27 Launchpad bug 1599489 in kolla "Neutron Open vSwitch Agent hangs because of Sudo" [High,Incomplete] - Assigned to Michał Jastrzębski (inc007) 11:13:51 Vladislav Belogrudov proposed openstack/kolla: Allow Neutron to be run in DVR mode https://review.openstack.org/356544 11:32:25 Jeffrey4l__: ping 11:36:07 ping pbourke 11:36:13 yingjun: hi 11:37:02 hi pbourke, i answered your comment in this patch: https://review.openstack.org/#/c/358973/ 11:37:40 Could you recheck on that? 11:38:24 yingjun: hmm 11:38:58 hey guys, according to this https://bugs.launchpad.net/cloud-archive/+bug/1249708 neutron-ovs-cleanup should be run in the event of hosts reboot , just before neutron_server starts 11:38:58 Launchpad bug 1249708 in neutron (Ubuntu) "Missing neutron-ovs-cleanup service" [High,Fix released] - Assigned to James Page (james-page) 11:39:27 yingjun: ok maybe it's best to do it at runtime then, but could we do it in the kolla-toolbox start task rather than ceph? 11:39:28 I've looked through kolla sources, and cant find this being used - therefore having issues with my neutron network agents after reboos 11:39:45 should I open a bug in Kolla ? or you guys are aware of that ? 11:40:14 Mr_Broken: sounds like we need that functionality 11:40:17 Mr_Broken: i would log a bug 11:40:37 kk will do, thanks 11:41:26 pbourke, i think it could, i’ll try :) 11:47:33 pbourke, pong 11:48:03 Jeffrey4l__: was just looking again at https://review.openstack.org/#/c/311125/ 11:48:26 Jeffrey4l__: would be great to get that fixed. is it on your radar or would you like me to try submitting an update for it? 11:48:44 Jeffrey4l__: it looks good if we could avoid the extra site.yml 11:49:02 pbourke, OK. i can handle it. 11:49:07 Jeffrey4l__: thanks! 11:49:25 thanks for the reminder. 11:50:15 Li Yingjun proposed openstack/kolla: Fix toolbox permission issue when enabling ceph https://review.openstack.org/358973 11:51:53 pbourke thanks 11:52:02 yingjun: thanks also 11:53:48 Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 11:53:49 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 11:53:49 Mathias proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 11:58:56 What do I need to do to solve these "Patch in Merge Conflict" errors in gerrit? 12:02:42 Paul Bourke proposed openstack/kolla: Customizations for Nova https://review.openstack.org/351676 12:13:25 mewald, yes is needed because some change was merged in some place your patchset is changing 12:13:37 as example inventory files 13:07:08 Paul Bourke proposed openstack/kolla: Customizations for Nova https://review.openstack.org/351676 13:10:10 sean mooney proposed openstack/kolla: generate bifrost yaml configs https://review.openstack.org/349684 13:10:11 sean mooney proposed openstack/kolla: add bifrost deploy ansible support https://review.openstack.org/349685 13:10:11 sean mooney proposed openstack/kolla: introduce bifrost-deploy container https://review.openstack.org/319703 13:10:12 sean mooney proposed openstack/kolla: adds bifrost ssh key generation https://review.openstack.org/350692 13:10:12 sean mooney proposed openstack/kolla: add deploy-servers command https://review.openstack.org/349729 13:53:19 good morning 13:53:27 morning 13:53:30 morning inc0 13:59:44 pbourke, around? 13:59:49 inc0: hu 13:59:51 *hi 13:59:59 mind taking a look at https://review.openstack.org/#/c/358042/ 14:00:23 oracle linux is failing for me and to me it looks like repo issue...is this correct/incorrect? 14:10:11 looking 14:25:13 inc0: need your advise. I am chasing a nova bug and folks from nova channel suggested a change in os_vif package. How I could bring cutomized os_vif package into nova-compute image? 14:25:34 customized how? 14:26:56 inc0: line changed in python code 14:40:44 hmm 14:40:56 sbezverk_, so you can make use of our customization mechanism 14:41:12 to inject something like sed into build 14:41:46 also this will be trickier with binary build 14:41:47 s 14:41:49 meh 14:42:06 no need binary I am on source 14:43:05 yeah but people might be on binary 14:43:05 morning 14:43:07 pbourke where we at with osic? 14:43:15 who uses it at the end 14:43:19 anyway, worth to try 14:43:22 inc0 where we aat on osic 14:43:23 best bet - use customizations 14:43:30 rally is running 14:43:35 that's what I'm seeing now;) 14:43:37 for which scenario? 14:43:57 I have no idea 14:44:05 can you check the nextsteps file in the /root dir 14:44:17 todays deployment tho 14:44:25 you are last author man:P 14:44:38 anyway we probably killed our cloud 14:44:40 * sdake groans 14:44:47 why killed our cloud? 14:44:54 lots of timeouts and shit 14:45:05 timeouts are ok 14:45:06 I bet that space on disk ran out 14:45:13 how they are ok? 14:45:22 because we are running an overload test 14:45:33 hence killed our cloud 14:45:35 i just fnid it hand to believe its been running for 12 hours 14:45:42 as overloaded it 14:46:21 so are you saying someone redeployed the cloud? 14:46:33 no idea 14:47:01 aaaand yeah 14:47:04 we ran out of disk 14:47:05 what type of scenario is it on? 14:47:16 told ya we should mount big disk to var/lib/docker 14:47:19 ran out of disk in ceph? 14:47:23 no 14:47:30 elastic 14:47:34 ceph is ok 14:47:45 but on controls / is on 40G partition 14:47:55 and elastic stores a lot...a lot of data 14:48:40 I'll stop this stuff and fix our env if you don't mid 14:48:51 i'd like to know what scenario its on 14:48:59 run rally task list | wc -l 14:49:18 81 14:49:36 somehting deinatley wrong it was at 60 when i left 12 hrs ago... 14:49:52 it's new deployment 14:49:58 first task was finished 2hrs agio 14:50:01 so it's not that horrible 14:50:02 inc0: the problem is "7server" in http://mirror.centos.org/centos/7Server/virt/x86_64/kvm-common/repodata/repomd.xml 14:50:27 pbourke, so it's not issue with my code? 14:50:37 inc0: there's a sed to swap that with '7' but whatever repo provides that one above must be getting missed 14:50:44 was any datacaptured in scenario-2 directory? 14:50:48 I see 14:50:55 inc0: your change seems to be fine but there must be something odd with it as the other patches aren't failing 14:51:28 sdake, onlu what I was doing yesterday 14:51:30 inc0: I'll try your change locally to see if I can pin it down 14:51:48 why would soemone redeploy the cloud? 14:52:13 as it stands today, it's broken 14:52:59 I'm stopping this crap, fixing env and redeploying 14:53:24 sdake: yeah I just saw rally running and assumed someone was on it 14:57:03 Is it fair to say we should have a '/var/lib/docker is huge' pre-deployment check? 14:57:13 YES 14:57:44 inc0: weird its building locally for me 14:58:16 britthouser: its sort of dodging the issue that we're lashing a lot of large data into one partition though 14:58:51 britthouser: we have ops who dont like this about kolla but I dont have a good answer yet 14:59:29 pbourke first time i've ever heard the complaint 14:59:41 Is it basically all the logs go into volume, and all volumes live in /var/lib/docker? 14:59:47 sdake: yeah i haven't been vocal enough about it 14:59:54 britthouser: yes 15:00:55 one directory or spread across the system - i don't see the delta 15:01:16 its common enough to mount /var/lib/nova/instances on a separate large disk 15:01:19 how do I do this with kolla? 15:01:41 any dont say use ceph ;) 15:01:54 mount /var/ib/kolla on a large disk 15:01:58 sorry v/ar/lib/docker 15:02:23 along with /var/lib/cinder ? and /var/lib/glance ? 15:03:45 it is true there is no spindle separation of storage to specific disk 15:03:57 but that is an orthogonal problem to running out of disk space in general 15:04:06 pbourke: second for question? 15:04:30 I wondering would it be possible to mount /var/lib/docker/volumes/_data/nova on separate disk for example 15:04:33 i haven't tried yet 15:06:47 Jeffrey4l__ still here? 15:06:49 Typically on systems I deploy, /var is my largest partition 15:06:57 sdake, yep 15:07:08 Jeffrey4l__ did you redeploy the cluster last night? 15:07:14 sdake, no.. 15:07:33 ok - there is a file clled NEXTSTEPS 15:07:43 it contians directions for what to do next 15:07:45 sdake, writing the rally script to test the io 15:07:47 we now have 5 days 15:08:14 yep.. 15:09:56 inc0 if you rerun tests amke sure to wipe out db 15:12:32 well fuck 15:12:43 I might have broken our controllers 15:12:50 I'm going to redeploy baremetal 15:12:56 it's long overude anyway 15:13:05 how long will that take? 15:13:14 hour or so 15:13:24 just deploy xenial then 15:13:26 gonna fix the storage issue when I'm at it 15:13:31 not that easy 15:13:41 I don't have preseed xenial image handy 15:14:25 if we can skip to xenial before having done any work 15:14:27 we should do so 15:14:35 at this point we haven't done any work... 15:14:39 centos is same issue exacrly 15:14:48 ok - well wen eed to sovle those 15:14:50 I don't agree 15:14:51 britthouser is your man 15:15:04 I'm about to start writing best practices guide 15:15:05 tell me how long it takes to make an image 15:15:09 if its a 3 day process 15:15:09 lot's of stuff we learned 15:15:11 then we have problems 15:15:43 my point is i'd rather have xenial data then 14.04 data 15:15:55 and at this pace we will not hae time for xenial data 15:17:36 can i get an estimate on time needed to get xenail deployed? 15:17:39 i realize its more work 15:18:10 inc0 britthouser ^^ 15:23:10 so because elasticsearch is full we're tearing down and going ubuntu? 15:23:12 what? 15:23:15 sdake: ^ 15:23:24 pbourke isn't it alrady ubuntu? 15:23:31 oh right 15:23:35 i dont get the teardown 15:23:45 inc0: why are you tearing down 15:23:45 why not just fix the 3 controlers 15:23:47 but whatever 15:23:56 pbourke he cratered the controllers 15:23:59 why still need to do upgrade/stress test scenarios 15:24:00 pbourke, I ran rm -rf /var/lib on controllers;) 15:24:14 but on the other hand controllers were out of space 15:24:22 and thrown a lot of errors 15:24:39 so I'm going to redeploy whole thing and make controllers disks...bigger 15:24:53 pbourke check out etherpad - i tuned our test sceanrios 15:25:08 but woudl like estimate on exenail deployment 15:25:12 if we dont do it now 15:25:18 its not going to happen at all 15:25:27 The import of xenial shouldnt take too long sdake. *but* the risk of going to xenial now is we had to install special driver for the NICs. Not sure how/if/etc that works in xenial 15:25:29 these teset scenarios take6-8 hrs to run 15:25:59 britthouser it wasn't just a kernel update? 15:26:22 chaosmonkey tests would be useful 15:26:29 sdake: sure you want to drop those? 15:26:35 ya lets make it harder 15:26:48 pbourke - we have 5 days to run tatleast 2 platforms worth of tests 15:26:49 I wasn't around for the specifics of what it required to get the NICs to work right 15:26:50 do the math 15:26:54 I came back from travel after the fact 15:27:05 if it was just new kernel, and xenial has kernel, then pobably low risk 15:27:06 britthouser pretty sure itw as just a ernel update 15:28:12 Ok, then probably lower risk. 15:28:21 Lemme login the cobbler container and work on importing it. 15:28:25 britthouser is that something you an hep with ? 15:28:32 yes 15:28:54 thank god ;) 15:29:35 pbourke sorry for being so short - just runnign out of time trying to reduce our scope as much as possible 15:30:03 sdake: no prob 15:30:16 sdake: final question though. benefit of testing on xenial over 14.04 is? 15:30:29 pbourke master onl yruns on xenial 15:30:36 sdake: thanks 15:35:23 britthouser osic uses a cobbler container? 15:35:48 Yeah...an LXC container with cobbler in it 15:36:05 Its running on teh deploy node 15:36:19 cool 15:36:24 root@729494-comp-s3500-002:~# lxc-ls 15:36:24 osic-prep 15:40:42 hmm..how did we add first IP for nodes? 15:41:29 taht was from spreadsheet? 15:41:59 You're talking about the 10.15.243.X inc0? 15:42:36 The ILO addresses were in the email 15:42:48 I cut/pasted them into ilo.csv 15:43:05 no, I already deploy first node 15:43:05 which is just a compliation of m9*.txt files 15:43:09 OS 15:43:13 trying to get to them 15:43:16 Paul Bourke proposed openstack/kolla: Import the Murano core libary as part of setup https://review.openstack.org/360002 15:43:26 then I'm not sure what you're asking 15:44:03 hey rhallisey_ or sdake, have a question as a new guy.... 15:44:26 looking through blueprints...and this one came up (has my interest): https://blueprints.launchpad.net/kolla/+spec/consul-container 15:44:57 could we just use etcd though? how can i make suggestions to the author huikang27? 15:45:19 v1k0d3n, that bp doesn't look very active 15:45:22 etcd is already in the review queue 15:45:23 you can probably take it 15:45:38 there is already an implemtnation of consul in teh queue as well 15:45:41 good. i think etcd meets the request and better aligns. 15:45:44 its just that we dont see a good need for it 15:45:52 ^^ agreed. 15:45:52 v1k0d3n: and depends on if your looking at ansible or k8s. the k8s side is doing service descovery via k8s. 15:45:58 https://review.openstack.org/#/c/355156/ 15:46:11 well, etcd could be used for either, no? 15:46:35 it could, but its extranious in k8s as k8s provides dns based service discovery. 15:46:42 yeah, that review is good. like i said...i'm new to this fulltime role. just taking things in right now. 15:46:49 v1k0d3n, ya I guess we could go with etcd 15:46:53 it's in much better shape 15:47:01 that's my thought too. 15:47:10 v1k0d3n i'd recommend having a look at the review queue 15:47:14 to see whta peopel are working on 15:47:38 reusable configs/skills if we just go with etcd...not that i'd want to limit options later though. 15:47:48 ok. 15:48:53 sdake / rhallisey wanted to also ask about something like LKVM or hypervirtualization of the containers. do we have that on the roadmap at all (support some form of hypervirtualization of the containers)? 15:49:21 youo mean like intel's clear containers ? 15:49:35 yes 15:49:42 increased security 15:49:44 v1k0d3n this project doesn't have a super roadmap :) 15:49:52 v1k0d3n basiclly people work on what interests them 15:49:56 and that defines our roadmap 15:50:08 but i'd liek to see clear containers optionally implemented if its possible 15:50:12 not sure if that iss the case 15:51:05 i think it can be done, but the prep would most likely be out of kolla or kolla-kubernetes. at the kolla level, it would be using the flags to support what's been prepared. 15:51:35 does clear containers support host bindmounts? 15:51:42 does clear containers support host netowrking mode? 15:51:46 or ipc mode 15:51:53 i was going to check on this. 15:51:55 or pid=host 15:52:07 thse are things we need to hve any kind of extra barrier 15:52:15 i probably need to come up with a list of kolla requirements based on what's in place, and then compare. 15:52:22 right, agreed. 15:53:17 it 'should' support docker: https://clearlinux.org/blogs/clear-containers-docker-engine 15:53:56 but does it support all docker flags needed 15:53:58 unknown 15:54:08 right. i will look into it. 15:54:17 rut row 15:54:18 something to blueprint maybe? 15:54:43 i'm 1000% new to blueprints. and i don't want to put something out there just to put it out there. 15:55:06 v1k0d3n that is usually how it wors - think of blueprints as a brainstorming area 15:55:28 britthouser sup problem? 15:55:52 freenode disconnect me. Back now. =) 15:56:17 I need to get the contents of the ISO into the LXC container. Does LXC have something equivalent to 'docker copy' ? 15:56:40 lxc may have binddmounts? 15:56:50 docker copy only copie from insdie to outside the container 15:57:15 I tried to mount the ISO while in the LXC, and got errors, just like I do with docker. 15:57:40 another option is curl 15:57:59 https://paste.fedoraproject.org/413935/14721406/ 15:58:15 Yeah I curl'd the ISO down...or do you mean somethingelse? 16:05:27 thats what i meant - curl inside the lxc ontainer 16:06:11 Sure...I can curl down the ISO to host or LXC 16:06:14 britthouser t oget that to work in docker - you have to bindmount /dev 16:06:22 but I need the files inside the ISO 16:06:35 oh you need to modify the iso? 16:06:49 modify outside - then curl inside? 16:06:50 no cobbler just needs to pull files out of it 16:06:57 i see 16:07:09 does cobbler use mount to do that? 16:07:25 typically user does the mount, and then points cobbler at the mount point 16:07:26 a loopback mount? 16:07:29 right 16:07:42 is ubuntu mounted atm? 16:07:55 I think i found a way 16:08:15 if it just needs the files in the iso, loopback mount on the host, then tar em up then curl the tar in 16:08:26 From host I can mount ISO, and then copy contents into /var/lib/lxc/osic-prep/rootfs/blah 16:08:35 nice 16:09:24 hey folks i am in flagstaff today and have a 2-3 hour travel time back to phoenix around noon pst 16:09:36 fly like the wind! 16:11:05 =) 16:11:26 Eduardo Gonzalez proposed openstack/kolla: Ansible config for nova-serialproxy console https://review.openstack.org/358839 16:12:14 Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 16:13:12 so steve, if i put a blueprint out there about hypervirtualization of the container, would that work? it may actually be a better fit into kolla-kubernetes rhallisey, since kubernetes will soon (or at somepoint) support a runtime interface for other runtimes. 16:13:32 even those runtimes can support docker format, like hyperd 16:13:33 so guys our quickstart gets about 2k page views a month 16:13:41 not bad 16:14:00 oh i noticed that the order was... well... problematic 16:14:09 the python stuff should come frist 16:14:18 i'll fix it if i can ever get down the whole process 16:14:28 sdake, what 16:14:29 but basically ansible needed a dev package that is part of the python setup 16:14:30 wow. that's good. sdake how are you getting those stats? 16:14:37 v1k0d3n magic 16:14:40 sdake, how did you find that out 16:14:49 dont ask questions you dont want the answers to 16:14:49 google analytics? :) 16:14:53 hahaha 16:15:29 i use analytics on my blog. really helps because then i know how users got there, and demographical information :) 16:15:50 i use it on most sites i manage, fairly useful 16:16:36 it may help understand if they're searching for docker on openstack or visa versa :) 16:17:17 well if you're at the quickstart, you're there for a reason 16:17:29 i can imagine you stumble on to kolla, but the quickstart is intentional 16:22:49 Eduardo Gonzalez proposed openstack/kolla: Ansible config for nova-serialproxy console https://review.openstack.org/358839 16:23:32 jrich523: maybe, but not really sure about that. i think there's still some unfortunate misunderstandings about containers/openstack and the use of each. 16:23:52 i'm realizing this at our kubernetes meetups. there's a lot of misunderstanding out there. 16:24:50 i know im completely lost 90% of the time, and im a fairly smart guy... some times.... 16:25:07 but analytics could show new/return traffic. i would imagine a high percentage of return traffic is keenly interested in Openstack on Docker. 16:25:27 jrich523: ha! so true. easy to get lost. 16:25:29 lol actually a chunk of those hits are me :) 16:25:40 yesterday i saw 3 browser windows open to the quick start 16:25:45 not sure why... just too many open i guess lol 16:29:34 ok slight bad news sdake, the cobbler (2.4.1) that came in the container only supports up to trusty 16:31:05 gonna look to see if it easy to make it recognize xenial or not 16:35:26 hi guys, how can I check all logs like nova log, neutron log,? I login different container, but where is the log 16:36:29 wally__ docker exec heka ls -l /var/log/kolla 16:36:37 britthouser thats annoying :( 16:36:41 sdake - cobbler 2.6 supports up through Wily 16:37:05 so there is no way to deploy xenial? 16:37:14 no easy way 16:37:25 we can try to upgrade cobbler in the lxc container. 16:37:42 but even then it will probalby require some small hacking to get it to recnogize xenial 16:38:22 is the cobbler container easy to reproduce? 16:38:23 thx got it 16:38:26 or can it be backed up 16:39:05 Paul Bourke proposed openstack/kolla: Add option ceilometer_database_type for ceilometer https://review.openstack.org/345881 16:39:06 Paul Bourke proposed openstack/kolla: Allow mysql to be used as a database for Ceilometer https://review.openstack.org/360669 16:39:29 the container we pulled down had cobbler pre-installed. It was pretty straightforward to launch. 16:39:42 cool lets try upgrding then and hacking it 16:39:47 by lets i mean yuo :) 16:39:48 we addeda bunch of scripts specific to the topology that we'd want to save off if we had to re-deploy 16:40:05 can you back up that stuff? 16:40:08 Ok let me first see if 2.6 is even available for the distro the LXC is running 16:40:24 I want to restart horizon apache server, I login to the horizon container, but do not see the apache service 16:40:35 how to restart horizon apache 16:40:46 docker horizon restart 16:41:00 or docker restart horizon 16:41:05 ok, thx 16:42:09 yeah sdake the container is trusty, so its repo only holds cobbler 2.4 16:42:39 is cobbler c or python or what? 16:42:40 I think to get 2.6 we'd need a xenial container. 16:43:04 mostly python I think? 16:43:13 try a pip install of it 16:43:37 best to make a backup 16:43:40 Cobbler (0.6.3-2) - Provisioning control application & library for PXE, Virtualization, and Re-Provisioning 16:46:08 sdake: bit tired today so sorry if Im getting in the way here. But - we've been doing kolla master on top of 14.04 fine up till this point 16:46:15 when I do the kolla delploy, in the global.yaml file , I enable enable_murano: "yes", but when the murano pannel, not showing on the horizon dashboard 16:46:37 sdake: your earlier statement about why we need 16.04 actually still doesn't make sense to me 16:46:53 pbourke the master containers run xenail now rather then 14.04 16:47:05 yes but it runs fine on top of a host 14.04 16:47:36 here is the deal 16:47:40 Yeah I feel like we're right on the edge of "this is too risky too late in the game" 16:47:43 if we ar egoing to do any xenail teting at all 16:47:50 but I'm a risk averse person 16:47:50 it needs to be setup now 16:47:56 so lets make a choice as a team 16:48:05 no xenial testing or xenail teseting 16:48:25 before we that we need to ask why we're abandoning 14.04 half way through 16:48:31 *we ask that 16:48:39 we aren't abanondineding 14.04 halfway through 16:48:47 so we have all scenarios done for 14.04? 16:48:48 we have no *resULts* for 14.04 as of yet 16:48:59 we have done no scenarios! 16:49:07 what have we been doing that past few days 16:49:09 but we ehave a fully automated setup for doing the testing 16:49:19 we have been automating evreything so that anyone can run the scripts 16:49:26 and making rally work properly with kolla 16:49:38 tick tock ... one week left, folks! ;) You're welcome, inc0 16:50:02 What if we do this: keep on with 14.04. I'll stand up a second OSIC-prep container, and just play with getting cobble 2.6 and xenial loaded. that would mitigate the upgrade risk while letting 14.04 work continue? 16:50:13 britthouser yes plz 16:50:18 hacking cobbler at this stage in the game is bad idea 16:50:25 pbourke tend to agree 16:50:33 yeah I don't want planA to depend on my hacking cobbler. =) 16:50:39 there's no gain in upgrading the host OS 16:50:44 Happy to make that planB,C,D =) 16:50:46 perfomrnace deltas 16:50:59 we started with 14.04, that's what osic recommend 16:51:07 britthouser - pleae do a plan b for xenial - we may or may not have time ot get to testing it if you get the osic prep container working 16:51:09 stick with it, plough through the scenarios 16:51:31 yeah, I need to prep ansible inventoryu 16:51:33 nodes are back up 16:51:37 i honestly thought we were on like #3 as of today before the controllers went down 16:51:39 just ansible doesn't understand this 16:51:40 inc0 ue same inventory 16:51:50 it is setup for the node counts i wwant 16:51:55 3 contor 20 storage 100 compute 16:52:13 pbourke we ehave done a bare metal deploy 16:52:19 pbourke we are on ceph + cinder + gathering data 16:52:24 that's what I'm saying, ansible doesn't understand it for whatever reason 16:52:27 pbourke and our deployment crater 16:52:56 inc0 ok - well whatever you need ot do to get to 3 control 20 stoage 100 comput e;) 16:53:11 i.e scenario #2 16:53:17 pbourke all documentd in nextsteps file ;) 16:54:15 britthouser also if you could get centos sorted out in cobbler 16:54:25 that is actually more important then xenial at this point 16:54:57 centos is part of our plan a - and we have basically 3 days per distro 16:55:08 Ok, the cobbler in the current OSIC-prep should be able to import centos7. I'll do that first. 16:55:24 hi sdake, when I do the kolla delploy, in the global.yaml file , I enable enable_murano: "yes", but when the murano pannel, not showing on the horizon dashboard 16:55:46 wally__ - we dont have plugins directly implemented for services at this time 16:56:05 wally__ but newton will have the ability to do whatever type of plugin you desire ;) 16:56:22 ok i see, so the murano gui not working at this time, right 16:56:47 wally__ - yup i know not ideal for stable branches 16:57:05 wally__ - we are not superpeople :) takes time to develop this stuff 16:57:32 ok, another quesiton, how to restart all different kind of service, like nova-api, neutron, l3, dhcp,, serivce, 16:57:35 pbourke may have some murano gui workaround he can provide you 16:57:52 wally__ our api at present doesn't include a restart option 16:58:09 wally__: we dont ship the dashboard by default unfortuantely 16:58:11 you can use docker manually of course t odo that 16:58:24 wally__: I have a patch but not ready yet. Luckily it's easy to do via templates 16:58:44 wally__ templates are the thing we implemented in newton to make plugins possible for any scenario 16:59:02 wally__: create a file called 'template-overrides.j2' and put the following in it http://paste.openstack.org/show/563425/ 16:59:05 how do i install ansible 1.9.4, i thought it was pip install ansible==1.9.4 ,or yum install ansible-1.9.4 16:59:11 pbourke wally__ i think is running table 16:59:16 ok, thanks, i will try 16:59:21 jrich523 pip install ansible==1.9.4 16:59:27 crap... thanks 16:59:30 then rebuild horizon with --template-overrides template-overrides.j2 16:59:52 wally__: if you're using stable the templates aren't there, but you can put the same code right into the dockerfile (minus the jinja parts) 17:00:14 ok, thanks. I will try, 17:00:22 wally__: you also need to add a plugin block to kolla-build.conf to clone the repo 17:00:35 wally__: lot of steps I know :/ PM me if you need more help. but will have to be tomorrow! 17:00:55 okok i will take to you tomorrow 17:01:32 so to restart all the different serfvice, I need to restar the docker, right, like nova, nova-api, neutron,,..l3 dhcp...etc 17:01:39 add {% block horizon_footer %} at the begining of the file 17:01:48 egonzalez90 he is using stable 17:02:10 up sorry 17:02:59 inc0 when you get the cluster deplyed again please let me know 17:03:07 so our plan a is as follows 17:03:09 settying up networking now 17:03:42 1) deploy all scenarios on 14.04 2) deploy all scenarios on centos 7 3) hack the osic-prep container to support xenial 4) try to get through xenial testing 17:03:48 3 and 4 are stretch goals here :) 17:04:47 fuck 17:04:55 I messed it up again 17:05:21 i've nuked /var/lib several times during kolla dev 17:05:25 it happens :( 17:05:36 but I have most of stuff in place now 17:05:49 so I can nuke it up and make it back up in couple mins now 17:06:10 please dont nuke the deployment host 17:06:15 i haven't backed it up lately 17:06:18 ^^ this 17:06:22 lol 17:07:40 inc0 - need ceph+cinder 17:07:58 as our rallly test cases are selected for that combo 17:10:28 hi guys, is there a network architecture diagram for the VM in kolla. my VM still have no internet access, and canot ping and cannot ssh into the VM, I just check the l3 log, there are some errors. 17:10:49 I think somthing wrong with my network 17:17:05 sounds like it :) 17:21:43 well fuck 17:21:52 deploy ndoe ran out of space 17:24:13 hi guys, in my dnsmasq.log. I always got this (heka)[heka@kolla neutron]$ tail -f dnsmasq.log Aug 25 10:22:12 dnsmasq-dhcp[87]: DHCPDISCOVER(tapf44fe0a4-95) 2a:50:bd:00:71:94 no address available Aug 25 10:22:15 dnsmasq-dhcp[87]: DHCPDISCOVER(tapf44fe0a4-95) 2a:50:bd:00:71:94 no address available 17:24:25 anyone know what is wrong? 17:24:51 dhcp now working? 17:24:51 its looking for a pool of addresses, and you've got none 17:24:58 basically 17:25:12 u mean when I setup the network ? 17:25:16 for vm? 17:25:30 im not really sure, i imagine thats the case tho 17:26:19 so because of that, the vm cannot go to internet? is the the reason? 17:27:38 yeah 17:27:49 my instances isnt setup, so cant really provided any more info 17:27:53 you've gotten ahead of me :-/ 17:28:41 ok, please setup, man 17:28:51 im working on it now :) 17:28:58 after i submit my damn timecard 17:29:17 let me know if your instance can go outside or now, and ping and ssh to the vm 17:29:42 will do 17:32:05 inc0 17:32:10 inc0 there is a /backup.tar file 17:32:11 remove it 17:32:23 I removed all the images besides registry 17:32:35 inc0 there is a hue ass tarfile that i a bkackup 17:32:38 freed up 50% of space 17:32:43 okthen allgodo :) 17:34:07 feels like friday today 17:34:55 jrich523,. how is your network setup, do u have a network digram? 17:35:08 well its really flat 17:35:24 as in, its all in VMWare Workstation with a single subnet for internal/external 17:35:27 how is your netowrk setut , how many nic 17:35:30 so my setup wont be as useful 17:35:39 but i'll probably be able to figure out how the IP pool works 17:35:48 i use two nics, because you have to 17:35:52 how many nodes? 17:35:54 but each nic is the same subnet 17:36:10 there will be 2, a controller node (basically all of kolla) and a compute/storage node (to build VMs on) 17:36:10 how many interfaces? 17:36:34 ? 17:36:38 what sort of interfaces? 17:36:52 how many netowrk interfacce, like eth0 eth1...? 17:36:58 2 17:37:02 because its a must 17:37:10 mgmt, and external 17:37:13 can u send me your ifconfig output 17:37:24 nothing useful t here 17:37:28 because its flat 17:37:30 so for example 17:37:35 ok, pleaes 17:37:37 i use my host IPs as 2* 17:37:46 guys, let's start some flame war 17:37:51 so the controller is 192.168.11.20, node1, is 192.168.11.21 17:37:57 I just grabbed popcorn and need moral reason for it 17:38:27 so 192.168.11.20 is your eth0 and this is management interfface? 17:38:30 everything i have is 192.168.11.* 17:38:35 inc0 i have nthing to complain about atm :) 17:38:36 yes 17:38:51 so its 21 and 31 17:38:58 what about eth1? no ip assigned? 17:39:12 correct (based on our conversation yesterday) 17:39:35 i forget who told us, but.. yeah.. only assign the mgmt ip 17:39:35 ok , ,my is the same setup,,.., 17:40:04 so you ssh into your server using ssh 192.168.11.20 right 17:40:10 yes 17:40:14 ok, i got it 17:40:33 i used bridge mode for the adapters, so my pc is .99, super flat :) 17:40:51 ok 17:40:58 i mostly need to learn HEAT and other automation type stuff, so i could care less about how the network is setup 17:41:37 we have a client who wants to use openstack as their main entry point for system managment, but they also use puppet and ansible 17:41:47 and they want me to tie it all together 17:41:50 * jrich523 shurgs 17:41:50 i got the same setup, but my vm cannot go to internet, let me know if your are ok 17:41:56 ok 17:42:03 might take an hour or two before im up and running 17:42:08 got a couple of actual work things to do first 17:42:24 ok 17:46:35 irtermite, btw one node seems fubared 17:46:54 729583-comp-s3700-023.cloud7.osic.rackspace.com not that it hits us badly, but take a look at it 17:53:47 fuck the servers. 16 managed to stand up with ssh 17:53:50 rest did not 17:55:24 nvm, fuck ssh not servers, servers are cool 17:56:15 inc0 jefrey4l pointd out 23 has a nocarrier on one of its nics 17:56:35 hence why its not in the inventory file ;) 17:56:35 yeah I already moved it out of stuff 17:56:43 just want irtermite to know that it has issues 17:56:59 as I'm sure he'd be interested;) 18:00:52 ok, sdake wanna see Seans playbook in action? 18:02:44 inc0 yes 18:02:48 moment let me switch vpns 18:08:29 Afternoon 18:12:23 howdy kbaegis 18:28:26 inc0 that was a kernel update no? 18:28:33 #yeah 18:28:47 cool, now I start to chat with # in irc 18:28:52 tmux broken me 18:29:27 are we waiting on a reboot 18:31:13 correct 18:37:21 inc0 how long does this take 18:37:44 no clue, put time on it 18:37:49 so we'll have benchmark of it 18:37:57 never run it before 18:37:58 ? 18:38:06 it was merged 3 days ago... 18:38:09 or so 18:38:15 not on 100+ nodes 18:38:20 looks jammed up to me 18:38:57 let's wait, it's including stuff in loop 18:39:02 that's good learning too btw 18:39:19 btw rhallisey wirehead_ we have kolla-k8s talk 18:39:25 so one woot for kolla k8s;) 18:39:28 Just saw. 18:39:29 woot 18:39:32 woot 18:40:05 inc0 our other talk gte rejected? 18:40:35 general kolla - yeah 18:41:38 wow 18:42:03 My cynical prediction is that Barcelona will be a Kubernetes conference with an OpenStack sideshow. 18:43:01 Because that's what OpenStack Days SV kinda devolved into. :/ 18:45:00 inc0 two paths - repave the way you hae in the past - or use the playbook 18:45:24 nah, I make notes on what to improve on playbook 18:45:30 it's actually important learining 18:45:46 yes - but it can be done later not now ;) 18:49:19 yeah this part is suboptimal, we need to fix it 18:52:33 why did the playbook implode? 18:52:51 not sure 18:53:09 hi guys, I just check my neutron agent-list, Metadata agent | kolla | | xxx | True L3 agent | kolla | nova | xxx | True DHCP agent | kolla | nova | xxx | True 18:53:31 these service not start, how to start the docker container? 18:55:11 scroll u plets find source of that problem 18:55:26 since we a re now debugging the host playbook... 18:55:38 inc0 18:55:56 i think its an undeifned variable buty ou move around too fast for my screen size 18:56:02 cannot start the docker start neutron_metadata_agent..Error response from daemon: linux mounts: Path /run/netns is mounted on /run but it is not a shared mount. Error: failed to start containers: neutron_metadata_agent 18:56:06 any idea? 18:56:35 wally__ you hve to read the quickstart dude ;) 18:57:35 this playbook isn't idempotent 18:57:46 it is, I just messed it up 18:58:16 kolla_ssh_key 18:58:21 just disable the ssh key instlalation 18:58:37 kolla_ssh-key and whatnot go in passwords.yml 18:58:44 ok i gotit 19:01:13 vim is hard 19:02:32 inc0 did ou time it? 19:02:56 yeah, but it wont be that relevant now 19:03:00 I changed it 19:03:17 give us a rough idea 19:03:34 yeah after I'll upstream these changes 19:03:42 so it will be close to what we have now 19:04:48 wonder why python-dev and libssl-dev get installed 19:08:37 ok, we should be ready to deploy 19:08:59 so what went wrong with the playbook there? 19:09:08 ok in meeting atm, can you deploy 19:10:27 bare metal config should be under 15 mins i think 19:10:46 1. it has loop with it to fill up hostfiles 19:11:02 and it is O^2 currently 19:11:10 we need to make it just for controller nodes 19:11:30 other than that it's pretty much ok 19:11:55 that was't the only problem 19:11:57 oh and this first task is strange 19:12:09 well, I messed up dpkg by breaking it mid-work 19:12:16 so that was partially my bad 19:17:46 use lei's labeling script dude 19:18:27 that's not it 19:18:35 something is fucked up on networking layer 19:18:37 yay 19:22:26 Yay! 19:22:52 YaY! 19:24:02 inc0: maybe networking problems are similar to this? http://lists.openstack.org/pipermail/openstack-dev/2016-August/102211.html 19:24:12 dasm, no, it's low lever 19:24:15 bonding kind of stuff 19:24:20 mhm 19:24:23 britthouser, you there on tmux? 19:24:37 oh..no I closed it out 19:24:43 lemme login 19:24:50 wally__: did you setup a docker registry? 19:25:12 no 19:25:15 ok I'm there inc0 19:25:28 you have two nodes tho right? 19:25:37 sooo...how did we break this britthouser ? 19:25:53 i am useing all ini one, now, just to make sure i have everything right 19:26:08 bond-slaves none looks suspicious 19:26:26 britthouser inc0 and I are struggling after a repave 19:26:37 ahh ok, the all-in-one is a bit different, not easy to add compute nodes.. cuz... its obviously no longer all in one :) 19:26:39 this is the deploy node? 19:26:41 it has something to do with bond0 showing no carrier 19:26:59 not sure - wasn't driving 19:27:15 yeah i know, i just want to try first, then I will go to multinode 19:27:27 modprobe bonding ? 19:27:58 well damn, I did that! 19:28:07 =) 19:28:29 I still can't cat /proc/net/bonding 19:28:32 which is odd 19:29:13 it works 19:29:25 ok. =) 19:29:26 check ip a again 19:30:31 redeploying 19:30:31 dont you need to relabel the disks? 19:33:56 britthouser its still busted 19:34:40 take it for few minutes, gotta take some air;) 19:34:45 get some air* 19:34:51 and swear a lot 19:35:19 i dont know anything about ubuntu 19:35:33 i am pretyt sure the slaves are not being assigned tho from that config file 19:36:31 hi guys, I got some error in the neutron-l3-agent.log , can anyone please have a look for me,http://pastebin.com/nbxPp24t 19:37:28 i got this error when i create a subnet for demo, and create demo-router, and assign the demo subnet interface to the demo router 19:38:16 sdake are these problems on teh deploy node, or all the other ndoes? 19:38:26 other nodes i think 19:38:27 i am not realy sure 19:38:50 Tehre isa $ on the end of that p1p1 interface 19:38:54 that doesn't look right 19:39:04 in the right hand tab 19:39:11 got it 19:39:16 that was me typoing 19:39:26 on the right is the deply node 19:39:30 on the left is the node not working 19:39:59 looks right. lemme drive a secon 19:42:03 can I reboot it/ 19:42:10 free free 19:42:16 feel free 19:44:28 pokey reboot times 19:45:30 is it kickstarting again? 19:45:37 no idea 19:45:41 keep trying to ssh in 19:46:20 usually when you can ping but no ssh, its b/c its kickstarting 19:46:33 is there nay wy to tell for sure? 19:46:37 console 19:46:53 what was the name of that node? 19:46:58 .23 19:47:24 729517-comp-s3500-023 ? 19:47:49 http://729537-comp-s3500-047.cloud7.osic.rackspace.com/ 19:48:27 without the http part i think 19:48:56 atleast thtat is what the email says.... 19:49:07 ok we are in 19:50:15 ok so no bonding was loaded on reboot i modprobe bonding bonding0 comes up 19:50:20 but in a down state 19:51:02 weird....the bonding driver is loaded 19:51:20 but its not not configuring the bond according tot eh config file 19:51:56 that would be my impression 19:59:01 * britthouser scratches my head 20:02:44 gotta meeting...bbiab 20:07:52 https://help.ubuntu.com/community/UbuntuBonding 20:09:27 ahh...good call sdake 20:09:31 need that ifenslave 20:09:37 right 20:09:48 now its doing something else annoying 20:15:23 nice 20:18:49 hi anyone know what the issue is http://pastebin.com/nbxPp24t 21:18:01 Michal Jastrzebski (inc0) proposed openstack/kolla: Architectural guide doc https://review.openstack.org/360796 21:23:17 sdake, can you take rally from there 21:23:22 I'm out of juice today 21:23:31 btw take a look at this awesome doc I submitted 21:23:36 https://review.openstack.org/360796 21:23:41 i am not sure where its at at preent 21:23:53 did the deploy work? 21:23:58 yeah 21:24:12 so should be able to run script and off itgoes/ 21:24:39 I'm running init runonce now 21:28:36 ok i'm out for a bit as well - traveling on the road 21:28:48 looks like its ticking along 21:28:59 awesome doc inc0 21:29:05 exactly what is needed 21:29:15 :) 21:29:32 clears alot of the networking questioned I had up 21:30:12 yeah also storage driver for docker seems to make people confused to say the least 21:30:42 the storage part - we actually ran out of storage on osic cluster 21:31:25 well, why'd you go and do that, inc0 21:31:27 ? 21:31:29 ;) 21:31:43 do what? 21:32:21 inc0 run out of space 21:32:33 irtermite because someone thought 40g was plenty for kolla.. 21:33:07 40g? you have a LOT more than that there 21:33:09 which is fine for the entire os and kolla on any node but a control node ;-) 21:33:11 as it turns out 130 nodes of openstack does produce rather significant amount of logs 21:33:27 irtermite, yeah, but we didnt mount it in correct place;) 21:33:30 irtermite / was 40g 21:33:34 hahahah 21:33:38 irtermite /var/lob/docker was in / 21:33:56 logrotate 21:34:09 yup we need to look into that for elasticsearch 21:34:20 we already have fs log rotation 21:34:23 boy, that sure sucks. do you have to start over, or are you going to dump and roll forward? You guys only have until Tuesday left. 21:34:33 irtermite, issue is elasticsearch and kibana 21:34:36 right - so we reloaded 21:34:52 scenario 2 is running of 12 ;) 21:34:57 but everything is automated 21:34:59 irtermite, deployment of fresh openstack is 20min, today we repaved it from scratch 21:35:12 wow 21:35:13 not that horrible 21:35:24 so assuming someone is around to run the script and do he setup of the scenario, it takes about 20 mins to do that 21:35:24 nice 21:35:27 well done guys 21:35:27 once you get hang of it 21:36:18 still i am concerned about the amount of time available 21:36:32 because scenario run time is 6-8 hrs 21:36:48 if we'll be nice and community open - we're open, not sure about nice part 21:37:06 we might ask OSIC for another installment somewhere in future:) 21:37:20 irtermite, does osic have rules against asking for it twice? 21:37:29 if we provide good reason to ofc 21:37:36 now it would take much less time that we have figured out how to operate in osic ;-) 21:37:49 now that we 21:37:54 inc0: nope... but it won't be consecutive, nor is it guaranteed to be approved. 21:38:04 its all good 21:38:05 yeah bulk of today was dealing with 14.04 21:38:11 this is a great great gift irtermite - appreciate it alot 21:38:45 sdake: you can thank Intel and Rackspace. I merely manage the deal. :) 21:38:46 this will be the first public scale testing doucment i've heard of of openstack being published 21:39:03 for kolla that is, sdake 21:39:11 ya not sure of any others 21:39:23 not saying there isn't 21:39:26 just don't knowof any 21:39:37 there are plenty of other white papers that are coming out of OSIC access 21:39:48 cool 21:40:00 i dont know everything ;) 21:40:28 everyone who uses these environments is expected to upstream commit and to contribute a writeup 21:40:44 that is part of the agreement. nothing is free ;) 21:40:56 we have a review in work that will be going on docs.oo 21:41:04 so i think that is satisified 21:41:13 I say "expected," but... it really is actually "strongly suggested" as part of the acceptance criteria. 21:41:18 once merged of course 21:41:18 once filled with data ;) 21:41:48 don' worry... inc0 will be bugged quite often by his co-workers and me for the docs 21:42:11 Michal Jastrzebski (inc0) proposed openstack/kolla: Architectural guide doc https://review.openstack.org/360796 21:42:59 irtermite, couple of learinings have jsut been published in patch above 21:43:02 feel free to review;) 21:43:27 You know Marketing won't read that inc0 21:44:09 irtermite, for marketing we have this https://www.youtube.com/watch?v=RNZMtym5x1c 21:44:35 caution, slavic accent included 21:44:52 They are going to want a PDF as well, I'm sure of it. 21:45:01 t-28 minutes is where it climaxes 21:45:06 .rst is best they can get 21:45:21 sdake, when deploy is finished 21:45:23 pretty sure rst converts to pdf 21:46:04 Yea, this .rst looks like a technical review of work. I am sure they are going to want business case outputs as well along with this when it is done. 21:46:23 sdake, btw mind reviewing my doc patch? 21:47:16 irtermite, hey, openstack from naught to running in 20min, that's a business case output right there! 21:47:18 irtermite - not sure the broader community can help with business case outputs 21:47:28 oh, it does sdake... but, it's more of the content I am talking about. There is one that is about to be posted from one of the last users. When it is, give it a look to give you an idea of what I am talking about. 21:48:05 It is... so make a pretty graph of it, comparing to other methods, inc0 that is what they will be looking for 21:48:15 Control, another example, and your example. 21:48:27 we'll figure it out 21:48:41 I'll keep leaving next revisions on your desk 21:48:55 irtermite would you mind defining for me "business case output" 21:49:46 sdake: What was your testing to prove? What did it confirm? Give comparisons to other options. Etc etc. 21:50:19 Saying it deployed in 20 min needs to be compared to another deployment for baseline and to actually prove something. 21:50:41 not interested in comparing kolla to other tools 21:51:21 our testing is to prove that kolla is fast and works properly at scale as I promised originally 3 years ago when I signed up for the PTL gig 21:51:22 then what is the use-case here? 21:51:49 comparing to anytning else will cause a flame war we don't want 21:51:55 we will just shovel up data 21:51:59 But "fast" relative to what? I'm preparing you for the feedback that will come. 21:52:00 Tin Lam proposed openstack/kolla: Add ansible role for vmtp container https://review.openstack.org/358305 21:52:10 let people compare themself 21:52:28 fast compared to manual install 21:52:42 let people judge that baseline on their own 21:53:04 never mention your competition in your marketing ;-) 21:54:18 i hear mirantis getting shot at osic gear with fuel 21:54:38 they can provide a comparison if they like -or users of the tools can 22:02:15 sdake OSIC Project is about community, not competition. Baselines are indeed useful and can be positioned to not be viewed as slander. If the compared group feels they are being stomped on, that is another matter. 22:03:29 perhaps a future osic study by a group of deployment tools could be a baseline on deployment times from nothing to running 22:03:34 Hi all, my deploy with ceph is failing at fetching ceph keyrings task and when i run docker exec ceph_mon fetch_ceph_keys.py on terminal i see it return "failed":"false" in the json. Should it have "changed" in the json 22:03:55 that gets returned? 22:05:18 our philosophy specificlaly states our objective: 22:05:23 irtermite : Kolla has an objective to replace the inflexible, painful, resource-intensive deployment process of OpenStack with a flexible, painless, inexpensive deployment process. Often to deploy OpenStack at the 100+ node scale small businesses may require means building a team of OpenStack professionals to maintain and manage the OpenStack deployment. Finding people experienced in OpenStack deployment is very difficult and expensive, resulting in 22:05:24 a big barrier for OpenStack adoption. Kolla seeks to remedy this set of problems by simplifying the deployment process but enabling flexible deployment models. 22:05:58 basically, what I am saying is... whether you are comparing to another tool or not, the proposal for the kolla testing on OSIC was to test time of deployment and scaling beyond what was previously available. Even if you had metrics that showed how quickly kolla deployed before any work that was done here, or how it performed before throwing a larger node count at it; that would be good for your baseline and t 22:05:58 est case results. 22:06:34 the error I am getting is this http://pastebin.com/rXhP8gAf. When I checked distribute_keyrings.yml I see it is failing on docker exec 22:06:48 ya i can compare to my 3 node cluster I use typically :) 22:13:18 inc0 27 rally tasks already run 22:13:18 out of about 400 22:17:25 any idea about the error http://pastebin.com/rXhP8gAf 22:22:57 inc0: lol yeah, that'll be a lot of logs. :) how many gigs you generate? 22:23:45 kbaegis, after a day around 30+gig 22:24:10 Sounds about right. I had 3/day with a 5 node deployment, so makes sense. 22:24:28 I was also hammering it with requests though 22:25:24 so are we 22:25:32 stress testing 130node cloud 22:25:40 30 gb of logs 22:25:43 Wow 22:26:03 Did you grep for any shakespearean clauses? 22:26:48 shakespearean? Like "you will all die in horrible way" ? 22:26:51 You might find some quotes from the tempest in there 22:27:18 Like in enough monkeys, infinite time kind of way 22:28:10 ahh, this kind, no 22:28:15 I don't believe this theory 22:28:22 existance of internet proven it wrong 22:28:38 lol Yeah, but we've only had 40 years or so 22:29:40 Think of all the great things said on twitter and youtube comments. ;) 22:29:41 Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 22:29:54 but trends looks like it goes more towards sexist and racist flame wars than poetic 22:31:51 iambi pent-up-meter? 22:32:12 Okay, that was a stretch :) 22:33:37 I do not have slighest idea what you're talking about 22:33:57 Quipping iambic pentameter. Ignore me. :) 22:35:52 Is the testing going well? Any unforeseen reason not to use kolla at-scale? 22:36:02 Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 22:39:00 thats 10 terabytes a year inc0 - we need to work on gc ;) 22:41:18 sdake logrotate? 22:41:30 kbaegis nah its elasticsearch that is storing the logs 22:41:33 rather something for elastic 22:43:52 aNupoisc thats easy enough to fix 22:44:06 aNupoisc make your authorized_keys file match your public key in ssh 22:44:50 Hey, is there a convenient way to deploy on a different ssh port? I know ansible supports it 22:45:00 kbaegis dont know 22:45:03 <— Making things difficult for himself :) 22:45:48 I found that changing from 22, 99% of your spam brute-forcing at the edge disappears 22:47:22 its the 1% i'd be worried about in that case - as they had to run a port scan to determine your ssh port to brute force 22:48:03 sdake Yeah, that's true. Portknockers help there 22:48:07 Or ids 22:49:39 I just disable password logins. Unless there's a 0-day against openssl, I'm probably fine. If they want to use one of those, well I have tcpdump. :) 22:50:44 sdake: the private key that i am using matches with the public key in authorized_keys 22:51:20 authorized_keys = public key contents? 22:51:50 and mode on .ssh is 700 22:51:59 and mdoe on all files in .ssh dir is 600? 22:54:02 aNupoisc you will probably need to run that as sudo for the moment 22:54:28 aNupoisc and make sure your authorized_keys in ~root/.ssh/authorized_keys = public key of private key your using 23:01:49 sdake: yes the modes are correct but let me check the modes and all under root 23:30:13 sdake: so the authorized_keys is matching public key of private key i am using but what i noticed is that my authorized_keys is owned by different user than root 23:30:27 sdake: will that cause a problem 23:32:08 hi guys, I am just wondering it use ovs-bridge in the neutron config , right? 23:33:10 aNupoisc yes that will not work if sshing in via root 23:33:21 wally__ yes ovs or liuxbridge is selectable 23:33:25 bridge_mappings = physnet1:br-ex 23:33:45 how to select linuxbridge when do the deploy 23:33:54 by default it use ovs, right 23:34:20 _wally: if you are using devstack then in local.conf you have to specift Q_type=linuxbridge 23:34:49 i am deploying all in one 23:35:26 in the global.yaml, can I set to linux bridge? 23:37:17 if I use ovs to deploy then, my server need to create ovs bridge "br-ex" before doing the deploy, right ? 23:38:58 wally__: http://docs.openstack.org/developer/devstack/guides/neutron.html#using-linux-bridge-instead-of-open-vswitch 23:39:03 here is the pointer 23:40:13 wally__: i am not sure what option should be used when using globals.yml 23:40:54 thx i got it, but I dont use devstack, then how to deploy it witll linux bridged, with kollo-ansible deploy 23:42:53 sdake: it seems some different problem because even I changed the ownership still it is giving the same error. Is this correct output for docker exec ceph_mon fetch_ceph_keys.py correct? http://pastebin.com/WrRxu7vM 23:43:39 ok.. using the quickstart, multinode... prechecks were happy, went to deploy and i got this 23:43:47 The requested image does not exist: 192.168.11.20:4000/kolla/centos-binary-heka:2.0.2 23:44:07 well, I can use ovs , so I need to create ovs br-ex, is that correct? 23:44:19 shouldnt it auto populate the registry? 23:46:37 jrich523: I believe you have to run docker that populates registry 23:47:02 jrich523: I am not 100% sure I am also playing with kolla 23:48:08 hi jrich523, how is your deployment 23:51:43 jrich523 is your registry (192.168.11.20) accessible? 23:51:50 great question 23:51:54 i wasnt really able to find a way to test that 23:52:28 run docker ps -ef | grep docker on the deploy target 23:52:37 and paste output 23:53:30 hmm so its telling me it doesnt know what e is 23:53:36 jrich523 did you build images? 23:53:39 docker ps works fine 23:53:48 sorry ps -ef | grep docker 23:54:18 k 23:54:32 also, as far as the build, i did the pip install of kolla, based on the quick start, didnt seem like i had to build it? 23:54:50 you hae to build images if you want to use your own registry 23:54:51 ps output: http://paste.openstack.org/show/563697/ 23:55:00 we have nothing at present that takes the docker hub and replicates it to your local docker registry 23:55:23 dockerd needs the --insecure-registry flag as well 23:55:35 ok so that was something i asked the other day 23:55:57 thats my node01, but ctrl01 has the docker settings for the registry (hosting it) 23:56:21 yes the deploy target should have --insecure-registry flag 23:56:33 so basically every node needs that done manually 23:56:38 and you need to build images (which will psuh them to hte registry) 23:56:39 weird that it says it looked there tho? 23:56:53 wait nvm 23:56:58 ok 23:57:00 jrich523 your globals.yml has that address in teh egistry location 23:57:04 so for the build, just kolla build? 23:57:05 it does 23:57:06 if you just want to try multinode without a registry 23:57:20 just comment out the registry option 23:57:20 and it will pull from docker hub 23:57:30 no user info tho, that was a tad murky to me, didnt see any specific info on setting security so i assume its just open based on the directions? 23:57:48 well i did all this work to get the registry setup, i might as well use it now :) 23:57:50 to secure docker registry - you have to use rls 23:57:58 ok - to use the registry 23:58:06 well i dont want to, just making sure i dont need to add any creds in globals 23:58:20 kolla-build --type source --base ubuntu 23:58:20 or if on centos 23:58:26 kolla-build --type source --base ubuntu 23:58:29 you need more options too 23:58:43 --registry (the ip of your registry):4000 23:58:46 hi sdake, so I need to create ovs bridge br-ex before doing the deployment? 23:58:50 and --push 23:58:56 wally__ dont think so - pretty sure kolla does that for you 23:59:21 kolla-build --registry registry_ip_address:registry_ip_port --push 23:59:33 based on the dev build in quickstart ^ 23:59:35 jrich don't forget the build types 23:59:35 but after the dopoymnet, I check the ifconfig on my sever, i do not see the bridge 23:59:38 so put it all togther 23:59:39 yeah 23:59:41 that will build centos binary 23:59:56 wally__ not sure - ask sbezverk_ 00:00:15 hi sbeverk, there 00:00:27 how can I just ping someone here 00:00:36 i am off to get dinner 00:00:43 sbezverk_ is probaby asleep by noow - on eas tcoast 00:00:44 bbiaf 00:00:44 enjoy sdake 00:00:52 ok, 00:01:03 hi jrich523, there 00:01:10 that pings, but its based on my client 00:01:26 did u create a ovs bridge when u do the deploy? 00:01:31 nah 00:01:35 im still struggling with other parts 00:01:40 mostly the docker/ansible multinode part 00:01:42 but, im close 00:01:46 real close 00:01:55 did u do the kolla-ansible deploy? 00:01:58 if this populates my registry correctly, then i'll be able to deploy 00:02:04 not yet 00:02:07 ok i c 00:02:17 just finished building the docker registry 00:02:27 now im building the images and pushing them in to the registry 00:02:30 then i'll do the deploy 00:03:59 jrich523: you need to run docker, build images and then run deploy 00:04:43 yeah docker is setup no the controller, its compiling now and populating registry 00:04:48 anyone know do i need to create ovs bridge be-ex when do the deploy? 00:04:52 im configuring the registry on the next node now 00:06:00 ok, node01 is configured for registry 00:06:09 just waiting on the image creation 00:06:43 wally__ what are you trying to deploy? Openstack right? 00:06:52 wally__ or is it ceph 00:06:54 or something 00:06:56 hey he has a all in one kolla setup 00:06:56 yeah, openstack 00:07:02 use kolla, AIO 00:07:33 wally__ yeah then you don't need create ovs bridge 00:08:13 I know it use ovs, but i check the openvswithc.config file, it will create a bridge "br-ex" 00:08:14 right 00:08:45 bridge_mappings = phynset1-ex 00:08:50 bridge_mappings = phynset1: br-ex 00:08:51 wally__ this was my globals http://pastebin.com/BM1W6LC5 00:09:18 but i do not see the br-ex is create in my server when i do ifconfig 00:09:31 wally__ also there is a option # Valid options neutron_plugin_agent: "openvswitch" 00:09:42 where you can specify linuxbridge if you want 00:09:59 wally__ ahh 00:10:15 ok i c 00:10:15 wally__ i never faced that. Interesting 00:10:25 so u have br-ex create? 00:10:37 can u output your ifconfig for me please 00:10:51 wally__ i am actually right now trying to deploy ceph so don't have setup for openstack 00:11:12 wally__ but yeah when i tried couple of weeks back that time the globals. yml above worked 00:11:40 on your globla,ayml file 00:11:44 neutron_external_interface: "p787p1" 00:11:50 the interface is a bridge? 00:11:59 that's my bridge 00:12:04 or a phycial net working interface? 00:12:21 wally__ kolla should build the br-ex interface based off the interface you use as neutron_external_interface setting in globals 00:12:52 and neutron_external_interface should be named as the what the host sees the physical interface 00:13:02 wally__ its a physical one 00:13:06 wally__ that i had 00:13:07 oh u mean, kolla will create a br-ex and add the br to the port:neutron_external_interface? 00:13:15 eth, en, etc.... 00:13:59 no what ever interface you set in your globals.yml file kolla will use to to make the br-ex interface off of 00:14:21 like #ovs-vsctl add-port br-ex neutron_external_interface? 00:14:54 no, what interface did you set in globals.yml as your neutron_external_interface? 00:14:59 eth1 00:15:15 then what is the bridge, 00:15:36 then on the network node you should be able to do a ifconfig or ip addr and see br-ex which is bridged to eth1 00:16:27 i use allinone, do i need to long to the neutron docker container? 00:16:34 physical to logical it should go eth1 -> br-ex -> ovs -> br-int 00:17:20 so for allinone all interfaces inside globals.yml should be the same interface 00:18:00 so eth1 is my pyhsical network interface, and i should br-ex showing when i check ifconfig, right 00:18:12 yes 00:18:30 if there isn't a br-ex or ovs-bridge then something went wrong 00:19:02 i have 4 nics, eth0, eth1. eth2, eth3, i use eth0 for management. so I can use eth1 or eth2,, eth3 for the bridge? 00:19:05 correct? 00:19:39 not sure for allinone deployment 00:19:51 I think allinone deployment uses the same interface for all 00:20:08 ie.... as the title says allinone 00:20:17 u mean, use eth0 as well 00:20:46 yes 00:21:23 so but if this is the case, then during the deploying, it will disconnect when it create bridge, 00:22:44 for a mill sec yeah, but when its creating this section all the cmds its needs is already there and should reconnect afrer 00:22:46 after 00:24:38 I could be completely wrong here but thats how I see it 00:24:53 I haven't rried allinone deployment yet 00:24:58 tried* 00:25:33 ok, starting the deploy! 00:25:40 ugh 00:26:17 rewind! 00:40:47 Li Yingjun proposed openstack/kolla: Add option ceilometer_database_type for ceilometer https://review.openstack.org/345881 00:45:25 wally__ maybe this article will help you 00:45:27 http://mntdevops.com/2016/08/08/iaas-2/ 00:46:53 Li Yingjun proposed openstack/kolla: Fix toolbox permission issue when enabling ceph https://review.openstack.org/358973 02:11:51 hello 02:14:51 zhubingbing: Hello 02:19:49 sdake， morning 02:34:13 Kevin Fox proposed openstack/kolla-kubernetes: Doc for multi node development https://review.openstack.org/360853 02:39:11 Jeffrey Zhang proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 02:42:34 Jeffrey4l__ did you do any work on the oosic cluster? 02:42:45 sdake, no. 02:42:54 Jeffrey4l__ - ok we are on scenario #2 atm 02:42:58 it is almost finished 02:43:06 cool. 02:43:09 may need hep setting up next scenario 02:43:14 OK. 02:43:17 i can help today. 02:43:39 i dont know what is required to make ceph run with caching 02:43:44 which is scneario 3 02:43:57 i have been thinking we may just cut this scenario out 02:44:01 and go straight to linuxbridge 02:44:08 I can try ( but never tried) 02:44:55 we all all ssd, the ceph cache should not improve the performance very much. 02:45:20 The disks we use are all ssd.. 02:45:25 yup 02:45:48 we are also running different scenarios now 02:45:56 128x96, 128x48, 128x24 02:46:21 and tempest 02:46:25 there is one script that runs these 02:46:41 i wrote a script to get the data out but it doesnt seem to work with deployments 02:46:47 so i will ahve to sort that out aftre this rally finishes 02:47:20 we are running tempest now? 02:47:25 rally 02:47:33 tempest is part of the script to run all 4 cases 02:47:42 look at the script 02:47:48 so you understand whatit does 02:48:10 i think we need a rally deployment use operation in the reporting 02:48:22 it takes about 3 hours to run rally now 02:48:23 maybe 4 02:48:30 all 4 scenarios 02:48:39 all 4 cases we test 02:49:48 lost the context. 02:49:51 :( 02:50:02 need me to do something on my side? 02:50:22 not clear what should do next. 02:50:36 look at scripts in ~/rally.git 02:50:47 attach to tmux 03:05:11 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 03:11:34 Jeffrey4l__ just lot access to the clussteer for some reaosn 03:12:19 sdake, do not get the point.. 03:13:32 sdake, do not get the point. :( 03:13:47 OK lot-> lost. 03:13:58 point of which 03:14:04 oh i'm back in 03:14:06 nm 03:23:07 Tin Lam proposed openstack/kolla: Add ansible role for vmtp container https://review.openstack.org/358305 03:32:09 Anyone played around with novnc baseurl? 03:32:49 I'm actually hiding horizon behind an apache reverse proxy. UI works perfectly, but vnc is still broken 03:34:29 I'm noticing that clients are still redirected to the old url 03:34:49 kbaegis, change it to the url u want. 03:35:05 I did, restarted the container for novnc 03:35:18 Do I need to do that in nova-api instead? 03:35:29 Cause clients are still getting redirected to the old url 03:35:45 kbaegis, how did u changed it? 03:36:00 kbaegis, you should change the nova_compute 03:36:40 Jeffrey4l__: Thanks. I have copy_always, and was just editing /etc/kolla/nova-novncproxy/nova.conf 03:36:45 restart the container 03:37:15 Jeffrey4l__, sdake , coolsvap_ , can you please review https://review.openstack.org/#/c/350629/ 03:37:56 kbaegis, check this http://docs.openstack.org/developer/kolla/advanced-configuration.html#openstack-service-configuration-in-kolla 03:38:55 mdnadeem, ok 03:42:24 So that's the correct way to do it, but mine works too, no :) 03:42:49 It's ultimately restarting the container and copying it from a shared directory 03:43:12 nova compute it is :) 03:45:41 Okay, did that in nova-compute 03:45:49 Horizon is still redirecting to the wrong host 03:54:51 Jeffrey4l__: So on that link- is it based on fqdn or hostname? 03:55:02 'myhost' 03:59:23 use fqdn, or use the custom configuration file. 03:59:45 that's what I was asking. Tied to the path :) 04:00:11 change nova_base url in the /etc/kolla/config/nova.conf and run kolla-ansible reconfigure 04:10:03 Jeffrey4l__: Thanks. Just did that 04:14:36 how about cpu_allocation_ratio? Anyone play with that? 04:17:14 Didn't work. Old url is still being used 04:19:54 File was '/etc/kolla/config/crucible.lab/nova/nova.conf', aio 04:24:38 Well, copied nova.conf to the root config directory and symlinked the fqdn to the hostname 04:24:40 retrying 04:27:45 kbaegis: what are you doing? 04:28:26 duonghq: trying to alter the baseurl for novnc proxy. I'm hiding it behind a reverse httpd proxy 04:28:58 kbaegis: roger 04:29:11 how many node you have for now 04:29:41 duonghq: aio still 04:30:01 okay 04:30:27 Jeffrey4l__ we need the json output 04:30:34 sdake, yep. 04:30:35 Jeffrey4l__ so we can parse it and produce rsts with it 04:30:43 just test the html group report. 04:30:45 also would like html output 04:31:04 result seem have no such parameter. 04:33:20 Yay! That worked 04:33:34 So either the path is not accurately linked to fqdn (maybe just hostname) 04:33:50 Or it's only reading out of /etc/kolla/config/ 04:33:52 Jeffrey4l__ i'm all done, i wantedd to see if switching deployments got us at the other data 04:33:58 sdake, ok 04:41:09 ervice-level agreement 04:41:09 Criterion Detail Success 04:41:11 something_went_wrong Unexpected error: Invalid scenario argument: 'Image with pattern '^cirros.*uec$' not found' 04:52:45 Jeffrey4l__ aftere this finishes i want an hour with oprofile 04:52:49 then on to next scenario 04:52:55 sdake, the script seems good now. will check later. need be out. brb 04:52:57 sound good? 04:53:21 sdake, good. no idea about the oprofile :( 04:53:26 never used it. 04:53:28 i got it 04:53:28 brb 04:54:06 need check the collected report. should OK. but double check is batter :) 05:27:42 Jeffrey4l__ to get any good data out of oprofile we will need dbg info files in them 05:27:52 Jeffrey4l__ so for hte moment going to just run a quck test 05:28:05 and then if we have time i'll build images with dbginfo in them 05:28:10 i dont want to break the images we have ;-) 06:28:16 Md Nadeem proposed openstack/kolla: Added Ansible playbook for congress deploy https://review.openstack.org/350629 06:33:44 Jeffrey4l__ - something is busted on the test cluster - complaining about domian not found in all client calls 06:34:04 Jeffrey4l__ inc0 said he fixed this by adding a user domain 06:34:32 but we already have a default domain 06:34:46 and i read the relevant bugs, our admin-openrc.shlooks ok 06:45:34 post deploy is broken 06:45:42 instead of ID shoudd be NAME for the domains 06:47:02 sdake, back 06:47:14 Jeffrey4l__ i am running an oprofile test atm 06:47:16 sdake, still failed? 06:47:18 as soon sa rally wraps up 06:47:25 time to move on to scenario 3 06:47:34 can you get that kicked off, getting pretty late here 06:47:41 i think this rally job will take about 45 mins or so 06:47:50 cool. is the domain not found solved? 06:47:59 sdake, i think i can fix it. 06:48:10 i solved it 06:48:26 %s/ID/NAME with domain in the lines 06:48:39 yep 06:48:41 i'll get it fixed 06:49:04 or change the `default` domain name to `Default` 06:51:46 Jeffrey4l__ jump in tmux - I want ot show you oprofile 06:52:02 this is something you need to know how to use 06:53:18 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 06:59:03 what is #3 in cpu cycles 06:59:08 Jeffrey4l__ ^ 06:59:39 thats right, #1 is java 06:59:56 30% of control node consumed by java 07:00:21 25% python 07:00:39 13% by hekad 07:00:42 OK. clearly about his.\ 07:00:43 this 07:00:47 there was one above hekad 07:01:06 rabbitmq 07:01:06 beam.smp 07:01:11 ya rabbitmq 07:01:14 pretty useful huh? 07:01:21 yes. 07:01:28 tells me we need to eject java from our system if it were possibel :( 07:01:57 when you get done with that simple view i'll show you a more comlex one 07:01:58 let me know 07:02:00 in channel 07:02:23 sdake, ok. go on 07:04:03 pretty flat across symbols 07:04:09 that is what you like to see when optimizing 07:04:18 yep 07:04:27 it typically means those symbols are highly optimized already 07:05:05 6687 0.2704 vmlinux-4.4.0-34-generic sudo /vmlinux-4.4.0-34-generic 07:05:08 busy users of sudo :) 07:05:27 hmm. Cool 07:07:01 sdake, but few we can do in Kolla, right 07:08:11 not a whole lot unless it involves ejecting elasticsearch 07:08:57 ok. 07:09:06 hekad cpu hog 07:09:20 uses the kernel way too much 07:09:38 yes. 07:09:56 7% of the total sample time of the systme is spent processing hekad systemcalls 07:10:06 can we get more info about what kernel call he used? 07:10:15 try --callgraph 07:11:03 on centos the system call names are printed out 07:11:12 but on 14.04 looks like that is no bueno 07:11:46 nope it is treating the kernel as one blob 07:11:55 ok 07:11:56 annoying 07:12:28 but very useful tools :) 07:12:56 lets go back to the run of rally 07:16:29 git noob needs help :D https://review.openstack.org/#/c/346450/ How can I solve this "Patch in Merge Conflict" thing? 07:17:37 mewald, make a local rebase origin/master and solve the merge conflict, then `git review` 07:24:48 morning 07:48:33 snake ping 07:49:11 snake ping 07:49:21 snake ping 07:49:40 sdake ping 07:49:56 snake here 07:49:57 sup 07:50:01 hi, 07:50:28 mewald - git pull --rebase 07:50:28 https://review.openstack.org/#/c/348388/ 07:50:46 snake, where is snake 07:51:32 Sdake, you look at this branch can not hurry review merger, because I role gnocchi rely on this branch 07:52:21 - -duonqhq sorry, I'm spelling mistakes. 07:52:34 just kiding, don't mind 07:52:52 --） 07:54:15 zhubingbing: As I understand: you have other ps depend on this ps? 07:54:25 or you just want to make this feature running? 07:57:34 role gnocchi branch before the dockerfile gnocchi branch was carried out with the rebase operation 07:58:25 Now I submit the role gnocchi branch, which will cause the 2 branch to submit, 07:59:19 ping Hui Kong 07:59:36 ping Hui Kang 08:00:11 zhubingbing: seem that he don't online right now 08:00:41 and I think you should ping using his irc nick than his name 08:01:16 ok， thanks 08:01:53 I need to talk to him about the issue of etcd. 08:26:00 Christian Berendt proposed openstack/kolla: Add cloudkitty containers for type source https://review.openstack.org/349295 08:26:34 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 08:28:32 Li Yingjun proposed openstack/kolla: Add option ceilometer_database_type for ceilometer https://review.openstack.org/345881 08:36:01 Li Yingjun proposed openstack/kolla: Add option to support dhcp/l3 agent ha https://review.openstack.org/352180 08:41:14 what happen if a host is in compute group but network group? 08:46:56 Andreas Jaeger proposed openstack/kolla-kubernetes: Properly use constraints https://review.openstack.org/360984 08:47:32 pbourke, ping 08:47:41 Jeffrey4l__: hey 08:48:16 do u have any idea the default gw 172.21.112.1 can not reach on isic pbourke 08:48:40 osic 08:49:16 what do you mean it can not reach osic? 08:49:40 pbourke, i can not ping the 172.21.112.1 ip 08:49:46 it should be the default gw. 08:49:54 for p4p1 interface 08:50:28 with vlan 733 08:52:05 Andreas Jaeger proposed openstack/kolla-kubernetes: Cleanup tox.ini constraints https://review.openstack.org/360984 08:52:19 Jeffrey4l__: on now 08:52:25 im not sure but can have a look 08:52:34 Jeffrey4l__: whats the latest in the testing 08:52:51 OK. thanks. 08:54:10 pbourke, i need setup a external network now for some rally tests. They need connect the vms through ssh from the deploy node. 08:57:03 Jeffrey4l__: there is a script to do that 08:58:35 pbourke, this init-runonce-osic ? if so, it set up a wrong external network. 08:59:51 pbourke, found some thing wrong. fixing.. 09:06:11 ok 10:28:37 Good morning everyone 10:29:52 morning kbaegis 10:30:13 egonzalez90: We're up early :) 10:30:34 not really, EMEA time ;) 10:31:49 Oh, gotcha 10:31:56 'round noon there? 10:33:52 midday 10:37:40 zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 11:08:59 zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 12:23:33 is it possible to overide network names for particular hosts? i.e. in globals i have network_interface: "bond0" 12:23:47 but a slew of my machines will not have a bonb0 12:23:50 bond0 12:24:17 they will just have eth0 and eth0. 12:24:19 etc 12:46:42 hello 12:46:52 I believe you can do it in ansible Inventory file 12:46:56 look at this : 12:46:57 https://github.com/openstack/kolla/blob/master/ansible/inventory/multinode 12:47:15 # When compute nodes and control nodes use different interfaces, 12:47:15 # you can specify "api_interface" and another interfaces like below: 12:47:16 #compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1 12:47:55 you can also, change the names of your interfaces on the machines, and standarize then across the infrastructure ... but thats more hustle ;] 12:54:06 Mr_Broken: https://review.openstack.org/#/c/354173/2/etc/kolla/globals.yml 12:54:37 bjolo, ^ 12:55:21 uuu new documentation changes :D 12:56:13 did it get to stable/mitaka too ? 12:56:20 or just master branch ? 12:56:57 Anyone have issues with scheduling boot from images? 12:57:04 I can boot from volumes just fine 12:57:53 sounds like glance issues 12:58:12 had different issues depending on glance backend and OS release 12:58:27 finally in stable/mitaka on ceph - everything seems to be working fine 12:59:00 are you trying to use multinode topology and glance with file backend ? 12:59:26 Mr_Broken: glance logs look clean 12:59:38 Mr_Broken: I just have a little aio install 13:00:16 so what are the issues exactly ? 13:00:37 in what state is the Instance after trying to boot ? 13:00:41 what image are you using ? 13:01:25 Mr_Broken: Checking now. I'm guessing either a cinder-volume timeout or something else I just fixed 13:01:38 I'm running kolla on gentoo, so I have to configure firewall_driver to noop 13:02:07 But I just persisted it in /etc/kolla/conf/ml2_conf.ini 13:02:20 Forgot to do that last time I reconfigured :) 13:03:01 yae, might also be a network issue :] 13:03:37 Mr_Broken: ovs runs on the host uncontainerized. Works just fine. iptables is the issue 13:04:16 something about the syntax w/ iptables_manager in neutron doesn't work well with the 4.8 kernel or utilities 13:04:19 and neutron_server (with agents) still on container ? 13:04:24 Yeah 13:04:29 and it works ?! :P 13:04:32 Everything except ovs is managed by kolla 13:04:36 including ovs-db 13:04:44 Mr_Broken: I made it work, yes :) 13:04:56 you managed to get ovs uncontenerized with contenerized ovs-db ?! 13:05:05 I'm just that good ;) 13:05:09 my hero ! 13:05:13 how ?! :P 13:05:15 share ! 13:05:26 I was strugling with the same problem weeks ago 13:05:41 <--- gave up and uncontenerized ovs-db and some neutron_agents ;] 13:06:11 I have ovs running in foreground in screen. Command: http://hastebin.com/ebijijevex.vbs 13:06:19 Hi All 13:06:51 I've some trouble with the reconfigure action, should I expose here or in launchpad ? 13:08:33 @kbaegis: heh, good idea :] 13:08:50 @kbaegis: did you run into any issues across reboots ? 13:09:03 Let's see 13:09:15 or with ovs-ports not being removed when instance is being removed ? 13:09:17 etc 13:10:05 guess my multinode topology also adds a little network complexity 13:10:31 "mount --make-shared /run" 13:10:32 since my l3 agents and dhcp agents actually have some work to do, and are "trying" to work in HA :P 13:10:41 classic :} 13:10:44 , firewall noop 13:10:49 Those were the only 3 issues 13:11:00 ovs, noop firewall, and mount —make-shared /run 13:11:15 Also disabled haproxy/keepalived 13:11:20 yeah ;] 13:11:25 I don't need them, since my other nodes have <4GB ram 13:11:29 and master branch ? 13:11:31 Yes 13:11:41 yeah, a lot is fixed there 13:11:54 Only serious drawback I'm running into: no ui support for designate, magnum, etc 13:12:14 Also, w/ my setup I was having issues w/ my reverse proxy and novnc 13:12:24 do those even fully work in master branch already ? :P 13:12:29 Yes 13:12:29 magnum / designate ? 13:12:35 You can make bays from cli 13:12:47 The BIND backend I haven't tested w/ designate 13:13:02 i think there are some ansible roles still missing for designate 13:13:20 Makes sense 13:13:48 i am kind of forced to use stable/mitaka branch, so I might be wrong though 13:13:53 ipv6 would be nice with neutron too. Also dvr 13:14:13 dvr support is being worked on currently 13:14:20 should be merged soon 13:14:23 seen some blueprints 13:15:57 https://blueprints.launchpad.net/kolla/+spec/support-network-ha 13:16:17 That'll be nice. That's when I'll really move from AIO 13:16:37 we actually have internal stable/mitaka branch that support dvr's in HA 13:16:48 I need about 10k worth of equipment before I'll even get much use out of dvr 13:16:56 but dont have approvals from my company yet , to publish into community 13:16:59 morning 13:17:02 morning 13:17:03 sdake: morning 13:17:06 Jeffrey4l__ - hows testing going 13:17:13 sdake_: Wait, are there two of you? 13:17:26 sdake_: Underscore the difference? 13:17:30 connecte to vpn 13:17:44 disconnects my original connection - it willl ping timeout 13:17:49 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Getting NOVA pieces together to launch VM https://review.openstack.org/358911 13:17:55 sdake_, fixed some network issue. Just try to start the test 13:18:06 start linuxbridge test 13:18:08 or start fio test 13:18:28 sdake: vm in kube is running, how is this for good morning ;-) 13:18:37 pbourke i gathered oprofile data 13:18:42 pbourke interested in seeing? 13:18:50 sbezverk_ fantastic 13:18:57 sbezverk_ the talk on kube was accepted 13:19:06 so I guess we should have something working by summit then ;) 13:19:52 Merged openstack/kolla: Fix toolbox permission issue when enabling ceph https://review.openstack.org/358973 13:19:57 kbaegis the way to tell its me is my cloak will be @fedora i.e. sdake@fedora/dake 13:20:21 i do this so people dont gather my home ip address and harass my gear 13:20:42 it is infinately more feasible to hack soemone you know 13:20:44 sdake_, test started 13:20:45 then someone you don't ;) 13:20:54 jefwhich test, fio or the scenario 3? 13:21:05 or rather scenario 4 13:21:22 made some small fio test. 13:21:28 no is scenario 4 13:21:32 now is 13:21:37 fio worked on scenario 2 13:21:39 ? 13:21:54 yep 13:21:59 network issue. 13:22:00 fixed. 13:22:05 we might as well be capturing thse fio tets 13:22:07 (sorry for interrupting) I've some issue with the reconfigure action when using fake driver and neutron_openvswitch_agent : can anyone help me here or should I go to launchpad ? 13:22:12 the network issue- was it codified in init-runonce-osic? 13:22:22 sdake_, yes. 13:22:26 msimonin does it just fail? 13:22:36 Jeffrey4l__ i mean did you codify the fix 13:23:16 sdake_, a fixed version is here /root/osic_playbooks/init-runonce-osic 13:23:32 sdake_: yes, just like this (stable/mitaka) : 13:23:33 TASK: [neutron | Restart the neutron_openvswitch_agent container] ************* 13:23:33 skipping: [parasilo-28-kavlan-4.rennes.grid5000.fr] 13:23:33 skipping: [parasilo-8-kavlan-4.rennes.grid5000.fr] 13:23:33 fatal: [parasilo-5-kavlan-4.rennes.grid5000.fr] => error while evaluating conditional: openvswitch_agent_envs['KOLLA_CONFIG_STRATEGY'] != 'COPY_ONCE' 13:23:57 msimonin file a bug pleae 13:24:11 msimonin may I suggest you use stable? 13:24:30 check the fio test html report on the deployment node:/tmp/a.html 13:24:36 msimonin are are you aprt of the project to test openstack at 5k node scale? 13:24:37 sdake_, ^ 13:24:49 sdake_: yes 13:25:03 I think the test here https://github.com/openstack/kolla/blob/stable/mitaka/ansible/roles/neutron/tasks/do_reconfigure.yml#L237 is wrong 13:25:05 msimonin i'd still recommend stable unless you want to test all of openstack master 13:25:07 I can fill a bug 13:25:18 I'm using stable/mitaka 13:25:34 there should be no problem with stable/mitaka with that code 13:25:40 hm 13:25:49 but pleawe filea bug 13:26:02 Actually I've some more traces I can attach them in the bug 13:26:13 different issues? 13:26:27 no same on with more debug to help identifying 13:26:38 same one* 13:26:47 msimonin - attaching your globals.yml would be helpfu a well 13:26:55 or linking to it 13:26:56 ;) 13:26:57 sure 13:27:12 what scale are you at currently with kolla? 13:27:20 1000 compute fake driver 13:27:37 interesting 13:27:41 and that works? 13:27:41 actually I had also an issue with the mariadb configuration 13:28:22 Jeffrey4l__ did you get fio output in json? 13:28:27 that works after removing all wrep* configuration from galera.cnf 13:28:38 (I was using only one mariadb node) 13:28:50 so without ha, mariadb scales just fine :) 13:28:55 how many control nodes? 13:28:56 sdake_, just small test, to test my script work. 13:29:04 i will get the detailed test result later. 13:29:12 sdake_: only one control 13:29:23 1 control node for 1000 computes? 13:29:38 yes 13:29:53 I don't say the rally benchmarks were smooth :) 13:30:00 we can help theree 13:30:07 ok cool ! 13:30:12 I'll be pleased 13:30:13 we hae just been jering aroudn with getting rallly running for over 2 weeks 13:30:35 on a 130 node physical cluster 13:30:42 so when I created /etc/kolla/config/ml2_config.ini, it's options didn't overwrite openvswitch-agent 13:30:42 without fake driver 13:30:49 What did I do wrong there? 13:31:05 maybe m2_config.ini isn't customizable 13:31:09 but i'm not certain 13:31:09 sdake_: how things are going with this deployment ? 13:31:34 msimonin we can do a 123 node deployment in roughly 20 minutes 13:31:47 w ehae 6 scale scenarios we are testing across two different oses 13:31:54 i think we are eliminating one of the scale test 13:31:59 so we will haea total of 10 tests 13:32:05 we are on test #2 now 13:32:46 it takes about 5 hours to run rally scenarios e have defined 13:32:52 sdake_: What strategy do you have to speed up the deployment (prefetched images ? ) 13:33:19 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Getting NOVA pieces together to launch VM https://review.openstack.org/358911 13:33:21 the tactical plan we executed on was to automate testing, and automate information gathering 13:33:36 the automated testing takes about 4-5 hours 13:33:55 sbezverk_ does that nova vm have ovs connectivity? 13:34:59 sdake_ I do VM interface plugged into ovs bridge, but I need to bring up vnc pod to actually test it 13:35:00 Nice. boot from image is working 13:35:19 sdake_: I meant I do see VM interface .. 13:35:28 msimonin wee can share our automatation and test selection that works 13:35:57 Think I'm still having issues with root volumes from flavor though. Can those timeout? 13:36:07 Jeffrey4l__: you still on osic? need any help? 13:36:09 msimonin i am happy with teh peed of deployment - 20 minutes for 123 nodess 13:36:38 pbourke if your on osic check out the file in there ending in .txt startin with oprofile 13:36:43 sdake_: That would be great if you can share 13:37:01 msimonin ya give me a moment to get on the test cluster 13:37:12 sdake_: we don't have yet a real strategy to deploy at scale just a local prefecthed registry 13:38:38 pbourke, no right now. the test is running. 13:38:54 we are testing scenario 4, now. 13:40:34 msimonin: Use bittorrent! ;) 13:40:48 msimonin: Docker containers can be spread by torrent, right 13:40:55 Kevin Fox proposed openstack/kolla-kubernetes: Document some Conventions https://review.openstack.org/361186 13:41:00 Use riffle just to make it complicated 13:42:30 kbaegis: wow right ! 13:43:03 msimonin : http://paste.fedoraproject.org/414343/21896914/ -> http://paste.fedoraproject.org/414343/21896914 13:44:25 sdake: bookmarked :) 13:44:53 sdake: what is the diff between the two ? 13:46:11 msimonin i'd download it - fpaste doesnt' keep data around for a long time 13:46:14 same link 13:46:28 msimonin fpaste is supposed to url shorten things 13:46:34 but that doesn't seem tobe working lately 13:46:37 ok 13:46:39 :) 13:48:49 Jeffrey4l__ can you update nextsteps file pls 13:49:12 sdake, hmm, what's the detail? 13:49:30 next scenario? 13:53:56 Paul Bourke proposed openstack/kolla: Architecture guide doc https://review.openstack.org/360796 13:55:11 exact steps 13:55:17 clearly the next scenario isn't 2 its 3 i think 13:55:24 check etherpad 13:55:27 its about setting up tls 13:56:09 https://etherpad.openstack.org/p/kolla-N-midcycle-osic 13:57:15 msimonin there are several scripts in there as well as our"ignore" file 14:01:18 Kevin Fox proposed openstack/kolla-kubernetes: Document some Conventions https://review.openstack.org/361186 14:03:28 sdake: I filled a bug report -> https://bugs.launchpad.net/kolla/+bug/1617334 14:03:28 Launchpad bug 1617334 in kolla "reconfigure action fails on [neutron | Restart the neutron_openvswitch_agent container] " [Undecided,New] 14:04:34 Li Yingjun proposed openstack/kolla: Add option ceilometer_database_type for ceilometer https://review.openstack.org/345881 14:13:03 kbaegis: to distribute over torrent do you use a modified registry ? 14:14:20 msimonin: The only place I've seen that model is with cisco phones 14:14:35 they set up p2p connections to redistribute firmware. Never seen it w/ docker 14:15:03 My point is how kolla team achiev 20min deployment for 123 nodes ? 14:15:13 10gb/s connections 14:15:19 And network copy 14:17:48 I don't think ansible hyperscales 14:18:02 I haven't looked into tower's architecture 14:20:19 irtermite are we on 40gig network in the osic lab, or 20gig or 10gig? 14:27:10 good morning 14:35:17 sup inc0 14:35:22 we finally have one scenario captured 14:35:27 scenartio #3 in progress now 14:35:28 yay 14:35:36 pbourke, ping 14:35:40 inc0: yo 14:35:41 so i need some shell help 14:35:50 pbourke, why do we want to move haproxy to control? 14:35:52 I've got a document that has as its first column a list of numbers 14:36:02 inc0 i asked the same hting :) 14:36:11 pbourke said it was ineffecient 14:36:12 it'll make upgrades harder, security weaker...and I can't see much value in it 14:36:22 inc0: it's standard convention to run it on the control nodes 14:36:32 it should never have been placed on the network 14:36:39 but why? 14:37:02 if you want to have public endpoints, you don't really want to expose your controllers to external net 14:37:04 requests on control nodes have to go to another node 14:37:21 and you always need to do this for network node 14:37:45 well, 2/3 of controllers will go on a different node anyway 14:38:02 actually more 14:38:35 8/9 of requests will cross between nodes with 3 controllers 14:39:05 ok im happy enough to leave it if you dont think it makes sense 14:39:20 well, changing arch is hell of a migration 14:39:32 control downtime will be required 14:39:47 maybe even long as ARP tables would need to reload 14:40:16 also it might break if somebody would make their arch with external network only connected to network node 14:40:31 and just ran upgrade with pre-defined inventory 14:40:54 actually no, if they have pre-defined inventory then it's ok 14:41:13 but moment they rebase their inventory and miss this change 14:41:18 things might break 14:41:48 granted, it's not super trivial to understand why haproxy+keepalived is on network node, but it does make a lot of sense 14:41:51 from security standpoint 14:42:53 in fact, I wanted to suggest people in this arch guide 14:43:03 to connect external facing network just to network node 14:43:03 s 14:43:12 and hide control plane from the world alltogether 14:43:19 kfox1111: ping 14:45:28 inc0 use my diagram pls 14:45:36 inc0 it explains the network architecture clearly 14:45:44 sdake, mind linking it? 14:45:51 inc0 moment 14:50:14 so folks, here are oprofile results of one kolla controller node under rally 128x96 test case 14:50:17 https://drive.google.com/file/d/0B8q6xDPETSkHY3RFWE51U1J3WlE/view?usp=sharing 14:52:44 inc0: https://www.gliffy.com/go/publish/10486755 14:53:25 that's incorrect sdake 14:53:31 pbourke just looking at that diagram, it appears the controller nodes will rquire *4* IPS 14:53:35 inc0 feel free to fix it 14:53:43 inc0 i want it to be correct - its part off our security analysis 14:54:15 it took about 6 hours to make 14:54:19 so my point is, you don't need to connect to external network from controllers at all 14:54:20 so i'd suggest just finetuningthat one 14:54:45 also we're missing few networks 14:54:47 the api network shoudl connect to routed internet for CSPs 14:55:17 maybe we need anothe rline "externally routed API network link" 14:55:30 so it runs well in lab enironments as well 14:55:43 i can make you a collaborator on this document 14:56:41 tha tlittle ebox on the api network line is a router with firewlal ;) 14:57:56 notice the title of the document "Deployment Architecture" 14:58:05 that is probably what you should call your doc in the queue 14:58:52 inc0 do you know if osic is 40gig, 20gig, or 10gig? 14:58:55 the nic in use is 10gig 14:59:01 rather 40gig 14:59:06 butnot sure about the actual gear behind it 15:05:32 sdake: the OSIC uses 10GBE devices. The nics are intel x710, the switches are cisco nexus and are setup to allow for LACP if you configure it 15:05:47 cloudnull thnks bro 15:06:37 all of the nics are the same using: X710 for 10GbE SFP+ (rev 01) 15:06:40 anytime 15:09:41 cloudnull thanks :) 15:09:44 brb need to relog networks 15:11:01 sdake, 10gig all the way 15:11:21 sdake, API network should *never* connect to external 15:11:48 public endpoints sits on HAProxy which sits on network node 15:12:11 hey folks 15:12:17 so from my oprofile analysis, and using top, we are only using about 10-30% of the control node's capacity 15:12:28 under full load 15:14:11 for folks that wnat to see oprofile summary report: https://drive.google.com/open?id=0B8q6xDPETSkHY3RFWE51U1J3WlE 15:18:39 Kevin Fox proposed openstack/kolla-kubernetes: Document some Conventions https://review.openstack.org/361186 15:19:45 sdake - I'm not gonna be able to pitch in on OSIC today. Gotta take my kid to doc appt in 30min. 15:19:54 sdake_ - I'm not gonna be able to pitch in on OSIC today. Gotta take my kid to doc appt in 30min. 15:20:14 Paul Bourke proposed openstack/kolla: Move rabbitmq precheck from start https://review.openstack.org/361298 15:20:20 britthouser I htink we have it under control 15:20:27 britthouser I do hae a huge favor to ask 15:20:38 sure 15:20:45 britthouser assuming we get ubuntu finished this weekend, I'm going to need centos7 deployed 15:20:53 (this weekend) 15:21:09 I started on that yesterday, lemme see if I can wrap it up. 15:21:21 i'll also needsoemone to do the actual deploy 15:21:26 as I honestly dont know how :) 15:21:47 what i need is 2-4 hours of your time this wekeend to sort this out 15:22:00 i'll get inc0 lined up on the timing as well so we can all3 bang on it 15:22:01 This weekend I'm not gonna be home. =/ Heading to Atlanta 15:22:09 oh bummer 15:22:29 ok - well then if you can finish anything you have around centos import 15:22:39 and provide some instructions somewhere 15:22:42 on how to get rolling 15:22:47 that would be super appreciated then :) 15:23:03 the reason i asked for weekend time - we only he cluster until tuesday 15:23:22 so we need to make sure its fullly utilized the next 4 days 15:23:26 Yeah I understand. If I was gonna be at home I would. 15:23:31 np 15:23:41 ifyou could get it staged and ready - that owuld be a huge help too :) 15:23:51 thats something you may be able to do today? 15:24:10 after retrieving munchkins from dr? 15:26:22 inc0 pbourke open up this document: https://drive.google.com/open?id=0B8q6xDPETSkHY3RFWE51U1J3WlE 15:27:42 what is this showing 15:27:54 total cpu cycle count on controller node 15:28:09 divided by cateegory of service 15:29:27 wow ehkstack taking almost 25% 15:30:05 load avg was 9-20 15:30:12 so we have plenty of cpu cycles to burn 15:30:31 load avg measures running processes sor processes waiting to be schedueld 15:30:39 but ya, heka piggy 15:31:09 heka must have implementation problems, it spends alot of time in kernel space 15:32:16 elasticsearch - well thats java for you 15:32:30 pehraps we can do some jre optimziation 15:52:45 Kevin Fox proposed openstack/kolla-kubernetes: l3/metadata DaemonSets + DVR, & labels https://review.openstack.org/357557 15:56:35 sean mooney available anywhere? 16:03:14 what would be nice is an elasticseach log forwarder written in c specifically for kolla's needs 16:03:58 we are not running a log forwarder in java 16:04:09 and the heka implementation seems pokey 16:04:54 possibly because of the lua parsing 16:05:09 hard to tell, I couldn't get a symbol list of heka because it is stripped... 16:10:26 Merged openstack/kolla: Add etcd container https://review.openstack.org/355156 16:13:14 ok, so everything got installed on multiple nodes, now horizon is giving me a 503 'no server availble to handle this request' 16:15:00 Tin Lam proposed openstack/kolla: Add ansible role for vmtp container https://review.openstack.org/358305 16:15:05 which based on coversations from other folks a few days ago, its a known issue? 16:17:38 back - was pulled out to meeting 16:18:28 inc0 https://drive.google.com/open?id=0B8q6xDPETSkHY3RFWE51U1J3WlE 16:18:33 look 16:18:38 what's that? 16:18:44 click and you will see 16:19:05 yeah, I still don't know what these numbers mean 16:19:18 a pie chart, nice colors 16:19:20 percentages of that components systemwide cpu count 16:19:27 cpu cycle count 16:19:39 the components include their kernel access 16:20:01 with a 128x96 rally run 16:20:25 it waits on IO... 16:20:30 which is kinda expected 16:21:09 waiting on io count doesn't generate cycle count 16:21:26 this is pure execution - no waiting included 16:21:35 ok, we do share controllers and network nodes 16:21:39 this isn't runtime, but clock cycles used by binary 16:21:40 we might want to separate these 16:21:51 and re-bench 16:22:09 its not a benchmark its a performance profile 16:22:11 (just to be accurate:) 16:22:30 ok, rerun benchmark and get new performance profile 16:22:31 brb 16:22:32 anyway i thiink we wont have time to do that, but if we do i can do it 16:22:59 inc0 i am going to need your help this weekend to redeploy the nodees to centos most likely 16:25:09 inc0 also Jeffrey4l__ fixed a network issue in the osic setup script 16:25:13 so now vms can access the interneets 16:25:20 and wrote a specific io benchmark test for rally 16:25:44 using fio 16:25:48 its really a nice peice of work 16:25:51 hope he submits to rally 16:34:40 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Getting NOVA pieces together to launch VM https://review.openstack.org/358911 16:36:13 Michal Jastrzebski (inc0) proposed openstack/kolla: Architecture guide doc https://review.openstack.org/360796 16:44:42 when i do a pip install, where do the tools end up? specifically the admin-openrc.sh 16:52:42 jrich523: You have to run the post-install script to generate that one pretty sure 16:52:57 you sir are correct 16:53:01 jsut did that 16:53:29 jrich523: noice 16:53:42 indeed! 16:53:48 and im in! 16:53:52 lets see if both nodes show up 16:54:12 nice :) 16:54:36 jrich523: happy stacking! 16:54:46 lol oh i dont think we're there yet :) 16:54:50 i've made it this far before 16:56:31 hmm so i noticed in the globals, that heat was an options (just like horizon, which i turned on) 16:56:43 i left heat off for now, cuz.. why add complexity... but i noticed it built the docker images 16:56:59 which is cool cuz i want heat, but... shouldnt it have not done that? 16:57:42 jrich523 all images are built unles you specify the --profile otpion 16:57:50 oh cool 16:58:20 hmm built, deployed and configured? 16:58:40 jrich523: also I could be wrong, but it looks like unless you explicitly uncomment the 'enable_heat: no' in globals it's enabled by default 16:58:44 jrich523: https://github.com/openstack/kolla/blob/master/ansible/group_vars/all.yml#L222 16:59:07 oh cool 16:59:08 thanks 16:59:23 yeah, this is all the defaults, good to look at, thanks 17:00:17 np 17:13:23 Christine Hicks proposed openstack/kolla: Changed admin-openrc from DOMAIN_ID to DOMAIN_NAME https://review.openstack.org/361390 17:16:05 pbourke mind reviewing the above change ^^ 17:16:11 dont workflow a +2 would do ;) 17:16:19 want to wait for gate on workflow 17:18:11 inc0 ^^ 17:39:41 rhallisey ping 17:41:18 sdake_, yo 17:41:40 can you review https://review.openstack.org/361390 17:41:41 a +2 will do 17:41:45 and i'll workflow after gate shows success 17:42:25 hi aNuposic, there 17:43:05 look fine 17:44:11 I use the global.yaml file you send to me yesterday, and tried to deploy again, but I still not see the bridage was created..:( 17:59:54 hi , i just check the debug log ,there is no bridge was create, http://pastebin.com/qz13FwGm 18:00:07 can anyone please take a look for me 18:01:11 this is my global.yaml 18:01:12 http://pastebin.com/RLU1zEZh 18:42:06 hi guys, I am trying to use linux bridge to deploy, but I got some error when i did the kolla-ansible deploy 18:42:08 changed: [localhost] TASK: [haproxy | Waiting for virtual IP to appear] **************************** failed: [localhost] => {"elapsed": 301, "failed": true} msg: Timeout when waiting for 10.145.213.62:3306 FATAL: all hosts have already failed -- aborting 18:42:19 Merged openstack/kolla: Changed admin-openrc from DOMAIN_ID to DOMAIN_NAME https://review.openstack.org/361390 18:42:24 the precheck is fine, is anyone know what happen 18:42:53 Michal Jastrzebski (inc0) proposed openstack/kolla: Customization for base https://review.openstack.org/358042 18:43:31 wally__, you check if keepalived managed to setup floating ip 18:43:38 go to first controller node and do ip a 18:43:50 see if ip lands on network_interface 18:44:58 hi inc0 can you take alook at this for me http://pastebin.com/87zzLEta 18:45:07 i paste the ip a output 18:45:23 can I see your globals.yml? 18:45:29 ok one sec 18:46:05 http://pastebin.com/Pas5XYPP 18:46:12 this is one, I use linux bridge, 18:46:31 before I use ovsbridg,but i dontknow why it did not create ovs bridge for me 18:47:37 how many controllers do you have? 18:48:04 i am install AIO 18:48:07 just one 18:48:29 please paste output of docker ps -a 18:48:36 ok 18:49:05 http://pastebin.com/0X1SSvET 18:49:14 in the middle of deploy 18:49:43 ok, docker logs keepalived 18:50:31 http://pastebin.com/Q8fdXnGR 18:50:35 somthing wrong 18:51:00 hmm...yeah 18:51:42 how do deploy kolla with linux bridge 18:51:52 just change driver to linuxbridge 18:51:55 but that's not this issue 18:52:03 keepalived is the same regardless 18:52:21 do u have your global.yaml 18:52:26 can i see 18:53:01 do you have another kolla running in same network? 18:53:09 yeah, 18:53:10 or rather, another keepalived running? 18:53:13 there ya goi 18:53:18 yeah, how do u know 18:53:22 #keepalived_virtual_router_id: "51" 18:53:30 yeah, that is another one 18:53:30 uncomment this line and change a number 18:53:32 to something else 18:53:43 then cleanup and redeploy 18:55:44 hi inc) 18:55:49 hi inc0 18:55:59 you were out for a minute...or less;) 18:56:02 but hello;) 18:56:06 yeahn 18:56:11 so hwat do u need to do 18:56:14 i did not see it 18:56:20 waht do i need to do 18:56:26 #keepalived_virtual_router_id: "51" 18:56:27 [13:53] uncomment this line and change a number 18:56:41 this line is #keepalived_virtual_router_id: "51" 18:56:42 alrady 18:56:42 after that cleanup and redeploy 18:56:55 uncoment 18:56:56 okok 18:56:57 change number to anything else 18:57:24 what is this for 18:57:42 keepalived_virtual_router_id: "61" 18:57:42 keepalived communicates with each other using broadcasts 18:57:45 is thati ok? 18:57:47 yeah 18:58:04 ok let me try again 18:58:09 so if old keepalived, from another kolla, will broadcast its shit, new one will get confused 18:58:19 this identifies mutliple keepalived instances 18:58:25 ok, i c , 18:58:45 so number itself doesn't really mean anything, it just has to be unique 18:58:55 so if i have one than one kolla in the same subnet, I need to uncommend this 18:59:04 and change number, yeah 18:59:08 ok 18:59:20 let me know if it fixes your issue 18:59:49 ok i am doing the cleaning 18:59:55 good luck:) 19:00:28 another quesiotn, i use ovs-bridge before, but why there is no br-ex create when i do the ifcofig 19:00:31 so confused 19:00:41 so I now use linux bridge 19:02:28 wally__, it's inside container 19:02:38 try docker exec -it openvswitch vsctl show 19:02:40 it's there 19:02:41 oh,, 19:03:20 but after that why my VM has not internet accesss, cannot reach outside, 19:03:51 then we should debug this:) 19:04:03 ok, let me try again, 19:04:17 do u have a globla.yaml file , can you send to me 19:04:42 Merged openstack/kolla: Move rabbitmq precheck from start https://review.openstack.org/361298 19:08:56 hi inc0 19:09:42 sorry I wasnt looking at irc 19:09:50 I don't have any linuxbridge conf atm 19:09:58 http://pastebin.com/t43YNKzM 19:10:15 I just check the ovs bridge , seems it there. can you plese check 19:10:26 but seems something wrong 19:10:30 wally__, inside docker 19:10:40 yeah, i login the docker 19:10:53 (openvswitch-vswitchd) 19:10:59 a yeah I see that 19:11:03 http://pastebin.com/t43YNKzM 19:11:07 can you cehck this 19:11:14 seems like something broke during deployment 19:11:23 it is down 19:12:02 ovs-vsctl ovs-vswitchd (openvswitch-vswitchd)[root@kolla /]# ovs-vsctl show 2016-08-26T19:08:26Z|00001|ovsdb_idl|WARN|Interface table in Open_vSwitch database lacks error column (database needs upgrade?) f554bb5b-4dba-42d1-ae59-3352997afe21 ovs_version: "2.0.2" 19:12:07 somthing wrong? 19:12:13 yeah 19:12:25 can you do docker logs openvswitch_vswitchd 19:12:31 and openvswitch_db? 19:12:57 once sec 19:13:19 http://pastebin.com/AHXgMXi5 19:13:53 http://pastebin.com/yppcfXdN 19:13:57 thses all 19:14:12 somehting wrong with db? 19:14:18 can you look at /var/log/openvswitch...? 19:14:20 inside containers 19:14:25 one sec 19:15:44 http://pastebin.com/1L8k6XtV 19:15:46 this is the one 19:17:34 all the br-ex, br-int are down,,, it created the bridge,, 19:18:14 inc0, I almost have my cluster working... But have a couple of Neutron questions. Does neutron/openvswitch need to own a NIC on all six server or only the three controller running neutron-network? 19:18:17 hmm logs looks good 19:18:49 jemcevoy, nic has to be owned really on compute 19:19:01 and network 19:19:03 :( then waht is the problem, the error 19:19:09 pretty much wherever openvswitch is running 19:19:11 I need to know If I need 3 cables run from eth1 to the switch or six 19:19:21 so it is six cables 19:19:24 yeah 19:19:29 computes needs it 19:20:24 is my globla.yaml ok? 19:20:41 well you've shown me one for linuxbridge, but looks correct 19:20:50 problem maybe somewhere else, hard to say now 19:20:53 Cool... And how do I force the uplink to use VLAN 10. the only way to the outside of my dev rack is on VLAN 10 19:21:11 jemcevoy, there 2 ways 19:21:30 either you make it native vlan on neutron_external_interface 19:21:33 and use flat network 19:21:53 or you use tagged vlan networking, but for that you need to change neutron config slightly 19:21:53 where? 19:22:14 neutron_external_interface: "eth1" 19:22:21 this is for my ovs deploy 19:22:24 wally__, so, logs of ovs looks good 19:22:43 eth1 is another interface with no ip assign 19:22:53 can you show me logs of ovs_db too? 19:23:01 ok 19:23:25 (openvswitch-db)[root@kolla openvswitch]# vi ovsdb-server.log 2016-08-26T17:36:23.651Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovsdb-server.log 2016-08-26T17:36:23.657Z|00002|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 2.5.0 2016-08-26T17:36:33.660Z|00003|memory|INFO|8388 kB peak resident set size after 10.0 seconds 2016-08-26T17:36:33.660Z|00004|memory|INFO|cells:232 monitors:1 sessions:1 19:23:27 only this 19:23:30 seem good 19:24:27 but when i do ovs-vsctl list-br, in the container ,nothing 19:26:57 Update cadvisor to v0.23.9 to fix a problem where attempting to gather container filesystem usage statistics could result in corrupted devicemapper thin pool storage for Docker. (#30307, @sjenning) 19:27:07 thats probably the issue that broke my box before... :/ 19:27:30 yay 19:27:37 well, you're using devicemapper 19:27:41 you're kinda asking for it 19:27:52 heh. I switched to btrfs this last time... 19:28:03 not sure which one I trust more though... 19:28:25 ubuntu seems to trust btrfs enough to put it's os on top of it by default 19:28:36 overlay is there by default in xenial too 19:30:42 wally__, tbh I don't have idea now 19:30:51 you need to debug, check perms of ovs db 19:31:06 although we had some issues with perms before...just didn't show always 19:31:10 which is strange 19:31:22 ..... 19:31:47 so .did u try deploy with ovs before? 19:31:59 lest time yesterday on 130 nodes... 19:32:13 oh, really, it work? 19:32:17 ofc 19:32:28 anyway, you might have hit some bug 19:32:37 can u see your globla.yaml file 19:32:41 which we haven't seen before 19:32:44 please 19:32:50 evening everyone and happy friday 19:33:04 nothing too fancy there, let me see if it's published in our review:) 19:33:24 anyway, could you try to deploy master please? 19:33:28 I might have theory 19:33:36 yeah, I am goinng to do it again 19:33:45 try master+ovs 19:33:49 and let's see 19:34:01 waht do u mena master + ovs 19:34:09 kolla from master 19:34:11 waht do u mean, mastger +ovs 19:34:17 how many nodes, 19:34:24 https://github.com/openstack/kolla 19:34:26 one for kolla 19:34:38 similar setup as one broken 19:35:32 i am using allin one 19:35:41 that's ok, try it 19:36:09 ok, can you please send me your globla file 19:36:27 nice... k8s 1.4 alpha 3 has scheduled jobs. 19:36:41 we can do fernet token roling with that. :) 19:37:23 wally__, I really don't want to send this config, I have several of them with all sorts of different stuff in it 19:37:34 ok, i c 19:37:50 also we need to figure out what's wrong with yours if anything 19:37:56 I couldn't find anythign wrong wit it 19:37:58 with it 19:38:03 before I do it agin, i just want to make sue my network setup is good 19:38:18 how is your netowrk setup in your master 19:38:20 for aio it doesn't mean that much anyway 19:38:32 same as mitaka, also again, for aio it's not a big issue 19:39:05 wally__ did you read the article I posted for you lastnight? 19:39:17 it was about aio deployments 19:39:23 Yeah, i following the same 19:39:33 ok cool 19:39:45 but there iss some probloem about the bridge 19:40:00 have you tried with stable branch? 19:40:07 or master? 19:40:17 http://pastebin.com/t43YNKzM 19:40:21 can you pleas cehfck this 19:40:24 @pavo 19:40:44 looks right to me 19:40:59 the bridge is down 19:41:10 inc0, Do all [compute] nodes as well as [network] nodes need to have "neutron_external_interface="enp1s0f1" defined? 19:41:19 and you should use the openstack clients and sourcing your keystone rc to check ovs cmds 19:41:31 jemcevoy, yeah 19:41:37 bridge status might say down but thats might because of how the bridge works 19:41:52 jemcevoy, https://review.openstack.org/#/c/360796/4/doc/architecture-guide.rst 19:42:04 (openvswitch-vswitchd)[root@kolla /]# ovs-vsctl show 2016-08-26T19:24:01Z|00001|ovsdb_idl|WARN|Interface table in Open_vSwitch database lacks error column (database needs upgrade?) f554bb5b-4dba-42d1-ae59-3352997afe21 ovs_version: "2.0.2" 19:42:05 take a look at this 19:42:14 and i did not see any bridge 19:42:51 so, u tried it and it working for you right? 19:42:52 wally__ install the openstck-clients and do a kolla-ansible post-deploy on master node 19:43:08 inc0, Thanks that doc looks very helpful... 19:43:14 and then source keystone rc file and then from master node use openstack ovs blah blah 19:43:15 hehe 19:43:17 I did the post deploy 19:43:33 just wrote it yesterday;) I figured it'll gonna be useful 19:43:52 ok source /etc/kolla/rc file 19:44:07 then do openstack ovs-vsctl list 19:44:09 openstack ovs blah? 19:44:53 root@kolla:/etc/kolla# openstack ovs-vsctl list openstack: 'ovs-vsctl' is not an openstack command. See 'openstack --help'. Did you mean one of these? 19:44:55 ?? 19:44:57 I don't have my environment up right now so don't know the rc file name or ovs cmds right off bat sorry 19:45:36 ok try ovs-vsctl list 19:45:41 without openstack 19:45:47 I can't remember 19:46:26 root@kolla:/etc/kolla# ovs-vsctl list-br root@kolla:/etc/kolla# 19:46:32 show nothing, 19:46:48 but this is only for the master node, right 19:47:03 I'll throw up a aio VM later tonight and see what I can do 19:47:14 when you insall the kolla, did u use stable branch or the master? 19:47:20 I do stable 19:47:51 unless you plan on doing development and testing I would use stable 19:47:56 how did u insall kolla 19:47:58 pip install kolla? 19:48:01 yeah 19:48:16 then I use the same 19:48:16 and pip install ansible==1.9.4 19:48:21 yeah, the same 19:48:39 ok, man, can you plase try aio again? 19:48:47 just take 15 mins to do 19:48:57 yeah I will once I get done with dinner 19:49:00 wally__, also, which distro you have on host and which one in container? 19:49:13 I use centos for host and centos for containers 19:49:13 ubunt 1404 19:49:20 all ubuntu 19:49:31 you did mount --make-shared /run? 19:49:32 I never use ubuntu 19:49:34 right? 19:49:38 yeah 19:49:48 i use mount --make-shared /run? 19:49:48 ok, it would break otherwise 19:50:06 yeah i run mount --make-shared /run 19:50:07 well without ? but yeah you need this in ubu14 19:50:15 yeah i did 19:50:30 can you pleaes try with ubuntn 19:50:37 Hi all, is it always required to have 2 or more nics on the machine where kolla is deployed? http://docs.openstack.org/developer/kolla/quickstart.html 19:50:39 yesterday - that was all ubuntu 19:50:41 but master 19:50:58 Merged openstack/kolla: Add full support for fernet https://review.openstack.org/349366 19:51:00 aNupoisc, yes, if you're running neutron that is 19:51:15 inc0, nice! 19:51:19 fernet got in 19:51:30 inc0, https://review.openstack.org/#/c/351139/25 19:51:32 yeah with one caveat 19:51:42 what's the caveat 19:51:50 I don't really like keystone_ssh 19:51:54 we have nova_ssh 19:52:09 which kinda became a standard 19:52:17 but I guess we need to figure out this stuff 19:52:18 I don't know what you mean 19:52:30 fernet requires token distribution 19:52:33 it uses ssh here 19:52:36 it should use ansible 19:52:44 but I don't want to stop whole feature on this 19:52:46 inc0: ahh good to know. I am doing all-in-one setup with just ceph 19:52:58 and also, pavo, do u have your init-runonce file 19:53:03 can I see it 19:53:11 to create the network 19:53:15 inc0: so i will be enabling ceph and ceph_rgw from globals.yml 19:53:21 rhallisey, can you take a look at it? https://review.openstack.org/#/c/360796/ 19:53:46 inc0: so I assume that requirement is not meant for me. 19:53:46 I get your point, but how would that look with anisble 19:53:48 I'm not seeing it 19:53:57 inc0, was going to ask you to rebase on my doc struacture 19:54:03 aNupoisc, no, you should be ok without this setting 19:54:08 let me fix my doc structure though real quick 19:54:13 sure 19:54:18 inc0, I have a few things that i have stumbled over that may be useful in a doc like yours. Can I email then to you next week? IRC seems too ephemeral 19:54:33 inc0: cool. 19:54:35 jemcevoy, either that or write a patch with it 19:55:02 I'd prefere latter ;) you should take credit for this stuff too 19:55:24 but if you don't want to submit it for whatever reason, sure mail would be ok 19:55:35 That would mean that I would need to signup with OpenStack? 19:55:48 yeah, it only cost a soul 19:56:39 but jokes apart, it's really cool when you see your patch merged 19:56:51 I really like this feeling;) 19:57:02 I'll ask my buddy from Marantis if he can help me through that hurdle 19:57:18 we can help you if you need help 19:57:42 there is guide for it actually, hold on 19:58:05 http://docs.openstack.org/infra/manual/developers.html 19:58:29 First question... Should I use my work or personal email? 19:58:55 depends, is your company contributing to openstack? 19:59:34 They may have done a little... Penguin Computing not sure 20:00:21 well, let me put it this way, if it's just you, I'd go with personal mail and individual contributor agreement 20:00:33 there is paragraph about this in guide I've linked 20:01:02 I will get stared on that over the weekend... 20:01:07 thanks! 20:01:23 it's not as bad as it seems, and you'll be listed as one of authors of kolla 20:01:53 also contributors get free access to summit 20:02:02 full pass 20:02:14 Cool.. 20:03:12 hi, pavo, can I give you a server, and you can try on it? 20:03:24 i will give you a public ip to loing 20:03:27 login 20:03:48 I have never used ubuntu before so not sure if that would be good idea 20:04:01 wally__, I can do it if you want 20:04:05 ok 20:04:22 good, I will set up a ubuntnu 1404 server and give you public ip later 20:04:30 give me 2 hours 20:04:48 ubuntu 1404 server, right? 20:04:51 ahh 2 hrs...Friday evening:( 20:04:56 ok, ik c 20:05:02 next monday 20:05:03 I'll play over weekend if that's ok 20:05:05 or Monday 20:05:05 is it ok for your? 20:05:10 sure, thanks 20:05:33 i will set it up fist , then email to you? can u tell me your email? 20:08:42 i will send you all the detail include network subnet range, then you can assign for the floating ip 20:09:35 Merged openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 20:18:14 Merged openstack/kolla: Fix bug in Elasticsearch role https://review.openstack.org/359596 20:26:01 inc0, One more question... Can neutron/openvswitch share a NIC with haproxy, horizon, etc? Or will I need two NICs on VLAN 10? 20:26:18 you want 2 nics 20:26:24 neutron will bind bridge to the port 20:26:32 which will render port unreachable 20:26:47 there are ways to make them share network, but not supported here 20:26:58 however 20:27:04 what you can do 20:27:10 is to make 2 nivs 20:27:12 nics 20:27:17 OK... I am going to need another 1Gig switch 20:27:25 one for vlan 10 another in whatever you want 20:27:30 nah 20:27:44 can you trunk your netowork therE? 20:28:09 hi inc0, can you gvie me you email, then I send you the server info later 20:28:17 wally__, sent over in pm 20:28:42 elemoine_, hey Eric, long time no see:) 20:28:42 in pm? 20:28:54 private message, you should have it in irc 20:28:56 somewhere 20:29:20 how to send in pm? 20:29:41 I will need eth0.10 for haproxy horizon, etc and eth1 native VLAN10 for neutron and the rest of the traffic on the 10Gig internal network 20:31:26 hmm, sooo 20:31:31 let me think 20:31:45 you want flat network to be in same net as haproxy 20:31:50 ok, so what we can do 20:32:09 is change your inventory in a way to put haproxy and keepalived on control nodes 20:32:19 they don't need connection to public network at all 20:32:36 so you can just create port.10 there and bind haproxy to irt 20:34:02 either that or you just create virtual interface in network node and manually add it to br-ex 20:34:05 and use it for haproxy 20:38:02 Michal Jastrzebski (inc0) proposed openstack/kolla: Architecture guide doc https://review.openstack.org/360796 20:38:20 jemcevoy, ^ added note about kolla external interface 20:38:25 you might want to laverage that 21:03:07 Ryan Hallisey proposed openstack/kolla: Add docs for host setup https://review.openstack.org/343223 21:03:08 Ryan Hallisey proposed openstack/kolla: Shrink the quickstart massively and break it into 2 https://review.openstack.org/343224 21:20:33 Ryan Hallisey proposed openstack/kolla: Add docs for host setup https://review.openstack.org/343223 21:20:33 Ryan Hallisey proposed openstack/kolla: Shrink the quickstart massively and break it into 2 https://review.openstack.org/343224 21:20:50 down to 277 for devs 21:20:57 would like under 150.. 21:33:53 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Getting NOVA pieces together to launch VM https://review.openstack.org/358911 21:38:09 rhallisey 27 comments in the quickstart guide change 21:38:13 look at comment #7 21:38:31 rhallisey like under r15 0what rhallisey ? 21:39:59 150 lines 21:55:42 150 line of what? 22:03:22 Hi guys, i am not able to take a snapshot of my instance. I have this issue : http://paste.openstack.org/show/564088/ . Do you have an idea what happen ? 23:23:42 inc0, I got those eth1 ports plugged in to a switch... How do I restart to get neutron running properly... 23:46:00 jemcevoy best hting to do is probably redeploy but i don't know what state your in 23:46:16 what version are you running? 03:09:08 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Getting NOVA pieces together to launch VM https://review.openstack.org/358911 07:29:50 Shaun Smekel proposed openstack/kolla: Customizations for MariaDB https://review.openstack.org/354422 08:14:42 hi 09:19:42 morning folks 13:59:01 morning everyone 15:42:48 Pavo morning 18:34:17 anyone around that can hep with osic work? 18:38:09 sdake, I'm here, logging to osic 18:38:15 yo 18:38:25 so deployment currently jammed 18:38:43 i think a reboot may fix - but not certain 18:39:50 did this - see instructions below 18:39:54 in bottom window 18:41:50 isn't htis using master playbooks? 18:41:58 we want to use the system playbooks 18:42:06 hmm 18:42:22 i hae pip installl 1.1.2 18:42:26 but your in the kolla dir 18:42:35 not sure whats in there but if its kolla i think it will preer those playbooks 18:49:09 I think it's going on, just slower 18:50:13 it had red on the screen before dude 18:50:25 or was completel yblcoked 18:50:26 ok we'll see 18:50:28 not sur which 18:50:37 is it progressing? 18:51:29 i saw faults in the osd 18:54:43 yeah see whats wrong? 18:55:50 no tls ws wnabled 18:55:53 i disalbed it again 18:55:59 but that isn' the cause here 18:56:21 what is on 6789? 18:56:29 perhaps we cn cheeck that on one of th etarges that failed 18:56:42 wasn't 26 missing a 6789 process inc0? 18:57:11 no 18:57:16 something else is wrong 18:57:19 digging through code 19:00:12 we can add a mon oveerride 19:06:28 I don't want to add override here, we need to figure out why it's there 19:06:30 this is critical. 19:06:57 as long as it doesnt' take forever ;) 19:07:03 I dont want to block here for days 19:11:42 did you make any change? 19:12:02 no 19:12:10 it's generated incorrectly! 19:12:27 hmmm...let me check sth 19:12:33 let me ask what should it be 19:12:45 our prechecks are failing in bonding mode 19:12:49 maybe this a bonding mode problem 19:15:22 we never picked up on this because we deploy storage alongside control annd compute 19:16:10 for some reason it didnt override with globals 19:16:45 what is it spposed to be 19:16:52 it works now 19:16:58 when I changed it in all.yml 19:17:06 no diea what's happending 19:17:12 looks like ansible issue 19:17:12 missed our change 19:17:49 that was change 19:18:24 doing that in globals.yml had no effect? 19:18:37 for some reason wrong interface was used 19:19:30 well it was set to the default 19:20:47 someone fixed the defaults or the storae interface in master iirc 19:21:02 perhaps a missingbackport 19:21:41 this one didnt seem to change 19:25:47 old passwords.yml die hard 19:25:51 inc0 i left the password file in place 19:26:42 we might consider adding that variable back to passwords.yml 19:26:53 we don't use it any more 19:26:57 i know 19:27:00 or OpenStack for that matter 19:27:02 mark it as deprcated 19:27:12 and deprecate it appropiately 19:27:37 it's keystones job really 19:27:49 its part of our interface 19:28:13 anyway, deprecate properly means it disapears with newton 19:28:24 which is week from now 19:28:25 oh was this changed in mitaka? 19:28:28 yeah 19:28:33 k nm then 19:28:46 something to keep in mind for t he future 19:28:57 agree 19:29:19 so got osme intreesting data 19:29:32 tls is 20% slower then non-tls under very heavy api load 19:30:56 interesting, my bet would be that it would be slower, but not as much as 20% 19:31:10 unless you wer ehammering API with just read calls 19:31:18 its more like 0% slower if ther are not alot of api operations 19:31:25 ya just hammering api with just validatin calls 19:31:47 thats the perfoamnce impact for tls 19:31:54 0->20% depending on workload 19:32:39 kolla can be tuned here to handle more api call as well 19:32:45 that may or may not help 19:39:07 how do we get bck to hte other window inc0 19:39:16 if a service comes with a default configuration, do you need to define a config for those values inside an ansible role? 19:39:32 sdake ctrl-b -> z 19:47:03 https://github.com/openstack/vmtp/blob/master/vmtp/cfg.default.yaml#L4-L7 19:47:34 does this mean those values do not need to be duplicated inside the ansible cfg? 19:47:52 reconfigure taks 58 minutes inc0.. :( 19:48:13 how? it's 3 times longer than deploy! 19:48:57 inc0 no idea 19:49:04 inc0 check out the file scenario5 19:49:35 lrensing_ what it means is to override them you put them in a special file 19:49:50 lrensing_ as in a config file that is loaded along side vmtp 19:49:57 correct 19:50:19 https://review.openstack.org/#/c/358305/5/ansible/roles/vmtp/templates/cfg.default.yaml.j2 19:50:45 sdake: vmtp automatically puts in a cfg.default.yaml that it uses by default, and if a user wants to change that setting, they can pass that in via the -c option. 19:51:25 right - so to configure it to run in a cloud just make a cfg.deault.yaml.j2 file in defaults dir 19:51:35 and use the template function 19:51:51 then people can reconfigure or override the variables as they see fit 19:51:55 function/module 19:51:58 template/merge_configs 19:52:07 actually merge configs may noot handle j2 19:52:12 may just want to use the copy operation 19:52:31 okay - I was wondering if we want to change that file or let users pass it their configuration via -c 19:56:06 lamt let people pass it into the container when it is deployed 19:56:32 via a copy operation 19:56:35 moment: 19:57:49 https://github.com/openstack/kolla/blob/master/ansible/roles/nova/tasks/config.yml#L74 20:00:05 sdake: thanks 20:00:22 np 01:07:29 Merged openstack/kolla: Set 'balance source' for Horizon in haproxy https://review.openstack.org/356514 02:42:11 Larry Rensing proposed openstack/kolla: Add ansible role for vmtp container https://review.openstack.org/358305 03:47:13 hi sdake 03:47:33 zhubingbing_ shoot 03:48:12 happy weekend 03:49:04 i'll let you know when I have one of those ;) 03:56:13 Tin Lam proposed openstack/kolla: Add ansible role for vmtp container https://review.openstack.org/358305 04:48:21 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 05:05:04 holly batman the profile finallly finished 05:10:53 zhubingbing proposed openstack/kolla: Add Barbican container dockerfile https://review.openstack.org/351822 05:15:01 zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 05:25:17 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 08:00:17 Jeffrey4l__ about? 08:31:59 Hey... sshd on 172.21.8.253 not working? Getting the protocol banner, but no login? 08:33:11 sdake: ^ ? 08:33:44 Daviey let me check 08:35:45 sdake, yep 08:36:27 david-lyle_ wfm 08:36:29 Daviey wfm 08:36:50 wtf 08:36:51 Daviey tests are running atm 08:36:57 cool 08:37:13 Daviey i've been watching it all day - almost done with ubuntu test cases 08:37:31 we found tls adds 20% performance hit 08:37:45 yeah.. mental it has that cost 08:37:51 on very high rate API calls without a lot of backend processing 08:38:00 i was expecting worse 08:38:15 i think 20% is actually pretty good considering how its implemented :) 08:39:21 Jeffrey4l__ can you look at the gate patch 08:39:54 https://review.openstack.org/#/c/353942/ 08:40:05 Kolla has 420 stars on github ;) 08:40:08 OK 08:41:35 actually that probably needs a bug id 08:42:37 Daviey can you modify that patch with a bug id so jeffrey and i can approve it 08:44:36 sdake_: it isn't my changeset, i just fixed the pep8 issue that you introduced :) 08:44:56 Daviey would you mind modifying it? Else we wont be able to merge it 08:45:10 i'll file the bug if you like ;) 08:45:12 Here add a non-sense blank line, should be removed. https://review.openstack.org/#/c/353942/7/docker/swift/swift-base/Dockerfile.j2 08:45:32 sdake_: ok 08:45:39 and might as well get that blank line 08:45:43 moment filing bug 08:46:15 brb 08:48:29 ah, i just did that.. but LP 500'd on me 08:48:29 Launchpad bug 500 in kdegraphics (Ubuntu) "Package description does not match content/dependencies" [Medium,Fix released] https://launchpad.net/bugs/500 - Assigned to Kubuntu Bugs (kubuntu-bugs) 08:48:40 (shut up bot) 08:49:20 sdake_: bug 1617713 08:49:20 bug 1617713 in kolla "bandit gate tests are failing" [Undecided,New] https://launchpad.net/bugs/1617713 08:52:15 Daviey ok triaged 08:53:01 Dave Walker proposed openstack/kolla: Fix bandit gate jobs https://review.openstack.org/353942 08:55:05 ^^ 08:58:16 Jeffrey4l__: Is SSH working to root@172.21.8.253 for you? Seems wedged to me, i've tried from 2 different networks 09:00:58 Daviey re osic machine, did you launch the vpn software from f5? 09:01:00 it launches in a browser via plugin 09:01:00 for me (mac) it only works in safari 09:01:31 basically its a web based login to a vpn 09:01:40 sdake: yes... this isn't my first rodeo :) 09:01:46 Daviey oh and welcome to the core review team :) 09:01:55 sdake: thanks :) 09:01:57 Daviey cool - not totally obvous so wasnt sure 09:02:35 the vpn window should hae a green thing on it 09:02:41 indicating things are good after you login 09:02:55 sdake: On linux, you need to use the command line tool... the web based thing doesn't work 09:02:56 and showing network data transfer 09:03:08 right - no plugins in chrome 09:03:11 or firefox 09:03:23 sdake: I can get the ssh server login banner.. so it isn't vpn related 09:03:26 didn't realize you use linux as a desktop 09:03:34 which password you using? 09:03:52 sdake: not getting that far.. 09:04:16 how do you get a login banner but not ente a password? 09:04:41 sdake: sorry, not login banner.. protocol banner 09:05:35 sdake: http://pastebin.com/raw/kepASqin 09:05:51 sdake: (this has been working for weeks.. not sure why it isn't now) 09:06:45 let me log back into the machine 09:06:49 brb 09:06:56 hae to switch vpns :( 09:09:57 Daviey firewall o your side? 09:10:10 Daviey reboot required? 09:10:21 I don't know anything about the f5 vpn agent 09:10:51 the vpn agent cli tool probably has stats 09:10:58 so you can see transferred packets and whatnot 09:12:21 sdake_: Gah. It was my local MTU. 09:13:06 you have jumbo frames implemented in your internal network? 09:13:27 sdake_: Seems not. :D 09:13:55 ya - i'm not sure i culd get that to work 09:14:07 my cluster runs on 9k mtus 09:14:21 but all my gear is 1500 mtu on my main network 09:14:28 my cluster is on separate 10gig net 09:15:21 Jeffrey4l__ - sometime tomorrow (in 8-12 hours) the cluster is being reloaded with centos 09:15:39 Jeffrey4l__ os if you wwant to make an fio test, yu will have to do so as soon as the current test finishes 09:21:00 sdake_: I really wanted to try and squeeze in storage/ceph co-located with compute on all nodes and see how if that model worked at scale. Happy to do it on centos, after all the pre-planned stuff is finished 11:33:15 pbourke, do you have any idea why the tests is stop on the osic? some one ctrl+c it 12:36:19 Jeffrey Zhang proposed openstack/kolla: Fix removing nova_libvirt container issue https://review.openstack.org/361728 12:46:24 Jeffrey Zhang proposed openstack/kolla: Fix removing nova_libvirt container issue https://review.openstack.org/361728 13:13:06 Daviey around? 13:13:34 morning folks 13:31:48 sdake: good morning :) 13:31:58 sup 13:33:57 ooc is anyone working actively on horizon? Getting senlin/magnum/lbaas/sahara integrated? 13:44:54 that is what the third party plugins blueprint is all about 13:45:07 kbaegis but is anyone implmented all the pllugin logic, not at the moment 13:46:16 Gotcha. Thanks :) 14:39:47 disk full = bad things happen (tm) 14:41:57 wow mitaka deploys on 100 nodes in 9 minutes 14:42:05 newton deploys in 20 14:44:34 good morning everyone 15:00:46 hey pavo 15:00:59 irtermite around? curious what time on the 30th we lose access to the osic gear 15:01:29 so it seems 100 node deployment only takes 9 minutes 15:01:35 very nice 15:01:45 on mitaka 15:01:54 more concerned that it takes 20 on master with same configs 15:02:13 what do you mean by master? 15:02:21 as in image building? 15:02:30 i mean master takes 20 minutes to deploy without coutning imae buid 15:02:41 hmmm 15:03:35 well there are a few changes in the way it deploys and it is using a newer ansible right? 15:03:54 yup newer ansible version 15:04:04 reconfigure takes an hour on master 15:04:32 maybe something in the newer version is what is taking longer to deploy 15:04:46 hard to tell 15:04:51 yeah 15:05:01 but still 100 nodes in 20 isn't bad 15:05:04 ara wuld tell us but we don't have enough time to run thrug hthat 15:05:18 100 nodes in 20 minutes with ceph is fantastic 15:05:45 i think we can get to under 1 hour once ironic merges 15:06:01 for from bare metal to fully operational openstack cloud 15:06:24 so what are you using to provision the bare metal? 15:06:50 ironic in container for dep management 15:07:22 wait what, I thought ironic was only for provisioning instances using bare metal 15:07:44 not provisioning actual host node bare metal 15:11:03 so was horizon fixed on master I take it? 15:37:06 pavo we use bifrost for actual host node bare metal deploy 15:37:16 ironic thin containers are for BMAAS 15:37:20 that isn't ready to go yet 15:37:32 pavo see mailing list 15:51:00 Merged openstack/kolla: Fix bandit gate jobs https://review.openstack.org/353942 15:53:39 Merged openstack/kolla: Pin Ansible version to 2.1.1.0 in kolla-toolbox https://review.openstack.org/359124 15:53:46 Merged openstack/kolla: Create the _member_ role in the horizon role https://review.openstack.org/358298 16:04:07 Merged openstack/kolla: Customizations for RabbitMQ https://review.openstack.org/354409 16:08:25 Merged openstack/kolla: Customizations for Ironic https://review.openstack.org/354427 16:16:54 sdake_: here 16:17:39 hey Daviey 16:17:47 re deploying kolla across all storage nodes 16:17:51 if we hae time we can set that up 16:17:55 i think we wont have time though 16:18:17 the scenario tests take about 6 hours to run 16:18:22 and we have about 8 o them to do 16:18:30 8*6=48 hours 16:18:35 sdake_: okay, either way 16:18:47 you can have a crack at it once we get the main data collected 16:19:03 sdake_: wfm 16:19:24 Merged openstack/kolla: Customizations for MariaDB https://review.openstack.org/354422 16:25:37 Merged openstack/kolla: Customizations for kibana https://review.openstack.org/346252 16:45:01 sdake_: ping 16:45:08 wound me vhosakot 16:45:53 I didn't find you online yesterday.. I texted you.. :) 16:47:01 sdake_: I'll log into tmux.. 16:47:12 tests are running 16:47:16 it probably takes about 6 hours 16:47:24 is it scenario 6 ? 17:01:13 sdake_: I am in tmux... I see the steps.. cool.. after scenario 6 (which is running currently running), I'd like to help run scenario 7 (after 6 hrs once scenario 6 ends)... 17:01:27 did anyone sign up for scenario 7 ? or, can I run it ? 17:03:20 sdake_: let me know.. I'll be on IRC 17:11:06 we are on scenario 8 atm 17:11:11 you are welcome to watch me run scenario 9 17:11:22 its a simple shell script 17:14:45 cool.. you're already in 8.. I'll watch scenario 9.. ok 17:15:36 sdake_: yep, I'm seeing the shell script.. cool 17:17:57 sdake_: any idea how long scenario 8 takes and when you'll start 9? 17:19:48 no idea 17:20:00 run rally deployment list and tell me how many deployments y ou see 17:20:10 in the bottom window or right window 17:20:47 vhosakot_ - scenario #9 is benchmarking the various operations of kolla (pull, deploy, etc) 17:20:56 i wrote a script that does all the operations and captures them 17:21:58 sdake_: cool... I'll help with 9.. sure.. I see the steps open in VI editor in top left pane.. rally is running in bottom left pane (I wont touch it).. I will run "rally deployment list" in the right pane.. 17:24:55 oundsgood 17:28:35 sdake_: http://paste.openstack.org/show/564348/ scenario4-128x48 is running now 17:28:38 cool 17:31:14 cool shell script 17:37:37 now run rally task list | wc -l 17:37:52 128x48 is the second scenario 17:37:53 we have 3 17:38:05 each one takes about 2 hours i think 17:38:20 ther e are 107 tasks total in each deployment of rally 17:38:51 vhosakot_ if you want ot help - please review peoples' patches 17:39:05 our queue is super long and i'd like to get appropriat efeedback to people 17:39:19 sdake_: yep, will review patches... 17:41:03 sdake_: "rally task list | wc -l" shows 27 17:42:10 ok its aout 2/5ths done 17:42:13 so another 2-3 hours to go 17:43:04 sdake_: ah ok cool... 9 is upgrade to Mitaka and capture time right ? 17:43:18 I'll be here after 2-3 hrs.. cool... I'll review patch sets in queue.. 17:47:49 35 tasks are done 17:50:23 9 is run all commands and time them 17:50:35 there is a script in the rally.git dir that does this 17:50:37 dont run it 17:50:39 but youcan look t it if yo ulie 17:50:58 sdake_: cool, yeah, I wont run anything 17:51:16 9.2 is running mitaka to master upgrade 17:52:01 sdake_: I dont see 9.2 in https://etherpad.openstack.org/p/kolla-N-midcycle-osic :) you mean 10 ? 17:52:38 please add it the n:) 17:53:25 cool... since there is no 9.1, I'll add 9.1 as Upgrade Mitaka to master (Newton) and capture times 17:53:26 sorry it should be 11 17:53:37 ah, ok.. cool.. I'll add as 11 then 17:54:37 cool.. added 11. Upgrade mitaka to master (Newton) and capture execution time of all mitaka kolla-ansible operations 17:54:48 sdake_: what about reconfigure ? 17:55:17 I remember seeing it few days ago.. may be it is removed now 17:55:37 its in there 17:55:43 in my script 17:55:59 looka round for kolla* files 17:56:09 there is one with bunch of steps in it with comments 17:56:17 ok cool.. 19:14:02 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 19:15:50 Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 23:38:22 evening/morning/lot track of what time it is 23:38:28 alteat th sun still out 00:29:48 Daviey about? 00:30:12 Jeffrey4l_ ping re osic 00:30:29 Jeffrey4l_ i've captured all data for ubuntu 00:30:34 now we need to do the same for centos 00:30:38 sdake_, cool 00:30:48 Jeffrey4l_ do you hae any idea how to deploy centos? 00:31:06 no. :( never touch the deploy OS scripts. 00:31:32 it has soething to do with the lxccobbler container 00:32:23 let me check the doc and the script. 00:45:24 sdake_: regarding https://review.openstack.org/#/c/346214/ - you didn't want to use the $PATH method, and suggest I talk to pburke about it. 00:45:32 Steven Dake proposed openstack/kolla: Add OSIC Scale Testing Documentation https://review.openstack.org/352101 00:45:42 sdake_: I mentioned it a couple of times without response. 00:46:15 sdake_: so we ok with just using $PATH or what ? 00:47:01 Mech422 just rebase - and our nv gate will fial 00:47:06 and we can resolve that as a separate patch 00:47:44 Mech422 if pbourke_ gives the go-ahead we can merge it 00:47:54 sdake_: ok - I'll try to get to that later this week 00:47:58 even though it breaks the gate 00:48:11 sdake_: still doing stuff for $BOSS atm 00:48:12 it takes 30 seconds to rebase - like me to do so? :) 00:48:49 if you want - I'm actually setting up oVirt atm - after having done cloudstack and archipel 00:49:12 boss seems to want to try out everything with 'virtual' in the name 00:49:30 cool - hae any feedback from that? 00:50:37 Jeffrey4l_ can you chane mtu on two boxes without cratering the environment to 9k? 00:50:39 sdake_ where did OSIC test go? are you nearly done with this 00:50:40 and rerun test? 00:50:47 duonghq i am done with ubuntu 00:50:49 now ffor centos 00:50:58 need someone to deploy centos - he no idea how 00:51:02 nd test cases takes about 40 hours tor un 00:51:07 we he about 45 hours left 00:51:18 sdake_, we can try. but change the MTU need change the physical switcher too. 00:51:29 sdake_: we had high hopes for cloudstack in our small pop 'clusters'... 00:51:35 no idea what's the MTU on the switcher sdake_ 00:51:38 ok - lets try on two non-essential nodes then? 00:51:47 ok 00:51:48 sdake_: but he seemed to lose patience waiting for me to figure out the cloudstack networking 00:51:50 i assume 9k 00:52:42 sdake_: archipel is based on xmpp - very thin layer over libvirt, easy to deploy but seems a bit 'chatty' on the network side, and I'm not sure the agents are 'production grade' yet 00:53:08 sup v1k0d3n 00:54:49 Jeffrey4l_ both ends need to be mtu 9k 00:55:00 sdake_, it is now. 00:55:08 what is the second end? 00:55:43 0.23 vs 0.66 00:55:49 ok 01:01:11 sdake_, http://paste.openstack.org/show/564397/ 01:02:39 Ross Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 01:05:53 Steven Dake proposed openstack/kolla: Add OSIC Scale Testing Documentation https://review.openstack.org/352101 01:06:46 sdake_: here 01:06:58 sdake: I hope that you can leave a brief comment on implementation direction of bp/ansible-specific-task-become implementation @ https://review.openstack.org/#/c/358539/7 when you have spare time 01:07:06 sdake_, 01:07:09 Daviey - it appears we have run out of forwrad movement 01:07:40 Daviey i'm not sure if Jeffrey4l_ can sort out deploying centos or not 01:07:50 hmm 01:07:53 if not, your welome to use the cluster for a full ceph test 01:08:02 i think we need pbourke_ and inc0 01:08:07 sdake_: What is the issue with centos? 01:08:07 they know how to do this 01:08:17 Daviey - not sure how to deploy it to our 130 nodes 01:08:33 sdake_: Are we using bifrost or still cobbler? 01:08:34 britthou_ was working on it 01:08:38 cobble 01:09:03 (I hate cobbler) 01:09:42 part of the issue is it is in an lxc thing 01:09:46 which i don't know much about ;) 01:10:13 lets wait until Jeffrey4l_ gts back from breakfast and see if he can sort it out 01:10:28 if not, we can setup your scenario as #11 01:10:43 since i need cento to proceed 01:15:07 sdake_: well, it is 2:15am here. I'm reluctant to start something i can't reasonably finish before i need to sleep 01:16:40 sdake_ you have a link for this bifrost thing you been talking about? 01:18:16 Pavo: bifrost is just ironic, but ansibleised 01:18:23 and standlone 01:18:35 hmmm any info on it? 01:19:33 Pavo: http://docs.openstack.org/developer/bifrost/ 01:21:41 so this is a way to basically net install OS on bare metal using ironic from Openstack without making apart of resources in openstack? 01:28:23 Pavo: yeah, ironic standaone 01:28:23 Pavo: yes. if I understand correctly this does not necessarily deploy Ironic with the OpenStack deploy. It just means that if you want to deploy OpenStack using Ironic you get a chicken-and-egg problem trying to use an existing Ironic install. Hence Bifrost 01:29:12 correction *this does not necessarily _mean you will not_ deploy Ironic 01:30:06 ok I can see its use 01:31:13 Pavo: it does make some tradeoffs that require, for example, different methods of security. for example the API has much simpler (if any) auth applied to it, because Keystone isn't there to ask 01:31:30 pavo yes that is correct 01:31:38 Daviey wront file 01:31:43 moemnt reconnecting to vpn 01:32:35 could see the usefulness to be able to login into alot of bare metal machines using the same keypair 01:33:05 Pavo: that is the least of the benefit! 01:33:06 Ryan Hallisey proposed openstack/kolla-kubernetes: Update the kubernetes aio guide https://review.openstack.org/361787 01:33:15 Daviey is that you on irc 01:33:17 i am back. 01:33:27 sdake: yes 01:33:37 rather on tmux 01:33:40 sound slike both of you are back 01:33:53 so here is the deal, i want to see if jeffrey can get a node deployed with centos 01:34:01 if he can, lets proceed down centos path 01:34:08 the scenario tests take about 6 hours to run Daviey 01:34:09 centos is awesome 01:34:11 lol 01:34:25 thats what I use on stable 01:34:26 let me check the osic deploy doc. 01:34:59 i was trying to texxt inc0 01:35:06 but he appears mia and I don't hve his cell # 01:35:14 so... how does kubernetes work with kolla? 01:35:30 pavo - we dont hae a cocrete answer to that yet 01:35:40 Pavo kolla-kubernetes 1.0.0 = occata at best 01:35:45 mosst likley p cycle 01:35:49 I haven't really understood what kubernetes is anyways lol 01:36:01 only thing I see from kubernetes is self healing 01:36:03 pavo just stick with ansible version of kolla for now 01:36:03 sdake: hmm.. so what is happening ? 01:36:07 pavo that is right 01:36:16 sdake: should i proceed or hold fire? 01:36:18 Daviey - we are redeploying the bare metal with centos if its possible 01:36:32 Daviey Jeffrey4l_ is going to deploy only one node hopefully 01:36:39 if it doesn't work you can do your test case 01:36:48 sdake: Okay, works for me 01:36:50 Ross Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 01:36:51 i'd recommend doing the fio test scenario that jeffrey ha sorted out 01:37:09 so with self healing if a container crashes or freaks out it will bring up another container to replace it without losing any data, is that correct? 01:37:11 sdake: can you or Jeffrey4l_ provide an update for my morning, to know what i can do tomorrow? 01:37:24 sdake: I'm happy to try my scenario on centos or ubuntu 01:37:29 Daviey its your end of day? 01:37:51 sdake: it is 2:37am, so yeah. 01:37:59 Daviey i think we will kknwo in 1-2 hours if Jeffrey4l_ is able to deploy cenots 01:38:03 Daviey roger - feel the pain mysself 01:38:14 i've been up for what seems like 6 days straight 01:38:40 sdake: This isn't 'our' cobbler is it? 01:38:49 our as in kolla's? 01:38:50 sdake: this was provided by osic already, right? 01:39:00 yes its the osic cobbler 01:39:22 good. So hopefully they already have a functional centos config 01:39:29 they dont 01:39:31 otherwise, 1-2 hours is probably optimistic 01:39:43 britthou_ worked on it or 3-4 hours already 01:39:51 ok, cool 01:40:45 don't redeploy the deploy node Jeffrey4l_ 01:40:47 plz :) 01:41:04 sdake, yes. 01:41:15 will not touch that one 01:44:11 Jeffrey4l_ a preseed is an ubuntu thign i think 01:44:16 i think we need a kickstart or something 01:44:24 sdake, yes. 01:45:03 sdake: but for cobbler, it is basically the same hook 01:45:13 but not familiar with how cobbler profile works. need check the code and some doc 01:45:27 Jeffrey4l_: don't read cobbler code, it will make you cry 01:45:27 Daviey - do you create a preseed for centos then? 01:45:40 sdake: well yeah 01:45:41 or a kickstart 01:45:50 need a ks file 01:46:29 Have we imported the iso? 01:46:37 or kernel? 01:48:04 iso is imported 01:48:41 iso is in the filesystem atleast 01:48:45 i dont know what importing means 01:49:26 sdake: cobbler doesn't know about it 01:49:33 sdake: cobbler only knows about ubuntu here? 01:50:40 right 01:52:34 (i hate cobbler) 01:55:43 Daviey dns i sbust on these machines i htink 01:55:47 you may have ot setup resolv.conf 01:55:50 to 8.8.8.8 01:58:42 whisky tango foxtrot 02:03:29 victory 02:03:30 nothing like a reboot to fix things :) 02:05:03 i will try my fio test now. 02:05:29 wonder what that hp raid is about 02:05:59 Jeffrey4l_ not sure if th ecluster is operational atm - but your welcome to try while daviey sorts out a centos deploy 02:06:38 tried boot a vm, it works. 02:06:59 sdake / Jeffrey4l_: hmm, we need a cobbler kickstart file now 02:07:15 the legacy.ks almost certainly will not work ootb 02:07:32 is there somewehre we can get one 02:07:39 or is it roll our own? 02:08:05 sdake: Would be good if OSIC already had one 02:08:18 i think thta is us :) 02:08:20 (i'm surprised there isn't one that is known to work) 02:08:37 Jeffrey4l_ did you see anything in the docs about centos? 02:08:45 no. 02:09:10 sdake, Daviey this is the one i used for pxe http://paste.openstack.org/show/564449/ 02:09:13 https://github.com/os-cloud/osic-ref-impl/blob/master/bare_metal_provisioning.md 02:09:16 may be help 02:09:41 just install the os and setup a ssh public key. 02:10:29 That works 02:10:35 Jeffrey4l_: do you want to run with that? 02:10:54 hmm, url looks bad tho? 02:11:02 Daviey, yep. 02:11:18 it should work. Just provide what can I help. 02:11:25 no familiar with cobbler... 02:11:34 The url should be configuration in cobbler? 02:12:05 see? there is a $tree variable. 02:12:08 Jeffrey4l_: well it is just kickstart... that also supports templates 02:12:10 and snippets 02:12:34 yep 02:14:09 Jeffrey4l_: the password variable comes from /etc/cobbler/settings file 02:14:24 and other $variables 02:14:45 Daviey if we can et centos on there and accessible via password login of some type, we ccan handle the rest of the setup i think 02:14:59 but it woudl be handy if the network got automatically steup ;) 02:15:03 sdake: well lets just try legacy and see what happens 02:15:05 we have ilo 02:15:17 doing all nodes? 02:15:23 lets do 1 first 02:15:29 agree 02:15:30 i dont know cobbler 02:15:35 so dont know how it works - haveo nly used the web ui 02:15:43 please do not touch the controller node. 02:15:46 now. 02:15:59 use 23 daviey 02:16:04 Jeffrey4l_ 23 is safe right? 02:16:46 yep 02:17:46 it is runing. just he p4p1 nic broken 02:19:39 24-26 are controller nodes Daviey 02:19:46 23 should be good to try out 02:29:44 sdake: In my commit https://review.openstack.org/#/c/358539/7 , Jeffrey and inc0 have some comments about my approach. Can you review that? 02:30:19 duonghq occupied atm 02:32:59 know ilo password? 02:33:49 root:calvincalvin 02:34:44 Lu lei proposed openstack/kolla: Use six.StringIO https://review.openstack.org/361797 02:35:45 sdake: can you look at the ilo console, i don't have java setup on my local machine 02:35:51 (and .net.. well Linux) 02:36:01 trying - failling 02:36:04 idont have .neteither 02:36:12 java luanch button not working 02:38:14 trying safari 02:38:17 needto update java 02:38:53 sdake: trying serial console 02:39:18 Daviey how do you know all this stuff :) 02:40:10 sdake: :) 02:40:33 the console on ucs gear is annoying to use as well 02:40:38 java crap 02:41:09 hmm. doesn't look like it net booted 02:41:47 wouoldn't appear so 02:42:00 sdake: If only we had some oracle people here to help with java 02:42:47 in cobbler dont you hveto tellt he machine to save and reboot orsomhting 02:42:59 i recall on the gui there was some magic button 02:43:22 sdake: i did, i thought 02:43:24 try this 02:44:48 esc-@? 02:44:53 sdake: cobbler sync, copies all the files around 02:45:03 thats it sync 02:45:08 hang on, i set one time boot from net 02:45:33 here we go 02:45:59 nice 02:46:10 now no console ;( 02:46:41 That you are seeing is because i tail'd -F cobbler log file server side 02:47:02 but yeah, seems centos doesn't export the serial console by default 02:47:48 so the bad news is that it is unlikely the preseed is working 02:48:13 so the kernel command line probably needs help to export the tty over serial console 02:48:32 or.. get ilo / java working 02:48:49 let me try new java moment 02:50:03 sdake: I should really go to bed soon, it is silly o'clock 02:51:49 sdake: Okay, see the command there to set it to boot from pxe? 02:56:32 sdake / Jeffrey4l_: Okay, i need to head to bed. If someone else can take over this, it would be good 02:56:36 let me know how you get on. x 02:56:58 OK. 02:57:16 how do i get the console again? 02:57:29 and trigger a reboot? 02:57:39 sdake: i sent it to you in pm 02:57:41 Daviey roger - understand need for sleep :) night 02:58:11 sol activate opens up a serial console? 02:58:13 chassis bootdev pxe # set boot from local 02:58:15 but it seems to not be ddoing so 02:58:23 sdake: it should be? 02:58:39 what should be 02:58:46 it should be opening a serial console 02:58:48 * sdake strangles inc0 02:58:57 mc reset warm # power reset 02:59:07 but why is there no bios booting? 02:59:29 sdake: we saw BIOS POST checks, no? 02:59:40 oh i thought you warm reset it 02:59:52 so we should see the bios again right? 03:00:00 if it is rebooted yeah 03:00:21 mc reset warm = reboot? 03:00:35 right 03:00:43 so you rebooted, and no serial console 03:00:49 hmm 03:05:57 sdake: power reset, works :) 03:06:45 right, really going now 03:07:21 Sridar Kandaswamy proposed openstack/kolla: Customizations for dind https://review.openstack.org/353203 03:09:31 Jeffrey4l_ how do I get to Daviey's tmux? 03:09:59 sdake, see my tmux 03:10:51 igot it thanks 03:11:04 you young whipersnappers and your tmux tools 03:11:13 and us old whippersnappers and our ipmi tools 03:11:23 rather old dogs 03:11:29 ;) 03:11:40 well i'm stuck ;) 03:11:48 java console not working 03:18:20 whatever we do, lets multiply it by 9.2 03:19:02 pbourke_ when your alive - need centos running on the gear if you can make ethat happen 03:19:05 britthou_ same story 03:19:12 daviey imported centos 03:19:20 I think the kickstart file needs some love but I am not sure 03:20:29 Jeffrey4l_ - do you plan to submit this fio test uptream to rally? 03:20:43 sdake, yes. 03:21:22 sdake, another test is `booting multi instances test` 03:21:41 well you hve th gear until centos is on it 03:21:44 whenever that is 03:21:59 i dont nwo how to get centos rolling 03:22:03 daviey imported the imge 03:22:07 and did some magic ipmi commands 03:22:35 where are we now? does the node boot from pxe? 03:23:14 not really 03:23:19 i dont now what its oding 03:23:26 because i can't get a console 03:23:56 i tried to get the console by java. bug failed, too. 03:24:12 we can get a console via ipmi 03:24:23 but the kickstart file needs attention to output to ttyS0 03:24:57 why? 03:25:03 is it required? 03:25:10 cobbler need get the console? 03:25:20 seeing the console woudl let us know if its working 03:25:29 so getting a console is next step 03:25:57 ok.. 03:26:21 do you know how to get back inside the lxc container? 03:26:27 yes. 03:26:41 can you join othe rtmux or busy with fio testing? 03:27:18 sdake, i am in the tmux 0 03:27:24 sdake, please try tmux att -t 0 03:34:00 we need to get the kickstart file to have console=ttyS0 in some way 03:34:32 sdake, pxe config file? 03:34:45 goes in kickstart 03:35:03 i think 03:35:07 maye its pxeconfig 03:46:45 Jeffrey4l_ i want to find the kickstart file under use 03:46:53 so we can have that console=ttys0 in there 03:48:05 sd i think we need change the pxe menu file directly. the ks control the final OS kernel parameter. 03:48:27 sdake, ctrl+b ] then you can scroll 03:48:37 i id that 03:48:38 wasn't working 03:51:16 Hui Kang proposed openstack/kolla: Add missing container image names to build config file https://review.openstack.org/356854 03:53:35 default_kickstart : /var/lib/cobbler/kickstarts/ubuntu-server.preseed 03:53:37 this is wrong 03:55:29 kernel_options : ksdevice=bootif lang= locale=en_US text interface=p1p1 kssendmac priority=critical 03:59:43 Jeffrey4l_ it hsould be spttting out post stuff now 04:03:43 Lu lei proposed openstack/kolla: Use six.StringIO https://review.openstack.org/361797 04:25:17 sdake: for scroll in tmux: C-b [ 04:44:10 Duong Ha-Quang proposed openstack/kolla: Fix Kibana image build error https://review.openstack.org/361833 04:44:50 Merged openstack/kolla: Customizations for Nova https://review.openstack.org/351676 04:46:23 Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Dockerfiles for footer related changes https://review.openstack.org/357746 04:58:35 Sridar Kandaswamy proposed openstack/kolla: Customizations for dind https://review.openstack.org/353203 05:03:29 coolsvap do you know anything about cobbler setup in osic? 05:03:41 sdake: no 05:04:23 but I know there are issues with rolling out centos using cobbler in osic 05:04:31 do not know the details 05:04:49 you found these issues from whom? 05:05:31 there was a discussion on channel at start between inc0, britt and paul 05:05:48 when we first rolled out ubuntu on all nodes 05:07:42 ya - centos is not preconfigured 05:15:58 bedtime 05:16:03 * sdake ptfos 05:16:23 bye sdake 05:31:29 Eduardo Gonzalez proposed openstack/kolla: Change source with dot at extend_start files https://review.openstack.org/359926 05:43:04 Eduardo Gonzalez proposed openstack/kolla: Remove RUN macro install from Kibana Dockerfile https://review.openstack.org/361846 06:11:46 OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/361865 06:22:09 morning 06:51:51 jenkins gates failed again, no idea 06:53:38 Tin Lam proposed openstack/kolla: Chage with_ loop variables syntax to ansible 2 https://review.openstack.org/361886 07:03:20 Matthew Taylor proposed openstack/kolla: Fix nova_ssh container shell to allow instance resizes. https://review.openstack.org/361891 07:13:47 Can someone else review this PS? https://review.openstack.org/#/c/361833/ 07:14:37 - -hello 07:14:49 zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 07:16:26 ｐｉｎｇEduardo Gonzalez 07:16:36 hi zhubingbing 07:16:37 ping Eduardo Gonzalez 07:16:40 hello 07:16:56 https://review.openstack.org/348388 07:17:23 i have fixed it , 07:17:36 can you help review it ? 07:24:56 zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 07:37:17 ping Hui Kang 07:38:11 Jeffrey Zhang proposed openstack/kolla: Fix the wrong install_packages usage in kibana Dockerfile https://review.openstack.org/361909 07:39:42 Jeffrey Zhang proposed openstack/kolla: Fix removing nova_libvirt container issue https://review.openstack.org/361728 07:46:03 Morning everybody. 07:47:38 morning 07:52:14 Merged openstack/kolla: Fix Kibana image build error https://review.openstack.org/361833 08:12:17 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 08:14:47 Hey guys, I'll have time to work on the whole telegraf + influxdb + grafana thing this week before I go on vacation. Would be awesome to have fast review cycles to get it merged this week :) Sorry for not being on it recently … was busy with customers and stuff ^^ 08:27:45 zhubingbing_ can you please check https://review.openstack.org/#/c/352120/2 and remove your -1 08:33:41 sure i will review it 08:37:59 Jeffrey Zhang proposed openstack/kolla: Fix removing nova_libvirt container issue https://review.openstack.org/361728 08:43:17 morning 08:46:24 morning pbourke_ 08:46:42 it seems like sdake would like osic reinstalled now with centos 08:47:26 im not sure how difficult that will be but will give it a go soon if anyone wants to help 08:48:38 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 08:53:21 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 08:56:55 Lu lei proposed openstack/kolla: Fix using filter() to meet python2,3 https://review.openstack.org/361950 08:58:08 Merged openstack/kolla-kubernetes: Cleanup tox.ini constraints https://review.openstack.org/360984 09:09:19 Prithiv proposed openstack/kolla: Dockerfiles for collectd Container https://review.openstack.org/358632 09:23:14 * Daviey checks in 09:30:04 pbourke: Hey. So last night i set up cobbler to support centos.. but i don't think the ks was suitable 09:30:16 Daviey: ok 09:30:18 pbourke: not sure if sdake did anything after i went to sleep 09:31:00 Daviey: i dont think so 09:31:40 Daviey: where is the kickstart 09:31:51 In cobbler 09:32:21 pbourke: cobbler system getgs --name=SERVERNAME 09:32:23 getks* 09:34:27 fair bit in there 09:36:58 pbourke: make sure you are looking at a node using profile centos., otherwise you are looking at a ubuntu preseed 09:51:55 hi coolsvap 09:51:56 Hi All 09:52:13 i was trying to do a sudo on nova_compute 09:52:15 https://review.openstack.org/#/c/355408/ 09:52:24 not sure if there is a default password 09:52:41 how can i get the password of the nova user in the nova_compute container? 09:57:22 Daviey: have you tried kickstarting a machine with this profile? 10:00:22 Merged openstack/kolla: Updated from global requirements https://review.openstack.org/361865 10:15:24 Merged openstack/kolla: Change source with dot at extend_start files https://review.openstack.org/359926 10:18:01 pbourke: ping 10:18:10 coolsvap: hieulq 10:18:14 coolsvap: *hi 10:18:15 :) 10:18:26 autocompleted by accident there 10:18:34 pbourke: regarding the footer review 10:19:01 though deprecated we need to keep the {{ include footer }} blocks for backward compat 10:19:27 and inc0 mentioned there should be no unnamed footer blocks 10:21:03 coolsvap: there needs to be unnamed ones in non base images 10:21:13 coolsvap: he's incorrect - how else do I unset the proxy for example 10:23:10 so it is needed for a change that can be applied across all images like e.g. unset proxy 10:23:23 pbourke: honestly, the proxy thing is a distraction.. why bother putting it into dockerfile templating, when you can just set and unset proxy settings in the shell? 10:23:39 so we have two different footer blocks 10:24:03 both unnamed 10:24:06 Daviey: there's other use cases, it's about setting something for the duration of the build 10:24:09 coolsvap: yes 10:24:23 Daviey: but not having it persist in the final images 10:24:41 pbourke: right... but the proxy example everyone bounces around is annoying. 10:25:03 pbourke: If we have better examples, we can solve it better :) 10:26:11 potentially it could be removed in favour of build-args 10:26:34 there probably should have been a spec for the customisations 10:26:38 pretty messy as is 10:27:08 is there a documentation about the proper use of the footers in the dockerfiles? 10:27:17 if not maybe we should first write the docs..? 10:27:22 berendt: I have a patchset up 10:27:30 berendt: I also just sent a link to the ML 10:27:47 let's review and merge the docs first and continue with the pending reviews afterwards 10:28:34 +1 10:28:44 Daviey: any idea how close the current centos ks is to working? 10:28:51 Daviey: has it being modified at all? 10:29:04 pbourke can you link the review here 10:29:13 https://review.openstack.org/#/c/361253/2/doc/CONTRIBUTING.rst 10:30:25 Paul Bourke proposed openstack/kolla: Add guidelines on adding a new service https://review.openstack.org/361253 10:33:12 pbourke I noted some syntax issues 10:35:32 Paul Bourke proposed openstack/kolla: Add guidelines on adding a new service https://review.openstack.org/361253 10:39:01 Christian Berendt proposed openstack/kolla: Fix using filter() to meet python2,3 https://review.openstack.org/361950 10:45:06 pbourke: well.. the installer starts and the kickstart is pulled in 10:45:33 Lu lei proposed openstack/kolla: Fix using filter() to meet python2,3 https://review.openstack.org/361950 10:45:34 i couldn't easiy use the iLo client last night.. (java bad version and .net, well linux desktop) 10:45:53 so i was using serial console, and centos wasn't setup to export the progress 10:46:15 so it just needs someone to look at the iLo vnc like interface and see where it is blocking 10:53:16 i'll try take a look 10:53:23 Daviey: which node? 11:03:30 hmm 11:05:43 pbourke: I was using 729583-comp-s3700-023.cloud7.osic.rackspace.com 11:05:47 as a test node 11:20:57 dupengfei proposed openstack/kolla: * Fix lost parameter --remote in generated start_ovsdb_server.sh https://review.openstack.org/362027 11:56:34 please review https://review.openstack.org/#/c/346449/19 :) 12:00:47 Lu lei proposed openstack/kolla: Fix oslo.i18n in kolla project https://review.openstack.org/355342 12:15:55 also need reviews for https://review.openstack.org/#/c/346448/ 12:32:54 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 12:33:41 Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 12:44:32 Lu lei proposed openstack/kolla: Fix oslo.i18n in kolla project https://review.openstack.org/355342 13:01:49 Sridar Kandaswamy proposed openstack/kolla: Customizations for dind https://review.openstack.org/353203 13:05:08 pbourke any luck with centos on osic? 13:05:34 sdake: looking at it breifly this morning with Daviey 13:05:44 sdake: need to figure out why the kickstart isn't working, I haven't had a chance to check yet 13:07:03 is britthou_ in? 13:07:23 if we ont have centos soon wewont have any centos at all 13:09:49 sdake: I'm distracted today.. so don't rely on me 13:10:00 I think TBH, the centos test isn't going to happen 13:10:17 Daviey well your right it isn't going to do itself.. 13:11:26 we have roughly 30 hours left of osic time - i had really expected centos to be installed when I woke up this morning 13:12:05 i wouold do it myself if I knew hwo, but I don't know how t odo so 13:12:50 Did anyone get onto the iLo display? 13:13:15 Jeffrey4l_ was unable to doso 13:13:24 unable?! 13:13:28 i triedturning on ttyS0 from kernel command line 13:13:47 Daviey the java didn't work for him 13:13:53 *sigh* 13:13:56 that ilo java thing is a big gPILE OF SHIT 13:14:17 console=ttyS0 had no effect 13:14:47 did 23 not build with centos last night? 13:17:28 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Getting NOVA pieces together to launch VM https://review.openstack.org/358911 13:18:41 serial console=tty0 console=ttyS0,115200n8 13:23:13 sdake: that would work on debian / ubuntu, no idea about centos 13:24:11 Jeffrey4l_ are you running a test atm? 13:24:54 sdake_, yep 13:25:05 Jeffrey4l_ when wil it complete 13:25:13 less than 30 mins 13:25:26 will there be rally data? 13:26:24 sorry? sdake_ 13:26:45 Jeffrey4l_ will there be rally data after the test 13:26:55 yes. 13:27:04 the io test reasult 13:27:18 ok i guess that can be scenario #11 13:28:15 how goes, folks? 13:28:35 ok. the result will be a single html file and a single json file. 13:28:45 irtermite ubuntu testing is aboutwrapped up - however, there is no cookbook for centos for osic 13:29:03 irtermite so we are making one it appears 13:29:40 or not making one depending on your interpretation 13:30:41 cookbook for osic? you mean for the osic environment? 13:30:47 I see... 13:30:53 irtermite i mean to launch centos on the nodes 13:31:04 ...just a reminder, tomorrow is the last day 13:31:08 got ya 13:31:10 i speent 4 hours jeriking aroudn with it last night before i hit hte wall 13:31:11 yes i know 13:31:27 re last day - tuesday morning, evening what time exacty? 13:32:03 Paul Bourke proposed openstack/kolla: Add a check before wiping Ceph disks in bootstrap https://review.openstack.org/337594 13:32:33 typically, it would be morning of due date (ie: midnight tonight) but, I may be able to stretch it into the business hours of the day 13:32:36 irtermite: Adding a few days would be great :) 13:33:05 Daviey: sorry, but that is not an option. You all already had an extra week from the start, and we have to get it rebuilt for the next group in line. 13:33:16 =( 13:33:57 irtermite: What do other users tend to use to deploy the OS? 13:33:58 sdake_: tomorrow afternoon is fine by me 13:34:24 Daviey: it all depends on the user and their need, why do you ask? do you need assistance with something? 13:34:40 yes getting centos to deploy would be fantastic 13:34:45 osic-prep doesn't include such support 13:34:49 irtermite: Well, so far all of the tests have been Ubuntu based 13:34:59 irtermite: we wanted to smash out a few centos tests 13:35:03 Paul Bourke proposed openstack/kolla: Add a check before wiping Ceph disks in bootstrap https://review.openstack.org/337594 13:35:10 personally, we tend to build a cobbler system on one of the nodes and use that to push all the systems 13:35:18 irtermite: right 13:35:30 irtermite the cobbler system you build (osic-prep) has no implementation for centos 13:35:32 irtermite: Do you have a centos kickstart template that is known to work? 13:35:39 I built 13:35:40 ? 13:35:48 we did not provide you with any such thing. 13:35:51 you/osic 13:36:10 someone provided us with osic-prep ;) 13:36:32 its an lxc container with cobbler buit in 13:36:38 sorry, we provide you with a blank slate. if there is anything installed on the environment that was one of the kolla members working on this project with you 13:36:55 pretty ure osic-prep comes from the directions 13:37:09 what directions? 13:37:22 Daviey where were those osic-prep instrutions? 13:37:30 github 13:37:42 daviey something more specific as in a link? 13:37:42 sdake_: https://github.com/os-cloud/osic-ref-impl/blob/master/bare_metal_provisioning.md 13:37:45 ah that 13:37:58 yes, that is what we use for cleaning up and re-building the environemnt 13:38:27 it doesn't build centos environments.... 13:38:34 it isn't necessarily something we 'provide' for the users; though we do share it from time to time. Michal probably shared that with you all 13:39:04 Daviey Jeffrey4l_ how do i connect to the osic prep container 13:39:45 sdake_, lxc-ls -f : show all lxc 13:39:50 correct. that is a rackspace tool that we use to clean up the environment. it wasn't intended for any kind of production or other os use. but, it would be cool to see it start to grow for others to use more regularly 13:39:54 sdake_: tmux 0 ? 13:40:04 sdake_, lxc-attach -n osic-prep : attach the lxc 13:40:17 I actually shared it with another team that was using ubuntu. and their feedback was that it was helpful 13:40:35 irtermite works well enough on ubuntu 13:40:42 no centos implementation in it at all 13:40:47 if you all do plan on adding centos support to it, I would certainly appreciate a pull request to help future users of the clusters! 13:40:55 no doubt 13:41:14 this is great feedback too 13:41:38 I can certainly see a need for a repo of tools for others that snag access 13:42:10 good morning 13:42:23 look who decided to join us 13:42:24 ;) 13:42:41 uh... 13:42:46 ok, hit me, wassup? 13:42:48 inc0 hope you had a nice peaceful weekend 13:43:10 inc0 need centos on the nodes 13:43:11 notta. just giving you crap, inc0 13:43:18 inc0: Hope you are well rested :) 13:43:34 ya i'm not well rested and in a really angry mood atm 13:43:39 sdake_: Hmm... we might run into problems with centos 13:43:45 (and might be the issues we saw) 13:43:53 which type of problems? 13:44:03 The ubuntu image is mofified to include i40e driver 13:44:08 I bet centos doesn't have this 13:44:11 the first step is getting a console log 13:44:18 sdake_: I believe you just need to create a new preseed/kickstart file and upload a centos image, then you should be good to go 13:44:21 i40e is not in kernel? 13:44:25 Daviey, nah, i40n comes with new kernel 13:44:28 i40e 13:44:30 irtermite eaier said then done 13:44:32 ok, cool 13:44:47 Someone needs to use windows to connect to the iLo 13:45:02 I have a meeting now, but if not - i will spin up a VM with windows on after 13:45:41 irtermite - yell at the hp dudes to fix their java implementation of console btw ;) 13:45:58 irtermite what htey have donesn't work on mac or linux in any browser 13:46:25 sdake_: and create centos drivers? yea... after I finish yelling at them for their raid firmware BS 13:46:48 sdake_: works perfectly fine on safari 13:46:50 nah this is an LO thing 13:46:55 yeah safari is ok 13:46:56 ilo thing 13:46:58 doesn't work for me 13:47:11 not the f5 firewlal, but the ilo console 13:47:40 yea, I know it's an ilo thing... was just saying I've been yelling at them about something else for a few months now 13:47:41 i worked on raid in my early career 13:47:54 my only recommendation would be use JBOD with software raid these days ;-) 13:48:06 sdake_: ilo on safari works for me. what is the trouble you are seeing 13:48:29 intel integrated raid - what a charlie foxtrot that program was 13:49:15 inc0: give this a try https://www.centosblog.com/centos-7-minimal-kickstart-file/ 13:49:51 uhh...that's gonna be fun 13:51:41 damn, mass bare metal provisioning is fubar, we need something bifrosty 13:54:56 Merged openstack/kolla: Add missing container image names to build config file https://review.openstack.org/356854 13:55:11 inc0 there are two tmux sessions 13:55:17 inc0 connect to mine pls 13:57:49 sdake i opened the console http://storage7.static.itmages.com/i/16/0829/h_1472479081_3201804_2be9cb4902.png 13:58:23 is that on node 23 Jeffrey4l_ ? 13:58:35 sdake, yep 13:58:41 did you just rboot it? 13:58:50 we need the os console 13:58:52 you can see the ip address. 13:59:01 sdake, i do nothing. just connect it. 13:59:18 whichmeans we need this in the boot command lien: serial console=tty0 console=ttyS0,115200n8 13:59:38 sdake, caffeine is yet to kick in 14:00:49 inc0 http://i2.kym-cdn.com/entries/icons/original/000/003/730/cocaine-bear.jpg 14:02:18 sdake, how did tests go btw? 14:05:31 inc0 testing is done on ubuntu 14:05:46 inc0 hwich is why we needed centos yesterday not today 14:06:05 note the kickstart file referenced in the osic config is not correct 14:06:30 and cobbler needs the kernel command line appended with 729517-comp-s3500-023.cloud7.osic.rackspace.com 14:06:37 rather that serial line I pastd above 14:07:40 thattook all of 3 minutes 14:07:45 now for serial console 14:15:38 back 14:16:15 ping Mathias 14:16:18 inc0: Jeffrey4l_: need a bit of help. In kube when I start instance, I do not see that security group associated with instance gets programmed on linux bridge where the instance's tap interface plugged. WHich process is responsible for programming security group, is it neutron openvswitch agent or nova libvirt? 14:16:39 sdake: sitrep? 14:16:59 Daviey we need to write teh pxeboot filee - how to do so? 14:17:20 https://review.openstack.org/#/c/346448/ 14:17:23 I will download it for debugging, if there is a problem I will be review 14:17:26 sbezverk_, should be the neutron openvswitch. 14:18:05 nova libvirt's security group only work with nova-network now. 14:18:39 why are we removing all of the cobbler systems? 14:18:59 sdake: generate the files for tftp? 14:19:03 sdake: cobbler sync 14:19:17 inc0 just run cobbler sync 14:19:30 inc0 rather then removing everything...........! 14:19:50 I can generate it back in 10s 14:20:26 ping egonzalez90 14:20:40 sdake: where have all the systems gone?! 14:21:09 inc0 removed them 14:21:11 no idea why 14:21:34 Did anyone look at the iLo web based console? 14:21:35 probablly needs more cocaine bear imo 14:21:44 Daviey the os output nothing 14:21:59 sdake: not the serial console, but the vga one? 14:22:09 Daviey - ya there is a screenshot from Jeffrey4l_ in scrollback 14:22:23 sdake: directly after pxe? 14:23:03 i see it 14:23:17 Jeffrey4l_: thank you.. will keep digging then in neutron openvswitch logs 14:23:32 the problem now is centos and kickstart configs are gone 14:23:47 what? 14:23:52 what is gone, everything is there. 14:24:18 i mean gone from the configuration of 23 14:24:39 both the image and kernel command lines need to be set for 23 14:24:44 and the kickstart file as well 14:24:58 sdake: no, the issue is that we are missing the systems 14:25:14 Daviey - inc0 removed all the systems... 14:25:23 Daviey we can dd them back easily 14:25:30 Daviey the only system we care about atm is 23 14:26:08 btw I removed all the *systems* not *profiles* 14:27:09 sdake, we have a call with att now 14:27:11 serial console=tty0 console=ttyS0,115200n8 14:31:16 rhallisey ping 14:37:13 sdake: I can access now the instance from outside :-) with floating ip 14:37:19 * Daviey needs to go again 14:37:48 sdake: found a bug in neutron which was preventing it 15:04:23 sbezverk_ hot 15:06:45 good morning everyone \o/. does anyone have an idea of something i could work on similar to the recent vmtp work? 15:07:12 ping Mathias 15:07:15 there ae probbaly some big tent projects which can be integrated 15:07:22 just a moment - let me see 15:07:50 srwilkers__ we generaly go from the user surveyt o pick out features 15:08:09 cool, good to know 15:08:29 also makes sense :D 15:08:30 https://www.openstack.org/assets/survey/April-2016-User-Survey-Report.pdf 15:08:47 tbh, I think there are better ways to learn than create new Dockerfile 15:08:57 but that depends on how your brain operates 15:09:06 I learn by trying stuff and fixing what's broken 15:10:33 srwilkers__ page 31 15:10:58 srwilkers__ first you will need to hve a thoorough understanding of how aio works - have you got an aio deploy oging? 15:11:20 yeah 15:11:42 then deploy ha multinode;) 15:12:35 page 33 15:12:58 rally seems similar in nature to vmtp 15:13:03 and has alot of community interest 15:13:24 okay cool 15:13:33 thanks guys :D 15:13:40 sahara trove 15:13:46 i think there isn't implementations for these yet 15:13:48 but htey maybe in queue 15:13:59 (the review quue) 15:14:10 ill take a gander at those 15:14:28 make sur eto check that your not duplicating work already in flight 15:14:37 i am pretty sur enobody is working on rally 15:14:39 yeah, definitely 15:14:45 not sure about the rest on page 33 - i think most of those are in progress 15:16:46 afk - relocationg:) 15:18:32 sdake, inc0, fyi, the test is done. you can start to re-install the OS 15:19:00 srs- read https://review.openstack.org/#/c/361253/2/doc/CONTRIBUTING.rst 15:19:02 Jeffrey4l_ how many iops we get? 15:19:02 Jeffrey4l_ can you amil me the html file of results ? 15:19:10 stdake@cisco.ccom 15:19:37 i will push them to the PS. 15:19:51 i will send it to u later. 15:19:54 i need it now though 15:20:02 where is it on disk - i'llcopy it off 15:20:03 let's me check the iops. 15:20:05 np 15:20:22 scp the html to mybox if you can? 15:21:24 ok 15:21:56 sdake_, which folder? 15:24:06 sdake_, i put them here /home/jeffrey/testresult 15:25:07 sdake, i put them here /home/jeffrey/testresult 15:26:02 Jeffrey4l_ what is the delta betwen the boot files? (.1, .2) 15:26:52 i tested twice using the the same rally tasks. i re-deployed openstack before .2. 15:27:00 .1 has much failed tasks. 15:27:10 .2 is more better. 15:29:16 2 doesntshow iops just cumulative run time 15:29:39 nova.boot test just boot instance test. 15:29:57 for the io test. check the secnario11 folder 15:30:14 i made two test: boot multi instances and io test 15:30:18 i did - html doesn't have iops printed out :) 15:30:44 Jeffrey4l_ ca nyou tell me how to interepet that data from scenario11 15:30:47 sdake, check the 15:30:47 Scenario Data tab 15:32:14 Jeffrey4l_ i am there now 15:32:28 5796 iops? 15:32:38 is that the average ? 15:32:40 sdake, the usage full data is bw and iops. 15:32:52 sdake, which task are u checking? 15:33:02 boot_runcommand_delete [?] 15:33:11 VMTasks.boot_runcommand_delete (164.512s) 15:33:18 which i assume is an average 15:33:38 it do not calculate the sum iops. 15:34:05 we should sum the ios when neccessary. 15:35:28 sdake, but you are checking the 1 node test. so the iops is the final value: 5795 15:36:00 for randwrite + 4K + 4 numjob + direct io + libaio engine 15:36:59 other test may be using multi node, which we need sum all iops on each node. 15:37:57 Jeffrey4l_ so tests 2-24 are multinode, tst 1 is single enode? 15:38:33 btw master upgrade is broken 15:38:38 if anyone cares to fix it ;-) 15:39:00 sdake, no. I will such a test http://paste.openstack.org/show/564594/ 15:39:40 there should be 4 1 node test. for each 4k rand write, 4k rand read, 4m rand write and 4m rand read. 15:39:56 sdake, bug link? 15:40:01 Jeffrey4l_ didn't file bug 15:40:14 Jeffrey4l_ noticed last night when trying to time upgrade for master 15:40:20 what about it is ? 15:40:41 upgrade from mitaka to master does not work - related to the bond interfaces not being fund 15:40:57 ansible variable error of some kind 15:41:20 sdake, could u or any other file a bug with some log? 15:41:49 Jeffrey4l_ yes - if your done testing i'll do now 15:42:00 are ou running master or mitaka? 15:42:34 no now. I need to bed, soon. I can test it tomorrow :) 15:42:58 Jeffrey4l_ we are redeplooying the cluster with centos 15:43:05 Jeffrey4l_ so if you want any tests nows the time to do em 15:43:33 The total iops > 3w for 4k read write 15:43:35 i'll run a master upgrade and get a traceback 15:43:44 what is 3w? 15:44:06 io test. the 4k rand write iops arouch 3w 15:44:10 around. 15:44:19 what is w in that sentence? 15:44:21 you mean weeks? 15:44:38 when u will destroy the current os? 15:44:51 my bad.... the iops is around 30000 15:44:58 as soon as we can figure out how to get centos installed on on enode 15:45:06 but we may not be able to figure that out 15:45:11 its not looking promising 15:45:22 since i dont see it happening on the gear 15:45:37 Jeffrey4l_ how did you calculate IOPS? 15:45:41 OK. i will try another test. 15:45:52 sdake, check this boot_runcommand_delete [6] in the html 15:46:02 open the scenario data tab. 15:46:32 i've got that open 15:46:37 check the fio: write graphic. click the iops on the top 15:46:43 let is show the iops only. 15:46:59 then you will found all the iops is above 1000 15:47:14 so 32 nodes * 1000 > 30000 15:48:14 it was run on 32 nodes? 15:48:29 yes. see the input task tab sdake 15:48:40 concurrency: 32 times: 32 15:48:58 cool 15:49:07 nice scenario teset jejff 15:49:10 Jeffrey4l_ 15:49:22 but two nodes are failed. so should be 30 nodes * 10000 15:49:26 * 1000 15:49:30 so 30k 15:49:39 yep. 15:49:44 hotness 15:49:50 that is smoking fast 15:50:17 are u saying it is good? or bad? 15:50:22 good 15:50:25 :) 15:50:59 i will continue the test: deploy ceph-osd on all node. then we will get a huge cluster of ceph. 15:51:04 then test the performance. 15:51:18 Jeffrey4l_ daviey also wanted to do this same thing 15:51:18 until u guys can deploy the centos . 15:51:42 sdake, same thing mean? ceph-osd? 15:51:49 same scenario test 15:51:52 lots of osds 15:52:11 just deploy on compute - not control 15:52:15 OK. I can deploy it now. 15:52:17 on compute and storage 15:52:17 yes. 15:52:21 got it. 15:52:32 OK. start to deploy 15:52:37 note we have one less machine 15:52:39 one of them died 15:52:44 so I took it out of the inventoyr file 15:53:02 I marked that. in the osic_playbook/inventory file 15:53:15 hmm. which one? from when? 15:54:06 i dont remember 15:54:08 140 i think 15:54:13 but it culd lhave been 40 15:54:22 dunno Jeffrey4l_ - had 8 hours of sleep in last 5 days 15:54:28 i am beat 15:54:42 OK should be it. i have comment it out. 15:55:02 sdake, take care :) 15:55:14 i'm doing what I can :) 15:55:40 Jeffrey4l_: please do carry on 15:55:54 i already made an inventory that folded storage into compute 15:57:14 fold storage into compute? or vice versa? Daviey 15:57:46 Daviey, start to destroy current openstack. OK? 15:59:28 Jeffrey4l_ yes go for it 15:59:39 Jeffrey4l_ quick q - how would i run that fio test on my own hardwre? 15:59:51 Jeffrey4l_ i'm interested in a bare metal baseline 16:00:16 sdake, run fio directly? 16:00:19 yup 16:00:22 you can get the fio command from the html 16:00:28 single node 16:00:30 ok - just run tht then? 16:00:39 Runing fio by: ['fio', '-thread', '--output-format=json', '--group_reporting', '--filename', '/data', '--readwrite', 'randwrite', '--runtime', '120', '--size', '20G', '--iodepth', '64', '--ioengine', 'libaio', '--blocksize', '4k', '--numjobs', '4', '--name', 'fio test', '--direct', '1'] 16:00:41 like this. 16:00:43 yes. 16:01:02 just notice the --filename , you may need change the value. 16:01:35 install it from the epel-release directly. yum install fio 16:01:47 Jeffrey4l_: sure, if you want 16:02:08 Daviey, np. started to destroy and re-deploiy 16:02:30 Jeffrey4l_: diff inventory-ips-only inventory-ips-only-storage-colo 16:04:09 Daviey, you left the original storage nodes. 16:04:26 Jeffrey4l_: hmm? 16:04:31 Jeffrey4l_ - any thoughtso n this? http://paste.fedoraproject.org/416959/72486655/ -> http://paste.fedoraproject.org/416959/72486655 16:04:41 Jeffrey4l_: I moved all the storage nodes into compute, and then made storage == compute 16:04:48 Daviey, got it. 16:05:08 You are right. I am doing almost the same things. 16:05:23 cool 16:05:36 Jeffrey4l_ it appears fio isn't doing anything useful :) 16:05:59 sdake, yep. you can remove the --output-format=json paramater, to generate a readable report. 16:06:15 it exits in about .5 seconds 16:06:43 some wrong. 16:06:48 ya perm denied 16:07:00 got it with sudo ;) 16:07:21 yep. remove the --output-format=json , too. 16:07:28 yup i did 16:09:28 this is my bare metal disk: 16:09:29 write: io=39090MB, bw=333455KB/s, iops=83363, runt=120041msec 16:09:57 ssd 16:10:02 nvme ;) 16:10:17 lOl 16:10:30 going to try on the teset gear 16:12:18 write: io=2315.8MB, bw=162390KB/s, iops=40597, runt= 14598msec mine. ssd 16:32:22 Jeffrey4l_ where are the ssd's on 253? 16:35:53 sdake, no idea. :( 16:36:12 i need a baseline on the ssds for fio 16:36:28 is there one i can use? 16:38:54 you can use 0.23 for the test. 16:40:05 no we nuked 23 16:40:38 we are using 124 nodes, 2 are down that should leave 5 extra nodes that are around 16:40:53 have you started your test yet on the other nodes? 16:41:55 sdake, try this ssh root@172.21.0.66 16:42:37 that works thanks 16:43:02 in inventory file, it say: DNS broken for some reason 16:47:21 Jeffrey4l_ in all of our tests, /var/lib/docker is not on an ssd... 16:48:02 umm 16:48:09 sdake: I thought all disks were ssd on these? 16:48:17 i ont think /dev/sda is 16:48:19 i dont really know 16:48:21 perhaps it is 16:48:58 so here is ssd baseline in osic: 16:48:58 write: io=18417MB, bw=157153KB/s, iops=39288, runt=120004msec 16:49:09 my workstations do almost double iops :) 16:49:37 sdake, that's OK. we write data to ceph/ssd directly 16:49:58 Jeffrey4l_ i know but data volumes are not on ceph 16:50:09 and ddepoy would surelygo faster to ssd 16:50:28 butI think sda is an ssd - going to find out 16:50:35 sdake, few thing are wrote in the volumes. 16:50:59 [ 2.266417] sd 0:0:0:0: [sda] 1562758832 512-byte logical blocks: (800 GB/745 GiB) 16:51:18 all the diks are the same type 16:51:30 inc0, just mount a large volume to /var/lib/docker 16:51:49 yeah 16:52:06 we now have 600+gigs for elastic \o/ 16:52:28 lol, so. two weeks :D 16:52:41 i disabled the debug log. so the log will not grow that much. 16:52:46 i looekd into gcing es 16:52:48 its super easy 16:53:13 Jeffrey4l_ debug was on this entire time? 16:53:28 sdake, seems yes. :( 16:53:46 I disabled it today. 16:53:47 thats ok - most people want to run their cluster with debug 16:57:23 how many more days left with the cluster? 16:57:25 2 or 3 right? 16:58:08 I haven't seen ceph with that many OSDs before 16:58:11 gonna be interesting 16:59:49 1 16:59:52 1 17:00:15 more osd means more fast ( i think) 17:00:25 not necessarly 17:00:32 mons can be overwhelmed 17:00:43 also wont really affect speed 17:00:44 maybe . 17:01:01 network for replication can become issue as well, if we save a lot of data there 17:01:09 this will be huge cluster as ceph goes 17:01:31 well, biggest one I've ever seen 17:01:46 me too. :) 17:02:28 3GHZ 17:02:33 3 mon + 110 * 10 nodes 17:02:42 hey lyncos :) 17:02:46 Hey guys 17:02:47 3 mon + 110 * 10 disks ( 1100+ osd ) 17:02:52 1k+ OSDs 17:02:54 gonna be fun 17:02:56 You all miss me 17:03:27 I want to get ceph -s after this 17:04:15 i want to get a tesla s after this 17:04:26 oh wait.... completely different set of desires 17:04:26 we need to up placement groups....a lot 17:04:51 inc0, yes. 17:05:13 Jeffrey4l_, clock skew detected 17:05:26 did you install ntp? 17:05:34 inc0, no.. 17:05:47 let's install. 17:11:12 inc0, we need 40K pg in sum 17:12:10 nearest power of 2 17:12:12 of this 17:13:24 i got this 17:13:25 PG num = (OSDs * 100/max_rep_count) 17:13:30 PG per pool = (PG num)/NB_pools 17:13:31 kk 17:13:45 so the pg num should be 1200 * 100 /3 = 40K 17:13:58 this thrown errors 17:14:29 ahh you added different pools? 17:14:43 inc0, yeah. 17:14:47 ok 17:14:53 inc0, increate little by little 17:15:36 so are we giving up on centos then? 17:15:46 so it seems 17:15:46 set vms to 40K? no need touch other pools. 17:16:00 yeah go ahead 17:16:04 ok 17:16:49 inc0 i'd really like a centos baseline 17:16:59 atleast one test case 17:17:15 is it not possiblee to eeploy centos within the neext 12 hours? 17:19:41 inc0 ^^ 17:27:48 inc0 tell me now - so i can either stay and help or go back to bed 17:29:55 sdake, I don't think it's feasable 17:30:08 I would need to learn this stuff 17:30:14 and I'm not good at centos 17:31:14 are there instructions? I'll help deploy centos 17:32:03 instructions 17:32:04 lol 17:32:28 if there were instructions it woud already be done 17:33:30 the only person i know that knows this stuff well is britthou_ 17:33:57 this is the sort of thing that turns into one rabbit hole after the other. If there is some way to segment (even by manual firewalling or etc.) 1 or 2 nodes and have a few people work on it then you could keep at the ceph stuff in the mean time. 17:34:09 That or accept the risk that you put 14 hours into it and get nothing further done on the cluster 17:34:38 we have a node segmented to deploy centos to 17:34:45 oh :D 17:34:47 so there is no risk of not getting anything furhter done 17:35:09 britthou_: ping 17:35:47 sdake: "node segmented to deploy centos to" means there is a PXE server to load centos onto all the target nodes ? 17:36:03 vhosakot no it means we have one node (0.23) to load centos onto 17:36:11 via cobbler 17:36:12 ah ok 17:36:16 then wecan expand to every otherndoe 17:36:22 if we can get one node working 17:36:45 ah got it... it is the cobbler node 17:36:51 hi inc0,, r u there 17:37:23 the fio test is started and will be finished in 2 hours 17:37:50 if we plan to do centos, I can help whoever is doing it 17:38:28 vhosakot inc0 seems to know th most about cobbler 17:38:35 vhosakot but hwere we er jammed up is in a kickstart file I think 17:38:37 but I'm not relaly sure 17:38:48 it seems like everyonegve up - nobody to lead the rest of the effort 17:39:04 I've got nothing left - selpt at most 8 hours since wednesday 17:39:18 AND Its mONDAY 17:39:43 so if someone sets it up, grand 17:39:44 hi inc0. there 17:39:51 when i wake up i'll do a baseline 17:40:08 sdake: cool.. I'll check with inc0 17:40:31 vhosakot inc0 gave up - look at britthou_ - he cn get you going 17:41:07 sdake: cool... I'll check with britthou_.. about cobbler kickstarts and PXE... 17:42:06 sdake: when do you plan to finish the baseline on Ubuntu ? when is the cluster free for cenot re-image? 17:42:50 sdake: also, until when do we have the cluster for us ? end of day tomorrow, or end of day 31st ? 17:44:33 sdake, did you look at pxe we produced? 17:46:11 inc0: do we need to create the centos kickstarts for cobbler ? 17:47:04 vhosakot, yeah, I did something 17:47:14 like copy pasted from inet 17:47:23 inc0: where is it.. I dont see it in tmux.. is it on the deploy node 002 ? 17:47:27 but as I said - I'm not centos guy, and not really a cobbler guy either 17:47:45 join session 9 in tmux 17:48:00 ah, session 9.. ok 17:48:12 let me know when you're in 17:48:31 yep.. 2 mins 17:57:02 inc0: I'm in 17:58:27 hi inc0, where r u 18:00:15 I'm here now wally1 18:00:23 but a bit busy atm 18:05:39 need go bed. bye guys. 18:05:39 inc0: Jeffrey4l_ showed me the config 18:05:44 Jeffrey4l_: good night 18:05:57 inc0: how can I access the cobbler node (0.32) ? 18:06:02 0.23 I meant 18:06:52 centos is pretty easy, never used cobbler though, but I do use ks files to install centos using dnsmaq 18:07:36 thats all we use is centos 18:11:57 .... hi inc0 there? 18:12:23 wally1 what do you need 18:12:34 inc0: thanks for the pointer.. I'll load it onto 0.23 and let you know 18:12:49 0.23 is where centos should be loaded to 18:13:19 last friday, I asked inc0 help for deploy the kolla all in one, I repare a server for him, he said he can helo me to instlal 18:13:32 sdake: yes, I'll load centos on 0.23 as a test.. and if it works, we can use the same ks to load centos on all the 130 nodes.. this is what inc0 told 18:13:33 since I cannot get my workinng, 18:13:42 the VM cannnot go out 18:13:49 sdake, any chances you could try to help wally in my stead? 18:13:51 vhosakot cool soundsgood 18:13:57 I'll stick with osic cluster 18:13:59 inc0 what do you think I'm doing :) 18:14:10 i culdn't sleep becausei just woke up 18:14:14 you're the man sdake 18:14:17 need to take some mad sleep meds :) 18:14:18 inc0: I need the interface connected to the cobbler server, MAC address, IP address for the "cobbler system edit" command 18:14:31 wally1 tell me what type of serveroy uhae 18:14:40 vhosakot, once you setup the profile 18:14:42 ubunt 1404 servers 18:14:48 i have the server reade 18:14:49 more then one? 18:14:50 let me know, I'll show you how to cobller it up 18:14:57 all in one 18:15:04 i havef the server reazdy with public ip 18:15:05 multiple nics? 18:15:09 inc0: profile ? meaning kickstart, right ? 18:15:14 i have 2 config 18:15:40 inc0: ah got it.. thanks 18:15:41 you mean 2 nics? 18:15:47 yeah, 18:16:05 ok - follow quick start guide for evaluation/deployment 18:16:12 you have been through it once 18:16:19 should be straightforward to run thorugh again 18:16:22 inc0: the profile points to ubuntu seed... I'll first update that 18:16:27 i have install so many times, but nothing worikking 18:16:28 yeah 18:16:32 the VM canot go out side 18:16:42 ya - tht is because your vm is not setup properly 18:16:48 kolla can definately network outside 18:16:50 that is why in0 said he can help me to depploy one for me 18:16:54 :( 18:17:18 do u have time to insttall one for me? 18:17:21 so go ahead and get to the point where you would deploy on the baremetal from the quickstart 18:17:29 yup i hae time 18:17:35 but wouldn't you rather do it? 18:17:43 inc0: I dont see any centos preseends in /opt/osic/preseeds 18:17:57 i have following everything step in the guide 18:18:08 wally1 on the baremetal node? 18:18:09 but no idea why not working 18:18:12 yeah 18:18:19 physical server 18:18:21 dell server 18:18:22 what specifically isn't working? 18:18:28 does nova list work? 18:18:44 the VM cannnot reacch outside, cannot ping and cannot ssh into thte VM 18:18:53 yeak 18:18:58 wally1 did you run init-runonce? 18:18:59 nova agent list working 18:19:03 yeah, 18:19:04 i did 18:19:13 ok - init-runonce is for my personal home lab environment 18:19:22 init-runoonce is not a generic tool for end user usage 18:19:28 i change the ip in the innit-runonce 18:19:30 its meant to serve as a guide to get going 18:19:59 that is why, can you help me to deploy all in one on my server, than i can take a look 18:20:02 if you haveg time, 18:20:07 only 15 - 20 mins right 18:20:21 yup - hae you gone throug hthe uickstart on your bare metal? 18:20:24 I don't think I can re-use an ubuntu seed for centos.. we'll need a new centos seed with all the YUM packages.. the ubuntu seed in debian-based 18:20:45 yeah, I have followdd the link and do many times 18:20:58 vhosakot i was pretty sure a preseed was an ubuntu/debian only thing but i could be wrong 18:21:12 wally1 are you able to connect to webex? 18:22:03 sdake: ah ok... seed is ubuntu's kickstart? inc0 showed me the centos ks.. 18:22:13 yup 18:22:28 cool.. got it.. I'll point the cobbler profile to the centos ks 18:25:21 uhm, what is the situation? 18:25:34 We had a kickstart for centos already, but it wasn't functional 18:25:37 Daviey - cluster running for 2 hours a massive ceph test 18:25:47 ceph io test 18:25:58 also trying to boot .23 18:25:58 sdake: yeah, saw that. looking forward to benchmarks 18:26:00 Daviey th ekickstart for .23 was not set 18:26:11 sdake: are we also loading it with sporadic nova vm's? 18:26:20 vhosakot you might have mor eluck if you turn on serial output 18:26:32 Daviey i dont know mechanics of the test case 18:26:42 sdake: serial output where ? how ? 18:26:51 vhosakot no idea 18:26:55 sdake: erm, there was a kickstart set.. that is what i set before i went to bed 18:27:08 Ryan Hallisey proposed openstack/kolla-kubernetes: Update the kubernetes aio guide https://review.openstack.org/361787 18:27:08 Ryan Hallisey proposed openstack/kolla-kubernetes: Add a script that will get the latest kubectl https://review.openstack.org/362312 18:27:13 https://cisco.webex.com/ciscosales/e.php?MTID=m659c63768c3e0e10eeed6121bc1add01 18:27:25 sdake: you mean output from the PXE port incase PXE loading crashes.. 18:27:28 wrong link 18:27:28 not even sure what that is 18:27:44 sdake: yes, we'll need serial access to debug boot crashes as the box wont even have an IP address to SSH into 18:27:46 vhosakot serial console=tty0 console=ttyS0,115200n8 18:27:58 vhosakot: we have IPMI serial console, and iLo VGA 18:28:00 vhosakot that needs to go in the kernel command line in some way 18:28:11 v1k0d3n, ^ just pushed some aio docs changes 18:28:25 those should be far more straightforward 18:28:28 although nobody but Jeffrey4l_ has been abble to get the ilo vgat o work 18:28:57 sdake: got it.. "serial console=tty0 console=ttyS0,115200n8" musy be added as kernel option when booting.. cool.. I'll add this in the centos ks 18:29:06 thanks Daviey 18:29:47 sdake: the cobbler profile already has it --> Kernel Options : {'serial': '~', 'console': ['tty0', 'ttyS0,115200n8']} 18:29:59 ya i set it 18:30:04 not sure if i st it correctly 18:31:10 Daviey the kickstart was instead pointing to a prseed 18:31:32 anyway inc0 deleted all the sytems and strted with uust 0.23 18:33:08 wally1 -> see privat emessage with webex location 18:37:47 sdake: it wasn't... 18:38:10 i checked it was kickstart before i went to bed. :/ 18:38:12 oh well 18:39:33 hi sdake 18:39:38 sorry, just went out 18:39:48 how do see private message with webex? 18:40:02 on irc i sent you a privat emessage 18:40:09 webex is a paltform where i can see your screen 18:40:11 on your server 18:40:34 oh, do i need to install this? 18:41:50 never use it before 18:44:43 I see we can use "cobbler profile edit --kickstart=" to update the ubuntu seed to centos ks.. I'll try it now 18:46:22 wally1 just click the link 18:46:37 wally1 http://cisco.webex.com/join/stdake 18:47:44 ok let me get my headset 18:47:45 moment 18:47:45 yeah, just did 18:48:18 did u see my screen 18:48:58 yup 18:49:05 really 18:49:36 so ... what to do now 18:49:40 wally1 click "Quickstart" tab and then click "connect to audio" 18:49:53 wally1 i just got my headset - hang tight - you need to get on audio so i can speak with you 18:50:13 i have my headset 18:50:15 too 18:50:26 where is teh quickstart tab 18:50:58 on the top o f the screen 18:51:01 try stop sharing 18:51:23 or exit full screen mode 18:51:46 or exit entirely -don't share screen and click "connect to audio" 18:52:45 wally big blue button on left of screen says "connect to audio" 18:53:28 i use my compute rfor this function 18:53:28 ory ouc an dial in 19:00:37 Ryan Hallisey proposed openstack/kolla-kubernetes: Add a script that will get the latest kubectl https://review.openstack.org/362312 19:00:37 Ryan Hallisey proposed openstack/kolla-kubernetes: Update the kubernetes aio guide https://review.openstack.org/361787 19:04:15 Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 19:06:04 Ryan Hallisey proposed openstack/kolla-kubernetes: Update the quickstart https://review.openstack.org/362325 19:13:57 Ryan Hallisey proposed openstack/kolla-kubernetes: Update the quickstart https://review.openstack.org/362325 19:16:46 hi sdave thanks for your help 19:17:02 i will re-configure my network 19:18:36 but one last question, if i change my kolla_internal_vip_address: to be "192.168.X.XX", then I cannot loign to my horizon dashboard from outside. 19:18:53 but you said I can set the external ip for the horizion dashboard? right 19:19:03 where in the golabl;ymal file 19:24:16 wally1 its clled kolla_external_vip_address 19:29:11 inc0: ping 19:29:16 I'm here 19:29:21 I ran "cobbler profile edit --name Centos-7-x86_64 --kickstart=/var/lib/cobbler/kickstarts/centos7.ks" 19:29:29 but only one ks got changed 19:29:35 I still see some ubuntu seeds 19:30:16 I see we can do "cobbler profile getks" to get the centos ks and load it onto 0.23 19:30:24 vhosakot, tbh...I dont know how to cobbler:/ 19:30:36 Merged openstack/kolla: Fix removing nova_libvirt container issue https://review.openstack.org/361728 19:31:11 inc0: how was the Centos-7-x86_64 profile created in the first place ? was it copied from the ubuntu profile using "cobbler profile copy" ? 19:31:28 tbh I don't know 19:31:30 wasn't me 19:31:37 Daviey created that profile 19:32:29 sdake: cool.. Daviey: did you copy an Ubuntu profile and made the Centos-7-x86_64 profile.. or, you created Centos-7-x86_64 profile from scratch using "cobbler profile add" ? 19:32:43 vhosakot don't recall what ommand he used 19:33:02 Michal Jastrzebski (inc0) proposed openstack/kolla: Customization for base https://review.openstack.org/358042 19:33:02 cool 19:33:03 vhosakot: that is created automagically when you create a distro 19:33:22 Daviey: the Centos-7-x86_64 profile points to ubuntu seeds and I'm not able to change it 19:33:37 er 19:34:15 Daviey: http://paste.openstack.org/show/564642/.. see the line with <----- 19:34:54 I was able to change only one ks of the Centos-7-x86_64 profile using the command "cobbler profile edit --name Centos-7-x86_64 --kickstart=/var/lib/cobbler/kickstarts/centos7.ks" but not all ks 19:34:58 vhosakot: that is available profiles 19:35:39 Daviey: it is the report from just Centos-7-x86_64 , right ? I ran "cobbler profile report Centos-7-x86_64" 19:36:28 try with --name Centos-7-x86_64 19:37:18 sdake: yes.. passing --name shows the right output! looks like the centos profile does point to the centos ks.. 19:37:21 cobbler profile report --name Centos-7-x86_64 | grep -i kick 19:37:21 Kickstart : /var/lib/cobbler/kickstarts/centos7.ks 19:37:21 Kickstart Metadata : {} 19:37:49 what is the next step? "cobbler profile getks" to generate the centos ks ? 19:37:58 i need to feed my brain 19:37:58 i'll be back 19:38:07 I mean "cobbler profile getks --name Centos-7-x86_64" 19:38:13 sdake: cool... later 19:38:36 vhosakot: see the screen? 19:38:59 vhosakot: this is a distraction, you can see the kickstart there 19:39:02 That isn't the issue 19:39:19 Daviey: which screen? I'm in window 4 in tmux on the osic-prep machine 19:39:27 rhallisey: aio for kolla-kube? man...was just going to try and work on that :) 19:39:44 v1k0d3n, I had a patch up for it last night :) 19:39:56 just tweaked it a little this morning 19:40:24 v1k0d3n, see if you can get the vagrant stuff from kargo working 19:40:35 vhosakot: tmux session 0 19:40:36 It didn't work for me 19:41:06 awesome. 19:41:09 Daviey: I'm in tmux window 0 19:41:25 do you see me? I typed" #vhosakot here" 19:41:44 nope 19:41:57 vhosakot: are you on tmux session 0, window 0? 19:42:18 window 0 19:42:31 Daviey: what command should I use to go to tmux session 0? 19:42:45 ssh, tmux att -t 0 19:43:28 Daviey: I'm in 19:46:07 Michal Jastrzebski (inc0) proposed openstack/kolla: Architecture guide doc https://review.openstack.org/360796 19:52:48 vhosakot: are you conneted to iLo vga? 19:56:40 vhosakot: ?? 19:57:13 Daviey: yes, I'm in tmux session 0 19:58:04 hey all any results on ceph yet? curious if the osic hardware puts up some good numbers 19:59:52 vhosakot: hey, so... serial console is great... but you don't always get all messages - or the option to really play with the bios 20:00:00 VGA iLO is pretty good to use aswell 20:00:13 .net windows support works well.. Java somewhat. 20:00:25 Daviey: cool 20:00:27 But, i am a linux desktop person.. which really gives me the shitty end of the stick on these 20:01:18 vhosakot: so we have successful install 20:01:40 Daviey: so, once the boot process in the bottom pane finishes, will we see the login prompt? 20:02:38 Daviey: why is the test target machine called 0.23? it the IP it gets x.y.0.23 ? 20:03:58 Daviey: got it.. I see its IP is 172.21.0.23.. cool 20:04:08 right 20:04:11 looks like it booted but password is not working? 20:04:13 vhosakot: The password was badly set 20:04:20 someone set it to --iscrypted cobbler 20:04:27 which means cobbler is the result of the salting 20:04:36 (and i'm guessing that isn't true :) 20:04:38 Daviey: so, it means, it wont accept plaintext password from a keyboard ? 20:04:51 vhosakot: it means i have NFI what the password is 20:06:08 Daviey: ah ok :) 20:06:10 you guys should use the other nodes to crack it 20:06:36 can we change it to a good password, and re-run the ks again? 20:09:22 vhosakot: done 20:09:45 Daviey: cool.. we'll see how it boots this time.. 20:10:09 so, the cobbelr hirarchy is system --> profile --> ks --> image... is this right ? 20:10:53 vhosakot: think so 20:10:56 (I hate cobbler) 20:12:02 there is PXE and DHCP as well in that hierarchy somewhere 20:14:00 Merged openstack/kolla-kubernetes: Getting NOVA pieces together to launch VM https://review.openstack.org/358911 20:14:46 vhosakot: well yes, but that isn't inherent to cobbler 20:15:06 there is a dhcp server, which has dhcp option 66 pointing to the cobbler servers tftp server 20:15:25 in /var/lib/tftp/ there is the tftp files that the bios pulls down 20:15:32 Daviey: ah ok cool 20:15:34 they are created by cobbler 20:17:32 Daviey: are the steps you run documented anywhere ? 20:18:14 vhosakot: don't think so 20:23:28 Daviey: I see the boo process in the bottom pane empty/ended ? 20:23:33 boot* 20:24:09 Daviey: wow, what is this animation?!? :) 20:25:14 it's coming back 20:25:54 yeah I see it 20:28:17 vhosakot: ok 20:28:21 vhosakot: it worked 20:28:31 vhosakot: are you ok to take over now? 20:29:13 vhosakot: next step is to create a new inventory file, and try and get docker setup and installed on this node 20:29:20 copy over the ssh key etc 20:29:23 Daviey: great! thanks a lot for your help... 20:29:45 we can use the same inventory file we used for ubuntu 20:29:48 vhosakot: Once we ar ehappy that docker is working on this single node, we can reproduce the whole thing across the entire cluster 20:29:56 Hi all 20:30:03 vhosakot: yeah, but start with JUST this node 20:30:19 docker needs to be setup slightly differently etc 20:30:22 Daviey: should I install kolla on this node and test? or just install docker and test ? 20:30:25 facing an issue where nova resize failed as nova is not a part of sudoer 20:30:30 just docker 20:30:30 can anyone help? 20:31:06 Satya: try rebuilding your images... we've seen this happen a few times now 20:31:24 Daviey: I know the steps to install docker (it is the kolla quick start guide).. is it enough if I do just that ? 20:31:26 vhosakot: kolla can stick on this the same control node we used for ubuntu 20:31:43 Daviey: you mean the deploy node ? 20:31:55 vhosakot: Althought it would be nicer to prove it works with centos as a control node, i don't think that is part of the test 20:32:01 vhosakot: yeah 20:32:29 Daviey: right, I think CentOS on target nodes is a good test of OpenSTack deployed by kolla on centos 20:32:30 'sudo', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--privsep_context', 'os_brick.privileged.default', '--privsep_sock_path', '/tmp/tmpTeH3JX/privsep.sock' 20:32:44 this is the command which run on the compute node 20:33:01 which exited as non zero 20:33:16 who is in tmux 20:33:16 added nova to sudoer and it works 20:33:24 but is there a fix in kolla 20:34:48 any help? 20:35:16 britt u there? 20:36:10 vhosakot: master has a playbook for setting up the nodes 20:36:19 britthou_ 20:39:51 yep 20:42:33 Hello, can anyone help me with kolla deployment? I got stuck for a while 20:43:19 I'm trying to run # kolla-ansible deploy 20:43:48 But at the step "TASK [neutron : Waiting the openvswitch_db service to be ready]" 20:45:09 It kept failing 20:45:14 TASK [neutron : Waiting the openvswitch_db service to be ready] **************** FAILED - RETRYING: TASK: neutron : Waiting the openvswitch_db service to be ready (29 retries left). FAILED - RETRYING: TASK: neutron : Waiting the openvswitch_db service to be ready (28 retries left). FAILED - RETRYING: TASK: neutron : Waiting the openvswitch_db service to be ready (27 retries left). FAILED - RETRYING: TASK: neutron : Waiting th 20:46:53 The error message is fatal: [localhost]: FAILED! => {"changed": false, "cmd": ["docker", "exec", "openvswitch_db", "ovs-vsctl", "--no-wait", "show"], "delta": "0:00:00.013437", "end": "2016-08-29 12:54:46.933598", "failed": true, "rc": 1, "start": "2016-08-29 12:54:46.920161", "stderr": "Error response from daemon: Container 37e16719a963ae2561db6df932ee274f55164577d6db4c5e73cbdf8ee469381b is restarting, wait until the contain 20:48:05 vhosakot: whatcha trying to do? 20:48:27 Daviey: sorry, I was trying to see which servers got bootstrapped 20:49:41 I was trying "kolla-ansible deploy" on a virtual machine 20:49:48 vhosakot: just the one 20:49:58 vhosakot: the one we are working on at the moment 20:50:01 Daviey: yep.. cool 20:50:04 vhosakot: the one we just installed 20:51:49 vhosakot: ok, we look good now 20:52:02 vhosakot: are you ok to reinstall all the machines doing what we just did? 20:52:39 Daviey: is this now installing all-in-one kolla on 0.23 ? 20:53:19 vhosakot: no! 20:53:24 then ? 20:53:31 vhosakot: I just kicked off an image build for centos 20:53:39 and bugger, i didn't push them to the registrt 20:53:55 Daviey: on the deploy node 20:54:24 Daviey: what about installing CentOS on the 130 nodes... ? I'll get started with that 20:54:31 that is what i mean 20:54:34 the cobbler stuff 20:54:48 we've proved it works for 1, so i'm happy it will work for all 20:55:13 Daviey: well, I didn;t watch the commands you ran.. I'll need your help or can you point me to the commands in history... 20:55:20 you already did cobbler sync right ? 20:56:17 vhosakot: cobbler sync will need to be run before deploy 20:56:25 Daviey: cool.. 20:56:33 vhosakot: inc0 removed all the nodes earlier, not sure why 20:56:41 but all the systems will need to be created in cobbler 20:57:41 Daviey: "cobbler system list" shows just "729583-comp-s3700-023.cloud7.osic.rackspace.com".. is there a script to add the 130 cobbler systems? 20:59:49 Daviey: ^^ 21:00:45 vhosakot: yeah... but i don't have it 21:00:55 would be really helpful if inc0 woke back up 21:01:02 right 21:01:22 inc0: is there a script to add the 130 systems in cobbler 21:01:59 Daviey: what IP did you enter in the kolla inventory file that is currently building ? 21:02:14 we can use the same inventory file (for 130 nodes) we used for ubuntu ? 21:03:19 who is in tmux now ? 21:03:44 me 21:04:08 vhosakot: we can yes, but i wanted an inventory file with just one host in to test instaling docker 21:04:26 cool 21:11:10 vhosakot: are you following? 21:14:10 Daviey: yes, I see 129 systems now... generated form the CSV.. cool 21:14:32 thanks a slew Daviey vhosakot 21:14:39 sounds like you are both close :) 21:15:16 the hard pat will be networking config on each node 21:15:28 sdake: nah 21:15:52 Kevin Fox proposed openstack/kolla-kubernetes: Remove unneeded config overrides. https://review.openstack.org/362449 21:17:03 Daviey: cool, the boot devices are now set... 21:18:12 vhosakot: hopefully 21:18:58 vhosakot: so we are reinstalling the one we already installed aswell 21:20:17 Daviey: cool... 21:20:30 bugger didn't work 21:26:10 Kevin Fox proposed openstack/kolla-kubernetes: Make libvirt over tcp an option https://review.openstack.org/362457 21:26:57 Daviey: the tftpboot config for all the 130 nodes are same except the MAC address and the IP address, right ? 21:27:08 vhosakot: yes 21:27:11 cool 21:39:33 Daviey: I see they are powercycling.. cool 21:40:55 vhosakot: yeah, i think it is workin 21:42:09 Daviey: once done, can we login into few nods and check.. ? 21:42:11 nodes* 21:42:17 vhosakot: sure 21:42:29 vhosakot: infact, copying the ssh key to all nodes is a good test :) 21:42:58 Daviey: yep, I think there is a playbook fo that 21:43:51 vhosakot: ansible -k -i ./inventory -m authorized_key -a "key=\"{{lookup('file','~/.ssh/id_rsa.pub')}}\" user=$USER" all 21:44:20 cool 21:51:05 Daviey: can we try pinging 151 ? 21:51:18 sure 21:51:25 vhosakot: err that works 21:51:27 i just did 21:51:45 Daviey: may sshd is down on the targets 21:51:58 vhosakot: i suspect it is still instaling 21:52:06 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Script generating kubernetes secrets from passwords.yml https://review.openstack.org/354199 21:52:14 Daviey: yep.. CentOS baremetal boot takes few mins... 21:52:39 Merged openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 21:55:09 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Script generating kubernetes secrets from passwords.yml https://review.openstack.org/354199 21:57:11 vhosakot: there we go 21:57:15 :) 21:59:27 Ryan Hallisey proposed openstack/kolla-kubernetes: Update the quickstart https://review.openstack.org/362325 21:59:28 Ryan Hallisey proposed openstack/kolla-kubernetes: Add a script that will get the latest kubectl https://review.openstack.org/362312 21:59:28 Ryan Hallisey proposed openstack/kolla-kubernetes: Update the kubernetes aio guide https://review.openstack.org/361787 21:59:29 Ryan Hallisey proposed openstack/kolla-kubernetes: Massive rework of the kolla-kubernetes docs https://review.openstack.org/362467 22:01:02 Daviey: !!! 22:01:13 vhosakot: ?? 22:01:22 I see it worked on all 151 nodes! 22:01:29 129 I mean 22:01:31 vhosakot: yeah, looks good 22:01:45 vhosakot: Are you ok to take over now? 22:02:26 yes, now, once the images are built on the deploy node (that is currently going on), I'll install kolla on all the nodes.. 22:02:59 vhosakot: the images might need to be rebuilt and pushed to the registry 22:03:09 Daviey: yep, I can do that 22:03:17 i fear the registry might not have enough space 22:03:26 vhosakot: do you know the command to push? 22:03:45 Daviey: "729494-comp-s3500-002" is the deploy node right.. I 've opened it in the tmux session 22:03:56 Daviey: cool I'll use --push 22:04:35 Daviey: can we check the space of /var/lib/docker to make sire deploy node has enough space 22:04:44 coo, I see you already started the push 22:04:58 vhosakot: sorry, you really do have the driving seat now 22:05:15 Daviey: yep, I'll take it from here.. thanks a lot Daviey! 22:05:23 vhosakot: cool 22:05:49 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Script generating kubernetes secrets from passwords.yml https://review.openstack.org/354199 22:05:50 Daviey: no space :( 22:05:52 # df -h /var/lib/docker 22:05:52 Filesystem Size Used Avail Use% Mounted on 22:05:52 /dev/mapper/lxc-root00 47G 45G 0 100% / 22:06:02 vhosakot: delete the ubuntu images 22:06:23 Daviey: r u sure.. we dont need them right ? 22:06:35 vhosakot: we need to give the cluster back tomorrow, so unlikely 22:06:41 and if we do, they build in 20 mins or so 22:06:41 Daviey: cool.. thanks! 22:06:45 right 22:07:14 hmm 22:08:56 "rmi -f" will delete forcefully 22:09:18 cool! 22:10:41 delete the volumes and the stop the running containers as well 22:10:50 and delete everything in "docker ps -a " 22:14:22 Daviey: why is 75% still used ? 22:14:54 Daviey: let us restart docker and try 22:15:00 yep 22:15:01 restarting docker frees up some space sometimes 22:15:33 ah! 22:16:45 vhosakot: lets see if we have enoug already 22:16:52 Daviey: cool 22:17:02 25% of 47G should be good I think 22:17:15 Daviey: the build output looks good! 22:17:26 I will check later tonight.. thanks a lot for your help! 22:18:01 vhosakot: i might be in bed then 22:18:09 vhosakot: let me know how you get on 22:18:16 Daviey: no.. we'll chat tomorrow.. thanks 22:18:40 vhosakot: keep in mind we are supposed to hand them back tomorrow 22:18:47 Daviey: what time ? 22:18:51 vhosakot: so as much as can get done in the next few hours is worth it 22:18:57 Daviey: we can run one baseline test tonight 22:19:00 vhosakot: during the business day :/ 22:24:23 Kevin Fox proposed openstack/kolla-kubernetes: Remove unneeded config overrides. https://review.openstack.org/362449 22:36:05 evening everyone 22:54:44 Hello 22:54:53 Evening Pavo 23:23:39 vhosakot: ping 23:35:25 Merged openstack/kolla-kubernetes: Update the kubernetes aio guide https://review.openstack.org/361787 23:40:07 Merged openstack/kolla-kubernetes: Add a script that will get the latest kubectl https://review.openstack.org/362312 23:40:12 Merged openstack/kolla-kubernetes: Update the quickstart https://review.openstack.org/362325 00:15:00 yo 00:20:01 sdake: need help from vxlan expert, do you know any? 00:20:11 god i wish 00:20:17 daneyon is your best shot there 00:20:30 but i you dont know it, then i ont think daneyon would either 00:21:01 :-( 00:21:22 he is a ccie and pretty strong network guy - worth asking 00:21:31 sdake: this one of those x-files type of issue 00:21:51 when all config looks ok, no any errors in the log but it does not work.. 00:22:00 where is the vikram 00:22:38 no idea, tried to ping him earlier 00:23:10 he sent me mail 30 minutes ago 00:26:12 Hi Folks, When i am trying to deploy ceph with kolla it is forever staying in "Looking up disks to bootstrap for Ceph OSDs" task. I have one partition named as "KOLLA_CEPH_OSD_BOOTSTRAP". 00:26:30 Any idea whether it is looking for something 00:26:39 it should be fast at that step 00:26:52 are you using mitaka branch? 00:27:09 sdake: yes 00:27:14 how many nodes? 00:27:26 this is the paste where it is sitting forever http://pastebin.com/rYqvXjgA 00:27:30 single node 00:27:38 single node requires special ceph config 00:27:42 i am using all-in-one inventory file sdake 00:27:57 sdake: I am following https://github.com/openstack/kolla/blob/master/doc/ceph-guide.rst#managing-ceph 00:28:38 sdake: what is that can you tell? I have changed globals.yml and ceph.conf to tell one pool 00:28:40 for osd 00:28:48 [global] 00:28:48 osd pool default size = 1 00:28:50 osd pool default min size = 1 00:29:04 sdake: yes doing that 00:29:07 ok 00:29:13 also running this "docker exec ceph_mon ceph osd pool set rbd size 1" 00:29:16 to tell one copy 00:29:30 you should not need to do that if you set the globals.yml 00:30:00 okay let me paste my globals.yml 00:31:17 sdake: http://pastebin.com/gbPE33hC 00:31:23 that is my globals 00:32:29 sdake: i ran docker exec ceph_mon ceph osd pool set rbd size 1 too. 00:32:47 sdake: but that won't affect right as it will read globals.yml 00:32:50 what is that spposed to do 00:32:55 sdake: centos is ok? 00:33:11 duonghq no iea just woke up 00:33:15 oops 00:33:23 duonghq i was hoping Daviey or vhosakot would e up still 00:33:31 i got a email indicating the nodes are deplouyed 00:33:41 sdake: it is just tells the number of copies should be one inside pool 00:33:42 sdake: you just took a quick nap or full sleep? 00:33:50 2-3 hrs 00:34:08 2 hrs sleep 12 hrs work in cycle since wednesday 00:34:21 sdake: you are what timezone? 00:34:31 arizona gmt-7 pst 00:34:36 hope that next time I can join OSIC driver team 00:34:49 sdake: ahh 00:35:08 aNupoisc i need to get back to you - today is our lat day on the osic gear 00:35:16 aNupoisc and i guess it has beeen deplooyed on centos 00:35:21 so I need to measure some baselines 00:35:31 sdake: how many hours do we left? 00:35:38 about 6-12 00:35:50 enough to run 1 baseline 00:35:55 tight schedule 00:36:01 sdake: i am running on ubuntu 00:36:23 aNupoisc its not that i dont want to help yu - osic gear disappaers tomorrow 00:36:29 but, what wrong with you aNupoisc? 00:36:31 and i need to make use of it while we have it 00:36:36 see if I can help 00:37:02 sdake: Oh okay i don't have an idea what is osic gear? 00:37:14 sdake: okay np 00:37:27 duonghq: Hi yeah so 00:38:15 duonghq: I am running kolla with ceph and my deploy is sitting at " Looking up disks to bootstrap for Ceph OSDs" forever 00:38:37 duonghq: I am running all-in-one node setup 00:38:44 which ceph backend do you use? 00:38:54 radosgw 00:39:00 aNupoisc osic gear is 131 node cluster we have to scale tet kolla 00:39:14 sdake: Ahh awesome good luck :) 00:39:30 we have done alot of measurement already 00:39:36 20 minutes to deploy 121 nodes 00:39:39 123 nodes that is 00:39:55 duonghq: here is my globals.yml http://pastebin.com/gbPE33hC 00:40:47 sdake: I am trying to bring up ceph for 2 days and some thing is failing. We can talk on it tomorrow if not resolved by today 00:40:55 sdake: good luck 00:41:18 I do not ceph yet, sorry 00:41:27 but did you check the container log? 00:41:47 duonghq: not sure where they go 00:41:51 let me do that 00:42:12 docker exec -it heka /bin/bash 00:42:18 and look in /var/log/kolla/ceph 00:44:47 duonghq: not returning anything 00:45:09 i have two files ceph-client.admin.log and ceph-mon.log 00:45:14 both are empty 00:45:33 hmm, did you try: docker logs ceph-osd? 00:45:38 or bootstrap ceph 00:45:48 do not remember exact name 00:46:01 should those be inside docker 00:46:02 ? 00:46:11 no, the outer shell 00:46:47 saying no such container 00:47:03 docker ps -a | grep cehph 00:47:08 *ceph 00:47:11 what does it say? 00:50:55 Hey guys. From current master, horizon 00:52:36 kbaegis? what's wrong? 00:53:40 500's. Traceback: http://hastebin.com/dafahevori.rb 00:55:23 duonghq: It says ceph is running 8d4f59263708 kolla/ubuntu-binary-ceph-mon:2.0.2 "kolla_start" 00:55:35 from 3 days 00:56:02 duonghq: it is somehow unable to find the partition 00:56:07 for osd 00:56:30 thats the problem even though i have the partition named as it is looking 00:57:46 kbaegis?latest master? did you try clean and rebuild image 00:58:13 aNupoisc: can you try docker logs 00:58:22 ceph-mon, I guess 00:59:04 duonghq: Did rebuild, didn't clean 00:59:36 it is unable to find ceph-mon container 00:59:43 duonghq 01:00:05 duonghq: i forget to mention the name at start. lol 01:00:47 duonghq: do you know what extra things are required when doing ceph on single node 01:01:42 kbaegis: sometime docker image and ansible need to in sync :) 01:02:03 aNupoisc: I don't know, sorry 01:02:19 but ceph-mon is running but cannot docker logs is wired 01:03:18 duonghq: I am not sure. Do we need to mention seperately in any conf file or in globals for that? 01:03:38 duonghq: yeah seems ceph-mon is started but logs are not available 01:04:11 afaik it's quite unussual 01:04:23 not sure, sorry 01:04:31 duonghq: hmm. 01:04:34 no problem 01:04:37 thanks 01:05:30 sdake: ping.. should we run scenario # 2 on centos ? 01:05:51 vhosakot need to stup network first 01:06:14 sdake: yep, I'm searching that playbook.. do you know its name? 01:06:23 i dont know i there is one 01:06:34 sdake: I'll check.. 01:06:42 the interfaces are named differenelty as well 01:06:52 1 min... switching VPNs 01:16:13 Has anyone seen neutron agent errors like RTNETLINK answers: Invalid argument with Kolla and Centos 7? 01:16:23 It seems specific to running neutron in docker atm. 01:22:19 vhosakot - ok was delaing with yard dude 01:22:23 i am back 01:22:36 sdake: cool, I'm still searching the playbook... 01:22:49 do we have the globals.yml ready for scenario 2 ? 01:22:56 not really ready 01:23:00 but its prtty rclose 01:23:09 sdake: ok... 01:23:10 the interface names are diferent on that machine 01:23:16 on centos the interface names ae different 01:23:35 that machine = ? deploy node? 01:23:40 253? 01:24:10 all machines deployed with centos have a differnet interface name 01:24:12 enp1s0 01:24:15 vs eth0 01:24:16 etc 01:24:26 ok.. 01:24:34 sdake: 1 sec... i'll be back... 01:27:41 sdake: back 01:27:42 vhosakot_ did you capture ipconfig on one of the nodes before resetting them? 01:27:48 ifconfig thatis 01:27:59 sdake: no 01:28:10 britthouser would be helpful here 01:28:11 ca nyou call him 01:28:12 he knows how the networks are setup i htink 01:28:40 yes, britthouser said he can help after 9 pm EST today and it is 9:28 pm EST here now 01:29:33 p4p1.702 Link encap:Ethernet HWaddr 68:05:ca:38:65:64 01:29:33 inet addr:172.21.8.253 Bcast:172.21.11.255 Mask:255.255.252.0 01:29:53 p4p1.733 Link encap:Ethernet HWaddr 68:05:ca:38:65:64 01:29:54 inet addr:172.21.112.250 Bcast:0.0.0.0 Mask:255.255.252.0 01:30:23 no idea what network topology of this gear i 01:30:24 wow, that is the sub-interface for VLAN 702 I think... I doubt the cobbler ks has VLAN info... will the playbook set this up ? 01:30:24 is 01:30:40 there is no playbook that i can see 01:30:46 it was deployed by ansible i'm certain of it 01:30:59 becuse the interface file aid at the top "Deployed by ansible" 01:31:29 there are other vlans we have as ewll 01:32:24 setup-insecure-inv.yml 01:32:28 that is one file 01:32:31 I think it needs modification 01:32:45 ok 01:32:47 let me check 01:33:47 how aobut we run the host setup playbook 01:33:50 and see what happens 01:33:52 moment 01:34:25 2: ens1f0: mtu 1500 qdisc mq state UP qlen 1000 01:34:26 link/ether 3c:fd:fe:9c:6f:1c brd ff:ff:ff:ff:ff:ff 01:34:26 inet 172.21.0.24/22 brd 172.21.3.255 scope global dynamic ens1f0 01:34:27 valid_lft 18192sec preferred_lft 18192sec 01:34:28 inet6 fe80::3efd:feff:fe9c:6f1c/64 scope link 01:34:28 valid_lft forever preferred_lft forever 01:34:30 3: ens1f1: mtu 1500 qdisc mq state UP qlen 1000 01:34:30 link/ether 3c:fd:fe:9c:6f:1d brd ff:ff:ff:ff:ff:ff 01:34:32 4: ens4f0: mtu 1500 qdisc mq state UP qlen 1000 01:34:32 link/ether 3c:fd:fe:9c:73:14 brd ff:ff:ff:ff:ff:ff 01:34:34 5: ens4f1: mtu 1500 qdisc mq state UP qlen 1000 01:34:34 link/ether 3c:fd:fe:9c:73:15 brd ff:ff:ff:ff:ff:ff 01:34:36 those are all the interfaces on the machine 01:34:42 sdake: setup-insecure-inv.yml starts the docker insecure registry 01:34:49 i know - i looked at it 01:34:58 sdake: who setup the netwporking in ubuntu 01:35:00 it doesn't start the registry 01:35:06 it sets up insecure registry on all the nodes 01:35:10 no idea 01:35:13 i think inc0 or pburke 01:35:15 pbourke 01:35:28 but britthouser was involved 01:35:34 right, setup-insecure-inv.yml updates DOCKEROPTS /etc/default/docker... 01:36:04 sdake: in centos, the config file is not /etc/default/docker... I think it is different.. let me check 01:36:10 would have been handy to keep an interface file on the ubuntu machines 01:36:13 are there any machines you didn't reload? 01:37:30 sdake: only deploy node (253) was not touched 01:38:20 i dunno - dead in the water 01:38:42 need britthouser 01:39:01 shall I setup a webex with you and britthouser ? 01:39:24 i think irc should be good enough 01:39:36 cool 01:39:40 is britthouser here? 01:39:57 my cll is out of juice - so i can't call him 01:40:42 cell 01:41:10 1 sec... need to switch VPN again to find Britt;s cell number 01:44:36 sdake: ok back 01:45:37 sdake: wow, britthouser does not have his cell number in the compant dir 01:45:42 company* 01:47:45 sdake: I emailed britthouser 01:47:55 I am here now sdake and vhosakot 01:48:01 :) 01:48:10 cool - wife just brought home dinner 01:48:18 britthouser do you recall how the networks were configured for kolla 01:48:32 britthouser: so, all the 130 nodes have CentOS now and need networking/VLAN settings 01:48:34 there were some vlans and stuff 01:49:04 I took some notes, lemme look 01:49:12 1 sec.. switching VPNs... I'll be back 01:49:33 Cisco<-->F5 VPN switching is a game! 01:49:49 So have a look at the etherpoad 01:49:55 there is a section called "Network Layout" 01:50:07 p4p2 and p1p2 should be bonded into bond0 01:50:33 p1p1 is our external 01:50:45 p4p1 is the ceph cluster (on vlan 733) 01:52:02 line 128 01:52:40 britthouser do you know how to do that on centos on 123 nodes via ansible 01:53:08 note centos has different interface names 01:53:20 britthouser so weprobaby need a mapping of some type 01:53:44 back.. 01:53:56 britthouser: which tmux session are you in? 0 or 9? 01:55:14 the centos source images built can be seen in tmux session 0 window 3 01:55:54 I'll login to zero 01:56:02 cool 01:58:40 cn you cats make yur screens bigger 02:02:08 Agreed sdake and vhosakot - lets stick with the interface names we have 02:02:24 too complicted to rename 02:02:28 and doens't work particualrly well 02:04:28 cool... 02:04:41 britthouser: so, what "extra" networking is needed than how it is now ? 02:04:43 sdake - my proposal is that we work out the bondign config on one host, and then write the ansible todo the rest 02:04:51 lets login to one of the nodes and see 02:04:53 britthouser: should we bond and sub-inerface for vlan ? 02:04:53 wfm 02:08:58 britthouser: window 3 was deploy node.. now, it is not.. what happened ? 02:09:13 * britthouser shrug 02:09:32 anybody know where the baremetal roles is for ubuntu 14.04 or 16.04 one? 02:09:38 britthouser: :) cool... 02:09:39 *whether 02:09:57 duonghq its for 14.04 02:10:28 thank sdake 02:11:20 which interfaces aer e bonding 02:11:34 ens1f1 and ens4f1 02:11:45 old p1p2 and p4p2 02:11:49 britthouser: ah, did you SSH into one of the target nodes (.45) in window 3 ? 02:12:02 someone must have vhosakot_ 02:12:16 cool.. 02:12:40 may have been me looking for interface files for ubuntu 02:13:01 cool.. 02:13:53 Lu lei proposed openstack/kolla: Fix oslo.i18n in kolla project https://review.openstack.org/355342 02:15:41 zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 02:16:26 britthouser: of the 130 nodes, on which nodes are the bonds configured? 02:16:33 britthouser pretty sur eyu can get rid of uuid as well as long as hou ahve the name 02:16:37 all 02:16:51 wow, all 130 needs bond.. ok 02:16:53 I think you need either HWADDRES or UUID 02:17:29 dupengfei proposed openstack/kolla: * Fix lost parameter --remote in generated start_ovsdb_server.sh https://review.openstack.org/362027 02:17:42 * britthouser crosses fingers 02:17:56 SSH did not drop. cool! 02:18:39 nice! 02:18:56 oh duh 02:19:26 britthouser: we can refer VIM's bond0 as sample? 02:19:31 cool 02:19:37 that's where I'm cutting pasteing 02:19:42 :) 02:20:08 nice mii statu ftw 02:20:28 Partner Mac Address: 00:23:04:ee:be:b9 02:20:31 we are bnoded with teh switch 02:20:44 britthouser: is this mac the mac of the remote end ? 02:21:00 right...that is the switch_id sent by lacp process on the ToR 02:21:04 cool 02:21:13 if its all 00:00:00:00 02:21:15 shouldn;t bond0 have an IP ? 02:21:29 no 02:21:35 we now need to create bond0.702 02:21:40 the power of 100 lemons!! 02:21:40 bond0.730 02:21:46 bond0.731 02:21:49 ROFL 02:21:53 bond0.732 02:21:56 yes, I saw that issue the other day.. partner MAC was 00:00:00:00 and port-channel on the other end was down due to bad networking 02:22:01 and ens4f1.733 02:22:41 britthouser: how do we redo these steps on the other 129 nodes... should we write a bash script and use sed to replace lines in files ? 02:22:52 probably ansible lineinfile? 02:22:56 cool 02:23:00 we should look back how it was done in ubuntu 02:23:08 lineinfile 02:23:09 it will be similar, just different files instead of all in one file 02:23:41 britthouser i don't see any networ playbooks on 253 02:23:43 are 730, 731 and 732 the VLAN Ids trunked on bond0? 02:24:56 hmm....not sure how inc0 did that then. 02:25:05 oh...lets check the lxc-container 02:25:10 he might have done it in there 02:28:15 Hello, everybody, I am a newer in openstack contributor. 02:28:53 bingo 02:30:52 hey guys need to run to store for 10 mins bbiaf ok? 02:30:58 sure thing! 02:33:45 So in ubuntu, there is just a single file which has all the interfaces 02:34:01 In Centos, each interface has its own file. 02:34:06 britthouser, /etc/network/interfaces 02:34:22 britthouser: did the steps you did on the targetnnode .45 work ? If yes, can we try to add those steps in lineinfile ? 02:34:27 Right duonghq, taht the ubuntu file 02:34:43 vhosakot_ - they worked, but they aren't everything 02:34:52 britthouser: ahok 02:35:08 we need to divide the playbooks into two categories: 1) interfaces files that already exist and just need some lineinefile plays 02:35:09 ah, I just blackout for awhile, so I lost the context 02:35:21 2) interfaces that don't yet exist and we can use templates to create 02:35:29 make sense? 02:35:59 So we can ansiblize the ens1f1 and ens4f1 config changes using lineinfile 02:36:15 and the templatie the bond0, bond0.whatever and ens4f1.whatever files. 02:36:16 britthouser: yes, for 2), do we write a shell script and call ansible's "shell" module to run the shell script on all the nodes ? 02:36:36 no, lets use template simliar to what we have with interfaces.j2 02:36:45 b/c the Ip address logic will be the same 02:36:48 we can reuse 02:37:13 we'll just have interface.bond0.j2 and interface.bond0.733.j2 etc 02:37:13 right right.. cool 02:37:16 make sense? 02:37:19 yes... 02:37:32 which host distro do we support? Ubuntu 14.04/16.04 and??? 02:37:32 so, three templates to create 3 new files on each node 02:37:35 right ? 02:37:53 I think 6? 02:38:19 bond0, bond0.702, bond0.730, bond0.731, bond0.7732, ens4f1.733 02:38:36 7732=732 02:38:50 cool, and edit for the ens1f1 and ens4f1 files are done using lineinfile ? 02:38:58 that's my thoughts. 02:39:08 cool, yes.. thats a great idea... 02:39:08 You wanna start with the lineinfile part, and I'll watch? 02:39:20 or vice versa? what do you think? 02:39:34 yes... I need to know the lines to change... I can do the lineinfile part.. cool 02:39:57 just diff the file and file.orig 02:40:05 I think you can pick out the differences? 02:42:46 back 02:42:55 vhosakot_ I think instead of commenting out, we can just remove 02:42:58 Hey sdake 02:43:01 I think we have a plan 02:43:11 need to me re-iterate? or do yo have scrollback? 02:43:20 let me read 02:43:21 sdake: hey.. still playing with interface config files 02:43:30 britthouser: cool 02:43:40 ya keep at it - if we finish by midnight pst i should be able to run a baseline :) 02:43:41 vhosakot_ lets copy the current files to .bond 02:43:44 britthouser: can I delete the lines commented out (with #) 02:43:58 Yeah vhosakot_ lets delete the files with # 02:44:03 and then copy to *.bond 02:44:04 coolsvap: 02:44:05 cool 02:44:12 then copy the orig back 02:44:22 and we can run our playbook and diff against the .bond file to make sure it turned out right. 02:45:12 vhosakot_ if your using a tempalte you just use template and a globals..yml to go with it 02:45:49 cool 02:46:05 lemme change one more thing vhosakot_ 02:46:13 britthouser: sure, go ahead 02:46:45 ok that's cleaner 02:46:54 Jeffrey Zhang proposed openstack/kolla: Add OSIC Scale Testing Documentation https://review.openstack.org/352101 02:47:56 Ok so it looks like we need to remove 10 file, change 2 lines, and add 3 lines. yes? 02:48:06 remove 10 lines* 02:48:23 britthouser Jeffrey4l_ is here 02:48:33 Jeffrey4l_ can probbly helpl very fast get tihs problem sorted 02:48:40 Hey Jeffrey4l_ 02:48:48 sdake, britthouser hey 02:48:52 what's wrong? 02:48:55 Jeffrey4l_ centos is deployed 02:49:02 now need to configure netowrkign on 130 nodes 02:49:13 we have a reference impementation on one of the nodes of the cnfig files 02:49:20 need that rolled across 130 nodes 02:49:42 write a ansible templates? 02:49:50 dupengfei proposed openstack/kolla: Fix Bug #1618294, Live migration failure bug. Modify vncserver_listen and server_listen to 0.0.0.0. Closes-Bug: #1618294 https://review.openstack.org/362577 02:49:50 bug 1618294 in kolla "Live migration failure bug." [Undecided,New] https://launchpad.net/bugs/1618294 - Assigned to Pengfei Du (58750307-3) 02:49:52 that is what vhosakot_ is doing atm 02:49:53 If you're in tmux 0, then on the screen you see the diff of the manual changes we made that need ansiblizing 02:49:53 use ansible tempalte? 02:50:09 We discussed, and I think there are two sets of changes we need. 02:50:11 Jeffrey4l_ yup - any chance yuo can jump in and write it quckly 02:50:12 i am in the tmux 0 now. 02:50:18 1) changes to existing files (what you see now) 02:50:23 2) additional files that don't exist yet 02:50:31 so I think templates for #2 02:50:32 britthouser: I'm writing lineinfile to edit lines 02:50:34 you cn doo all of those with tmeplates 02:50:49 I think we need UUID or HW_ADDRESS 02:51:02 ok it sounds like vhosakot_ is writing some ansible not in the tmux. =) 02:51:10 dont need uuid but may need mac address 02:51:28 britthouser: I'm in tmux session 0 window 3 :) 02:51:34 it works with UUID, so rather not rock the boat 02:51:46 wow, mac address need to be read from cobbler config for each node then 02:51:53 uuid and mam doesn't matter. i think 02:52:02 mam/mac 02:52:15 if you do not point the mac, it use the physical mac address. 02:52:28 well, what we narrowed down for these files Jeffrey4l_ was we need to remove 10 lines, change 2 lines and add 3 lines 02:52:34 britthouser: shall I write ansible lineinfile code to remove lines first? 02:52:38 so it should be pretty easy playbook, and I think vhosakot_ is already writing it. 02:52:47 go for it vhosakot_ 02:52:50 britthouser: cool 02:52:58 ok 02:52:59 Are you writing it not in the tmux? 02:53:09 vhosakot_ ^^ 02:53:24 if so, then Jeffrey4l_ and I can look at the 6 files we do need to create from template? 02:53:25 I am.. session 0 window 2 02:53:29 britthouser: ^^ 02:53:31 btw Jeffrey4l_ i captured results for the fio test 02:53:51 sdake, cool. and i captured too. and push it into the PS just now 02:53:55 britthouser: aren't you in session 0 window 3? 02:53:58 ok 02:54:20 Ok, vhosakot_ I'll let you work on that uninterrupted. Ping us when you get it written and we can test 02:54:20 sdake, i added scenario-11 ( fio test) and scenario-12( nova boot test) 02:54:41 britthouser: cool... I'll let you guys know.. 02:56:04 Jeffrey4l_: do you know ryu? 02:56:34 sbezverk_, no :( what happen? 02:57:36 Jeffrey4l_: I am struggling with vxlan tunnel between two compute nodes in kube, suspecting ryu which is openflow controller.. 02:58:11 Jeffrey4l_: but I have never dealt with it before, so I was hoping you might know.. 02:59:36 sbezverk_, why introduce it? is it necessary? 03:03:24 Jeffrey4l_: I did not introduce it, it is in neutron 03:03:45 it is in kolla's build image 03:05:08 sbezverk_, but i find nothing in the kolla' code about ryu? which line install that? 03:06:40 @vhosakot_ feel free to overwrite the ifcfg-ens1f1 adn ifcfg-ens4f1 files in your play. I have made backup copy. 03:06:42 Jeffrey4l_: do you have by any chance ml2_conf.ini from multinode with vlan used for tenants and vxlan for tunnel? 03:06:56 britthouser: yep... cool 03:07:02 here you go: INFO:kolla.image.build.neutron-base:Installing collected packages: waitress, beautifulsoup4, WebTest, httplib2, logutils, neutron-lib, oslo.cache, psutil, oslo.reports, oslo.rootwrap, oslo.versionedobjects, ovs, singledispatch, pecan, ryu, neutron 03:07:04 Dupengfei hey there, looking to contribute to kolla or looking to find where you would like to contribute? 03:07:04 sbezverk_, yep. I have a env use that env 03:08:21 Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 03:08:55 Jeffrey4l_: as you can see it gets installed as part of neutron.. if you can share that ml2 conf, that would be awesome.. 03:09:56 sbezverk_, http://paste.openstack.org/show/564724/ 03:10:25 Duong Ha-Quang proposed openstack/kolla: Change package installation to use ansible package task https://review.openstack.org/362579 03:10:34 Jeffrey4l_: thank you very much 03:11:24 sbezverk_, neutron depends ryu. the requirements.txt has: ryu!=4.1,!=4.2,!=4.2.1,!=4.4,>=3.30 # Apache-2.0 03:13:06 OK. I got the reason. the openvswitch use the native openflow driver, which depend on ryu packages. 03:13:36 Jeffrey4l_: confused by your config. physnet1 is flat and vlan, never seen it before 03:14:14 this one from compute node right? 03:14:30 sbezverk_, compute node and control node are the same 03:14:49 I use physnet1 as the flat interface and value interface, that's OK 03:14:56 172.18.2.101 this ip address is assigned to which interface? 03:15:30 sbezverk_, customize the config by /etc/kolla/config/neutron/ml2_conf.ini check here http://paste.openstack.org/show/564725/ 03:15:49 sbezverk_, eth5, another interface. 03:16:19 Duong Ha-Quang proposed openstack/kolla: Change package installation to use ansible package task https://review.openstack.org/362579 03:16:25 sbezverk_, br-ex bridged to eth1 03:16:53 Jeffrey4l_ you up for debugging the host install playbook for centos 03:17:22 sdake, vhosakot_ is working on it. 03:17:25 Jeffrey4l_ or seeing it in action 03:17:29 no he is working on the network stup 03:17:41 our host playbook doesn't do host network sstup 03:17:41 OK> install playbook. 03:17:49 sdake: I'm working on it.. almost done 03:18:05 hmm. my osic_playbooks should work with centos. 03:18:08 let me check 03:18:14 Merged openstack/kolla: Make the kolla_keystone_service can update fields https://review.openstack.org/348382 03:18:43 Jeffrey4l_ - we have full hot playbooks i'd like to tyr out 03:18:55 Jeffrey4l_ installs docker, configures docker, intalls /etc/hostss etc 03:19:03 Jeffrey4l_ how do you creat ea new tmux? 03:19:07 Merged openstack/kolla: Enable the nova microversion api https://review.openstack.org/348432 03:19:14 nm i got new tmux open 03:19:18 feel free to join :) 03:23:53 @sdake, hello , I am in 03:26:31 vhosakot_ - It hink I forget we need to also edit the ifcfg-enp4f0 file 03:26:44 I 'd like to contribute to kolla and kolla-k8s preffered. 03:27:05 britthouser: cool..once the playbook is done.. we can run add more interfaces,... cool 03:28:50 sounds good vhosakot_ 03:28:56 I think I have the templates created for the other interfaces 03:29:17 so should be pretty easy to add plays to render those when you're ready 03:29:41 Dupengfei cool - so best thing to do is ramp up 03:29:45 cool.. I'm almost done with lineinfile to remove, edit and add lines in ifcfg-* 03:29:50 Dupengfei have you got an aio deployment under yourbelet? 03:29:52 that is step 1 ;) 03:30:52 sdake: I have a multinode deployment 03:31:06 sdake: and work well. 03:31:10 Dupengfei nice! 03:31:34 sdake: I tested most features on openstack. 03:32:16 sdake: Sorry, I say I had a multinode deployment with kolla-ansible 03:32:18 Dupengfei we are rihgt at the end of our cycle 03:32:27 milestone 3 is the 31st 03:32:46 Dupengfei so as far as feature dev goes we dont need alot at this point - mostly testint bug fixes for kolla-ansible for newton 03:32:48 sdake: kolla-k8s is not ready. 03:33:03 ya kolla-kubernetes wont be ready for some time 03:33:11 that is why its not 1.0.0 :) 03:33:26 its at pre-release versioining 03:33:54 sdake: I see, my team is willing to use and update kolla-k8s. 03:34:03 nice 03:34:24 we need devs on that deliverable 03:34:26 looks right vhosakot_ ? 03:34:40 rhallisey or sbezverk_ are the guys to tlak to about what work needs to be done 03:34:43 sbezverk_ is here right now 03:35:14 sdake: so as far as feature dev goes we dont need alot at this point - mostly testint bug fixes for kolla-ansible for newton 03:35:28 sdake: What's meaning, I can't understand. 03:35:49 we are not doing feature dev on the ansible code base for about another 6 weeks until we releases 03:35:52 or whenever it is 03:35:59 just bug fixes 03:36:34 sdake: OH, I see. Forgive my poor english 03:38:40 britthouser in the other tmux - you can see the bootstrap playbook we hae in kolla proper 03:38:42 which installs the deps and whatnot 03:38:49 anddoes baic configuration 03:41:09 took about 7 minutes to run 03:41:32 "in the other tmux" ? sdake? 03:41:55 ok britthouser, the lineinfile now works anf edits config files for ens1f1 and ens4f1.. should we edit files of ens1f0 and ens4f0 as well ? 03:42:02 anf=and* 03:42:08 just ens4f0 03:42:15 and it will be similar but subset of what you did vhosakot_ 03:42:26 check out the ifcfg-ens4f0 and ifcfg-ens4f0.orig 03:42:37 it will be the same line changes, lien remove, but NOT the line adds 03:42:44 ah ok.. 03:42:51 ok I'm in the other tmux sdake 03:43:00 britthouser: ens1f0 has the IP 172.21.0.45 from cobbler.. no need to edit its file ? 03:43:03 it already finished 03:43:11 correct no need to edit that file 03:43:55 yes britthouser, I see the different edits needed in ifcfg-ens4f0.orig... just edits and removals, no additions.. 03:44:08 I'll add them to the inlineinfile playbook 03:44:08 Duong Ha-Quang proposed openstack/kolla: Change package installation to use ansible package task https://review.openstack.org/362579 03:44:11 yeah and should be same edits and removals as the other files 03:44:16 easy cust/paste It hnk 03:44:19 britthouser: cool 03:44:20 or start using loops 03:44:35 i.e. the ansible with: 03:44:57 What are you the two issues you're seeing sdake? 03:45:19 docker-py is not installed thus far 03:45:32 openssl-devel and libffi-devel are installed as well which is wrong 03:45:59 the playbook looks like it installs docker[-py, but it actually doesn't? 03:46:13 its called python-docker-py 03:46:27 yes...RH renamed that 03:47:04 were openssl-devel and libffi-devel installed as dependancies? 03:50:23 sdake: I'm headed back to tmux0 to see how vhosakot_ is getting along 03:50:34 sounds good 03:50:40 i think we are done in this one 03:50:46 britthouser: Im done.. testing on 45 03:52:25 looks good vhosakot_ ! 03:52:33 wanna do a live run? 03:53:23 britthouser: two things.. need to point to real files instead of *.orig.test files.. and need to point to entier inventory with 130 nodes, not to just 45 (test target node) 03:53:40 well, lets get the other tempaltes in there too before the 130 nodes 03:54:35 also, wanna clean it up a bit and make a loop? 03:54:49 britthouser: yep.. 03:54:51 that's really up2 you. it works, don't break it. =) 03:55:18 yeah.. it works as is.. :) we can add loop and variable if you'd like 03:55:32 meh...I'm on the fence vhosakot_ 03:55:42 lets clean up later 03:56:00 the famous last words of every great inventor ;) 03:56:05 ROFL 03:56:12 50 years later people still cleaning up their messes :) 03:56:28 but in this case - no need to be perfect 03:56:31 whatever works ;) 03:56:31 True. =) Fortunately (unfortunately?) we have a short deadline. 03:56:34 yeah 03:56:36 britthouser: cool, functionality first, looks later :) 03:56:45 ok, setup_centos_networking.yml is ready and points to real files 03:56:52 shall we run on all 130 nodes 03:56:55 not yet. 03:57:07 Lets add 6 more plays to do the templates 03:57:07 sdake: you can see in tmux session 0 window 3 if you'd like 03:57:09 britthouser: cool 03:57:14 get the anestaphine from the medlock 03:57:23 lol 03:57:37 britthouser: can't we run setup_centos_networking.yml first and make the edit/replces/additions ? 03:58:08 yes lets test on real files on one host 03:58:15 then test adding templates on one host. 03:58:16 ya there has to be a better way long term 03:58:26 i think we will need network setup in our playbooks at some point 03:58:29 then once we're happy, fire off the 130 03:58:32 so keep a copy of that 03:58:53 lemme revert the ifcfg-interfaces files 03:58:58 and then you can try your places on real files 03:59:43 ok go ahead with your live test vhosakot_ 03:59:50 on just that one host 04:00:12 britthouser: sdake: inventory-ips-only is the right inventory file ? 04:00:16 roger vhosakot_ 04:01:24 britthouser: why do you want to revert your changes on the test target node (45)... if no matches found, lineinfile will not fail and continue to next line/file 04:01:45 Call me extra careful 04:02:06 britthouser: cool :) yeah, actually back them up.. they are working configs 04:02:20 They are backedup as .bond 04:02:29 cool.. shall i run for all 130 nodes then ? 04:02:34 no 04:02:37 britthouser: ok 04:02:41 run just that one node again 04:02:47 britthouser: one node.. ok 04:02:48 and then lets add all the vlan interfaces 04:02:59 cool 04:03:00 on just that one node too 04:03:07 and THEN go 130 nodes 04:03:46 britthouser: ran on one node 04:03:55 +1 04:04:03 ok lets add the playbooks to render those tempaltes 04:04:23 Since the interfaces.j2 tempalte was in teh container 04:04:23 britthouser: should I add the command to restart network service in the playbook ? 04:04:29 byby, every one 04:04:41 Yeah vhosakot_ 04:04:50 britthouser: cool 04:04:50 also to modeprobe bonding 04:04:57 modprobe 802.1x 04:05:34 errr 04:05:37 modprobe 8021q 04:07:59 vhosakot_ I copied all the templates into /root on your deploy node 04:08:07 have a look at one of them and tell me what you think 04:08:21 britthouser: cool.. I see ens4f0 is not part of bond0.. is this fine ? 04:08:29 correct 04:08:33 cool 04:08:35 only the *f1 interfaces are bonded 04:09:05 britthouser: did you delete the setup_centos_networking.loop.yml.. I dont see it 04:09:12 yeah 04:09:14 cool 04:09:17 since we weren't gonna do that 04:09:25 cool.. we can do later.. sure... 04:09:27 you caught me. =P 04:09:40 ok, I now see you *.j2 files britthouser on the deploy node.. cool 04:09:56 so we need to have plays to render those tempaltes on each node 04:10:44 yes.. there is a playbook.. 1 sec.. let me check 04:10:59 in the osic-prep 04:11:05 there is configure-bonds.yml 04:11:16 which we can cut/paste the play 04:12:03 it looks like this vhosakot_ 04:12:04 https://paste.fedoraproject.org/417104/47253031/ 04:12:34 britthouser: cool.. thanks! 04:12:45 so we just add 6 templates to your existing playbook 04:13:00 we can ignore the ifdown/ifup since we restart networking 04:13:00 britthouser: should we run your templates on just one node first to test on the test targte node (.45) ? 04:13:06 yes please. =) 04:13:14 britthouser: cool.. I'll do it now 04:13:20 If you haven't caught on yet...I'm risk averse. =P 04:14:01 dupengfei proposed openstack/kolla: Modify Live migration failure bug. https://review.openstack.org/362577 04:16:12 dest is wrong @vhosakot_ 04:16:25 britthouser: what should it be ? 04:16:37 the older path was ubuntu's 04:16:48 the first one should be /etc/sysconfig/network-scripts/ifcfg-bond0 04:16:59 morning pbourke ! 04:17:11 britthouser my first director at intel called me a risk taker and green 04:17:21 second one same but bond0.732 04:17:38 britthouser: ah ok .. filename 04:17:49 I've never been called a risk taker sdake. =P Probably why I haven't advanced as fast career wise. But for me, slow and steady wins the race. =) 04:18:10 britthouser the key with risk taking is its painful when your green 04:18:24 but when your not green and you take risks all your life - itleads to a 80-90% hit rate 04:18:40 all those risks taken = serious punishment 04:18:43 you know when you can win 04:18:48 and when to play no foldem holdem ;) 04:18:59 With house paid for, I could probably stand to take more risk, but unfortunately I'm also change-averse. 04:19:02 I really like my ruts 04:19:19 i dont mind change 04:19:26 britthouser is a rockstar :) 04:19:34 ROFL 04:20:03 I could eat the same cereal for breakfast my entire life and be happy. Drives my wife cray 04:20:07 crazy 04:20:45 one more vhosakot_ 04:20:52 ifcfg-ens4f0.733.j2 04:21:01 britthouser: ah ok 04:21:03 let me add it 04:21:24 You like my back seat driving? I feel kinda like hyacinth bucket 04:21:45 lolol 04:22:13 ok britthouser... does it look good now ? 04:22:15 * britthouser crosses fingers 04:22:28 britthouser: let me add the task to restart network service 04:22:33 and modprobe? 04:22:40 maybe you added that and I didn't see it. 04:23:13 I'm 90% sure CentOS will do that automagically, but better safe than sorry 04:23:47 you want bonding and 8021q 04:23:52 britthouser: just "modprobe bonding" is good 04:23:53 ? 04:23:59 vhosakot_ he menas in the plaboook 04:24:09 yeah ^^ 04:24:10 i already insserted that module on45 04:24:17 sdake: yes.. I asked it for the playbook :) 04:24:27 centos does not do it automatically - atleast not on systemctl restart network 04:24:31 cool.. let me add it 04:25:21 vhosakot_ add 8021q after bonding 04:25:46 britthouser: like "modprobe bonding 8021q" 04:25:47 ? 04:25:52 yup vhosakot_ 04:26:08 britthouser: cool looks good? 04:26:08 * britthouser crosses fingers again 04:26:10 yup 04:26:21 ok, shall we run the play book on 45 ? 04:26:29 yup 04:26:40 and may be on 46 as well (that has all the origin configs from cobbler).. 04:26:48 sounds good 04:27:03 britthouser: cool.. here I go! 04:28:00 I htink we just ignore errors there 04:28:07 b/c they all got added 04:28:35 hmm...the bond0 didn't get their IPs 04:29:51 double check their content 04:29:59 ya yu cn't just add bonds and systemctl restart network 04:30:01 hve to reboot 04:30:01 there is no IP inifcfg-bond0 britthouser 04:30:11 because network tries to down the interface 04:30:15 ah ok 04:30:19 and the interface is new 04:30:25 but make sure the contents o the file are correct 04:30:28 or the reboot will crater 04:30:51 i mean on the target.. 04:31:02 yeah the files look good 04:31:15 we're missing something though.... 04:31:29 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 04:31:32 britthouser: yeah.. none of the sub-interafces for IPs 04:31:35 hmmmm 04:31:40 why? 04:32:02 ens4f0.733 does 04:32:04 just none of teh bonds 04:32:34 good morning 04:32:42 hey berendt 04:32:51 good morning berendt 04:33:02 if files look good reboot 04:33:12 ON THE TARGET 04:33:16 if they dont look good on the target 04:33:17 dont reboot 04:33:29 sdake: britthouser: shall I delete all the interfaces and rerrun 04:33:35 no 04:33:38 ok 04:33:56 # ifup bond0.731 04:33:57 Error: Connection activation failed: Connection 'Vlan bond0.731' is not available on the device bond0.731 at this time. 04:33:59 you cannot down a network with a bond after adding a bond later to the system 04:34:13 see I cannot retsrat network.serice manually as well 04:34:15 I wonder if this is network manager again. 04:34:35 lets try adding NM_CONTROLLED=no 04:34:39 and restart networking 04:34:49 I think netowrking manager is trying to interfere 04:34:57 hmmm. I have seen it in past 04:35:25 yeah taht is it. 04:35:30 check out bond0.731 04:35:35 it had ip address now 04:35:45 so lets edit our tempaltes 04:35:45 add NM_CONTROLLED=no 04:35:47 and rerun 04:35:55 you are one step ahead of me vhosakot_ 04:35:58 britthouser: looks good? 04:36:00 haha 04:36:01 yes 04:36:06 cool, let me save and rerun 04:37:01 should we retsart NetworkManager as well ? 04:37:07 all of the tempaltes need it 04:37:10 one sec I'll add 04:37:15 cool, please 04:37:15 uninstall network manager more like it :) 04:37:20 lol... 04:37:40 NetworkManager is really a misnomer 04:37:43 it should be called: 04:37:53 NetworkMismanager 04:37:54 don't write it 04:37:56 :) 04:38:06 yeah, I have seen two services trouble netowking in Centos.. why does centos have two..network.service and NetworkManager.service 04:38:11 hahaha 04:38:15 NetworkMismanager.. lol 04:38:19 ok try again vhosakot_ 04:38:31 britthouser: cool.. what did you do ? 04:38:36 if we uninstall network manager, we'll probalby lose the DHCP address on ens1f0 04:38:45 I added NM_CONTROLLED=no to all the tempaltes 04:38:45 ah right 04:38:55 some already had it right ? 04:39:01 yeah 04:39:05 cool.. re-running 04:39:20 worked! 04:39:28 yes!! 04:39:33 hmm...730 has two addresses 04:39:48 hmmm 04:39:48 I think I might have a typo in the template 04:39:49 lemme check 04:39:52 cool 04:40:00 # grep 730 ifcfg-* 04:40:00 ifcfg-bond0.730.j2:DEVICE=bond0.730 04:40:10 i know whats coming, cultavanue 04:40:11 ifcfg-bond0.732.j2:DEVICE=bond0.730 04:40:11 yup 04:40:11 woops 04:40:11 the serpents 04:40:16 ok one more time 04:40:22 cool.. 04:40:28 shall I run britthouser? 04:40:31 yes 04:40:33 cool 04:41:07 w00t 04:41:15 Ok just for good measure, lets reboot .45 04:41:15 !!! 04:41:16 vhosakot_: Error: "!!" is not a valid command. 04:41:24 I can do that while you run on .46 04:41:26 sound good? 04:41:41 !/bin/bash 04:41:42 sdake: Error: "/bin/bash" is not a valid command. 04:41:52 .45 going down..... 04:41:56 britthouser: yes.. I'll run on 46 and may be 47 too.. can you check the bonding in 46 and 47 after I run.. 04:42:04 britthouser: r u rebooting 45 ? 04:42:09 yes 45 is rebooting 04:42:12 go ahead iwth 46 04:42:12 cool.. 04:42:15 ok.. 04:42:29 once 45, 46, 47 are good... we'll blast it for all 130 modes 04:42:31 cool 04:43:28 you config file is hardcoded vhosakot_ 04:43:32 there you go. = 04:43:44 britthouser: .. we need to chanage it to "all" 04:44:22 running on 46 04:44:33 ah, modprobe bombed 04:44:58 * britthouser scratches my head 04:45:01 it worked fine manually 04:45:08 yeah 04:45:09 hmmm 04:45:21 lemme remove and try it again? 04:45:37 britthouser: yes 04:46:30 add them one at a time 04:46:53 all 130 ? 04:47:01 its putting them in the same command that is messing it up 04:47:04 no the modprobe 04:47:08 split it into two commands 04:47:27 reboot on .45 looks good 04:47:40 britthouser: cool.. all the IP and bonds r good on 45 after reboot ? 04:47:45 correct vhosakot_ 04:47:57 britthouser: cool.. do the bonds have partner MAC ? 04:48:03 yes 04:48:15 britthouser: cool! what do you mean by add two modprobe commands 04:48:26 vhosakot_ ope nplaybook 04:48:33 so just split the modprobe command into two 04:48:35 run on 46 04:48:47 and I thnk we'll be ready to shoot all 130 of our toes off. =P 04:49:22 britthouser: ah, add just the modprobe command in a separate playbook? got it.. cool 04:49:23 Your play where you do the modprobe vhosakot_ 04:49:30 need to split it into two 04:49:32 yes 04:51:15 done.. shall I run on 46 again britthouser? 04:51:27 you didn't make two modprobe command yet 04:51:42 if this didn't work try using shelll 04:51:45 shell module 04:51:52 I didn't mean put it in a different playbook 04:51:54 I moved modprobe and networkstart to separate playbook.. 04:52:01 haha.. then what? :) 04:52:05 I mean run two commands in same playbook 04:52:08 modprobe bonding 04:52:15 modprobe 8021q 04:52:20 ah ok ok 04:52:24 insert each kernel module as its own command 04:52:40 or use shell module ;) 04:53:04 sdake: yes.. we can shell module as well to run them on all nodes.. :) 04:53:08 even when I try to load both manually from shell I get error sdake 04:53:09 just adding to playbook 04:53:16 britthouser roger 04:53:54 * britthouser crosses fingers 04:54:07 clear to luanch on 46 04:54:18 cool.. yep.. lauching now on 46! 04:54:21 vhosakot_ a bit cavalier with that -rf operation ;) 04:54:34 haha.. yeah... :) 04:54:36 lol 04:55:13 +2 04:55:18 clear to launch on all 130 04:55:33 i'e learned not to use -rf because it results in thigns like rm -rf /var/lib/docker * 04:55:35 britthouser: does it all look good? 04:55:40 yes vhosakot_ looks good 04:55:49 just becuae i like typing extra spaces! 04:55:52 extra space there sdake ? 04:55:54 lol 04:55:59 sdake: yes.. yeah, I deleted evetying once using rm -rf.. 04:56:03 happend more then once 04:56:19 britthouser: should we ping the switch (the other end of bond0) ? 04:56:21 i use -r, but not -f 04:56:28 no vhosakot_ 04:56:59 I'm satisifed we'll have a clean run on the rest 04:57:00 britthouser: sdake ok! should I run on 47 as well.. or, shall we fire on 130 ? 04:57:10 do all 130 04:57:10 I feel good about doing the 130 04:57:19 time it plz 04:57:24 yep 04:57:31 should we reboot 46 and test britthouser sdake? 04:57:40 no...the 45 reboot was sufficient 04:57:57 dno't forget to remove your hardcode 04:58:00 host 04:58:08 britthouser: yes 04:58:21 sdake: inventory-ips-only is the right inventory pointing to 130 nodes woth 130 IPs, right ? 04:58:28 with* 04:58:32 vhosakot_ more or less 04:58:35 coo 04:58:36 cool 04:58:40 some nodes are ubuntu - we have those commented out 04:59:11 sdake: cool.. I do see 170 IPs in inventory-ips-only.. cool 04:59:25 britthouser: yeah.. I'll remove the hardcoded IP and change it to "all" 04:59:40 170 ips? 05:00:03 sorry 130 05:00:16 "cat inventory-ips-only | more | grep 172 | wc -l" returns 130 sdake 05:00:31 cool 05:00:44 britthouser: sdake: shall I fire for the entire cluster? 05:00:49 yes 05:01:05 You are clear for launch vhosakot_ 05:01:17 cool and thanks for adding the "time" sdake! :) 05:01:41 started!! 05:01:56 * britthouser britthouser is disconnected 05:02:00 ....j/k 05:02:02 noooo.. 05:02:03 :)_ 05:02:19 this is the power of a config mgmt tool like ansible.. :) 05:02:47 althought this would have been totally posisble using python, pexpect, TCL, bash... Ansible is very easy 05:03:06 I sorta wish you could make libraries out of plays. But the plays are so simple to write, its not that much effort to re-invent the wheel 05:03:17 right 05:03:43 done! 05:03:51 cool 2 min 05:03:53 dang....my screen is still scrolling 05:03:55 an d that oculd be highly optimized 05:04:01 * britthouser shakes fist at DSL 05:04:10 britthouser are yo urnning through the docker proxy? 05:04:10 true 05:04:15 i.e. 28.8k modem 05:04:15 LOL 05:04:18 did 130 nodes run in parallel? 05:04:24 its about that sdake. 05:04:26 vhosakot_ default is 8 threads i think 05:04:32 ah ok sdake 05:04:38 now I see its done 05:04:41 ok now i guess its tie to deploy 05:04:45 sheesh 05:05:08 sdake: should we add the OSIC inventory file to the kolla repo incase someone wants to deploy multinode with 100+ noes to scale 05:05:10 after we conigure globals.yml 05:05:11 sdake: let us deploy... 05:05:21 scenario 2 ? is the globals.yml ready ? 05:05:22 need to configure gloals.yml 05:05:25 So this is where my expertise drops off, but I'd like to watch 05:05:30 yup 05:05:32 let me drive 05:05:37 sure, please sdake 05:06:05 britthouser: sdake: are you sure we dont need to check networking on the nodes after the playbook ended? 05:06:53 I think we can do that in background while sdake forges ahead 05:07:10 nah i need help on globals.yml 05:07:13 sdake: I think we need to add this globals.yml and inventory we used in OSIC 05:07:26 vhosakot_ i am making a tarball of /root when we wrap up 05:07:27 sdake: neutron_external_interface shold be the interface without any IP, right ? 05:07:28 p4p1 == ens4f1 05:07:33 so we will have a full archive 05:07:36 sdake: cool 05:07:54 4 before the f sdake 05:08:09 sdake: copy from ip a | grep ens 05:10:33 doh 05:10:39 no apt-get? 05:10:43 yum 05:10:51 ah 05:10:54 cool :) 05:11:27 sdake: on which OS do you expect better kolla performance ? centOS or ubuntu ? 05:11:42 centos vs 14.04? 05:11:43 centos 05:11:47 more modern compiler 05:11:51 cool 05:12:09 was the ubuntu on the target nodes in OSIC 14.04 or xenial (16) ? 05:12:46 sdake: ^^ 05:13:09 sdake: password? 05:13:10 14.04 05:13:11 xenail we can't get to work 05:13:22 sdake: cool.. what about kolla-genpwd? 05:13:29 its already configured 05:13:33 cool 05:13:44 nice!! 05:13:55 sdake: how long will with deploy take ? 05:14:10 20 minutes typically 05:14:35 cool.. i'll be there.. after that, will you run your script that calculates time taken by each kolla-ansible operation ? 05:14:49 nah - i'll run a rally report 05:14:55 cool 05:15:00 that takes 6 hours 05:15:10 hopefully we will still hvae the cluster by then 05:15:25 Jeffrey4l_ need help 05:15:29 Jeffrey4l_ something failed in deploy 05:16:26 i've seen this before on the bootstrap playbook 05:16:41 when I add 'ip addr' 05:16:46 kernel is calling that interface bond0.731@bond0 05:16:58 maybe that what should be in the play? 05:17:05 sdake ^^ 05:17:08 sdake: are the ceph storage nodes setup for ceph ? ceph-osd failed on storage nodes 05:18:25 probably the rest there too 05:19:13 britthouser: cool catch on @bond0 05:19:20 britthouser is james bond :) 05:19:40 well don't thank me yet 05:19:45 could be red herring 05:22:54 sdake: what is wrong? 05:22:55 could the @ be throwing it off sdake? 05:23:05 who knows 05:23:08 maybe \@ 05:23:15 unlikely 05:23:16 * britthouser throws some darts at well 05:23:28 i saw the interface in a lower-up state 05:23:31 then it went to up 05:23:40 so i dont get how that happened 05:23:43 sdake: do you haven working gloabsl.yml with bond from ubuntu testing? 05:23:57 no 05:24:10 sdake: did you @ bond0in ubuntu 05:24:42 no ubuntu didn't have @bond0@ in the interface name 05:24:47 that is something rhel kernel is adding 05:24:51 britthouser: ah ok 05:25:11 can we see the error again sdake? 05:26:29 can we get the inventory of one of those nodes 05:27:10 moment thinking 05:28:32 no idea why its running that on .23 05:28:40 Jeffrey4l_ wtb some assisstance 05:28:48 Jeffrey4l_ i knwo your beat - i am too, the pain is almost over 05:29:01 whenever I've seen that error, its b/c the var I'm referencing isn't in teh inventory 05:29:15 maybe we can browse the inventory and see what it thinks that interface name is 05:29:18 how would bond0 be in the invetory? 05:29:29 intreface names are on globals.yml 05:29:45 no i mean where you say, "ansible tell me everything you know about this host" 05:29:51 maybe inventory isn't the right term 05:30:28 all interfaces in LOWER_UP.. is this good? 05:30:58 ansible -m setup 05:31:03 will dump the vars 05:31:20 sdake: can I rerun with verbose? -vvvv ? 05:31:40 you can - but you need to cleanup and relabel 05:31:42 who is driving atm? 05:31:47 not me 05:31:50 sdake: yes, doing it 05:31:54 runinng cleanuo 05:32:24 try ansible 172.21.0.27 -m setup -i inventory-ips-only 05:32:41 it shoudl be ansible_bond0.731 05:33:01 britthouser: ah ok.. I'll run it once the playbook ends 05:33:01 I sent us on wild goose chase with teh @bond0@ nonsense 05:33:02 it definately shouldn't e ansible_bond0 :) 05:33:15 right sdake 05:33:21 that is the variable name 05:33:28 what I mean is it shuldn't be prefixed with ansible_ 05:33:32 right 05:33:42 it should just be bond0.731 05:33:54 and then playbook should be referencing ansible_bond0.731 05:33:56 that is what we had originally 05:34:02 yeah. =/ 05:34:11 the playbook automatically references ansible_ 05:34:19 If you look at ansible 172.21.0.27 -m setup -i inventory-ips-only 05:34:20 hi 05:34:23 you can see all variables 05:34:59 i have sseen this problem and inc0 fixed it 05:35:02 but not sure what he did 05:35:09 britthouser: I ran it.. 05:35:11 I see ansible_bond0.731 05:35:18 so in etherpad 05:35:27 we have p4p1.733 being the cluster network 05:35:39 is that what storage_network is? the ceph cluster network? 05:35:57 where is the depploy -vvv output 05:36:06 britthouser: but, the storage node does not have any itnerface ncamed p4p1.733 05:36:12 sdake: further up 05:36:24 ok taking a break - keep a tit guys :) 05:36:27 sdake: see now 05:36:31 get Jeffrey4l_ involved - he can debug easily 05:36:43 sdake: this is the -vvv output 05:36:45 ok thx sake 05:36:48 sdake 05:36:55 i'll be back in 15 mins or so 05:36:59 sdake: cool 05:37:25 can we look at the error again vhosakot_ ? 05:37:33 britthouser: sure... 05:37:50 the error and debug (-vvvv) is in tmux session 0 window 2 05:37:54 window 3, not 2 05:38:17 ok taht's stil got teh @bond0@ in the name 05:38:20 lets revert that 05:38:24 and rerun 05:38:45 britthouser: cool.. let me rerun 05:39:31 britthouser: should I remove all @bond0, or just the storage network? 05:39:37 all the bond0 05:39:41 britthouser: cool 05:39:53 and make cluster_interface ens4f0.733 05:40:04 no wait 05:40:06 vhosakot_ 05:40:16 britthouser: ah ok 05:40:30 lemme just dobule check all these one sec 05:40:43 britthouser: sure, please 05:41:26 britthouser: is it good? 05:41:45 neutron external should be bond0.732 I think 05:42:03 unless the etherpad is out of date.... 05:42:44 let me change it to bond0.732 britthouser 05:43:11 ok give it a shot 05:43:54 cool 05:45:59 britthouser: now it complains that ansible_bond0.731 is not found 05:46:10 my screen still scrolls..... 05:46:17 which host/ 05:47:35 britthouser: all storage hosts complain the same.. same error 05:48:09 but taht is cleary in the variables for that host 05:48:15 Jeffrey4l_ got a minute? 05:50:17 britthouser: I'll add a debug in the ceph playbook 05:50:53 do wehave the correct vereson of anible 05:51:28 I'm printing all the variables in the playbook itself.. 05:51:32 yeah its sthe same deploy ndoe, right? 05:51:53 # pip list | grep ansible 05:51:53 ansible (2.1.1.0) 05:53:13 lets run yum upgrade on all the nodes 05:53:16 and reboot them 05:53:37 I'll run manually on .45 first 05:53:42 ok 05:54:00 yum -y upgrade should do it 05:55:02 yum update && yum -y upgrade 05:55:05 is what I did 05:55:29 yeah that should do 05:55:39 and then probably need touch /.autorelabel and reboot 05:55:41 I'm runnign with debugs in playbook 05:55:45 but I'll wait till yoru plays are done vhosakot_ 05:55:54 britthouser: cool 05:56:58 can i reboot vhosakot_ ? 05:57:06 just .45 05:57:21 might as well wait until its up 05:57:24 nvm on the autorelabel 05:57:25 and run playbook 05:57:33 selinux is disabled 05:57:45 ya we do that atm 05:57:57 although msater can run with xelinux enabled 05:58:11 if the disk is labeled properly ;-) 05:58:22 taht's what /.autorelabel is for. =) 05:58:35 i'd lilke something more surgical ;-)o 05:58:48 i.e. if you're been running with it off for awhile, or sometimes when your docker selinux policy changes drastically 05:58:54 we've hit both cases 05:59:18 ok, rebooting vhosakot_ 05:59:24 britthouser: wait 05:59:36 * britthouser takes finger off return key 05:59:59 britthouser: ok reboot... got for it! 06:00:10 I do see the variabel ""storage_interface": "bond0.731", " at runtime in the playbbok 06:00:22 so not like the variable is missing at runtime 06:00:44 could be nasible bug 06:00:53 i'd like Jeffrey4l_ to take a look 06:00:59 he would have a good handle on thedetila of this one 06:01:07 he is typically around this time 06:01:11 maybe at lunch 06:02:24 sdake, what's the bug? 06:02:45 Jeffrey4l_ ansible is spitting back it can't find a dict value 06:02:45 Jeffrey4l_ we are doing our laswt centos test for the osic cluster 06:02:56 but we easily find it in the host invenetory 06:02:58 Jeffrey4l_ and ansible is complaining about bnod0-731 06:03:00 britthouser, could u show me? 06:03:11 bound0.731 i mean 06:03:15 Jeffrey4l_: this is error 06:03:15 failed: [172.21.0.143] (item=ceph-osd) => {"failed": true, "invocation": {"module_args": {"dest": "/etc/kolla//ceph-osd/config.json", "src": "ceph-osd.json.j2"}, "module_name": "template"}, "item": "ceph-osd", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute u'ansible_bond0.731'"} 06:03:51 but then in the variables: 06:03:51 # ansible 172.21.0.143 -m setup -i inventory-ips-only | grep bond0.731 06:03:51 "ansible_bond0.731": { 06:03:53 "device": "bond0.731", 06:03:53 "bond0.731", 06:05:20 sdake: did you use bond0.731 as storage_interafec in globals.yml on ubuntu ? 06:05:33 vhosakot_ no idea 06:05:36 cool 06:05:56 maybe version mismatch in the fact caching 06:06:03 which sucks 06:06:03 hmm 06:06:12 ansible fact caching? 06:06:30 python 2.7.6 on the deploy node, and 2.7.5 on centos nodes 06:06:39 that's should be OK 06:06:46 Jeffrey4l_ ifyou log into the tmux0 06:06:56 i am in britthouser 06:07:09 sorry guys i can barely see straight 06:07:10 Ok.... do you want to drive for a bit Jeffrey4l_ 06:07:11 i removed the /tmp/ansible_fact_cache folder. 06:07:18 cool 06:07:22 I re-ran 06:07:26 * britthouser crosses fingers 06:07:47 mtaylor22, ping still around? 06:07:50 * vhosakot_ vhosakot has bitten his nails and fingers 06:08:12 one odd things that we think is just a red herring, the otuput of 'ip addr' shows 'bond0.731@bond0@' 06:08:25 somthing different the rhel kernel is doing 06:08:35 looks like is it working 06:08:42 it never went this far, right? 06:08:56 annoying 06:08:58 Jeffrey4l_: what is this /tmp/ansible_fact_cache folder ? 06:08:58 ahh...so since we ran ansible to create those interfaces 06:09:03 well let see if it runs rally now ;) 06:09:03 the cached inventory didn't have them 06:09:03 Please review https://review.openstack.org/#/c/346449 06:09:23 mewald i know you need reviews 06:09:26 we should add removing that to the cleanup. 06:09:36 mewald core team supe rbusy with osic - we have 6-12 hours left of time 06:09:38 there must some thing in ansible. 06:09:48 mewald then we are burning through the review queue - ok? 06:09:51 the fact cache should only keep 3600s 06:10:21 yes, britthouser is right.. I think we must delete the /tmp/ansible_fact_cache folder in the cleanup script 06:10:22 this is configuration in /root/.ansible.cfg, to speed up the deployment 06:10:23 is it still deploying? 06:10:24 sdake: yeah I am just pinging once in a while to remind everybody :) sure take your time! The chance to have OSIC must be used :) 06:10:30 sdake: yes.. 06:10:45 Thanks Jeffrey4l_ you da man! 06:10:46 it did not come this far at all.. it dies within a min 06:11:08 ya we need to ctrol -c this 06:11:11 remove the debug junk 06:11:14 sdake: why? 06:11:17 sdake: ok 06:11:23 so i can get a deploy time 06:11:25 shall I remove my debug as well and re-run? 06:11:30 yes pleae 06:11:33 my debug will add time 06:11:35 but don't git reeset 06:11:36 sdake: ok, will do it 06:11:39 sdake: cool 06:13:18 run deploy in script please 06:13:23 and save output 06:13:46 i nee to grab a 30 min nap - and then i'll caputre the data 06:14:03 sdake: which script? 06:15:55 Jeffrey4l_: yep sorry, I didn't have notifications enabled for irc 06:16:37 vhosakot_ and sdake 06:16:39 Jeffrey4l_ - any chance you can finishthe rally tests 06:16:45 should we yum upgrade and reboot all hosts 06:16:58 or just leave .45 as the sole one upgraded? 06:17:06 upgrade all please 06:17:23 here is what I'd liek 06:17:30 just out of curiosity, anyone had dramas with nova resize + ceph (volumes) not actually resizing the root fs? 06:17:53 upgrade lall nodes, reboot, run script (the command) to capture he deploy output (type exit to exit wscript) 06:17:58 sdake: the ceph cleanup complained that partion cannot be cleaned up and a reboot is needed.. hence, I'm rebooting the deploy node.. all tmux sessions will die 06:18:00 run it timed 06:18:08 no 06:18:14 that isn't the deploy node 06:18:18 that is the ceph nodde 06:19:06 all of the cluster nodes need to be rebooted 06:19:10 and deploy time caputred 06:19:24 jeffrey we are all supe tired here 06:19:47 Jeffrey4l_ any chance you could do what needs to be done her ewith the benchmarking? 06:20:14 sdake: I rebooted the deploy node.. the error showed the IP of the deploy node, not the ceph node 06:20:33 sdake: the deploy node is rebooted and back up fine 06:21:07 britthouser: I think all nodes must be upgraded 06:21:15 yes 06:21:24 yum update && yum -y upgrade 06:21:28 reboot 06:21:42 that's what I did on .45, and it went fine 06:21:42 sdake: which script must be used to deploy? 06:27:01 sorry need be out for about 2 hours. 06:27:23 i can take over this after back. 06:28:11 vhosakot_ cool thanks - i think Jeffrey4l_ can take it over nad i can get some real rest and hou gusy can go to bed :) 06:28:17 its like 3am there 06:28:37 sdake: yep.. going to bed.. good night.. thanks for all the help! 06:28:58 good night vhosakot_ 06:29:06 good night sdake 06:29:22 good night 06:29:35 Jeffrey4l_ - just to reiterate: all ndoes need to be yum upgraded, and then rebooted before deploy 06:31:33 Jeffrey4l_: you can add the commands to setup_centos_networking.yml in /root 06:59:16 Ok I'm off to bed everone 07:00:39 anyboday try sean k moony's boostrap-servers 07:01:55 ping Eduardo Gonzalez 07:03:14 he is not online zhubingbing 07:03:31 thanks 07:04:16 duonghq I found you always on line. 07:05:40 zhubingbing: our timezone is closer than American guys 07:15:47 Merged openstack/kolla: Dockerfiles for collectd Container https://review.openstack.org/358632 07:17:14 ping egonzalez90 07:21:04 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 07:23:26 morning zhubingbing 07:23:45 morning 07:24:03 https://review.openstack.org/349351 07:24:11 can you review it 07:24:51 hey guys 07:25:11 anyone have time to look into this ? https://bugs.launchpad.net/kolla/+bug/1617188 07:25:11 Launchpad bug 1617188 in kolla "neutron-ovs-cleanup missing" [Undecided,New] 07:30:19 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 07:32:38 zhubingbing, gnocchi_statsd_resource_id are random uuids or something like domain_id=default? 07:33:10 random uuids 07:35:26 https://github.com/openstack/gnocchi/blob/master/gnocchi/statsd.py 07:36:01 egonzalez90 you can read gnocchi code 07:36:54 You can look at this section of the code on the resouce_id ， link https://github.com/openstack/gnocchi/blob/master/gnocchi/statsd.py 07:43:15 * Daviey checks in 07:44:04 ok 08:09:22 Yao Lu proposed openstack/kolla: delete virtual bridge artifacts when cleaning hosts https://review.openstack.org/358924 08:30:37 ping egonzalez90 08:30:41 https://review.openstack.org/#/c/349351/ 08:30:50 can you review it ? 08:31:36 https://review.openstack.org/#/c/348388/ 09:23:17 Paul Bourke proposed openstack/kolla: Allow use of database for Horizon sessions https://review.openstack.org/359794 09:33:45 ping sdake 09:35:50 ping Daviey 09:41:00 zhubingbing: OSIC made them very tired now 09:41:33 － － 09:45:47 zhubingbing: hey 09:46:28 I have a question for rally that needs your help. 09:47:52 Rally install one step to initialize the database operation (rally-manage db recreate), this step operation, how should I add? 09:48:59 umm 09:49:07 But he's not as complicated as the other components, and he's just executing a command. 09:49:11 zhubingbing: sorry, can you phrase the question again? 09:50:02 rally-manage db recreate you think we should how to add for kolla 09:50:31 zhubingbing: I'm sorry, but i'm not understanding the question 09:50:36 Add rally role 09:50:39 - - 09:50:41 Oh! 09:50:52 You want to add a permanent rally role to kolla 09:50:57 yeah 09:51:01 i see 09:51:10 sorry, i thought you were trying to use rally 09:51:27 zhubingbing: so, why does this differ from other services? 09:51:32 zhubingbing: rally supports mysql, right? 09:51:46 yes 09:52:10 zhubingbing: so create the database, and then bootstrap rally-manage db recreate ? 09:53:58 zhubingbing: I mean, you just need to mimic this to create the database: https://github.com/openstack/kolla/blob/0ff90a6f10494f9f33d665a3367dfaf0486b94df/ansible/roles/watcher/tasks/bootstrap.yml 09:54:05 It seems as long as the implementation of this command rally-manage db recreate, do not need to create a database 09:55:43 If this is so, I have no doubt, but rally initialization database on a command, do not create their own database 09:59:08 zhubingbing: rally can use sqlite or mysql right? 09:59:22 use qlite 09:59:25 zhubingbing: I think it is reasonable for kolla rally role to use mysql 10:00:26 i see， you advice to use mysql 10:00:49 ok 10:15:07 zhubingbing: i think so... it makes sense to use the databases we already use 10:15:17 zhubingbing: and sqlite is going to be a pain 10:15:35 ok 10:15:37 good 10:29:09 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 10:35:33 Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 10:51:22 Hey Jeffrey4l_ - how did OSIC go overnight 10:56:57 britthouser, i got a success deploy 10:57:05 Hurray! 10:57:15 Thanks for taking over last night 11:09:36 britthouser, do u change anything in osic? 11:38:06 I have not Jeffrey4l_ 11:39:38 ok. thanks. 12:43:16 Jeffrey4l_: hey, did you see that nova change for vnc ip? 12:45:55 Daviey, yep. I have -2 for my PS. let's wait the nova bug is fixed. 13:03:43 Jeffrey4l_: Are you in a position to be able to validate the nova fix before it lands? 13:03:49 Jeffrey4l_: i sensed they wanted us to 13:04:54 Daviey, maybe when i am free :) 13:07:43 Jeffrey4l_: ok, let me know.. as i will try and do it later in the week if not 13:10:31 OK 13:32:58 Did a full kolla rebuild. Horizon master is broken. 13:33:13 Returning 500 for any query, even curl from localhost 13:34:07 traceback: http://hastebin.com/ezixuzeqil.vbs 13:52:54 mornin all... how goes? 13:53:42 irtermite: Morning 14:05:29 good morning 14:05:54 inc0_: Good morning_ 14:20:18 good morning inc0 14:23:23 sdake_, did you secure all the data we have on OSIC? 14:23:50 inc0_ yes just now 14:24:14 the /root directorry only 14:24:28 yeah I think that's enough 14:24:43 we need ton of time to analyze our results too 14:25:33 the sad part is centos was deployed lasst night 14:26:13 yeah, that was also bad organization from out side 14:26:35 kinda sucks that we got this at the end of cycle, but well, that's on me 14:27:01 sdake_, we need to tag this week right? 14:27:32 yup 14:28:17 https://review.openstack.org/#/c/358042/ then this is important 14:28:27 unless you want your repo customization 14:36:40 sdake_, I hope that we did copy globals.yml along with the tests? 14:36:57 in any case, please copy /etc/kolla too 14:45:08 mewald, ping 14:46:53 inc0_: ping 14:47:10 https://review.openstack.org/#/c/346449/23 I wanted to ask you about inventory 14:47:19 sure 14:47:33 so you created brand new group for monitoring, why? 14:47:44 why not use control:children? 14:48:23 and if it actually worth to create new group for it, why not add monitoring01 as example hostname? 14:49:45 Merged openstack/kolla: Customization for base https://review.openstack.org/358042 14:50:19 I don't really love the idea to running ElasticSearch and Kibana on the controller nodes. Imho they should only have to deal with OpenStack controller stuff. In my first patch I added a monitoring01 host to deploy to, but then someone gave -1 for adding new server types. So I ended up reusing the control group and only deploying to the first node, as I am currently not clustering influxdb and Grafana. 14:51:45 so this someone made a wrong call in my opinion;) I agree that monitoring can produce enough data to be on a dedicated node, however if we want to encourage that, we should 14:52:15 good :D I will go that way then. I like it better anyway :) 14:52:16 can we cluster influx tho? isn't this paid license? 14:52:30 what do you think about enable_performance_monitoring? 14:52:33 I'll make comment in review so you can blame it on me 14:52:46 ok ^^ 14:52:54 as single command to rule them all? that's ok, we did something like that for central logging 14:53:19 I think you can cluster influx using that relay service but it's very complex and I personally wouldnt need that kind of availability for monitoring 14:53:50 about enable_performance_monitoring: full ack, I have seen that for logging and kept the pattern 14:53:56 well, at least some sort of data replication across servers would be nice 14:56:32 berendt, ping;) 14:56:41 (I'm doing house cleaning, sorry guys) 14:58:14 berendt, https://review.openstack.org/#/c/350777/ will you continue? If not that's ok, but let me know, this one is extremely important for me. I can finish it off if you don't have time for it 15:12:08 hi sdake , i have deployed a openstack env in osic now. 15:12:21 boot a instance and pint that vms successfully. 15:12:26 *ping 15:13:11 Jeffrey4l_ tests take 6 hours to run 15:13:19 i think we have to give backgear now 15:13:21 But rally seem has some issue. I can not run rally test now. no idea the root cuase. 15:13:43 OK.. 15:13:56 can you copy /etc/kolla? 15:14:09 sure. copy to where? 15:14:17 anywhere - my computer is fine 15:14:48 ok. i will keep one copy in my local. and make a copy to you node.. 15:18:45 copied to /home/jeffrey/testresult/kolla 15:18:47 sdake, ^ 15:22:03 inc0_ take over if you need to finish it today, i'm end of business now 15:22:13 inc0_ i can take care of it tomorrow morning 15:22:13 berendt, more like end of week 15:22:34 inc0_ ok, than have a look on it tomorrow 15:22:41 but yeah, I'll fix it up and you can look tomorrow morning for any incomming reviews 15:22:48 ok? 15:22:57 we I'll co-author it if that's ok with you 15:23:08 inc0_ of course this if fine for me 15:23:40 ok, so let's do that, I think this is only customization patch left from core services 15:23:56 great 15:24:54 sbezverk: alive? 15:27:18 kfox1111: yes 15:27:48 I can take a quick stab at doing a deployment for novncproxy if you haven't started one. 15:28:18 kfox1111: when I switched to vxlan for tenants as well, now instances from both compute nodes have connectivity.. 15:28:33 very cool. 15:28:37 kfox1111: I have not started, I was busy with vxlan 15:29:01 so multinode breaths life. :) 15:29:51 kfox1111: oh yes.. the pulse is normal ;-) 15:30:05 hehe. but its a start. :) 15:31:35 kfox1111: about your ps for changing vncproxy.. I still think we need to use compute node's ip address.. the reason being, it is used in openstack documentation.. 15:31:37 Merged openstack/kolla: Vagrant doc updates https://review.openstack.org/353240 15:32:07 Jeffrey4l_: did you get a chance to deploy kolla on centos? 15:32:10 for which field? I'm absolutely positive on base_url's. 15:32:36 I've manually had to fix every novnc setup I've deployed with packstack. 15:32:50 I meant in the OSIC cluster 15:33:07 kfox1111: do you have your PS link handy? 15:33:14 yeah. sec 15:34:12 https://review.openstack.org/#/c/362449/ 15:36:21 kfox1111: here is what openstack doc says http://paste.openstack.org/show/564848/, we should probably follow it.. 15:36:57 looking. 15:38:24 yeah, I have vncserver_proxyclient_address overriden still in the PS. 15:38:40 and novncproxy_base_url is set to the controller in that example. which is what kolla should be generating. 15:38:55 same with vncserver_listen. 15:39:07 so I think that snippet is how we are configuring it. 15:39:21 vnc_listen is bound to ip address, we should change it to 0.0.0.0 as per doc 15:39:53 here is from your ps 15:39:54 crudini --set /nova/nova.conf vnc vncserver_listen $IP; 15:40:03 well, it shouldn't matter. for the container there are only 2 addresses. 127.0.0.1 and the eth0 one. 15:40:21 nothings going to touch it on 127.0.0.1 so they are basically equiv. 15:41:17 is that correct? 15:42:10 kfox1111: in this specific case you are right, but I think with 0.0.0.0 it will be more flexible for other possible confiugration 15:42:16 which we do not consider 15:42:24 true. 15:42:42 I mean it is more to be inline with opensatckj doc that is all 15:42:53 If its not specific to kolla-kubernetes, then I think it should be set in kolla's genconfig so we should just remove the line entirely. 15:43:17 fyi, the docs arn't aways the most security hardened of things 15:43:26 they focus on getting somethign working. 15:43:37 not nessisarily how you'd want to run it in production. 15:43:51 Michal Jastrzebski (inc0) proposed openstack/kolla: Customizations for Neutron https://review.openstack.org/350777 15:43:55 they have gotten better over the years though. 15:44:39 kfox1111: agree, but config should not force people, there should be a choice if they want to make it more secure.. 15:45:13 kfox1111: agree that it make more sense to address from genconfig 15:45:20 agreed. 15:45:34 sbezverk: uhm, what is this about? 15:46:02 Daviey: kubernetes things :-) 15:46:19 Daviey: we've been having to do some config overriding in kolla-kubernetes init as only kubernetes knows a few things like what ip the container has at runtime. 15:46:46 we're still coming up with the right balance of having kolla do almost all of the config generation and having kolla-kubernetes+k8s do a few overrides at runtime to get stuff working. 15:47:04 kfox1111: ah, i see 15:47:20 sbezverk: ok. so I'll remove vncserver_listen from my ps? 15:47:32 I thought it might have been in reference to vnc listen address being 0.0.0.0 15:48:03 Daviey: yeah. it is in this particular case. we're deciding whether to set it to 0.0.0.0 in kolla and not override it it k8s, 15:48:10 or set it to the specific ip of the container in k8s. 15:48:43 Daviey: I am for following openstack docs on this regard 15:49:00 and give people a choice if they want to change it 15:49:10 for this case, I don't think it matters much. 15:49:18 the only extra interface 0.0.0.0 gives is 127.0.0.1. 15:49:33 if its less code to maintain in kolla-kubernetes though, I'm for it. :) 15:50:09 sbezverk: oh, one thing 15:50:13 novncproxy_base_url = http://controller:6080/vnc_auto.html is wrong. 15:50:19 in two ways... 15:50:27 sbezverk: Hardened by default if possible is never a bad idea :) 15:50:33 1. the address is handed directly to a browser via horizon. 15:50:53 so it at minimum has to be http://:6080/vnc_auto.html 15:51:03 but probably should be a dns entry. 15:51:25 2. its a console where passwords tend to go across. it is really really important to use https for it. 15:51:39 either way, I think kolla genconfig shoudl create the value. 15:55:00 kfox1111: since you will be taken care of vnc, I will start working on cinder 15:55:21 k. 15:56:00 when you do cinder-api, can you use glance-registry as a starting point? 15:56:10 that will be one less thing I'd have to convert later then. 15:56:58 it should mostly be a copy and a sed 's/glance-registry/cinder-api/' 15:58:43 we really need to get conventions agreed to. its really hard to debug all this stuff when the names are so inconsistant all over the place. :/ 15:58:50 kfox1111, quick question...when you would want to expose libvirt over tcp? 15:58:56 live migration? 15:59:04 inc0_: personally? never. 15:59:30 I've only seen it configured to ssh tunnel the traffic from local to local. 16:00:01 thats why I added the switch to disable libvirt tcp. I don't need/want it in my env. 16:00:10 well, do you know if tunneled LM requires that? 16:00:38 I don't believe it does. 16:00:52 exposesd libvirt tcp is a security issue imho. 16:01:12 yeah I don't think there is any auth out there right 16:01:12 ? 16:01:21 as far as I know, no auth. 16:01:26 yay 16:01:41 then I'd go for "don't expose it at all" 16:02:04 on that note...let's see if Kolla exposes it over tcp 16:02:07 thats where I was at, but I gave up fighting it and opted for a config option to at least disable it. 16:04:25 well there is some auth 16:04:28 configurable 16:04:56 is it more then just plain text over the wire? 16:05:53 https://libvirt.org/auth.html hard to tell 16:06:09 kfox1111: ok I will use glance-apoi as base 16:06:26 I doubt it uses any encryption over network 16:06:31 uhh kolla suffers from it too 16:06:53 hmm.... its using sasl for basic username/password 16:06:53 inc0_ Jeffrey4l was advocating tcp socket as more efficient way 16:07:07 sbezverk, but much less secure 16:07:08 inc0_: sdake: i figured out why the bifrost change is broken. currently you cannt install mysql-server on any image built off of kollas base image on ubuntu 16.04 16:07:23 libvirt communication doesn't have to be very performant I think. 16:07:24 sean-k-mooney, duh...why is that? 16:07:29 secure on the other hand. 16:07:30 inc0_: hence config option, people after secuirty, can use it 16:07:57 to disable it, I do not care about secuirty, so I am after efficiency 16:08:04 sbezverk: -1. ok with a config option, but security shoudl be the default. performance for those at extreme scale that may need it. 16:08:21 btw sean-k-mooney https://review.openstack.org/#/c/359364/ mind taking a look? I answered your comment 16:08:28 "premature optomization is the root of all evil". 16:08:40 inc0_: http://paste.openstack.org/show/564851/ 16:08:47 and sure ill take a look now 16:08:51 kfox1111: I am fine as long as config option exists 16:08:52 on a small cloud, the performance overhad of it is unnoticable. the extra security is noticable. 16:08:58 k. 16:09:01 agree with Kevin here, everybody should be after security 16:09:18 and we don't require our deployers to know every config option to increase it 16:09:32 we should default to as secure as we can (within reasonable limits) 16:09:39 I'm ok with an option so long as its defaulted to security. if you want to turn it off, thats ok with me. 16:09:54 kfox1111: agreed 16:10:37 sean-k-mooney, can we use mariadb btw? I know you had issues with it, mind reminding me which were these? 16:11:07 btw I see issue with your configs guys:) 16:11:10 inc0_: i replace mysql with mariadb later but mysql-server is installed by bifrost and i dont think i can change it 16:11:27 inc0_: which issues? 16:11:28 you use "yes" "no" config type without ansible's | bool filter 16:11:50 inc0_: its j2 not ansible I think. 16:11:54 people might write true there and that would work with ansible 16:12:10 bool filter is in ansible unfortunately 16:12:15 yeah, we probably shoudl fix that. 16:12:29 I added another ansible like filter to the system already. 16:12:36 it woudl be easy to add a | bool 16:12:42 maybe we make a blueprint for it? 16:12:42 yeah, that's super simple one 16:12:50 inc0_: ill apply https://review.openstack.org/#/c/359364/ locally and test it with my 3 vm setup and give you my +1 if it works. i just want to be sure that it work when you have different barmetal nics on each host 16:12:52 bp will take longer than this filter;) 16:13:08 inc0_: filter yes, rewriting everything to use it, no. 16:13:18 sean-k-mooney, sure thing, just make sure to remove it from globals 16:13:23 and add it in inventoryu 16:13:41 kfox1111, fair enough 16:14:20 https://blueprints.launchpad.net/kolla-kubernetes/+spec/bool-filter 16:14:45 inc0_: yep that what i have currently as i have a ubuntu 14.04 vm, 16.04 vm and centos7 so the values are different and set in my inventory. 16:15:19 ok 16:15:40 btw sean-k-mooney this play took little over 5min on 130node osic:) 16:16:01 ok, I take that back, I need to optimize one thing that was horribly long:P 16:16:07 after optimization it was 5min 16:16:21 oh cool. and after it was done was it fuctional? 16:16:35 yeah 16:16:45 :) oh what was the slow part out of interest 16:16:48 issue is hostfile 16:16:54 ah 16:17:03 only controllers needs to be talking to each other 16:17:15 you did that for everything and it was N^2 connections 16:17:26 with 130 nodes it became...significant;) 16:17:49 yes that n*n in current implemntation. i guess its simple to restict that to nodes in the controler group only 16:18:15 yeah 16:18:25 that's what I'm going to do 16:19:00 inc0_: does it need to be done for any of the other services. its mainly for rabbit but does ceph or any of the storage backend require hostnames to be resolveable? 16:20:12 I don't think ceph even needs that now 16:20:20 afair it's only rabbit 16:20:55 cool well its easy to rexpand if needed 16:21:13 also I think we can optimize it yet again 16:21:24 create file once and copy over 16:21:46 that will lower the complexity to N 16:21:50 darn... gota rebase this... 16:22:31 yes though instead of a copy it would be better to append to the end of the existing file. 16:22:39 yeah, that's true 16:23:01 inc0_: what do you think of https://review.openstack.org/#/c/362443 16:23:39 inc0_: do you want me to do that? i can open a tech debt bug and submit a patch. 16:23:43 kfox1111, let me put it this way - make configs right >> crudini >> sed 16:23:43 I need to rebase the libvirt tcp one, and it would be much easier if review 362443 was in. 16:23:58 inc0_: agreed. 16:24:00 sean-k-mooney, I'll write it down later today 16:24:14 at least restrict to control one 16:24:19 this will make it managable 16:24:26 inc0_: if it looks good enough to you, can you +2/workflow it? then I can rebase the libvirt tcp one easier. 16:24:45 there ya go 16:24:56 inc0_: ok cool, for bifrost i need to overide https://github.com/openstack/bifrost/blob/1ddc801c65c8c8c8ba28caf9cc7f1e8ec3a6ad71/playbooks/roles/bifrost-ironic-install/defaults/required_defaults_Debian_family.yml#L13-L44 16:25:14 thx. 16:25:30 sean-k-mooney, btw...can bifrost work on sqlite? 16:25:37 I see little value in having mysql there tbh 16:26:00 inc0_: if ironic can work with sqllite the bifrost should be able to 16:26:17 bifrost jsut installs ironic in standalone mode. 16:26:29 worth to try later 16:26:36 seems like it can 16:26:43 would be much easier to deploy tbh 16:27:18 btw I'd be ok with seding this mysql installation off. I think requiring mysql is bad in the first place 16:27:23 inc0_: ya it would i just implmented as vanila a bifrost install as i code but im sure there is lots we can do to optimise 16:27:52 ofc, let's get it working, we still need our jobs following cycle;) 16:28:04 so let's not do everything today;) 16:29:40 inc0_: so for the mysql issue what do you think is the best approch. fix kolla base so installing works, overdied the package list in kolla so it uses mariabdb or submit a change to bifrost to change to mariadb? 16:30:23 in fact, best solution would be to submit change to bifrost to make this configurable, and for kolla use sqlite 16:30:43 setting up mariadb is complicated compared to just use local sqlite 16:31:38 yep thats an option too though i think we are pasted the ironic code freeze so not sure it can be done this cycle but i can ask in #openstack-ironic 16:32:59 we can do this next release imho 16:43:12 Merged openstack/kolla-kubernetes: Switch to crudini for easier readability https://review.openstack.org/362443 16:47:50 Michal Jastrzebski (inc0) proposed openstack/kolla: Limit hostfile generation to control nodes https://review.openstack.org/363028 17:10:47 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 17:31:03 sdake, can you plz take a look? https://review.openstack.org/#/c/360796/ 17:31:19 people keep asking same questions;) 17:33:17 hi sdake, I re-config my network, and re-do again now 17:34:04 just finish, going to run the init-runonce, do u have a copy that I can have a look? 17:34:09 hey cats- taking today off mostly - been grinding away on osic for 6 days straight- palan on sleepingtoday 17:35:16 sdake, https://www.youtube.com/watch?v=HKtsdZs9LJo 17:39:05 take rest sdake 17:48:50 hi sdaked, i got my vm workking, it can go out now,:) 18:16:16 wally2, what was the issue? 18:21:18 just network configure problem 18:21:39 ok I'm glad you got this fixed 18:22:10 sbezverk: I just tried firing up the nova-scheduler. 18:22:20 it fails because command: ["sh", "-c"] isn't a valid shell command. 18:22:35 it looks like this was debug code left in by mistake? 18:22:38 should I just pull it? 18:31:09 kfox1111: there should not be any sh other than in init container 18:32:10 kfox1111: yep, damn left overs :-( 18:32:16 will fix it 18:36:08 k. thx. 18:36:23 Serguei Bezverkhi proposed openstack/kolla-kubernetes: PS cleans up nova scheduler container from left over command https://review.openstack.org/363113 18:37:13 do we need 2 +2's in this case? or is it trivial enough/critical enough to just workflow now? 18:37:36 always 18:37:42 but just acked it 18:37:46 k. 18:38:38 ok, I need to make at least one commit in kolla-k8s, I have +2 powers - it is my moral obligation;) 18:39:08 :) 18:39:25 bool? :) 18:39:45 duh 18:40:27 or fencing script for ceph 18:40:45 ah. ceph support for k8s would be sweet. :) 18:40:57 well, it is our requirement right? 18:41:03 to have ceph pre-installed? 18:41:23 for now. we'd like not to have to require that long term. 18:41:37 have a way to use kolla-kubernetes to deploy ceph if you don't already have one. 18:41:58 but we want to keep ceph as requirement right? for HA 18:42:12 yeah. 18:42:14 btw I'm not super convinced if ceph should be deployed by k8s 18:42:21 not much value 18:42:31 I've got it running in k8s on my laptop. :) 18:42:36 using the kolla containers. :) 18:42:52 ok, but why, that's my question:P 18:43:02 it's not like osds will float around 18:43:08 why containers, or why k8s? 18:43:13 why k8s 18:43:23 because its one tool for an op to learn. 18:43:57 its easier to debug if you already have to learn how to debug things in k8s. 18:44:08 fair enough 18:44:48 but I don't think we' 18:44:53 we coudl do this without petsets 18:45:05 ceph.conf needs list of running mons 18:45:27 petsets or net=host. 18:45:40 well... daemonsets with net=host? 18:45:54 however, how will you know the IPs prior to scheduling? 18:46:18 I haven't tried much, but can you use dns? 18:46:33 a petset would work then. 18:46:34 I think ceph likes dns 18:46:48 yeah that's what I mean 18:46:49 if not, a daemonset + net=host would work as its the host ip which is static. 18:47:05 well, a petset won't work as it will float around though. 18:47:07 yeah, but not really k8s'y 18:47:14 with petsets at least mons can float around 18:47:15 it really really needs to be pinned to the particular host. 18:47:20 ah. true. 18:47:23 kfox1111: for sure 18:47:34 well... 18:47:44 although...not sure, it does have persistent data (monmap) 18:47:50 a petset mon would work only if you could persist the data 18:47:52 (hehe) 18:47:53 soooo...we can't put it on ceph;) 18:47:55 unless you have shared storage and it can be access from multiple compute nodes 18:48:08 so we need ceph to deploy ceph;) 18:48:11 deamonset + net=host then. 18:48:27 it should work in all cases. 18:48:35 tbh kfox1111 I'm not big fan of using k8s just to use k8s 18:48:40 kfox1111: why do you need daemonset? 18:48:47 if something doesn't make sense to be ran on k8s, I'd not run it on k8s 18:49:02 sbezverk: so you can alwys land a container on a host where the data resides. 18:49:21 what about using labels and nodeselect? 18:49:31 I guess you could do it with a host selector label on a deployment too and just do one deployment per mon/osd 18:49:32 you mark compute with label, say compute 18:49:37 :) 18:49:43 sorry I meant storage 18:49:46 and that is it 18:50:03 daemonset + hostlabel + net=host, yeah. 18:50:17 not that I against daemonset, I love them :-) 18:50:24 sooo....you'll reimplement ansible in k8s 18:50:26 not a fan. 18:51:04 inc0_ do not you think that ansible actually is kind of kube killer? 18:51:20 because ansible cannot keep up with dynamic nature of kube 18:51:44 the issue with it is, requiring ansible when you only need it for the last 2%. maybe that's ok, but it is a potentially largish cost for an op to learn. 18:51:48 sbezverk, there are services that can't keep up with dynamic nature of kube:) 18:51:49 example you want to scale up, do you need ansible for this in kube? absolutely not 18:51:52 ceph being an example 18:52:10 kfox1111, I was rather thinking...you also need to deploy k8s 18:52:19 so if you use ansible to deploy k8s itself + ceph 18:52:22 that makes more sense to me 18:52:23 inc0_: other way around I think. kube 1.0 wasn't designed to handle it. they are working on growing backwards to support more pet kind of things. hence daemonsets/petsets. 18:52:48 inc0_: ansible is overkill I think to deploy kubernetes. its really easy to do. 18:53:05 heat template or kickstart file and good to go. 18:53:11 rhallisey, ^ ;) 18:53:34 kolla-kubernetes can deploy the rest on top. 18:54:46 sooo...I think you need to tell kubelets where kube mons (or whatever it's called) are 18:54:52 which means you need orchiestration 18:54:55 no. 18:55:07 it detects it? 18:55:15 ceph support in k8s is done at the pod or pvc level. 18:55:20 Merged openstack/kolla-kubernetes: PS cleans up nova scheduler container from left over command https://review.openstack.org/363113 18:55:26 you don't have to preconfig the kubelets. 18:55:41 in fact, because of that, you can have one kube system target multiple cephs, potentially one per tenant. :) 18:56:39 I think ceph externally is perfect. If you want to deploy with ceph and have it manged by kube go ahead 18:57:02 I don't see a big gain in having k8s manage ceph 18:57:13 other than convince 18:57:21 thats ok. :) 18:57:31 but that's not a reason not to have it 18:57:33 I may take a stab at it after a bunch of other stuff is working. :) 18:57:47 but my clouds all have ceph external, so I can wait. 18:59:08 rhallisey: what about secret generator, inc0 is pushing for integrating it with password gen tool of kolla. Does it make sense to merge these two? I think they should be separte. 18:59:41 depends, is kolla moving that out of kolla-ansible? 18:59:42 cuase the same password gen will be used for ansible as well 19:00:02 I could go either way. but right now, it kind of feels like a premature optomization? 19:00:13 rhallisey, I don't believe it is 19:00:15 what there is now is fine 19:00:35 especially I do not see anybody taking to actually do it ;-) 19:00:37 inc0_, Ideally kolla-kube would consume kolla-host, kolla-config, and kolla 19:00:44 and kolla-ansible would do the same 19:01:06 unless kolla-kube and kolla-ansible are intertwined 19:01:21 but kolla-ansible doesn't need anything from kolla-kube 19:01:38 I think kolla-kube will have issues gaining traction if it depends on kolla-ansible. 19:01:48 ya I agree 19:01:52 rhallisey: exactly my point too 19:03:15 well genconfig is a scrtipt that sits in kolla 19:03:16 not ansible 19:03:25 so I see no reason to not consume it 19:03:43 we already established that passwords.yml and globals.yml will look more or less similar 19:03:45 to me, there are 4 parts to the equation. 19:03:46 ya but we talked about splitting out config 19:03:59 workflow, config, deployment tool, containers. 19:04:00 yeah I hope config split will happen too 19:04:03 probably next release 19:04:08 Sandhya Dasu proposed openstack/kolla: Removing development packages from Baremetal install defaults https://review.openstack.org/363138 19:04:11 kolla, kolla-config, kolla-host, kolla-ansible, kolla-kube 19:04:20 kolla-ansible tends to do three things, workflow engine, config and deployment. 19:04:33 well I wouldn't make 3 projects 19:04:35 kolla-kubernetes should minimially do just deployment and containers. 19:04:42 rather kolla-config and kolla-host will be simply kolla 19:04:46 I plan on only using it in that mode for a while. 19:05:12 I've got system configs already and will use genconfig once and overwrite the configs with my own for now. and do manuall workflow. 19:05:13 inc0_, kolla-host would have docs/scripts to setup kolla-kube and kolla-kube would reuse the bm 19:05:21 kolla-host could be merged with kolla-config 19:05:39 next step will be getting configgen to be able to produce configs like mine. then I might replace my hand crafted ones with kolla configgen ones. 19:05:52 and last would be workflow, once that matures enough I'm confortable with. 19:06:08 but just like the containers are seperate from ansible, I kind of think kolla-kubernetes should too. 19:06:20 workflow is a different step that can be done by many differnet tools. 19:06:24 btw since kolla-host is well...ansible;) 19:06:29 my preference I think would be ansible too. but, 19:06:32 we can keep it there 19:06:36 it could be mistral, or any number of other things. 19:06:48 kfox1111: generating good config for kube would be huge work, right now there is no a signle service which would have its configs consumable by kube without manual editing 19:06:49 inc0_, but it's a different step in the deploymen tprocess 19:06:58 different playbook 19:07:01 like it is today 19:07:04 1) vm deploy 2) configure 3) deploy 19:07:10 sbezverk: yup. I've got it mostly working though. :) 19:07:10 s/vm/bm/ 19:08:41 inc0_ are you ok now for secret generator? 19:09:01 sbezverk, I just don't want to have x scripts doing the same 19:09:03 thing 19:09:08 it's not scallable 19:09:26 inc0_: agreed. its not actually generating secrets though, or shouldnt. 19:09:35 it should just take the ones created by kolla and upload them. 19:09:42 its just a re-formatter. 19:10:08 from yaml to whatever-k8s-likes? 19:10:12 as the secrets it uploads into k8s should match the ones in the configs kolla generates. 19:10:22 that's ok 19:10:36 I just dont want x generators and possibly keep same config files 19:10:58 +1 19:16:32 ok... got nova api services working on my test box. :) 19:18:56 Kevin Fox proposed openstack/kolla-kubernetes: Make libvirt over tcp an option https://review.openstack.org/362457 19:19:53 kfox1111 - nova + ovs is a huge accomplishment 19:20:03 much further then we got in 9 months of doing kolla work initially 19:20:22 yes, but getting the containers there was a huge amount of work too. don't sell that short. :) 19:20:47 ya or the structure of the containers so they are esaily consumable 19:20:56 yeah. 19:21:52 Kevin Fox proposed openstack/kolla-kubernetes: Nova Services https://review.openstack.org/363152 19:22:21 ok. with that one, I can replace my on host nova-api with kube-k8s container. 19:23:57 kfox1111, fix your git;) 19:23:59 inc0_: oops 19:24:02 yeah. :) 19:24:31 I'm usually so careful of that too. :/ 19:25:25 Kevin Fox proposed openstack/kolla-kubernetes: Make libvirt over tcp an option https://review.openstack.org/362457 19:25:26 ok. take 2. 19:30:09 there is 1 project - deliverables inside a project 19:36:05 sdake: o/ this is whats breaking the bifrost container build https://github.com/openstack/kolla/blob/master/docker/base/sources.list.ubuntu#L14 19:36:35 sean-k-mooney hrm 19:36:55 sean-k-mooney there is an internal mirror f or mariadb now 19:37:36 can I get some more eyes on this: https://review.openstack.org/#/c/361186/ 19:38:32 sdake: bifrost is trying to install mysql which i replace with mariadb at a later date but because we enable that repo it case the install of mysql-server to fail in bifrosts install playbook 19:39:20 sdake: ubuntu provides mysql-server-5.7 this is enableing mariadb version 10 19:40:46 can bifrost run without mariadb 10? 19:41:27 biforst is not enableing mariadb 10 kolla is 19:41:30 if so - ust remoe the mariadb repo file at the stat of the container 19:41:44 start of container build i mean 19:41:56 oh am well i had issue using mysql instad of kolla in the past but i can try that 19:42:11 /kolla/mariadb/ 19:42:29 maybe removal needs to be conditional 19:42:43 or maybe there is a way to versoinpin on ubuntu 19:42:49 i am not an ubuntu expert unfortunately 19:43:45 sean-k-mooney in the gate log where is the output which makess you think that is the problem? 19:44:02 http://paste.openstack.org/show/564851/ 19:44:13 sean-k-mooney - btw tested bootsstrap - works weell 19:44:20 i ran it locally and found the real error. 19:44:22 sean-k-mooney we got a few patche up for bootstrap 19:44:35 sdake: inc0_: has a few tweaks to it to make it faster 19:45:14 sean-k-mooney its already fast - 5-7 minutes 19:46:12 sdake: mainly around how the hosts file is generated. currently its O(n*n) but it can be made O(1) 19:46:21 yes i saw his change 19:47:11 we were in tmuxtogther when it was discussed 19:47:23 whatever numbers we produce we have to multiply by 9.2!! 19:48:37 dave-mccowan interesting result with tls no? 19:49:10 sdake, it wasn't 5min 19:49:17 I hacked it to be 5min 19:49:23 why 9.2? 19:49:25 now I proposed my former hack to upstream 19:49:27 ran again o centos 19:49:39 sean-k-mooney, inside joke, I'll tell you over beer 19:49:46 ok 19:50:00 beer good :) 19:50:12 cervesa con limon - I'm drinking that in Barcelona 19:50:25 and ciara 19:50:33 man I love this city 19:51:19 sean-k-mooney, btw I just limited hostname generation to controllers for now 19:51:24 quick fix and get's us there 19:51:30 we'll improve later when we need 19:51:57 sean-k-mooney, https://review.openstack.org/#/c/363028/ 19:53:22 cool ya the change makes sense to me 19:54:26 sdake: ill try to revert to mysql instead of mariadb and disable the mariadb repo in the bifrost containter for now and see if that works. 19:55:13 sdake: diableing the repo allows mysql-server to be installed but im not sure if i will hit the service file issue i had before. 19:56:04 sdake: im heading home to grab something to eat but ill try and test this soon and push a patch tonight with a fix 19:57:54 before i run did support for using msdos partitions for ceph ever land? 20:12:49 sean-k-mooneyAFK souns good 20:17:05 damn internet, dropped right in the middle of message I typed, appologies.. 20:21:54 irtermite thanks for access to the osic gear - I am pretty sur we had everything we neededd around 8am 20:22:08 irtermite so feel free (if you haavent already) to take back control :) 20:35:48 Merged openstack/kolla: Removing development packages from Baremetal install defaults https://review.openstack.org/363138 20:40:35 sdake: much thanks! it's still yours for the next 2-3 hours or so. All good! 20:40:42 glad it worked out for you all 20:42:46 irtermite: Great. deploying bitcoin mining now 20:43:51 hah 20:44:36 * irtermite blocks port 8333 20:45:03 Merged openstack/kolla: Customizations for dind https://review.openstack.org/353203 20:45:49 Daviey: ^^ 20:46:26 :) 20:52:35 Daviey 130 nodes of that gear of bitcoin softare would make yo uabout 2 cents a day 20:52:50 Daviey i had a knc 600mhz miner - was makingabout 400bucks a day for the first month 20:53:05 asic based 20:53:10 sdake: My mum always told me.. look after the pennies and the pounds will look after themselves 20:53:23 Daviey gret idiom ;) 20:53:38 Daviey i wish i coudl get my wife to stop spendin money 300 bucks a crack 20:53:49 300 bucks 300 bucks adds up over a month 20:54:05 sdake: I get my crack much cheaper 21:04:08 Hi All, my bootstrap_osd.yml seems to run a command "docker exec -t kolla_toolbox sudo -E /usr/bin/ansible localhost 21:04:09 -m find_disk" which is requesting password for ansible, do we need to create seperate user as ansible? I am trying to deploy ceph with kolla on a single node 21:19:52 Merged openstack/kolla-kubernetes: Script generating kubernetes secrets from passwords.yml https://review.openstack.org/354199 21:35:55 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 21:57:58 Kevin Fox proposed openstack/kolla-kubernetes: Make libvirt over tcp an option https://review.openstack.org/362457 22:10:24 kfox1111: ping 22:17:20 Merged openstack/kolla: Architecture guide doc https://review.openstack.org/360796 22:26:10 Merged openstack/kolla: Add defaults for interfaces to all.yml https://review.openstack.org/354173 23:20:35 sean mooney proposed openstack/kolla: generate bifrost yaml configs https://review.openstack.org/349684 23:20:36 sean mooney proposed openstack/kolla: add bifrost deploy ansible support https://review.openstack.org/349685 23:20:36 sean mooney proposed openstack/kolla: introduce bifrost-deploy container https://review.openstack.org/319703 23:20:37 sean mooney proposed openstack/kolla: adds bifrost ssh key generation https://review.openstack.org/350692 23:20:37 sean mooney proposed openstack/kolla: add deploy-servers command https://review.openstack.org/349729 23:40:47 kfox1111: I see issue with haproxy http://paste.openstack.org/show/564919/ 00:10:19 Li Yingjun proposed openstack/kolla: Add option to support dhcp/l3 agent ha https://review.openstack.org/352180 00:17:33 Li Yingjun proposed openstack/kolla: Add option ceilometer_database_type for ceilometer https://review.openstack.org/345881 00:27:28 Daviey around? 00:27:42 anyone know how many mons we were running in our tesst scenrio 00:29:21 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 00:55:23 morning 01:10:24 Duong Ha-Quang proposed openstack/kolla: Add ansible raw task note, add apt-get update https://review.openstack.org/362579 01:17:50 hello 01:51:33 sdake: have you done with OSIC? 01:51:41 yup 02:00:42 sdake: can you take a glance at this bp implement: https://blueprints.launchpad.net/kolla/+spec/ansible-specific-task-become 02:01:03 duonghq tbh - I am feeling a little troubled by all the late submisssions to the code base 02:01:15 duonghq i'd sort of like to wait until after newton t omerge that 02:01:47 especially on a change like that - i thought it would be fairly straightforward a coupel weeks ago 02:01:59 sdake: roger 02:02:03 but it has not been tested or reviewed enough from my pov for a major architectural change 02:02:23 after we branch - seems good to me to have this kind of functionality 02:02:36 not really much change here, but agree with you 02:02:46 too tight schedule for this kind of change 02:03:00 ya but atleawst the work is not wasted - it will ust go into next cycle 02:03:45 but after other review, I wish that you can take a brief look at my work, I need some comment about my direction 02:06:39 will do 02:07:03 thank you, sdake 02:13:09 well duonghq i've got to the first become patch 02:13:09 and my brain is not cooperating 02:13:11 too tired 02:13:18 can I have a look tomorrow instead? 02:13:28 it's up too you 02:13:31 thank you very much 02:42:52 sdake: can you change the milestone target of my bp? due to the work is defered to next cycle 02:45:02 duonghq yup but dont give up on it 02:45:04 k? :) 02:45:30 ok, I will follow this bp until it's finished 03:11:50 sdake: as I understand, the baremetal role is one of a newton feature? 03:12:02 duonghq roger 03:12:31 duonghq i you want t o shift focus to that or a couple weeks wfm ;) 03:12:44 that is our #1 most important deliverable 03:13:02 i you/if you 03:13:10 third party blueprints is #2 03:13:17 plugins rather 03:13:20 * sdake brain cooked 03:13:45 I think next cycle we need factor out some of sean-k-moony's work to more general role, it's really valuable work 03:14:08 yup it needs more work 03:14:13 but its a really good starting place 03:14:52 i'd like to see the tig stack working as well 03:15:16 I'm sorry but what is "tig stack" 03:15:20 we have until the 15th to choose our features wisely we intend to keep in the release for newton 03:15:31 telegraph, influxdb, grafana 03:15:33 mewald working on it 03:15:48 hi sdake 03:15:58 i have a probleme 03:16:36 ping 03:16:42 ping sdake 03:16:46 interesting stack, but what is it for? or just prepare for other feature? 03:17:35 Aodh log need to add to Heke？ 03:18:30 duonghq its to present performance metrics of the system 03:18:59 for central logging facility? 03:19:57 duonghq - it doesn't integrate with elasticsearch 03:20:05 given that influxdb dropped open-source official support for distribute deployment (iirc), I don't really like it. But it's still good time-series db 03:20:08 instead telegraph integrates with influxb 03:20:32 duonghq mind gonig into more detail there 03:21:02 you mean influxdb has no ha? 03:22:47 sdake: give me few second, looking for the annoucement 03:22:59 but I don't mean no HA, just no distributed 03:23:15 what is distributed deployment 03:23:36 ah, my fault, no clustering 03:23:45 ya clustering typically means ha 03:23:58 ha, lb, etc... 03:24:08 the duty is pushed to community 03:24:19 #link https://docs.influxdata.com/influxdb/v0.13/concepts/clusters/ 03:26:36 ping hiking 03:26:42 ping huikang 03:26:51 Hi 03:26:55 zhubingbing 03:27:02 * sdake angered 03:29:08 Duong Ha-Quang proposed openstack/kolla: Add Ansible become to quick start guide https://review.openstack.org/358374 03:29:32 I want to talk to you about etcd. 03:30:25 sdake: the original HA feature is fork at https://github.com/influxdata/influxdb-relay but there is no much attention here. 03:44:32 duonghq that is no beuno 03:50:30 sdake: it seems that checking if user has access to docker daemon is complicate 03:50:50 duonghq ya - docker version and parsing output is viable 03:51:23 could bindmount /run and check the socket file directly 03:51:30 sdake: so, we can only know if user can connect to docker but not know the real reason, 03:51:33 that is what docker versoin does 03:51:49 sdake: the dockerd has option to run docker at other sock 03:51:56 *socket 03:52:08 duonghq yup - another option is to mount /proc in the container from the host and parse it looking for dockerd 03:52:21 a third failure scenario - tls is enabled 03:52:32 unfortunately error reporting in ansible is terrible 03:52:39 many edge case here 03:52:45 it either works or doesn't work - no rationale given as to why :( 03:53:40 and, currently, we only support connect to docker at default sock @ localhost 03:54:50 zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 03:55:12 Md Nadeem proposed openstack/kolla: Added Ansible playbook for congress deploy https://review.openstack.org/350629 03:58:06 sdake: I think it is rationale to check default socker due to it is only support configuration in Kolla right now 03:58:11 *socket 03:58:36 have hard time seing someone configure the socket location differently then the default in a kolla deployment 03:58:42 machines willbe purupose built for the job 03:58:49 (of running openstack) 03:59:55 sadly but I do not have much experience with deploy docker 04:00:02 (in production) 04:00:14 ya - i think not checking seems fine personaly 04:00:25 notssure hwy that is in that patch set - seems orthogonal to the problem your tryign to solve 04:01:16 yup, just a try to clarify fail reason for user 04:01:37 should probably be a precheck rathe then a runtime check 04:01:56 I think the 'docker version' in precheck is acceptable for now 04:02:05 yeah, I do the user check in precheck stage 04:02:10 cool 04:02:28 I already here but I have trouble when see it first time 04:03:01 for user familiar with docker, it's not surprise, of course 04:04:57 I'll save the user check of later decision 04:05:00 *for 04:25:10 hey pbourke 04:25:15 pbourke - quick q if you dont mind 04:25:26 pbourke do you have any idea how many mons were running in our test sceanrios 04:25:32 or Jeffrey4l_ ? 04:36:42 sdake if you have the inventory file 04:36:49 see which nodes were set has ceph-mon ? 04:37:02 sup coolsvap_ 04:37:07 britthouser i do - there were 20 storage nodes 04:37:19 i think there is multile ceph mons deplyed pernode hto 04:37:21 but not sure 04:37:37 sdake: hi 04:37:39 So with ceph there is one OSD process per drive that you alloate to Ceph 04:37:49 But it looks like from ansible/inventory/multinode 04:37:58 that ceph-mon runs on control node 04:38:14 [ceph-mon:children] 04:38:15 control 04:38:21 which is pretty standard 04:38:29 y ou don't typically run mon and OSDs on same node 04:38:42 so if you had 5 controllers, then you had 5 ons 04:38:44 mons 04:38:47 sorry i meant osds 04:38:53 Matthew Taylor proposed openstack/kolla: Fix nova_ssh container shell to allow instance resizes. https://review.openstack.org/361891 04:39:02 Does that answer your question? 04:39:12 ya 20*16 osds then 04:39:19 correct 04:39:44 MON process typically needs same quorum requirements as galera and other processes on control nodes 04:39:50 so it make sense to place them there 04:39:57 i.e. need an odd number 04:40:53 heh 04:44:05 and on that bombshell, I'm headed to bed. =) 04:46:20 mtaylor22 - check out reiew in your ptch 04:47:08 yeah jsut saw it 04:47:26 i thought about doing it in the dockerfile too 04:47:40 bit i figured i'd have a bit more control in the extended start script 04:47:44 much better place to do it 04:47:46 then its permanent 04:48:02 mmmmmmm hokay, easy fix anyway :P 04:48:05 one moment 04:48:48 just testing locally first 04:49:15 coolsvap_ caan you help me clean up launchpad in prep for releasing newton3 tomorrow? 04:50:32 sdake: yes will do 04:50:55 coolsvap_ how youc ould help is by moving bugs that dont seem critial to occata 04:51:11 coolsvap_ i.e. breaks kolla in some way 04:51:23 i'll be moving blueprints shortly 04:52:05 or maybe i'll be moving the blueprints tomorrow 04:52:12 i am super beat from the beating on osic 04:52:29 sdake: will do 04:54:16 sdake, coolsvap_ , please review it when get time : https://review.openstack.org/#/c/350629/ 04:56:52 Matthew Taylor proposed openstack/kolla: Fix nova_ssh container shell to allow instance resizes. https://review.openstack.org/361891 04:57:47 oh wait 04:58:29 sdake: i figured its best only in the nova-ssh dockerfile rather than base (ie. only change where required) 04:58:30 brb sec 04:58:50 mtaylor22 - right only changei n nova_ssh container 05:00:45 mdnadeem_ looks good :) 05:00:52 mdnadeem_ nice job! 05:01:00 sdake, Thanks :) 05:01:11 didn't persoanlly test it of course 05:01:19 but implementation looks correct 05:05:42 time for bed 05:05:45 night all :) 05:05:59 sdake, good night 05:06:36 night sdake 05:08:22 sdake: i tested it locally lol 05:43:22 mtaylor22 does that nova ssh container resize operation now work for yu? 05:43:41 yeah it does ever since i updated the shell 05:43:46 with nologin = no go 06:13:54 Li Yingjun proposed openstack/kolla: Add option to support dhcp/l3 agent ha https://review.openstack.org/352180 06:31:25 hello 06:31:30 who 06:32:54 who know what is add releasnotes command? 06:33:42 zhubingbing_: guide: http://docs.openstack.org/developer/reno/usage.html 06:34:02 thanks 06:34:08 i forget it 06:49:09 Duong Ha-Quang proposed openstack/kolla: Check if ansible_user is in docker group https://review.openstack.org/358432 06:59:39 morning 07:16:49 morning o/ 07:48:19 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 07:50:27 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 08:04:31 zhubingbing proposed openstack/kolla: repair rally dockerfile https://review.openstack.org/363466 08:10:22 zhubingbing_, ping 08:10:30 here 08:10:46 i wil change rally 08:29:26 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 08:31:30 Li Yingjun proposed openstack/kolla: Add option to support dhcp/l3 agent ha https://review.openstack.org/352180 08:35:58 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 08:39:04 hi , everyone , I failed to build rabbitmq image. 08:40:21 ERROR:kolla.image.build.rabbitmq:The command '/bin/sh -c rm -rf /var/lib/rabbitmq/* && curl -o /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/plugins/rabbitmq_clusterer-3.5.x-189b3a81.ez http://www.rabbitmq.com/community-plugins/v3.5.x/rabbitmq_clusterer-3.5.x-189b3a81.ez && /usr/lib/rabbitmq/bin/rabbitmq-plugins enable --offline rabbitmq_management rabbitmq_clusterer && /bin/true' returned a non-zero code: 23 08:42:57 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 08:45:50 try agin 08:46:20 harley 08:49:01 again 08:55:45 Paul Bourke proposed openstack/kolla: Allow use of database for Horizon sessions https://review.openstack.org/359794 08:56:24 egonzalez90: any chance you could review https://review.openstack.org/#/c/337594/ ? 08:57:33 egonzalez90: needs to approve for tagging today 09:08:31 coolsvap_: ^ 09:09:06 pbourke: ack 09:10:52 pbourke, doing it 09:10:57 thanks guys 09:20:34 if ye have time to review https://review.openstack.org/#/c/319703/ also that is the only change blocking bifrost support form merging. all the rest depend on it but are already approved 09:21:40 pbourke: im glad that ceph check is merged though. since we were labeling partions i did find it non intutive that we were going to nuke the whole disk at first 09:22:05 sean-k-mooney: cool cheers, yeah your input confirmed to me that we needed it 09:22:44 sean-k-mooney: a manager in oracle was freaking out when they found out the possibility of nuking a disk ;) 09:23:14 thats the joy of linux 09:23:55 at least windows won let let you nuke the os while its running 09:24:50 actully maybe you can do it with diskparted... 09:25:18 * sean-k-mooney fight uerge to test for science on work laptop... 09:29:23 duonghq: can you add explanation why https://bugs.launchpad.net/kolla/+bug/1618722 is incomplete or do you think it is addressed with https://review.openstack.org/#/c/363466/ 09:29:23 Launchpad bug 1618722 in kolla "repair rally dockerfile" [Undecided,Incomplete] 09:30:43 sean mooney proposed openstack/kolla: introduce bifrost-deploy container https://review.openstack.org/319703 09:32:29 pbourke: the latest version should adress your commet. 09:32:46 got to run for a bit. will be back in 30 mins or so 09:35:09 coolsvap_, zhubingbing_: I think zhubingbing_ should give little more detail on that, but it's ok, maybe I miss understood his idea at that moment, sorry 09:35:38 duonghq: np, just wanted to check if you had any other things in mind 09:35:53 i am just updating the bugs 09:36:00 sorry again, zhubingbing_, coolsvap_ 09:36:11 duonghq: no need for sorry :) 10:28:53 pbourke: do you think https://bugs.launchpad.net/kolla/+bug/1604602 can be resolved? 10:28:53 Launchpad bug 1604602 in kolla "No build output in deploy gates" [Medium,Triaged] 10:29:20 coolsvap_: im sure it can 10:29:31 coolsvap_: I dont have time right now to look into it though 10:29:42 pbourke ack 10:29:48 hi 10:29:50 i will update 10:30:45 coolsvap sean－k－mooney can you help me review for rally 10:31:06 https://review.openstack.org/#/c/363466/ 10:31:28 https://review.openstack.org/#/c/363456/ 10:31:43 pbourke 10:49:54 ping sdake 10:50:01 hi 10:51:50 ping sdake 10:52:01 dude its 4am chill out 10:52:03 whats up 10:52:09 wa 10:53:15 barbican role need you review it 10:53:34 can you help me review it 10:55:37 need brain toboot 10:56:31 sure 10:57:29 Sorry, I always forget the time zone. 10:57:30 － － 11:00:57 coolsvap_ around? 11:04:41 sdake: yeah 11:04:50 coolsvap_ thanks for the list of bugs 11:04:57 coolsvap_ mind moving the rc1 bugs over? 11:05:02 sure will do 11:05:07 coolsvap_ looks like there is plenty 11:05:10 in hour or so 11:05:12 yes 11:05:15 i'll create a palceholder o1 11:05:21 ya no rush 11:05:27 and there are still in progress bugs 11:05:34 lee those as is 11:06:17 yeah 11:11:27 coolsvap_ this needs a review if you can get ot it - or egonzalez90 https://review.openstack.org/#/c/350692/5 11:18:00 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 11:22:05 sdake: bugs moved to rc1 11:34:40 tx 11:34:51 sbezverk_ on a tear :) 11:38:47 sdake_: what is up? 11:39:17 sbezverk_ just commentary on how your doing good fast work ;) 11:40:28 sdake: thanks, it is just so exciting to see kolla-kube is taking shape 11:41:28 Paul Bourke proposed openstack/kolla: Customizations for Neutron https://review.openstack.org/350777 12:19:45 Christian Berendt proposed openstack/kolla: Enable CADF events in Keystone and introduce enable_cadf_notifications https://review.openstack.org/349006 12:20:22 can you please review https://review.openstack.org/#/c/349006/ , would be nice to have this in m3 12:29:15 Christian Berendt proposed openstack/kolla: Fix service_type of mistral endpoint https://review.openstack.org/352120 12:34:54 Christian Berendt proposed openstack/kolla: Add cloudkitty containers for type source https://review.openstack.org/349295 12:41:15 berendt, done. a nit 12:42:11 Jeffrey4l_ thanks 12:44:06 Jeffrey4l_ I have not added it to the list of enable_* because of the description: "Additional optional OpenStack services are specified here" 12:45:05 Christian Berendt proposed openstack/kolla: Enable CADF events in Keystone and introduce enable_cadf_notifications https://review.openstack.org/349006 12:45:51 Merged openstack/kolla: introduce bifrost-deploy container https://review.openstack.org/319703 12:47:38 Christian Berendt proposed openstack/kolla: Add cloudkitty containers for type source https://review.openstack.org/349295 12:48:18 Christian Berendt proposed openstack/kolla: Install neutron-lbaas-dashboard from source in horizon container https://review.openstack.org/336943 12:48:50 berendt, we have enable_neutron_lbaas. So i think it should be OK. 12:49:09 Jeffrey4l_ LBAAS enables a service in OpenStack, neutron-lbaas-agent 12:49:17 Merged openstack/kolla: Fix nova_ssh container shell to allow instance resizes. https://review.openstack.org/361891 12:49:34 Jeffrey4l_ cadf notifications are only a alternative messaging fomat for keystone and barbican 12:50:02 ok. 12:52:59 Merged openstack/kolla: Add aodh role https://review.openstack.org/351027 12:56:01 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 12:57:51 Merged openstack/kolla: Customizations for Neutron https://review.openstack.org/350777 13:06:41 sean mooney proposed openstack/kolla: generate bifrost yaml configs https://review.openstack.org/349684 13:08:37 im going to quickly rebase the full bifrost chain again the first patch whcih merged was ahead of the rest 13:11:39 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 13:17:11 http://paste.fedoraproject.org/418051/14726493/ -> http://paste.fedoraproject.org/418051/14726493 13:17:21 sean mooney proposed openstack/kolla: add bifrost deploy ansible support https://review.openstack.org/349685 13:17:21 sean mooney proposed openstack/kolla: adds bifrost ssh key generation https://review.openstack.org/350692 13:17:22 sean mooney proposed openstack/kolla: add deploy-servers command https://review.openstack.org/349729 13:21:50 sean-k-mooney - why rthe rebase? 13:23:32 sdake: the base commit was ahead of the rest so the chain could not merge automatically 13:24:01 sean-k-mooney so you just reordered the commits? 13:24:24 sean-k-mooney i had left workflow off one patch 13:24:26 sdake: i litrally use rebased them on master no code chage 13:24:29 because i wass waiting for gate 13:24:52 sean-k-mooney cool 13:24:58 well i just checked this one https://review.openstack.org/#/c/349684/10 13:25:02 i'll wait for gate and workflow them 13:25:21 night 13:25:21 sean-k-mooney a rebase removes workflow+1 13:25:27 night duonghq 13:26:05 sdake: yes it does but you had given https://review.openstack.org/#/c/349684/10 workflow +1 but jenkins failed to merge it for some reason 13:26:42 Patch Set 9: Verified-1 13:26:42 Merge Failed. 13:26:43 This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. 13:26:48 yup - it needed a rebase ;) 13:26:49 thanks 13:27:13 i'll still wait ofr gate 13:27:30 sdake: no worries 13:27:56 Merged openstack/kolla: Import the Murano core libary as part of setup https://review.openstack.org/360002 13:30:47 our ci needs love 13:34:35 Paul Bourke proposed openstack/kolla: Add a block around oraclelinux base package install https://review.openstack.org/363659 13:36:14 pbourke where is inc0's patch to add custoizations tobase? 13:36:22 pbourke retty sure he had submitted one 13:36:28 sdake: yeah it's merged 13:36:35 this is an addition 13:36:40 hmm oesn't look like it 13:36:49 using rpm still in base? 13:37:05 because oraclelinux is doing funky stuff (nodeps) 13:37:13 not compatible with our standard macros 13:37:30 got it 13:37:31 thanks 13:37:38 though... 13:37:44 it looks like I may need to rebase this patch 13:38:24 Paul Bourke proposed openstack/kolla: Add a block around oraclelinux base package install https://review.openstack.org/363659 13:38:41 Hey all. Still getting null module errors from horizon. Is the person who fixed that last time around? 13:38:58 It may very well be horizon's fault. Just wondering what the fix/workaround was 13:39:13 Traceback ending 'ValueError: Empty module name' 13:39:24 it looks like inc0 may have missed a few oppurtunities to use the macro 13:39:28 in other places 13:39:41 kbaegis, iirc last week the latest image worked 13:40:12 hasn't tried this week 13:41:04 Jeffrey Zhang proposed openstack/kolla: Add warning to kolla-ansible destroy action https://review.openstack.org/356957 13:42:25 pbourke you hae a +1 on https://review.openstack.org/#/c/357015 13:42:29 any reason not a +2? 13:43:08 hmm cant remember! 13:43:13 I think Im good with it 13:44:13 Merged openstack/kolla: Add option to support dhcp/l3 agent ha https://review.openstack.org/352180 13:45:29 Jeffrey4l_ since you hae tht patch open - one little wording chngee 13:47:00 Jeffrey4l_ atually i just added naother comment there 13:47:43 sdake, ok. i will add that. thanks. 13:48:26 lets try to wrap up as many reviews today as possible 13:48:39 at 5pm pst I am submitting the milestone 3 release review 13:48:52 Paul Bourke proposed openstack/kolla: Allow operators to override versionlock for redhat in base https://review.openstack.org/363671 13:48:55 then we can leak into rc1 bit 13:49:07 for important features 13:49:16 OK 13:49:33 rc1 is 15th 13:49:58 Well, if anyone cares here's the traceback :) http://hastebin.com/kexoporilo.rb 13:51:31 Jeffrey Zhang proposed openstack/kolla: Add warning to kolla-ansible destroy action https://review.openstack.org/356957 13:51:34 kbaegis is this centos on centos with master? 13:52:39 sdake: Correct 13:52:59 I'm trying to figure out what's going on there, but I'm not very good with django 13:53:12 Looks like it's trying to load the keystone portion? 13:53:25 Low priority if you guys are preparing an rc 13:53:31 But help is always appreciated :) 13:54:42 [06:53:22] iirc this is a new thing 13:54:43 [06:53:29] amotoki (~amotoki@sp1-79-81-235.msb.spmode.ne.jp) joined the channel 13:54:43 [06:53:35] <+tsufiev> wat? 13:54:43 [06:53:43] is it an old thing? :) 13:54:44 [06:54:02] <+tsufiev> as of now we just added osprofiler requirement to horizon 13:54:45 [06:54:08] esp (~dnguyen@c-67-183-15-117.hsd1.wa.comcast.net) left IRC (Quit: esp) 13:54:46 [06:54:09] <+tsufiev> no other commit has been merged yet 13:55:07 kbaegis - looks like horizon regression to me - join #openstack-horizon 13:57:03 sdake: Oh, you're already on it :) 13:58:27 kbaegis - ts* will look into it 13:58:44 kbaegis i suspect if its broken for us its broken for eveyrone because we just synced with their defaults 13:58:58 sdake: horizon still has not closed https://bugs.launchpad.net/horizon/+bug/1585761 13:58:58 Launchpad bug 1585761 in OpenStack Dashboard (Horizon) "failed to boot instance from image without cinder volume" [High,In progress] - Assigned to Rob Cresswell (robcresswell) 13:59:19 sdake: I was actually playing around with that. I pulled git into the container and tried to sync back to an older tag 13:59:42 without that you cant boot vm without cinder deployed from horizon 13:59:43 unlinked the /horizon symlink, pointed it at the repo and did pip install -U /horizon-source/git 13:59:57 Still doesn't work 14:00:16 well bottom line is horizon is busted :) 14:00:25 i thinkk our implementation is correct 14:00:26 yep 14:00:41 so no sense burning cycles on debugging a different project's new code changes 14:00:55 that is why they have a core team :) 14:13:29 Good catch, horizon bug. Workaround is to $(rm -rf /var/lib/kolla/venv/lib/python2.7/site-packages/osprofiler/) 14:16:25 kbaegis a better workaround would be to remove osprofiler from requirements.txt 14:16:37 kbaegis but not somethign we will do at this point 14:16:52 kbaegis will wait on horizon team to fix the bug 14:20:47 sdake_: They have a fix in review 14:20:52 https://review.openstack.org/#/c/363701/1/osprofiler/__init__.py testing now 14:21:18 How do I redeploy just horizon from image again? 14:21:24 Is that the —tag horizon? 14:22:37 kbaegis today is busy day 14:22:48 kbaegis i got the ball rolling but need to focus on othe rthings 14:23:06 sdake_: Thanks. I'll leave you guys be then :) 14:24:21 kbaegis: what do you men reploy just horizon? 14:24:24 *mean 14:24:58 I think you can pull the horizon image than restart the container 14:25:11 due to horizon is single container component iirc 14:26:14 kfox1111: ping 14:26:37 Jeffrey4l_ re https://ask.openstack.org/en/question/96303/kolla-all-in-one-failed-with-openvswitch-db-container/?sort=latest 14:26:45 is this the docker 1.12 bug you fixed? 14:27:15 sdake_, no. 14:27:50 but he is use latest kolla playbooks to deploy old images. 14:27:58 he should build the image again. 14:28:02 i see 14:28:10 he should be following the instructiosn then 14:28:38 sdake_ seem familiar, 14:29:06 clean everything and rebuild image and redeploy is ok, 14:29:18 newbie get it easily 14:29:37 i'd like to make kolla newbie-proof :) 14:29:53 sdake_, we are not handle the mitaka branch doc properly 14:30:17 Jeffrey4l_ you mean master? 14:31:09 Jeffrey4l_ please expand - i don't understand what you mean 14:31:19 sdake_, yes(doc.openstack.org only show the master doc, this is talked) 14:31:34 iirc, in the mid-cycle? 14:31:37 yes that is right 14:31:49 docs.oo only show master not per tag 14:31:53 for ALL projects 14:32:08 And in the mitaka/quickstart.rst, when installing kolla 14:32:10 so our docs.oo content needs to cover master and all branches still in existance 14:32:16 it say: git clone https://git.openstack.org/openstack/kolla 14:32:19 i see 14:32:20 pip install kolla/ 14:32:33 which is wrong too. :( 14:33:06 we may need some better solution for this :) 14:33:19 Jeffrey4l_ do you have any suggestion? 14:34:15 openstack doc has some better solution ( which is talked, too ) 14:34:23 let me find the link. 14:43:54 duonghq, sorry can not find the link right now. The solution is: put all branch doc in one file and render different branch doc by using that. 14:44:13 then we can put all branch's doc into doc.openstack.org. 14:44:18 something like that. 14:46:53 Jeffrey4l_ thank you, I understood 14:47:08 by egonzalez90 is a champ on ask.oo 14:47:15 by/boy 14:48:58 id like to get rhallisey's docs changes merged once they are correct 14:49:02 and then rehash docs at summit 14:49:13 i have some new ideas there 14:49:27 sdake_, would that be merge them and keep the quickstart there? 14:49:30 rhallisey - any chance you can take over the meeting today for kolla at the 25 minute mark 14:49:41 ya 14:49:44 just make me a chain 14:49:47 chair* 14:49:48 will do 14:49:51 when you start it 14:59:46 Merged openstack/kolla: Add a block around oraclelinux base package install https://review.openstack.org/363659 15:02:11 good morning 15:02:13 good morning Kolla 15:02:30 inc0: Morning! 15:02:31 ahh sorry for double post, make it a very good morning then 15:02:53 kbaegis: did you success upgrade Horizon from image? 15:03:35 duonghq: No, that didn't work. The directory was still missing after pull. Had to full redeploy 15:04:13 hmm, which directory? I success with this method before 15:05:22 Merged openstack/kolla: repair rally dockerfile https://review.openstack.org/363466 15:05:54 duonghq: The issue was osprofiler. Removing it from the site-packages is a kludge 15:06:09 duonghq: Reviewing a horizon patch currently 15:07:04 kbaegis: as I understand, you want to remove the osprofiler in docker image 15:07:16 and then deploy this image to the target nodes? 15:07:34 correct? 15:07:59 duonghq: No, I'm testing a patch from the horizon team 15:08:14 duonghq: The kludge works though. No more 500's after removing osprofiler 15:09:42 kbaegis: I mean you want to test it by apply thi patch to horizon image? 15:09:56 or just external horizon? 15:17:15 sdake: I can bp https://bugs.launchpad.net/kolla/+bug/1616155 this afternoon 15:17:18 Launchpad bug 1616155 in kolla mitaka "Unable to import database dump with ansible module in kolla-toolbox due to missing mysql binaries." [Critical,Confirmed] - Assigned to Steven Dake (sdake) 15:17:22 sdake: I kinda lost track of that, sorry! 15:18:04 michauds no sweat 15:19:01 michauds typically the stable team takes care to backport 15:19:07 michauds but we take where we can get it ;) 15:19:16 take help that is ;) 15:19:25 i'e assinged those backports to you - tia :) 15:19:56 sdake_ Awesome thanks! I'll get them done by eod 15:20:00 :D 15:20:24 inc0 morning fine sir 15:21:35 ah I woken up, and 3 my reviews are merged 15:21:38 how fun 15:22:10 sean-k-mooney, did you test the interface name thingy by any chance? 15:24:43 michauds - re losing track- tell me about it - this is why we have all this nonsense with bug tracking in git commits ;-) 15:25:08 Merged openstack/kolla: Add warning to kolla-ansible destroy action https://review.openstack.org/356957 15:28:07 sdake_ definetly, I only remembered because the milestone changed ahaha 15:28:20 kfox1111: need to chat 15:30:23 Merged openstack/kolla: generate bifrost yaml configs https://review.openstack.org/349684 15:31:07 Jeffrey4l_ when you get to submitting the fio change to rally could you link the review 15:31:16 Jeffrey4l_ could be whenever - two-four weeks out is fine :) 15:31:39 i'm pretty sure rally will want that scenario 15:36:42 important team meeting in t-25 minutes 15:37:20 roger 15:37:43 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 15:38:03 zhubingbing - if you make a change to the rneo note for barbican we can merge it 15:38:17 ok 15:38:59 iHow should I do it 15:43:06 snake make a change to the rneo note , i should add something? 15:44:23 zhubingbing change it to somethign like "Implement Barbican playbook in Kolla. Barbican's mission is XYZ" 15:44:30 where XYZ is a cut and paste from thegovernance repo 15:44:38 http://github.com/openstack/kolla 15:44:39 rather 15:44:52 http://github.com/openstack/governance 15:44:59 look for a file called projects.yaml 15:45:14 ok 15:46:23 Is barbican working w/ designate? Can I create certs automagically with the fqdn? 15:46:39 inc0: am not i have not had a change i have been fight with an os install 15:46:52 kk, I'll try it then 15:47:10 inc0: ill do it now as well and let you know. 15:47:20 it should only take a few minutes to do 15:48:04 kbaegis doubtful 15:48:17 kbaegis we take patches :) 15:50:25 inc0: it seams to be failing for me. it looks like "{{ baremetal_nic }}" is resolved before the deleation to "{{ baremetal_node }}" happens 15:50:41 inc0: i need to double check my config though 15:52:19 zhubingbing your kiling me dude :) 15:57:48 sdake, the fio should be not accept by rally i think. 15:58:41 Jeffrey4l_ why not? 15:59:02 Jeffrey4l_ i don't want proprietary tests necessarily unless they re in some kind of repo 15:59:44 meeting time? 15:59:59 sdake, i can try that. 16:00:52 we can release our report along with the test method 16:02:12 team meeeting now 16:02:22 #openstack-meeting-4 16:02:46 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 16:03:29 Vladislav Belogrudov proposed openstack/kolla: Allow Neutron to be run in DVR mode https://review.openstack.org/356544 16:05:04 Paul Bourke proposed openstack/kolla: Allow operators to override use of get-pip.py https://review.openstack.org/363775 16:06:22 Vladislav Belogrudov proposed openstack/kolla: Allow Neutron to be run in DVR mode https://review.openstack.org/356544 16:07:28 Paul Bourke proposed openstack/kolla: Allow operators to override use of get-pip.py https://review.openstack.org/363775 16:13:45 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 16:13:56 Vladislav Belogrudov proposed openstack/kolla: Allow Neutron to be run in DVR mode https://review.openstack.org/356544 16:16:25 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 16:21:30 https://review.openstack.org/345751 ready for review. 16:25:11 Merged openstack/kolla: Chage with_ loop variables syntax to ansible 2 https://review.openstack.org/361886 16:26:08 Merged openstack/kolla: add bifrost deploy ansible support https://review.openstack.org/349685 16:30:13 Merged openstack/kolla: Use dumb-init to manage the pid 1 process https://review.openstack.org/357015 16:46:44 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 16:54:05 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 16:54:25 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 16:58:41 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 16:59:47 ok... I need to spawn a vm on k8s 16:59:53 to make my day brighter 16:59:57 sbezverk_, https://bugs.launchpad.net/kolla/+bug/1616268 16:59:57 Launchpad bug 1616268 in kolla mitaka "Stale namespace removal causing "RTNETLINK answers: Invalid argument" errors" [Critical,Confirmed] 17:00:13 didn't you see something like that? 17:00:18 rhallisey: wasnt that fix before 17:00:23 I haven't seen that appear in latest dockers 17:00:34 it may caused by zombie process. But we are not sure. 17:00:52 that too, l3 agent can create zombies 17:00:58 Jeffrey4l_: this is only showing up with k8 though right 17:01:08 inc0, rhallisey it happened on both master and mitaka. I see it many times. 17:01:14 hmm 17:01:29 sean-k-mooney, i see this in ansible deployed images. 17:01:29 qdhcp would suggest zombies 17:01:31 sean-k-mooney, I think sbezverk_ saw it last week in k8s, but it may also be showing up here 17:01:44 not sure it's the same bug. sbezverk_ can confirm 17:01:49 as from my talk with Sam, it's neturon that spawns dnsmasq that spawns agents that's issue 17:01:50 could this be a regression in docker 1.12 17:01:57 grandchildren aren't reaped in docker 17:02:01 sean-k-mooney, yep 17:02:02 inc0, ah ya 17:02:13 I remember he mentioned this in Austin 17:02:29 may be https://review.openstack.org/357015 this fix it . But i am not sure. 17:02:54 i can not re-product this issue steadily. 17:03:02 re-produce 17:03:04 well either the dumb-init or bash would solve that issue. just need a pid 1 other then the l3 agent 17:03:18 brb 17:03:19 Jeffrey4l_, it makses snese, let's see if dumb init will fix it 17:03:47 yep. If so , we may need backport the dumb init to mitaka. 17:03:57 probably good idea 17:04:23 does dumb-init effect the loging? 17:04:29 e.g. docker log x 17:04:43 sean-k-mooney, so we have mostly stuff logging to file anyway 17:04:48 we rarely use docker logs 17:04:52 sean-k-mooney, no. 17:04:54 but no, stdout should stay the same 17:05:04 ok cool 17:05:09 brb need food 17:05:32 i have kibabn deployed just dont know how to use it so nomally check docker log or the heka container 17:06:14 you can search all the log in kibana. 17:12:47 sean-k-mooney you will need to configure it - or load a preconfig 17:13:02 akwasnie has one she produced for austin summit 17:14:21 sdake: i when through the kibana guide a few months ago but honestly i jsut have not had time to sit down and look at it properly. 17:14:38 sean-k-mooney you need a preocnfigure 17:14:44 its not easy to configure 17:15:40 is anyone working on it next cycle? 17:16:40 its one of those things for me anyway that would definetly be nice to have just havent had the free time to learn 17:22:32 sean-k-mooney its a high priority for this cycle but it didn't happen 17:22:42 so we bounced it to ocatta 17:22:47 the implementation is almost done 17:22:53 it needs to be made to load automatically 17:22:59 get the setup from akwasnie :) 17:26:13 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 17:26:41 Kevin Fox proposed openstack/kolla-kubernetes: Generic Services, external ip & nodeport https://review.openstack.org/363152 17:29:13 Merged openstack/kolla: Allow operators to customise rabbitmq plugins https://review.openstack.org/363720 17:38:56 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 17:40:40 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 17:42:32 rhallisey: where can i find tasks to participate? is there an etherpad? 17:43:01 imcsk8, https://blueprints.launchpad.net/kolla-kubernetes 17:43:04 sean-k-mooney, is "add deploy-servers command" review dependant on gnocci ansible role? 17:43:05 thanks! 17:43:17 can you remove this dep if it's so? it will block your merge 17:43:25 no 17:43:35 imcsk8,kolla-kubernetes or kolla? 17:43:38 its depdent on the other bifrost revew 17:43:38 or both 17:43:43 imcsk8, ^ 17:43:59 rhallisey: are they different projects? i thought kolla-kubernetes was under kolla 17:44:09 kk then my gertty is borked 17:44:34 imcsk8, they are both under kolla, but are differnt in how they deploy openstack 17:44:35 i'm interested in both, i've had my share of kuernetes also 17:44:43 inc0: the deploy-servers is the last in the chain so once the ssh key gen patch is merged it should merge 17:44:53 I just +w botrh 17:44:55 both 17:44:59 so yay for bifrost 17:45:13 imcsk8, k8s is super new so there is lots of work to be done there 17:45:22 amazing job Sean 17:45:23 kolla ir more stable, but still lots of work 17:45:38 rhallisey: cool i'll check that out, i've been into kubernetes for a while 17:45:42 imcsk8, do you want to pick a service to implement 17:45:43 ok great 17:45:54 I'll buy you cava in Barcelona for it;) 17:45:56 i will :) 17:46:26 imcsk8, if the service you want to add isn't there, you can add it as a bluepring 17:46:29 blueprint 17:46:38 imcsk8, and feel free to ask questions if you get stuck 17:46:53 recoomend following the existing pod/service templates 17:46:59 they should guide you 17:47:09 v1k0d3n, ^ 17:47:27 a good way to dig in 17:47:35 inc0 thanks for the help in getting it done. 17:48:01 Merged openstack/kolla: adds bifrost ssh key generation https://review.openstack.org/350692 17:48:08 Merged openstack/kolla: Allow operators to override versionlock for redhat in base https://review.openstack.org/363671 17:48:16 merge fest 17:48:18 inc0: i tested your patch by the way for https://review.openstack.org/#/c/359364/ 17:48:25 update the bps if they get completed :) 17:48:29 inc0: it broke on my setup 17:48:33 duh 17:49:05 ok I guess we can require baremetal_interface for now 17:49:56 it actually makes sense because ssh can be on different interface than APIs 17:50:12 Merged openstack/kolla: Allow operators to override use of get-pip.py https://review.openstack.org/363775 17:50:13 Vladislav Belogrudov proposed openstack/kolla: Allow Neutron to be run in DVR mode https://review.openstack.org/356544 17:50:14 in fact....we could make this a feature and make ssh listen *only* on baremetal_interface 17:50:20 as a security feature 17:50:24 Kevin Fox proposed openstack/kolla-kubernetes: Make libvirt over tcp an option https://review.openstack.org/362457 17:51:22 yes well this is currently used to make the hostnames resolveable for rabbit so it should be the same as the api network interface 17:51:44 inc0: https://review.openstack.org/#/c/362457/ ? 17:52:16 well i guess it dose not have to be the api network interfce but it should be the same interface used for rabbit 17:52:58 well, tbh that's a bug 17:53:03 rabbit needs API interface 17:53:12 so let's change it to api interface instead of baremetal 17:53:14 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 17:53:35 sure ill submit a patch for that now then 17:53:42 thanks 17:54:15 I'll update docs to make baremetal required and add this ssh config 17:54:17 it's cool. 17:55:17 hang tight folks - i'll be on in about 30 mins 17:55:46 in fact this will make us probably most secure deployment tool;) 17:55:49 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 17:55:53 yup 17:57:21 inc0: thx 17:59:01 sdake ping 18:00:51 Merged openstack/kolla-kubernetes: Make libvirt over tcp an option https://review.openstack.org/362457 18:01:22 inc0: one more please if you have a sec? https://review.openstack.org/#/c/363152/ 18:02:08 nm. merge conflict... let me rebase. 18:05:48 hmm... not sure why that wasn't an automatic rebase... 18:07:44 oh... I know why... it didn't notice the parent already merged... 18:08:41 kfox1111: check this out :-) 18:09:02 http://paste.openstack.org/show/565268/ 18:09:10 now with petset 18:09:24 cinder is ready for final review 18:09:42 sbezverk_: nice. :) 18:09:47 service and api optimization will do after corresponding changes 18:09:56 in the trunk 18:10:11 I'd like to get #363152 in real quick and rebase the cinder one on it. give me a couple minutes and I think #363152 will be ready. 18:12:53 Kevin Fox proposed openstack/kolla-kubernetes: Generic Services, external ip & nodeport https://review.openstack.org/363152 18:12:57 k. rebased. 18:19:34 rhallisey thanks for finishing the meeting out 18:19:43 rhallisey ++ :) 18:19:51 no problem 18:19:59 inc0: ok, can you review this real quick if you got a moment? https://review.openstack.org/#/c/363152 18:20:04 rhallisey: you too? :) 18:20:12 yessir 18:20:17 thx. 18:20:23 sbezverk_: will there be a demo of kolla-k8s at the summit? 18:21:26 rhallisey: saw your note...as a guideline/suggestions for adding blueprints? 18:22:00 v1k0d3n, I'll add them in 18:22:04 for all the services 18:22:09 rc1 looks fairly feasible 18:22:10 and you can pick one 18:22:19 v1k0d3n, will do that asap 18:22:27 kfox1111, gerrit is slow:() 18:22:32 core reviewers - please focus your reviews on stuff related to rc1 blueprints 18:22:39 (or bugs) 18:22:44 v1k0d3n, https://blueprints.launchpad.net/kolla-kubernetes/+addspec 18:22:55 that's how you fill out a bp 18:23:09 just fill in the fields describing what you're working on 18:23:41 sdake: where does kolla-kubernetes fit in with that? 18:24:10 Christian Berendt proposed openstack/kolla: Enable CADF events in Keystone and introduce enable_cadf_notifications https://review.openstack.org/349006 18:26:08 kfox1111 fit in with which? 18:26:15 kfox1111 rc1? 18:26:38 kfox1111 - kolla-kubernetes is on a release_independent cycle 18:26:39 yeah. reviewers targeting rc1 things. or was your comment about just kolla reviews and not kolla-kubernetes reviews? 18:26:46 ah. ok. 18:27:10 kfox1111 ya don't ignore other reviews folks, but fosuc on those blueprints :) 18:27:23 kfox1111 for kubernetes what i'd like to see is something demoable for summit 18:27:34 i think we are close to that 18:27:53 i'd also like to present our technical foundations 18:28:00 but i'm not doing the presenting 18:28:05 sdake: yeah. I think we're in pretty good shape now for demoing something. 18:28:24 sdake: I think we have a bunch of cleanup that should happen before we add more devs though, as it will be painful churn. 18:28:40 kfox1111 devs will come and we will take them as they do :) 18:28:43 so I'm pushing to get those changes agreed to/applied as quickly as I can. 18:28:57 kfox1111 roger - so what do you need 18:29:01 agreed. but its more like, if we reaname a service, then the devel has to delete x and recreate y. 18:29:09 that kind of thing is not bad, but painful if its churning a lot. 18:29:21 churn is normal in an early code base 18:29:23 better to now if possible. 18:29:26 rhallisey: i submitted a blueprint the other day for LKVM stage 1. 18:29:58 I saw 18:29:59 nice 18:30:20 is that good/bad? i don't really know what it would take now...probably has to sit for a little while. 18:30:28 until i'm at a place where i can start testing. 18:31:47 v1k0d3n, if you want to got for that one wfm, but it's not super high prioprity because there still needs to be a lot of work on the foundation 18:32:00 still missing services 18:32:13 my suggestion would be tog et heat going 18:32:15 easy to demo 18:32:17 CLI needs some work and we need to disguss ansible 18:32:24 createse fantastic demonstrations 18:32:57 ceph + cinder + heat is where I'd focus sservice wise 18:32:57 v1k0d3n: lkvm for the tenenat of the kolla contaienrs? 18:33:20 https://blueprints.launchpad.net/kolla-kubernetes/+spec/rkt-lkvm-support 18:33:29 true it should probably go into kolla 18:33:48 though it may span both tools 18:33:54 lkvm will really reduce your network perfromce 18:34:03 what is lkvm 18:34:17 oh i see link reading 18:34:19 it like a really light qemu 18:34:26 you mean intel clear containers? 18:34:39 lkvm is what clear container is based on 18:35:00 got it - so buliding block 18:36:19 so for atleat the cc condition, I think we need to consider it 18:36:34 but I don't know if it supports the fetaures we need to run inside 18:36:40 rhallisey: i was checking, the kolla-k8s stuff, i think if it's combined with cockpit is a good alternative for ooo 18:36:50 for eample host bind mounts, host ipc mode, host pid mode, and host networking mode 18:37:22 sdake: host networking cant be done with lkvm 18:37:51 well it can but your container will see the vms network stack not the baremetal servers 18:37:52 what about the others? 18:38:15 imcsk8, I have a theory on how to re use a bunch of the tripleo work within a k8s framework 18:39:51 sdake: with lkvm you are running the container in a hardware enforced vm context with its own kernel so container in the same lkvm instance could use pid=host ... but they would not be shared outside the lkvm instance 18:40:26 sean-k-mooney - need the functionality to work as is does today to embrace cc 18:40:44 imcsk8, it's still in theory state though 18:40:45 sounds like there are a slew of technical brrier 18:40:46 s 18:40:55 imcsk8, interesting observation though 18:41:39 Kevin Fox proposed openstack/kolla-kubernetes: l3/metadata DaemonSets + DVR, & labels https://review.openstack.org/357557 18:41:42 sdake: there is a nice diagrame at the bottom of https://coreos.com/rkt/docs/latest/running-lkvm-stage1.html 18:42:00 well ok its not really a diagram but you can see the layers 18:43:30 runnin kolla on lkvm would almost be like running kolla on magnum. 18:44:14 Is it fair to say that kolla is one of the most complicated docker projects currently? 18:44:45 kbaegis: i dont know if i would acall kolla a partcarlly complicated docker project. 18:45:02 kbaegis doubtful - only 35k lines of code 18:45:21 kbaegis I'd say a different way to put it is we produce alot of functionality in avery compact (35kloc) form factor 18:45:38 places like netflix/amazonetc have more complex solutions 18:45:51 you know, internet 15.0 companies :) 18:45:59 kbaegis: kolla has lots of containers but they all follow a similar patteren in how they are used,deployed and created 18:46:58 sean-k-mooney and managed long term ;) 18:48:00 Hi Folks, Does anyone know what extra settings are required for deploying ceph on single node? I am following https://github.com/openstack/kolla/blob/master/doc/ceph-guide.rst#managing-ceph and stuck at deploy 18:48:16 sdake: do you think that long term management will be solve in both k8s and ansible or just one or the other 18:48:48 sean-k-mooney its already solved in kolla ansible afiac 18:48:51 adnavare: what are you stuck on 18:48:52 ceph is sitting forever at "Looking up disks to bootstrap for Ceph OSDs" 18:49:01 sean-k-mooney kolla-kubernetes requires more work to reach parity 18:49:18 adnavare your the second person that has reported a similar problem 18:49:35 I did do "parted /dev/sdb -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -1" so it should correctly look for that partition 18:49:40 adnavare we have been in release mode and osic mode for about a month, perhaps its time we tried that after we tag milestone 3 today 18:49:54 sdake: ha ha that is me only ;) 18:50:08 adnavare ok :) 18:50:31 i thought anscopic reported it as well 18:50:31 sdake: i have given two names for preference other is aNupOsic 18:50:43 oh thatsy o u cool :) 18:50:49 say where did the kuryr blueprint go rhallisey 18:51:00 uh 18:51:39 sdake: you said you were doing osic tear something with 131 nodes 18:51:42 https://blueprints.launchpad.net/kolla/+spec/kuryr-docker-plugin 18:51:44 sdake, ^ 18:51:54 so - this used to be at 420 stars lsat week: https://github.com/openstack/kolla 18:53:50 sdake: sean-mooney: so any idea about the issue? 18:53:51 thanks rhallisey 18:53:56 adnavare no idea 18:54:12 adnavare also we have a release today - but I can give it a go tomorrow afternoon 18:54:13 sdake:hmmm 18:54:20 adnavare and see if it can be reproduced 18:54:26 can you file a bug please and link it? 18:54:57 sdake: Yeah i can definitely reproduce. But it is not throwing any error it is just sitting 18:55:35 sdake: also what i notice is when it is trying to run "docker exec -t kolla_toolbox sudo -E /usr/bin/ansible localhost 18:55:35 -m find_disks" it is asking for ansible user password 18:55:46 sdake: so do we need such user? 18:56:00 sdake: I used to never create ansible user with Kolla 18:56:14 adnavare - not sure what the problem is 18:56:25 adnavare possible it was broken during our third party blueprints work 18:56:31 inc0 may be able to help 18:56:44 or i'll be available tomorrow 18:56:59 sdake: ok sure 18:57:08 adnavare I will need to see it for myself 18:57:17 adnavare one ething that could help is a webex ssession if your availalbe for that 18:57:26 sdake: sure 18:57:29 ut tomorrow - not today ;) 18:57:49 sdake: ha ha no problem, I want to set up ceph :) 18:58:43 rhallisey i see some blueprints in milestone 3 which are not in implemented state 18:58:49 rhallisey what was thinking on those? 18:58:56 sdake, they are close 18:59:03 the ones that are there 18:59:06 could be done today 18:59:52 who do we need to haraas? :) 18:59:56 harass that is 19:00:08 smekel_ - ping re fernet 19:05:29 sean-k-mooney - think for bifrost we need your patch to go thorugh ci 19:05:38 sean-k-mooney and for inc0 to rebase his patch 19:05:46 sean-k-mooney anything else outstanding? 19:15:41 steak -> shower brb 19:28:17 syo 19:36:24 sdake: for bifrost there should just be https://review.openstack.org/#/c/349729/ 19:37:33 inc0 i can't upload to google drive - out of quota 19:37:45 I just upgraded to the 100gb plan, but quota hasn't updated 19:37:55 so as soon as that happens, I'll upload those files for you 19:38:53 sdake: for kolla host inc0 change https://review.openstack.org/#/c/359364/ dose not work and https://review.openstack.org/#/c/362739/ is a duplicate that also does not work 19:39:20 fantastic 19:39:25 sean-k-mooney you mind taking a crack at it? 19:39:31 sdake: ill be uploading a small bugfix that just gets rid of the baremetal_nic variable and uses the api_interface instead 19:40:12 Kevin Fox proposed openstack/kolla-kubernetes: Generic Services, external ip & nodeport https://review.openstack.org/363152 19:40:24 the code on the master branch works without any change but you just needed to define baremetal_nic in the gloabl.yaml or in the inventory file. 19:40:55 right - so didn't we decide in meeting we wanted to be able to specify that? 19:41:22 baremetal_nic? 19:41:27 yup 19:41:33 you tell me - you added it :) 19:42:31 so it will be used to generate the host name to ip mappings for rabbitmq so infact it should always be the same as api_interface 19:42:58 so talking to inc0 i was jsut going to remove it and use api_interface instead 19:43:26 inc0, kfox1111 https://blueprints.launchpad.net/kolla-kubernetes/+spec/rabbitmq-single-queue 19:43:33 sdake if we want to keep both we can we just need to document the use of baremetal_nic 19:52:33 sean mooney proposed openstack/kolla: replace baremetal_nic with api_interface https://review.openstack.org/363916 19:55:14 sdake: https://review.openstack.org/#/c/349729/ seams to be stuck 20:10:15 rhallisey: kfox1111: gents after cinder control plane is merged, I want to look into cinder volume with iscsi. SInce there is not much time, I want to do a really limited version just to be able to have real cinder volumes, maybe even on a single compute node and to be able to boot instance from it. WOuld you be ok with this approach. I think global development might take too much 20:10:16 time.. 20:10:20 what do you think? 20:11:05 sure 20:11:27 getting it functional first is a great step 20:12:06 rhallisey: cool :-) I hope Kevin agrees also 20:12:32 we 20:12:42 otherwise he will torture me with flexibility and covering all possibilities 20:13:05 comments 20:13:11 we'll have to get there at some point, but don't need to asap 20:13:38 100% agree, it is more like a prototype or a baseline 20:24:56 Michal Jastrzebski (inc0) proposed openstack/kolla: Release notes for customization https://review.openstack.org/363924 20:26:09 rhallisey: while working on cinder, Kevin adn I tested cofigmap replace with new config. works fine, so for reconfiguration we need to replace existing configmap and restart container.. 20:26:28 sbezverk_, yup 20:26:48 CLI/workflow will run that 20:27:40 rhallisey: it would be fun to be able to catch that event and maybe restart container automatically ? 20:27:54 sbezverk_, kubernetes is going to build in that feature 20:28:01 when the configmap changes restart 20:28:24 rhallisey: super.. 21:04:31 sbezverk_, kfox1111 did any of you found good way to deploy k8s on multinode? 21:11:28 anyone know why the rabbitmq part is failing? http://paste.openstack.org/show/565319/ 21:11:31 it doesnt really say much 21:12:01 inc0: The only reliable way to deploy multinode I found was manual.. none of the automated worked and I was spending more time troubleshooting installation than working on kube 21:12:50 duh, last thing we need it to be deployment tool for a deployment tool on top of which we would deploy 21:13:53 i believe thats an initialization tool for a deployment tool :) 21:14:46 inc0: last week rhallisey was playing with kargo 21:14:46 jrich523, issue with rabbit is that every rabbit host has to communicate to other over hostnames 21:15:04 rabbitmq doesn't know how to IP 21:15:16 errr so this is the 3rd time i've set this up 21:15:26 and the last two times didnt care about the ip usage 21:15:41 it was almost success.. you need to check with him for the final verdict 21:16:11 so way to fix it is ensure that every controller can ping every other controller over it's hostname 21:16:26 and it has tob be api_interface 21:16:29 address 21:18:50 also it seems to be using a name, 'localhost' 21:19:09 which, cuz its all in one, localhost will resolve correctly from "all hosts" :) 21:19:23 inc0: check this link, I used it as a guideline: https://github.com/kelseyhightower/kubernetes-the-hard-way 21:25:05 trying cargo 21:25:07 kargo 21:28:30 it's buggy 21:29:15 I deployed manually too. just because I wanted to learn how to debug it, and didn't want scripts in my way. 21:29:30 (ops... :) 21:41:08 so i used the multinode inventory, and used the machine name, making sure it was pingable, and, still the same thing 21:52:36 hi; newbie question - i trying to deploy liberty (1.1.2) - i’ve `git checkout tags/1.1.2`, did the prereq’s and i now get an error pulling in kolla-toolbox image: TASK [common : Pulling kolla-toolbox image] ************************************ 21:52:36 An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: 'NoneType' object has no attribute '__getitem__' 21:52:37 fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Traceback (most recent call last):\n File \"/tmp/ansible_9npklE/ansible_module_kolla_docker.py\", line 703, in \n main()\n File \"/tmp/ansible_9npklE/ansible_module_kolla_docker.py\", line 673, in main\n module = generate_nested_module()\n File \"/tmp/ansible_9npklE/ansible_module_kolla_docker.py\", line 659, in generate_nested_module\n 21:52:38 new_args.update(module._load_params()[0])\nTypeError: 'NoneType' object has no attribute '__getitem__'\n", "module_stdout": "", "msg": "MODULE FAILURE", "parsed": false} 21:52:50 (sorry for mesy dump of code) 22:03:27 so kargo kinda works 22:03:45 cool. :) 22:03:45 I had to dig a bit into it, but that's not horrible, I got k8s up and running 22:04:35 now, how do I deploy... 22:04:49 nice :) 22:05:11 yee379 follow instriuctions 22:05:21 instead use sudo pip install kolla-ansible==1.1.2 22:05:40 yee379 first step is pip uninstall kolla 22:05:40 sorry 22:05:47 instead step 1: pip uninstall koll 22:05:54 step 2: pip install koll==1.1.2 22:05:57 kolla==1.1.2 22:06:28 ah, i was trying the developer install - i’ll give the pip install a try :) 22:06:55 yee379, also make sure to have ansible 1.9.4 for liberty 22:07:26 inc: ryans been working on revised docs: 22:07:34 https://review.openstack.org/#/c/362467 22:08:31 thanks inc0: out of interest any reason why ansible 2 wont’ work? 22:09:20 yee379, it will with newton 22:09:28 ansible 2 is not backward compatible 22:13:50 ugh, cant figure this out 22:14:19 Stephan Michaud proposed openstack/kolla: Add mariadb client binaries to kolla-toolbox https://review.openstack.org/363962 22:15:25 thanks sdrake and inc0; a pull seems to be doing a lot more now. i think it was the ansible version issue 22:15:43 yeah probably 22:15:53 with newton it's ansible 2+ 22:16:28 sdake, mind acking? https://review.openstack.org/#/c/363924/ 22:17:25 anyway, I'm off 22:17:26 cyas 22:23:30 kfox1111: still around 22:23:31 ? 22:24:05 inc0 got it 22:24:33 yee379 name is sdake 22:24:40 drake is a pirate ;) 22:25:11 sorry! no pirates allowed him here then? 22:25:13 sdake: you have something in common with Drake ;-) 22:25:29 aye matey :) 22:27:41 yarrr 22:28:33 sbezverk_: yeah. whats up? 22:29:31 michauds ping 22:29:44 michauds pleae usse the cherry-pick button for cherry-picking patches in gerrit 22:29:57 michauds or use the proper cherrypick command (git cherry-pick -x) 22:31:17 sdake The cherrypick button would throw errors because of merge conflicts 22:31:42 kfox1111: I plan to do cinder volume with just iscsi 22:32:06 I think that's probably the harder thing to get working? as you got to do the tgt stuff? 22:32:11 michauds understoood 22:32:17 michauds can you use git cherry-pick -x then 22:32:22 I know I did that for kolla 22:32:30 sdake git didn't seem to appreciate the Dockerfile syntax change on neutron ;) 22:32:31 the direct rbd backend should just need the rbd config right, and nothing more. 22:32:57 michauds oh - did you do after the third party plugins? 22:33:28 I chattaed with Ryan and to be able to do something before the summit the scope will be just iscsi and probably on the single storage node 22:33:30 michauds its critical we keep the chnge ids intact 22:33:46 michauds and cherry-pick -x can do the job 22:33:59 its also iport we record which commit id was cherry-picked 22:34:01 sbezverk_: sure, if you want to try. I'm just saying, its probably a lot easier to do ceph backed, based on personal experience. 22:34:04 even if it conflicts all over the place 22:34:06 so once it is done and demonstrated, we can add all flavors we want.. 22:34:23 we are cherry-picking the concept, not the implementation so much this time 22:34:34 as it just pokes entries into ceph and lets ceph handle it. while the iscsi stack involves other daemons, the kernel, etc. 22:34:47 kfox1111: just FYI so you would not be surprised not to see ceph 22:34:56 k. 22:34:58 sdake Ack! Should I just drop the review or is it possible to ammend the metadata through an update to the review? 22:35:02 would like ceph +cinder + heat if possible 22:35:16 michauds drop review, start over 22:35:23 michauds i can walk you through one ove rwebex so you can learn 22:35:54 go for it. :) if you can get it to work, adding ceph support should be easy. :) 22:36:08 sdake I think it's good, I just noticed that the -x flag was missing on the gerrit generated cherry-pick link :) 22:36:14 kfox1111 - lol too busy unforutnately 22:36:35 kfox1111 one advantage of not being ptl is I will have more time to spend actually doing implementation work ;) 22:36:45 hehe. yeah. 22:37:58 michauds nah - you have to do it differenty 22:38:01 michauds in summary 22:38:03 checkout master 22:38:05 git review -s 22:38:12 git log to find the commit id 22:38:19 git checkout stable/mitaka 22:38:25 git cherry-pick -x commitid from mster 22:38:40 fix cherry-pick to mege correctly 22:38:40 then git review -s 22:38:42 then git review 22:38:44 i think that should do it 22:38:54 you may have to do the git review -s in the stable/mitaka branch 22:38:59 michauds ^^ 22:39:40 kfox1111: if it was working for kolla, should do for kube ;-) 22:40:23 sbezverk: yeah. Not saying its insurmountable. just level of effort. :) 22:40:42 Merged openstack/kolla-kubernetes: Generic Services, external ip & nodeport https://review.openstack.org/363152 22:42:06 the gate's slow today. :) 22:42:17 kfox1111 ^^ 22:42:33 kfox1111 releae week is always slow 22:42:42 yeah. 22:42:42 especially mielstone 3 22:43:02 hence the comment earlyer about stuff like fuelccp consuming resources in crunch time. :/ 22:43:29 sdake: Essentially, it's to make sure the the proper commit id is shown in the git message, right? ie: (cherry picked from commit 29c542186b8bc1b3d54611574aa953a03039d2fe) 22:43:31 kfox1111 check out this chart of just kolla: http://stackalytics.com/?module=kolla&metric=commits 22:43:49 michauds yes and the change id is correct (as cheryr-picked0 22:43:59 the patch should I think end up with two change ids 22:44:03 nice. 22:44:10 kfox1111 you notice the peaks 22:44:12 that is milestone 3 22:44:44 kfox1111 the chart at the top of commits over time 22:45:17 * kfox1111 nods 22:45:45 Stephan Michaud proposed openstack/kolla: Add mariadb client binaries to kolla-toolbox https://review.openstack.org/363968 22:51:33 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add Cinder API/Backup/Scheduler components https://review.openstack.org/363044 22:52:14 kfox1111: please check the latest cinder PS. 22:53:30 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add Cinder api and cheduler components https://review.openstack.org/363044 22:53:52 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add Cinder api and scheduler components https://review.openstack.org/363044 22:54:47 Stephan Michaud proposed openstack/kolla: Add mariadb client binaries to kolla-toolbox https://review.openstack.org/363962 22:57:57 sbezverk: reviewed 23:01:24 me again: trying to get 1.1.2 installed in an all-in-one - hitting issues with accessing the mysql database 23:01:32 unable to connect to database, check login_user and login_password are correct or /home/ansible/.my.cnf has the credentials. Exception message: (1045, \"Access denied for user 'root'@'dhcp-os-196-85.slac.stanford.edu' (using password: YES)\") 23:02:08 this is for [mariadb | Creating haproxy mysql user] 23:03:15 password its trying to use is the same as that in passwords.yml / database_password 23:04:35 sbezverk were ou the lsat one to touch magnum 23:05:02 yee379 did you run kolla-genpwd? 23:05:16 sdake: yep 23:05:40 yee which distro? 23:05:55 all defaults - so i guess centos / binary 23:06:13 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add Cinder api and scheduler components https://review.openstack.org/363044 23:06:15 so you didn't modify the globals.yml file at all? 23:06:20 sbezverk ping 23:06:47 v1k0d3n ping 23:06:54 yo 23:07:01 So I've been trying to wrap my head around how to use Kolla to install the OpenStack controllers (and compute nodes) under CoreOS on the bare metal hosts. 23:07:11 i had to change the openstack_release to 1.1.2 (it complained that 1.1.0 didn’t exist) and i added the kolla_internal_vip_address 23:07:20 v1k0d3n i've got a kubernetes project for the guy on your team hat likes to mentor others 23:07:22 if yu have one of those 23:07:37 lol 23:07:40 v1k0d3n it involves implmeenting heat in kolla-kubernetes 23:07:40 thats me man. 23:07:47 v1k0d3n ok - then project for you :) 23:08:00 that works. i would be all over that. 23:08:04 v1k0d3n kfox1111 will mentor you on how to get it rolling 23:08:12 (has volunteered atleast) 23:09:42 yeah that will be great. so reach out to kfox1111? i'm down. 23:18:46 Using physnet1, I can't seem to ping my routers interface 23:19:12 I can do it from inside the netns 23:20:54 I see br-ex, br-int, qbr from global 23:21:15 none have ipv4 addresses assigned 23:21:21 I'm trying to remember if that's new 23:22:09 Yup. Issue w/ L3 agent. Possibly my own, but let's see :) 23:22:59 v1k0d3n yup - as soon as your ready to begin, i think kfox will be ready 23:23:10 v1k0d3n you might need to get your test bed sorted out first 23:23:15 rather kfox1111 ^^ 23:28:23 v1k0d3n: hi. :) 23:28:45 we're still trying to nail down all the install instructions to get a base system working. that part is still very green/in progress. 23:29:19 ryan is working on a much more up to date version of the docs here: https://review.openstack.org/#/c/362467 23:29:25 so that might be a good starting point. 23:44:09 Merged openstack/kolla: Release notes for customization https://review.openstack.org/363924 23:52:05 kfox1111 getting a dev environment rolling should be a top priority i tink 23:52:13 aftre summit there will e many people attracted to the project 23:52:33 could alwys use a heat template ;-) 23:52:48 there is one kicking around that sets up kubernetes for you automaticalaly 23:53:01 larsks is the person to talk to on that point 23:53:34 you give it a ndoe count and it creates the appropriate number of kubernetes minions 00:01:15 sdake: +1 for a heat template. 00:01:31 that would be awesome. :) 00:05:11 sdake: howdy. 00:05:23 larsks thanks for coming 00:05:44 larsks hated to summon yu like that, but was curious, do you still maintian the kubernetes heat template that an be used standalone? 00:05:54 an/can? 00:06:01 Noooooooopity nope! 00:06:17 Haven't looked at it in a while, actually... 00:06:24 do you hve a link to your repo ? 00:06:32 rather can you provide one 00:19:33 kfox1111 another optoin is magnum - they do a decent job of maintaining their heat teplate for kubernetes 00:19:53 i dont know what state its in at present 00:20:21 whats the clean up/removal command? 00:21:04 jrich523 destroy 00:21:11 kolla-ansible destroy 00:21:14 sdake: sorry, distracted by local conversation. Let me grab that link... 00:21:16 jrich523 that only works on mster though 00:21:16 thanks 00:21:21 larsks cool thansk 00:21:27 sdake: https://github.com/larsks/heat-kubernetes 00:21:33 Like I said, haven't touched that in *ages* 00:21:35 kfox1111 ^^ 00:21:41 yeah no go on this version 00:21:49 do i just blow away the containers? 00:21:54 larsks roger - i think between that and magnum, kfox1111 can sort it out 00:22:00 jrich523 single node? 00:22:03 yup 00:22:19 jrich523 is there a cleanup-containers tool in /usr/share/kolla 00:22:23 or /usr/local/share/kolla? 00:22:30 /usr/share i think, but i'll check, thanks 00:22:41 run cleanup-containers -f 00:22:50 then run cleanup-host 00:22:57 make sure to back up /etc ;) 00:23:09 jrich523 a reboot in between can sometimes be hepful 00:23:14 larks how rht treating you 00:23:17 long time no chat 00:23:55 yeah i've rebooted, maybe i'll try that again... was having rabbit issues, now marie is timing out... 00:24:39 also i've noticed there are some tricks to getting this setup on vmware workstation, explains some of the much earlier problems :) 00:24:41 jrich523 we have a solution for that problem but its going throug hthe gate now 00:24:54 oh cool 00:25:02 well i'll likely rebuild again, in a couple of days 00:25:03 lol 00:25:12 i will be masterful at this in no time! 00:26:08 i did the PIP install, looks like the tools arent there (Either location) 00:31:25 jrich523 ok clone repo 00:31:35 jrich523 we need to fix those tools so they are put in the pip archive 00:31:46 git clone http://github.com/openstack/kolla 00:31:46 im sure its on the list :) 00:31:48 cd kolla 00:31:56 tools/cleanup-containers -f 00:32:12 maek sure not ot overwrite yur existing dir 00:32:25 yeah i'll clone it in to my home 00:35:00 ugh, well that fixed the marie db issue, but now im back to the rabbit issue 00:35:03 jrich523 we typically expect people to deploy over nad over to gt a feel for how oll works 00:35:12 which im starting to think i've had before, howeveer dont have a clue what i did to resolve it :-/ 00:35:18 oh im doing that 00:35:20 lol 00:35:24 which rabbit issue 00:35:33 well the error is fairly worthless, but... 00:35:40 failed: [localhost] => (item=localhost) => {"changed": false, "cmd": ["getent", "ahostsv4", "base"], "delta": "0:00:01.654630", "end": "2016-08-31 17:34:45.360238", "item": "localhost", "rc": 2, "start": "2016-08-31 17:34:43.705608", "stdout_lines": [], "warnings": []} 00:35:56 [rabbitmq | Check if all rabbit hostnames are resolvable] 00:36:00 do you ony have ipv6 configuredd? 00:36:07 is your hostname in etc/hosts? 00:36:10 hmm i dont think so, but let me double check 00:36:16 its using localhost (Based on the task data) 00:36:35 jrich525: every related host name should in /etc/hosts 00:36:39 jrich523 hostname still must be in /etc/hosts 00:36:49 run gtent ahostsv4 base 00:36:52 hmm well dns works 00:36:54 on your base os 00:37:17 hmm dont seem to have gtent? 00:37:29 nvm 00:37:30 yum search getent 00:37:58 you missed an e, but, it returns nothing 00:38:09 also it looks like ipv6 is there 00:38:09 ya its supposed to return your host name :) 00:38:22 but do you hae na ipv4 address? 00:38:26 oh crap 00:38:36 i think i know why now... 00:38:45 base was the name i used for the template i created 00:38:48 do share 00:38:50 which... wont resolve :) 00:39:04 cool 00:39:15 which teplat exactly? 00:39:22 (template an overloaded term in our architecture) 00:39:37 it sure is 00:39:43 just a vm template for centos7 00:39:47 so i can build nodes quickly 00:40:01 also less space required since they are linked clones 00:40:10 got it 00:40:19 deploying on virtual machines bound to be painful 00:43:02 zhubingbing sup dude 00:43:16 hello 00:43:17 zhubingbing yo uhave alot of work on your plate between now and sept 15th :) 00:43:24 ok 00:44:09 I am now in a hurry 00:44:16 you hae plenty of time i think 00:44:30 barbican seems most pressing 00:45:39 ok 00:45:48 i see 00:46:27 its super painful, but im getting close :) 00:47:01 sdake: did you clear out all bugs? 00:47:13 duonghq no not yet 00:47:20 duonghq i haven't taggedthe releae 00:47:55 sdake: the release is tagged after rc1 or next few days? 00:48:30 duonghq mailinglist has full explination 00:48:43 check for last msg from me 00:49:06 ah, ok 00:49:41 that fixed it... damn it 00:50:22 good to hear :) 00:52:16 hi kbaegis 00:52:19 Well, pidgin is definitely different than adium :) 00:52:21 Hi 00:52:35 Is the Horizon running now? 00:52:38 https://review.openstack.org/#/c/352515/ 00:52:47 mine is :D 00:52:49 duonghq: Yeah. The patch they proposed didn't work out for me 00:53:05 But basically removing the profiler from sites did it 00:53:14 who can help see，Gate error is a good strange 00:53:50 ah, deploy failed, interresting 00:53:59 so 00:54:01 yes 00:54:06 duonghq 00:55:32 It looks like the nova is wrong. 00:57:29 oh, novncproxy broke 00:57:58 yes 00:58:16 rechecked? 00:58:56 ping coolsvap 00:59:06 help recheck 00:59:17 zhubingbing: he is not here right now 00:59:54 ok 01:00:42 anyone can recheck 01:00:54 just type "recheck" into the comment box and click post 01:01:29 ok 01:01:32 i try 01:01:49 zhubingbing the job timed out 01:01:55 see the 1 hour 2 minutes thing 01:02:09 our gate jobs have a 60 minute timer on them 01:02:18 yes 01:03:17 gating is running slow because openstack-infra is at capacity 01:03:31 or over possibly ;) 01:04:06 Oh, I see 01:04:20 for my ps #link https://review.openstack.org/#/c/358539/ 01:04:34 given sdake and other PTL comment 01:04:35 I always thought I was a problem 01:05:09 zhubingbing huh? 01:05:14 zhubingbing somteimes patches are broken 01:05:18 sometimes gate is broken 01:05:19 sometimes both 01:05:24 sometimes inra is over capacity 01:05:33 － －ok 01:05:37 so I think there is should be pre-deploy step for setting up user, group 01:06:18 my idea is the ssh user do not need to be the config owner user, it should be in this group only 01:06:40 but if the group is add in ansible's play time, the ssh user is not have the group 01:06:55 I have been tested in the local environment, before uploading 01:07:00 so I need another ssh session, mean that another step 01:07:30 https://review.openstack.org/#/c/352515/ 01:08:19 and for remove become in every config task, I still need a finalize task in every config playbook to fix owner and group of config file 01:08:21 recheck ， no response 01:08:21 duonghq sounds like a special inventory file for setting up the local bulid node 01:08:30 and a special playbook 01:08:33 zhubingbing recheck takes awhile 01:08:40 probably 4-5 hours with current pace of infra 01:09:01 you can see status at zuul.openstack.org 01:09:13 ok 01:09:23 sdake: I do not get you idea, I think it is need only for target node? 01:09:41 duonghq oh i see wht you mean 01:09:56 duonghq is the issue you want to setup the deploy node first 01:10:02 and then the deployoment targts second? 01:10:40 sdake: the deploy node do not need many setup, imo 01:10:55 assume that image building is in other node 01:11:08 the deploy node only use ansible to deploy Kolla 01:14:02 duonghq makes sense 01:14:30 duonghq i htink that all belongs in the baremetal setup then 01:14:40 or is that not possible? 01:15:05 sdake: I saw, so I think it is need to factor out general task to other role than in baremetal 01:15:26 which general task 01:15:49 leave it in baremetal is somewhat misleading 01:15:52 at least to me 01:16:05 its called "bootstrap" 01:16:12 that is what yur doing - bootstrapping the servers 01:16:48 for the kolla-ansible tool, I agree, the only confuse is the playbook is placed in baremetal role 01:17:11 kolla-ansible bootstrap-server is totally fine 01:17:20 just the playbook placement 01:17:23 so you suggest a different role then bremetal 01:17:28 such as bootstrap? 01:17:50 I suggest different role, same level with baremetal 01:17:54 like the precheck one 01:18:13 something likes "bootstrap" role 01:18:46 duonghq - you can get a refrence of this log by clicking in the subject on "IRC channel is logged @ 01:18:49 the link is here: 01:18:52 http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2016-09-01.log.html#t2016-09-01T01:05:37 01:19:00 i'd suggest putting that in whtever work your referencing 01:19:09 so people can reference it 01:20:22 thank sdake 01:20:42 but, it's should be another bp or just stick into my bp? 01:21:51 I prefer new bp: bootstrap bp (even it's only goal is factor out sean-k-mooney work), 01:25:30 hey kfox1111 sorry late, grabbed some dinner. 01:25:50 sure, i'd actually like to work on some of the docs...i learn the most that way. 01:26:19 but dev is probably pretty high with summit soon 01:26:27 v1k0d3n i think kfox1111 bounced fo the day 01:26:46 v1k0d3n ryan ha a review in teh queue for docs changes 01:26:54 v1k0d3n you could alway review it and provide feedback 01:26:58 the docs need more love 01:27:10 larsks: man, it's been a while! anything i can help with? (sorry catching up from earlier messages). 01:27:24 ok 01:27:45 * sdake needs some anestaphine rom the medlock 01:31:03 Li Yingjun proposed openstack/kolla: Add option ceilometer_database_type for ceilometer https://review.openstack.org/345881 01:32:00 v1k0d3n review.openstack.org 01:32:01 log in 01:32:10 set watched projects to kolla and kolla-kubernetes 01:32:19 then click "my->review" iirc 01:32:27 or my watched i think 01:53:44 time to take wife out to birthday dinner 01:53:45 bbl 01:55:19 snake 01:55:29 snake happy birthday 01:55:46 snake 01:56:18 Enjoy the birthday dinner 02:00:02 its her birthday 02:02:28 sounds good. thanks sdake i'll get on it in the morning. have fun wtih the wife! 02:04:08 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add Cinder api and scheduler components https://review.openstack.org/363044 02:04:55 sdake: have a good time ;) 02:14:24 v1k0d3n: hey! :) just passing through on sdake's request... 02:20:10 zhubingbing: I made it 02:20:13 *you 03:10:13 inc0, sean-k-mooney: in case you can read this message: the "get real node ip" task still fail, due to 'dict object' has no attribute 'baremetal_nic', don't known which Ansible version do you use? 03:10:58 zhubingbing: are you there? 03:11:19 there 03:11:34 have you tried the bootstrap-servers? 03:11:40 kolla-ansible bootstrap-servers 03:11:40 yes 03:11:46 duonghq: I'm pretty sure you have to configure baremetal_nic. It's for Bifrost 03:11:47 the get real node ip work? 03:11:56 work 03:12:12 hmm, I expected it can get value from baremetal/default 03:12:47 can you take a look at ansible/roles/baremetal/defaults/main.yml 03:12:56 I think it specifies a nic on the deploy host not on the nodes 03:14:24 but does it should have a default value? 03:14:32 api_interface is reasonable choice? 03:14:58 rephrase: if the playbook cannot get value from defaults, so why it is placed here? 03:27:20 zhubingbing proposed openstack/kolla: Add Barbican container dockerfile https://review.openstack.org/351822 03:27:59 oh 03:28:45 duonghq: hmm. I'm not sure why you don't have a default. it looks like there is a default in defaults 03:29:22 I mean this one mark-casey 03:29:30 but the playbook cannot get this value 03:29:43 due to Ansible natural 03:29:55 don't know why everybody can use it 03:30:23 natural? 03:30:39 mark-casey: You can see explanation here: https://launchpad.net/bugs/1618381 03:30:39 Launchpad bug 1618381 in kolla "Baremetal playbook append_to_etc_hosts variable error" [Undecided,In progress] - Assigned to Duong Ha-Quang (duonghq) 03:30:58 https://github.com/ansible/ansible/issues/13343 03:31:46 it's about how hostvars work 03:34:01 * duonghq going to take a lunch for awhile, hope that somebody has good idea about this case 04:09:00 zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 05:00:21 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 05:06:17 morning 05:13:26 Md Nadeem proposed openstack/kolla: Added Ansible playbook for congress deploy https://review.openstack.org/350629 05:17:19 ping Mathias 05:25:17 Md Nadeem proposed openstack/kolla: Update doc for congress https://review.openstack.org/364074 05:25:45 ping Mathias 05:27:08 hey folks 05:27:48 zhubingbing mathias is mewald 05:27:58 ok 05:28:02 ping mewald 05:28:07 thanks 05:28:12 zhubingbing shouldn'tnaked ping imo 05:28:18 ping mewald subject works better 05:28:26 o 05:28:44 unless its personal in nature 05:28:50 then nakid pingis fine in y book ;) 05:28:52 my 05:30:47 i 05:30:48 see 05:31:34 morning guys 05:31:52 zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 05:32:00 hey mewald 05:32:12 zhubingbing: I read your mail. You're welcome to support my patches. I'll be leaving for vacation on Friday so we need to get going :) 05:32:16 we made decision today to keep your blueprint in the releae 05:32:29 sdake: cool! 05:32:31 the deadline is Spet 15 not aug 15 05:32:35 cool 05:32:37 yeah though so ;) 05:32:41 +t 05:32:44 sorry aobut that 05:33:01 but as sept 1 rolls around i'll start to kick things out of the release 05:33:06 I need people to review you know, I would like to have it merged already 05:33:07 I try to finish it today. 05:33:14 mewled 05:33:22 mewald roger - i've sked the core team to focus on our blueprints for review 05:33:30 the rc1 blueprints thatis 05:33:34 nice! let's do it 05:33:39 mewald also wehave two new core reviewers 05:33:42 ok 05:35:04 my focus is on influxdb-role (346449) and telegraf-role (346448) patches now. Once they're accepted it makes sense to worry about grafana. Otherwise I'll be rebasing most of my time 05:36:25 mewled ，I will make some changes on your basis. 05:36:40 sdake: problem with the gate failing for telegraf is that there is no centos image for influxdb. I only developed for ubuntu 05:36:59 i can fixed it 05:37:21 zhubingbing: you submit patches for centos? 05:37:56 ubuntu and centos 05:38:07 every thing 05:39:19 mewald there is an exception liston the buids in the test gate 05:39:37 we eliminate failed builds for binary in some cases 05:39:42 moment - link coming 05:40:43 https://github.com/openstack/kolla/blob/master/tests/test_build.py#L68 05:41:16 mewald is there a different choice then infuxdb? 05:41:21 influxdb has no clustering support 05:41:29 they are an opencore model company 05:42:34 sdake: I think it can work like this https://github.com/influxdata/influxdb-relay/blob/master/README.md 05:43:15 mewald i think logn term that model wont work 05:43:31 because influxdb co has said no open source implementation of influxdb will be available 05:44:31 i think zhubingbing had a link on it 05:45:09 lets go with it for now, and we can rework in the future if needed 05:45:23 hmm, atm I don't see a lot of other options I could help with. I mean that Gnocchi thing: the other guy from the team posted on our mailing list it wouldnt perform. Graphite is a lot worse in performance in my experience and about clustering I dont know 05:45:25 but i'd start looking for an alternatie 05:45:40 ok we have to keep that in mind 05:45:48 not concerne dwith performance as much as ha 05:46:17 really? for monitoring? imho performance data may be lost, no problem 05:46:38 possible 05:46:40 i've been drinking 05:46:44 so judgement off :) 05:46:53 :D cheers 05:47:13 mewald you mentioned vacation 05:47:17 when will ou return? 05:47:35 Sept 19th 05:49:09 so what's required for the patches now? implement centos support or add an exception for the tests to have everything green and shiny in Gerrit? 05:49:30 if we need centos support now, we need further patches for already merged stuff 06:02:10 mewald centos source support is required 06:02:14 centos binary support is not required 06:02:21 exception list for binary builds is acceptable 06:03:08 sept 19th - we may have ot remove this from rc1 then 06:03:24 hey egonzalez90 06:03:34 depends on how many review I get today and tomorrow ;) 06:03:39 morning sdake 06:03:52 mewald another optoin is for someone to take over your work 06:03:57 while your out 06:08:38 sup coolsvap 06:08:49 sdake: hey 06:09:10 sdake: moving the bugs to o-1 06:09:21 mewald we made that decision without knowledge ofyur impending pto 06:09:46 sdake: somebody else taking over is fine with me 06:09:51 mewald would you be kind enough to state on the mailing list a link to the blueprint and when you depart for pto and return, and sk for someone to take it over then? 06:10:08 mewald maybe someon will bite :) 06:10:27 coolsvap which bugs? 06:10:45 the ones i sent you yesterday 06:12:21 sdake: we need to do some serious documentation 06:12:26 there are around 15 doc bus 06:12:29 bugs 06:14:18 coolsvap pretty sure we have a pla nin place for that 06:15:00 these are mostly docimpact bugs 06:15:22 a reminder is required imo 06:15:52 yup lets keep the reminder in place 06:16:08 (i.e. the bugs) 06:16:10 i think https://bugs.launchpad.net/kolla/+bug/1613292 can be moved 06:16:10 Launchpad bug 1613292 in kolla mitaka "Failed to open some config files: /etc/ironic/ironic.conf" [High,Confirmed] - Assigned to Waldemar Znoinski (wznoinsk) 06:16:12 but kick them to ocata 06:16:23 all doc bugs? 06:16:38 i think so 06:16:46 sdake: alright 06:16:49 even though docs are high priority 06:16:57 mdnadeem: congress dockerfiles merged? 06:16:58 we need to fix bugs in the software first 06:17:15 sdake: will do and will send a mail on list 06:17:22 for reminder 06:17:28 coolsvap mail related to docs? 06:17:37 yea 06:17:39 coolsvap at midcycle we had a pretty well thought out docs session 06:17:57 coolsvap the problem is we can't gte all the core team in the same place at the same time to hash over the docs design 06:18:08 we hae not yet designed our documentation 06:18:12 although ryan is working on it 06:18:28 sdake: the doc bugs i am talking about is not about design 06:18:43 coolsvap you misunderstand me 06:18:53 what I mean is desinging documentation is an engineering taask 06:19:01 we need people to write docs corresponding to docimpact they wrote in commit messages for things 06:19:19 we can update and move things around during design 06:20:30 sdake: i agree, yes design is engineering task 06:21:01 but what i want is docimpact bugs and the engineering task saperated 06:21:17 coool email sounds good there 06:21:20 when you say you need documentation for some stuff you have added/updated 06:21:22 maybe you can tag the bugs with docimpact 06:21:26 so peopel can find them more easily 06:21:34 Takashi Sogabe proposed openstack/kolla: Keepalived leaves VIP on the host in case of accidental removal. https://review.openstack.org/364085 06:21:34 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 06:21:34 yes i will 06:21:41 i dont really like docimpact at all 06:21:49 i think docs should go in with the code personally 06:22:03 add the documentation, if it need design update, person responsible will take care of it or contact you 06:22:08 but a battle or a different day 06:22:09 Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 06:22:12 mewald1 fwiw gate going slow 06:25:20 Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 06:27:09 Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 06:27:17 I just added the exceptions for centos binary 06:28:57 coolsvap, yes 06:29:45 mdnadeem: in n-3, i cannot see as part of bp/bugs 06:31:23 coolsvap, https://blueprints.launchpad.net/kolla/+spec/enable-congress-container 06:32:57 coolsvap we moed some features around 06:33:09 but if congress is ready to go might as well move it into rc1 06:33:37 coolsvap it was part of our triage the blueprints day in our team meeting. 06:35:48 sdake: yeah i missed update for congress thanks mdnadeem 06:36:34 sdake, coolsvap , https://review.openstack.org/#/c/350629/ 06:36:49 sdake: should we move the needs code review bps and in progress bugs to rc1? 06:37:13 yes plese 06:37:18 the release hs been submitted 06:37:39 i'd do it but out of steam for the day 06:37:45 been long and laoriosu :) 06:37:50 laborious 06:53:06 sdake: done now n-3 looks good 06:53:27 zhubingbing, ping 06:53:34 here 06:54:15 coolsvap nice 06:55:16 coolsvap was congress part of milestone 3? 06:55:39 containers are part of rc1 06:55:50 what about playbooksk? 06:55:56 both 06:56:20 ok 06:56:22 wait 06:56:31 i want to keep the feature creep in rc1 to a minimum 06:56:38 we have alot of work to do on the bug front 06:58:08 coolsvap all i see or congress is a containe in rc1, what about the playbooks - where is that blueprint? 06:58:26 congress container + playbooks in a single bp 06:58:34 sdake: ^^ 06:58:40 thats what i was also confused about 06:59:13 afaics, it looks in good shape, haven't seen in detail 06:59:17 can be part of rc1 06:59:37 * coolsvap brb lunch 07:03:24 Merged openstack/kolla: add deploy-servers command https://review.openstack.org/349729 07:04:13 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 07:16:22 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 07:24:19 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 07:36:05 zhubingbing: you are doing rally? nice! in case you look at tempest too: The container is broken imho 07:36:14 yeah 07:37:06 tempest ubuntu ？ 07:37:27 tempest ubuntu or cents binary? 07:39:23 ubuntu 07:39:36 something is wrong with dependencies of python modules or so 07:43:18 Is https://review.openstack.org/#/c/346449 good enough now regarding the testing stuff? 07:44:05 Merged openstack/kolla: Update doc for congress https://review.openstack.org/364074 07:45:17 zhubingbing proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 07:45:26 mewald1 07:45:45 hi i have change influxdb 07:53:03 morning 07:53:17 Merged openstack/kolla: replace baremetal_nic with api_interface https://review.openstack.org/363916 07:53:56 morning 08:01:48 zhubingbing proposed openstack/kolla: Add Barbican container dockerfile https://review.openstack.org/351822 08:05:16 Yao Lu proposed openstack/kolla: delete virtual bridge artifacts when cleaning hosts https://review.openstack.org/358924 08:06:41 Mathias proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 08:07:08 zhubingbing proposed openstack/kolla: Add Barbican container dockerfile https://review.openstack.org/351822 08:08:32 Merged openstack/kolla: Fix service_type of mistral endpoint https://review.openstack.org/352120 08:10:11 Mathias proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 08:11:43 submitted for 3 fb 15 wr, 1 half day contributor meetup 08:11:45 Jeffrey4l ^^ 08:11:48 coolsvap ^^ 08:11:50 Merged openstack/kolla: Fix using filter() to meet python2,3 https://review.openstack.org/361950 08:11:50 egonzalez90 ^^ 08:12:03 and anyone else that may cre ;) 08:12:43 sdake: looks good 08:12:54 warning we wont et all tht pce 08:12:57 space 08:13:00 coolsvap: can you review it: https://review.openstack.org/#/c/362579/ 08:13:02 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 08:13:18 i asked for fb's ahead of workroom sessions and priority on half day contributor meetup on friday faternon 08:13:33 so we can shift wr sessions into the half day contributor meetup if needed 08:13:55 sdake: need your view on https://review.openstack.org/#/c/349295/ 08:14:41 sdake: yea frm past experience we need more time for some sessions and move things into contributors meetup 08:16:36 coolsvap view made available 08:21:35 coolsvap: ping 08:21:46 duonghq: pong 08:21:59 can you review this: https://review.openstack.org/#/c/362579/? 08:22:59 duonghq: on it 08:27:04 thank coolsvap 08:27:35 Jeffrey4l: ping 08:37:43 Mathias proposed openstack/kolla: Added grafana role https://review.openstack.org/346450 08:37:44 Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 08:37:44 Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 08:37:45 Mathias proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 08:38:15 holy crap, hope I did the rebase of https://review.openstack.org/#/c/346450/ right … so many notifications xD 08:54:59 Christian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container https://review.openstack.org/335495 08:55:35 zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 08:56:32 mewald1 looks ok to me - but i'm beat and need to hit the rack 09:02:22 Nguyen Phuong An proposed openstack/kolla: Ensure doesn't use LOG.warn https://review.openstack.org/364168 09:06:01 Paul Bourke proposed openstack/kolla: Check volume group exists for Cinder in prechecks https://review.openstack.org/354154 09:06:14 Christian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container https://review.openstack.org/335495 09:06:49 pbourke did you want to have that discussion about the releae process now? 09:06:58 sdake: sure why not 09:07:13 coolsvap: https://bugs.launchpad.net/kolla/+bug/1585955 would the merged patchset be enough to mark it as done in newton? 09:07:13 Launchpad bug 1585955 in kolla " add support for non-default database_port in kolla/ansible" [High,Confirmed] - Assigned to Waldemar Znoinski (wznoinsk) 09:07:21 there is a repository called http://github.com/openstack/releases 09:07:40 Paul Bourke proposed openstack/kolla: Add guidelines on adding a new service https://review.openstack.org/361253 09:07:57 if you take a look in there, there is a directory called deliverables 09:08:50 for us to release a milestone or rc, the release liason (in this csae, me) submits a change like: https://review.openstack.org/364007 09:09:13 the release team has pretty much automated everything 09:09:28 once a workflow occurs on the patch, the release is tagged and pushed to pip etc 09:09:34 pypi 09:09:44 we also release z streams every 45 days 09:09:56 a z stream = x.y.z 1.1.2 2=x 09:09:59 2=z i mean 09:10:16 we have z streams due next week for 1.2.3 and 2.0.3 09:10:59 in order to facilitate the backport process from mster to the stable releases, we use launchpad to trrack the status of the bug 09:11:43 mewald1 i need ot hit the rack 09:11:59 but i'll grab all the material i can scour off my disks and send to you tomorrow morning 09:12:29 if its in msater, we use the target to series feature of launchpad 09:12:49 openstack relesae team mechanics integrate with launchad to tell us which bugs are fixedi n a zstream and which ones re not fixed 09:13:04 we find this information via launchpad 09:14:00 for example: https://launchpad.net/kolla/+milestone/1.1.3 09:14:19 we are behind on backports 09:14:35 as you can see from that milestone list for 1.1.3 09:14:44 sdake: so that seems ok so far, but doesn't address why we need to enforce a bug for every feature 09:14:47 by using launchpd, we can esily see what needs to be backported 09:15:07 we dont need da bug for every feature 09:15:12 only things thta need bckports 09:15:17 ok 09:15:31 starting to make more sense :) 09:15:31 i thought this was pretty clear but apparenty not :( 09:16:16 i f its a blueprint its typicl to use the bluerpint tracking which is also integrated with launchpad 09:16:37 openstack s a whole is going to bugs only for features and deprecating the blueprint model 09:17:00 how that effects what we currently do, no idea 09:17:02 zhubingbing proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 09:17:17 Christian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container https://review.openstack.org/335495 09:17:18 Christian Berendt proposed openstack/kolla: Integrate neutron-vpnaas-agent into the neutron Ansible role https://review.openstack.org/335502 09:17:37 the reason we want to track patches in a blueprint is to help do what we did today in our team meeting - triage status of blueprints 09:18:26 this is why we have work items 09:18:32 so we can compare work item list to patch set 09:18:47 and make a determination if the blueprint has met its definition of done 09:19:13 pbourke i need to hit the rack - would you mind posting a link to this irc discussion on the thread 09:19:26 can somebody review https://review.openstack.org/#/c/336943/, one +2 missing, small change 09:19:26 will do, thanks sdake 09:20:02 same for https://review.openstack.org/#/c/349006/ 09:21:22 pbourke any questions about all that? 09:21:29 sdake: dont think so 09:21:43 I did want to understandd daviey's idea of using change ids to track backports 09:21:59 sdake: how does reno fit in 09:22:08 sdake: does release teams tools publish those somewhere? 09:22:16 wznoinsk: The merged changed adde in comment #1 is the same which created this documentation bug. This bug needs documentation update. 09:22:16 reno is a tool for publishing release notes 09:22:18 yes moment 09:22:48 https://releases.openstack.org/teams/kolla.html 09:23:43 see mitaka: http://docs.openstack.org/releasenotes/kolla/mitaka.html 09:23:44 can somebody approve https://blueprints.launchpad.net/kolla/+spec/cloudkitty ? i changed series and target to occata 09:24:08 sdake: cool 09:25:47 can somebody review this: https://review.openstack.org/#/c/364168/1 09:25:49 berendt: one question related to https://review.openstack.org/#/c/349006 added in comment 09:25:57 see current: http://docs.openstack.org/releasenotes/kolla/unreleased.html 09:26:19 sdake: we can probably update the guidelines then as follows: 09:26:22 berendt: did you create a blueprint for ansible-cloudkitty? 09:26:32 sdake: trivial bugfix = add TrivialFix 09:26:41 sdake: non trivial bugfix *or* backport: needs bug id 09:26:55 sdake: feature that does not need backport: no tags necessary 09:27:18 pbourke i'd like to continue using blueprints for the 3rd line you posted 09:27:27 sdake: ++ 09:27:31 makes sense 09:27:34 i was going to say the same 09:28:04 pbourke there is also another consideration 09:28:13 if an operator reports a bug 09:28:18 we should usse tha tbug id even if its trivial 09:29:32 pbourke what we need really is a policy document 09:29:37 sdake: agree 09:29:43 as was previously suggested on the ml 09:29:51 i keep meaning to get to that 09:29:57 as soon as fire #999999 is put out 09:30:25 we dont have a whole lot of policies 09:30:43 i think a project runs best when it uses as little process as is necessaray to make the mechanics work 09:30:53 (process/policy) 09:31:29 my fire availability is only 6 9s :) 09:31:37 fire-fighting that is 09:32:18 if you look at neutron for e.g. they have a great policy document 09:32:32 but they have a much mor ecomplex project to deal with 09:32:43 3x as many contributors 09:32:48 3x as many reviewers 09:33:11 but i dont htink we need to go bonkers on the policy document 09:34:43 coolsvap i am not sure about a 2nd blueprint because i think nobody will implement the binary type for the docker images until rc1 09:35:21 berendt that is ok - binray typically trials source 09:35:30 trails 09:35:53 berendt: we need one for ansible as well 09:36:16 coolsvap so always 1 bp for docker and 1 bp for ansible? 09:36:34 berendt: not necessarily 09:36:43 berendt we are operating this way becaue we are planningto split the repositories 09:36:57 ansible review seems is not active for long 09:37:13 so there will in the future be separte launchpads - one or ansible one for docker containers 09:37:34 which mostly will be bumped to o-1 09:38:03 Christian Berendt proposed openstack/kolla: Add cloudkitty containers for type source https://review.openstack.org/349295 09:38:11 Ugh, lots of scrollback 09:38:35 Daviey its my magnum opus about our process 09:38:43 but i'm going to bed 09:38:44 feel free to hash on the ml :) 09:38:52 i'll read in morning 09:38:54 coolsvap yes, i am waiting for gnocchi until i continue the ansible part 09:40:26 coolsvap i renamed the existing blueprint to cloudkitty-docker and created https://blueprints.launchpad.net/kolla/+spec/cloudkitty-ansible (needs approval) 09:40:53 berendt: cool 09:40:54 coolsvap i set milestone to occata-1 and series to occata for cloudkitty ansible and will change cloudkitty-docker to newton-rc1 and newton. that's correct? 09:42:53 berendt: yeah 09:43:13 coolsvap done 09:47:41 Merged openstack/kolla: Install neutron-lbaas-dashboard from source in horizon container https://review.openstack.org/336943 09:49:03 Yao Lu proposed openstack/kolla: delete virtual bridge artifacts when cleaning hosts https://review.openstack.org/358924 09:49:09 coolsvap can you have a look at https://review.openstack.org/#/c/359451/. at the moment i am not sure why deploy_aio tries to download images from docker.io. 09:50:26 berendt: sure but in half n hour or so 09:54:15 why deploy gate error 09:55:19 berendt looking moment 09:57:11 snake 09:57:18 coolsvap no rush.. 09:57:44 sdake can you look sahara role and dockerfile 09:58:03 https://review.openstack.org/#/c/351294/ 09:58:05 zhubingbing going to bed 09:58:16 ok 09:58:17 berendt the issue is the profile your buliding 09:58:25 night sdake 09:58:30 zhubingbing would like to hep - but 3am and need to protect my sleep :) 09:58:37 - - 09:58:43 berendt ping pbourke - he can get you going on the profiel stuff 09:58:51 sdake go to bed :) i will check this later in more detail 09:59:00 berendt note - i doubt we can enable heat and horizon in the gate because of memory concerns 09:59:17 concerns/constraints 09:59:19 sdake hmm. that's bad. 09:59:20 can't compose a sentence 09:59:24 ya 8gig ram 09:59:36 we need to break our gate jobs into seprate gates for arious thigns 09:59:56 i wish we could get 16 gig vms 10:00:03 that woudl solve alot of stuf for us 10:00:12 i am going to havea look at multinode gating next cycle 10:00:43 and see if I can get it running in such a way as to have 2 nodes to deploy across 10:00:44 Nguyen Phuong An proposed openstack/kolla: TrivialFix: Ensure doesn't use LOG.warn https://review.openstack.org/364168 10:00:47 1 compute and 1 controller 10:01:01 with 8gb of ram we can deploy a whole lot of controller 10:01:13 its when we launch a nova vm the system implodes 10:01:21 with 8gb and heat 10:01:25 het has a lot of threads 10:01:34 horizon is probably safe to enable 10:01:36 night all :) 10:20:36 Christian Berendt proposed openstack/kolla: Enable deployment of horizon in tools/deploy_aio.sh https://review.openstack.org/359451 10:34:44 Christian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container https://review.openstack.org/335495 10:35:28 Christian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container https://review.openstack.org/335495 10:46:45 Christian Berendt proposed openstack/kolla: Remove fedora from Dockerfile templates https://review.openstack.org/359447 10:48:44 narasimha18sv proposed openstack/kolla: Updating ironic configurations in nova.conf https://review.openstack.org/364228 11:25:05 Merged openstack/kolla: Use six.StringIO https://review.openstack.org/361797 11:41:31 Merged openstack/kolla: Add ansible raw task note, add apt-get update https://review.openstack.org/362579 12:17:06 zhongshengping proposed openstack/kolla: Deprecate old auth parameters of ironic in nova.conf https://review.openstack.org/364259 13:20:39 Merged openstack/kolla: Added Ansible playbook for congress deploy https://review.openstack.org/350629 13:29:50 Jeffrey Zhang proposed openstack/kolla: Use a local docker image namespace in gate https://review.openstack.org/364310 13:38:48 Kevin Fox proposed openstack/kolla: ip tool to the toolbox https://review.openstack.org/364317 13:57:53 good morning 14:00:45 morning. 14:00:59 inc0: can we get this through quickly? https://review.openstack.org/364317 14:01:16 I was reallly surprised neither ip or ifconfig was in kolla-toolbox 14:01:41 hmm...hold on I think I've seen this in base lately tho 14:02:19 https://github.com/openstack/kolla/blob/b4c505a6c8da19be438cbf03e28a4d3f82099a11/docker/openstack-base/Dockerfile.j2#L9 hmm 14:02:53 odd. 14:03:10 this is only for several distros and in openstack-base 14:03:22 which toolbox doesn't inherit from afair 14:03:35 maybe it's worth to install it in base image 14:03:36 ? 14:03:56 kfox1111: if it is not too early for you ;-) https://review.openstack.org/#/c/363044/ with last comments 14:03:58 but my question is 14:04:12 ah, well k8s will need it 14:04:19 as it's not net=host everywhere 14:05:25 inc0: either base image or toolbox works for me. but its broken right now for some reason. 14:05:55 sbezverk: let me double check the serviceName thing... I thought it had to be unique. 14:07:40 kfox1111: nova uses the same approach 14:07:48 http://kubernetes.io/docs/user-guide/petset/ 14:07:58 $(petname).$(governing service domain), where the governing service is defined by the serviceName field on the Pet Set. 14:08:29 and since the petname is the petset name with index, that should be unique. 14:08:45 yeah. that should work. 14:09:27 ok. loos good. :) 14:09:30 looks 14:09:35 thanks 14:09:59 plesae act accordingle ;-) I want to get my hands on cinder volume 14:10:47 I just got nova-compute in container talking to libvirt on the raw host. 14:11:03 will document how to do that... 14:12:34 Kevin Fox proposed openstack/kolla: ip tool to the toolbox https://review.openstack.org/364317 14:12:37 kfox1111, I'd put it in base tbh, make sure to include info that this is needed for kolla-k8s and since kolla-k8s doesn't use net=host, you can't do stuff from outside of container 14:13:00 k. 14:13:10 but move it to base, I'm sure it will come in handy in other places too 14:15:00 sbezverk, what do you think about an in cluster container to work with the clients? 14:15:11 much harder in base. :/ 14:15:33 Jeffrey Zhang proposed openstack/kolla: Use the standard start method for kolla-toolbox container https://review.openstack.org/364344 14:15:34 rhallisey: I'm for it. :) 14:15:43 kfox1111, ok 14:15:51 would make for a nice demo tbh 14:15:57 also for basic testing 14:16:03 it can maintain some state and lock things so if multiple admins are doing stuff at the same time, badness doesn't happen. 14:16:04 and it could generate and openrc! 14:16:32 start your container, exec in, and your good to go 14:16:36 I'd have a look at the new helm. they do the same, and use configmaps for storage. 14:16:44 so its pretty stateless. 14:17:12 yeah. I'd like to see a container with kolla/kolla-kubernetes in it so you can just run it and go. 14:17:22 Ryan Hallisey proposed openstack/kolla-kubernetes: Extra space in generate-passwords.py https://review.openstack.org/364346 14:18:00 inc0: will putting it in base slow down the review? 14:18:11 Jeffrey Zhang proposed openstack/kolla: Use a local docker image namespace in gate https://review.openstack.org/364310 14:18:12 theres a lot more places I have to touch and following the if logic is messy. 14:18:22 doing it for now in just kolla-kubernetes its just in 2 places. 14:18:26 kfox1111, well, I don't think so, but if you need it *right now* you can use customizations 14:18:44 before change merges 14:19:22 kfox1111, http://docs.openstack.org/developer/kolla/image-building.html#dockerfile-customisation 14:19:31 well even before the custimisations you could always install addtional packages using the header support 14:19:39 no, I'm more worried about getting in for 3.0.0. 14:20:04 kfox1111, I think we can merge it, I'm saying you can do this at this moment 14:20:14 k. 14:20:16 I'd rather you put it to base as it can be useful 14:20:39 I'm pretty sure it will be at some point and I want to avoid having multiple installs of it 14:22:04 rhallisey: it could be an option but then how you want external clients/people to use this container? 14:22:21 sbezverk, they won't 14:22:29 it's for in cluster development 14:22:40 no for production 14:22:49 rhallisey: got it. 14:23:19 rhallisey: as you know I resolved it by using external haproxy, Kevin also has a solution.. 14:23:33 nice 14:23:37 should document them 14:24:05 rhallisey: but that container will only be testing internal interactions 14:24:27 ya that's enough for development 14:24:35 or even an AIO setup 14:24:57 rhallisey: which image you plan to use for it as base? 14:25:06 kolla-toolbox 14:25:07 toolsbox? 14:25:12 cool 14:25:16 it has all the clients 14:26:42 Kevin Fox proposed openstack/kolla: ip tool https://review.openstack.org/364317 14:26:59 inc0: I think that should move it to base. 14:27:50 Jeffrey4l, so with kolla-k8s without net=host this thing will be needed every time you need to check local IP 14:27:55 which can be pretty often 14:28:19 inc0: even without net=host you may need to figure out what ip you have. 14:28:36 well not that often tho - we have ansible for that 14:28:37 inc0, the best solution is adding iproute in base-image 14:28:39 the container is light weight though. 14:28:41 right? kfox1111 14:28:56 Jeffrey4l, check the latest review 14:28:59 it's in base 14:29:01 OK. 14:29:16 well, for what I need, actually the best solution might be to put crudini/jq/ip in its own super lightweightcontainer. 14:29:32 let's optimize later 14:29:41 inc0: ansible has no visibility of kube cluster, and in reality it does not need it 14:29:52 sbezverk, that's why I'm saying we didn't need it before 14:30:04 Merged openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 14:30:15 ansible will use just one single entry point to api server, everything else kube takes care of it 14:30:21 ansible was our tool for the job before, and with k8s we need different 14:30:58 kfox1111, a nit 14:31:09 see the comments in https://review.openstack.org/364317 14:32:25 Kevin Fox proposed openstack/kolla: add iproute pacagke in base image. https://review.openstack.org/364317 14:32:31 ammended 14:32:41 thanks for the quick reviews. :) 14:36:04 kfox1111, hmm typo still exist https://review.openstack.org/364317 :) 14:36:59 I admit to the second typo. the first was copy paste. :) 14:37:00 fixing... 14:37:14 i am sorry for that. :) 14:37:42 Kevin Fox proposed openstack/kolla: add iproute package in base image. https://review.openstack.org/364317 14:37:43 no worries. :) 14:42:52 ok. the patch seems to work. 14:43:11 and it fixed the kolla-kubernetes issue. 14:44:25 cool 14:45:15 sbezverk: with the init containers, I think we want to change /bin/sh -c ... to 14:45:21 /bin/sh -ec ... 14:45:39 so if any of the script fails, the init container fails and blocks to whole pod creation. 14:45:42 much easer to debug. 14:47:58 Li Yingjun proposed openstack/kolla: Add option ceilometer_database_type for ceilometer https://review.openstack.org/345881 14:56:02 yingjun, ping 14:56:16 Jeffrey4l, pong 14:56:28 yingjun, about this https://review.openstack.org/345881 14:57:02 could u remove remove the ceilometer_database_type variable? I do not think we need add this now. 14:57:18 we can add this when we really have two options. 14:57:24 yingjun, ^^ 14:57:57 Jeffrey4l, well technically we have mongo, galera and now gnocci 14:58:28 inc0, but we are not implement those now. the only one valid option is mongodb 14:58:29 anyway, I'd wait for Ocata with this merge 14:58:42 I don't want to add new config options 14:58:45 in newton 14:59:19 so I agree, let's merge this just after newton tag 14:59:21 inc0, OK. but add the check and fail fast ( enable ceilometer without enable_mongo ) is nice to have in newton. 14:59:56 nah, I agree we shouldn't have it in newton. 14:59:57 so i perfer to remove the variable and just add the check task. 14:59:57 I’m ok removing the config, and upload another patch for the new option 15:00:23 yingjun, cool. thanks. 15:01:13 kfox1111: hm whenever my init was failing, it was bloking pod from being created, it was stuck on somehting like waiting for init.. 15:02:36 Jeffrey Zhang proposed openstack/kolla: Liberty is now on 1.1.2 prerelease https://review.openstack.org/334637 15:05:49 Merged openstack/kolla: Correct permissions to /etc/keystone/domains https://review.openstack.org/350484 15:07:21 pbourke, about https://review.openstack.org/#/c/345881, it was suggested that a simple pre-check is not enough 15:08:28 yingjun: Jeffrey4l: I dont see point in duplicating prechecks like that 15:08:47 if user runs deploy without prechecks then of course there could be a prolem 15:08:49 *problem 15:09:02 pbourke, i think it is necessary to add the the check in bootstrap. 15:09:22 the pre-check only work before deploying. 15:09:24 whats the point in having a separate precheck playbook then 15:09:40 after deployed, pre-check is not work. 15:10:12 pbourke, and some guys may do not run pre-check and run deploy directly. silence ignore is not good. 15:10:39 ok 15:12:50 Yeah, that happens often for operators using kolla not running pre-check before deploying. 15:12:58 at least for me... 15:13:27 pre-check is not good. we need improve it in the O 15:18:51 Li Yingjun proposed openstack/kolla: Add precheck for ceilometer backend https://review.openstack.org/345881 15:21:03 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 15:21:49 sbezverk: yeah. but I think without the -e, it only does that if the last command in the script fails. 15:22:01 the -e causes it to always fail if any of the lines fail. 15:22:20 kfox1111, may be we need add set -e to all scripts. 15:22:26 guys, what's best way to deploy all the stuff with k8s? 15:22:36 inc0: isn't one yet. 15:22:40 fast failure is good 15:22:42 duh 15:22:45 thats the workflow stuff. :/ 15:22:54 so ansible is not there yet? 15:22:59 not yet. 15:23:03 need a dev to work on that. :) 15:23:04 inc0: your hands is trhe fastest way to deploy ;-) 15:23:28 can you type in commands in order that needs to happen? 15:23:33 Merged openstack/kolla-kubernetes: Add Cinder api and scheduler components https://review.openstack.org/363044 15:23:41 from "I installed k8s" to "yay it works" 15:23:56 morning folks 15:23:58 inc0: the link I pasted to you yesterday 15:24:00 Li Yingjun proposed openstack/kolla: Add precheck for ceilometer backend https://review.openstack.org/345881 15:24:04 sdake: Morning 15:24:10 inc0: theres a doc in proress for that. but its still in review and in progress. 15:24:11 gives you very close to what you asked 15:24:32 inc0: https://review.openstack.org/#/c/362467/ 15:26:00 inc0: need https://review.openstack.org/#/c/364317/ applied currently. 15:28:16 kfox1111, I feel like your private ack machine for last 2 days:P 15:28:21 there ya go 15:28:43 Kevin Fox proposed openstack/kolla-kubernetes: Fail nova-compute init container always https://review.openstack.org/364387 15:28:51 inc0: sorry. :/ 15:28:56 nah, don't be 15:29:01 I want k8s to happen 15:29:11 after all...I'm supposed to demo it in Barcelona 15:29:17 heh. 15:29:21 with wirehead and rhallisey 15:29:23 then we better make sure it works. :) 15:29:32 yup 15:31:13 pbourke, please review one more time for https://review.openstack.org/#/c/345881/, new patch uploaded for code cleanup, thanks! 15:32:43 current the gate bust. error message is (1832, u"Cannot change column 'standard_attr_id': used in a foreign key constraint 'qos_policies_ibfk_1'") 15:32:54 it should be a bug in neutron. any one know the bug id? 15:33:33 morning 15:33:53 morning :) 15:33:56 Jeffrey4l ask on #openstack-neutron? 15:34:02 ok. 15:34:27 Jeffrey4l sounds like a databae migration script problem 15:34:54 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 15:35:56 sdake, yep 15:37:09 wow. almost all of the kolla-kubernetes patches are merged. :) 15:37:10 Merged openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 15:37:20 * kfox1111 makes note to submit some more... ;) 15:37:36 kfox1111, sbezverk I broke it:( 15:37:54 Error from server: configmaps "mariadb-configmap" already exists 15:38:08 I did bootstrap maria and run maria 15:38:32 inc0, nah that's fine 15:38:33 I think that isn't fatal. 15:38:42 inc0, you need the password patch I put in 15:38:48 well kubectl get pod is empty 15:38:51 or your passwords wil be empty 15:38:59 brb 15:39:11 heh. messed up that order... oh well. 15:39:45 morning guys 15:40:08 good morning 15:41:17 morning 15:42:26 fyi: relate PS: https://review.openstack.org/#/c/364336/ 15:42:35 for the gate bust issue. 15:44:32 Merged openstack/kolla: add iproute package in base image. https://review.openstack.org/364317 15:44:39 Merged openstack/kolla-kubernetes: Extra space in generate-passwords.py https://review.openstack.org/364346 15:52:59 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 15:53:46 Hello - is there a way to have Kolla set up Ceph w/o dedicating an entire disk to it? like to have Ceph use an LVM volume for example? 15:56:18 sbezverk: I can do that, but requeres a bunch of changes... 15:56:25 log has to be mounted to the init container. 15:56:37 the nova subdir doesn't exist so I've got to create it. 15:57:03 kfox1111: I think it would be good for troublehooting, I was using it and it was very helpfull 15:57:08 probably not too bad though... 15:57:11 k. 15:57:21 what about nova-init for the subdir though? 15:57:36 if I do nova, it will run into a permmission issue later. 15:57:45 unless I make another init container to fix the permissin. 15:57:48 permission 15:57:49 kfox1111: does not matter where as long as in a pre-defined location :-) 15:57:52 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 15:58:02 say /var/log/kolla/nova-init/ip.log ? 15:58:30 kfox1111: super, we could then replicate it everywhere 15:58:38 k. 16:00:49 Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: test the neutron db upgrade fix https://review.openstack.org/364419 16:01:18 hello guys 16:02:21 ping caoyuan 16:03:18 pbourke: fyi, review https://review.openstack.org/#/c/363968/ fixes bug https://bugs.launchpad.net/kolla/+bug/1616155 which is ranked critical. 16:03:18 Launchpad bug 1616155 in kolla mitaka "Unable to import database dump with ansible module in kolla-toolbox due to missing mysql binaries." [Critical,In progress] - Assigned to Stephan Michaud (michauds90) 16:03:51 michauds: Im not sure I agree that is a critical bug though 16:04:21 pbourke ohhhh alright! 16:04:54 why it's toolbox issue tho? 16:05:41 michauds: im assuming you have some internal ansible play that can restore a database dump? 16:05:45 also, http://docs.ansible.com/ansible/mysql_db_module.html 16:05:56 this seems to suggest that ansible can indeed do this 16:06:07 inc0: im reading it as the ansible module needs those binaries available 16:06:29 ok, interesting, we use these modules all the way 16:06:43 maybe this part is buggy 16:07:00 Jeffrey Zhang proposed openstack/kolla: Use the standard start method for kolla-toolbox container https://review.openstack.org/364344 16:07:45 Kevin Fox proposed openstack/kolla-kubernetes: Fail nova-compute init container always https://review.openstack.org/364387 16:08:07 pbourke, yes I can post an example rule that fails 16:08:51 michauds: sure that would be helpful 16:09:31 pbourke Alright, I'll attach it to the bug on launchpad. 16:10:39 pbourke, i replied https://review.openstack.org/364333 16:12:02 sooo kfox1111 sbezverk kubectl get pods return nothniing 16:12:13 how to debug this 16:12:14 ? 16:13:17 hey - re the bandit gate 16:13:26 now tht it is operational 16:13:29 i was thinking of making it voting 16:13:36 (and we are past the milestone 3 bump) 16:13:41 sbezverk, you may need check this https://review.openstack.org/362027 16:13:43 any objections? 16:13:50 sdake, no 16:14:16 no objections or no dont do that :) 16:14:28 pbourke: I added the problematic playbook rule to https://bugs.launchpad.net/kolla/+bug/1616155 16:14:28 Launchpad bug 1616155 in kolla mitaka "Unable to import database dump with ansible module in kolla-toolbox due to missing mysql binaries." [Critical,In progress] - Assigned to Stephan Michaud (michauds90) 16:14:28 no objections :) 16:14:43 sdake, ^ 16:15:30 michauds fwiw wee dont expect operators to interact with kolla via docker exec 16:15:41 michauds: monasca is not in kolla though? 16:15:44 michauds but i dont see harm in adding a package to enable your effforts 16:15:53 pbourke: not yet :) 16:16:08 I dont see us backporting it 16:16:26 right we wont backport monasca 16:16:35 pbourke i thught it was more a matter of people wantin to dump their mysql db 16:16:41 for backup purposes 16:17:07 sdake: even if so, my understanding of stable is we dont change it unless we have to 16:17:10 Merged openstack/kolla: Use dumb-init to manage the pid 1 process https://review.openstack.org/364333 16:17:18 zhubingbing proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 16:17:28 pbourke rogere - good understanding 16:17:41 maybe other projects manage their stable differently, if so we can chat about changing this policy 16:18:05 pbourke - we sosrt of make our own ruels up here 16:18:22 as long as they fit in with the stable:follows-policy framework 16:18:23 pbourke no, monasca would be new stuff 16:18:33 Merged openstack/kolla: fix sahara dockerfile https://review.openstack.org/351320 16:18:56 pbourke i guess i fundamentally didn't understand the monasca link 16:19:07 pbourke again i thought it was more about db backups 16:19:20 which i think we do want to enable 16:19:26 especialy since its a two liner 16:19:40 people can backup without kolla-toolbox in stable 16:19:56 how? 16:20:07 (just woke up - your my googel:) 16:20:27 had rough night of drinking too much to top it off 16:20:34 hmm 16:20:41 maybe i was too quick with that last statement :) 16:20:54 but i have a feeling there's a way. this feels like a feature to me 16:21:00 i could see making a copy of the docker volume 16:21:33 well pbourke make a call 16:21:48 I'm sticking with my -1 on this 16:22:01 wfm - then michauds - i'm sorry for wasting your time on this :( 16:22:21 michauds often we have to make quick judgements on backport bugs and sometimes we are imperfect 16:22:39 michauds in this case, I misunderstood the situation - so its my blame 16:22:41 sdake, no problem. It was a good learning experience :) 16:22:50 pbourke can you fix up the bug id 16:22:57 pbourke to not target it to backports 16:23:09 and leave a note in the backport that it was determined not necessary at this time 16:23:33 michauds: thanks for raising it 16:23:35 we may need to revisit that in the future if we have a storming horde of operators angry 16:23:44 pbourke it has gone into mster - which is fine from my pov 16:23:48 we may revisit, im known to be easily swayed :p 16:25:18 sdake: i may have missed something are we holding on +W ? 16:25:27 i think we are not super good at handling backports yet 16:25:38 coolsvap relating to which? features? 16:25:54 changes in master 16:25:56 coolsvap or this pbourke conversation? 16:26:04 bugs -> master 16:26:11 blueprints in rc1 -> master 16:26:19 bugs in master 16:26:39 alright got it 16:27:12 blueprints not in rc1 -> -2 with the note "Procedural -2 to block merge until rc1 is tagged on September 15th, please keep working on this work. The -2 here does not indicate the work is not vaued." 16:27:36 any bug is free reign in master 16:27:44 even if its a little featury 16:27:50 but not alot feataury 16:27:57 Merged openstack/kolla: Remove unnecessary host at haproxy in site.yml https://review.openstack.org/355861 16:28:49 coolsvap ify ou could make a summary out of this discussion and mail to ml it owuld be appreciated 16:29:48 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 16:29:48 sdake: sure 16:30:02 coolsvap thx :) 16:30:13 zhubingbing where are you with barbican? 16:30:13 pbourke, inc0 coolsvap sdake please review this. a critical one https://review.openstack.org/357898 16:30:22 Jeffrey4l on it 16:30:27 thanks. 16:30:34 keystone 16:30:49 keystone +barbican 16:31:16 i plan in magnum use barbican 16:31:42 Jeffrey4l you get rid of the nova volume? 16:32:02 sdake, no the nova_compuate volume will be created automatically. 16:32:03 zhubingbing sweet I was looking for someone to tidy up magnum to use x509 certs or barbican conditionally 16:32:26 Jeffrey4l but it doesn't get cretaed in a persisstent way? 16:32:35 where is it created 16:32:41 * sdake just woke up - still warming up ;) 16:32:49 and seriously hung over :( 16:32:57 when you run `docker run -v nova_compute:/var/lib/nova ...` 16:33:01 snake，sure i can help you ？ 16:33:15 no need to create manually. 16:33:38 when staring/creating the container, it will be created automatically. 16:33:49 other containers use this pattern. 16:34:21 we create the volume explicit when we need in some case. 16:34:58 cool didn't know that Jeffrey4l 16:35:10 :) 16:35:17 Jeffrey4l so jut to clarify - your 100% ceertain after this patch merges we will still have a nova data volume? 16:35:25 i also didnt know that 16:35:27 yep 16:35:33 :o 16:35:38 cool 16:35:47 totally not what I would have expected 16:36:06 Jeffrey4l so docker volume ls will show the nova volume then? 16:36:09 check the glance volume, we never create it explicit. 16:36:15 sdake, yep. 16:36:59 Mathias proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 16:37:00 Jeffrey4l - again goes against my thinking of how this stuff should work - so if its bust during the rcs, we may have to be open to a revert here 16:37:09 pbourke, sdake when removing container, the volume will not be removed, unless you add the -v paramater when delete. like `docker rm -v nova_compute` 16:37:31 Jeffrey4l yes but what about first time startup? 16:38:13 Jeffrey4l e.g. do a cleanup-host 16:38:15 and then deploy 16:38:19 is nova volume still there? 16:38:44 sdake, when you need it for the first time. for the nova. when start the nova-compute container, it will create the automatically. 16:39:17 what triggers that? 16:39:23 sdake, yes. check the glance role. we never create the glance volume explicit. But the glance volume exist all the time. 16:39:26 sdake, docker-engine. 16:39:44 Jeffrey4l got it - so maybe its new feture of docker? 16:39:50 Jeffrey4l can you run docker --version ? 16:40:08 our min requirement is 1.10.0 and I'd like to keep it that way if possible 16:40:22 sdake, hmm this should exist for a long time. i am sure this work at least since docker 1.9 16:40:34 ok good enough for me 16:41:40 Jeffrey4l reading through bug log i see what problem yu want to solve 16:41:56 Jeffrey4l if this solves it, then sweet :) 16:42:03 i think we have some other areas to tackle with this as well 16:42:12 ya. remove the creating named volume is side effect :) 16:42:21 sdake, means? 16:42:26 more test? 16:42:35 Jeffrey4l maybe this is not the only scenario where this problem exists 16:42:58 (as described in the bug log) 16:43:06 clearly we don't want the volume only created once if its missing 16:43:19 btw, anyone care to explain to me how runonce owrks? 16:43:28 (under the covers) 16:43:56 Jeffrey4l also I +w your patch 16:44:49 sdake, hmm it is the /var/lib/nova/instances folder we need create all the time, not the named volume. :) 16:45:24 right - i get that we only want it to run once 16:45:38 what I dont understand bout runonce is how ansible knows to only run the bootstrap task one time 16:45:56 since ansible keeps no state 16:45:57 Merged openstack/kolla: Create the nova instances directory when need during starting https://review.openstack.org/357898 16:46:09 i asked sam once and he couldn't gie me a good explination 16:47:56 https://github.com/openstack/kolla/blob/master/ansible/roles/nova/tasks/bootstrap.yml#L78 here how ansible control the bootstrap just run once all the time. 16:48:04 sdake, ^ 16:48:56 Jeffrey4l how does this task know only to run only one time: https://github.com/openstack/kolla/blob/master/ansible/roles/nova/tasks/bootstrap.yml#L75 16:49:01 or does runnce mean run only on one host? 16:49:12 sdake: that. run on one host 16:49:37 and the changed when: sorts out when to run bootstrpaping 16:49:56 mark-casey cool think i've finaly got thatone now :) thanks 16:49:57 been wondering that for years :) 16:49:58 sdake, the runonce you mean is `run_once` key word in ansible or others? 16:50:14 Jeffrey4l yup - i understand it now 16:50:15 thansk 16:50:23 cool 16:51:10 sdake: the changed_when only controls when Ansible reports "ok:" or "changed:" for a task, assuming it wasn't "skipped:" 16:51:28 mark-casey yup i understan hw that part works 16:51:32 its come together 16:51:34 thnks ;) 16:51:48 * sdake STILL learning ansible 16:52:26 btw folks, milestone 3 had soething like 470 commits in it 16:52:38 tremendous output! 16:52:56 last cycle we had something like 1k commits in total 16:52:59 I think we all are :), but yea was just clarifying that part didn't prevent repeated runs of the task - it's only an output thing 16:53:00 this cycle we hve something like 800 commits thus far 16:53:04 and a few hundred more to go i think 16:53:07 wow 16:56:41 my sons' birthday today - 12 years old 16:56:45 * sdake getting old 16:57:27 give it another 6 years and see how you feel :) 16:59:49 Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: test the neutron db upgrade fix https://review.openstack.org/364419 17:01:15 can anyone give me some guidance on setting up the networking? im getting an error about no host resources, and google is telling me its likely network related, which, makes sense since thats a hot mess for me :) 17:01:15 my son started school today. 17:01:23 * Jeffrey4l getting old too. :( 17:01:29 oh damn, summer is over 17:03:37 Jeffrey Zhang proposed openstack/kolla: Create the nova instances directory when need during starting https://review.openstack.org/364466 17:05:52 wow sdake! My son turns 11 tomorrow 17:07:27 britthouser i think you are either younger then you look or got started earlier then I did :) 17:07:56 He was born when I was 25 17:09:21 sdake: sent 17:09:39 ya i started around 30 17:09:50 britthouser so I think its a case of both :) 17:09:57 gn guys 17:10:13 my first child was born on a arm shot birth control, and second one on an IUD or whtever it is called 17:10:25 whhhhaaaat 17:11:11 ya - i've got strong swimmers I guess ;) 17:11:12 Sounds like they were meant to be sdake. =) 17:11:26 lol or she had some plans she didnt let you in on 17:11:27 i always wanted kids but was never ready 17:11:41 fortunately it was thurst upon me, or I may never have had em :) 17:11:52 no one is ever ready :) 17:12:06 jrich523 I find children fairly esy to deal with :) 17:12:31 usually it isnt the 'dealing with' thats the problem, typically more time/money related 17:13:29 jrich523 ya - have dual income high tech worker household 17:13:36 so swimming in cash 17:13:44 and time - well our kids take care of themselves :) 17:13:51 lol 17:14:11 whats cool is I work from home, which means I got to spend alot o time with them hen they were growign into young adults 17:14:27 https://review.openstack.org/#/c/352515/ stake can you help review barbican role and barbican docker file 17:14:37 yeah working from home is nice :) 17:14:59 zhubingbing_ after meeting (1 hour) you got it 17:15:06 Jeffrey4l or pbourke can you also review that 17:15:08 ok 17:15:11 or both of you actually :) 17:15:15 need barbican for magnum 17:15:24 np 17:15:31 zhubingbing_ can you also add the x509 cert db method for magnum 17:15:36 madhuri implmeneted it 17:15:40 yes 17:15:43 ok 17:15:46 she can guide you - tell her I sent you her way 17:15:56 by ping her in #openstack-containers not privatey 17:16:13 zhubingbing_ as a separate patch 17:18:02 madhuri what mean ? 17:19:59 madhuri is a people, she can guide me complete to add the x509 cert db method for magnum 17:20:39 and i can in #openstack-contariners contact she？ 17:20:48 zhubingbing_, i think so 17:21:01 ok 17:22:27 Eduardo Gonzalez proposed openstack/kolla: Add Senlin Ansible role https://review.openstack.org/339725 17:25:33 egonzalez90 say - wil lsenlen land before sept 15th? 17:25:46 I'm not sure why that was descoped in our last planning meeting 17:25:57 but seems reasonable to introduce into rc1 if its feasible 17:26:39 if gets reviewed can be before 15th 17:26:41 egonzalez90 also - it isn't necessary to sign off on blueprints 17:26:56 with git that is 17:27:00 rather reviews. 17:27:04 * sdake going backto bed ;) 17:30:39 wish i could go back to bed 17:35:54 fyi: the neutron bug is fixed. the gate should be all green now. 17:38:11 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 17:39:53 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 17:45:37 Jeffrey4l that was fast 17:45:41 jrich523 - ya I was just joking 17:45:52 Eduardo Gonzalez proposed openstack/kolla: Add Senlin Ansible role https://review.openstack.org/339725 17:45:58 yaaaaaaaay, they just detected Ebola in Spain 17:45:59 ya 17:46:02 happy summit folks 17:46:09 inc0 ya i got that same warning rom csco internal 17:46:14 bring your DEET 17:46:26 tick bite = deadly 17:46:31 if you see someone convulsing and spreading blood all around 17:46:33 don't approach 17:46:36 inc0, really? 17:46:40 egonzalez90 serious 17:46:59 2 people hae disease, 1 dead I think now, and 200 people in quaranteen atm 17:47:26 first case of ebola in europe - the notice said 17:48:17 well you do work from home, so... its possible :) 17:48:56 im not sure how ticks and ebola are related lol 17:49:03 jrich523 ticks carry the virus 17:49:18 they carry a lot of things 17:49:31 Two cases of Crimean-Congo haemorrhagic fever (CCHF) have been reported from Spain. A 62-year-old man is thought to have been infected via tick bite during a walk in the Castilla-Leon region. A nurse who treated him prior to his death has also tested positive for the disease. She is currently in a stable condition in an isolation unit in Madrid. Authorities are monitoring about 200 people who had contact with either of the individuals. It is likely 17:49:31 this is the first locally acquired case reported in Western Europe. CCHF is a potentially fatal viral illness transmitted by tick bites or through direct contact with infected body fluids. Symptoms include flu-like illness, abdominal pain and bleeding often occurs. There is no vaccine. Protect yourself against tick bites to avoid the illness. 17:49:42 i dont think that is ebola 17:49:44 but it doens't sound good 17:50:13 lyme will do similar 17:50:17 however it doesnt kill you in days 17:50:18 spanish papers says are not ebola, is a similar bloody fever disease 17:50:26 ya - we have alot of lyme disease in arizona 17:50:37 i have it :-/ 17:50:38 egonzalez90 ya - see above 17:51:03 they suspect 75-80% of the population has lyme 17:51:06 jrich523 don't know much about it othe then it can be treated within 48 hours if yu have a bullseye after being bitten 17:51:17 yeah, thats basically wrong 17:51:20 interesting 17:51:34 jrich523 well I am not a MD/DO/etc :) 17:51:44 it requires a lot of antibiotics, like 6 months worth 17:51:57 which they wont give you unless you are certain you have it, which means a test, which takes a month to come back 17:52:26 at any rate, there is no killing it, just subduing it.. 17:52:28 can get antibiotics in mexico and come back across the border ;) 17:52:41 lol honestly, if you think you had it, i'd advise that 17:53:12 i've seen pictures of what the tick bite looks like 17:53:19 it looks like a big bullseye 17:53:41 the bullseye is rare, and if you see it, itsl ike 99% you have lyme... if you dont see it, its like 75% chance you have lyme lol 17:53:51 so, rash or not, you're likely screwed 17:54:18 which is why its believed such a high portion of the population has it 17:54:50 since symptoms are really across the spectrum, its hard to identify 17:55:22 they believe the vast majority of hypochondriac cases are actually lyme 17:56:20 all the ticks in arizona carry lime dissease 17:56:29 or most of em anyway 17:56:46 thats the other common misconception, only deer ticks carry it 17:57:06 all ticks do, hell even mosquitoes do 17:57:36 we're all gonna die! 17:57:40 wait, i mean, we're all dying! 18:00:56 my kids think immortality is on the horizon 18:01:10 it is, sorta 18:01:11 I think its within human's grasp to live much longer then we do 18:01:15 we can keep bodies going 18:01:21 thats "easy" really 18:01:27 its keeping the brain functioning that we cant do 18:01:39 my brain works better now then it ever has 18:01:49 kidney, liver, heart, all replaceable, either with mechnical things or cell printed versions 18:01:59 well wait till you're 80 18:02:04 and then we can talk about your brain lol 18:02:07 i hope I dont live to 80 tbh :) 18:02:27 i want to live to a ripe old age of 65-70 and then die in my sleep ;) 18:02:29 lol you will 18:02:39 for our age (you're about 40?) 18:02:40 my poor mother - she is in bad shape - broke hip and back 18:02:43 80 will be young 18:02:45 42 18:03:05 im fairly certain 100 will be our common dying age 18:03:05 i dont want to live without mobility when i age 18:03:18 my grandmother lived to 101 18:03:30 meh, self driving cars, exo skeletons, im sure you'll be fine :) 18:03:50 jrich523 frightening :) 18:03:55 besides, you're rolling in cash, you can buy all thse fancy things to keep you moving 18:04:01 "what can go wrong" ;) 18:04:05 hahaa 18:04:08 nothing, nothing at all :) 18:04:11 jrich523 would could possibly go wrong ;) 18:04:19 "imagine everything on fire...." :) 18:04:25 I feel like I've taken enough punishment for two lifetimes :) 18:04:46 and unfortunaely its just beginning 18:04:58 well if you plan to live to 80, its half way over :) 18:05:05 you're almost there! 18:05:50 coolsvap nice email dude :) 18:06:26 ya its the second half thta sucks 18:26:56 can anyone provide some help with setting up networking? i think its the last part to getting this fully working 18:30:32 as my wife says, cisco didn't build their empire because networking is easy ;) 18:30:53 lol 18:30:56 jrich523 fwiw netorking works perfectly in msater as well as stable branches with either ovs or linuxbridge 18:31:03 im fairly certain they made it hard on purpose 18:31:04 she's not wrong. :) 18:31:11 jrich523 unfortunately I am not a networking expert - so idon't know exactly how to set up your environment 18:31:23 jrich523: that, sometimes is true too I think. :) 18:31:40 jrich523 sbezverk may be able to help 18:31:43 well really the problem is that there are no hosts with available resources 18:31:55 jrich523 another person is jeffrey4l - he undrestands neutron inside and out 18:31:56 which in reading most people get that because of network issues 18:32:08 run nova hypervisor-list 18:32:35 it shows my one host, up/enabled 18:33:02 i think what i need to do now can all be done in horizon, but, im doing something wrong i think 18:33:08 yeah. if the vm takes a while to launch and finally gives up after 3 attempts, thats usually a network issue. 18:33:30 i think it was 5 attempts, but yeah... a really horrible msg that provides no details 18:33:38 even poked around the nova logs, nothing all that useful in there 18:34:10 that, is something openstack really suffers from. lots of really unhelpful errors. :/ 18:34:41 right now i have no networks defined, the host is a VM with two network adapters, eth0 is the admin, has an IP assigned to it 192.168.11.20 (vip 30) 18:34:58 there is no IP assigned to eth1 18:36:49 both adapters are sitting on the same network (bridge to my local network, 192.168.11) 18:37:20 ping inc0 18:37:38 so i assume if i create a 'flat' network, that should work out ok? 18:38:26 jrich523 did you take a look at init-runonce script? 18:38:33 jrich523 it needs to be customized foryour environment 18:38:41 i'll take a look now 18:38:50 jrich523 it sets up all the networks as necessary 18:38:58 also make sure nova hypervisor-list shows your hypervisors 18:39:06 it does 18:39:08 and nova hypervisor-show shows its up and running 18:39:35 the reason it is not working is probaby because you have not set up the neutron networks 18:39:41 init-runonce does this 18:39:46 but it does it for *my* personal environment 18:39:52 perfect, i'll dig in to that 18:39:55 you will have to customize or your environment 18:40:01 or/for 18:40:04 the nova cmds are looking good 18:40:27 thats good news 18:40:33 that means you have a functional deployment 18:40:37 just user error at this point 18:40:43 yeah im getting close 18:42:19 ok so one thing i noticed before, is that a lot of things are referencing br100 18:42:45 which if i do ip a, i see i have a br-ex, br-int and br-tun 18:46:46 yaay neutron is broken 18:47:07 br100 isnt that a linux bridge thing? 18:47:15 no, migration issue 18:47:22 dbsync is not working 18:47:37 sean-k-mooneyAFK: from what i've read, yes it is 18:47:56 all the docs i've come across assume a bunch of things... like.. mostly importantly, that i know linux :) 18:48:03 i know that br100 is one of the default bridges created when using linux bridge in devstack 18:48:51 jrich523: youll use have to start working on the windows install of kolla 18:49:27 i'd certainly be more comfortable :) 18:49:39 with windows nano server you actully might be able to pull it off at somepoint 18:49:51 however if its docker, it still uses nix under the covers, so i suspect i'll have similar issues 18:50:09 no there is a native windows runtime for docker 18:50:26 its part of windows server 2016 18:50:29 right, but, when you create containers on windows, that host apps for linux 18:50:38 it basically builds a linux kernel to run them 18:50:53 so... its sort of garbage 18:51:05 yes but if you were doing a windows version of kolla you would not be using linux at all 18:51:06 i mean, its cool, but if performance matters, not so much 18:52:03 unfortuetly most of openstack does not run of windows currently so you could only have compute nodes 18:52:34 right 18:53:35 so the only windows i could run, would be hyperv instead of kvm 18:53:39 which, i guess thats cool 18:53:40 but meh 18:54:16 in that init script, i see it mentions physnet1 18:54:20 where is that? 18:54:22 or what is that 18:54:41 physnet1 one is the default provider network in neutron 18:55:09 hmm so when i went to do this via the gui (horizon) it populated that as "default" 18:55:20 is it smart enough to pull a useful name, or is default just... default 18:55:32 basically i think kolla maps physnet1 to the br-ex 18:55:48 no horizon is not that smart 18:55:51 can i "look up" physnet1? 18:56:01 how do i see if its there? 18:56:24 you can check the neutron conf 18:56:55 well technically the ml2_conf.ini 18:57:32 i really like how i can use powershell on linux now, at least something im use to :) 18:58:02 i like windows but always hated powershell 18:58:13 whhhaaaatttt?!?! 18:58:17 how can you dislike objects?! 18:58:31 i dont but if im going to use them ill use c# 18:58:42 well powershell is real time C# 18:58:59 sean-k-mooneyAFK do yo uhave remaining patches for bifrost needed? 18:59:09 or the bootstrap playbook 18:59:37 sdake: i dont think soo at least not for this cycle. ill be addressing the techdebt bugs in the next few weeks but they will be ocata i guess 19:00:09 http://paste.openstack.org/show/565859/ 19:00:23 so yeah, looks like physnet1 is mapped to br-ex 19:00:23 sean-k-mooneyAFK you have free reign to do whatever needs to be done to make bifrost and the bootstrap playbooks work correctly all the way up to 3.0.0 19:00:30 that is about 3 days prior to summit 19:00:55 sean-k-mooneyAFK that includes any tech debt or redesign that is needed 19:01:08 sean-k-mooneyAFK i want to maek sure first and foremost the api is solid (the api being kolla-ansible's commands) 19:01:35 sdake: oh ok i taught it all had to completed for n3 e.g. today 19:02:53 sdake: in this init file, is 10.0.2 your external network? 19:03:09 which for mine would be 192.168.11 19:04:09 sdake: i should be starting on the tech debt items next week. i dont think the interface will change much if at all between now and 3.0.0 19:05:43 jrich523 10.0.2.z is my extrenal network 19:05:49 10.0.0.0 is my tenat network 19:05:53 k, thanks 19:05:57 192.168.1.z is my management network 19:06:26 Hi 19:06:47 sdake the only thing that i was considering currently was should the image build for the baremetal image be a seperate step from the deployment of the bifrost container 19:06:59 I am facing a issue where the cinder is not able to connect to ceph cluster in kolla 19:07:02 sean-k-mooneyAFK cool well thats sort that out 19:07:08 september 15th is deadline for rc1 19:07:12 can anyone help? 19:07:13 after rc1, bug fixess only 19:07:40 sean-k-mooneyAFK we moved some blueprints into rc1 via FFE 19:08:03 bah, still no luck 19:09:33 sdake: ok cool. the only part of the original bifrost blueprint that was not implmented in whats merged was bring your own image. 19:10:15 sdake: currently bifrost build the image it deploys as part of the bootsrap of the bifrost container and you can customise that 19:11:06 2016-09-01 19:02:58.428703 7fa718ff9700 0 -- 10.241.70.95:0/1000487 >> 10.242.193.230:6789/0 pipe(0x7fa710008280 sd=3 :48845 s=1 pgs=0 cs=0 l=1 c=0x7fa7100055d0).connect protocol feature mismatch, my 83ffffffffffff < peer 483ffffefdfbfff missing 400000000000000 19:11:25 when i do a ceph -s from cinder-volume kolla 19:12:52 Satya_: i have not deployed ceph recently but if you go to one of the ceph contianer and do a ceph status is the ceph cluster running correctly 19:15:24 my ceph is external 19:15:57 but i can run ceph -s on the ceph cluster and thats running fine 19:17:54 Satya_: my guess is that there is a version missmatch between the version of ceph you have deployed externaly and what kolla/cinder expects 19:19:45 hmm so according to docs, i should have a /etc/sysconfig/network-scripts/ifcfg-br-ex, which i do not 19:20:49 i installed jewel 19:21:33 and on the cinder volume ceph-common-0.94.5-1.el7.x86_64 19:23:07 how come no one told me its 12:30 :O 19:23:45 hmm so i think i need to create the ipcfg-br-ex which binds it to eth1... maybe... 19:24:00 cuz physnet1 -> br-ex which currently goes no where 19:30:19 Satya_: it looks like kolla expects the hammer release https://github.com/openstack/kolla/blob/master/docker/base/sources.list.ubuntu#L20 19:30:48 .. that broke everything 19:31:11 jrich523: no you should not need too 19:31:21 jrich523: are you using ovs 19:31:31 i dont have a clue 19:31:34 can we upgrade the release to jewel? 19:31:49 as my ceph cluster is running on jewel 19:31:51 dose cinder support jewl 19:31:59 'yes 19:32:11 i tried with kilo release 19:32:22 it was working like charm 19:32:26 kilo release of kolla or cinder 19:32:41 both 19:33:00 oh well in that case im not sure why its not working now 19:33:30 jrich523: i belive ovs is the default in kolla 19:33:31 :( 19:33:44 i think i've seen mention of it 19:33:59 so its probably that 19:34:11 jrich523: if you do a docker ps you should see an openvswitch container 19:34:45 yup, a _vswitched and _db 19:35:31 jrich523: if you docker exec into th _db container and run ovs-vsctl show it will show you the bridge confiugration 19:36:02 jrich523: you should see whatever interface was set as neutorn_external_interface is added to br-ex 19:37:04 yup, 19:37:10 that is mapped correctly 19:37:37 hmm 19:37:56 external ceph may be incompatible unless its the same version (hammer) 19:38:12 Satya_: i dont know if the ceph driver for cinder use any ceph specifc packages form the ceph repo but you could try chaning https://github.com/openstack/kolla/blob/master/docker/base/sources.list.ubuntu#L20 to jewle and rebuild the cinder container and see if that resolve the issue but unfortnetly my ceph knolage is limited 19:38:48 Satya_ - the issue is the protocols change 19:38:55 and they re not ersioned 19:38:57 so ceph hammer and ceph jewel are incompatible 19:39:08 may I suggest installing hammer - or use built-in ceph 19:39:09 ohh ok 19:39:11 got it 19:39:47 sdake: is there a reason we have not bumped the ceph version in kolla to Jewel yet? 19:39:51 sdake is there any way we can change the cinder volume container to have jewel package? 19:40:09 im going to grab lunch, i'll bbl, thanks 19:40:33 Satya_: you can specify your own source.list to use when building the containers 19:40:49 ok 19:40:52 will try that 19:41:43 Satya_ yes it can be done - but not easily 19:41:56 sean-k-mooneyAFK no good reason other then hasn't been planned and capacity constraints 19:42:16 sean-k-mooneyAFK and it appears an upgrade might be incompatible from Satya_ 's report 19:42:37 although I htin ceph upgrade is a good goal to have and may be worth looking at for rc1 19:42:55 i wonder if we would have to upgrade the ceph contaienr between intermediat version of ceph 19:42:56 sean-k-mooneyAFK but we have alot on our plates bug wise 19:43:31 i also dont know if ceph has sorted out an upgrade technique 19:43:59 anyone using the kube-dashboard? 19:44:03 sdake: well i would have hoped they would have at least n-1 upgrade supprot? 19:44:25 sean-k-mooneyAFK no idea - but worth lookign into 19:45:23 should i raise a bug again it 19:45:27 against 19:45:32 and another issue 19:45:37 nova resize fails 19:45:53 as nova is not added to the sudoers list 19:46:04 adding nova to sudoers list works fine 19:46:39 this is mainly because while resize vm it runs privsep-helper code as sudo 19:46:44 Satya_: what conatieer didn you added nova to the sudoer list? 19:46:58 nova_compute 19:47:38 Satya_ we have that one in the queue i think 19:47:41 thinking of fixing it 19:47:52 after raising a bug :) 19:48:03 Satya_: the issue is https://github.com/openstack/kolla/blob/master/docker/nova/nova-base/nova_sudoers#L1 19:48:04 Satya_ it may have already merged - moemnt 19:49:01 Satya_: you would need to also allow sudo to the privsep helper 19:49:40 yes 19:50:02 else resize will fail 19:54:34 sdake looking at the reviews i dont see one for the suduers issue 19:55:09 sdake: i also dont see a bug for it but i could just be blind 19:55:37 :) 20:00:20 anyone know if this is currently being worked on. https://bugs.launchpad.net/kolla/+bug/1545389 20:00:20 Launchpad bug 1545389 in kolla "Multiple drivers support for nova_compute container" [High,In progress] - Assigned to Lingfeng Xiong (xionglingfeng) 20:00:48 it would be nice if we could have ironic and kvm in the same could 20:01:01 /could/cloud 20:35:24 sdake: are you there 20:35:33 aNupoisc yes but busy atm 20:35:40 aNupoisc you sorted out your problems didn't you? 20:35:55 aNupoisc if not give me 20-30 mins 20:36:24 sean-k-mooneyAFK totally agree :) 20:36:45 sdake: no i tried couple of things but that did not resolved the problem 20:36:59 sdake: okay hover me when you get free 21:25:31 aNupoisc kids birthday and i'm jammed up on time 21:25:42 aNupoisc i may have to get back to you later tonight or tomorrow 21:26:15 sdake: oh wish him from me. Ahh okay 21:26:26 sdake: do we create an ansible user? 21:26:32 i do not 21:26:40 i run sudo kolla-ansible deploy 21:26:49 sdake: because docker exec command is asking me to enter password for ansible 21:27:01 and that's where it is sitting forever 21:27:11 yup i recall seeing error 21:27:18 sdake: ahh then what do you call? 21:27:35 what do you mean what od i call 21:28:08 sdake: i mean how do you deploy then 21:28:38 sdake: do you mean you deploy ceph using some different technique? 21:45:12 aNupoisc sametechnique 21:45:30 but i have my public key in authorized keys for both my normal user and root for an aio deploy 21:47:13 hey sean-k-mooneyAFK, you around? 21:53:47 sdake: oh okay seems i have that in mine too. But seems ssh is not giving a problem. 21:54:09 sdake: we can find out that. I am bothering you from your son's bday celebration 21:54:41 sdake: we can have webex session later tonight if possible 21:55:00 mmmm cake 21:58:20 jrich523: cake yeah.. 21:58:27 aNupoisc my schedule jut oepned up - parents cancelled on me 21:58:30 i hope its chocolate cake 21:58:45 no cake? :( 21:58:48 jrich523: I love chocolate 21:59:04 as do i 21:59:05 sdake: :( 21:59:16 aNupoisc so in 1 hour will yoube around? 21:59:20 he gets home from school 21:59:26 then he will want to open his presents 21:59:27 sdake: yes of course 21:59:45 how about this 21:59:47 lets start now 21:59:55 and when I hvet o bounce I have to bounce 22:00:00 and can be bak after i get done with setup of his new display 22:00:18 his presents are not yet wrapped 22:00:27 sdake: yes of course 22:00:30 and his mother is the wrapper in the household 22:00:32 i will come on webex 22:00:42 or google talk? what do you prefer 22:00:43 http://cisco.webex.com/join/stdake 22:00:47 webex pls 22:00:54 okay 22:03:30 click connect to audio 22:03:34 the blue button on the left 22:03:58 sdake: i am calling you 22:05:39 hey steve, are you at all involved with metapod? 22:13:28 *Pavo posting sdake's webex on 1000 social sites* lol 22:14:35 lol 22:14:47 im fairly certain he'll get an email for everyone that joins it when he isnt there 22:14:55 and i feel like if he doesnt show up it boots you out 22:15:04 however im not sure, i do have one of my own tho, and always wondered about it 22:15:12 lol yeah was just messing around 22:16:55 my url isnt nearly as cool tho :( 22:17:57 fusionstorm.webex.com/meet/jrich2fusionstorm.com 22:18:01 seems a tad over kill 22:20:02 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 23:01:19 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 23:19:09 sdake: it worked!! 23:19:25 aNupoisc as i would expect 23:19:30 grats :) 23:20:29 sdake: thanks. Versions are indeed very important to look 00:03:20 sdake: seems wlan0 did not cause a problem 01:58:32 so.... this might be a newb question but whats the difference between kolla and kolla-kubernetes? 01:59:04 kolla-kubernete is openstack on Kubernetes 01:59:39 guess I need to readup on kubernetes then because I don't fully understand it 02:01:01 only thing I understand about kubernetes is that it is orchestration for docker containers 02:01:40 it's great for container management 02:05:23 so how many nodes would be the ideal number to use kolla-kubernetes over just plan kolla? 02:05:44 pavo kolla-kuberntes is not usable in the field for some time 02:05:55 pavo so that quesetion is nonsensical :) 02:05:58 oh ok 02:06:06 ya It's super early 02:06:56 ok but when it is to the point of working what would be that ideal number, something like datacenter size? 02:07:23 you need ceph a few compute and controllers 02:07:25 pavo kubernetes itself scales to 1k nodes 02:07:43 kolla-anible has been tested at 123 node count 02:07:47 ok cool think I am getting the gist of it 02:08:20 although I personally believe kola-ansible could scale indefinately to very large sizes if openstack itself were scalable 02:08:22 this whole docker, ansible, kolla, kubernetes is all kind of new to me so slowly learning them 02:08:32 but openstack has single points of service - so its not fully horizontally scalale 02:08:55 for example rabbitmq, an dmaraidb in particular - only one proce running 02:08:59 so the load can't be shared 02:09:00 but couldn't the openstack REST APIs be load balancible? 02:09:11 it has nothing to do with opentack proper 02:09:14 ah 02:09:17 it has to do with the backends opentack uses 02:09:37 but yes the rest apis are horizontally scalable for the most part in openstack 02:10:06 so you saying multiple rabbimq services can not work together 02:10:26 rabbitmq possibly 02:10:28 mariadb no 02:10:57 what is stopping mariadb from being able to load balance? 02:11:06 the design 02:11:10 ah 02:12:54 sdake, single queue rabbitmq on kubernetes 02:13:00 could be the solution 02:16:44 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 02:30:30 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 03:13:05 Jeffrey Zhang proposed openstack/kolla: Add nova-serialproxy container https://review.openstack.org/358822 03:28:07 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 03:36:43 ` 03:42:25 Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 04:04:50 Wei Cao proposed openstack/kolla: Add solum ansible role https://review.openstack.org/358938 04:11:24 zhubingbing proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 04:16:18 ping hiking 04:16:24 ping huikang 04:19:31 ping pbourke 04:19:33 hi 04:21:20 zhubingbing_: pbourke will not be available for atleast 2 hrs iirc 04:21:42 ok 04:21:45 sure 04:22:29 coolsvap have you free time ？ 04:22:50 can you help review ？ 04:22:57 zhubingbing_: i am running in a meeting, which review? 04:23:10 bibican 04:23:14 barbican role 04:23:33 i need some more time to review it, sometime in later afternoon i suppose 04:23:55 ok， thanks 04:24:25 https://review.openstack.org/#/c/351822/ 04:25:44 https://review.openstack.org/#/c/352515/ 04:52:00 Hui Kang proposed openstack/kolla: Add etcd ansible role https://review.openstack.org/364662 05:02:20 Jeffrey4l: https://blueprints.launchpad.net/kolla/+spec/nova-proxies is not targeted for rc1 05:26:59 morning 05:31:29 zhubingbing proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 05:33:21 hi, why gate have POST-FAILD 05:36:02 zhubingbing_: ^^ 05:37:01 ^^ 06:19:23 Mathias proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 06:24:47 coolsvap_, ok. let merge this in next cycle. 06:41:03 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 06:49:34 pbourke, could u review this ? https://review.openstack.org/356449 06:51:44 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 07:50:10 ping Md Nadeem 07:58:19 Wei Cao proposed openstack/kolla: Add solum ansible role https://review.openstack.org/358938 08:06:55 Martin André proposed openstack/kolla: Fix oslo.i18n in kolla project https://review.openstack.org/355342 08:06:56 Martin André proposed openstack/kolla: Consistently set sys path when using local tools https://review.openstack.org/364735 08:48:00 Takashi Sogabe proposed openstack/kolla: Keepalived leaves VIP on the host in case of accidental removal. https://review.openstack.org/364085 09:17:51 Paul Bourke proposed openstack/kolla: Allow use of database for Horizon sessions https://review.openstack.org/359794 09:22:12 Merged openstack/kolla: Move to transport_url for rabbitmq configuration https://review.openstack.org/356449 09:22:53 Merged openstack/kolla: Create the nova instances directory when need during starting https://review.openstack.org/364466 09:31:49 Merged openstack/kolla: Use the standard start method for kolla-toolbox container https://review.openstack.org/364344 09:56:27 Christian Berendt proposed openstack/kolla: Remove dev/heat note from READE file https://review.openstack.org/364799 10:00:00 Christian Berendt proposed openstack/kolla: Add missing Docker images to README file https://review.openstack.org/364804 10:25:08 Merged openstack/kolla: Remove dev/heat note from READE file https://review.openstack.org/364799 11:50:49 ping Paul Bourke 12:35:12 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 12:44:52 morning guys :) 12:54:46 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 13:19:39 Jeffrey4l: hey, puppet CI also have https://bugs.launchpad.net/neutron/+bug/1619367 13:19:39 Launchpad bug 1619367 in neutron "neutron db upgrade raise error " [Undecided,Confirmed] 13:19:50 same thing http://logs.openstack.org/46/364746/2/check/gate-puppet-openstack-integration-3-scenario001-tempest-centos-7/c5136ac/console.html#_2016-09-02_12_47_20_864790 13:58:04 EmilienM, this is fixed yesterday. it is OK now. 13:58:13 Jeffrey4l: ack 14:03:18 Paul Bourke proposed openstack/kolla: Update destroy include images arg to LONGOPT https://review.openstack.org/364983 14:05:21 Jeffrey4l: our Ci still have the problem with latest neutron 14:07:21 EmilienM, seems it is. let me check this. 14:08:55 Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: test the neutron db upgrade fix https://review.openstack.org/364419 14:16:46 Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH https://review.openstack.org/326307 14:30:36 good morning 14:40:11 Merged openstack/kolla: Use a local docker image namespace in gate https://review.openstack.org/364310 15:12:44 good morning inc0! 15:13:19 hey Britt:) 15:18:55 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 15:21:18 yo guys 15:21:26 morning 15:21:50 anyone have any info on what happens if we enable the user namespace in docker for kolla 15:22:10 kfox1111: qq yesterday's fix for iproute, was done for toolbox to be able to rup "ip a" ?? 15:22:36 sbezverk, it's in base dockerfile 15:22:52 inc0: I think it went into openstack-base 15:23:03 and toolbox is based on cenots-base 15:23:39 last night after that fix merged I could not run ip a using toolbox until I added it into toolbox Docker file 15:28:58 sbezverk: yeah. 15:29:22 sbezverk: I think it was dropped somehow out of the toolbox. 15:29:33 I went to add it back in, and it was suggested to put it in base instead. 15:30:07 kfox1111: I think when somebody suggested iproute should be a part of base, you removed it from toolbox but added to opensatck-base 15:30:21 :-) yep 15:30:58 so a newer build should put it in base and it gets back into toolbox by that route. 15:31:33 kfox1111: comments not always do good ;-) 15:32:17 yeah. we need to get some gating up soon. so that breakages are cought earlier. :/ 15:32:25 I've started looking at it a bit. 15:34:52 kfox1111: that would be awesome the real gate.. 15:37:21 I think if nothing else doing a few basic checks like, does the template generate a valid kube yaml file would go a long way. 15:38:19 can do some basic sanity checks on the init containers too. replace the main container with a dummy that reads in the config files and does sanity checks too. 15:38:46 then we can work on some full blown checks. 15:40:09 I've got a basic sanity check script I've used to check if the json in the yaml is valid json too. that would help. 15:43:03 kfox1111: you think it would be possible to integrate this script into the gate? 15:43:57 yeah. 16:26:42 sbezverk: does this fix the unmount issue? https://github.com/kubernetes/kubernetes/pull/12502 16:31:01 kfox1111: do not think so, last message in that thread was august of last year, but I still have unmount issue 16:40:39 thats a very old thread. :/ 16:41:19 is it possible it only affects very old cephs like what kolla is providing? 16:43:45 Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 16:44:06 kfox1111: do not think so, I saw another thread talking about the same issue even with the latest ceph 16:46:31 kfox1111: check this out: https://github.com/openshift/origin/issues/7983#issuecomment-243760943 17:02:44 sdake: Hi, regarding the yesterday's issue I see "osdmap e10: 1 osds: 0 up, 0 in" on running ceph -s. So OSDs are not up 17:05:34 adnavare, did you label disks correctly? 17:06:23 inc0: yes they are labelled as KOLLA_CEPH_OSD_BOOTSTRAP 17:06:52 they're still labelled? did ansible failed in any way? 17:11:34 inc0: no ansible did not failed. 17:11:44 let me print the partitions 17:12:40 inc0: here is that http://pastebin.intel.com/view/05453c9a 17:12:48 inc0: I see journal created 17:14:19 inc0: I have attached a pen drive as a block device. Also the neutron_external_network = wlan0 which is down i am assuming that would not be a problem 17:19:53 hmm...pendrive as block device is kinda unorthodox;) 17:20:08 I have no idea how udev treats it...also...you have linux right? 17:21:51 Eduardo Gonzalez proposed openstack/kolla: Add Senlin Ansible role https://review.openstack.org/339725 17:27:19 Eduardo Gonzalez proposed openstack/kolla: Add Senlin Ansible role https://review.openstack.org/339725 17:28:39 inc0: I just need a basic setup not with much space so thought to use PD ;) Yes I have linux 17:32:11 I have no idea if udev works with pendrive tbh 17:38:07 Eduardo Gonzalez proposed openstack/kolla: Add nova-serialproxy container https://review.openstack.org/358822 18:24:02 sbezverk: yeah, I saw that one. That involves node failover. which I get is still a problem. 18:24:32 but you've talked about things not releasing when you delete a pod (job) which is a very different thing I think. 18:29:55 adnavare: last I looked wlan devices couldn't bridge. 18:30:27 well, unless you had a wlan device that could. they use to be special. not sure if the newer stuff does now. 18:49:44 inc0: It is mounting that. 18:51:38 kfox1111: but do you think wlan is linked to bringing OSDs up correctly? I mean it is assigned to neutron_external_network so will make instances not able to communicate with outer world but I am not getting is that be the reason of OSDs not to start 18:57:20 that, I'm not sure about. if its on the same host, I'd think it would work. 18:58:00 just giving you a heads up I've seen wlan0 not pass on multiple macs in the past. 19:04:32 kfox1111: Ahh okay 19:05:17 kfox1111: And the other thing what I noticed is that I just used 1OSD and thinking that might be the reason ceph -s is showing HEALTH_WARN 19:05:26 kfox1111: do you think so? 19:06:08 is there a good way to tear the ceph down, now that it is not correctly up I am trying to use more OSDs and try with that 19:13:05 yeah. ceph doesn't much like 1 osd. it should still work fine though. 19:13:20 I do a one node osd cluster on my laptop for testing. 19:14:38 adnavare: I haven't used kolla to deply ceph yet, so I don't know much about it yet. 19:15:57 kfox1111: i node osd thats cool 19:16:20 kfox1111: actually I am also trying to setup a simple setup 19:16:30 but giving heck ups 19:24:07 pbourke: i know its late but are you still around? 19:46:45 has anyone here done a gpt install of ubuntu or centos succesfully 20:12:03 that depressing moment when your realise that the centos installer support gpt installes but cant create the partiion layout your want at install time and the ubuntu installer can create the layout but then wont boot for the gpt partion... 20:13:26 i think i will have to go with centos or drop ceph external journals and cache layer but thats a problem for futrure me 20:14:41 sean-k-mooneyAFK: I've done gpt with centos7. 20:15:20 it depends on if you are efi or bios too. :/ 20:15:46 the servers support both 20:16:00 bios you gota do a weird hybrid fat/gpt partition. 20:16:12 efi's smarter about gpt. 20:16:28 switching to efi was a bit of a pain, but I've been pretty happy with it once I transitioned. 20:17:39 well i just spend 2 and a half days on and off trying to get ubuntu 16.04 installed so that i can deploy a kolla cluster and stil having grub issues so if centos works on monday i guess ill stick with that 20:18:53 the only reason im using gpt is because kolla's ceph support currently needs it. 20:18:59 yeah. centos7's installer seems to be pretty smart about gpt. 20:19:18 it autobuilt the hybrid partion once when it detected bios with a 4t root drive and it booted fine. 20:20:55 i currently have 2 disks a 2TB hdd for ceph osd and a 240GB ssd for os + journal + ceph cache + cache journal 20:21:38 ah. 2t shouldn't throw much fight with a bios then. 20:21:52 fat should work. 20:22:18 well the 2TB dis is for a ceph osd so that can be GPT 20:22:30 the 240GB ssd has the os 20:22:48 ah. ok. 20:23:06 but since it also has the ceph journals and ceph cache it also needs to be gpt because the kolla lables are too long for msdos 20:23:19 oh... 20:23:46 fat + lvm? then lvm volumes for the ceph stuff? 20:23:49 if the kolla lables were sorter then you could just use an msdos partition layout 20:24:23 wouldn't be quite as fast, but might be fast enough, and would be easier to manage. 20:24:40 can you have a seperate partition table for lvm volumes? 20:24:49 i did not think so 20:25:08 yeah, you can. 20:25:19 I have a ceph osd on a lvm volume. 20:25:27 running in k8s. :) 20:25:37 using the kolla container. :) 20:25:53 its pretty hacky at the moment though, so havent contributed it upstream yet. 20:25:58 but that was not using kolla ansible to deploy the ceph correct 20:26:17 right. I've got a k8s pod in /etc/kubernetes/manifest that starts it up. 20:26:35 and I manually ran the bootstrap stuff for it as the logic in the kolla container isn't quite right. 20:27:16 im currently trying to set ups a 9-15 node devcloud for our team so i need someting that is production readys so unfrotunetely kolla-k8s is not an option currently 20:27:31 ah. yeah. 20:27:54 so kolla-ansible or ceph-deply 20:28:12 ya i was planning on using kolla ansible 20:28:40 i have used ceph deploy in the past but i hate puppet so kolla-ansible is much more appealing 20:29:07 that is espically true since in plan on using the ceph with kolla-ansible deployed openstack anyway 20:29:49 makes sense then. yeah. 20:31:29 anyway my vm based test install of centos finished fine with gpt so im going to head home. catch you next week. enjoy the weekend. 20:32:05 yup. have a great one. :) 20:44:43 sbezverk: at least on my one node system, 20:44:57 deleting a ceph backed pod and having the deplyment recreate it worked ok. 20:45:01 no ceph locking issues. 20:45:18 was your test multinode? 20:45:39 this was an infernalis ceph though. so it could be a hammer issue. 20:49:28 kfox1111: yes 20:50:35 k. I'm bringing this test to a multinode cloud soon, so I'll retry it there and let you know what I find. 20:52:17 kfox1111: sounds good. What I do now, when I shut down cluster and before bringing it up, I delete locks on ceph monitor, then start cluster. I this case kube cluster comes up fine 20:53:07 that would work... 20:53:24 have you tried deleting the lock after the job is done with k8s still live? 20:53:42 that might free it up too. 20:54:00 still weirded out that deleting the pod but leaving the node up would fail. 20:54:24 can you please file that as a bug in kubernetes? I think that one is much different then the currently registerd bug, and won't get attention until specifically registered. 20:54:54 kfox1111: I tried but as long as job existed kube was rebuilding lock 20:55:48 I will run a couple of scenarios to confirm and then file a bug 20:58:40 oh. I thought part of the bug was the lock existed after the job was deleted? 20:58:45 the lock should be held as long as the job exists. 20:58:59 as a volume is exclusive to a pod. 21:14:44 sbezverk: if it is just the volume being blocked while the job exists, that is not a bug but a feature. :) 21:15:12 our workflow should collect the job result, delete the pod, and move on in that case. 21:16:23 kfox1111: well if job has finished, why would it keep the volume attached? I agree this is probably by design, but I do not think it is right.. 21:17:39 hmm... I guess I can see that... you could consider the pod done and just the result needs collecting. 21:18:25 so it might be considered a feature too... do you want to keep the pod around so you can look at its logs on completion, or do you want the pod deleted so resources are freed up. 21:19:00 Either way though, I think its really workflows job to wait until the job is done, and only launch the pod that depends on the job succeeding when its done/deleted. 21:22:14 kfox1111: agree. 21:24:12 hey folks 21:24:49 hey sdake 21:32:54 hey. :) 21:34:06 hi sdake 21:34:23 sup dudes 21:45:33 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 22:22:50 happy labor day folks in the us in case I don't see you monday :) 22:22:59 same to you :) 22:23:12 sdake: same to you 22:23:24 !! 22:23:25 adnavare: Error: "!" is not a valid command. 22:23:34 lol 22:23:36 lol 22:23:39 haha 22:23:48 :) 22:23:55 it did not gave for this 23:38:59 sbezverk there? 23:39:23 sdake: I use this in a cron job to clean out my elk: docker run --net=host --rm bobrik/curator:3.4.0 --host www.xxx.yyy.xxx --port 9201 delete indices --older-than 3 --time-unit days --timestring '%Y.%m.%d' --regex '^(logstash-.*)|(topbeat-.*)$' 23:39:40 when kubernetes 1.4 comes out, I'm going to create a scheduledjob that does that. :) 23:39:54 and contribute it to kolla-kubernetes. :) 23:40:12 we don't use logstash anywhere in our architectur 23:40:15 will that model work with just elasticsearch? 23:40:34 its just a regex. you can put whatever you want in it. :) 23:40:56 what is curator 23:42:09 https://www.elastic.co/guide/en/elasticsearch/client/curator/4.0/about.html 23:42:16 curator's really nice. :) 23:43:53 nice a license we can use 23:44:00 * kfox1111 nods 23:44:33 probably need to get that into a container and integrated in newton 23:44:53 +1 :) 23:45:37 looks pretty simple: https://github.com/bobrik/docker-curator/blob/master/Dockerfile 23:49:33 what is apk 23:49:46 oh alpines packager i think 23:49:47 oh. the distro used looks to be alpine. 23:49:48 yeah. 23:49:57 similar to yum or apt. 23:51:05 kfox1111 are u using beat? 23:53:34 a bit. been testing it out. 23:53:51 we didn't have grafana for a while, so it gave an easy alternative. 23:54:04 I think we're likely to go the grafana route though, as its more feature rich. 23:54:24 you can cram that data into elk but its a questionable fit. 23:58:22 sdake: sorry missed your ping.. still chasing cinder bug 23:59:12 sbezverk do yo uhae 3 minutse to help me with something 23:59:19 sbezverk if so restart jabber 00:01:58 sdake: my jabber is not working :-( 00:02:11 how about outlook? 00:02:24 i'll send you a mail - i want to see if my photo shows up 00:02:43 that works fine so far ;-) 00:03:16 nothing attached so far 00:03:28 I mean email came with out picture 00:04:59 try a restart 00:07:53 sbezverk which versoin of outlook are you on 00:10:16 outlook 2016 00:10:21 I think 00:10:36 the one that came with windows 10 image 00:12:43 Hi Folks, my deploy is failing when trying to get cephx keyrings. Here is the pastebin http://pastebin.com/sQiRg0eD 00:12:59 can anyone take a look at it and give an input? 00:13:10 it is saying permission error but I am running using sudo 00:13:36 and also my /etc/kolla/ceph-mon/ has ceph.client.admin.keyring 00:14:01 is kolla 1.1.2 tied to docker registry version 1? 00:14:14 adnavare pretty sur we had you operational yesterday? 00:14:18 yee379 1.10..0 00:14:28 yee379 is the minimum version 00:14:56 sdake: hey nope ceph -s had shown HEALTH_WARN with osd not up 00:15:05 sdake:mon were up 00:15:34 sdake: i am now using 3 OSDs as seems ceph don't like 1 OSD 00:16:34 so i get `ERROR:__main__:Get https://172.23.67.28:4000/v1/_ping: http: server gave HTTP response to HTTPS client`, apart from the fact that i haven’t setup https on registry:2 container; a curl to http://172.23.67.28:4000/v1/_ping returns 404 00:38:33 hi 00:38:36 hi guys 01:17:51 sup zhugaoxiao 01:18:20 wrong haplo37 :) 01:18:24 worng zhugaoxiao :0 that is 01:57:19 $your_local_time-based salutation folks 02:50:56 yo mark-casey 02:52:58 evening sdake 04:47:54 holy crap a VM is booting! 04:56:27 yo 05:16:07 Hui Kang proposed openstack/kolla: Add etcd ansible role https://review.openstack.org/364662 15:22:38 Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 13:05:49 hello gus 13:05:53 hello guys 21:09:19 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 21:22:13 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 01:39:47 Merged openstack/kolla: Add precheck for ceilometer backend https://review.openstack.org/345881 01:53:40 {"failed": true, "item": "elasticsearch", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute u'ansible_em2'"} .. getting this for all "control" hosts 01:53:53 except, 2 of the hosts actually have em2. 1 of them does not 01:54:10 i have api_interface=p1p1 in the inventory file for that different host 01:54:29 (everything worked fine before introducing the host with the different interface) 01:56:14 mnaser: which task? 01:56:27 elasticcache (first one) 01:56:29 Kolla has 3 interfaces definitions 01:56:44 duonghq, oh, the api_interface one 01:57:21 duonghq, i removed the extra host and it just went by with no problem. if i add the new host api_interface=p1p1 .. it fails 01:58:06 you get this error when deploy or...? 01:58:15 yes 01:58:44 what is the task's name? 01:59:16 TASK [elasticsearch : Copying over elasticsearch.yml] ************************** 02:00:00 it needs data in ansible_em2 (which is right, but only for 2/3 of the hosts, it should get ansible_em2 for 2 hosts, ansible_p1p1 for the other) 02:00:28 yeah, so the variables is not passed to the template 02:00:49 is that a bug or? i can submit a fix if i can see a working "example" 02:01:34 you can add a debug to config.yml playbook to print out the api_interface value 02:03:15 duonghq, rerunning with debug print api_interface 02:04:59 duonghq, reporting em2 for all 3 nodes 02:05:18 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 02:05:23 even if in my inventory file.. i have this: x.x.x.x:5555 api_interface=p1p1 02:05:54 seem that Ansible's related problem 02:06:22 ansible 2.1.1.0 on this host hmm 02:07:28 let me recheck for a moment 02:09:05 duonghq, oddly enough .. ansible -m debug -a "var=hostvars[inventory_hostname]" -i inventory all => shows "api_interface": "p1p1", 02:09:25 stranger 02:10:33 whats even more strange is that all 3 hosts report AnsibleUndefinedVariable for ansible_em2... 02:10:42 even though 2 of those hosts actually have em2 02:11:53 can you print the hostvars[inventory_hostname] in above playbook 02:14:45 duonghq, ansible_em2 exists in both first hosts, and ansible_p1p1 exists in the 3rd host 02:15:08 but, api_interface set to em2 02:15:59 you use latest master? 02:16:04 mnaser? 02:16:45 yes duonghq 02:23:03 mnaser: I need rebuild docker image, it take some time 02:24:35 np duonghq .. maybe building elasticcache only to replicate might be enough 02:25:26 my images are outdate for a while, Kollish has updated base and kolla-toolbox image 02:27:41 ah 02:27:56 thats gonna be quite the images to rebuild then 02:27:58 ill keep looking at this 02:33:11 zhongshengping proposed openstack/kolla: Deprecate old auth parameters of ironic in nova.conf https://review.openstack.org/364259 02:39:17 this seems similar duonghq https://review.openstack.org/gitweb?p=openstack/kolla.git;a=commitdiff;h=a98628453b0c1fc72692376abd873cbccb5a35a9 02:40:24 discovery.zen.ping.unicast.hosts: [{% for host in groups['elasticsearch'] %}"{{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}"{% if not loop.last %},{% endif %}{% endfor %}] 02:40:28 i think this is where things are going wrong 02:42:39 they're all breaking cause api_interface for that host is wrong 02:44:30 the site.yml is re-arranged 02:45:56 does the order of my inventory file matter duonghq ? 02:46:47 iirc: no 02:46:51 duonghq, tell me if this makes sense.. according to playbook docs .. http://docs.ansible.com/ansible/playbooks_variables.html 02:47:22 "The idea here to follow is that the more explicit you get in scope, the more precedence it takes with command line -e extra vars always winning. " 02:47:38 when kolla-ansible runs, it does this.. ansible-playbook -i inventory -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e action=deploy /usr/share/kolla/ansible/site.yml 02:47:51 which basically means that globals.yml can never be overridden 02:48:07 i think the problem here is that i have to comment out api_interface in globals.yml IF i want to use it in each host 02:48:35 iirc, the api_interfaces is already commented? 02:48:35 i cant use api_interface in globals.yml and then override it in the inventory file 02:48:49 well, not in my case, i changed it (because 2 of my hosts had a specific api_interface) 02:50:42 there we go, so I commit to a single api_interface in globals.yaml OR configure them individually in inventory.. 02:51:42 mnaser: I have not tried to set api_interface yet, only in inventory 02:53:17 looks like that was the reason :0 02:53:43 mnaser: good to heard that 03:11:36 Yao Lu proposed openstack/kolla: delete virtual bridge artifacts when cleaning hosts https://review.openstack.org/358924 03:33:37 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 03:54:32 sean-k-mooneyAFK: ping 03:59:33 Jeffrey Zhang proposed openstack/kolla: Change Mariadb default character set to utf8 https://review.openstack.org/365428 04:27:33 Nguyen Phuong An proposed openstack/kolla: Ensure doesn't use LOG.warn https://review.openstack.org/364168 07:15:37 Lu lei proposed openstack/kolla: Add oslo.i18n module in kolla project https://review.openstack.org/355342 07:17:21 Lu lei proposed openstack/kolla: Add oslo.i18n module in kolla project https://review.openstack.org/355342 07:28:33 duonghq hi not sure if your still about but i just got in 07:29:08 sean-k-mooney: hello 07:29:18 duonghq: hi 07:29:33 duonghq: did you want to talk to me earlier? 07:29:46 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 07:29:49 yes, waiting for a while 07:30:24 I want to diccuss about your barematel 07:30:30 sure 07:31:31 I see your pre/post install playbook is more general than only barematel? 07:31:54 I think it should be placed in its own role 07:32:03 like pre-install or pre-deploy? 07:32:15 it is its own role 07:32:40 birforst which does the baremetal install is a seperate role 07:32:58 afaik: not every server/node is baremetal? 07:33:01 we could chagne the name of the baremeatl role to kolla-hosts 07:33:46 but the bifrost is Ansiblize Ironic 07:33:53 and Ironic is for barematel 07:34:06 duonghq: the only reason i called it baremetal was to signify that it was altering the host unlike the rest of kolla which does not 07:34:38 sean-k-mooney: ok 07:35:30 just forget bifrost role 07:36:03 so, I think we should change the the name to make it more clearly? 07:36:14 I confused for a while 07:36:22 there are 3 things that will hopefully be available in newton. 1 we are fixing ironic as a service to provide baremetal host to tenents, 2 we are providing contianerised bifrost to give baremetal provisioning to operators and 3 we are provideing a kolla host playbook(the baremetal role) to install kollas depencies 07:37:15 roger 07:37:33 thank you 07:37:33 duonghq: sure i can do that, when i started on the playbook we had not started calling it kolla hosts so ill change it to the host role? 07:38:03 or would you perfer pre-install or bootstrap? 07:38:48 Jeffrey Zhang proposed openstack/kolla: Remove the useless host in site.yml https://review.openstack.org/365483 07:39:26 sean-k-mooney: I think small voting in ml is more appropriate 07:39:58 ok cool will do ill send a mail later today 07:40:34 personally, I prefer bootstrap due to the kolla-ansible command is boostrap-server, so it make new-comer easier to get into your works :) 07:42:18 i guess it dose not help that i was also working on the containerisation of bifrost this cycle too, i can see why beremetal might be confusing in that context 07:43:30 yep 07:46:40 sean-k-mooney: one more question: why you assign only /etc/kolla to kolla user, not group too? 07:47:00 *assign /etc/kolla only to user 07:47:26 that may be a bug. i taught i assigned it to kolla:kolla 07:48:06 duonghq: it currently does not break anything but i probably intended kolla:kolla 07:48:29 sean-k-mooney: would you mind if I fixed that 07:49:25 go ahead, you found it :) 07:49:39 okay 07:51:44 ok ill be back soon got to run to the lab for a bit 07:53:28 ok 08:00:37 Lu lei proposed openstack/kolla: Add oslo.i18n module in kolla project https://review.openstack.org/355342 08:06:13 Duong Ha-Quang proposed openstack/kolla: Init group owner /etc/kolla to kolla https://review.openstack.org/365494 08:09:10 Duong Ha-Quang proposed openstack/kolla: Init group owner of /etc/kolla to kolla https://review.openstack.org/365494 08:11:32 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 08:20:34 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 08:44:02 morning 09:07:43 Lu lei proposed openstack/kolla: Add oslo.i18n module in kolla project https://review.openstack.org/355342 09:11:16 good morning/evening everyone 09:11:34 is it only me who sees doubled output during build? 09:12:26 I tried in two different environments (both centos) 09:19:54 https://bugs.launchpad.net/kolla/+bug/1620239 09:19:54 Launchpad bug 1620239 in kolla "build process outputs each line twice" [Undecided,New] 09:21:29 hi! 09:45:08 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 09:46:01 zhubingbing is the sahara role part of rc1? 09:46:16 yeag 09:46:17 yeah 09:46:53 Do you have interesting for sahara? 09:47:20 zhubingbing not at the moment, I am waiting for your barbican role :) 09:47:30 sure , 09:47:39 barbican will be ok 09:47:52 yes, just nits... 09:48:01 you plan use to barbican for something? 09:48:17 yes, I want to use it for signed images 09:49:01 ok 09:50:07 I have used the keysthone and Barbican together. 09:51:29 I think there are a lot of use cases for barbican, notably in public cloud environments 09:52:06 i agree 09:52:23 have you worked with signed images? 09:52:38 i don't test it 09:53:40 If there is a need I can cooperate with the test for worked with signed images 09:56:48 Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (1) https://review.openstack.org/358539 09:58:32 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 09:58:49 Are there any plans to support the ec2-api project in kolla? 09:59:09 NachoDuck no open blueprint at the moment 09:59:25 Ok thanks 09:59:48 NachoDuck but we can open one, this is not a big deal 10:03:19 Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (2) https://review.openstack.org/359031 10:05:02 zhubingbing can you set the implementation status of https://blueprints.launchpad.net/kolla/+spec/rally-role? 10:05:16 ok 10:06:25 zhubingbing priority is also missing, i think low.. 10:07:05 i can't setting it 10:07:07 =- 10:07:35 sdake can you set the priority of https://blueprints.launchpad.net/kolla/+spec/rally-role 10:08:43 berendt you should be able to do so 10:08:49 zhubingbing I think you forgot to upload your recent update of https://review.openstack.org/#/c/351822 10:09:46 sorry 10:09:51 i forget it 10:10:05 i fixed it now 10:10:48 zhubingbing "Implementation" is still missing 10:11:13 ok 10:11:17 thanks 10:11:22 i add it 10:11:45 cool. than all blueprints for rc1 have useful values ;) 10:11:49 hi, all. I got a patch :Add oslo.i18n module in kolla project . Here is links:https://review.openstack.org/#/c/355342/ . But I not sure we need to internationalize only in kolla directory or in all directories. Any suggestion ? 10:13:31 lulei the assigned blueprint is not approvied, I think we should discuss this feature first 10:13:44 lulei I am not sure if we need i18n at all for the kolla commands 10:14:54 ping hiking 10:15:05 ping huikang 10:15:10 hi berendt, OK. I got it . here is bp links: https://blueprints.launchpad.net/openstack/?searchtext=kolla-stadium-i18n 10:17:42 lulei i commented in the review itself.. 10:18:12 zhubingbing proposed openstack/kolla: Add Barbican container dockerfile https://review.openstack.org/351822 10:18:33 Christian Berendt proposed openstack/kolla: Add missing Docker images of OpenStack services to README file https://review.openstack.org/364804 10:18:56 lulei i've loe to hae i18n translations for build tools and kolla-ansible (which is currently shell) 10:19:03 ping Christian Berendt 10:19:12 lulei to make that happen, we need to rewrite kolla-ansible (the shell script) in python 10:19:35 lulei i think for ocata that should be something we tackle upfront 10:20:34 lulei ithink what probably won't happen is rewritting all tools/* files in python - so some things will remain uninternationalized 10:20:52 sdake. So, I'm not sure how to do next. 10:20:56 lulei are you inovlved with the i18n tteam? 10:21:13 lulei here is my concern- has to do with timing 10:21:33 we are in rc1, and adding internationalization to he code bsae is surely todisrupt i18n translators 10:21:41 wee hve already missedthestring freze 10:21:45 Christian i see you submit add cloudkitty role, and clouldkitty can use gnocchi . so can you help me review gnocchi role ? 10:22:24 sdake, yes, I know a guy which is a oslo core review. I will ask him about this. 10:22:44 snake , freze ? 10:22:47 zhubingbing use berendt instead of christian, this way i will receive an highlight. yes, i will have a look on gnocchi until the mid of this week 10:22:54 lulei i think theimpact is more on the i18n team 10:23:11 which i haven't reached out to yet - too much stuff on our plates to sort out internationalization this cycle 10:23:15 berendt thanks 10:23:46 sdake, I got it. Thanks:) 10:23:48 eveer watch survivorman? 10:24:03 it not freezer ? 10:24:03 i sawthis cat come toarizona and "survive" 10:24:19 he came duringthe mildest part of the year 10:24:20 lol 10:24:33 go out there when its 115f, and i'll take your seouslly :) 10:24:56 he did go to a desert where it was 120 at night though 10:25:03 so iam sure he could handle it 10:25:10 and tthe deserthe was in hadno resources at all 10:25:12 － － 10:25:40 i've spent weeks in the desert "living off the land" 10:25:53 but it was september - arguably the best time to do such a thing 10:26:13 october mightbebetter 10:26:23 part of feagle scout training 10:26:32 living off the land what mean ? 10:26:43 zhubingbing using resources in the environment to survive 10:26:47 ok 10:26:56 setting traps for mice and whatnot 10:27:06 eating crickets 10:27:08 that sort of thign 10:27:35 nothing as good as 6-10 cooked crickets after not havng had a meal for 3-4 days 10:27:45 protein- yum ;) 10:27:46 Wow that sounds fun 10:28:25 we were permitted t o bring 3 things 10:29:37 i chose lighter, pocket knife, and water purifier 10:29:53 (plusclothes on our back) 10:30:12 some dudes washed out early 10:30:17 they didnt bring those 3 things 10:30:37 －wow 10:30:48 it is too funny 10:31:06 (you were also give an emegency radio for them to comoe carry you out and you had to check in everyday) 10:31:33 i want to take part in tia activity 10:31:36 ya eagle scouts was a fantastic exeprience 10:32:53 the best meal was a snake ;) 10:33:12 i want to do it 10:33:20 takes alot of training 10:33:34 where you go ？ 10:33:36 99% of people would fil 10:33:43 arizona desert 10:33:48 fil/fail 10:34:31 yeah， 10:34:55 lots o wateer in the dsert 10:35:00 It is difficult for most people 10:35:06 i found a spot with water and stayed there ;) 10:35:10 Vladislav Belogrudov proposed openstack/kolla: Build process outputs each line twice https://review.openstack.org/365567 10:37:00 Do you take part in this activity alone 10:37:27 ya solo 10:37:37 there were 6 cats in total 10:37:40 4 washed out 10:38:16 the other guy that made it got eagle scout at same time as i did 10:38:31 the other 4 dudes made it the next year 10:38:43 It looks really fun, I also want to participate in the back of your success 11:18:41 hi, sdake ,berendt. Thanks for your suggestion. Now I think we need to internationalize in kolla directory, other directories may be discuss next time. BTW, I will rewrite the bp for this. Thanks guys:) 11:19:47 lulei there is a dependent blueprit of writing kolla-ansible in ansible 11:19:50 rather in python 11:21:35 sdake, ok 11:22:26 the ui is really waht needs to be i18n'ed 11:22:41 not the build tool 11:25:19 sdake , yes, I think test directory need't to internationalize . 11:27:16 Hmm something going wrong for me with this attempt to get version, any suggestions ? 11:27:16 self._version = self._retrieve_server_version()\n File "/usr/lib/python2.7/site-packages/docker/client.py", line 105, in _retrieve_server_version\n \'Error while fetching server API version: {0}\'.format(e)\nDockerException: Error while fetching server API version: (\'Connection aborted.\', error(111, \'Connection refused\'))\n' 11:27:45 jmccarthy context? 11:29:18 When trying to deploy, the Creating log volume task create_volume library/kolla_docker.py: - create_volume tries to do looks like 11:30:14 I mean: roles/common/tasks/bootstrap.yml:- name: Creating log volume 11:30:53 (I think I have latest ansible from master) 11:33:12 the error is probably raised by "Starting heka bootstrap container".. have you checked that your nodes can access the configured Docker registry? 11:34:00 I believe they can, I'll double check 11:34:29 zhubingbing proposed openstack/kolla: xAdd gnocchi ansible role https://review.openstack.org/349351 11:34:53 if the error is raised by "Creating log volume" then the task cannot access the local Docker service 11:35:16 berendt: You may have hit it in one ! I upgraded to newer docker but seem to have lost some settings in the process 11:39:17 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 12:03:16 now neutron db have problem 12:07:03 morning guys 12:22:53 Merged openstack/kolla: Add missing Docker images of OpenStack services to README file https://review.openstack.org/364804 12:50:01 Jeffrey Zhang proposed openstack/kolla: DO_NOT_MERGE: Use better URL for China https://review.openstack.org/329060 12:50:53 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 13:48:04 just wanted to give feedback and say that the work done in kolla is absolutely great 13:48:14 and we're in the final/close stages of moving a pretty large production workload on it 14:02:16 Kevin Fox proposed openstack/kolla-kubernetes: Docker systemd fix https://review.openstack.org/365664 14:09:50 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 14:46:11 Jeffrey Zhang proposed openstack/kolla: DO_NOT_MERGE: Use better URL for China https://review.openstack.org/329060 15:12:04 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 15:39:20 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 15:49:58 Jeffrey Zhang proposed openstack/kolla: [wip] using ci source mirror https://review.openstack.org/365721 16:04:01 hello all 16:08:00 Merged openstack/kolla: Updating ironic configurations in nova.conf https://review.openstack.org/364228 16:08:16 @pbourke, are you here and have a while? 16:12:32 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 16:19:55 Kevin Fox proposed openstack/kolla-kubernetes: Host Labels https://review.openstack.org/365741 16:44:32 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 17:32:17 Jeffrey Zhang proposed openstack/kolla: [wip] using ci source mirror https://review.openstack.org/365721 17:51:09 Bartłomiej Daca proposed openstack/kolla: Update murano service name https://review.openstack.org/365761 18:58:33 Hi, guys! I am in progress of testing Kolla and it looks good! Just wanted to check if there is any high availability guide or experience? 19:07:35 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 19:25:18 YefimG, i mean everything seems to be prewired for HA so im unsure what you're looking for? 19:28:22 mnaser: Yes, OpenStack endpoints are highly available out of the box because of haproxy, but I am looking for instance (nova-compute) failover 19:29:01 nova-compute fail over in what case? 19:29:10 nova-compute is homed on a physical compute node 19:29:28 if the physical compute node crashes, nova-compute serves no purpose 19:30:24 Mauricio Lima proposed openstack/kolla: Fix Typo in quickstart https://review.openstack.org/365777 19:31:26 mnaser: sorry, I mean instance failover in general. If a compute node fails - instances are automatically restarted on another compute node 19:31:40 Something you would generally achieve with pacemaker, I guess 19:32:08 oh, that's something else, i dont know if the kolla team has this in the scope of the project 19:32:10 YefimG this is not a kolla feature and not a nova-compute feature and not a cloud feature at all 19:32:25 this is not in the scope of this project and it will hopefully never be in scope 19:32:56 YefimG: https://ask.openstack.org/en/question/63031/automatic-failover-instances/ 19:34:04 if you want to do this, some google-ing will help you find scripts for pacemaker that accomplish that automatically 19:35:15 berendt, bdaca, mnaser, Thanks a lot for clearing this out, will definitely do some google-ing :) 19:36:09 YefimG read about it but please do not do it, it is not a feature that you want to use in a cloud enviroment 19:37:08 there are reasons why nobody implemented it and why only nasty workarounds are available to do it 19:37:15 and it will NOT result in high available instances 19:41:16 berendt: Thanks for detailed answer! 19:41:37 YefimG you are welcome. do you know the 12 factor app? 19:45:22 berendt: Yes, but unfortunately in my case there are needs in hosting old monolith applications. There are few of them, so building a separate enterprise-like virtualization would be cost-effective, that's why I wondered about the OpenStack case. But it is clear for me that I should search for a better approach. 19:45:46 would not be cost-effective* 19:46:14 the problem is that this enterprise -like high available virtualization makes no sense, it should be a no-go to start a dead instance somewhere else without taking a look on it 19:46:49 but a lot of manager think that this will solve there issues with existing legacy applications 19:49:11 While most of the new deployments are cloud-aware, there are some P2V migrations for customers with old and unique software, that just can not be re-written :( 19:54:37 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 20:10:06 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 20:16:09 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 20:19:19 rhallisey: hey 20:19:26 kfox1111, hey 20:20:32 you familior with the gate? 20:21:09 ya 20:21:22 I'm really close I think: https://review.openstack.org/#/c/365733/ 20:21:33 it runs on my machine, but not on the gate boxes. 20:23:08 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 20:23:22 it's beacuse the gate it looking for a different script 20:23:27 let me find it 20:25:47 kfox1111, https://github.com/openstack-infra/project-config/blob/5c5f70bf3ce48ee1872432705b5dbf9857d9e01b/jenkins/jobs/kolla-kubernetes.yaml 20:26:07 the gate also doesn't trigger unles you add 'check experimental' 20:27:20 rhallisey: I heard from someone on infra, that that yaml file isn't actually used. 20:27:41 hmm.... maybe that's why? 20:27:50 I guess I mean checks, not gate. 20:28:12 like sanity checks? 20:28:18 I'm confused what you're after 20:29:02 I want jenkins to stop failing on that review :/ 20:29:10 oh 20:29:29 and it updates jenkins to fail if the templates are invalid. 20:29:48 jinja2.exceptions.UndefinedError: 'hostvars' is undefined 20:29:54 kfox1111, I'l comment on there sec 20:29:59 k. thx. 20:30:35 hmm... a little farther now I think. 20:36:48 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 20:40:53 kfox1111, it seems like one of the templates being rendered is no valid.. 20:40:58 I don't know which it is 20:41:11 its valid. it passes on my host. 20:41:22 here's the current issue: http://logs.openstack.org/33/365733/8/check/gate-kolla-kubernetes-python27-ubuntu-xenial/347ca2b/console.html 20:41:29 its like its not finding all-in-one now. 20:42:10 which kind of makes sense, as I've never told it which to use, but I didn't on my machine either which is confusing. 20:44:49 hmm... has all the kolla address patches we need merged? Maybe its that. 20:45:18 I don't think the nova one has 20:45:39 its failing on the first template it tried. ceph-secret 20:46:02 ya 20:48:01 hmm... fresh checkout of kolla works ok. 20:48:18 something's differnet but really not clear what... :/ 20:48:35 maybe part of the jinja engine isn't detecting the checkout of kolla? 20:51:50 kfox1111, mabe a different version of python-jinja2? 20:52:09 I did a virtualenv with whats in deps. 20:52:16 so should be the same. 20:52:38 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 20:53:04 do you know if we can get an ssh into the failing machine? its kind of hard to debug this way. :/ 20:53:16 fortunately it only takes about 2 min per test though. 20:54:16 going to need some prints 20:54:23 that's the best way to gate test 20:55:48 yeah... :/ 20:55:53 maybe a strace too... 20:56:22 ah... here we go... 20:56:57 I added grep api_interface_address ../kolla/etc/kolla/globals.yml || echo api_interface_address: "0.0.0.0" >> ../kolla/etc/kolla/globals.yml 20:57:03 and it got a bit further... 20:57:16 ok 20:58:32 it would be nice if we could make a test globals.yml that we could spicifically load in... 20:59:45 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 20:59:47 you could 21:01:49 we'd have to add a way to override which one was used instead of the probing code... 21:02:09 maybe we get this one working, and do that one as a followon patch. 21:04:33 nice... got it to pass 2.7. 21:05:08 ok. the last one seems to be a python exception throwing string difference.... 21:05:19 stupid lack of standard error strings or something better. :/ 21:05:26 but totally fixable. 21:07:11 ya closer 21:07:48 oh... its failing in the TypeError... no .message.... 21:07:49 bleh. 21:08:55 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 21:16:22 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 21:18:21 once this test is working, I can add tests for some of the stuff like ensuring hostselectors and namespaces are always in the template to prevent regressions. 21:21:30 ok, there's 34... 21:21:36 just a pep8 issue... 21:22:08 Kevin Fox proposed openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 21:27:16 yay! :) 21:27:46 that was harder then it should have been. :/ 21:27:54 oh well. this should help a lot. :) 21:31:38 nice :0 21:31:40 :) 21:33:44 I tried hooking it directly into kubectl to have it validate too, but there is no way currently to have it validate and not upload. 21:34:02 I may need to create a kubectl patch... 23:54:58 Jeffrey Zhang proposed openstack/kolla: [wip] using ci source mirror https://review.openstack.org/365721 23:55:27 Jeffrey Zhang proposed openstack/kolla: [wip] using ci source mirror https://review.openstack.org/365721 00:12:31 Kevin Fox proposed openstack/kolla-kubernetes: Checks to start reducing Technical Debt https://review.openstack.org/365805 01:03:35 sean-k-mooney: can you review this ps: https://review.openstack.org/#/c/365494/ 01:09:09 Ross Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 01:24:46 Wei Cao proposed openstack/kolla: Add solum ansible role https://review.openstack.org/358938 01:33:09 Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 01:52:09 sean-k-mooney: ping 01:53:43 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 02:02:03 Kevin Fox proposed openstack/kolla-kubernetes: Checks to start reducing Technical Debt https://review.openstack.org/365805 02:39:52 Kevin Fox proposed openstack/kolla-kubernetes: Checks to start reducing Technical Debt https://review.openstack.org/365805 03:02:27 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 03:14:04 Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (1) https://review.openstack.org/358539 03:14:04 Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (2) https://review.openstack.org/359031 03:16:19 Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (last) https://review.openstack.org/359096 03:17:25 <5EXABAVU9> ping Mathias 03:22:40 Lu lei proposed openstack/kolla: Use oslo.i18n to translate messages in kolla project https://review.openstack.org/365829 03:23:53 Jeffrey Zhang proposed openstack/kolla: [wip] using ci source mirror https://review.openstack.org/365721 03:43:10 Kevin Fox proposed openstack/kolla-kubernetes: Add namespace and test https://review.openstack.org/365836 04:36:30 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 04:38:07 Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (last) https://review.openstack.org/359096 04:52:26 sdake: is there any way to add new user to dsvm? 04:52:30 before test is run 06:31:39 mistral doesnot work with keystone3, unless /v3 is appened to auth_uri in ansible/roles/mistral/templates/mistral.conf.j2 06:31:55 is there planned fix for this 06:32:27 kamal__: you can file a bug 06:33:10 ok thanks, I will 06:37:44 morning people 06:44:19 Merged openstack/kolla: Fix Typo in quickstart https://review.openstack.org/365777 06:44:39 Takashi Sogabe proposed openstack/kolla: Keepalived leaves VIP on the host in case of accidental removal. https://review.openstack.org/364085 06:44:57 morning 06:53:38 morning o/ 06:53:57 morning 06:54:21 hi 06:54:35 Christian Berendt proposed openstack/kolla: Add cloudkitty containers for type source https://review.openstack.org/349295 06:55:25 we should hire somebody who resolves merge conflicts.. 06:58:15 Christian Berendt proposed openstack/kolla: Enable CADF events in Keystone and introduce enable_cadf_notifications https://review.openstack.org/349006 06:59:10 Can somebody please check and approve https://review.openstack.org/#/c/335495/ 07:00:49 not yet.. looks like there is an issue with the new xenial images, i only tested on trusty 07:05:54 Md Nadeem proposed openstack/kolla: Replace deprecated rabbitmq parametrs https://review.openstack.org/365894 07:12:55 Christian Berendt proposed openstack/kolla: Improve script to check used source versions https://review.openstack.org/352640 07:15:53 Christian Berendt proposed openstack/kolla: Enable deployment of horizon in tools/deploy_aio.sh https://review.openstack.org/359451 07:18:09 Christian Berendt proposed openstack/kolla: Install neutron-lbaas plugin in neutron-server container https://review.openstack.org/329487 07:28:27 Jeffrey Zhang proposed openstack/kolla: Using CI mirror host https://review.openstack.org/365721 07:31:25 Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (1) https://review.openstack.org/358539 07:31:48 Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (2) https://review.openstack.org/359031 07:31:57 Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (last) https://review.openstack.org/359096 07:40:18 dupengfei proposed openstack/kolla: Fix lost parameter --remote in generated start_ovsdb_server.sh https://review.openstack.org/362027 08:27:42 duonghq: if you mean after devstack's finished and before tempest is run you may want to look into local.sh devstack is looking for - https://github.com/openstack-dev/devstack/blob/master/stack.sh#L1365-L1369 08:29:37 wznoinsk: thank you, I need setup non-root user for deployment due to the user is added in bootstrap-servers 08:30:26 right before kolla-ansible deploy is run 08:33:39 Christian Berendt proposed openstack/kolla: Improve README file https://review.openstack.org/365941 08:53:39 Christian Berendt proposed openstack/kolla: Fix keytone options in all.yml/globals.yml https://review.openstack.org/365950 08:56:45 Yao Lu proposed openstack/kolla: TrivialFix: Delete virtual bridge artifacts when cleaning hosts https://review.openstack.org/358924 09:02:15 Paul Bourke proposed openstack/kolla: Check volume group exists for Cinder in prechecks https://review.openstack.org/354154 09:05:34 Paul Bourke proposed openstack/kolla: Allow use of database for Horizon sessions https://review.openstack.org/359794 09:06:21 Yao Lu proposed openstack/kolla: TrivialFix: Delete virtual bridge artifacts when cleaning hosts https://review.openstack.org/358924 09:29:46 Paul Bourke proposed openstack/kolla: Allow mysql to be used as a database for Ceilometer https://review.openstack.org/360669 09:46:04 duonghq ye it canbe done 09:46:06 duonghq but needs to be done in your patch 09:46:48 moring folks 09:46:50 sdake: how can I do? 09:50:37 duonghq modify setup_gate.sh script 09:50:49 sdake: thanks 09:51:11 if you want it done each individual node, need to modifythe ansible code 09:52:28 sdake: you mean the playbook? 09:52:29 duonghq: https://review.openstack.org/#/c/365494/2 looks good to me. thanks for fixing it. 09:52:42 https://github.com/openstack/kolla/blob/master/tools/setup_gate.sh#L125 09:52:58 duonghq yess look at link above 09:53:07 that sets up each of the individual nodes 09:53:16 some day we hpe to have multinode gating 09:53:50 thank sean-k-m1 09:54:37 sdake: understand 09:54:51 hello guys 09:56:54 zup zhubingbing onfire ;) 09:57:08 lol 09:57:58 http://www.urbandictionary.com/define.php?term=on%20fire 09:58:00 sdake, do you think i should add ceilometer to support gnocchi 09:58:17 if it isn't too painful 09:58:22 for both of us :) 09:59:13 Md Nadeem proposed openstack/kolla: Correct Murano keystone_authtoken option name https://review.openstack.org/366019 09:59:38 sdake: can you review this very trivial fix: https://review.openstack.org/#/c/365494/2 10:00:48 http://www.urbandictionary.com/define.php?term=on%20fire what to do for this website 10:01:35 This site is used to do？ 10:02:46 － － 10:06:06 sdake: for my bp, I have some update here: https://review.openstack.org/#/c/358539/ new approach, if you have spare time, can you review this? 10:19:31 Merged openstack/kolla: Init group owner of /etc/kolla to kolla https://review.openstack.org/365494 10:21:02 * duonghq finish working day 10:21:04 bye bye 10:23:10 zhubingbing, ping 10:23:18 here 10:27:48 Vladislav Belogrudov proposed openstack/kolla: Cinder-backup misses client part for NFS backend https://review.openstack.org/366037 11:01:33 morning guys 11:02:23 morning mliima 11:27:34 Md Nadeem proposed openstack/kolla: Correct Murano keystone_authtoken option name https://review.openstack.org/366019 11:33:32 guys, we have some third party plugin implemented or with your implementation in progress? 11:42:00 Christian Berendt proposed openstack/kolla: Improve README file https://review.openstack.org/365941 11:42:42 Christian Berendt proposed openstack/kolla: Fix keytone options in all.yml/globals.yml https://review.openstack.org/365950 12:04:24 Prithiv proposed openstack/kolla: Ansible roles for collectd Container https://review.openstack.org/366082 12:05:58 Prithiv proposed openstack/kolla: Ansible roles for collectd Container https://review.openstack.org/366082 12:09:17 Mauricio Lima proposed openstack/kolla: Comment some lines in globals.yml https://review.openstack.org/366083 12:09:25 Vladislav Belogrudov proposed openstack/kolla: Cinder-backup misses client part for NFS backend https://review.openstack.org/366037 12:10:38 Mauricio Lima proposed openstack/kolla: Comment some lines in globals.yml https://review.openstack.org/366083 12:17:34 Jeffrey Zhang proposed openstack/kolla: Using CI mirror host https://review.openstack.org/365721 12:37:19 guys, we have some third party plugin implemented or with your implementation in progress? 12:49:50 mliima, I don't know if someone has one in production 12:50:11 should add a simple one 13:10:33 if someone has some spare time, would they mind reviewing https://review.openstack.org/#/c/358305/ 13:18:03 zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 13:18:07 hello 13:25:58 hi kola devs 13:26:16 I'm having trubles I've never seen before with kolla 13:27:24 few components refuse to start with an error : sudo: unknown uid 162: who are you? 13:27:35 does anyone have a clue? 13:34:33 ironic container logs are not getting logged in log files 13:34:57 while checking the ansible code the only change I see from other components is 13:35:08 log_dir variable in the conf files 13:35:33 is this the only change needs to be done or any other changes are required ? 13:39:42 hmm, ok rhallisey 13:39:45 thank you 13:45:34 Jeffrey Zhang proposed openstack/kolla: Using CI mirror host https://review.openstack.org/365721 13:52:21 Vladislav Belogrudov proposed openstack/kolla: Cinder-backup fails while backing up iSCSI volumes https://review.openstack.org/366146 13:58:33 hi huikang 13:58:47 Hi zhubingbing_ 14:31:46 good morning 14:32:05 morning. 14:32:14 got a bunch of reviews up. :) 14:32:29 yeah yeah, let me clear up my kolla-ansible queue first 14:32:35 hehe. k. 14:42:01 Merged openstack/kolla: Replace deprecated rabbitmq parametrs https://review.openstack.org/365894 14:45:10 sdake, howdy 14:45:15 tag happends today right? 14:45:22 yo 14:45:28 for miletstone 3? 14:45:41 that was last week 14:45:52 we tag zstreams this week 14:46:08 ok, I think dhellman wants to do this today 14:46:20 it has alrady beeon done 14:46:21 but we're not covered by that right? 14:46:43 https://github.com/openstack/kolla/releases/tag/3.0.0.0b3 14:46:55 covered by which 14:47:28 let me find mail 14:48:09 ahh this was about creation of stable/newton branch 14:48:21 that is supposed to happen in rc1 14:48:38 did you see something about branching of stable/newton this week? 14:48:52 Merged openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 14:49:30 http://lists.openstack.org/pipermail/openstack-dev/2016-August/102339.html sdake 14:49:32 we definately donot want to branch 14:49:45 ya that is libraries 14:49:50 not projects 14:50:14 ahh ok 14:50:15 We will wait to create server branches until the RC1 tag, as usual. 14:50:18 makes sense 14:50:36 do we need to re-iterate voting about kolla-ansible 14:50:37 ? 14:50:42 new repo 14:51:02 we did got majority afair when rhallisey got voting up 14:51:08 with or without end date 14:51:12 i think timing is not right 14:51:23 atleast i dont want to split right this second 14:51:31 atleast post rc1 14:51:37 well, sure 14:51:39 that's what I mean 14:52:17 we can setup repo and prep everything 14:52:34 the repo setup is a clone of kolla master 14:52:37 we can remove code from kolla itself after rc1 14:52:39 with a bunch of git rm after 14:52:52 or empty repo with one big commit 14:52:56 NO 14:53:01 but I guess git rm will retain history 14:53:03 that toaly loses all history 14:53:29 and damages our ability to apply for stabl etags in addition 14:54:00 ok, git logistics will be figured out 14:54:01 here is the deal 14:54:08 the split is fine with me 14:54:13 that's exactly why I want to start doing it asap 14:54:15 but lets do it around the end of the release 14:54:28 for backports? 14:54:30 if we do it before, we could potentiallly crater our ability to ci 14:54:36 I was thinking to just clone repos 14:54:38 backports will alsobe a problem 14:54:44 do backports without any issue 14:54:50 as repos will be exact clone 14:55:07 and then do git rm everything besides ansible from kolla-ansible and ansible from kolla itself 14:55:23 when we release 14:55:27 3.0 officially 14:55:39 no, we are using the current modele for 3.0 14:55:47 that is a major major change 14:55:49 yeah, that's what I mean 14:55:53 no way i'm introdcuing that at the end of the cycle 14:55:59 we do git rm *after* we release 14:56:03 oh 14:56:05 ok wfm 14:56:09 brb 14:56:14 i'd like to wait until h official releae is done 14:56:58 it takes about 1 week to go through the repo creation process (waiting on tc mostly) 14:58:14 ping Christian Berendt 14:58:22 zhubingbing_ pong 14:58:31 hello 15:04:33 sdake, have you ever eared about issues that I encounter with kolla deployments? 15:04:51 sdake, few components refuse to start with an error : sudo: unknown uid 162: who are you? 15:09:42 matrohon, what components? 15:10:16 vbel : nova-compute, nova-libvirtd, neutron ovs, neutron-l3, neutron-dhcp 15:10:43 vbel : looks like an unexpected sudo command is triggered 15:11:02 that's lots, what is your environment (host os, images)? 15:11:20 sdake, only thing we can't do before official release is git rm ansible from kolla repo 15:11:29 and that's literally single commit 15:11:49 vbel : host-os : debian-testing with docker 11, base image : centos 15:12:12 matrohon: what it tries to do with sudo? 15:12:59 vbel : I don't know! do you know how I can figure it out? 15:13:34 in heka container there are logs for the components in /var/log/kolla 15:13:46 hi sdake 15:14:42 matrohon: is it all-in-one deploy? 15:14:56 vbel, yep 15:15:43 i hope cloud kitty in newton merge， I'll finish the work with bearendt 15:15:45 cloud kitty 15:16:18 vbel nothing gets logged, the sudo cmd seems to happen before any logs gets sent 15:19:38 matrohon: can you paste your the deploy failure somewhere? e.g. paste.openstack.org 15:20:54 matrohon: another experiment I would try - run image by hand, e.g. nova and check if nova user is in (or anything with id 162) 15:21:08 vbel, well actually I don't have have much to past... deployment is ok, but containers fail to start, and I can see the sudo error with docker logs 15:22:14 hello guys 15:22:46 just a little question, kolla works fine with lvm? 15:24:34 matrohon: then I would just run container by hand, e.g. docker run -ti your_registry:port/kolla/centos-binary-nova-libvirt:3.0.0 bash , and check "grep 162 /etc/passwd", "sudo -u nova bash" 15:25:09 mliima: yes, it should 15:25:31 mliima: though this is for demo/toy clouds :) 15:26:13 or for clouds which doesn't give a remote crap about their data 15:26:49 ok vbel, thanks 15:27:18 Eduardo Gonzalez proposed openstack/kolla: Add Senlin Ansible role https://review.openstack.org/339725 15:27:26 on that note, we need to bump ceph 15:27:30 we're on hammer still 15:28:20 i'm running kolla deploy and i was using ceph, but now i have to use lvm :) 15:30:18 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 15:32:10 inc0: big +1 15:32:17 we need to get to jewel asap. :/ 15:32:23 vbel : it ends up with "Sorry, user nova is not allowed to execute '/usr/bin/bash'" 15:32:24 yeah agree 15:32:37 first thing in ocata 15:32:42 I'm going to have to patch all my containers with jewel until then. :/ 15:32:50 this has to be careful as upgrades will be fun fun 15:32:59 and I'm going to try it with mitaka. 15:33:04 yeah. 15:33:08 zhubingbing proposed openstack/kolla: implement Ansible rally role https://review.openstack.org/363456 15:33:35 the upgrades were'nt too bad. there's only one big change I knew of, which was which user it was running under, so a recursive chown. 15:33:51 I've heard good things about ceph upgrades, but we're jumping 2 releases 15:33:53 it use to be run as root. now its not. 15:34:07 inc0: hammer is an lts, and so is jewel. they have an upgrade path for that. 15:34:16 ok 15:35:33 vbel, http://paste.openstack.org/show/567195/ 15:35:38 can you help me? 15:35:39 :) 15:35:41 if we can squeeze that in in rc1- we should 15:35:50 (jewel) 15:36:04 matrohon: I can run the same on centos7 host 15:36:16 sdake: that would be awesome. :) 15:36:25 vbel : with the same error? 15:36:30 inc0, ya jewel would be nice 15:36:41 sdake: I got a bunch of gate tests up for kolla-kubernetes 15:37:04 kfox1111, you mentioned patches? 15:37:12 plus have whitelisted a bunch of existing issues. so we have a good list of todo's to clean up the codebase. 15:37:14 matrohon: no, I can "sudo -u nova bash" 15:37:15 inc0: yeah. sec. 15:37:33 https://review.openstack.org/#/c/365741 15:37:36 vbel : you run the docker run cmd as root? 15:37:40 matrohon, mliima : do you run any kind of selinux/apparmor? 15:37:50 matrohon: yes 15:37:57 https://review.openstack.org/#/c/365664/ 15:38:10 https://review.openstack.org/#/c/365733/ 15:38:26 https://review.openstack.org/#/c/365805/ 15:38:27 vbel : I mean you use 'docker run -u root' 15:38:33 and https://review.openstack.org/#/c/365836/ 15:38:52 vbel, apparmor is not installed, neither selinux 15:39:11 vbel, me too 15:39:13 :) 15:39:49 the first two patches are trivial. 15:39:56 the rest are test cases and a few fixes. 15:40:15 once the last 3 are through, I plan on going through and fixing the issues. 15:40:23 matrohon, I run as root, root@cent7 ~]# docker run -ti 1.2.3.4:5000/kolla/centos-binary-nova-libvirt:3.0.0 bash 15:40:47 and i get into container as root with that 15:42:21 vbel : intersting.. once you are in the container bash session, whoami tells you you are root? 15:42:26 matrohon: but that's for nova-libvirt, nova-compute lands me as 'nova' 15:42:31 vbel : whoami telles me I'm nova 15:43:09 each container has a default user it runs as. some of them run as nova. some as root 15:43:12 vbel : at the end of the dockerfile the is : USER[nova] 15:43:52 matrohon: yes, that should be, at least for nova-compute. if you run 'id' what you get? 15:44:41 kfox1111, yep but an unexpected sudo command seems to be triggered in the container 15:44:52 vbel : $ id 15:44:52 uid=162(nova) gid=162(nova) groups=162(nova),99(nobody),107(qemu),1000(kolla) 15:45:28 vbel :$ sudo -u nova ls 15:45:28 Sorry, user nova is not allowed to execute '/usr/bin/ls' as nova on c0780d04eb3a. 15:45:29 matrohon: vbel: it seems more like a docker uid mapping issue 15:45:43 matrohon: how did you install / configure docker 15:46:01 pbourke_, shouldn't it deactivated by default? 15:46:06 yeah. I don't think nova has sudo rights. 15:46:18 it usually uses rootwrap to elevate priv when needed. 15:46:21 pbourke_, apt-get install docker.io 15:46:36 docker version ? 15:46:59 kfox1111, except for sudo cmd from kolla scripts 15:47:07 pbourke_, 1.11.2 15:47:25 mliima: you probably have similar issue - it is docker related, may be old docker-py? If you try to upgrade: sudo pip install -U docker-py 15:47:36 ok 15:47:39 i'll try 15:47:41 thanks 15:49:53 mliima, matrohon : did you follow everything in http://docs.openstack.org/developer/kolla/quickstart.html#host-machine-requirements ? 15:49:56 vbel, kfox1111, pbourke_ tehr is a command in kolla_start that I can't run as a nova user : 15:49:59 sudo -E more /usr/local/bin/kolla_set_configs 15:50:54 vbel : gtg, you gave me new ideas to debug the situation 15:51:03 vbel, thanks 15:51:27 matrohon: you are welcome, have to go to 15:51:27 matrohon: I haven't come across someone using debian as a host OS for kolla so far. If you get really stuck Id suggest doing up a really simple dockerfile to make sure things are working at a fundamental level 15:52:03 matrohon: there are notes in the quickstart about issues with ubuntu and older kernels with certain backends, perhaps cross reference with that also (http://docs.openstack.org/developer/kolla/quickstart.html#install-dependencies) 15:53:20 Michal Jastrzebski (inc0) proposed openstack/kolla: Change ceph version to Jewel https://review.openstack.org/366230 15:54:52 kfox1111, ^ take a look plz 15:55:29 morning guys 15:55:51 hey 15:56:13 inc0: looks pretty good. 15:56:28 just need a migration job of some sort I think. 15:56:37 it has it 15:56:51 there are no notions of upgrade issues in ceph docs 15:57:26 hmm... there should be one... 15:57:30 http://ceph.com/releases/v10-2-0-jewel-released/ 15:57:45 under hammer, general, bullet 2. 15:57:50 let me see.... 15:58:30 here we go. that page, lower 'UPGRADING FROM HAMMER' 15:59:14 I guess we could use route 1 and add 'setuser match path' and not secure it more... 15:59:29 hmm, I wonder how this translates to container user namespace 15:59:31 might be good enough for newton, then we can secure better in o. 16:00:19 vbel, same error 16:00:21 ... 16:02:33 kfox1111, yeah, I'll do match path thing 16:02:43 and submit bug for drop root for ceph 16:02:48 k 16:04:59 I'm wondering, ceph will add it's user on startup? 16:05:11 no. its part of the rpm install script. 16:05:26 rpm -q --scripts ceph 16:05:30 so durin build 16:05:34 yeah. 16:05:35 should be ok then 16:07:23 inc0: kfox1111: we create the ceph user 16:07:25 via useradd 16:08:24 pbourke: the ceph user wasn't needed until jewel. why would we be createing it for hammer? 16:08:55 kfox1111: because we made a project wide change to not run as root in containers 16:09:20 which doesnt' affect ceph 16:09:28 it seems we created user for no reaosn whatsoever 16:09:37 :D 16:09:46 and this particular line breaks on build 16:09:49 yay 16:17:11 Michal Jastrzebski (inc0) proposed openstack/kolla: Change ceph version to Jewel https://review.openstack.org/366230 16:18:59 Michal Jastrzebski (inc0) proposed openstack/kolla: Change ceph version to Jewel https://review.openstack.org/366230 16:20:27 so, im getting an error about it unable to find the docker module for ansible 16:20:33 both docker and ansible are installed ok 16:20:55 jrich523, can you please paste the error somewhere? 16:21:02 I have theory what might be wrong 16:21:11 pbourke: oh, interesting. so the data might already be written under the correct user? 16:21:27 http://paste.openstack.org/show/567207/ 16:21:28 kfox1111, we didnt run ceph as ceph uset 16:21:31 just created it 16:21:51 jrich523, did you install docker-py ? 16:22:03 on every host? 16:22:05 i think i did... but.. its been a long weekend 16:22:10 it was part of the original vm template 16:22:23 it has to be installed on target hosts too 16:22:27 let me give that a spin tho, you might b e right 16:22:47 yeah, its all VMWorkstation hosted, so i created a vm template of a basic node and am doing linked clones 16:23:20 ok, make sure that python knows how to "import docker" 16:24:25 i think that was it 16:24:52 lets see how the deploy goes this time :) 16:28:53 ahh success... thanks inc0 16:29:21 yay 16:36:21 i can finally create VMs 16:36:30 that only took, what, 2 or 3 weeks lol 16:36:37 jrich523, one woot for kolla please 16:36:48 woot woot :) 16:37:09 thank you, your extra woot is appreciated;) 16:37:19 that second one goes to you guys ;) 16:37:33 cuz there is no way i could have done it without the help 16:38:20 jrich523, mind checking up our docs for what is lacking there? 16:38:35 i've been taking notes :) 16:38:35 you *should* be able to do it without our help 16:39:06 Mauricio Lima proposed openstack/kolla: Fix manila_share container isn't up after deploy https://review.openstack.org/366247 16:39:09 honestly i think half the problem is i suffer from an inability to read long paragraphs :) 16:39:18 however there are a few notes i have on ways to improve it 16:39:28 i'll probably rebuild one more time just to verify all of it 16:39:33 jrich523, to our defense, we now have playbook that will automate most of required steps 16:39:34 and then toss in my 2 cents with a PR 16:39:49 so i came to the party a little too soon? :) 16:40:11 someone came in here last week with a playbook that was doing that, it was fairly impressive 16:40:25 yeah, Sean's play is great 16:40:35 tested on 130 nodes already 16:41:02 i think it was some random guy on his own, but either way, most of the steps remaining are fairly basic, so it should be fairly easy to automate the remaining parts 16:42:14 there really should be a date on the docs pages to let you know when it was last updated, thats one thing i prefer about a wiki, you can easily (on the page) look at the changes, but the doc repo makes perfect sense too :) 16:44:22 oh, the only real problem i had with the docs this last go around, was that the 'install python clients' should be before installing ansible, because one of the libraries in that section (i for get which) was a dependency for ansible, that caused a weird error that took me a while to resolve 16:58:54 jrich523, you sure? I don't think ansible requires clients 16:59:07 it was the libaries 16:59:08 you should be able to deploy without any client 16:59:16 there are 3 listed in that section 16:59:27 yum install python-devel libffi-devel openssl-devel gcc 16:59:29 one of those 16:59:35 ahh these 16:59:35 yeah 16:59:43 so that threw me for a loop a bit 16:59:58 these can be problems every now and then, not much we can do about it 17:00:29 well thats easy to do something about, cut that section, paste it higher :) 17:00:36 lol, like i said, going to go through it one more time 17:00:46 jrich523 thos elibraries are for developing not for instlaling purposes 17:01:02 hmm well, ansible wanted one of them 17:01:04 * jrich523 shrugs 17:01:09 i used a minimal centos install 17:21:16 Mauricio Lima proposed openstack/kolla: Fix manila_share container isn't up after deploy https://review.openstack.org/366247 17:31:51 http://mirror.centos.org/centos/7/storage/x86_64/ oh the joy, centos doesn't have jewel ceph 17:31:58 kfox1111, ^ 17:32:20 sdake, ^ 17:39:16 they have it inc0 17:39:24 but its at http://buildlogs.centos.org/centos/7/storage/x86_64/ceph-jewel/ 17:39:34 no jewel release file 17:40:37 thanks 17:46:55 can someone help me here http://paste.openstack.org/show/567195/ ? 17:48:03 mliima, try to add become: True 17:48:14 bmm 17:48:17 ok inc0 :) 17:53:32 inc:really? 17:53:56 mnaser pasted repo where it actually exists 17:54:05 now I'm trying to figure out how to use it 17:54:06 oh. yeah. ok. :) 17:54:11 I'm a apt guy:( 17:55:12 same error inc0 17:55:13 nAPIError: 500 Server Error: Internal Server Error (\"{\"message\":\"mkdir /sys/kernel/config: operation not permitted\"}\")\\n'"} 17:55:15 :/ 17:55:58 inc0: sed 's/apt-get/yum/' :) 17:56:19 mliima, how about adding privileged to start_container task? 17:56:34 kfox1111, if only it would be so simple 17:56:36 sdake, around? 17:56:40 inc0: I know, right? :) 17:56:41 root inc0 17:57:08 why one should call it 'info' and the other 'show' is beyond me. 17:57:26 and worse, not alias each others. 17:57:31 sdake, how to use http://buildlogs.centos.org/centos/7/storage/x86_64/ceph-jewel/ instead of mirror.centos? 17:58:04 http://docs.openstack.org/developer/kolla/cinder-guide.html this guide don't have many details 17:58:54 mliima, sbezverk you guys, talk;) 17:58:57 sbezverk wrote the thing 17:59:11 inc0: what is up? 17:59:21 :) 17:59:30 hey sbezverk 17:59:33 mliima has problems with iscsi 17:59:39 yes :( 17:59:50 i have this error now 17:59:50 http://paste.openstack.org/show/567195/ 17:59:53 sbezverk, ^^ 18:00:32 btw i'm using ubuntu 16.04 18:01:19 mliima: are you on 16.04 or 14.04? 18:01:32 16.04 18:01:34 mlima: those steps were tested only on 14.04 18:01:40 oh god 18:01:50 ok, i'm testing it now haha 18:01:58 mliima: at that time 16.04 has not been around sorry 18:02:11 ok, its fine 18:02:28 i can change to 14.04 18:02:34 :) 18:02:49 while your doing that, you could change it to centos7 too. ;) 18:03:20 :) 18:04:58 inc0, i want to test some implementation of third party plugin 18:05:00 :) 18:06:09 afk meting tm 18:06:15 atm that is 18:06:22 do you know if someone already done something along those lines? inc0 18:07:03 mliima, we have our brand new customizations 18:07:55 yes yes, i know 18:08:13 :) 18:14:17 Kevin Fox proposed openstack/kolla-kubernetes: l3/metadata DaemonSets + DVR, & labels https://review.openstack.org/357557 18:18:29 whoah...kolla-k8s will run DVRs? fun fun 18:19:01 yeah. we run dvr at our site, so this should let us use it. :) 18:19:21 what scale kfox1111 ? 18:19:43 few hundred nodes so far. 18:19:51 more are coming in each month. 18:20:05 cool, I've heard of scalling issues of OVS 18:20:11 kfox1111: those are using kolla or openstack in general? 18:20:11 and I was wondering if you experience these 18:20:35 mark-casey: hand deplyed rdo for now. I'm working on getting kolla-kubernetes up to the point where I can migrate to it. :) 18:20:53 inc0: none really so far. 18:21:04 had a lot more problems with ceilometer. :/ 18:21:13 ahh the ceilometer 18:21:24 can't get that thing to scale/be stable to save my life. :/ 18:21:32 I remember silence when I asked "who runs ceilometer on prod" in Vancouver 18:21:38 kfox1111: neat :) 18:21:48 which is a sad statement... "dvr's easier then ceilometer...." :/ 18:22:14 some important people here bet some farms on ceilometer... :/ 18:22:24 we warned them... 18:24:28 meeting next... bbiab 19:04:39 lol ceilometer in prod 19:04:44 good one :-P 19:12:03 ahh the Openstack jokes, all the laughts 19:17:45 mnaser we dont claim non-compute kit projects work well at scale 19:17:54 but we includethem anyway so experts like yourseelf can make he judgement call 19:18:18 it is disabled by default 19:18:27 so no harm having in repo 19:19:27 mnaser we do claim we have tested kolla including ceph + compute kit + elasticsearch at 123 node scale 19:19:31 and it works fantasitcally well 19:20:06 100 compute 20 storagee 3 control 19:20:59 sdake, this is about ceilometer not kolla 19:21:08 some insider joke of people who actually tried 19:21:18 oh i thought it was ceilometer on kolla :) 19:21:22 inside joke* 19:24:51 hey guys - time to rewrite kolla in go! 19:25:10 can someone review this? https://review.openstack.org/#/c/358305/ 19:42:13 gotta jet - drive home ftl 19:51:26 back 20:00:57 Waldemar Znoinski proposed openstack/kolla: Non-default database port documentation. https://review.openstack.org/366315 20:06:21 kfox1111: ping 20:12:23 ping 20:12:53 I may have to duck back out in a few minutes. 20:13:04 kfox1111: which would be most relaible way from your point of you to get compute's mode management ip? 20:13:14 I meant node 20:16:34 guys, I need centos help 20:17:21 http://buildlogs.centos.org/centos/7/storage/x86_64/ceph-jewel/ <- how to use this as repo? 20:17:47 get to ipmi? 20:18:19 kfox1111, I think he meants IP in mgmt interface 20:18:31 not oob 20:19:44 kfox1111: management 20:20:14 since it would be used for intercompute node connectivity 20:25:50 hmm.... still trying to get a clear picture of your question. 20:26:09 what software A is trying to talk to software B? 20:26:28 ssh? ipmi? nova -> neutron? 20:27:20 oh... so, 'keystone admin port?' kinds of things? 20:27:30 kfox1111: if you read my reply to your comment re:cinder, it appears we need to customize cinder.conf at run time 20:28:01 for each compute node hosting cinder-volume 20:28:36 sbezverk: hmm... let me think about that for a little bit. 20:28:37 kfox1111: so each compute node will know about backends hosted by itself 20:28:46 this kind of thing hasn't been done with kubernetes yet... 20:28:52 kfox1111: it is what cinder folks told me 20:28:55 at least not without doing it as just pods. 20:29:07 the tricky bit comes when you throw daemonsets into the mix. 20:29:17 I have working prototype, all I need 20:29:34 most of the time, I see something like a single pod/deployment/rc per pet like thing. 20:29:35 from you before I push for review is the way to get ip address 20:29:40 then a configmap per thingy too. 20:29:59 are you serious configmap per comeput node? 20:30:04 Hello 20:30:05 it is management nightmare 20:30:08 sbezverk: it makes me very nervious to be doing config inside the pods other then what's strictly nessisary for stuff like dvr or getting the ip for neutron. 20:30:12 trying to get the vagrant setup to work 20:30:25 overriding an ip is not really config. 20:30:27 but having trouble with this line in particular 20:30:38 writing out a whole configfile at runtime is. 20:30:48 Steven Dake proposed openstack/kolla: DNM: Turn off bindeps https://review.openstack.org/366328 20:30:49 https://github.com/openstack/kolla/blob/master/dev/vagrant/bootstrap.sh#L171 20:31:15 seems like the intention is to install kolla into the KOLLA_PATH directory but that command seems off to me 20:31:46 inc0 thats easy dude 20:32:18 kfox1111: At this point I am going to go with building backend config on the fly, since no other solution exist 20:32:18 so, how does kolla solve this? 20:32:24 ok.. 20:32:35 how does kolla have a different config per lvm cinder-volume? 20:32:47 or is that unsupported? 20:33:08 kfox1111: they do not, they have all backends in the same file which is not recommended way 20:33:10 frankly, I don't know if any op would ever run more then one lvm cinder volume node anyway. 20:33:24 wow wwe have a foxtrot ton of containers 20:33:26 sdake, this is repo with jewel packages https://download.ceph.com/rpm-jewel/el7/x86_64/ , this is one we use http://mirror.centos.org/centos/7/storage/x86_64/ 20:33:29 and a slew of ansible to launch it all 20:33:47 inc0 hang tight dude 20:34:19 so the current kolla version generates one config with all the lvm nodes listed in it? 20:34:19 kfox1111: and nobody tested it, I mean multi backends in kolla 20:34:40 inc0 https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L88 20:34:45 change hammer to jewel 20:34:59 sdake, look at second link 20:35:06 no jewel there m8 20:35:28 looks like storage sig slacking 20:35:33 sbezverk: k, so it probably doesn't work there then. 20:35:53 hmm... 20:35:53 kfox1111: one last thing did iproute make to the base or it is still in openstack-base? 20:36:05 it should be in base I think. 20:36:10 yeah sdake, so how do I add a repo?;) 20:36:37 kfox1111: cool so then "ip a" shold be avaialble everywhere 20:36:54 * kfox1111 looks through the ansible cinder.conf code... 20:37:54 inc0 http://docs.ceph.com/docs/master/install/get-packages/#add-ceph 20:39:28 sbezverk: I haven't ever tried a multinode lvm storage cloud (its a pain to manage and risky or costly) but I think the cinder conf as is would work? 20:39:42 the hostnames are different, so I think you can have the same flavor with different hostnames and it will keep them seperate. 20:41:42 sdake, do I need to create repo file manually? 20:41:50 inc0 ust install that package 20:43:06 http://paste.openstack.org/show/567207/ 20:43:06 sbezverk: it looks like your tring to make it so you can override all of the backends in the cinder.conf. but do we need to do that? we pass in a pregenerated cinder.conf. unless the lvm-1 backend info needs to be different on each host, can't we just use the kolla one? 20:43:09 sdake, ^ 20:43:45 kfox1111: nope because each backend requires unique ip address 20:43:48 I think the only crudini thing we might need is the iscsi_ip_address override 20:43:59 yeah. 20:44:05 same as nova/neutron. 20:44:08 inc0 related to what 20:44:19 but I don't think we need specific overrides for any of the rest? 20:44:21 also forcing people to use the same VG name is limiting 20:44:29 I did docker run -it centos:7 bash 20:44:32 try to install ceph 20:44:36 packages 20:44:38 with jewel 20:44:41 and fail:( 20:45:18 inc0: need python-docker? 20:45:36 kfox1111, nah, I'm trying to get jewel installed in container 20:45:39 inc0 search that link for "RPM" 20:45:42 but rpm fails at repo 20:45:42 sbezverk: I kind of agree, but still think that should come from configgen somehow.. 20:45:47 from where ceph packages start 20:46:06 you have to add a repo file - they dont make one available 20:46:08 sbezverk: what I'm trying to avoid, is deplying with my premade config file and having some template screw up my carefully crafted config. 20:47:09 kfox1111: when it comes from config gen this part of comde will be easily skipped by not configuring backends in kolla-kubernetes.yml 20:47:13 easy :-) 20:47:35 sbezverk: yeah, but I do want to deploy cinder-volume... 20:47:43 for me to get this back and running quickly is very optimal way.. 20:47:53 what about a configmap per pod? 20:48:27 kfox1111: who and how it is goinf to be managed, I am not even talking about updating %() 20:48:28 or an override per host in a configmap? 20:49:21 kolla genconfig's what generates our config files. 20:49:44 kfox1111: if you have your config already placed in configmap, then you do not specify anything in kolla-kubernete.yml 20:49:44 I don't like having multiple sources of truth. :/ 20:49:54 then everything will work as before 20:50:39 that part of code works only if you put it in kolla-kubernetes.yml 20:50:49 k.. let me look back through the code with that in mind... 20:51:35 inc0 try this (documented in that link) : su -c 'rpm -Uvh https://download.ceph.com/rpm-jewel/el7/noarch/ceph-release-1-0.el7.noarch.rpm' 20:51:55 so there is a kolla cinder_enabled_backends too.... 20:52:33 ahh I failpasted. I tried to do this, got errors 20:55:45 sbezverk: the default if cinder_backends is not defined is for it to be defined to include rbd-1 and lvm-1 in kolla 20:56:02 if cinder_backed_ceph or enable_cinder_backend_lvm is true. 20:57:23 the kolla-kubernetes template looks to put all backends in all containers too? 20:58:09 oh... i see it chopping only one out I think.... 20:58:46 oh. no, its tacking on lvm to the end of enabled_backends... 20:59:54 this is complicated logic... 21:00:02 hmmm... 21:00:34 kfox1111: before I was preserving exisiting backends and I was adding new ones 21:00:53 now since I go with 1 backend per compute node 21:01:04 then it will make things more simple 21:01:19 I'm looking at version 10. 21:01:28 its appending I think. 21:01:29 I will find ip address of compute node in the list and if it is there I will create group 21:01:44 I have not pushed with new logic 21:02:01 k.... I think we need to think about this some more... 21:02:07 right 10 appends but new will replace and leave only 1 per compute node 21:02:13 sure 21:02:30 let me push the change and we can have another round of discussionas 21:03:13 as an op, I can look in /etc/kolla/* and look at config or just run kolla-kubernetes.py resource-template.... and see the pod description. 21:03:16 k 21:03:36 Michal Jastrzebski (inc0) proposed openstack/kolla: Change ceph version to Jewel https://review.openstack.org/366230 21:04:08 ops are going to naturally look at configfiles or at most configmaps and not so much init containers. 21:04:19 can we switch this over somehow to do stuff in configmaps still? 21:04:53 maybe move the templateted logic to generate the init container script into a configmap template generator instead? 21:05:01 then do a crudini --merge in the init-container? 21:08:05 btw, this is why I mentioned lvm was going to be harder then ceph. :/ cephs very streightforward here. :/ 21:24:18 sbezverk: I may be able to send an architectural diagram of our production cloud... I'm asking around to see if its possible. but it will make your eyes bleed. :/ 21:25:04 I'm trying to do the hostlabeling as flexable as possible as we've needed that kind of flexibility in our service placement and I'm sure other ops have similar strange looking placemnets that are required. 21:30:43 kfox1111: agree this is fundamental thing it would be goo to take time to come up with something flexible and easy undertsandable 21:31:59 can we punt and just support one node of lvm? that should work without anything special I think? (well, maybe except iscsi address override?) 21:32:21 that would buy us time till we can come up with a better solution. 21:32:33 really its the same problem ceph in kolla-kubernetes will have. 22:28:06 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 22:29:58 kfox1111: tested latest cinder PS, for now I would like to go with this solution. 22:32:07 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 22:41:55 sbezverk: reviewing... 22:42:16 can you review https://review.openstack.org/#/c/365733? 23:21:00 Merged openstack/kolla-kubernetes: Template validation/gating https://review.openstack.org/365733 23:27:40 inc0 yo 23:59:47 sdake, i wasnt throws jabs at kolla 23:59:58 as a fact we're moving our entire (pretty big) deployment from our puppet glue to kolla :) 00:00:09 mnaser its all good 00:00:24 so im just as excited to see this go up 00:00:28 mnaser tbh i have never ued ansible 00:00:39 well that partis true 00:00:39 i meant ceilometer 00:00:43 sorry just woke up from a nap :) 00:00:51 it's going to be interesting, we're going to deploy kolla and move service by service by migrating dbs and running db syncs afterwards 00:01:01 np :) but yeah, we're given ceilometer many tries (we've been running openstack since bexar) 00:01:15 wow long time - been in the pain house :) 00:01:29 we'll probably run it with like... 10-30 minute retention time to allow heat autoscaling to work 00:01:33 would love to pick your brain about top 3 pain points of kolla 00:02:04 i will be around more often here and part of us moving to kolla is: it fits with where we want to go, so we'll be part of it 00:02:16 nice 00:02:18 same reason why we ended up in openstack a few years ago 00:02:28 may i ask how many ndoes? 00:02:51 we're deploying it for a private cloud of one of our customers as a pilot which is around 40 nodes 00:03:02 cool thats no problem 00:03:16 it'll be a upgrade to mitaka from liberty and switch to kolla a the same time 00:03:26 that part will be tricky 00:03:29 i plan to fully install and setup kolla then move the dbs over 00:03:39 i'd separate the two operations persoanlly :) 00:03:49 we've done migrations from nova-network to neutron in our public cloud 00:03:58 can you handle vm or dataplane downtimem? 00:03:59 with 1 reboot as a downtime, we've seen some shit :-P 00:04:06 there will be dataplane downtime only 00:04:21 how handling the vm migration? 00:04:34 ceph ephermal = live migration 00:04:46 using external ceph feture then? 00:05:03 we haven't gotten that far yet, i WOULD like us to do the kolla one 00:05:12 we do converged compute/storage 00:05:31 that shouldbe fine as long asyou don't have ton of overcommit 00:05:39 but ymmv 00:05:47 ew have not done a greenfield migration in kolla as of yet 00:05:51 rather brownfield 00:06:00 rather time to go back to bed ;-) 00:06:05 we've planned for that 00:06:14 we overprovision memory and reserve cpu cores for ceph 00:06:42 have a good night! 00:09:23 Steven Dake proposed openstack/kolla: DNM: Turn off bindep https://review.openstack.org/366328 00:10:21 night mnaser 00:18:42 Steven Dake proposed openstack/kolla: DNM: Turn off bindep https://review.openstack.org/366328 00:18:54 so close - no matter how far 00:19:00 couldn't be much more from the heart 00:19:09 forever trust in who we are 00:23:32 Steven Dake proposed openstack/kolla: DNM: Turn off bindep https://review.openstack.org/366328 00:40:49 sdake: and nothing else matters. :) 00:41:02 kfox1111 you finally caught up;) 00:41:16 * kfox1111 chuckles 01:06:04 Hui Kang proposed openstack/kolla: Add etcd ansible role https://review.openstack.org/364662 01:08:05 Steven Dake proposed openstack/kolla: TEST OF MASTER GATE https://review.openstack.org/366430 01:08:07 Merged openstack/kolla: Update destroy include images arg to LONGOPT https://review.openstack.org/364983 01:08:55 Merged openstack/kolla: Comment some lines in globals.yml https://review.openstack.org/366083 01:22:38 morning 01:45:55 Kevin Fox proposed openstack/kolla-kubernetes: Reverse arg order on resource and resource-template https://review.openstack.org/366437 01:51:05 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 01:52:40 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 01:59:59 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 02:07:42 zhubingbing proposed openstack/kolla: Implement Ansible rally role https://review.openstack.org/363456 02:08:15 ping Mathias 02:08:45 sdake: are you there? 02:08:56 as always 02:10:23 for the gate setup: should we call bootstrap-servers before prechecks and deploy? 02:10:43 due to user setup,... is in bootstrap phase 02:10:55 we probably should 02:11:15 the code that is there was pre-bootstrap-seervers 02:12:12 sdake: you mean we should place something like pre-bootstrap-servers? 02:12:24 put it in setup_gate.sh 02:12:32 sorry pre bootstrap-servers implementation 02:12:33 but why pre-bootstrap? 02:12:55 that gate code that does bootstrapping was written prior to us having bootstrapping 02:13:50 got your point, sdake 02:16:48 good evening 02:17:01 morning inc0 02:20:20 duonghq, interesting, I just learned that in Polish we actually call China "a land of the center" which seems to be literal name in Chineese right? 02:20:40 duonghq arent you japanese? :) 02:20:42 I don't think there is equivalent phrase in English tho 02:21:23 duonghq, if you're Japanese, then I apologize 02:23:58 inc0, sdake: I'm Vietnamese :) 02:24:09 we both fail 02:24:40 Jeffrey4l ping re your patch to rework the gating - i don't really like it 02:24:48 i think your after getting mirroring working in build.py 02:24:50 which is good 02:25:00 but i am not keen to get rid of available.d dirctory 02:25:38 duonghq, yeah, I should recognize it, I'm terribly sorry 02:25:43 I have never image that my name is like Japanese one 02:25:43 Jeffrey4l: alive? 02:25:48 inc0: np 02:26:23 duonghq: 日本人？ 02:26:44 sdake: are you around mate? 02:27:00 mtaylor22, what's up? maybe I can help 02:27:12 one moment, just getting a link 02:28:12 wound me 02:28:26 lol. 02:28:33 https://bugs.launchpad.net/kolla/+bug/1617902 -- can this be backported to mitaka? 02:28:33 Launchpad bug 1617902 in kolla liberty "nova resize not working with KVM" [High,Triaged] 02:29:02 mtaylor22 its targeted as such 02:29:16 mtaylor22, let me take a look at code, if it won't break anything 02:29:28 ahh ok, was just curious because it has liberty/newton listed 02:29:30 inc0 that ship has sailed already been backported to mitaka 02:29:32 I'm not sure if we had ssh container back in Mitaka 02:29:45 thanks sdake :) 02:29:51 well I guess we had 02:30:39 mtaylor22: must put in google translate, haha 02:30:39 mtaylor22 all yours :) feel ffree to do the backport 02:30:49 yeah we had 02:31:00 so there shouldn't be problem 02:31:39 inc0 i already looked at tht before it ws proposed as a backport 02:31:46 sooo....rpm don't know how to install ceph deps 02:31:49 how fun 02:32:57 thanks! 02:33:07 ill see if i get a chance to poke around with this 02:33:43 just need to poke Jeffrey4l in regards to https://bugs.launchpad.net/kolla/+bug/1616268 02:33:43 Launchpad bug 1616268 in kolla mitaka "Stale namespace removal causing "RTNETLINK answers: Invalid argument" errors" [Critical,Confirmed] 02:33:47 from what Ross and I had found so far 02:34:43 mtaylor22, so docker doesn't clean after itself? 02:36:59 sdake, omfg...so remember how I sweared about centos people for not adding jewel repos? 02:37:16 they just got added? 02:37:18 http://mirror.centos.org/centos/7/storage/x86_64/ look what created couple hrs ago 02:37:31 like magic irc works ;) 02:37:48 I guess somebody was watching, or I'm just lucky 02:37:55 regardless, let's try if that works 02:38:28 hmm pretty much. it seems like the docker containers are locking, or holding on to the namespaces 02:38:42 we're able to remove them outside of docker 02:38:48 well, once the containers are stopped 02:39:06 ill asses Jeffrey4l's question later today (had no chance to touch since he asked) 02:40:18 oh 02:40:33 what's a decent irc client for osx? currently using limechat.. ? 02:42:59 mtaylor22, I use xchat 02:43:09 xchat azure I think is full name 02:43:46 ah interesting, thanks. 02:46:51 Steven Dake proposed openstack/kolla: Turn off bindep https://review.openstack.org/366328 02:47:14 anybody success with kubernetes's kube-deploy? 02:51:47 duonghq, I had success with kargo - ansible deploy of k8s 02:54:07 inc0: tyvm 02:55:43 Kevin Fox proposed openstack/kolla-kubernetes: Cleanup patch https://review.openstack.org/366450 02:56:34 soo, good news and bad news - good news is that this repo works and ceph installs well 02:56:53 bad news is that cinder can't build and I have no idea why 02:57:51 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 02:58:42 http://logs.openstack.org/30/366230/4/check/gate-kolla-dsvm-build-centos-binary-centos-7-nv/45d731c/console.html#_2016-09-06_21_19_26_712799 any centos experts around? 02:58:47 mtaylor22 textual is best 02:59:01 mtaylor22 it costs 5 bucks however 02:59:15 inc0 did you change the version pins 02:59:22 eh thats fine 02:59:26 looks god tho 02:59:29 good even 02:59:45 sdake, where are they? 02:59:49 STEP THROUGH THAT GATE 02:59:54 and your in a zero G environment 03:00:02 inc0 base image iirc 03:00:19 I used to use https://www.bitlbee.org/ with limechat for simplicity back in the day 03:00:40 https://github.com/openstack/kolla/blob/master/docker/base/versionlock.list 03:00:45 brb 03:01:50 inc0 fwiw i think its probably safe to remove version pins at this point 03:02:00 i'm not sure why they were added in the fifrst place 03:02:17 i think sam was concerned the ceph nerds would release non-lts stuf in the hammer repo 03:03:15 inc0: has you tried kargo with pre-download binary (etc, flanneld,...)? I do not see any related-settings in kargo.yml 03:04:09 hey guys - i'm starting a new project 03:04:31 its called "serveless stackless container openstack" 03:04:58 sdake, you want to run openstack without servers? like, put it on cloud? 03:05:11 and install stackless python inside? 03:05:58 ya calling it serverless tupperware 03:06:49 "there is too much at take to let your unfiltered thoughts to leave this facility." 03:06:54 take/stake 03:06:56 wtb new keyboard 03:07:52 stackless tuppervaporware 03:08:20 and its all based on kubernetes and docker and rancher and coreos 03:08:37 I know 03:08:39 and atomic and ansible and openstack 03:08:57 let's put coreos inside docker and run it on k8s which runs on...you guessed right! coreos 03:09:00 basically its a super mashup of evereything antastic 03:09:10 sdake: hah. I think there's real money there. ;) 03:09:16 if you forget ansible, then you have dinnerless sdake 03:09:22 thanks kfox1111 ! 03:09:27 :p 03:09:36 let's put rkt into docker and laugh the evil laghter 03:09:38 kfox1111 with your validation I am pinging my network now to find funding :) 03:10:07 * kfox1111 chuckles 03:10:20 find the right venture capitalist, and your in. :) 03:10:41 if only i was good friends with people that are now vcs ;) 03:10:52 lol 03:11:19 hipstack 03:11:25 ok I have no idea why do we even have these pins 03:11:34 in fact I think this is horrible 03:11:38 inc0 me either dude me either 03:11:48 since security patches will be hard to keep track of and so on 03:12:25 so how about versionlock serveless ansiblized dockerized containerized hyperkubed square rooted 03:12:38 answer = 2 03:12:59 sdake: i might say no on that trademark abuse 03:13:11 docker is a verb 03:13:16 those cats fucked up their trademark alrady 03:13:26 i was talking about ansible 03:13:41 the verb is cattle-ized 03:13:50 fun fact, we had ceph pinned in centos but not ubunut 03:13:51 so you know :p 03:14:01 kolla trivia vol 1 03:14:02 i dont think ubuntu has version pins 03:14:09 it can 03:14:44 anyway, I hereby remove version pins from centos, hate me all you wish 03:14:51 zhubingbing proposed openstack/kolla: Add cloudkitty ansible role https://review.openstack.org/349305 03:14:52 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 03:15:16 make sure to remove th versionlock plugin plz 03:15:46 hey folks - i optimized the gate run time by 3 minutes 03:15:51 hi 03:15:52 victory :) 03:15:53 snake 03:15:54 by plugin you mean package? 03:16:01 snake 03:16:08 inc0 bingo 03:16:09 zhubingbing steak dude steak 03:16:17 sdake, 03:16:17 i dont eat snake or dinner 03:16:21 -- 03:17:01 can you help me review barbican role and contaier 03:17:16 your wish is my command 03:17:21 have a link 03:17:40 ok 03:17:52 Kevin Fox proposed openstack/kolla-kubernetes: Add namespace and test https://review.openstack.org/365836 03:18:08 https://review.openstack.org/#/c/351822/ 03:18:29 https://review.openstack.org/#/c/352515/ 03:20:24 Michal Jastrzebski (inc0) proposed openstack/kolla: Change ceph version to Jewel https://review.openstack.org/366230 03:20:25 zhubingbing where is the release note in tehe first patch? 03:20:34 my patch set dependencies are starting to get pretty deep. if there are any kolla-kubernetes reviewers available to help, I'd appreciate it. :) 03:21:05 barbican container ? 03:21:29 sorry i should add release note 03:21:46 barbican role have release note 03:21:47 zhubingbing - need to wait for containe to merge so we can recheck barbican 03:21:55 zhubingbing don't sweat it 03:22:00 ok 03:22:02 the barbican role that is 03:22:25 i add it for barbican container 03:22:30 1 min 03:22:38 its already in the merge queue 03:22:39 dont worry about it 03:22:41 inc0: oh, I think there was some weirdnes with ceph back in the hammer days on redhat. that might be why they were pinned. 03:22:45 Michal Jastrzebski (inc0) proposed openstack/kolla: Change ceph version to Jewel https://review.openstack.org/366230 03:23:07 I think it was fixed in infernalis. 03:23:09 wow 03:23:17 ok thanks 03:23:56 kfox1111, entirely possible 03:24:13 btw we have an excuse why we waited so long with ceph bump 03:24:30 kfox1111, look at date modified (or added rather) http://mirror.centos.org/centos/7/storage/x86_64/ 03:25:00 inc0: looking good. 03:25:03 tomorrow? :) 03:25:08 Merged openstack/kolla: Add Barbican container dockerfile https://review.openstack.org/351822 03:25:23 this is what I call good timing 03:25:34 I'd guess that is continuously rebuilt. 03:25:55 or they just happend to cut a release at a perfect time. :) 03:25:59 kfox1111, no, it wasn't there today morning 03:26:06 oh. then great timeing. :) 03:26:13 yup 03:28:47 sdake, we don't have good bp to pin ceph to 03:29:08 inc0: can I undeploy the Kargo deploy cluster? 03:29:09 inc0 make one? 03:29:19 duonghq, no idea 03:29:28 sdake, I can, sure, do we need it tho? 03:29:38 inc0 ya we need it for ugrades 03:29:43 and release note might help too 03:29:49 release note is there 03:29:55 oh i missed that 03:30:06 Kevin Fox proposed openstack/kolla-kubernetes: Add namespace and test https://review.openstack.org/365836 03:30:07 i thought i reviewed the whoel thing 03:33:22 sdake_: you know whats realy going to be fun? 03:33:42 extending the kubernetes cluster your using to deploy openstack, 03:33:46 using that openstack's vm's. :) 03:35:15 inc0 is there an upgrade role work needing work here? 03:35:21 inc0 if so I'd say so 03:35:29 ya i looked at review saw release note 03:35:31 Michal Jastrzebski (inc0) proposed openstack/kolla: Change ceph version to Jewel https://review.openstack.org/366230 03:35:43 mtaylor22, ping 03:36:31 mtaylor22, do u try reproduce this issue after the dumb-init is merged? 03:36:35 I've already tried a hybrid kube with both bare metal and vms in openstack. it works well. :) 03:37:04 ok I'm off, have a good day/night/afternoon guys 03:37:08 mtaylor22, i think it may be the root cause. 03:37:20 good night :) 03:37:57 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 03:38:59 sdake_, about this https://review.openstack.org/365721 03:39:20 where is more detail about mirrors.availble.d? 03:39:28 i have no idea about this folder. 03:39:29 sdake_, ^ 03:39:37 Jeffrey4l i'm sure its documented omehre in infra manuals 03:40:03 Jeffrey4l that is the cp command 03:40:11 it copies from that directroy to the local repo locations 03:40:29 what's wrong with nodepool/provider? 03:40:36 ok. 03:40:53 Jeffrey4l personally i find that sed hard to read 03:41:03 cp from mirrors location is easier 03:41:17 and cp from mirrors location is guaranteed to be correct 03:41:22 because other people use that 03:41:29 that is the standard wayt o include mirrors in openstack 03:41:52 sdake_, but 1. it is in the docker file. in current customize implementation, it is hard to use COPY 03:42:22 2. we are using one source.lists file which container all source url. 03:42:52 but it is already working 03:42:52 hmm what happen if i add a two different url for ceph? ( no idea about this) 03:42:57 Jeffrey4l what problem are ou tyring to solve 03:43:39 the PS? url the ci provider's source mirror to build the docker image. 03:44:15 i thoguth youwere also trying to add external mirroring 03:44:29 maybe i read into the patch too much 03:44:35 with the change to build.py 03:44:52 no. just changed the setup_gate.sh script. 03:44:57 sdake_, check this https://github.com/openstack-infra/project-config/blob/master/nodepool/scripts/configure_mirror.sh#L165 03:45:21 the sources.list.available.d come from the nodepool/provider too. 03:46:06 yes and is much more understandable then a bunch of magic sed ;) 03:46:29 Jeffrey4l need to lower barrier to working in ci not increase it 03:46:33 in/with 03:47:08 sdake_, hmm means simplify? 03:47:21 Jeffrey4l right on 03:47:45 https://review.openstack.org/#/c/365721/8/kolla/image/build.py 03:47:53 maybe that was an extra artifact? 03:48:32 my main concern is nodepool will somehow change and break our gating coupled with complexity of the gate implementation 03:48:32 sdake_, it is a bug. the customization in Ubuntu not work now. 03:49:06 Jeffrey4l can you break that out separately 03:49:47 the use the ci provided mirror is the best solution right? if so, we have to use the nodepool/provider. the sources.list.available.d come from it, too. 03:49:58 np. I will push another PS> 03:51:02 hey coolsvap_ 03:51:12 sdake_: hi :) 03:51:29 Jeffrey4l yes we want to use mirrors 03:51:35 we are using now with the sources.available.d logic 03:53:24 ya. i will fix this. 03:53:35 try to use sources.avail.d 03:54:00 thx 03:57:31 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 04:00:23 Jeffrey4l: will play with it soon, have a few BAU things to sort out first. :) 04:02:43 ok. thank 04:17:27 Kevin Fox proposed openstack/kolla-kubernetes: Add namespace and test https://review.openstack.org/365836 04:27:06 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 04:32:35 3) To join a non-profit organization. Like Khan Academy, or Google Dot Org, or Yahoo! 04:32:48 lol 04:36:57 time for 300 04:37:04 nothing like 300 at 9:36 pm 04:46:03 can't believe kolla is up to 5k commits 04:51:41 http://img0.joyreactor.com/pics/post/funny-pictures-auto-birds-this-is-sparta-466890.jpeg 05:04:05 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 05:28:03 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 05:33:06 huikang do you need the pid file checking in that patch? 05:33:36 sdake_, any suggestion; still debugging the gate 05:34:11 huikang whts happening with the gate tht is problematic? 05:34:48 no suggestions other then would like a backport asap ;) 05:34:48 checking the existence of the pid file is not sufficient 05:34:59 i know, why even have that 05:35:09 is my quetion 05:35:32 if the cluster is in primary mode, we do not need checking the pid file 05:38:46 however, querying the database in primary mode needs both the socket file and the pid file. sdake_ 05:39:05 roger 05:39:57 if it is needed it is needed 05:40:06 a note to that affect would be helpful 05:40:30 k, will do. Thanks 06:25:26 hello 06:25:29 guys 07:06:47 morning guys 07:10:41 morning berendt 07:26:38 Merged openstack/kolla: Implement Ansible rally role https://review.openstack.org/363456 07:28:23 zhubingbing proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 07:36:13 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 07:38:57 hello 07:40:26 who can help me review it 07:40:30 https://review.openstack.org/#/c/349305/ 07:42:59 Merged openstack/kolla: Add cloudkitty containers for type source https://review.openstack.org/349295 07:59:34 zhubingbing_ have you already changed to gnocchi as backend? 07:59:42 yes 07:59:48 no 07:59:57 i have no change it 08:00:08 you can change it 08:00:22 I do not have the time to work on it at the moment :( 08:05:33 ok 08:05:42 i change it 08:06:21 Hi. Can anyone take a look at https://review.openstack.org/#/c/366082/ please. Cheers :) 08:06:52 groan failing gates 08:08:42 pbourke around? 08:09:01 prithiv I am missing the ceilometer integration of collectd, is this part of an other review? 08:16:54 Takashi Sogabe proposed openstack/kolla: Keepalived leaves VIP on the host in case of accidental removal. https://review.openstack.org/364085 08:17:51 vbel, hi 08:28:07 Jeffrey4l_ ping 08:30:59 good morning all 08:31:33 and evening to the asia/pacific 08:41:41 zhubingbing proposed openstack/kolla: Add cloudkitty ansible role https://review.openstack.org/349305 08:43:18 Steven Dake proposed openstack/kolla: DNM test of gate with 8 threads https://review.openstack.org/366591 08:47:12 sdake: what is approximate configuration of a gate node? 08:47:26 8 gig of ram 8 cores iirc 08:47:32 there are like 10 different clodu provides 08:47:34 providers 08:47:38 so its not clear to me 08:47:48 roger 08:47:52 the problem with 8 threads is as follows 08:48:02 if an imaege is not in the regex, it tries to queue it anyway 08:48:07 so it takes up one of the build threads 08:48:10 for only a few seconds 08:48:22 but that is enoug htime to be half done buildding the next ontainer 08:48:49 matrohon, hi, how are you? 08:48:52 what should probably happen is that bug should be fixed ;) 08:48:56 hey folks 08:49:06 hey wznoinsk 08:49:07 the build order is undeterminable? 08:49:15 vbel, hey! fine and you 08:49:16 good morning/evening 08:49:22 duonghq its ordered 08:49:29 matrohon: thanks, good :) 08:49:32 ah, understand 08:50:15 vbel : I finally figured out what was the root cause of my bug "who are you" when sudoing a cmd in a container 08:51:20 vbel : it happens only if the container is run with the option "--volume=/run:/run:rw" 08:52:23 vbel : we are going to report a bug for futher discussions 08:53:15 matrohon: interesting, looks like something new, yes, it is worth reporting 08:53:21 sdake: which is test deploy file? 08:53:31 duonghq huh? 08:53:49 sdake: the test for kolla-ansible deploy 08:53:52 tox -e cmd 08:53:56 where cmd is in tox.ini 08:54:26 roger 08:54:46 centos gates are pokey because they are not using mirrors 08:56:10 what does "pokey" mean? 08:56:14 zhubingbing proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 08:57:04 duonghq: I guess 'okey' 08:57:15 :) 08:58:23 vbel: thanks 08:58:45 sdake: could you have a look? (nit): https://review.openstack.org/#/c/366597/ 09:00:28 7 short days to go until releases 09:01:44 Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (1) https://review.openstack.org/358539 09:02:35 i am thinking about replacement of tgtd with targetcli (default since rh/centos7 and ubuntu 12.04). Should it be a complete replacement that eliminates tgtd as container everywhere or should we have a choice - what cinder-volume to use. E.g. new one could be called cinder-volume-lio and a user would opt. Since lvm backends are not really for massive deployments one could ask to kill tgtd container on upgrade and reattach disks if necessary 09:05:10 sdake can you see barbican role 09:05:12 vbel lvm is a new backend for us in newton 09:05:17 zhubingbing link? 09:05:37 https://review.openstack.org/#/c/352515/ 09:06:01 sdake: so it is worth to switch to targetcli now? 09:06:16 zhubingbing if gate fails, type recheck like i just did please 09:06:25 vbel totally the wrong time to make that chane 09:06:30 we are t-7 days to release 09:06:32 sure 09:06:45 or atleat release of rc1 09:06:46 sdake: yes, that's what I guessed, the same for other changes 09:06:56 bug fixes only pretty much 09:07:05 unless already marked as a blueprint for rc1 09:07:31 coolsvap_: https://review.openstack.org/#/c/366315/ whenever you have a moment pls 09:07:42 sdake: I see, thanks. So many review won't get in until new release is started 09:08:13 "O" release :) 09:08:34 vbel master will be open for business after rc1 09:08:40 we branch on rc1 09:09:13 but i'd like core team to stay focused on fixing bugs of which we have plenty :) 09:09:21 sdake: ok, great, no much wait time then. Thanks! 09:09:40 we have roughly 45 days to fix all bugs we want to fix 09:09:54 https://releases.openstack.org/newton/schedule.html 09:10:00 note we are cycle trailing 09:13:00 Waldemar Znoinski proposed openstack/kolla: Non-default database port documentation. https://review.openstack.org/366315 09:18:38 morning 09:19:23 im doing daily builds and testing of kolla newton. But kolla upgrade fails constantly on mariadb 09:19:36 is it just for me or are others experiencing this as well? 09:20:11 workaround is to shutdown mariadb and do recovery 09:20:31 bjolo i think huikang may be working on that 09:20:40 bjolo but please file a bug 09:20:53 ok 09:24:18 zhubingbing proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 09:31:29 wznoinsk: thanks for the doc update. 09:31:57 coolsvap_: cheers 09:32:40 Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (1) https://review.openstack.org/358539 09:46:15 look 142 people in channel 09:46:23 that is my age + 100 :) 09:46:27 boy I hope I don't live that long 09:46:30 night all 09:46:39 －－ 10:20:56 sean-k-mooney: ping 12:20:38 Jeffrey Zhang proposed openstack/kolla: Add extra methods in template when using template override https://review.openstack.org/366712 12:27:25 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 12:27:48 Hmm missing something obvious here, built images according to quickstart, but they are v2.0.2, I thought they would be something higher ? 12:33:04 Oh right ok I have run across 'operator workflow vs developer workflow' - my bad 12:50:02 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 13:04:55 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 13:16:41 good morning 13:17:02 hi! 13:19:00 pbourke, 13:19:06 around? 13:19:10 inc0: yo 13:19:36 https://review.openstack.org/#/c/366230/ <- can you validate that all works for you? 13:22:59 Hmm following the developer workflow from the quickstart guide, where does one get kolla-build from ? 13:23:38 Christian Berendt proposed openstack/kolla: Fix keytone options in all.yml/globals.yml https://review.openstack.org/365950 13:23:39 jmccarthy, if you pip install kolla, it will appear in your PATH 13:24:03 inco: Hmm yea the guide warns about pip installing vs use from git (which is what I want methinks) 13:24:20 if you pip install -e . in git repo, same thing:) 13:24:21 inc0: Hmm yea the guide warns about pip installing vs use from git (which is what I want methinks) 13:24:28 python is really simple in these terms 13:24:45 Ah ok I only installed the requirements.txt(s) from the guide .. 13:25:32 also tbh I usually just use tools/build.py 13:25:38 these are equivalent 13:25:54 Ok cool yea I may just do that then :) Thanks ! 13:39:29 Christian Berendt proposed openstack/kolla: Remove fedora from Dockerfile templates https://review.openstack.org/359447 13:42:15 hello guys 13:50:45 Merged openstack/kolla: Turn off bindep https://review.openstack.org/366328 13:54:19 Merged openstack/kolla: Correct Murano keystone_authtoken option name https://review.openstack.org/366019 13:59:00 vbel, BTW, what would be the config that would run the all-in-one config for sure? (os, base os...) 13:59:46 matrohon: centos7 / centos7 13:59:59 it is a default one 14:00:23 vbel : ok, taht's the one we are testing currently 14:00:45 matrohon: default for both building and testing :) 14:00:51 vbel : binary images fetched by the master branch are 2.0.3? 14:01:45 they are 3.0.0 so you need to have your own repo / built images 14:02:06 vbel : ok thanks 14:08:59 pbourke, so about ceph upgrade 14:09:21 issue is, with hammer ceph was running as "root" 14:09:28 in jewel it's running as "ceph" 14:09:48 so all the files in /var/lib/ceph should be rw for user ceph as well 14:10:22 zhubingbing proposed openstack/kolla: Add cloudkitty ansible role https://review.openstack.org/349305 14:12:23 inc0, and selinux is a thing now with ceph jewel centos 14:14:44 fyi 14:14:47 our upgrade is been 14:14:56 chown -R ceph:ceph /var/lib/ceph/bootstrap-* /var/lib/ceph/{mds,mon,tmp} /var/run/ceph /var/log/ceph/ 14:15:05 restorecon -Rv /var/lib/ceph/bootstrap-* /var/lib/ceph/{mds,mon,tmp} /var/run/ceph /var/log/ceph/ 14:15:06 inc0: I think I understand, just the comment in your patch is a bit unclear I think 14:15:21 so you might want to look at fixing those permissions too 14:15:34 the osds we do separately in multiple threads so it goes faster 14:15:49 restorecon might not be necessary as the rpms take care of that 14:15:55 mnaser, for now I just made this workaround https://review.openstack.org/#/c/366230/7/ansible/roles/ceph/templates/ceph.conf.j2 14:17:05 aand also the devices have to be chmod-ed under ceph:ceph 14:17:08 journals mostly 14:28:10 https://bugs.launchpad.net/kolla/+bug/1520728 - I suggest we don't fix this. Ansible cannot use multisection - if the same section appears again older hosts have troubles with facts 14:28:10 Launchpad bug 1520728 in kolla mitaka "fatal: [openstack002] => One or more undefined variables: 'dict object' has no attribute 'stdout'" [Critical,Triaged] 14:30:40 ping inc 14:30:49 wassup? 14:31:20 Michal Jastrzebski (inc0) proposed openstack/kolla: Change ceph version to Jewel https://review.openstack.org/366230 14:42:47 inc0 is it possible to run upgrade of hammer to jewel or does this have an impact on existing ceph clusters? 14:43:05 berendt, it should be safe 14:43:15 however let me test this thoroughly 14:43:57 right now my infra died and can't do anything:/ 14:44:22 ok but change itself looks good, i was not sure if the release note is sufficient 14:44:38 what would you add? 14:44:45 should we also do an upgrade of mariadb? at the moment we use 10.0, latest is 10.1 14:44:59 berendt, if we can squeeze it in 14:45:07 no i think release note is fine when an upgrade of hammer to jewel is no problem 14:45:38 if it isn't we can't release it 14:45:53 i will check mariadb 10.1 the next days 14:46:14 do we have a list of the used versions of infrastructure components in the docs? 14:46:18 cool, these are critical, so we need to be sure upgrades works 14:46:30 no, we don't, we should 14:46:41 berendt, can you post bug for it? 14:46:45 wishlist bug 14:46:48 yes i think so, will open a bug 14:47:15 thank you 14:47:36 agreed. this need well tested. 14:48:18 is there a reason tools/kolla_kubernetes ends in .py? 14:48:22 Jeffrey4l_, can you help with that? I'm strugging with my infra :S 14:48:33 it's written in python? 14:48:50 ;) 14:48:55 * inc0 being smartass 14:49:01 inc0, sure. I can test it. 14:49:58 this ownership change can break things if this workaround won't work 14:50:08 but at the end it gives us option to drop root 14:50:10 evening 14:50:20 good morning duonghq 14:51:13 https://bugs.launchpad.net/kolla/+bug/1621122 14:51:13 Launchpad bug 1621122 in kolla "Document versions of used infrastructure components" [Undecided,New] 14:51:17 i cannot set the importance 14:52:00 there ya go berendt 14:52:14 I made it medium because it is within my power mwhhahahaha 14:52:16 Kevin Fox proposed openstack/kolla-kubernetes: Add namespace and test https://review.openstack.org/365836 14:52:29 inc0 haha 14:55:48 rhallisey: alive? 14:56:05 kfox1111, what's up 14:56:12 what do you got for me :) 14:56:32 k. so, have you seen this one yet? 14:56:47 https://review.openstack.org/#/c/366437/ 14:57:17 I've got a slightly different idea to run by you once you have. 14:58:05 Vladislav Belogrudov proposed openstack/kolla: Cinder-backup misses client part for NFS backend https://review.openstack.org/366037 14:58:45 Bartłomiej Daca proposed openstack/kolla: Correct Murano keystone_authtoken option name https://review.openstack.org/366801 14:59:45 rhallisey: the alternate idea is, rather then rename the resources from openvswitch-vswitchd so that the command shrinks down from: kolla-kubernetes resource-template create pod openvswitch openvswitch-vswitchd to kolla-kubernetes resource-template create pod openvswitch vswitchd, 15:00:19 we requre resource_names to be unique, and then we can skip the service name entirely. 15:00:22 so: 15:00:39 kolla-kubernetes resource-template create pod openvswitch-vswitchd 15:01:01 I like that one a bit better as openvswitch-vswitchd should be the container name too. so it maps up very nicely. 15:01:29 kfox1111, I agree. I didn't like the order either 15:01:52 or the length of the commne 15:01:55 command 15:03:18 think I should rework it so that the cli commands take either service_name or resorce_name instead of both? 15:03:50 If so, I'll add a test too to ensure service_names and resource_names are all unique. 15:06:21 gtg. bbiab. 15:06:47 zhubingbing proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 15:21:44 Merged openstack/kolla: Add cloudkitty ansible role https://review.openstack.org/349305 15:22:22 kfox1111, I agree. I want to revist how the CLI looks 15:22:37 Md Nadeem proposed openstack/kolla: Test: make enforce_type=True in CONF.set_override https://review.openstack.org/366818 15:26:09 morning 15:26:34 hello 15:29:13 zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 15:29:20 moring 15:30:25 back. 15:31:05 rhallisey: you ok with the idea of resource names being unique? I can code something up if you are. 15:32:15 can you please review https://review.openstack.org/#/c/359447/ . i want to merge or abandon it, but i do not want to do daily rebases. 15:33:31 Christian Berendt proposed openstack/kolla: Fix keytone options in all.yml/globals.yml https://review.openstack.org/365950 15:33:38 kfox1111, so another example would be: kolla-kubernetes resource-template create pod nova-api 15:33:41 ^ yes 15:33:52 yeah. exactly. 15:34:01 yea that's way better 15:34:20 k. I'll see what I can do. :) 15:34:25 excellent 15:35:47 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 15:35:50 berendt, why do you want to remove fedora? 15:36:04 inc0 who uses fedora? 15:36:15 it's not that I like fedora, but we need to deprecate it first 15:36:20 i thank we should support 15:36:27 i think 15:36:30 from my POV openstack should not run on top of fedora 15:36:40 I'd be ok with deprecating it and removing it next release 15:36:53 berendt, agree, but if somebody actually deployed it, we can't screw them over 15:37:00 i think the fedora should be remove. no one use it and no one maintain it. 15:37:15 ok but than i will abandon the open review and will open another one to propose the deprecation 15:37:23 I agree, but with deprecation period of one release 15:37:31 this is talked in ml 15:37:37 in fact I'd take a careful look on which distros we want to support 15:37:51 ubuntu/centos/oracle 15:38:01 there is also debian 15:38:09 but I'd be ok with these 3 for now 15:38:11 and rhel 15:38:16 damn, we have lot of them 15:38:16 +rhel, yes 15:38:30 but i think it is better to have good support for 3 distros instead of supporting a mass of distros 15:38:36 I agree 15:38:38 and for fedora and debian we do not have gate jobs 15:39:07 potentially someday we could add some lightweight distro like snappy 15:39:42 I'd say we need to create policy on which distros we support and clad it in stone 15:39:52 because we're going exponential with this 15:40:15 anyway, fedora should get 1 release deprecation 15:40:31 just abandoned the review 15:40:33 berendt, which would translate to big red warning during build 15:40:55 deprecation warning into a release note + build warning should be sufficient..? 15:40:56 so let's mark it deprecate in this cycle. 15:41:05 agree 15:41:10 + prechecks 15:41:12 and remove it in Ocata 15:41:23 agree. 15:41:51 is there any other distro we want to get rid of 15:41:51 ? 15:42:04 debian from my POV.. 15:42:08 add warning in config.py to to warn the users when building fedora.. 15:42:30 Jeffrey4l_ inc0 i added both of you as adins of the luanchpad group 15:42:38 i think nobody is working on debian, at least a lot of docker image contain ubuntu instructions but no debian instructions 15:42:38 don't use delete please ;) 15:42:57 sdake, got emails. thanks. 15:42:59 and make certain your launchpad password is *INIQUE* 15:43:07 UNIQUE that is 15:43:11 what about https://bugs.launchpad.net/kolla/+bug/1616387 ? 15:43:11 Launchpad bug 1616387 in kolla "Remove base_distro type fedora from Dockerfile templates" [Wishlist,In progress] - Assigned to Tin Lam (tl3438) 15:43:22 should I close this? I think deprecation requires a blueprint 15:43:30 berendt yo ushould be all set on launchpad now 15:43:35 berendt, no. move it to O cycle. 15:43:43 Jeffrey4l_ no blueprint? 15:43:45 sdake thanks 15:44:11 hmm. bug or bp should both fine( i think ) :) 15:44:26 yaay my power grows mwhahaha 15:44:47 inc0 we have 6 dys to sort out an upgrae to jewwel 15:44:50 andd by we, i mean you 15:44:58 if it doesn'twork prior to rc1, i am reverting the jewel change 15:45:07 ofc 15:45:11 thanks ;) 15:45:16 well it's more about validation 15:45:20 right 15:45:31 sdake, i will test the upgrade ceph version tomorrow. 15:45:31 from what i hear upgrade is completely hosed atm 15:45:31 my infra is broken atm, but I'll work on setting new one 15:45:37 Jeffrey4l_ thanks dude 15:45:44 sdake, is it? 15:45:55 inc0 someone forgot who reported mariadb craters on upgrade 15:46:09 hey guys, anyone have a rst editor for windows they really like? 15:46:10 about the debian, i perfer to mark it deprecated too. 15:46:23 jrich523 vim? :) 15:46:28 deprecatd is a bit strong 15:46:38 jrich523, or sublime text if you're weakling and need mouse;) 15:46:39 i think not yet ready is better 15:46:40 i knew i would regret asking this in a heavily linux channel lol 15:46:40 we could make a pre-deprecation warning? 15:46:41 which I dop 15:46:43 wont we ever get to debian? 15:46:56 i do enjoy my mouse, but more so i love sublime due to its shortcuts :) 15:47:06 mostly i guess i wanted a preview window more than anything 15:47:09 no one maintain it and no bug is reported about it( mean no one use it.) 15:47:13 jrich523, my vim now has most of stuff I liked in sublime 15:47:14 jrich523 maybe atom could be an option 15:47:20 ok makes sense Jeffrey4l_ 15:47:34 Jeffrey4l_ confirmed. i think nobody takes care of debian 15:47:44 editor war :) 15:47:49 hmm atom looks neat berendt 15:47:50 jrich523 https://atom.io 15:48:08 OK let's do this :) 15:48:19 let's make it deprecated too and if we'll get uproar next release "but I love debian" at lest we'll have supporters 15:48:32 hmm i'll give that a try... hopefully it isnt as disappointing as Brackets 15:48:35 ok i will take care of both deprecations 15:48:43 blueprint or bug? not yet sure 15:48:59 for debian we have neither 15:49:18 sdake, the maraidb upgrade and reconfigure should be broken 15:49:56 berendt, i think both of them should be OK. but remember that add a release note for that. 15:50:51 berendt: it even has an rst preview package 15:51:12 jrich523 yes it is pretty nice 15:51:47 Jeffrey Zhang proposed openstack/kolla: Build process outputs each line twice https://review.openstack.org/365567 15:55:13 Jeffrey Zhang proposed openstack/kolla: Build process outputs each line twice https://review.openstack.org/365567 15:57:22 Merged openstack/kolla: Ensure doesn't use LOG.warn https://review.openstack.org/364168 15:57:57 meeting guys 15:58:17 Jeffrey Zhang proposed openstack/kolla: Build process outputs each line twice https://review.openstack.org/365567 15:59:18 https://bugs.launchpad.net/kolla/+bug/1621153 15:59:18 Launchpad bug 1621153 in kolla "Deprecate Debian" [Medium,New] - Assigned to Christian Berendt (berendt) 15:59:25 https://bugs.launchpad.net/kolla/+bug/1616387 15:59:25 Launchpad bug 1616387 in kolla "Deprecate Fedora" [Medium,In progress] - Assigned to Christian Berendt (berendt) 15:59:30 reviews will come tomorrow 15:59:47 cool 15:59:58 begin meeting？ 16:00:03 yep 16:00:04 Merged openstack/kolla: Update murano service name https://review.openstack.org/365761 16:05:36 we merged the cloudkitty role earlier as planned, can sombody approve https://blueprints.launchpad.net/kolla/+spec/cloudkitty-ansible ? 16:05:49 or can i self-approve it? 16:16:55 berendt typicallyblueprints should not be self approved 16:17:07 if I see someone do that, I typicallly just approve it myself 16:17:20 berendt do you need cloudkitty for newton? 16:17:28 I am concerned we have 6 days left and a whole lot of work to merge 16:17:40 sdake it is already merged.. :) 16:17:48 oh 16:17:52 well then :) 16:18:41 - - 16:18:52 i have help submit 16:18:57 sdake 16:19:01 sup zhubingbing 16:19:07 zhubingbing do you ever sleep ;) 16:19:08 rhallisey: ran into one issue... 16:19:25 - - 16:19:35 zhubingbing what hep do you need fine sir 16:19:53 there's one case where names are ambiguous with services. 16:20:34 wait to see Barbican. 16:20:35 － － 16:20:43 zhubingbing link? 16:21:00 kolla-kubernetes resource-tempalate create svc mariadb 16:21:05 wait gete 16:21:08 gate 16:21:12 zhubingbing roger 16:21:22 i am going to try to get the gating extended to 90 minutes 16:21:22 https://review.openstack.org/#/c/352515/ 16:21:28 we shouldn mae the svc service template without the -service as its redundant. but then its the same name as the service. 16:21:32 ok－ 16:21:47 by try I mean to say I am goin to submit a patch - and pray openstack-infra accepts it :) 16:23:00 sdake， I submitted the cloudkitty today, and I want him to support gnocchi backend 16:25:32 zhubingbing proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 16:27:06 i gotta give credit to the kolla team... 16:27:32 we were just looking at how we're going to move our telegraf/influxdb/grafana setup to the kolla architecture... reviews already exist for that 16:28:06 monitoring would be the one thing left (we use sensu, because autoregistration!).. we'll bring our stuff over hopefully soon when we can 16:28:17 Daviey was working on sensu 16:28:32 oh cool, i'll look at the work done and cooperate, we have a ton of really cool health checks 16:28:36 i thnk he abanonded in favor of a tig stakc 16:28:42 mnaser do you think both are needed? 16:29:28 https://review.openstack.org/#/c/351294/ 16:29:47 sdake, can you look it ? 16:29:48 telegraf is good for metrics, but for us things like: are the drives healthy? is the api responding? is X service up? more of the binary checks 16:29:54 those work better with sensu in our experience 16:30:28 i will submit telegarf role 16:30:46 wait influxdb role merge 16:32:32 mnaser does colelct integrate with sensu? 16:32:42 mnaser we are open to having both from prvious conversations we have had 16:32:53 mnaser however, would help me to understand the use cases 16:40:30 Kevin Fox proposed openstack/kolla-kubernetes: WIP: Reverse arg order on resource and resource-template https://review.openstack.org/366437 16:41:45 Kevin Fox proposed openstack/kolla-kubernetes: WIP: Reverse arg order on resource and resource-template https://review.openstack.org/366437 16:45:03 hi guys 16:45:04 : * https://review.openstack.org/#/c/366315/2 (wznoinsk) include or not a commented out 'Database options' section in globals.yml ? 16:45:11 i confirmi mnaser, i also use sensu in my environment, sensu is good fur health checks, collected is good to collect performance data 16:46:22 wznoinsk, it's always better to have it there, we'll need to have reference of all the options in the end as well 16:46:29 inc0, pbourke sdake cloud u review this https://review.openstack.org/345751 16:46:48 mnaser https://blueprints.launchpad.net/kolla/+spec/sensu is the spec for sensu 16:47:37 yeah imho monitoring will need to go to Ocata too 16:47:51 as for example influxdb is kinda fubar to make HA 16:47:54 i talked with a guy from evoila, they are also using sensu 16:48:13 i wanted to take over the sensu blueprint after his vacation 16:48:24 we are hopping to extend the collectd support to work with both influxdb at a later point 16:48:36 sean-k-mooney this would be great and is trivial 16:48:44 sean-k-mooney, there is also intel's own Snap 16:48:50 currentl the intiall contianer was being added to work with celiometer 16:48:54 i am using collectd with grafana at the moment and this is working pretty well 16:49:00 which is actually pretty cool 16:49:43 so on the monitoring side ideally we should be able to swap the collector betwen telegar/collectd and snap 16:49:47 inc0 does it makes sense to add several telemetry frameworks? 16:50:36 ideally we would be able to swap influxdb out for ceilometer or gnochii also 16:50:37 berendt, ideally I'd have one at the end of the day, just well versed 16:50:49 this is hard topic 16:51:01 i think we should focus on one implementation first 16:51:13 finish the influxdb integration and add snap later 16:51:36 well we should have too implenetaion hope fully in netwon 16:51:59 telegraph + influxdb + graphana 16:52:08 and collectd + ceilometer 16:52:24 well newton is done, we have what we have 16:52:36 inc0: yes true :) 16:52:49 reading buffer 16:53:01 all I'm saying is that in Ocata we should take a second look at it 16:53:17 Merged openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 16:53:18 collectd + should be easy to add and i think snap + influxdb + graphana would also be easy to add 16:53:32 sdake, we have two monitoring systems at the end of the day. sensu to do more boolean checks (up or down) but telegraf+influxdb+grafana for metrics 16:53:43 and then we havent gotten into using the alerting that is part of the tick stack 16:53:46 s/colledtd +/ collectd + influxdb + graphana 16:53:55 bbiab, need to move to office;) cya all in a bit 16:54:05 i think collectd is too heavy, telegraf has a ton of really cool built-in plugins 16:54:16 and it can fork/exec things too 16:54:21 collectd isn't really heavy 16:54:29 we've had problems with collectd somehow sucking up 40gb of memory on a compute node 16:54:39 we didnt look into why but yeah.. not a good day 16:54:46 mnaser: unless you had a memory leak that should not happen 16:54:51 Jeffrey4l_ ping do you know if we are using the centos and epel mirrorss yet 16:54:57 i figured :) but it was replicated across many compute nodes 16:55:35 telegraf has been working well for us and the fact it has built in influxdb integration worked well for us (and a ton of pre-made ready to use plugins) 16:55:46 sdake, no. we are only use mirror for ubuntu. 16:55:59 Jeffrey4l_ - we need to do our part on infra gating 16:56:21 Jeffrey4l_ since you understand how this stuff works, any chance you can get a patch in the queue today 16:56:23 or I can do so 16:56:29 I think it may be near bedtime fory ou 16:56:38 centos base and epel need to be in our mirror list 16:57:31 I have a PS to use the gate mirror, which using `sed` to change the file. we talked that. and i will move to the mirror.availabvle.d folder 16:57:37 I can add it. 17:05:19 Jeffrey4l_ can you add it today? 17:05:24 all our gate jobs are timing out 17:05:51 sdake, yep. I wil push a new PS soon. 17:06:22 btw. I increased the timeout time in project-config https://review.openstack.org/364377 17:06:27 sdake, ^^ 17:08:15 Jeffrey4l_ already on it 17:08:45 https://review.openstack.org/#/c/366859/ 17:08:47 sdake, mine is duplicated? 17:08:50 yup 17:08:59 mine already has s1 +2 :) 17:09:13 cool . I will abandon mine. 17:12:11 yo vhosakot 17:12:19 sdake: yo 17:12:43 sdake +2 battle? :) 17:14:15 sdake: here 17:14:46 Daviey i think we found a use case for both tig and sensu 17:14:49 read scrollback 17:14:57 ok, will do 17:15:28 sdake: Sorry for missing the meeting, i have a clashing standing real life meeting at the time 17:15:30 Hi guys 17:15:56 I'm troubleshooting some issues with L2 it looks like. My netns can't ping through to the host 17:16:00 Daviey its ok - i missed it too:) 17:16:30 Once kolla has deployed, you should be able to create a physnet1 external network and join a router to it as the default gateway, correct? 17:16:42 Paul Bourke proposed openstack/kolla: Add header blocks to all Dockerfiles https://review.openstack.org/366878 17:17:35 Paul Bourke proposed openstack/kolla: Add header blocks to all Dockerfiles https://review.openstack.org/366878 17:18:11 Daviey unfortunately it is very difficult to find a timeslot that works for everyone 17:18:25 re standing meeting - perhaps you will just have to read the meeting logs (which I would recommend) 17:18:43 Daviey i think changing the meeting time isn't going to operate well with all our contributors 17:20:18 Netns created, everything looks good. Here's my ovs config. Could someone spot check it for me to make sure it looks sane? 17:20:19 http://hastebin.com/yavuzumona.sm 17:20:55 Or ideally do a ovs-vsctl show from a test environment that I can compare to? 17:21:15 rhallisey: alive? 17:24:53 I'm back 17:25:05 inc0: wb 17:25:24 thank you 17:28:10 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 17:29:14 sdake: right.. not asking for it to be changed.. just letting you know why i often wont be there :) 17:30:04 Good night everyone. 17:30:09 So I get full network connectivity from inside the netns, but I can't talk between the host and the guests. Again, you just create a flat external network with physnet1, correct? 17:33:27 Jeffrey Zhang proposed openstack/kolla: Using CI mirror host https://review.openstack.org/365721 17:34:14 by the way is neutron fixed on master? 17:35:54 Could someone please change docker_restart_policy: "always" to docker_restart_policy: "on-failure"? Docker will start to complain about this or fail outright on future releases 17:36:22 # of retries and "always" are supposed to be mutually exclusive (according to docker) 17:37:18 kbaegis_: changeing to on failure would not restart the container on a system reboot 17:38:03 sean-k-mooney: Gotcha. I'm just telling you it will be an issue if you upgrade docker. It' breaks the playbook 17:38:28 kbaegis_: so really i guess the number of retires should be removed then 17:38:37 sean-k-mooney: That's probably right 17:40:31 Eduardo Gonzalez proposed openstack/kolla: Add Senlin Ansible role https://review.openstack.org/339725 17:40:49 kfox1111, yo 17:40:50 back 17:41:24 since I'm in refactoring the code, I think it would be better to fix this at the same time: 17:41:27 https://blueprints.launchpad.net/kolla-kubernetes/+spec/resource-magic 17:41:30 what do you think? 17:41:58 (I got kolla-kubernetes resource-template create svc mariadb to work. :) 17:42:08 kfox1111, I agree completely 17:42:13 kbaegis_: so we dont forget can you open a bug for the docker change? 17:42:14 it's not nearly granular enough 17:42:17 k. I'll fix that too then. thanks. :) 17:42:22 kfox1111, question though 17:42:31 the most granular of all is to just use kubelet 17:42:37 something to keep in mind 17:42:46 what do you mean? 17:42:58 like, not do the resource command at all? 17:43:13 well we would still have to render templates 17:43:20 it's more of a discussion topic 17:43:31 rendering is a CLI specific tool 17:43:37 but any orchestration is not 17:43:41 yeah. 17:44:10 it's something to think about because ultimatly to reach full granularity we get to kubectl 17:44:14 + a cli that renders 17:44:21 I'm wondering if we want to pull all the orchestration out to its own cli tool. 17:44:31 well 17:44:45 does the orchestration become part of the cli at all is the question 17:44:59 yeah. 17:45:13 I can see a case, where, while we provide an asible based worflow engine, 17:45:14 because the way it is now, specificlly bootstrap, it has to be done by hand 17:45:21 others may wantl, say a mistral one. 17:45:41 so having them be seperate commands like kolla-ansible and kolla-kubernetes are gives them room to innovate. 17:45:45 kfox1111, right, cli provides tools to render where an op can take that and use it to deploy with 17:45:53 yeah. 17:46:01 ansible/wahtever can orchestrate 17:46:09 * kfox1111 nods 17:46:45 kfox1111, something to think about I guess 17:47:02 yeah. I think we're on the same page though. 17:47:07 probably should file a blueprint for it. 17:48:00 ya for sure 17:48:12 by the way what is the best way to do a vote? ml, strawpool set to mailing list is there another method i should use? 17:48:23 sean-k-mooney: done https://bugs.launchpad.net/kolla/+bug/1621187 17:48:23 Launchpad bug 1621187 in kolla "fix ansible/group_vars/all.yml restart/retry policy" [Undecided,New] 17:48:48 kbaegis_: cool thanks 17:52:28 sean-k-mooney which kind of vote 17:52:35 sean-k-mooney only cores can trigger a vote 17:52:50 sean-k-mooney however, a core can proxy a vote for you if you obtain a proxy :) 17:53:12 sdake: i was asked a few days ago to create a vote on the name of the kolla-host role 17:53:25 huh? 17:53:36 sdake: currently the name is baremetal as originally i was calling it the baremetal playbook 17:53:37 why would that need a vote 17:54:13 it was suggested to change it to host or kolla-host role to avoid confution with ironic and bifrost 17:55:00 sdake: so basically i just wanted to as what shoudl the role be called and have people vote 17:55:28 sean-k-mooney did someone hae trouble with the baremetal name 17:55:36 sdake: i was just going to send a mail to the list with this link http://www.strawpoll.me/11175159 17:55:40 sean-k-mooney needd more context but have dr appointment in 10 mins 17:56:11 sean-k-mooney cool - that is a poll not a vote 17:56:15 am yes some one taught the baremental role was installing baremetal servers 17:56:25 just send a mail to the ailing with with [poll][kolla] Name of baremetla roel 17:56:54 sdake: ok cool will do and i will expaine the context in the mail thanks 17:57:25 sean-k-mooney a vote sets policy 17:57:40 a poll gains input into hard computer scieence problems like naming :) 17:58:19 sdake: ah cool i didnt know they were used differently in an openstack context 17:58:35 sean-k-mooney this is a koll athing i htink ;) 17:58:54 i voted for waffles :) 17:59:48 dont forget the chicken! 18:00:05 only if its fried! 18:00:34 me being the noob in this that i am... the two options 'baremetal' and 'pre-install' sound very different 18:00:36 sdake: and bacon 18:00:51 baremetal would imply only for baremetal (if thats true, id vote for that one) 18:01:07 we have bootstrap already jrich523 18:01:09 i dont see whatis wrong with baremetal 18:01:10 and the pre-install just sounds like a manditory step on all hosts (which also sounds good :) 18:01:21 jrich523 pre-install is clled bootstrap 18:01:35 sean-k-mooney whatever happens, these names are locked in after 3.0.0 18:01:37 so get em right :) 18:01:55 so then what is this for? the title isnt very helpful for someone like me :) 18:02:08 but my ignorance to all of this might help provide the most useful name 18:02:26 jrich523: the beremetal role today is used by the kolla-host playbook to install the kolla dependcies 18:02:36 jrich523: before you do a kolla deploy 18:03:03 jrich523: so it installs docker... on the baremetal host 18:03:05 so then its a pre-install for baremetal only 18:03:38 wouldnt the process be the same for all hosts? 18:03:50 well no it automate everyting in the quickstart e.g. the stuff you need to do on all host that you will use on your kolla deployed openstack cloud 18:03:58 yay :D 18:04:08 now that i have it down pat, you've automated it 18:04:09 lol 18:04:35 hmm i kinda like kolla-host i think 18:04:37 jrich523: i havent automated the build host yet but i will do that soon i hope 18:05:22 actually seems like it should be mildly easy 18:05:35 jrich523: ya the only reason i called it beremetal was i was origially calling the playbook the baremetal playbook 18:05:52 i feel like for the most part it shouldnt matter 18:06:02 and in the end should work on any host 18:06:53 jrich523: actully my plan is that for the build host i will reuse the current rould and just install the extra things need on the build host. like ansible and kolla iteslf 18:07:10 cool :) 18:14:35 i voted for kolla-host, and im winning! 18:16:20 :) i could not make up my mind so voted for kolla-host and host 18:16:33 host seems... dangerous 18:16:48 oh why 18:17:17 maybe it just has to do with my powershell experience, but there is an alias for host (get-host) which i've seen people.... well... struggle.. 18:17:43 its a common name, not uncommon for a cmd to be called that... just too general i guess 18:18:32 oh ok well the reason i lean towrods host instead of kolla-host is that it a role in the kolla repo so haveing kolla in the role name feels a little redundant 18:18:50 but this is why i create the poll 18:19:00 hmm i'll agree with that 18:19:07 so that people can tell me what they prefer 18:20:52 what about something like Node? because really its prepping any host to act as a node of sorts 18:21:20 Base, Node... hmm 18:22:02 ya maybe ill ask for other suggesting in the email too. 18:22:26 at the end of the day i dont really mind what the role is called as long as it works 18:23:00 yeah sort of unimportant, but meaningful names are always nice :) 18:30:19 sean-k-mooney: the second question in the poll is legitimately harder. So this is voting whether to rename ./ansible/../roles/baremetal? 18:31:58 sorry, not voting in that sense. heh. polling for names of that role... one of which (I think) is the current one 18:34:51 mark-casey: ya the question is should we change the name of ./ansible/../roles/baremetal given that kolla also has baremetal as a service(ironic) and birost for baremetal provisoning at deployment time 18:42:28 rhallisey: what should status look like then for resource? 18:42:45 kubectl get pod foo -o yaml? 18:46:29 kubectl describe pods nova-api-0 18:46:39 that would be the kubelet way 18:46:52 -o yaml adds the status too. and its machine parsable. 18:46:52 the more verbose way is contacting the endpoint 18:46:56 describe I don't think is. 18:47:06 oh true that drops it into yaml 18:47:07 sean-k-mooney: cool. I went with pre-install as I think 'kolla host' is often casually used to describe the host where kolla-ansible is run. 18:47:44 kfox1111, oh to describe the running resource 18:48:37 that output acutallu looks preety good 18:49:28 kfox1111, ya that's a way more descripive output 18:49:39 mark-casey: ya there are pros and cons to all the options hence the poll. 19:15:05 sean-k-mooney what about bootstrap? 19:15:42 berendt: that would make sense too yes :) 19:17:40 rhallisey: ok. I'll rewrite it to use that. 19:18:12 kfox1111, so what will the cli do in this case? 19:18:32 or are you making an iterative change 19:19:58 just mapping kolla-kubernetes resource create/delete/status X to equiv kubectl [create -f template]/[delete pod/job/daemonset X]/[get pod/job/daemonset X -o yaml] 19:20:09 ok cool 19:21:23 I'll have a revised ps up soon. 19:26:28 I got create hooked up. just doing delete/status now. 19:31:30 "string indices must be integers"..... not helpful... :/ 19:33:03 I really wish python had a gdb equiv that was easier to use. pdb can be a pain... 19:33:55 ah... there it is. 19:36:51 ok. this seems to do the trick. 19:37:24 Kevin Fox proposed openstack/kolla-kubernetes: Cleanup resource and resource-template https://review.openstack.org/366437 19:37:36 rhallisey: can you have a loook at ^ please and let me know what you think? 19:44:50 doh. pep8... will fix that 20:03:42 guys is it just me or neutron is broken? 20:03:54 inc0, I see it also 20:03:58 db sync error 20:04:07 rhallisey, rebuild and clenaup containerts 20:04:11 this one is fixed 20:04:17 now dhcp is acting up 21:08:50 Kevin Fox proposed openstack/kolla-kubernetes: Cleanup resource and resource-template https://review.openstack.org/366437 21:08:51 Kevin Fox proposed openstack/kolla-kubernetes: Check for service name in resource name https://review.openstack.org/366983 21:13:53 Eduardo Gonzalez proposed openstack/kolla: Add Senlin Ansible role https://review.openstack.org/339725 21:25:38 Kevin Fox proposed openstack/kolla-kubernetes: Cleanup resource and resource-template https://review.openstack.org/366437 21:33:04 sdake: ping 21:33:21 tonanhngo hello fine sir 21:33:58 Hi Steve, I am looking for a recommendation on installing OpenStack on a large cluster 21:34:36 I guess you just went through this exercise recently, so maybe we can learn a few things from you 21:34:59 We are getting a cluster of some 300 nodes for Magnum scalability testing using Rally 21:35:17 But first we have to install OpenStack on the cluster 21:35:18 from osic? 21:35:32 It's from CNCF, but it's similar 21:35:44 iteritems went away in python 3? 21:35:46 grr... 21:35:48 do they have predefined ways to install baremetal? 21:36:26 These are all baremetals, and we requested Ubuntu 14.04 with 2 NIC 21:36:40 kolla is easy 21:36:46 not sure if it scales to that size 21:36:53 we have implemented magnum already - but it may need love 21:36:58 how much time edo you have? 21:37:16 So if we want to install the Newton release, can we do this with Kolla? 21:37:27 well not sure on 300 nodes 21:37:34 i think that should work 21:37:34 but has not been tested 21:37:41 bleh.. stupid python 3. :/ 21:37:44 We requested 2 weeks, but I think they can be flexible 21:38:04 123 nodes has been tested 21:38:09 whe nis your window 21:38:20 i'd highly recommend preparing rally prior to starting 21:38:34 we have some work to start from 21:39:05 Did you drive Rally from your own server? or one of the server in the cluster? 21:39:14 from deploy node 21:39:22 to deply targets 21:39:31 ok 21:39:52 You use Ansible to deploy? 21:40:10 Kevin Fox proposed openstack/kolla-kubernetes: Cleanup resource and resource-template https://review.openstack.org/366437 21:44:40 tonanhngo yes 21:44:50 tonanhngo deploy on 123 servers takes 20 minutes 21:44:56 atleast in osic environmnt 21:45:12 100 compute 20 stroage 3 control with ceph 21:45:35 tonanhngo afaik no other tool can even scale to 100+ nodes 21:45:45 That's quite good 21:45:57 tonanhngo what you will be challenged with is running magnum outside containers 21:46:08 or alternately making the magnum in the containers work properly 21:46:30 So the containerized Magnum is not quite working yet? 21:46:36 tonanhngo the problem with our implementation is it is not integrated with anything but file backend 21:46:42 and i am pretty sure that works 21:46:59 sbezverk would know 21:47:08 he implemented lbaas v2 just for the magnum use case 21:47:27 which reminds me, is magnum workign with lbaas v2? 21:47:39 We are working on it. 21:47:46 the other problem you will have is getting correct routing of heat-api and heat-api-cfn 21:47:52 but you have that no mattere what too you use 21:47:58 that has to do with how the network is configured 21:49:01 Is there some documentation on how to work through these? 21:49:04 kolla will also setup your bare metal for you 21:49:15 the heat api network config? 21:49:19 right 21:49:21 i'd look to heat documentation for that 21:49:25 ok 21:49:38 we didnt test wait conditions in our environment 21:49:44 i know they work - I use them in my home lab 21:49:48 but my home lab is configured proerly 21:50:08 lbaasv2 - need to finih the job on magnum 21:50:15 you will probby be deploying master i assume 21:50:23 prior to newton magnum was non-functional 21:50:28 That's what I am thinking 21:50:42 ok, so if we are stuck with containerized Magnum, we can just install Magnum manually on some host? 21:53:34 tonanhngo yup you sure can 21:53:37 tonanhngo but it will be a manual process 21:53:48 the contianers run with net=hosst 21:53:59 so connecting to the network of the db/rabbit is straigghtforward 21:54:06 just configure it appropriately 21:54:15 ok 21:54:18 with the right passwords 21:54:29 anible itself can scale to 300 nodes no problem 21:54:41 whether rabbitmq and mariadb work at that scale with kolla is an unknown 21:55:53 Yeah that's one of my concern, we may hit other bottleneck outside Magnum or the container orchestration engines 21:55:58 the challenges in a non-containeized magnum are 1) manual config of mangum 2) manual install of magnum 3) getting heat-api or heat-api-cfn to communicate to the vms 4) getting lbaas v2 implemented 21:56:15 easy to configur earound 21:56:29 you just drop your custom config in /etc/kolla/config/nova.conf 21:56:34 or whatever 21:56:36 and it will override whata i already there for those values 21:56:59 tonanhngo i suspect some folks in olla would help you get going on a 300 node deploy 21:57:19 tonanhngo it would be a great proof point that kolla works a tthat scale for computekit 21:57:32 Prithiv proposed openstack/kolla: Ansible roles for collectd Container https://review.openstack.org/366082 21:57:34 how beefy are your controller nodes? 21:57:53 Good to know to watch out for these, and we would love to get help from the Kolla team. 21:57:58 let me check 21:58:23 Prithiv proposed openstack/kolla: Ansible roles for collectd Container https://review.openstack.org/366082 21:59:29 Here is the server spec: https://github.com/cncf/cluster/ 21:59:44 tonanhngo pbourke and inc0 did most of our bare metal networking setup 21:59:51 tonanhngo we used vlans and whatnot 21:59:55 hi. 21:59:56 all magic to me ;) 22:00:26 what we did was bond two of the networking interfaces 22:00:30 https://review.openstack.org/#/c/346842/ i see centos binary and deploy gate failing for this. 22:00:38 and setup vlans on them iirc 22:00:50 Daviey also has some understanding 22:00:53 yeah we probably won't have access to those, but we will see what they give us. 22:01:08 i have added exception for binary in test_build.py . but i still face the same issue. can someone help me out in this please 22:01:27 the lsat challenge is getting your bare metal in a shape is probably networking 22:01:50 tonanhngo with vlans, its fairly straightforward to get the api services sto communicate with the neutron network i think 22:01:59 which is what you need to get vms to check into heat's wait conditions 22:03:23 ok, guess I will need to get someone with networking expertise to stand by for help. 22:03:33 is that a known issue or am missing something here.. 22:03:33 So to try out Kolla on a small environment, I can just follow the documentation? 22:03:42 tonanhngo docs should get you there 22:03:45 i'd try all in one first 22:03:56 prithiv link to problem? 22:04:04 Sounds good, will do that 22:05:42 sdake: https://review.openstack.org/#/c/346842/ 22:05:50 this patch 22:06:26 has jenkins check failure. 22:06:39 https://github.com/openstack/kolla/blob/master/tests/test_build.py 22:06:49 sfc image has been added to exceptions also 22:07:45 prithiv http://logs.openstack.org/42/346842/7/check/gate-kolla-dsvm-deploy-centos-binary-centos-7-nv/c004054/console.html#_2016-08-31_19_40_09_344046 22:09:07 prithiv you hae added to exceptions here: https://github.com/openstack/kolla/blob/master/tests/test_build.py#L69 22:09:17 which is fine 22:09:34 your not bulding it in the deploy job 22:09:35 yet your trying to deploy it 22:10:03 oh.. how would i defer from that being executed 22:11:12 sdake: the networking-sfc-agent should not be enabled to deploy by default. 22:11:12 usually by enable_featurename 22:11:21 sean-k-mooney ya clearly :) 22:11:56 prithiv our gate jobs do not deploy non-compute kit servers at this time 22:12:47 sdake looking at the patach at first glance it should not be deploy this image in the gate https://review.openstack.org/#/c/346842/7/ansible/group_vars/all.yml 22:13:22 sean-k-mooney agree - I am stumped 22:13:34 main.yml looks ok to me but not certain 22:14:07 maybe woudl should expcitly test enable_neutron_networking_sfc_agent | bool == true? 22:14:19 i dont thnink that should change anyting though 22:18:40 ok folks nee to do laptop reapir shortly and ill be out 22:18:47 tonanhngo you all settled? 22:18:52 pbourke runs in the uk timezones 22:18:59 daviey runs in uk and early us timezones 22:19:16 tonanhngo inc0 runs in us timezones 22:20:04 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 22:23:44 Prithiv proposed openstack/kolla: Add networking guide for Neutron extensions https://review.openstack.org/355386 22:23:55 I think you give enough tips to get started 22:24:05 tonanhngo probbly too much :) 22:24:35 Thanks Steve, I will be back begging for more help :) 22:29:00 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 23:36:29 Kevin Fox proposed openstack/kolla: Fix tgtd config for kolla-kubernetes https://review.openstack.org/367024 23:38:02 is ceph broken on master or partically cinder+ceph? 23:38:56 im seeing this http://paste.openstack.org/show/568394/ 23:39:42 im also not able to upload an image to glance with the ceph backend. 23:40:01 im going to revert to mitaka and see if the same config works tomorow 23:59:45 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 00:11:37 Hui Kang proposed openstack/kolla: Add etcd ansible role https://review.openstack.org/364662 00:31:13 Hui Kang proposed openstack/kolla: Fix mariadb bootstrap error https://review.openstack.org/360218 00:58:59 Serguei Bezverkhi proposed openstack/kolla-kubernetes: Add cinder volume with iscsi and lvm backend to Kube https://review.openstack.org/364573 01:06:46 Jeffrey4l_, ping 01:06:53 ubuntu upgrades of ceph seems working 01:06:58 vm is alive, disk is there 01:07:03 all seems good 01:07:26 cool. i will test the centos today. 01:07:33 Jeffrey Zhang proposed openstack/kolla: Using CI mirror host https://review.openstack.org/365721 01:08:07 kk, remove -2 if it works, we should be good 01:08:08 thanks:) 01:08:19 inc0, OK> 01:26:46 Jeffrey4l_: testing now. 01:26:49 sorry, ran out of time yesterday 01:26:57 mtaylor22, cool. thanks :) 01:27:02 just rebuilding the docker images to include dumb init 01:27:33 yep. 01:28:03 also might want to ask you a few questions a bit later 01:28:11 ill ping you when ready etc :) 01:28:24 zhubingbing proposed openstack/kolla: Add Barbican ansible role https://review.openstack.org/352515 01:28:48 ok. 01:29:50 zhubingbing proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 01:32:28 mtaylor22, what do u think about Ross's explanation? https://bugs.launchpad.net/kolla/+bug/1616268/comments/8 01:32:28 Launchpad bug 1616268 in kolla mitaka "Stale namespace removal causing "RTNETLINK answers: Invalid argument" errors" [Critical,Confirmed] 01:32:49 yo 01:33:14 yo 01:33:17 sup 01:33:29 morning sdake 01:33:37 afternoon actually 01:33:38 6pm 01:33:47 9:30 am here :) 01:33:55 Jeffrey4l_: Ross sits next to me, so its hard for me to think of it from a different perspective (ie. fresh mind) 01:34:01 morning/afternoon 01:34:02 11:33am here (Sydney) 01:34:14 but i do understand what he pointed out 01:34:22 OK. 01:35:04 I couldn't figure out which container was holding onto the namespace, i mean lsof doesn't identify which particular container owns the namespace fd 01:35:15 and i couldn't find it in /proc either (ownership wise) 01:36:18 just rebuilding neutron containers atm too. 01:36:35 mtaylor22, i think it may be te dnsmasq process http://paste.openstack.org/show/568805/ 01:37:17 I will read in a few mins, just gotta deal with a customer issue.