*** bmace has quit IRC | 00:07 | |
*** bmace has joined #kolla | 00:07 | |
*** achanda has joined #kolla | 00:28 | |
*** ssurana has quit IRC | 00:36 | |
*** apuimedo has quit IRC | 00:37 | |
*** ntpttr has quit IRC | 00:37 | |
*** ntpttr has joined #kolla | 00:40 | |
*** unicell has quit IRC | 00:40 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Test reduce gate usage https://review.openstack.org/294854 | 00:45 |
---|---|---|
*** unicell has joined #kolla | 00:47 | |
*** unicell has quit IRC | 00:53 | |
*** rstarmer has joined #kolla | 01:00 | |
*** Jeffrey4l has joined #kolla | 01:05 | |
*** dims has joined #kolla | 01:12 | |
*** dims_ has quit IRC | 01:12 | |
*** iceyao has joined #kolla | 01:17 | |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Add check if all the hostnames are resolvable https://review.openstack.org/294713 | 01:22 |
*** sbezverk has quit IRC | 01:24 | |
*** sbezverk has joined #kolla | 01:25 | |
*** inc0 has joined #kolla | 01:26 | |
inc0 | hello | 01:26 |
*** inc0 has quit IRC | 01:29 | |
openstackgerrit | Hui Kang proposed openstack/kolla: Ceilometer needs mongoengine if build from source https://review.openstack.org/294725 | 01:29 |
*** rstarmer has quit IRC | 01:33 | |
*** dims has quit IRC | 01:50 | |
*** rhallisey has quit IRC | 01:53 | |
*** rstarmer has joined #kolla | 01:57 | |
*** weiyu has joined #kolla | 02:08 | |
*** weiyu has quit IRC | 02:09 | |
*** achanda_ has joined #kolla | 02:09 | |
*** achanda has quit IRC | 02:12 | |
*** Marga_ has quit IRC | 02:17 | |
*** achanda_ has quit IRC | 02:19 | |
*** achanda has joined #kolla | 02:20 | |
*** achanda has quit IRC | 02:24 | |
*** rstarmer has quit IRC | 02:34 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Allow external ceph journals https://review.openstack.org/294862 | 02:40 |
*** vhosakot has joined #kolla | 02:42 | |
*** banix has joined #kolla | 02:58 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Allow external ceph journals https://review.openstack.org/294862 | 03:07 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Allow user to choose filesystem for osd https://review.openstack.org/294863 | 03:07 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Fix permissions for log extraction in gate https://review.openstack.org/294864 | 03:09 |
*** dave-mccowan has quit IRC | 03:31 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: TEST debug glean in gate https://review.openstack.org/294868 | 03:32 |
openstackgerrit | Sam Yaple proposed openstack/kolla: TEST debug glean in gate https://review.openstack.org/294868 | 03:43 |
SamYaple | ~. | 03:47 |
openstackgerrit | Sam Yaple proposed openstack/kolla: TdEST debug glean in gate https://review.openstack.org/294868 | 03:47 |
*** Marga_ has joined #kolla | 03:53 | |
*** Marga_ has quit IRC | 03:55 | |
*** Marga_ has joined #kolla | 03:55 | |
openstackgerrit | Maru Newby proposed openstack/kolla: Update quickstart doc to use drop-in unit files https://review.openstack.org/294869 | 03:56 |
openstackgerrit | Maru Newby proposed openstack/kolla: Update quickstart doc to use drop-in unit files https://review.openstack.org/294869 | 04:01 |
*** rstarmer has joined #kolla | 04:09 | |
SamYaple | Jeffrey4l mandre nihilifer pbourke this patch is critical to fixing the UNSTABLE gate and should be merged ASAP https://review.openstack.org/#/c/294864/ | 04:11 |
*** rstarmer has quit IRC | 04:12 | |
*** banix has quit IRC | 04:24 | |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Add check if all the hostnames are resolvable https://review.openstack.org/294713 | 04:25 |
openstackgerrit | Sam Yaple proposed openstack/kolla: TEST reduce size https://review.openstack.org/294872 | 04:25 |
*** salv-orlando has joined #kolla | 04:27 | |
*** spisarski has quit IRC | 04:29 | |
*** salv-orl_ has quit IRC | 04:30 | |
*** rstarmer has joined #kolla | 04:43 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Fix 'optional' config.json values https://review.openstack.org/294350 | 04:45 |
*** Marga_ has quit IRC | 04:46 | |
*** rstarmer has quit IRC | 04:59 | |
*** Marga_ has joined #kolla | 05:00 | |
*** achanda has joined #kolla | 05:03 | |
openstackgerrit | Maru Newby proposed openstack/kolla: Update quickstart doc to use drop-in unit files https://review.openstack.org/294869 | 05:10 |
*** iceyao has quit IRC | 05:21 | |
SamYaple | ccesario_: i see the mariadb issue. i do not have a fix yet | 05:21 |
*** vhosakot has quit IRC | 05:25 | |
mandre | SamYaple: what's happening exactly with copying the logs on the gate? | 05:43 |
mandre | do you know why it started failing? | 05:43 |
*** Windir has joined #kolla | 06:33 | |
*** unicell has joined #kolla | 06:43 | |
*** akwasnie has joined #kolla | 07:21 | |
*** akwasnie has quit IRC | 07:36 | |
*** achanda has quit IRC | 07:38 | |
*** achanda has joined #kolla | 07:40 | |
*** jhyang has quit IRC | 07:41 | |
*** akwasnie has joined #kolla | 07:47 | |
*** achanda has quit IRC | 07:51 | |
*** akwasnie1 has joined #kolla | 07:52 | |
*** akwasnie has quit IRC | 07:55 | |
*** stvnoyes has quit IRC | 08:02 | |
*** stvnoyes has joined #kolla | 08:03 | |
*** jmccarthy has quit IRC | 08:05 | |
*** jmccarthy has joined #kolla | 08:06 | |
*** akwasnie has joined #kolla | 08:13 | |
*** akwasnie has quit IRC | 08:19 | |
*** cfarquhar has quit IRC | 08:21 | |
*** cfarquhar has joined #kolla | 08:24 | |
*** cfarquhar has quit IRC | 08:24 | |
*** cfarquhar has joined #kolla | 08:24 | |
*** achanda has joined #kolla | 08:51 | |
*** achanda has quit IRC | 08:57 | |
*** akwasnie has joined #kolla | 09:05 | |
*** mbound has joined #kolla | 09:08 | |
*** mbound has quit IRC | 09:13 | |
*** mbound has joined #kolla | 09:16 | |
*** akwasnie2 has joined #kolla | 09:19 | |
*** akwasnie has quit IRC | 09:19 | |
*** salv-orlando has quit IRC | 09:21 | |
*** akwasnie has joined #kolla | 09:24 | |
*** akwasnie2 has quit IRC | 09:25 | |
*** shardy has joined #kolla | 09:41 | |
*** akwasnie has quit IRC | 09:42 | |
*** shardy has quit IRC | 09:52 | |
*** achanda has joined #kolla | 09:54 | |
*** achanda has quit IRC | 09:59 | |
*** SiRiuS_ has joined #kolla | 10:13 | |
*** pbourke has quit IRC | 10:13 | |
*** pbourke has joined #kolla | 10:13 | |
*** mbound has quit IRC | 10:24 | |
SiRiuS_ | mandre, can you please look at my patch: https://review.openstack.org/#/c/294340/ | 10:26 |
SiRiuS_ | mandre, SamYaple requested it before the patch can be accepted | 10:26 |
SiRiuS_ | pbourke, dos you use virtualbox? can you please take a look at https://review.openstack.org/#/c/294340/ | 10:28 |
*** ccesario has quit IRC | 10:42 | |
*** athomas has joined #kolla | 10:42 | |
*** athomas has quit IRC | 10:46 | |
*** athomas has joined #kolla | 10:52 | |
*** dave-mccowan has joined #kolla | 11:53 | |
*** rhallisey has joined #kolla | 11:57 | |
*** achanda has joined #kolla | 11:58 | |
*** achanda has quit IRC | 12:03 | |
*** athomas has quit IRC | 12:07 | |
*** ccesario has joined #kolla | 12:09 | |
*** ccesario has quit IRC | 12:16 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE https://review.openstack.org/294906 | 12:20 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE https://review.openstack.org/294906 | 12:24 |
*** dims has joined #kolla | 12:25 | |
*** dims has quit IRC | 12:25 | |
*** achanda has joined #kolla | 13:00 | |
*** achanda has quit IRC | 13:04 | |
*** dave-mcc_ has joined #kolla | 13:17 | |
*** dave-mccowan has quit IRC | 13:18 | |
*** signed8bit has quit IRC | 13:20 | |
*** banix has joined #kolla | 13:38 | |
SamYaple | Jeffrey4l: i have responded to you, please merge the patch and if you have a better solution later submit it later, but get needs to be fixed asap | 13:39 |
SamYaple | remember, all the logs stuff is getting rewritten for multinode | 13:40 |
Jeffrey4l | SamYaple, i will merge it if I can not catch up a solution in 1 hours. is this ok? | 13:41 |
Jeffrey4l | SamYaple, btw, do you have any thoughts about the multinode logs? | 13:41 |
SamYaple | Jeffrey4l: the problem is we arent messin with the owner for the files from the host because that affects the owner of the files in the contaienr | 13:43 |
Jeffrey4l | agreed. | 13:44 |
*** banix has quit IRC | 13:46 | |
*** iceyao has joined #kolla | 13:49 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Test reduce gate usage https://review.openstack.org/294854 | 13:59 |
Jeffrey4l | SamYaple, see the comments in https://review.openstack.org/294864 | 14:03 |
*** eandersson has joined #kolla | 14:03 | |
eandersson | Hey | 14:04 |
SamYaple | Jeffrey4l: that wont work | 14:04 |
eandersson | I am trying to set up a all-in-one openstack instance using kolla | 14:04 |
SamYaple | the jenkins user is the one that doesnt have access to read | 14:05 |
eandersson | but getting stuck at waiting for virtual ip to appear | 14:05 |
SamYaple | changing dest perms wont help | 14:05 |
SamYaple | eandersson: waiting for virtualip appears is wiaiting on keepalived | 14:05 |
SamYaple | eandersson: did you set your vip to an unused ip on your network? | 14:05 |
eandersson | yep, but I don't see any ips on any of the docker containers | 14:06 |
Jeffrey4l | SamYaple, disagree. :( , i am runnnig some test. wait for a moment. | 14:06 |
eandersson | e.g. /keepalived - | 14:06 |
SamYaple | Jeffrey4l: you disagree that its breaking on read? or do you think updating the DEST perms fixes read issues? | 14:06 |
eandersson | So I am pretty sure I am missing something obvious | 14:07 |
SamYaple | eandersson: docker ps -a show haproxy and keepalived running? | 14:07 |
Jeffrey4l | SamYaple, I think change the DEST perms can fix the read issue. | 14:07 |
SamYaple | that makes no sense | 14:07 |
SamYaple | what would that be the case? | 14:07 |
eandersson | Yepo | 14:07 |
eandersson | Up 21 minutes | 14:07 |
eandersson | for both | 14:07 |
SamYaple | eandersson: can you post your globals.yml | 14:07 |
SamYaple | also make sure to check your keepalived and haproxy logs | 14:07 |
SamYaple | they are at /var/lib/docker/volume/kolla_logs/_data/* | 14:08 |
eandersson | Awesome SamYaple, so simple | 14:08 |
eandersson | Kernel is reporting: interface enp0s20f1 DOWN | 14:08 |
eandersson | :D | 14:08 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE https://review.openstack.org/294906 | 14:08 |
eandersson | Lets plug that network cable in | 14:08 |
SamYaple | eandersson: cool. that definetely could cause this issue | 14:09 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Test reduce gate usage https://review.openstack.org/294854 | 14:10 |
Jeffrey4l | SamYaple, a test PS is pushed. If it failed, i will merge you PS. | 14:10 |
SamYaple | Jeffrey4l: ok, but aside from that. im curious. why do you think dest file permissions affect source file permissions? | 14:11 |
SamYaple | im trying to follow your thought process | 14:11 |
Jeffrey4l | SamYaple, the root cause is that the jenkins scp the /tmp/log/ to a remote location. but the mariadb.log file(now in /tmp/log/kolla/mariadb/mariadb.log) can not read by jenkins user. So it failed. | 14:12 |
eandersson | That was simple. Would probably have taken me a while to figure out that I could even look at the logs using docker like that. :D | 14:13 |
SamYaple | Jeffrey4l: oh so youre just saying reorder the chmod operation? | 14:13 |
SamYaple | yea thats reasonable | 14:13 |
SamYaple | that should work fine | 14:13 |
Jeffrey4l | the minimal change which can fix the issue should be `chmod 777 /tmp/logs/kolla/mariadb/mariadb.log` i think. | 14:14 |
SamYaple | ill so that real quick | 14:14 |
Jeffrey4l | SamYaple, yea. thanks. | 14:14 |
eandersson | input file not found at /etc/kolla/config/swift/account.builder or /etc/kolla/config/swift/account.builder | 14:16 |
eandersson | interesting | 14:16 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Fix permissions for log extraction in gate https://review.openstack.org/294864 | 14:16 |
SamYaple | eandersson: are you trying to deploy swift? | 14:17 |
eandersson | Yea | 14:17 |
eandersson | NYI? | 14:17 |
SamYaple | eandersson: there is a swift guide online | 14:17 |
SamYaple | it requires additional work | 14:17 |
eandersson | ah kk | 14:17 |
eandersson | I disabled it for now | 14:17 |
eandersson | Will try after. | 14:17 |
eandersson | You guys going to Austin? | 14:18 |
SamYaple | ajafo: of course! | 14:18 |
SamYaple | eandersson: * | 14:18 |
eandersson | awesome | 14:18 |
eandersson | will be fun =] | 14:19 |
rhallisey | yo yo | 14:19 |
SamYaple | hey rhallisey | 14:19 |
SamYaple | gate all whackabusted | 14:19 |
rhallisey | active this morning eh | 14:19 |
SamYaple | indeed | 14:20 |
rhallisey | gate fixing is perfect for weekends.. that's also what I'm doing :/ | 14:20 |
rhallisey | just not for kolla | 14:20 |
SamYaple | awesome | 14:21 |
SamYaple | hey i got external journals working!!! https://review.openstack.org/#/c/294862/ | 14:21 |
SamYaple | for ceph | 14:21 |
rhallisey | thats for newton though right? | 14:22 |
SamYaple | no this was in mitaka | 14:22 |
SamYaple | blueprint is from early early mitaka | 14:22 |
rhallisey | didn't our rc1 already go out | 14:23 |
SamYaple | yup | 14:23 |
SamYaple | this is wrapping up the blueprints that are still open | 14:23 |
rhallisey | where will this fit in though? Seems like a feature | 14:23 |
SamYaple | this will fit in as finishing up blueprints like we are doing with all the other ones? | 14:23 |
rhallisey | thought only upgrade were give the priority here and the rest kicked to N | 14:24 |
SamYaple | feel free to no approve it, but this is to stablize the ceph bootstrap stuff, so i can easily make the case taht this is a bug | 14:25 |
eandersson | This was a pretty smooth experience! | 14:25 |
SamYaple | rhallisey: https://review.openstack.org/#/c/294862/2/docker/ceph/ceph-osd/extend_start.sh (see all that hackish stuff removed) | 14:26 |
SamYaple | eandersson: im glad you liked it! | 14:26 |
eandersson | I am guessing that it dosen't create any RC files automatically as it is right? | 14:26 |
SamYaple | eandersson: you have to run post-deploy | 14:27 |
SamYaple | kolla-ansible post-deploy | 14:27 |
SamYaple | it will geenerate one for you on the deploy now | 14:27 |
SamYaple | node* | 14:27 |
rhallisey | SamYaple, nice that's good. I'm just saying that's all. I guess we'll see what sdake or other community members have to say | 14:28 |
SamYaple | i never punted this blueprint, it was always landing in mitaka | 14:28 |
eandersson | awesome missed that | 14:29 |
rhallisey | I mean you can argue mitaka has passed though | 14:29 |
SamYaple | rhallisey: and i can argue this is a crtical bug because my ceph osds dont bootstrap too | 14:30 |
SamYaple | 90% of this fixes the bootstrap, the other 10% also looks for external journals | 14:30 |
rhallisey | it will have to go to the jury then | 14:30 |
SamYaple | see if you look at the find_disks.py youll see its the one finding hte partitions now | 14:30 |
SamYaple | whereas before it was the ceph-osd script and it had all kinds of race conditions | 14:31 |
rhallisey | SamYaple, how about I -1 for a bug. And you just attach that ^ so it's clear | 14:33 |
SamYaple | if you want me to create a bug for this i can i suppose | 14:34 |
SamYaple | thats not problem | 14:34 |
rhallisey | I think it's more clear | 14:35 |
SamYaple | rhallisey: https://bugs.launchpad.net/kolla/+bug/1558853 | 14:35 |
openstack | Launchpad bug 1558853 in kolla "Bootstrapping Ceph OSDs fails" [Critical,Triaged] | 14:35 |
SamYaple | already a bug | 14:35 |
rhallisey | cool | 14:35 |
rhallisey | thanks Sam | 14:37 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Allow external ceph journals https://review.openstack.org/294862 | 14:39 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Allow user to choose filesystem for osd https://review.openstack.org/294863 | 14:39 |
*** diogogmt has quit IRC | 14:41 | |
eandersson | looks like my rabbitmq server is crashing constantly | 14:46 |
eandersson | > ERROR: epmd error for host eu-admin-guardian01: timeout (timed out) | 14:47 |
SamYaple | eandersson: is your hostname resolveable to your ip (we have a fix for this in the queue) | 14:47 |
SamYaple | if it doesnt resolve to your internface ip, but rather 127.0.0.1 this wont work | 14:47 |
eandersson | oh | 14:47 |
eandersson | it is, but it actually resolves as the wrong ip | 14:47 |
SamYaple | yes | 14:48 |
SamYaple | the fix in the queue is a check for this because rabbtiqm requires it | 14:48 |
SamYaple | ccesario_: mariadb works when deploying AIO then scaling to multinode | 14:48 |
*** eandersson has quit IRC | 14:50 | |
*** eandersson has joined #kolla | 14:50 | |
*** diogogmt has joined #kolla | 14:55 | |
eandersson | I really need to change my domain to an internal sub-domain :p | 14:56 |
*** banix has joined #kolla | 14:59 | |
eandersson | Another silly question! =] What is the easiest way to view service logs? | 14:59 |
eandersson | Do you have to exec bash to the container? | 14:59 |
eandersson | for each service | 14:59 |
eandersson | or is there an easier way to view all logs? | 14:59 |
SamYaple | the logs are in /var/lib/docker/volume/kolla_logs/_data | 15:00 |
SamYaple | there is also elasticsearch/kibana | 15:00 |
eandersson | ah nice | 15:00 |
eandersson | I did a quick find /var -name neutron but it didnt come up with anything | 15:01 |
eandersson | which makes no sense, must have had a typo | 15:01 |
*** achanda has joined #kolla | 15:03 | |
SiRiuS_ | is cinder with lvm working? | 15:04 |
SamYaple | SiRiuS_: that will not be finished for master | 15:04 |
SiRiuS_ | oh, ok | 15:04 |
SamYaple | eandersson: im not sure i follow. cd to /var/lib/docker/volume/kolla_logs/_data | 15:05 |
SiRiuS_ | thanks | 15:05 |
eandersson | yep found it | 15:05 |
SamYaple | SiRiuS_: it will merge early on in netwon tough it seems | 15:05 |
SiRiuS_ | SamYaple, does it work with ceph ? | 15:05 |
eandersson | really feels like this is the future for deploys | 15:05 |
SamYaple | SiRiuS_: cinder+ceph has worked since pre-liberty | 15:06 |
SamYaple | eandersson: glad to hear you say that. weve spent alot of time to make it so | 15:06 |
SiRiuS_ | ok, thanks i'll give it a try | 15:06 |
*** banix has quit IRC | 15:07 | |
SiRiuS_ | this is the first time I'm enabling cinder so i tried lvm, but it's good to know that it works with ceph | 15:07 |
*** achanda has quit IRC | 15:08 | |
eandersson | What is the recommended way of restarting services? | 15:08 |
*** Jeffrey4l_ has joined #kolla | 15:08 | |
*** Jeffrey4l has quit IRC | 15:09 | |
SamYaple | docker restart <container> | 15:09 |
SamYaple | if you want to update configs we also have the reconfigure command for kolla-ansible | 15:09 |
*** rstarmer has joined #kolla | 15:09 | |
SamYaple | if the container is _really_ shot you canremove the container and run deploy again for a fresh copy | 15:10 |
*** Jeffrey4l_ has quit IRC | 15:13 | |
SamYaple | rhallisey: wierd bug with mariadb | 15:14 |
SamYaple | ARGS is set in the ENV, but start.sh isnt using the options | 15:14 |
*** vhosakot has joined #kolla | 15:15 | |
rhallisey | interesting | 15:15 |
SiRiuS_ | SamYaple, what is the main difference between running kolla-ansible deploy again and running kolla-ansible reconfigure ? | 15:16 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Test reduce gate usage https://review.openstack.org/294854 | 15:16 |
SamYaple | SiRiuS_: reconfigure looks for see if the config has changes and restarts the service if it has | 15:17 |
SamYaple | deploy actually does deployment | 15:17 |
eandersson | Is there a command to stop/start all containers? It's easy to do manually using docker, but after doing a restart all I realzied that its bad haha | 15:17 |
SamYaple | lots of tasks are shared | 15:17 |
eandersson | since db, rabbit etc needs to be started first | 15:17 |
SamYaple | there is not a command to stop all container | 15:17 |
SamYaple | starting all contaienrs in teh proper order can be done by rerunning deploy | 15:18 |
SiRiuS_ | SamYaple, ok,thanks | 15:19 |
SamYaple | ahhh rhallisey i found it. its getting blanked by start.sh | 15:20 |
*** Jeffrey4l_ has joined #kolla | 15:26 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Fix mysql bootstrap https://review.openstack.org/294921 | 15:30 |
*** vhosakot has quit IRC | 15:31 | |
*** vhosakot has joined #kolla | 15:31 | |
SamYaple | rhallisey: this is critical https://review.openstack.org/294921 | 15:32 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Fix permissions for log extraction in gate https://review.openstack.org/294864 | 15:36 |
SamYaple | rhallisey: this one is also critical https://review.openstack.org/#/c/294864/ | 15:37 |
SamYaple | Jeffrey4l_: i forgot to but kolla in the path aove ^ | 15:37 |
*** rstarmer has quit IRC | 15:40 | |
Jeffrey4l_ | SamYaple, yea. I saw that. But i have no idea why we have no permission to /tmp/logs folder? | 15:40 |
SamYaple | its a symlink Jeffrey4l_ | 15:40 |
SamYaple | the actual logs folder that jenkins pulls is in a long path name | 15:41 |
SamYaple | so i made a symlink for our convince | 15:41 |
Jeffrey4l_ | SamYaple, no. check this `drwxr-xr-x. 11 jenkins jenkins 4.0K Mar 19 14:34 ..` the .. means /tmp/logs root can not change the /tmp/logs permission. | 15:44 |
Jeffrey4l_ | root have no permission :D | 15:44 |
Jeffrey4l_ | SamYaple, http://logs.openstack.org/06/294906/3/check/gate-kolla-dsvm-deploy-centos-binary/86707e5/console.html#_2016-03-19_14_34_05_764 | 15:45 |
SamYaple | Jeffrey4l_: perhaps i can just switchback to my first patch and we can merge that? | 15:48 |
SamYaple | you can figure out the specifics later | 15:48 |
SamYaple | this needs to be fixed now | 15:48 |
eandersson | Network does not look healthy. Is linuxbridge fully functional in the latest version? | 15:48 |
SamYaple | eandersson: it should be, but mind you we do not gate on LB and most of us test OVS | 15:49 |
SamYaple | so it *may* be messed up | 15:49 |
SamYaple | likely not | 15:49 |
SamYaple | lets dig into your issue | 15:49 |
Jeffrey4l_ | SamYaple, no need. your PS3 will work. | 15:49 |
SamYaple | whats the problem | 15:49 |
SamYaple | Jeffrey4l_: ok | 15:49 |
eandersson | The network is created etc | 15:49 |
eandersson | but when I try to create a VM I get a MessagingTimeout | 15:49 |
Jeffrey4l_ | root have no permission for /tmp/logs, but he has permission for /tmp/logs/kolla folder. | 15:49 |
SamYaple | that is a rabbitmq issue eandersson | 15:49 |
eandersson | When nova tries to call the validate_networks function in the API. | 15:49 |
SamYaple | is rabbitmq running? | 15:49 |
eandersson | Yes. | 15:50 |
SamYaple | docker exec -it rabbitmq rabbitmqctl cluster_status | 15:50 |
eandersson | It's all fine. | 15:50 |
SamYaple | you fixed rabbit previously right? | 15:50 |
eandersson | Yes | 15:50 |
SamYaple | youll need to restart your openstack services. they likely stopped trying to reconnect | 15:51 |
eandersson | I then restarted all containers to make sure the bindings were in a good state. | 15:51 |
SamYaple | hmmm | 15:51 |
eandersson | I first brought it all down, then started rabbit, mysql, haproxy | 15:51 |
SamYaple | what do the nova logs have to say on this? | 15:51 |
eandersson | It's nova complaning that it isn't getting a reply. | 15:51 |
SamYaple | (and the rabbitmq logs for that matter) | 15:51 |
eandersson | I'll check the logs, but really it looks like either the queue isn't created. | 15:52 |
SamYaple | that normally means you need to restart a service | 15:52 |
eandersson | Yep, restarted Neutron a couple of times now. | 15:53 |
SamYaple | i would be digging into the logs to see why its failing | 15:53 |
*** Jeffrey4l_ has left #kolla | 15:53 | |
SamYaple | maybe bad rabbitmq auth | 15:53 |
rhallisey | SamYaple, is mariadb the only case that bootstrapping needs ARGS | 15:53 |
*** Jeffrey4l_ has joined #kolla | 15:53 | |
eandersson | Nothing in RabbitMQ | 15:53 |
eandersson | All good, only bad auth was me trying to login as guest/guest at first. | 15:54 |
eandersson | If I do a grep on the req id nothing shows up, so no service is receving it. | 15:54 |
*** ccesario has joined #kolla | 15:54 | |
SamYaple | rhallisey: yes, AFAIK | 15:54 |
eandersson | Restarting all Neutorn services again to make sure. | 15:54 |
SamYaple | i changed the ARGS thing in the lights out patch | 15:54 |
SamYaple | for mariadb | 15:55 |
rhallisey | I'm just wondering if this is the best way to do this | 15:55 |
SamYaple | rhallisey: well i cant put it into config.json because its not a permemnat thing | 15:56 |
SamYaple | im not sure there is a better way, at least not this late | 15:56 |
SamYaple | but im all ears | 15:56 |
SamYaple | i agree is not the best, but now mariadb bootstrap is clearer too! | 15:57 |
SamYaple | we have a true bootstrap_mariadb container | 15:57 |
SamYaple | should help with the understanding of the process now | 15:57 |
rhallisey | by permanent do you mean only used in bootstrap or that we'll remove later | 15:57 |
SamYaple | only used in bootstrap | 15:57 |
SamYaple | so we cant have the option in config.json | 15:57 |
SamYaple | because config.json is used for bootstrap _and_ normal operation | 15:57 |
rhallisey | maybe we could do like: exec ${CMD} ${ARGS} ${BOOTSTRAP_ARGS} | 15:58 |
SamYaple | well that affects the whole start.sh script which i wont want to touch so late in the game | 15:59 |
SamYaple | technically what we are doing is how its designed to work | 16:00 |
SamYaple | originally | 16:00 |
SamYaple | i can name NEW_ARGS to BOOTSTRAP_ARGS if you want | 16:00 |
eandersson | So I stopped all Neutron services and removed the logs. | 16:00 |
SamYaple | but im opposed to chanigng start.sh | 16:00 |
vhosakot | SamYaple: How about just skipping reading / copying mariadb.log to workaround temporary Jenkins read issue.... | 16:00 |
rhallisey | ya let's go with bootstrap args | 16:00 |
SamYaple | rhallisey: outside of start.sh right? | 16:01 |
rhallisey | ya | 16:01 |
SamYaple | ok that works for me | 16:01 |
SamYaple | vhosakot: nah we have a fix, just bikeshedding on the "best" fix | 16:01 |
SamYaple | vhosakot: its just a permissions issue | 16:01 |
eandersson | and I get no errors in Neutron | 16:01 |
vhosakot | SamYaple: ah ok.. | 16:02 |
SamYaple | eandersson: do you ahve DEBUG turned on for your openstack services? | 16:02 |
SamYaple | if not i would recommend you turn it on and recheck | 16:02 |
rhallisey | SamYaple, I'll file a tech debt bug for n | 16:02 |
rhallisey | should be good to good otherwise | 16:03 |
eandersson | Is the config files shared? | 16:03 |
eandersson | *are | 16:03 |
eandersson | Similar to etc | 16:03 |
eandersson | *log | 16:03 |
SamYaple | wow eandersson i legit dont knwo what you are trying to day | 16:07 |
SamYaple | say* | 16:07 |
eandersson | How do you enable debug logging on all services? | 16:07 |
SamYaple | in your globals.yml | 16:07 |
eandersson | I was asking if there is a shared volume that controls etc for all containers. | 16:07 |
SamYaple | no | 16:07 |
SamYaple | there are some shared folders between containers, but its mostly seperate | 16:08 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Fix mysql bootstrap https://review.openstack.org/294921 | 16:11 |
eandersson | http://paste.openstack.org/show/3MP196z4PmRSwIBdlRiO/ | 16:14 |
eandersson | That's from the Nova-API | 16:14 |
*** rstarmer has joined #kolla | 16:14 | |
*** iceyao has quit IRC | 16:16 | |
vhosakot | SamYaple: saw your 2nd patch set for "Allow user to choose filesystem for osd".. so, match_mode is not needed for ceph as it is a swift variable ? | 16:20 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Change deprecated options throughout Kolla https://review.openstack.org/294931 | 16:20 |
SamYaple | vhosakot: matchmode defaults to strict, but in this case I am setting the match_mode | 16:21 |
SamYaple | eandersson: yea nova got issue talking with rabbitmq | 16:21 |
vhosakot | ah I see it, it is used in the external ceph journals patch set.. cool, thanks! | 16:21 |
SamYaple | eandersson: you _should_ see some rabbitmq logs on this | 16:21 |
eandersson | SamYaple, sure, but there is nothing in the RabbitMQ logs. | 16:22 |
eandersson | > accepting AMQP connection <0.3647.0> (172.16.0.10:34663 -> 172.16.0.10:5672) | 16:22 |
eandersson | Filled with nothing but that | 16:22 |
eandersson | There was a ERR 2 hours ago due to the hostname issue | 16:22 |
SamYaple | is this multinode eandersson ? | 16:24 |
eandersson | all-in-one | 16:24 |
eandersson | Going to capture the AMQP message to see where it is going. | 16:24 |
eandersson | hmm it's going to nova-network | 16:26 |
eandersson | lol | 16:26 |
SamYaple | wierd. kolla doesnt do nova-network | 16:26 |
eandersson | lolk | 16:27 |
eandersson | found it | 16:27 |
eandersson | > linuxnet_interface_driver = nova.network.linux_net.BridgeInterfaceDriver | 16:27 |
SamYaple | oh yea thats not right | 16:27 |
SamYaple | no wait that is correct | 16:28 |
eandersson | > linuxnet_interface_driver = nova.network.linux_net.NeutronLinuxBridgeInterfaceDriver | 16:28 |
eandersson | That is right | 16:28 |
SamYaple | yea | 16:28 |
SamYaple | so that sounds like you get to write your first kolla patch! | 16:28 |
eandersson | =]] | 16:28 |
eandersson | This has been a great learning experience!! | 16:32 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Fix linuxbridge driver https://review.openstack.org/294936 | 16:33 |
SamYaple | there you go eandersson | 16:33 |
SamYaple | thanks for testing linuxbridge for us | 16:33 |
eandersson | Thanks SamYaple | 16:33 |
SamYaple | Jeffrey4l_: rhallisey https://review.openstack.org/#/c/294864/ | 16:34 |
SamYaple | ignore the gate fialure, that is an issue with infra glean. im aware and fixing it | 16:34 |
SamYaple | that patch needs to merge to get rid of those UNSTABLE gates | 16:34 |
Jeffrey4l_ | SamYaple, done. | 16:35 |
vhosakot | eandersson: thanks for reporting the linuxbridge driver issue! | 16:37 |
eandersson | Thanks for answering my (mostly stupid) questions! =] | 16:38 |
vhosakot | haha.. there is no stupid q | 16:38 |
eandersson | :D | 16:38 |
vhosakot | :) | 16:39 |
openstackgerrit | Merged openstack/kolla: Fix permissions for log extraction in gate https://review.openstack.org/294864 | 16:39 |
eandersson | SamYaple: Re-applying with the patch is cauisng this fatal: [localhost] => error while evaluating conditional: container_state.Running == false | 16:42 |
eandersson | For task [neutron | Ensuring the neutron_linuxbridge_agent container is up] | 16:42 |
eandersson | Better to just re-create from scratch? | 16:42 |
SamYaple | eandersson: you can remove the container and run redeploy yes | 16:47 |
SamYaple | Jeffrey4l_: reconfig is busted in ceph as well | 16:47 |
SamYaple | Jeffrey4l_: we need to fix reconfig | 16:48 |
SamYaple | eandersson: reconfig is about 2 weeks old as a feature | 16:48 |
eandersson | ah =] | 16:48 |
eandersson | Well it's pretty awesome already. | 16:48 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Add memcache_servers to keystone_auth section https://review.openstack.org/294937 | 16:49 |
SamYaple | eandersson: this makes me very happy to hear | 16:50 |
SamYaple | do you use ceph? | 16:50 |
eandersson | Nope | 16:51 |
SamYaple | ok | 16:52 |
eandersson | Cinder is disabled. | 16:52 |
ccesario_ | hey SamYaple | 16:54 |
ccesario_ | I saw your message now... did you gt problem with mysql!? | 16:55 |
Jeffrey4l_ | SamYaple, i will check the ceph reconfig stuff. | 16:58 |
SamYaple | ccesario_: yes | 16:58 |
Jeffrey4l_ | and try to fix it. | 16:58 |
SamYaple | ccesario_: https://review.openstack.org/#/c/294921/ | 16:58 |
SamYaple | Jeffrey4l_: its broken when rgw is not enabled | 16:58 |
SamYaple | Jeffrey4l_: its an easy fix, i just dont have time | 16:58 |
Jeffrey4l_ | SamYaple, yup. if so, it is easy to fix. | 16:59 |
eandersson | SamYaple: https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/tasks/do_reconfigure.yml#L40 | 17:05 |
eandersson | Looks wrong :D | 17:05 |
*** achanda has joined #kolla | 17:06 | |
SamYaple | rhallisey: what do you think about this patch https://review.openstack.org/#/c/294854 ? it was generated from a list of most installed packages in our gate. it reduces gate time ~5m and size about ~600MB | 17:06 |
vhosakot | eandersson: please file a bug, I will fix it | 17:06 |
SamYaple | rhallisey: i used a similiar method for what i used last time to generate that list | 17:06 |
vhosakot | eandersson: for https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/tasks/do_reconfigure.yml#L40 thanks for pointing | 17:06 |
SamYaple | eandersson: you keep breaking it, well keep fixing it | 17:06 |
vhosakot | eandersson: linuxbridge container cannot be reconfiuged.. is this the issue ? | 17:07 |
ccesario_ | SamYaple, nice... let me try apply it and test | 17:07 |
SamYaple | ccesario_: youll need contaienr rebuild for mariadb | 17:08 |
eandersson | vhosakot: Yea, that is the issue, but I found it odd that it had name: neutron_openvswitch_agent under linuxbridge. | 17:08 |
vhosakot | eandersson: yep, saw it.. I will fix... caould file a bug please.. and thanks! | 17:09 |
vhosakot | eandersson: do you use AIO or multi ? | 17:09 |
eandersson | AIO at the moment | 17:09 |
vhosakot | eandersson: cool, can you also add the output of "brctl show" in the bug please | 17:09 |
*** achanda has quit IRC | 17:10 | |
rhallisey | SamYaple, what about cenots | 17:12 |
rhallisey | centos | 17:12 |
rhallisey | SamYaple, you can leave that to me if you like | 17:17 |
SamYaple | rhallisey: if you mean packaging, this is just for source for testing | 17:18 |
SamYaple | rhallisey: the same applies for binary, but tracking down those packages is harder for me | 17:19 |
SamYaple | rhallisey: do you think its a good idea? that list _was_ autogenerated for packages that got installed 10 or more times in our containers | 17:19 |
rhallisey | ya I do | 17:19 |
rhallisey | that way containers are literally just getting very specific pacakges | 17:20 |
SamYaple | ok. do you want to do centos binary packages in a sperate patch or would you like to coauthor? | 17:20 |
SamYaple | your call | 17:20 |
rhallisey | I'll do it later | 17:21 |
rhallisey | in a sep patch | 17:21 |
eandersson | vhosakot: I already destroyed my enviornment unfortunately. | 17:21 |
vhosakot | eandersson: cool, np.. I got the output in my env... just filing bug would be great | 17:21 |
SamYaple | rhallisey: ok | 17:23 |
vhosakot | eandersson: I can file the bug myself.. np... :) I reproduced the issue | 17:24 |
eandersson | Thanks vhosakot | 17:24 |
vhosakot | np :) | 17:24 |
*** vinkman has joined #kolla | 17:28 | |
*** rstarmer has quit IRC | 17:28 | |
*** vhosakot has quit IRC | 17:33 | |
SiRiuS_ | mandre, ping | 17:36 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Increase usefulness of openstack-base https://review.openstack.org/294854 | 17:41 |
*** jpeeler has quit IRC | 17:53 | |
*** achanda has joined #kolla | 18:03 | |
ccesario_ | SamYaple, patch applyed... and rebuilding mariadb image | 18:20 |
*** vinkman has left #kolla | 18:49 | |
*** ccesario has quit IRC | 18:58 | |
*** ccesario has joined #kolla | 19:00 | |
eandersson | SamYaple: It looks like something else was missing. | 19:04 |
eandersson | > network_api_class = nova.network.neutronv2.api.API | 19:04 |
eandersson | use_neutron for Mitaka | 19:05 |
eandersson | nvm it was there | 19:05 |
*** achanda has quit IRC | 19:06 | |
eandersson | but all good now! | 19:06 |
*** vhosakot has joined #kolla | 19:20 | |
*** achanda has joined #kolla | 19:32 | |
*** Jeffrey4l__ has joined #kolla | 19:33 | |
*** Jeffrey4l_ has quit IRC | 19:34 | |
*** ccesario has quit IRC | 19:38 | |
*** ccesario has joined #kolla | 19:46 | |
*** mdnadeem has quit IRC | 19:51 | |
*** jasonsb has quit IRC | 20:07 | |
*** vhosakot has quit IRC | 20:17 | |
*** achanda has quit IRC | 20:26 | |
*** vinkman has joined #kolla | 20:35 | |
*** sdake has joined #kolla | 20:52 | |
sdake | hey folks | 20:52 |
*** ccesario has quit IRC | 21:07 | |
*** achanda has joined #kolla | 21:12 | |
*** jasonsb has joined #kolla | 21:13 | |
*** sdake has quit IRC | 21:15 | |
SamYaple | eandersson: glad you figured it out! we made that change a few weeks ago | 21:17 |
SamYaple | ccesario_: any luck? | 21:30 |
*** vinkman has quit IRC | 21:40 | |
*** achanda has quit IRC | 21:44 | |
ccesario_ | SamYaple, sorry by delay... I needed recreate my environment ..... | 21:48 |
ccesario_ | SamYaple, yes.... it worked as expected... the deployt works without problem :) | 21:48 |
SamYaple | wonderful | 21:49 |
ccesario_ | SamYaple, thank you man by his :) | 21:49 |
ccesario_ | this* | 21:50 |
*** DuncanT has joined #kolla | 21:57 | |
DuncanT | Hi. Quick question I can't find the answer to in the docker docs: what does setting MountFlags=shared do? (or indeed private, or slave, or any other valid values). | 21:57 |
SamYaple | it allows us to create shared mountpoints | 21:58 |
SamYaple | or docker specifically | 21:58 |
SamYaple | this is a requirement for our "thin" neutorn contaienrs | 21:58 |
SamYaple | you can access the containers network namespaces from the host and other contaienrs | 21:58 |
SamYaple | basically mounts inthe container propogate up to the host | 21:58 |
DuncanT | There's no particular downside to setting it on a docker host that might be used for other docker dev work? | 21:58 |
SamYaple | nope | 21:58 |
DuncanT | Great, thanks | 21:59 |
SamYaple | a potential security problem in some situations | 21:59 |
SamYaple | but thats are really uncommon (and require mounting things from within contaienrs) | 21:59 |
*** achanda has joined #kolla | 21:59 | |
DuncanT | Not bothered by that, thanks, I'm just teaching myself docker bit by bit and don't want to break the basics | 22:00 |
SamYaple | thats a fairly advanced docker thing that _most_ people arent aware of, so you should be fine | 22:00 |
SamYaple | it just landed in 1.10 | 22:00 |
*** mbound has joined #kolla | 22:21 | |
-openstackstatus- NOTICE: Gerrit is restarting to increase performance issues | 22:22 | |
SiRiuS_ | How do | 22:32 |
SiRiuS_ | How do I deploy again with cep, after a cleanup-containers? | 22:32 |
SiRiuS_ | *ceph | 22:32 |
SiRiuS_ | I get an error at this task TASK: [ceph | Generating initial Ceph keyrings and monmap | 22:33 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Allow external ceph journals and fix bootstrap https://review.openstack.org/294862 | 22:36 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Allow user to choose filesystem for osd https://review.openstack.org/294863 | 22:36 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Add memcached_servers to keystone_auth section https://review.openstack.org/294937 | 22:36 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Change deprecated options throughout Kolla https://review.openstack.org/294931 | 22:36 |
SamYaple | rhallisey: had a bad rebase on that ceph patch ^ | 22:37 |
SamYaple | rhallisey: 12 minutes!!! full container build in the gate in 12 minutes!!! | 22:38 |
SamYaple | all kolla built in the gate in 12 minutes! | 22:38 |
*** sdake has joined #kolla | 22:53 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Change deprecated options throughout Kolla https://review.openstack.org/294931 | 22:58 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Rework heka log copying https://review.openstack.org/294206 | 23:01 |
openstackgerrit | Sam Yaple proposed openstack/kolla: Fix 'optional' config.json values https://review.openstack.org/294350 | 23:01 |
*** achanda has quit IRC | 23:13 | |
*** sdake has quit IRC | 23:14 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Change deprecated options throughout Kolla https://review.openstack.org/294931 | 23:21 |
*** sdake has joined #kolla | 23:28 | |
openstackgerrit | Sam Yaple proposed openstack/kolla: Test commit https://review.openstack.org/294965 | 23:32 |
*** sdake has quit IRC | 23:32 | |
*** sdake has joined #kolla | 23:35 | |
*** unicell1 has joined #kolla | 23:52 | |
*** unicell has quit IRC | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!