| *** rhallisey has quit IRC | 00:00 | |
| *** tpot has joined #kolla | 00:10 | |
| *** sdake has quit IRC | 00:22 | |
| *** jpeeler has quit IRC | 00:28 | |
| *** diogogmt has quit IRC | 00:35 | |
| *** sdake has joined #kolla | 00:38 | |
| *** asalkeld has joined #kolla | 00:59 | |
| *** sdake has quit IRC | 00:59 | |
| *** jasonsb_ has quit IRC | 01:04 | |
| *** mbound has quit IRC | 01:09 | |
| *** sdake has joined #kolla | 01:18 | |
| sdake | SamYaple about? | 01:21 |
|---|---|---|
| kjelly | Hi sdake | 01:25 |
| sdake | hey kjelly | 01:25 |
| sdake | anyone have any luck with 520 t2 ethernet adapters working during bootup | 01:25 |
| kjelly | sdake: I have one question. If there are two image, nova-api wtih tag 1.0.0-release-001 and glance-registry 1.0.0-release-001, and we found glance-registry had bug and fix it, Is the glance-registry tag 1.0.0-release-002? | 01:26 |
| sdake | everything gets a new tag | 01:27 |
| kjelly | sdake: I see. I found launchpad support "Helpful guidelines for reporting a bug". And I will write some tip there. How to know kolla-ansible version ? It seems that we only know commit id about ansible script version. | 01:30 |
| mandre | SamYaple, regarding https://review.openstack.org/#/c/235777/ | 01:37 |
| mandre | I don't understand the change to site.yml, can you explain it like i'm 5? | 01:38 |
| *** diogogmt has joined #kolla | 01:52 | |
| *** dims has quit IRC | 01:53 | |
| *** achanda has quit IRC | 01:55 | |
| sdake | kjelly yes we dont have any versioning determination tools of the internal bits | 01:55 |
| sdake | maybe those will come at some later date ;) | 01:55 |
| SamYaple | mandre: since rabbitmq hosts are being used to generate a list of rabbitmq servers they have to be in the list of hosts. otherwise running with --tags glance would break if rabbitmq is not on your glance hosts | 01:56 |
| SamYaple | mandre: but you reminded me i need to add extra stuff into that patch | 01:56 |
| mandre | yeah, my question was more: why do we add rabbitmq to certain hosts and not to other | 01:58 |
| bmace | looks like it is specifically for services where the roles make reference to the location of the rabbitmq servers? | 01:58 |
| bmace | well, the role related tasks anyway | 01:59 |
| SamYaple | mandre: swift keystone horizon etc dont use rabbitmq | 01:59 |
| mandre | what about glance? | 01:59 |
| SamYaple | its added to glance | 01:59 |
| mandre | yeah, but a grep on docker/glance for rabbit returns no results | 02:00 |
| mandre | is it really using rabbitmq? | 02:01 |
| SamYaple | i dont follow the question | 02:01 |
| SamYaple | glance still uses rabbitmq for ceilometer and the likes | 02:01 |
| SamYaple | but rabbitmq isnt running the glance containers | 02:02 |
| SamYaple | or anything like that | 02:02 |
| kjelly | sdake: Ok | 02:03 |
| *** tummy has quit IRC | 02:03 | |
| mandre | ok, I was expecting to see an config for rabbitmq in glance, that's why | 02:03 |
| mandre | don't you need to add rabbitmq host to murano role? | 02:06 |
| *** mbound has joined #kolla | 02:10 | |
| *** unicell has quit IRC | 02:11 | |
| *** mbound has quit IRC | 02:15 | |
| *** sdake has quit IRC | 02:17 | |
| *** jasonsb_ has joined #kolla | 02:24 | |
| *** tummy has joined #kolla | 02:33 | |
| *** tfukushima has joined #kolla | 02:41 | |
| *** asalkeld is now known as asalkeld_afk | 02:45 | |
| *** sdake has joined #kolla | 02:48 | |
| sdake | yo quick question | 02:48 |
| sdake | docker0 bridge is not present on all my machines onlly one of them | 02:48 |
| sdake | and the one it is on is causing seriou sbooting problems | 02:48 |
| sdake | booting problems | 02:48 |
| sdake | anyone have idea how to remove it | 02:48 |
| *** rajalokan has joined #kolla | 03:11 | |
| *** diogogmt has quit IRC | 03:13 | |
| openstackgerrit | Kuo-tung Kao proposed openstack/kolla: DO NOT MERGE: bug report template in launchpad https://review.openstack.org/238757 | 03:14 |
| *** tummy has quit IRC | 03:14 | |
| *** gfidente|afk has quit IRC | 03:26 | |
| *** jpeeler has joined #kolla | 03:27 | |
| *** sdake has quit IRC | 03:28 | |
| *** asalkeld_afk is now known as asalkeld | 03:29 | |
| *** tummy has joined #kolla | 03:30 | |
| *** sdake has joined #kolla | 03:38 | |
| openstackgerrit | Kuo-tung Kao proposed openstack/kolla: DO NOT MERGE: bug report template in launchpad https://review.openstack.org/238757 | 03:41 |
| *** tpot has quit IRC | 03:48 | |
| *** tpot has joined #kolla | 03:48 | |
| *** dims has joined #kolla | 03:53 | |
| sdake | so tired | 03:53 |
| sdake | i've got one box + deployment box setup with all the new gear tho | 03:54 |
| sdake | yay for progress | 03:54 |
| sdake | and actually trying a proper deploy now | 03:54 |
| *** achanda has joined #kolla | 03:55 | |
| *** mbound has joined #kolla | 03:57 | |
| sdake | 10 minutes to deploy on 10GB network | 03:59 |
| sdake | prevously took 16 mins!! | 03:59 |
| *** dims has quit IRC | 03:59 | |
| *** achanda has quit IRC | 04:00 | |
| bmace | nice | 04:01 |
| *** mbound has quit IRC | 04:01 | |
| sdake | about what i expected | 04:02 |
| sdake | wonder how long multinode will take to deploy | 04:06 |
| sdake | time to go set up the rest of my gear | 04:06 |
| * sdake groans | 04:06 | |
| *** tummy has quit IRC | 04:19 | |
| *** sdake has quit IRC | 04:21 | |
| *** achanda has joined #kolla | 04:21 | |
| *** tummy has joined #kolla | 04:22 | |
| *** sdake has joined #kolla | 04:40 | |
| *** achanda_ has joined #kolla | 04:40 | |
| *** bmace has quit IRC | 04:41 | |
| *** achanda_ has quit IRC | 04:41 | |
| *** bmace has joined #kolla | 04:41 | |
| *** achanda_ has joined #kolla | 04:42 | |
| *** achanda has quit IRC | 04:43 | |
| *** achanda_ has quit IRC | 04:44 | |
| *** sdake has quit IRC | 04:44 | |
| *** sdake has joined #kolla | 04:45 | |
| *** achanda has joined #kolla | 04:45 | |
| sdake | god oh god I hope I get first class upgrade to tokyo | 04:47 |
| * sdake so sore and old and tired and stuff | 04:48 | |
| *** tpot has quit IRC | 04:55 | |
| *** tpot has joined #kolla | 04:58 | |
| *** sdake has quit IRC | 05:13 | |
| *** diga has joined #kolla | 05:19 | |
| *** unicell has joined #kolla | 05:20 | |
| *** sdake has joined #kolla | 05:26 | |
| *** dolpher has joined #kolla | 05:31 | |
| *** achanda has quit IRC | 05:37 | |
| *** unicell has quit IRC | 05:38 | |
| *** unicell has joined #kolla | 05:38 | |
| *** tpot has quit IRC | 05:40 | |
| *** achanda has joined #kolla | 05:54 | |
| bmace | how long is the flight? | 05:56 |
| *** tummy has quit IRC | 06:03 | |
| sdake | i think 20 hours | 06:06 |
| sdake | SamYaple need you asap plz | 06:07 |
| kjelly | inc0: Are you around ? | 06:24 |
| bmace | ouch.. that is a painful flight. i recall flying to australia, 14 hours straight... i watched so many bad movies | 06:35 |
| *** diga has quit IRC | 06:39 | |
| *** inc0 has joined #kolla | 06:42 | |
| inc0 | good morning | 06:42 |
| inc0 | kjelly, around? | 06:42 |
| kjelly | inc0: Are you starting working on the bug? I have writeen some unit test code. If you are starting writting, I will give up my code. https://bugs.launchpad.net/kolla/+bug/1487390 | 06:44 |
| openstack | Launchpad bug 1487390 in kolla "unittests for build script" [High,Triaged] - Assigned to Michał Jastrzębski (inc007) | 06:44 |
| sdake | yo i can't get multinode control to deploy | 06:44 |
| sdake | need help | 06:44 |
| inc0 | kjelly, if you want, I can assign it to you | 06:45 |
| inc0 | I haven't started it so if you want, feel free to do so | 06:45 |
| kjelly | inc0: Ok. Thanks. | 06:45 |
| inc0 | anyway, my bug didn't show up when you registered yours? | 06:46 |
| inc0 | reported* | 06:46 |
| *** alisonh has quit IRC | 06:46 | |
| inc0 | sdake, whats up? | 06:46 |
| sdake | my 2nd and 3rd nodes get conn refused | 06:47 |
| inc0 | containers started? | 06:47 |
| sdake | yes | 06:47 |
| kjelly | inc0: yes, so I report another bug. | 06:47 |
| inc0 | kjelly, don't do that, if that's a duplicate, don't report new one | 06:47 |
| inc0 | just ask me what is status and such | 06:48 |
| inc0 | if I would be near finish, you would waste time writing your own code | 06:48 |
| sdake | inc0 http://ur1.ca/o5pcb -> http://paste.fedoraproject.org/282808/44558286 | 06:48 |
| sdake | inc0 http://ur1.ca/o5pcr -> http://paste.fedoraproject.org/282809/58293114 | 06:49 |
| kjelly | inc0: Sorry, I use worng word. I means I don't see your bug when I registered bug. So I report another. | 06:49 |
| inc0 | ok | 06:49 |
| inc0 | hmm...failed to establish quorum | 06:50 |
| sdake | no firewall running, running in permissive, flushed all iptables rules, same /etc/hosts on all 4 machines | 06:51 |
| sdake | ntpd running | 06:51 |
| sdake | big change to 10gb network intel 520 t2 nics | 06:51 |
| kjelly | the answer of yes/no question between English and Chinese is different. Sometimes yes in chinese means no in english ... | 06:52 |
| inc0 | well, that would be strange if that'd be nic-level problem | 06:52 |
| sdake | just pointing out the differences | 06:52 |
| inc0 | keepalived seems to work right? | 06:53 |
| sdake | dont see any problems there | 06:53 |
| sdake | telnet 192.168.1.103 (minime-03) 3306 connection refused | 06:53 |
| inc0 | well galera didnt start | 06:54 |
| inc0 | so that's expected | 06:54 |
| inc0 | however, I don't have much experience with debugging it:S | 06:54 |
| sdake | going to try rc1, that works for sure with me | 06:54 |
| sdake | are there any other errors - perhaps missing tools in the distro | 06:55 |
| inc0 | yeah try it, although I can't remember any big change between rc1 and master in galera | 06:55 |
| sdake | can yo ucomb throug hthat error log, i've been up for 24 hours | 06:55 |
| sdake | base and openstack-base changed dramatically | 06:55 |
| inc0 | yeah I'll try | 06:55 |
| inc0 | you're running fedora? | 06:55 |
| inc0 | and centos-binary? | 06:56 |
| sdake | centos7 all the way down | 06:56 |
| inc0 | ok, I'll try to run it somewhere | 06:56 |
| *** dims has joined #kolla | 06:58 | |
| *** achanda has quit IRC | 07:00 | |
| *** jmccarthy has quit IRC | 07:02 | |
| *** alisonh has joined #kolla | 07:02 | |
| *** jmccarthy has joined #kolla | 07:03 | |
| *** dims has quit IRC | 07:03 | |
| *** achanda has joined #kolla | 07:04 | |
| sdake | i sure hope its a regression then atleast it can be solved before demo d-day | 07:10 |
| sdake | inc0 are you tring to deploy 1.0.0-liberty in multinode control? | 07:11 |
| inc0 | I'll try, I only just booted 3 nodes | 07:12 |
| inc0 | centos 7 it is | 07:12 |
| sdake | i didn't test those refactors on base and openstack-base in multilnode because my gear was afu | 07:13 |
| sdake | only single node | 07:13 |
| inc0 | do you have any specific changes to globals.yml? | 07:14 |
| sdake | no | 07:14 |
| sdake | deploying rc1 now, if this works, that means there is a regression in stable | 07:15 |
| sdake | if it doesn't work it means my gear is busted in some way | 07:15 |
| *** athomas has joined #kolla | 07:17 | |
| sdake | rc1 failed | 07:18 |
| * sdake groans | 07:18 | |
| inc0 | so it's something on your end? | 07:18 |
| sdake | likely - please try on your side | 07:18 |
| inc0 | uff, I'll try to deploy multinode anyway | 07:18 |
| sdake | let me try liberty-3 | 07:20 |
| inc0 | oracle announced their containers | 07:22 |
| sdake | saw that | 07:22 |
| inc0 | cool, shame that they didn't say it was kolla | 07:22 |
| sdake | pretty hot - already one product based upon kolla and it just hit release ;) | 07:22 |
| sdake | ya but industry knows | 07:22 |
| sdake | its not like oracle has been keeping it a secret either | 07:22 |
| inc0 | my folks just wrote me "oracle beat kolla to contenerize openstack" and I was "nope, they did not" | 07:23 |
| sdake | oracle had a fully functional deployment system before they engaged with kolla | 07:23 |
| sdake | they scrapped it | 07:23 |
| sdake | working system -> scrapped -> adopted upstream | 07:24 |
| sdake | wtb that samyaple guy | 07:29 |
| sdake | time to rise! | 07:30 |
| inc0 | hehe, he's not in this timezone any more;) | 07:30 |
| sdake | niether am i | 07:31 |
| sdake | yet, awake at 12:31 am! | 07:31 |
| inc0 | you don't sleep | 07:31 |
| sdake | apparently not :( | 07:31 |
| sdake | i leave in 6 hours | 07:33 |
| sdake | for japan | 07:33 |
| sdake | so we really need to get cracking on a soslution | 07:33 |
| inc0 | ahh | 07:34 |
| inc0 | I leave on Sunday | 07:34 |
| inc0 | building containers | 07:34 |
| inc0 | right now | 07:34 |
| sdake | 1.0.0-liberty? | 07:34 |
| inc0 | stable/liberty | 07:35 |
| inc0 | hold on I'll switch tags | 07:35 |
| inc0 | (aren't these same thing?) | 07:35 |
| sdake | no they are not the same | 07:35 |
| sdake | one is a tag one is a branch | 07:35 |
| sdake | the branch has a whole bunch of extra shit in it | 07:35 |
| sdake | i think we should revisit that concept for the next several months :) | 07:36 |
| sdake | one nice hting about 10gb nic, is docker push is a whole lot faster ;) | 07:37 |
| inc0 | right now it's disk that's bottlenect probably | 07:38 |
| sdake | i have 750 pcie nvm disks | 07:38 |
| sdake | those things have dramaticlly imrpvoed my well being | 07:38 |
| sdake | wish I had another pcie slot - i'd put another 1.2 tb in for windows gaming | 07:38 |
| inc0 | lol | 07:39 |
| inc0 | when I move I'm rebuilding my pc | 07:39 |
| sdake | instead i will be using my ssds that are old school | 07:39 |
| sdake | are you moving to the us? | 07:39 |
| inc0 | 99% that I am, mid Nov | 07:39 |
| sdake | cool | 07:39 |
| sdake | which state? | 07:39 |
| inc0 | San Antonio, TX | 07:40 |
| sdake | dusty | 07:40 |
| sdake | but bareable | 07:40 |
| inc0 | and hot | 07:40 |
| inc0 | dramatic change from where I live now | 07:40 |
| *** alisonh has quit IRC | 07:41 | |
| kjelly | Well, so many people will go to us. | 07:41 |
| inc0 | with a bit of luck I'll get to see Sam there | 07:41 |
| sdake | kjelly could you expand what you eman | 07:41 |
| inc0 | he's moving away | 07:41 |
| sdake | sam moving states? | 07:42 |
| inc0 | yeah, not sure when tho | 07:42 |
| sdake | interesting hadn't heard that | 07:42 |
| inc0 | I hope I didn't spill any big secret;) | 07:43 |
| inc0 | anyway, Sam and me are moving timezones...one way or another | 07:43 |
| inc0 | I think that's what kjelly meant | 07:43 |
| *** achanda has quit IRC | 07:43 | |
| kjelly | So many one go to us timezone. Finally, no one will be online in the time. | 07:44 |
| kjelly | :( | 07:44 |
| sdake | kjelly what is your locale? | 07:45 |
| kjelly | sdake: I live in Taiwan. utc+8 | 07:45 |
| inc0 | kjelly, there will be few cores left in Euro tz | 07:47 |
| *** achanda has joined #kolla | 07:47 | |
| sdake | deploying liberty-3 now | 07:49 |
| sdake | it hardly built | 07:49 |
| sdake | but built enough ;) | 07:49 |
| inc0 | it's still building for me, no problems so far | 07:51 |
| sdake | need pcie ssds dude ;) | 07:51 |
| sdake | 1 400gb will do :) | 07:51 |
| *** achanda has quit IRC | 07:51 | |
| inc0 | it's a vm on rax | 07:51 |
| sdake | ugh | 07:51 |
| inc0 | 1.4GB pcie ssd...duh that's pretty expensive hardware | 07:54 |
| sdake | what does this mean | 07:54 |
| sdake | 151023 7:51:04 [Warning] WSREP: access file(gvwstate.dat) failed(No such file o | 07:54 |
| sdake | r directory) | 07:54 |
| sdake | 151023 7:51:04 [Note] WSREP: restore pc from disk failed | 07:54 |
| sdake | they sell a 400gb model for 380 bucks | 07:54 |
| *** alisonh has joined #kolla | 07:56 | |
| inc0 | warning and a note...doens't seem too wrong | 07:56 |
| inc0 | but again, I have little experience with galera | 07:56 |
| inc0 | sdake, https://www.percona.com/forums/questions-discussions/percona-xtradb-cluster/29033-can-only-single-node-in-centos-7 | 07:58 |
| inc0 | well, it might not be this one | 07:58 |
| inc0 | but from reading docs, it seems that bootstrap didn't work well | 08:00 |
| inc0 | can you check logs of mariadb bootstrap? | 08:00 |
| sdake | that config option is not present in the image | 08:01 |
| sdake | i ahve tried liberty-3 it also fails for different reaosns | 08:01 |
| sdake | 151023 8:02:14 [Warning] WSREP: Member 1.0 (minime-02) requested state transfer from '*any*', but it is impossible to select State Transfer donor: Resource temporarily unavailable | 08:02 |
| sdake | goign to rebuild liberty-1.0.0 now | 08:02 |
| sdake | since it looks like not a regression | 08:02 |
| inc0 | yeah, seems like some sort of io fail | 08:03 |
| *** shardy has joined #kolla | 08:03 | |
| *** mbound has joined #kolla | 08:06 | |
| inc0 | gnocci didnt build, but that's normal I think | 08:17 |
| inc0 | sdake, I have bad news, connection refused happened to me as well | 08:26 |
| sdake | groan | 08:26 |
| inc0 | https://bugs.launchpad.net/kolla/+bug/1509247 more minor but still | 08:28 |
| openstack | Launchpad bug 1509247 in kolla "tagged release wants to pull liberty-1.0.0 images but builds latest" [High,New] | 08:28 |
| inc0 | I'll start debugging galera | 08:28 |
| kjelly | Are you pull images from docker hub now? | 08:30 |
| kjelly | I want to help you. But building images always take me long time. I have poor hardware. | 08:31 |
| inc0 | I'm building them right now from liberty-1.0.0 | 08:31 |
| inc0 | you can try to deploy dockerhub and see it it works | 08:31 |
| inc0 | centos-binary | 08:31 |
| kjelly | inc0: ok | 08:32 |
| inc0 | sdake, please call docker --version | 08:32 |
| sdake | Docker version 1.8.2, build 0a8c2e3 | 08:33 |
| sdake | we are hard pinned to 1.8.2 atm | 08:33 |
| sdake | nothign else works | 08:33 |
| inc0 | duh I installed 1.8.3 | 08:33 |
| inc0 | that was my mistake | 08:33 |
| kjelly | inc0, sdake: how many total nodes do you use? | 08:34 |
| sdake | 3 nodes plus 1 to deploy from | 08:34 |
| inc0 | 3 nodes, it's multinode issue | 08:34 |
| kjelly | thanks | 08:35 |
| sdake | WSREP_SST: [INFO] Using /tmp/tmp.bLuC8zx00p as xtrabackup temporary directory (2 | 08:36 |
| sdake | wonder how big /tmp is | 08:36 |
| sdake | WSREP_SST: [ERROR] innobackupex finished with error: 1. Check /var/lib/mysql//i | 08:37 |
| sdake | nnobackup.backup.log (20151023 08:27:24.198) | 08:37 |
| sdake | http://ur1.ca/o5qsw -> http://paste.fedoraproject.org/282826/58950414 | 08:38 |
| sdake | check that shit out | 08:38 |
| inc0 | looks bad | 08:40 |
| sdake | mariadb built aug 17 | 08:42 |
| sdake | so doubt thats the problem | 08:42 |
| sdake | possible config problem? | 08:42 |
| *** kproskurin has joined #kolla | 08:43 | |
| sdake | these dirs do not look correct xtrabackup: innodb_data_home_dir = ./ | 08:43 |
| sdake | xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend | 08:43 |
| sdake | xtrabackup: innodb_log_group_home_dir = ./ | 08:43 |
| sdake | xtrabackup: innodb_log_files_in_group = 2 | 08:43 |
| sdake | xtrabackup: innodb_log_file_size = 50331648 | 08:43 |
| inc0 | yeah, probably it's looking for these files on / | 08:43 |
| sdake | ibdata1 is present: http://ur1.ca/o5qyd -> http://paste.fedoraproject.org/282830/90062144 | 08:48 |
| sdake | how do we override that innodb_data_home_dir hting? | 08:48 |
| inc0 | my.cnf would be my guess | 08:49 |
| inc0 | https://github.com/openstack/kolla/blob/master/ansible/roles/mariadb/templates/galera.cnf.j2 | 08:50 |
| inc0 | try to add config there | 08:50 |
| sdake | https://www.percona.com/forums/questions-discussions/percona-xtrabackup/8920-xtrabackup-not-pulling-all-innodb-settings | 08:50 |
| inc0 | yeah saw this one | 08:51 |
| sdake | This is a known bug with the xtrabackup SST script - no known workaround so far except to consolidate everything on a single datadir, stop mysql after SST update configuration and move the files manually to the intended locations and restart the node again. | 08:52 |
| inc0 | strange that it didn't show up earlier | 08:52 |
| sdake | https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1098566 | 08:52 |
| openstack | Launchpad bug 1098566 in MySQL patches by Codership 5.6 "SST fails with innodb_data_home_dir/innodb_log_home_dir" [Medium,Fix released] - Assigned to Alex Yurchenko (ayurchen) | 08:52 |
| * sdake shakes fists! | 08:55 | |
| sdake | [sdake@minime-01 ~]$ docker exec 4d rpm -qi percona-xtrabackup-2.3.2 | 08:57 |
| sdake | Build Date : Fri Oct 16 18:50:17 2015 | 08:58 |
| sdake | regression | 08:58 |
| sdake | they built 5 days ago | 08:58 |
| inc0 | yay | 08:58 |
| inc0 | we need to pin these versions up | 08:58 |
| *** dolpher has quit IRC | 08:58 | |
| inc0 | and yeah, same thing happening to me | 08:59 |
| inc0 | they did bump up middle version | 09:01 |
| *** pbourke has quit IRC | 09:02 | |
| sdake | lets try this version http://repo.percona.com/release/7/RPMS/x86_64/percona-xtrabackup-2.2.9-5067.el7.x86_64.rpm | 09:02 |
| sdake | no they update their whole repo without changing the repo versioning | 09:02 |
| sdake | bunch of fail | 09:02 |
| *** pbourke has joined #kolla | 09:02 | |
| inc0 | omg | 09:03 |
| sdake | holy christ we are using the el6 version of percona | 09:05 |
| sdake | rather then the l7 version | 09:05 |
| inc0 | emmm... | 09:07 |
| inc0 | why? | 09:07 |
| inc0 | I mean in dockerfile it's just package name | 09:08 |
| inc0 | so it's not our fault...it's repo | 09:08 |
| *** gfidente has joined #kolla | 09:11 | |
| *** gfidente has joined #kolla | 09:11 | |
| inc0 | pinning to 2.2.9 didn't help either | 09:11 |
| inc0 | and it's also el6 | 09:11 |
| sdake | here is the el7 version http://ur1.ca/o5rep -> http://paste.fedoraproject.org/282843/44559206 | 09:21 |
| sdake | trying percona-xtrabackup-22 | 09:25 |
| sdake | deploying percona-xtrabackup-22 now | 09:28 |
| sdake | what other code do we use from the percona repo? | 09:29 |
| sdake | only xtrabackup? | 09:29 |
| inc0 | I dont think so, let me check | 09:30 |
| sdake | inc0 ^^ | 09:30 |
| inc0 | it might be only one...MariaDB-Galera-server I'm not sure if we don't use this one as well | 09:30 |
| sdake | yum list shows you where the package comes from in an interactive process in the container | 09:31 |
| sdake | docker run -it | 09:31 |
| sdake | good news pinning to -22 works | 09:31 |
| sdake | working being not crashing during deploy ;-) | 09:32 |
| sdake | we ought to try to make master work though | 09:33 |
| inc0 | duh | 09:33 |
| sdake | if we have time in the next 4 hours | 09:33 |
| sdake | i mean master of xtrabackup | 09:33 |
| inc0 | we can take over when you get to your flight | 09:34 |
| inc0 | I'd say let's pin version to working one and figure out how to do it better later | 09:34 |
| sdake | well we definately need the el7 bits in there not el6 | 09:34 |
| sdake | let me finish with this deploy and i'll push a patch for that | 09:35 |
| sdake | here is the log of the -22 version http://ur1.ca/o5rn6 -> http://paste.fedoraproject.org/282845/55930011 | 09:36 |
| inc0 | I'm wondering what has broken, there are el7 bits in repo | 09:36 |
| inc0 | yeah seems working | 09:37 |
| sdake | notice line 29 | 09:37 |
| sdake | cd to /var/lib/mysql | 09:37 |
| sdake | it doesn't do that in 32 | 09:37 |
| inc0 | that would explain a lot | 09:38 |
| inc0 | I'll try to change config | 09:38 |
| sdake | can you pull the code and see what config option its looking for at the print statement there | 09:38 |
| inc0 | and force innodb_data_home_dir | 09:38 |
| sdake | i'd prefer to keep all that the same and make the cd operation work correctly | 09:38 |
| sdake | pull code - grep for "cd to" see what config variable is :) | 09:39 |
| inc0 | yeah I know | 09:39 |
| inc0 | I'll look at github first | 09:39 |
| sdake | wfm | 09:39 |
| sdake | rest of deploy looks to be going well | 09:39 |
| sdake | seems to work STDAKE-M-J2VL:demo sdake$ nova list | grep ACTIVE | wc -l | 09:44 |
| sdake | 8 | 09:44 |
| sdake | although github is nonresponvie for me | 09:45 |
| inc0 | same here | 09:45 |
| inc0 | storage/innobase/xtrabackup/src/xtrabackup.cc:3398 | 09:45 |
| inc0 | that's where this line is | 09:45 |
| inc0 | mysql_real_data_home is name of variable | 09:46 |
| inc0 | yay github is down -.- | 09:49 |
| *** achanda has joined #kolla | 09:49 | |
| sdake | where woudl you speculate that config variable shoudl go | 09:50 |
| sdake | so I can test it out | 09:50 |
| inc0 | I'm not quite sure it is config variable at all | 09:50 |
| kjelly | Well, it seems that github often break in Friday. | 09:50 |
| sdake | drunken github staffers ;) | 09:51 |
| sdake | "shit I wasn't supposed to type that rm command!" | 09:51 |
| sdake | inc0 log bug info here https://bugs.launchpad.net/kolla/+bug/1509281 | 09:52 |
| openstack | Launchpad bug 1509281 in kolla "percona-xtrabackup is busted in master" [Critical,Confirmed] - Assigned to Steven Dake (sdake) | 09:52 |
| inc0 | sdake, I'd guess it's basedir config | 09:54 |
| inc0 | however it's odd that it would break there | 09:54 |
| inc0 | I mean...it's important enough to break at the begging | 09:54 |
| inc0 | beggining | 09:54 |
| *** achanda has quit IRC | 09:54 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use EL7 repostitory instead of EL6 for percona https://review.openstack.org/238862 | 09:55 |
| sdake | thats wierd the bot didn't post the change I just submitted which is here for your reviewing pleasure | 09:56 |
| sdake | https://review.openstack.org/238862 | 09:56 |
| sdake | oh it did nevermind :) | 09:56 |
| sdake | inc0 do we specify basedir atm? | 09:56 |
| sdake | or is that an option we are missing | 09:56 |
| sdake | if so where should it go and what should it be ;) | 09:57 |
| sdake | so I may test it without a hard pin | 09:57 |
| inc0 | we don't | 09:57 |
| inc0 | but I'll try to do it | 09:57 |
| inc0 | without pin | 09:57 |
| sdake | tell me config i'll do it here | 09:57 |
| sdake | or pull down that el7 patch | 09:57 |
| inc0 | basedir is the config | 09:58 |
| sdake | i'll add you as coauthor on the patch if you like | 09:58 |
| inc0 | or datadir | 09:58 |
| *** masterbound has joined #kolla | 09:58 | |
| inc0 | don't so I can +2 it and we'll get it in faster | 09:58 |
| inc0 | but thanks | 09:58 |
| sdake | ok | 09:59 |
| sdake | so that goes where? | 09:59 |
| inc0 | ansible/roles/mariadb/templates/galera.cnf | 09:59 |
| sdake | galera.cnf? | 09:59 |
| sdake | ok | 09:59 |
| inc0 | I think | 09:59 |
| sdake | cleaning up now should hae a deploy in 5 mins | 09:59 |
| inc0 | now when I think about it..maybe config is busted? I mean percona may look for my.cnf and there is none | 10:00 |
| sdake | you mean maybe xtradb doesn't read the config? | 10:00 |
| inc0 | just a guess | 10:00 |
| sdake | ok i'll try basedir first | 10:01 |
| sdake | does it look for it in [mysql] ? | 10:01 |
| inc0 | basedir failed for me, but try for yourself | 10:02 |
| inc0 | also, forcing this innodb conf didnt seem to work | 10:02 |
| inc0 | [mysqld] | 10:03 |
| *** Slower_ has joined #kolla | 10:03 | |
| sdake | i read that only my.cnf is pared by xtradb | 10:03 |
| sdake | not the include dirs | 10:04 |
| sdake | maybe we should >> into /etc/my.cnf | 10:04 |
| inc0 | I'll try another thing | 10:04 |
| inc0 | yeah | 10:04 |
| *** bigjools_ has joined #kolla | 10:06 | |
| *** mbound has quit IRC | 10:07 | |
| *** bigjools has quit IRC | 10:07 | |
| *** Slower has quit IRC | 10:07 | |
| *** ByPasS has quit IRC | 10:07 | |
| *** ByPasS has joined #kolla | 10:08 | |
| sdake | FATAL ERROR: Could not find mysqld | 10:09 |
| sdake | that is with basedir | 10:09 |
| sdake | basedir is the wrong variable imo :) | 10:10 |
| inc0 | try datadir | 10:10 |
| sdake | The following directories were searched: | 10:10 |
| sdake | /var/lib/mysql//libexec | 10:10 |
| sdake | /var/lib/mysql//sbin | 10:10 |
| sdake | /var/lib/mysql//bin | 10:10 |
| *** shardy has quit IRC | 10:12 | |
| inc0 | anyway sdake which config are you checking? | 10:12 |
| sdake | what do you mean | 10:13 |
| inc0 | docker log from bootstrap? | 10:13 |
| sdake | ls /var/lib/mysql | 10:13 |
| sdake | there is a backup log in there somewhere | 10:13 |
| sdake | forget name | 10:13 |
| sdake | that last oen was docker logs | 10:13 |
| sdake | datadir works | 10:15 |
| sdake | ;-) | 10:15 |
| inc0 | worked as "deployed ok"? | 10:17 |
| sdake | http://ur1.ca/o5s8r -> http://paste.fedoraproject.org/282860/95433144 | 10:17 |
| sdake | roger | 10:17 |
| sdake | in rabbitmq | 10:18 |
| sdake | i'll let you know when it finishes if i got a full deploy | 10:18 |
| inc0 | and that's without version pin right? | 10:18 |
| sdake | yup | 10:18 |
| inc0 | cool | 10:18 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Encode datadir for xtrabackup https://review.openstack.org/238867 | 10:20 |
| sdake | pbourke acks pelase | 10:20 |
| sdake | mandre ^^ | 10:21 |
| inc0 | uff | 10:22 |
| sdake | nothign like starting off a week of travel after having been awake for 2 days sstraight | 10:22 |
| inc0 | well at least you wont have problems in getting to sleep on plane | 10:22 |
| sdake | inc0 thanks for the help on this one | 10:24 |
| inc0 | no problem | 10:25 |
| inc0 | I'm glad that we solved this one | 10:26 |
| sdake | relatively easy once we isolated the problem | 10:26 |
| sdake | since i tore apart my whole environment i wasn't sure if my environment was busted or the code was | 10:26 |
| sdake | i was thinking my env was bustedd | 10:26 |
| sdake | until you confirmed it | 10:26 |
| inc0 | try my approach;) I created vms on rax public cloud | 10:27 |
| sdake | STDAKE-M-J2VL:demo sdake$ nova list | grep ACTIVE | wc -l | 10:29 |
| sdake | 8 | 10:29 |
| *** dims has joined #kolla | 10:29 | |
| inc0 | so it's solved | 10:29 |
| inc0 | damn you percona | 10:29 |
| sdake | it was a bug in our code | 10:30 |
| sdake | not theirs | 10:30 |
| inc0 | well, everything work with defaults | 10:30 |
| sdake | but they coudl have made a sane default | 10:30 |
| inc0 | it was a default | 10:30 |
| inc0 | so if they use my.cnf but don't use their defaults, it's their bug | 10:30 |
| sdake | ya | 10:30 |
| inc0 | anyway, explicit is better than implicit, I'm ok with confirming a config | 10:31 |
| sdake | now to deploy ceph | 10:36 |
| * sdake crosses fingers | 10:36 | |
| openstackgerrit | Merged openstack/kolla: Use EL7 repostitory instead of EL6 for percona https://review.openstack.org/238862 | 10:37 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Clean up memcached container https://review.openstack.org/238874 | 10:39 |
| pbourke | got tempest running against kolla :) | 10:41 |
| sdake | sweet | 10:41 |
| sdake | how does it fare | 10:41 |
| sdake | passing or failing? | 10:41 |
| sdake | now we just need that in the gate ;-) | 10:41 |
| pbourke | that's the plan | 10:41 |
| pbourke | so far just trying glance, all seem to be passing | 10:42 |
| sdake | nice | 10:42 |
| pbourke | you at the airport sdake? | 10:43 |
| sdake | 3am | 10:43 |
| sdake | airport is 8am | 10:43 |
| sdake | god willing ceph wiill just work | 10:43 |
| sdake | and i can grab a couple hours of sleep | 10:43 |
| sdake | pbourke there is another patch fi you ahven't acked it | 10:43 |
| pbourke | sdake: stick your gchat down here if you like https://etherpad.openstack.org/p/summit-gchat | 10:44 |
| sdake | then I guess I'll tag 1.0.1 | 10:44 |
| pbourke | sdake: checking now | 10:44 |
| sdake | enjoy | 10:44 |
| pbourke | think ive acked all your commits | 10:45 |
| sdake | thanks | 10:45 |
| pbourke | something wrong here though https://review.openstack.org/#/c/238874/ | 10:45 |
| pbourke | dupe? | 10:45 |
| openstackgerrit | Merged openstack/kolla: Encode datadir for xtrabackup https://review.openstack.org/238867 | 10:45 |
| nihilifer | yeah, I submitted identical change yesterday | 10:45 |
| nihilifer | I can backport it to liberty too | 10:45 |
| sdake | yup your right - needs a backport it looks like | 10:46 |
| sdake | nhi please do so now if you can | 10:46 |
| sdake | i wnt ot tag 1.0.1 | 10:46 |
| sdake | git cherry-pick -x HASH | 10:47 |
| sdake | on stable/liberty | 10:47 |
| sdake | and not the kind you smoke ;) | 10:47 |
| inc0 | that would be...an interesting release | 10:47 |
| inc0 | cherry-pick hash...I wonder what this cake was made of | 10:48 |
| pbourke | inc0: https://etherpad.openstack.org/p/summit-gchat | 10:48 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: Remove memcached container in cleanup-containers tool https://review.openstack.org/238876 | 10:48 |
| sdake | deploy is pokoey | 10:48 |
| sdake | pokey | 10:49 |
| inc0 | ow, I'm in same hotel as Ryan | 10:49 |
| inc0 | cool, drinking won't end in bars | 10:49 |
| *** achanda has joined #kolla | 10:52 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Use EL7 repostitory instead of EL6 for percona https://review.openstack.org/238880 | 10:52 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Encode datadir for xtrabackup https://review.openstack.org/238881 | 10:52 |
| sdake | kolla just locked up durign deploy here TASK: [neutron | Ensuring OVS bridge is properly setup] *********************** | 10:54 |
| * sdake growls | 10:54 | |
| inc0 | sdake, with ceph? | 10:55 |
| sdake | yup | 10:55 |
| inc0 | what does ceph has to do with it? oO | 10:55 |
| inc0 | it was working without ceph right? | 10:55 |
| sdake | possibly dirty environment | 10:55 |
| sdake | but we need to fix those orts of thigns | 10:56 |
| *** achanda has quit IRC | 10:58 | |
| sdake | here goes take 2 of deploy after a reboot ;) | 11:00 |
| *** cloudnautique has quit IRC | 11:04 | |
| *** exploreshaifali has joined #kolla | 11:09 | |
| sdake | 18min16seconcds to deploy everything but murano and swift | 11:19 |
| sdake | with 3 node HA | 11:19 |
| inc0 | uff | 11:19 |
| sdake | pretty smoking fast | 11:19 |
| sdake | now lets see if it works ;) | 11:19 |
| sdake | 55 seconds to spawn 32 vms into ACTIVE state | 11:22 |
| sdake | cirros image | 11:22 |
| * sdake wees | 11:22 | |
| inc0 | did we somehow fixed openstack in the process? | 11:22 |
| sdake | you mean did we make openstack rock by our mere presence | 11:24 |
| sdake | the answer to that is yes! | 11:24 |
| sdake | brb checking email | 11:25 |
| openstackgerrit | Merged openstack/kolla: Use EL7 repostitory instead of EL6 for percona https://review.openstack.org/238880 | 11:25 |
| SamYaple | morning | 11:25 |
| inc0 | hello Sam | 11:25 |
| openstackgerrit | Merged openstack/kolla: Encode datadir for xtrabackup https://review.openstack.org/238881 | 11:25 |
| *** sdake_ has joined #kolla | 11:27 | |
| *** exploreshaifali has quit IRC | 11:28 | |
| *** sdake has quit IRC | 11:29 | |
| SamYaple | did anyone check ubuntu before merging and backporting that xtrabackup stuff? | 11:31 |
| SamYaple | sdake ? | 11:31 |
| *** sdake has joined #kolla | 11:31 | |
| SamYaple | inc0: pbourke ? this was a clearly not tested backport for centos, did anyone test ubuntu? | 11:32 |
| inc0 | SamYaple, we tested it | 11:32 |
| inc0 | thing is, we it was completely broken on centos | 11:32 |
| inc0 | like, it didn't even start | 11:33 |
| inc0 | but well, I see how it can affect ubuntu as well | 11:33 |
| SamYaple | that was before or after the centos change? | 11:33 |
| SamYaple | so was ubuntu tested or not? | 11:33 |
| inc0 | ubnutu with this change was nto tested | 11:33 |
| *** sdake_ has quit IRC | 11:34 | |
| inc0 | centos without this change didn't work at all | 11:34 |
| sdake | cinder is busted | 11:34 |
| SamYaple | so for all anyone knows we could have traded broken centso for broken ubuntu :/ | 11:34 |
| SamYaple | sdake: cinder is not busted | 11:35 |
| SamYaple | you probably never setup the extra types | 11:35 |
| sdake | i know nothing of extra types so your probably correct | 11:35 |
| sdake | it is in a restarting loop | 11:35 |
| SamYaple | oh thats not normal. liberty or master? | 11:35 |
| sdake | liberty | 11:35 |
| SamYaple | sdake: did you see my comments about xtrabackup testing? | 11:36 |
| sdake | ImportError: No module named oslo_policy | 11:36 |
| sdake | did not see - was checking email | 11:36 |
| sdake | looks like busted rdo packaging | 11:36 |
| SamYaple | it was not tested against ubuntu | 11:36 |
| sdake | so i'm sure they will sort it out | 11:36 |
| SamYaple | you could have broken ubuntu :/ | 11:36 |
| SamYaple | im checking now | 11:36 |
| sdake | i assumed you would get to it in the morning | 11:37 |
| sdake | which is now apparentl y:) | 11:37 |
| SamYaple | but you just merged the patches :| | 11:37 |
| SamYaple | you should have held them up before backporting | 11:37 |
| SamYaple | thats not cool | 11:37 |
| sdake | tags are what people should run not branches | 11:37 |
| SamYaple | you dont break stable branch period | 11:37 |
| sdake | i didn't break anything | 11:38 |
| SamYaple | you dont know that because you didnt test ubuntu | 11:38 |
| sdake | gates are the solution to these problems | 11:39 |
| sdake | not arguments | 11:39 |
| SamYaple | right but until then please wiat for testing | 11:39 |
| SamYaple | dont do what was just done | 11:39 |
| sdake | i apologize if you feel i slightedy ou by merging a patch that could have possiblly broken ubuntu | 11:39 |
| sdake | no harm was meant | 11:39 |
| SamYaple | so until gating you arent going to be backporting more stuff until tis tested, yes? | 11:40 |
| sdake | if you test it | 11:40 |
| sdake | wfm | 11:40 |
| SamYaple | sure. but youll make sure its tested before merging? | 11:40 |
| sdake | i cant test 8 variants of distros on every commit | 11:40 |
| SamYaple | then you cant backport it :) | 11:41 |
| SamYaple | stable branch is stable | 11:41 |
| sdake | stable branch was totally broken, also broken for ubuntu | 11:41 |
| SamYaple | not as of last night for me | 11:41 |
| sdake | did you rebuild? | 11:42 |
| SamYaple | last night i did | 11:42 |
| sdake | maybe ubuntu on a different build schedule then el7 | 11:42 |
| sdake | it does mean whatever the bits ubuntu uses are different then el7 apparently | 11:42 |
| SamYaple | so you dont know if ubuntu was broken? | 11:42 |
| sdake | because the regression happened on the 16h | 11:42 |
| SamYaple | you were just saying that? | 11:42 |
| sdake | i am going to bed | 11:42 |
| sdake | night | 11:42 |
| SamYaple | yup. please dont backport things people unless you test them | 11:43 |
| SamYaple | that is horrible practice | 11:43 |
| *** openstackgerrit has quit IRC | 11:46 | |
| *** exploreshaifali has joined #kolla | 11:46 | |
| *** openstackgerrit has joined #kolla | 11:46 | |
| *** dwalsh has joined #kolla | 12:02 | |
| *** gfidente is now known as gfidente|afk | 12:04 | |
| *** inc0 has quit IRC | 12:21 | |
| sdake | nap to short | 12:33 |
| sdake | off to the airport | 12:33 |
| sdake | sam shoot me a note if there is a problem or isn't with ubuntu please | 12:33 |
| SamYaple | i thought you were leaving earlier | 12:33 |
| sdake | i was leaving to bed | 12:33 |
| sdake | now i am leaving to the airport | 12:33 |
| SamYaple | no i mean like 12 hours ago you said you were leaving in 8 hours | 12:33 |
| SamYaple | it doesnt matter | 12:33 |
| SamYaple | safe trip | 12:33 |
| sdake | thanks | 12:34 |
| SamYaple | i leave tomorrow morning | 12:34 |
| sdake | i think my flight is t 8am | 12:34 |
| *** dims has quit IRC | 12:34 | |
| sdake | nope just popped up on screen 7:35 | 12:34 |
| sdake | flyling through ca? | 12:34 |
| *** dims has joined #kolla | 12:34 | |
| SamYaple | canada? | 12:34 |
| sdake | california | 12:34 |
| SamYaple | that seems like the opposite direction... | 12:34 |
| SamYaple | oh | 12:35 |
| SamYaple | yea | 12:35 |
| sdake | are you | 12:35 |
| SamYaple | i think | 12:35 |
| sdake | got it me too | 12:35 |
| SamYaple | is san franscisco in cali? | 12:35 |
| sdake | yup | 12:35 |
| SamYaple | then yea | 12:35 |
| SamYaple | i have a pxe boot iso that will rekick my machine in 8 minutes :) | 12:37 |
| SamYaple | 2 ssd zfs raid on encrypted storage | 12:37 |
| sdake | nice it takes me like 6 hours to reload all my machines | 12:37 |
| * sdake groans | 12:38 | |
| sdake | although that is with gear changes | 12:38 |
| SamYaple | https://github.com/SamYaple/encrypted-zfs-ubuntu-bootstrap | 12:38 |
| SamYaple | ive been patching that thing for years | 12:38 |
| SamYaple | only recently uploaded it | 12:38 |
| *** diogogmt has joined #kolla | 12:40 | |
| *** sdake has quit IRC | 12:48 | |
| SamYaple | i love 10gb | 12:49 |
| SamYaple | i fogot i had it and 1.6gb transfered so quickly | 12:50 |
| *** achanda has joined #kolla | 12:55 | |
| *** asalkeld has quit IRC | 13:00 | |
| *** achanda has quit IRC | 13:01 | |
| *** sdake has joined #kolla | 13:01 | |
| *** kjelly_ has joined #kolla | 13:03 | |
| *** diogogmt has quit IRC | 13:04 | |
| *** gfidente|afk is now known as gfidente | 13:14 | |
| *** sdake has quit IRC | 13:15 | |
| *** diogogmt has joined #kolla | 13:16 | |
| *** dwalsh has quit IRC | 13:28 | |
| *** tfukushima has quit IRC | 13:29 | |
| *** dwalsh has joined #kolla | 13:46 | |
| *** stvnoyes has quit IRC | 13:50 | |
| *** stvnoyes has joined #kolla | 13:51 | |
| *** Greg__ has joined #kolla | 14:14 | |
| *** Greg__ has left #kolla | 14:14 | |
| *** exploreshaifali has quit IRC | 14:50 | |
| *** mimizone has joined #kolla | 14:52 | |
| *** mimizone has quit IRC | 14:55 | |
| *** mimizone has joined #kolla | 14:55 | |
| *** achanda has joined #kolla | 14:58 | |
| *** shardy has joined #kolla | 15:02 | |
| *** achanda has quit IRC | 15:04 | |
| *** rajalokan has quit IRC | 15:07 | |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Add new image for tempest https://review.openstack.org/238956 | 15:08 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: [WIP] Add tempest run to the gate https://review.openstack.org/238957 | 15:09 |
| *** CBR09 has joined #kolla | 15:12 | |
| SamYaple | pbourke: you know we dont have functional testing in the gate to "see what the gate says" right? | 15:16 |
| pbourke | SamYaple: is it not running deploy_kolla.sh ? | 15:16 |
| SamYaple | nope | 15:17 |
| pbourke | damn | 15:17 |
| SamYaple | https://review.openstack.org/#/c/237745/ | 15:17 |
| SamYaple | that just merged so soonish | 15:17 |
| pbourke | ok then s/see what the gate says/see what people say/ | 15:17 |
| SamYaple | zull still has the gates as NOT_REGISTERED | 15:18 |
| SamYaple | itll take a few hours to make them active | 15:18 |
| SamYaple | pbourke: | 15:25 |
| SamYaple | failed: [server01] => {"changes": ["{\"status\":\"Pulling repository server02:8082/kollaglue/ubuntu-source-data\"}\r\n", "{\"errorDetail\":{\"message\":\"Error: image kollaglue/ubuntu-source-data:latest not found\"},\"error\":\"Error: image kollaglue/ubuntu-source-data:latest not found\"}\r\n"], "failed": true, "status": ""} | 15:25 |
| SamYaple | what was the solution here? | 15:25 |
| SamYaple | it looks like its trying to pull from dockerhub not from localregistry | 15:25 |
| pbourke | run a registry :( | 15:25 |
| SamYaple | i have a registry | 15:25 |
| pbourke | oh | 15:25 |
| pbourke | I only got it when trying to use local cache | 15:25 |
| CBR09 | SamYample: you edit options in /etc/kolla/globals | 15:27 |
| CBR09 | to point pull images from private registry | 15:27 |
| SamYaple | CBR09: thats not whats happening here | 15:28 |
| *** tummy has joined #kolla | 15:28 | |
| *** mimizone has quit IRC | 15:29 | |
| CBR09 | oh, my sorry, what problem here? | 15:30 |
| SamYaple | i dont know :) | 15:30 |
| SamYaple | oh :( | 15:30 |
| SamYaple | i never built the data image | 15:30 |
| CBR09 | are you trying pull images from private registry? | 15:31 |
| SamYaple | yea | 15:32 |
| SamYaple | but i never pushed the data image | 15:32 |
| SamYaple | so thats whats happening :/ | 15:32 |
| CBR09 | so let push data image into private registry : ) | 15:32 |
| SamYaple | very funny | 15:33 |
| CBR09 | I tried build images, push it into private registry and pull it from kolla | 15:34 |
| CBR09 | it's ok | 15:34 |
| *** tfukushima has joined #kolla | 15:34 | |
| *** unicell1 has joined #kolla | 15:37 | |
| CBR09 | ah SamYaple: i have two problems, need you advise | 15:37 |
| SamYaple | ok | 15:37 |
| CBR09 | When i've one ceph cluster, I want to use kolla with my ceph | 15:37 |
| CBR09 | what should I do? | 15:38 |
| *** unicell has quit IRC | 15:38 | |
| SamYaple | CBR09: you have an external ceph cluster? | 15:39 |
| CBR09 | yea | 15:39 |
| SamYaple | i havent written all the pieces for that yet | 15:40 |
| CBR09 | i have existing ceph cluster | 15:40 |
| SamYaple | it should be a quick thing, two flags. i wrote most everything else with that in mind | 15:40 |
| SamYaple | give me a week and ill have a patch for oyu | 15:40 |
| SamYaple | CBR09: https://blueprints.launchpad.net/kolla/+spec/ceph-improvements | 15:41 |
| CBR09 | you can specify these thing need to change | 15:41 |
| CBR09 | I can do it | 15:42 |
| CBR09 | and let I help you do it : ) | 15:42 |
| SamYaple | it woudl be quicker if i do it, but how about you write teh docs for me :) | 15:42 |
| CBR09 | let me do yourself, I need it as soon as possible: D, I will show it for you : D | 15:44 |
| CBR09 | if you appro, it should be merge | 15:45 |
| SamYaple | CBR09: we probably wont backport that. so there shouldnt be a rush | 15:46 |
| SamYaple | well be discussing backports at the summit | 15:47 |
| CBR09 | ah, I know, I'm not backport it, it's only for improve ceph feature :D | 15:49 |
| SamYaple | pbourke: i believe we have talked about this before, what was your thoughts on installing all the openstack stuff in a venv | 15:51 |
| SamYaple | it would help us avoid package conflicts on the host | 15:51 |
| CBR09 | SamYaple: another problem, about network type (vlan, vxlan, gre), what bluprint for it? | 15:54 |
| CBR09 | now it only vxlan | 15:54 |
| SamYaple | CBR09: you can override that in the augment files | 15:55 |
| SamYaple | /etc/kolla/config/neutron* | 15:55 |
| CBR09 | about these bridge?, it's auto configure right? | 15:56 |
| *** dwalsh has quit IRC | 15:57 | |
| *** achanda has joined #kolla | 16:02 | |
| pbourke | SamYaple: do you mean within the images? | 16:03 |
| *** CBR09 has quit IRC | 16:04 | |
| *** achanda has quit IRC | 16:08 | |
| SamYaple | pbourke: i dont understand your question | 16:09 |
| pbourke | dont understand yours :) | 16:09 |
| SamYaple | what was my question | 16:09 |
| pbourke | "i believe we have talked about this before, what was your thoughts on installing all the openstack stuff in a venv" | 16:09 |
| pbourke | what do you mean by all the openstack stuff | 16:10 |
| SamYaple | oh yea | 16:10 |
| SamYaple | in the image | 16:10 |
| SamYaple | all the pip installed stuff | 16:10 |
| pbourke | hmm | 16:10 |
| SamYaple | so it doesnt conflict with the host | 16:10 |
| pbourke | are we seeing many of these type of conflicts? | 16:10 |
| SamYaple | yea | 16:10 |
| SamYaple | around ceph esspecially | 16:10 |
| SamYaple | it has its own version of python packages | 16:10 |
| pbourke | would be nice not to have to do it, but cant see any major issue | 16:11 |
| *** exploreshaifali has joined #kolla | 16:12 | |
| SamYaple | yea its a bit annoying for sure | 16:13 |
| SamYaple | but the package conflicts are a bad thing | 16:13 |
| *** kjelly_ has quit IRC | 16:13 | |
| *** jerome_ has joined #kolla | 16:15 | |
| jerome_ | hi | 16:15 |
| SamYaple | hello jerome_ | 16:15 |
| jerome_ | trying to get Kolla running but having extreme problems with the endpoint/service register part of Heat | 16:15 |
| jerome_ | I'm running the latest master mirrored yesterday | 16:16 |
| jerome_ | it seems the template regarding the auth parameter in register.yml renders improperly | 16:16 |
| jerome_ | I can't seem to figure out what's going on :( | 16:16 |
| *** slagle has quit IRC | 16:19 | |
| SamYaple | jerome_: what is the exact log? | 16:22 |
| SamYaple | typically this means keystone is not working | 16:22 |
| SamYaple | is the rest of the environemtn ruinning? | 16:22 |
| jerome_ | the ansible module is complaining about parameters being incorrect | 16:22 |
| jerome_ | SamYaple: keystone is running and works | 16:22 |
| *** jerome_ has quit IRC | 16:24 | |
| *** jerome_ has joined #kolla | 16:25 | |
| jerome_ | sorry, shitty internet connection :( | 16:26 |
| jerome_ | the template line I'm referring to: https://github.com/openstack/kolla/blob/master/ansible/roles/heat/tasks/register.yml#L13 | 16:27 |
| SamYaple | what is your version of ansible? | 16:27 |
| jerome_ | I've tried with 1.9.4 first, then used the one pulled in the kolla-ansible : 2.0.0. alpha something | 16:28 |
| jerome_ | both exhibit the same problem | 16:28 |
| SamYaple | yea 2.0.0 will definetely not work | 16:28 |
| SamYaple | 1.9.4 is what you should be using | 16:28 |
| SamYaple | i just did a fresh deploy and it works file | 16:29 |
| SamYaple | fine* | 16:29 |
| *** pbourke has quit IRC | 16:29 | |
| SamYaple | my guess is you have a dirst environemnt | 16:29 |
| SamYaple | jeez yping | 16:29 |
| SamYaple | dirty | 16:29 |
| *** unicell1 has quit IRC | 16:29 | |
| SamYaple | you should purge your images and configs and try it again from a vanilia install | 16:29 |
| jerome_ | it's the exact same problem actually | 16:30 |
| jerome_ | at first, I installed ansible using pip install (instead of the git clone) | 16:30 |
| SamYaple | you shouldnt be git cloning ansible to install | 16:30 |
| jerome_ | I stumbled upon this problem, and tried to stick to the version in the Dockerfile for kolla-ansible (2.0.0 alpha), but still no luck :( | 16:31 |
| jerome_ | SamYaple: ok at least that's a good point :) | 16:31 |
| *** mimizone has joined #kolla | 16:34 | |
| jerome_ | btw, rebuilding takes ages when using git as source | 16:34 |
| jerome_ | tarball seems to be recreated every time and leads to a different checksum, confusing docker that invalidates the cache | 16:35 |
| jerome_ | tar is really a PITA: creating two tar with the same content gives two archives with different checksums :( | 16:36 |
| SamYaple | jerome_: no the checksums are the same | 16:36 |
| SamYaple | the cache doesnt work for other stupider reasons | 16:36 |
| SamYaple | but you should be building _fesh_ images | 16:36 |
| SamYaple | not using the cache | 16:37 |
| SamYaple | using the cache could very well be your issue | 16:37 |
| jerome_ | are you sure about this? because that's the reason the guys from Debian had to code a small tool named "pristine-tar" | 16:37 |
| SamYaple | its the timestamps that get you | 16:37 |
| SamYaple | but i reset those when building | 16:37 |
| SamYaple | but you are pulling _master_ tarballs, those get updated several times a day | 16:38 |
| jerome_ | arg, anyway, the build process rebuilds the openstack-base container every time I rebuild something :( | 16:38 |
| SamYaple | yea because the master repo has changed | 16:38 |
| SamYaple | promting a rebuild | 16:38 |
| jerome_ | nono, I'm not pulling tarballs, I'm cloning from git directly, using a fixed ref | 16:38 |
| SamYaple | oh that will indeed get you different results | 16:39 |
| jerome_ | ?!? | 16:39 |
| SamYaple | the git clone gives the files different timestamps | 16:39 |
| SamYaple | its always going to force a rebuild | 16:39 |
| jerome_ | even when the git repo is up to date ??? | 16:40 |
| jerome_ | or maybe the build, git clones every time it needs the source maybe ? | 16:40 |
| SamYaple | thats what it does | 16:40 |
| jerome_ | ok, that explains it | 16:41 |
| SamYaple | but either way, yes | 16:41 |
| jerome_ | thx for clarifying this | 16:41 |
| SamYaple | why are you not using taballs btw? | 16:41 |
| jerome_ | meanwhile, I rebuilt the kolla-ansible with 1.9.4 | 16:41 |
| SamYaple | those respect the docker cache | 16:41 |
| jerome_ | now I get a different issue :/ | 16:41 |
| SamYaple | jerome_: no no no, you shouldnt be changing the containers. kolla-ansible needs ansible 2.0.0 | 16:41 |
| SamYaple | your deploy host should be ansible 1.9.4 | 16:42 |
| SamYaple | you should do a fresh checkout, clean your environment of all images and do a fresh rebuild | 16:42 |
| *** tummy has quit IRC | 16:42 | |
| jerome_ | ok... be back in a minute then..... . . . . . . | 16:43 |
| jerome_ | why no tarballs: because my machine building the images is (sadly) not connected to the internet | 16:44 |
| jerome_ | so I have to clone everything :'( | 16:44 |
| *** gfidente is now known as gfidente|afk | 16:46 | |
| *** mimizone has quit IRC | 16:50 | |
| SamYaple | jerome_: we can do a file adapter as well | 16:51 |
| jerome_ | you mean like using a local archive ? | 16:51 |
| SamYaple | yea | 16:51 |
| SamYaple | well i say we _can_ | 16:51 |
| jerome_ | that would be cool :) | 16:51 |
| SamYaple | like its not there but its just a few lines of code | 16:52 |
| jerome_ | np, I get what you meant | 16:52 |
| SamYaple | would you mind filing a bug and assign it to me? | 16:52 |
| SamYaple | otherwise ill forget | 16:52 |
| jerome_ | yes, I'll do that tomorrow | 16:52 |
| jerome_ | btw, how do I propose patches ? | 16:52 |
| *** achanda has joined #kolla | 16:52 | |
| SamYaple | have you ever contributed to openstack before? | 16:52 |
| jerome_ | I'm working with Debian Jessie and found a lot of stuff missing | 16:52 |
| jerome_ | I'd like to share patches :) | 16:53 |
| *** cloudnautique has joined #kolla | 16:53 | |
| SamYaple | yes debian isn't built out at all | 16:53 |
| jerome_ | it's not that complicated, when using source anyway | 16:53 |
| SamYaple | jerome_: https://wiki.openstack.org/wiki/How_To_Contribute | 16:53 |
| SamYaple | youll have to follow that | 16:53 |
| SamYaple | the setup process to being contributing is a bit tedious | 16:53 |
| SamYaple | but its an easy workflow afterward | 16:54 |
| *** gfidente|afk has quit IRC | 16:54 | |
| jerome_ | ok! | 16:54 |
| *** CBR09 has joined #kolla | 16:54 | |
| jerome_ | about the ansible version: every host needs to be running 1.9.4 except the kolla-ansible container, am I right? | 16:55 |
| SamYaple | ansible only needs to be installed on the host system of the deployment machine | 16:56 |
| SamYaple | the only requrements for the machines running the containres is docker and docker-py | 16:57 |
| CBR09 | Thank Sam for your advise, I go to bed | 16:57 |
| jerome_ | ok, in my case, I use an self-deploy, meaning I use the same machine for building and deploying | 16:57 |
| jerome_ | maybe that's the cause ? | 16:58 |
| jerome_ | it's working! | 16:58 |
| jerome_ | I rebuilt everything and now templates are working | 16:59 |
| *** CBR09 has quit IRC | 16:59 | |
| jerome_ | too many moving pieces, I guess that was the cause: thank you very much for your help! | 16:59 |
| SamYaple | youre welcome | 17:03 |
| *** unicell has joined #kolla | 17:03 | |
| *** mimizone has joined #kolla | 17:04 | |
| jerome_ | damn, running the generated ansible localhost command inside kolla-ansible works but now using tools/kolla-ansible I get a "error while evaluating conditional: heat_endpoint.stdout.split()[2]" | 17:06 |
| *** athomas has quit IRC | 17:08 | |
| *** achanda has quit IRC | 17:13 | |
| jerome_ | heat role is missing the meta/ common dependencies :( | 17:16 |
| *** shardy has quit IRC | 17:22 | |
| *** kproskurin has quit IRC | 17:22 | |
| *** mimizone has quit IRC | 17:25 | |
| *** achanda has joined #kolla | 17:25 | |
| *** mimizone has joined #kolla | 17:26 | |
| *** tfukushima has quit IRC | 17:32 | |
| *** cloudnautique has quit IRC | 17:40 | |
| *** cloudnautique has joined #kolla | 17:42 | |
| SamYaple | jerome_: sounds like a great first patch | 17:49 |
| *** dwalsh has joined #kolla | 18:05 | |
| *** vinkman has joined #kolla | 18:18 | |
| *** masterbound has quit IRC | 18:22 | |
| *** dims_ has joined #kolla | 18:37 | |
| *** dims has quit IRC | 18:40 | |
| *** dims_ has quit IRC | 18:42 | |
| *** cfarquhar has joined #kolla | 18:56 | |
| *** cfarquhar has quit IRC | 18:56 | |
| *** cfarquhar has joined #kolla | 18:56 | |
| *** jasonsb_ has quit IRC | 18:58 | |
| *** dims has joined #kolla | 19:10 | |
| *** achanda has quit IRC | 19:17 | |
| *** dims has quit IRC | 19:17 | |
| *** tummy has joined #kolla | 19:25 | |
| *** cloudnautique has quit IRC | 19:26 | |
| *** pbourke has joined #kolla | 19:33 | |
| *** jasonsb_ has joined #kolla | 19:36 | |
| *** jasonsb_ has quit IRC | 19:36 | |
| *** jasonsb_ has joined #kolla | 19:36 | |
| *** tummy has quit IRC | 19:58 | |
| *** tummy has joined #kolla | 19:58 | |
| *** tummy has quit IRC | 20:00 | |
| *** cloudnautique has joined #kolla | 20:01 | |
| *** tummy has joined #kolla | 20:01 | |
| *** pbourke has quit IRC | 20:06 | |
| *** dwalsh has quit IRC | 20:14 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix broken galera https://review.openstack.org/239098 | 20:15 |
| SamYaple | hey any cores around. need an immediate merge | 20:15 |
| SamYaple | coolsvap mandre ? | 20:16 |
| *** vinkman has left #kolla | 20:16 | |
| *** achanda has joined #kolla | 20:17 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Removed unneeded tox things https://review.openstack.org/239100 | 20:20 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Update the super function name https://review.openstack.org/239101 | 20:22 |
| *** achanda has quit IRC | 20:23 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Add mariadb to the gate https://review.openstack.org/239102 | 20:23 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Update the super function name https://review.openstack.org/239101 | 20:24 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Removed unneeded tox things https://review.openstack.org/239100 | 20:24 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Add missing images to default build profile https://review.openstack.org/239103 | 20:25 |
| *** dwalsh has joined #kolla | 20:32 | |
| *** dims has joined #kolla | 20:37 | |
| *** mbound has joined #kolla | 20:40 | |
| bmace | changes look pretty good sam | 20:46 |
| SamYaple | bmace: https://jenkins07.openstack.org/job/gate-kolla-deploy-images-ubuntu-source/1/console | 20:47 |
| SamYaple | successfully :) | 20:47 |
| *** bmace has quit IRC | 20:53 | |
| *** achanda has joined #kolla | 20:59 | |
| *** tummy has quit IRC | 21:01 | |
| *** bmace has joined #kolla | 21:09 | |
| *** dwalsh has quit IRC | 21:11 | |
| *** jasonsb_ has quit IRC | 21:32 | |
| SamYaple | jpeeler: https://review.openstack.org/#/c/239098/ | 21:46 |
| SamYaple | thats a "fix master and stable because its totally busted" patch | 21:47 |
| jpeeler | AH | 21:47 |
| jpeeler | i was looking for that other day, but my kolla time has been limited of late | 21:47 |
| SamYaple | well sdake fixed if for centos.... but broke it for ubuntu in the process | 21:47 |
| *** dims_ has joined #kolla | 21:49 | |
| *** dims has quit IRC | 21:52 | |
| *** dims_ has quit IRC | 21:53 | |
| *** dims has joined #kolla | 22:06 | |
| *** achanda has quit IRC | 22:07 | |
| *** exploreshaifali has quit IRC | 22:15 | |
| *** jasonsb_ has joined #kolla | 22:17 | |
| *** jasonsb_ has joined #kolla | 22:18 | |
| *** slagle has joined #kolla | 22:21 | |
| *** jasonsb_ has quit IRC | 22:22 | |
| *** achanda has joined #kolla | 22:50 | |
| *** cloudnautique has quit IRC | 22:55 | |
| *** dims has quit IRC | 22:57 | |
| *** cloudnautique has joined #kolla | 23:02 | |
| *** slagle has quit IRC | 23:05 | |
| *** dims has joined #kolla | 23:25 | |
| *** achanda has quit IRC | 23:29 | |
| *** achanda has joined #kolla | 23:29 | |
| *** dims_ has joined #kolla | 23:30 | |
| *** dims has quit IRC | 23:34 | |
| *** mimizone has quit IRC | 23:37 | |
| *** jongchoi has joined #kolla | 23:38 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!