| *** markvoelker has quit IRC | 00:08 | |
| *** dxiri has joined #openstack-ansible | 00:11 | |
| *** yifei1 has joined #openstack-ansible | 00:16 | |
| *** vnogin has quit IRC | 00:18 | |
| *** thorst has joined #openstack-ansible | 00:18 | |
| *** thorst has quit IRC | 00:20 | |
| *** Jeffrey4l has quit IRC | 00:42 | |
| *** Jeffrey4l has joined #openstack-ansible | 00:42 | |
| *** dxiri has quit IRC | 00:45 | |
| *** dxiri has joined #openstack-ansible | 00:47 | |
| *** dxiri has quit IRC | 00:58 | |
| *** markvoelker has joined #openstack-ansible | 01:05 | |
| *** kiennt26 has joined #openstack-ansible | 01:09 | |
| *** xingchao has joined #openstack-ansible | 01:21 | |
| *** markvoelker has quit IRC | 01:38 | |
| *** hw_wutianwei has joined #openstack-ansible | 01:42 | |
| *** xingchao has quit IRC | 01:45 | |
| *** xingchao has joined #openstack-ansible | 01:47 | |
| *** xingchao has quit IRC | 01:56 | |
| *** xingchao has joined #openstack-ansible | 02:00 | |
| *** gkadam has quit IRC | 02:06 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_freezer master: Updated from global requirements https://review.openstack.org/514258 | 02:19 |
|---|---|---|
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_keystone stable/pike: Add security headers to web accessable services. https://review.openstack.org/513973 | 02:20 |
| *** thorst has joined #openstack-ansible | 02:21 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Improve execution time by using dynamic groups https://review.openstack.org/513876 | 02:21 |
| *** thorst has quit IRC | 02:25 | |
| *** sxc731 has joined #openstack-ansible | 02:30 | |
| *** sxc731 has left #openstack-ansible | 02:32 | |
| *** markvoelker has joined #openstack-ansible | 02:35 | |
| *** chason has joined #openstack-ansible | 02:37 | |
| *** dxiri has joined #openstack-ansible | 02:46 | |
| *** xingchao has quit IRC | 03:03 | |
| *** markvoelker has quit IRC | 03:09 | |
| *** dxiri has quit IRC | 03:22 | |
| *** gkadam has joined #openstack-ansible | 03:32 | |
| *** gkadam has quit IRC | 03:32 | |
| *** gkadam has joined #openstack-ansible | 03:32 | |
| *** xingchao has joined #openstack-ansible | 03:43 | |
| *** udesale has joined #openstack-ansible | 03:45 | |
| *** dave-mccowan has quit IRC | 03:48 | |
| *** Chealion has joined #openstack-ansible | 03:52 | |
| *** xingchao has quit IRC | 04:01 | |
| *** markvoelker has joined #openstack-ansible | 04:06 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_nova master: Update test image for nova-lxd https://review.openstack.org/515020 | 04:07 |
| *** nshetty has joined #openstack-ansible | 04:08 | |
| *** Dinesh_Bhor has joined #openstack-ansible | 04:14 | |
| *** armaan has joined #openstack-ansible | 04:19 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Reduce the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 04:21 |
| *** thorst has joined #openstack-ansible | 04:22 | |
| *** huxinhui__ has joined #openstack-ansible | 04:26 | |
| *** thorst has quit IRC | 04:27 | |
| *** armaan has quit IRC | 04:37 | |
| *** chhavi has joined #openstack-ansible | 04:37 | |
| *** markvoelker has quit IRC | 04:39 | |
| *** pmannidi is now known as pmannidi|bomgar | 04:53 | |
| *** pmannidi|bomgar is now known as pmannidi | 04:53 | |
| *** udesale__ has joined #openstack-ansible | 04:59 | |
| *** udesale has quit IRC | 04:59 | |
| *** armaan has joined #openstack-ansible | 04:59 | |
| *** ivveh has joined #openstack-ansible | 05:29 | |
| *** markvoelker has joined #openstack-ansible | 05:37 | |
| *** sm806 has joined #openstack-ansible | 05:46 | |
| *** threestrands_ has joined #openstack-ansible | 05:47 | |
| *** sm806 has quit IRC | 05:49 | |
| *** sm806 has joined #openstack-ansible | 05:49 | |
| *** threestrands has quit IRC | 05:50 | |
| *** lhinds has quit IRC | 05:53 | |
| *** taseer4 is now known as Taseer | 05:58 | |
| *** hw_wutianwei has quit IRC | 05:59 | |
| *** br0k3n has joined #openstack-ansible | 06:08 | |
| *** markvoelker has quit IRC | 06:10 | |
| *** udesale__ has quit IRC | 06:14 | |
| *** thorst has joined #openstack-ansible | 06:23 | |
| *** thorst has quit IRC | 06:27 | |
| *** hamzy has quit IRC | 06:28 | |
| *** jbadiapa has joined #openstack-ansible | 06:29 | |
| *** sm806 has quit IRC | 06:39 | |
| *** sm806 has joined #openstack-ansible | 06:40 | |
| *** hamzy has joined #openstack-ansible | 06:42 | |
| *** threestrands_ has quit IRC | 06:51 | |
| *** drifterza has joined #openstack-ansible | 07:07 | |
| *** markvoelker has joined #openstack-ansible | 07:07 | |
| drifterza | morning all | 07:08 |
| *** Oku_OS-away is now known as Oku_OS | 07:08 | |
| *** pmannidi has quit IRC | 07:13 | |
| *** huxinhui__ is now known as huxinhui | 07:14 | |
| *** Miouge has joined #openstack-ansible | 07:24 | |
| *** nshetty is now known as nshetty|lunch | 07:32 | |
| *** hachi has joined #openstack-ansible | 07:38 | |
| *** markvoelker has quit IRC | 07:40 | |
| *** br0k3n has left #openstack-ansible | 07:40 | |
| *** jvidal has joined #openstack-ansible | 07:46 | |
| *** vnogin has joined #openstack-ansible | 07:59 | |
| *** drifterza has quit IRC | 08:02 | |
| Taseer | evrardjp: Good Morning. Can you please let me know about the next steps for Congress role ? | 08:03 |
| gunix | i rebooted the entire stack, all the VMs. i had 3 controllers. now (of course) galera doesn't work. is there any way to bootstrap it? | 08:06 |
| *** mcarden has joined #openstack-ansible | 08:18 | |
| *** mbuil has joined #openstack-ansible | 08:18 | |
| *** thorst has joined #openstack-ansible | 08:24 | |
| *** yifei has quit IRC | 08:24 | |
| *** yifei1 is now known as yifei | 08:24 | |
| *** yifei has quit IRC | 08:24 | |
| *** yifei has joined #openstack-ansible | 08:25 | |
| *** vnogin has quit IRC | 08:26 | |
| sm806 | gunix: try attaching to galera container and restart the service. I am not sure what the service name is called. You may execute this command to find out all the registered services in ubuntu "systemctl list-units --type service --all" | 08:26 |
| *** udesale has joined #openstack-ansible | 08:27 | |
| *** thorst has quit IRC | 08:28 | |
| *** hachi has quit IRC | 08:28 | |
| *** armaan_ has joined #openstack-ansible | 08:28 | |
| *** hachi has joined #openstack-ansible | 08:29 | |
| *** armaan__ has joined #openstack-ansible | 08:30 | |
| *** armaan has quit IRC | 08:30 | |
| *** armaan_ has quit IRC | 08:33 | |
| *** flaviosr has quit IRC | 08:35 | |
| *** drifterza has joined #openstack-ansible | 08:35 | |
| *** markvoelker has joined #openstack-ansible | 08:37 | |
| *** flaviosr has joined #openstack-ansible | 08:38 | |
| *** evrardjp has quit IRC | 08:40 | |
| *** evrardjp has joined #openstack-ansible | 08:41 | |
| *** mbuil has quit IRC | 08:43 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/pike: Use new zuul v3 git src location https://review.openstack.org/516213 | 08:45 |
| *** yifei has quit IRC | 08:47 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Zuul: add file extension to playbook path https://review.openstack.org/516068 | 08:49 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/newton: Zuul: add file extension to playbook path https://review.openstack.org/516070 | 08:50 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/ocata: Zuul: add file extension to playbook path https://review.openstack.org/516072 | 08:50 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/pike: Zuul: add file extension to playbook path https://review.openstack.org/516074 | 08:51 |
| *** mrch has joined #openstack-ansible | 08:54 | |
| SamYaple | odyssey4me: so i have my etcd+galera stuff going strong. it does need etcd3, which it looks like the etcd playbooks available are going to lay down etcd2. so to move forward that would need to be resolved. but otherwise its all good! | 08:56 |
| SamYaple | fully automated recovery/restart. can recover from split-brain and full cluster stopage automagically | 08:56 |
| odyssey4me | SamYaple I just looked through the scrollback - thanks for feeding back. I'll try and take a peek at the stuff asap. | 08:57 |
| *** SmearedBeard has joined #openstack-ansible | 08:57 | |
| *** nshetty|lunch is now known as nshetty|BMGR | 08:58 | |
| SamYaple | yep. it can be boiled down to a single service running along side mysqld in the container. its a touch more convoluted in my repo because im using upstream docker images without modifications, so i use two containers and file-based communication | 08:58 |
| SamYaple | OSA will not need that though | 08:58 |
| gunix | sm806: you normally start galera with new_galera_cluster after changing safe_to_bootstrap to 1 within "/var/lib/mysql/grastate.dat" | 08:58 |
| gunix | but that did not work | 08:58 |
| odyssey4me | SamYaple hmm, I see you're using autocluster for the rabbitmq clustering... I though that was deprecated for 3.7 and it now has a replacement built-in ? | 08:58 |
| SamYaple | odyssey4me: autocluster code is included in 3.7.0, yes | 08:58 |
| SamYaple | but im not running 3.7.0 | 08:59 |
| SamYaple | config is a bit different, but the acutal work is the same | 08:59 |
| odyssey4me | ah ok, makes sense | 08:59 |
| SamYaple | and autocluster as a plugin (same config) *does* work for 3.7.0, for a migration period | 09:00 |
| gunix | how do you guys start galera cluster after shutting down all 3 nodes? | 09:03 |
| *** pbandark has joined #openstack-ansible | 09:03 | |
| SamYaple | gunix: find the highest seqno and start that node | 09:03 |
| gunix | SamYaple: start the node how? galera_new_cluster> | 09:03 |
| gunix | ? | 09:03 |
| gunix | check above, i already changed safe_to_bootstrap to 1 and tried to start the cluster, but didn't work | 09:04 |
| SamYaple | gunix: https://github.com/SamYaple/home-salt/blob/master/salt/files/mariadb/lib.sh#L30-L33 those lines run on each node (while shutdown) will give you the seqno from each node | 09:05 |
| SamYaple | find the highest one(s), pick one to start by changing safe_to_bootstrap 1 in grastate.dat, then --wsrep-new-cluster *will* start it | 09:05 |
| SamYaple | barring other errors which should be evident in logs | 09:05 |
| gunix | mysqld --defaults-file=/etc/mysql/my.cnf --wsrep-recover --skip-networking | 09:06 |
| gunix | you mean this | 09:06 |
| gunix | what's the command to start the cluster | 09:06 |
| gunix | ? | 09:06 |
| gunix | i was able to do this on mariadb in my own environment. maybe it's different on mysql, no ideea | 09:06 |
| gunix | i don't know why osa uses mysql anyway. everybody uses mariadb now | 09:07 |
| *** SmearedBeard has quit IRC | 09:07 | |
| SamYaple | gunix: those 4 lines i pasted are for discovering the seqno from an improperly stopped database | 09:07 |
| SamYaple | gunix: it is mariadb silly | 09:07 |
| SamYaple | galera_new_cluster should be available on systemd hosts | 09:08 |
| gunix | ok, galera_new_cluster does not work | 09:08 |
| gunix | after changing safe_to_bootstrap to 1 | 09:08 |
| SamYaple | what do the logs say about ti not working? | 09:08 |
| SamYaple | you can always run it directly with `mysqld --wsrep-new-cluster` | 09:08 |
| *** markvoelker has quit IRC | 09:10 | |
| gunix | logs are useless: https://bpaste.net/show/b9a2b897c4f4 | 09:13 |
| *** mbuil has joined #openstack-ansible | 09:13 | |
| SamYaple | those arent the logs | 09:13 |
| SamYaple | journalctl -u mariadb and /var/log/mysql should give you more info | 09:14 |
| SamYaple | `systemctl set-environment _WSREP_NEW_CLUSTER='--wsrep-new-cluster' && systemctl start mariadb && systemctl set-environment _WSREP_NEW_CLUSTER=''` thats what galera_new_cluster is doing behind the scenes btw | 09:15 |
| odyssey4me | gunix um, this is mariadb ... not mysql | 09:15 |
| SamYaple | being a fork of mysql, mariadb still has lots of bits named mysql | 09:16 |
| gunix | ok i think i found the error withi mysql log | 09:17 |
| gunix | 2017-10-30 11:15:20 140100183202048 [ERROR] WSREP: failed to open gcomm backend connection: 98: error while trying to listen 'tcp://0.0.0.0:4567?socket.non_blocking=1', asio error 'bind: Address already in use': 98 (Address already in use) | 09:17 |
| gunix | mariadb is already linstening but i can't shut it down | 09:17 |
| gunix | i'll have to kill the service i guess | 09:17 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Use zuul v3 src location to install ARA https://review.openstack.org/515023 | 09:20 |
| *** SmearedBeard has joined #openstack-ansible | 09:21 | |
| *** evrardjp has quit IRC | 09:22 | |
| *** evrardjp has joined #openstack-ansible | 09:23 | |
| SamYaple | odyssey4me: so just a heads up, this is the meat of the logic. its actually quite simple https://github.com/SamYaple/home-salt/blob/master/salt/files/mariadb/etcd.sh#L31-L54 | 09:24 |
| SamYaple | it uses etcd3 locking to ensure only one is doing a bootstraping task at any time | 09:24 |
| SamYaple | as long as the gcomm list is up to date with all the nodes in the cluster, it should all just work fine | 09:25 |
| SamYaple | the only reason galera cant autorecover is because it doesnt know for *sure* what all the nodes in the cluster are, but if we do know for sure, then its all safe | 09:26 |
| SamYaple | discounting human intervention of course (we really cant prevent an operator from runnign --wsrep-new-cluster on multiple hosts, if they want to break it, they will) | 09:26 |
| gunix | SamYaple: it was some sort of bug with mariadb | 09:27 |
| gunix | it was refusing to properly close and still listening on tcp | 09:27 |
| gunix | but was not listening on unix socket | 09:28 |
| gunix | i had to kill the process and that allowed me to start galera | 09:28 |
| SamYaple | thats not a bug | 09:28 |
| SamYaple | you can start the service, but it wont listen on 3306 or the unix socket until it joins a cluster (or you start with --wsrep-new-cluster) | 09:29 |
| SamYaple | if systemd somehow got disconnected from the running process so it didnt know the status, that might be a bug. but thats not related to mysqld/mariadb itself | 09:29 |
| SamYaple | that would be packaging | 09:29 |
| *** shardy has joined #openstack-ansible | 09:34 | |
| gunix | SamYaple: what if it listents on 4567 but not on 3306 and not on unix socket, and it doesn't stop when you do systemctl stop. is that a bug? | 09:37 |
| SamYaple | gunix: no thats what i just described | 09:38 |
| SamYaple | it sounds like systemd got out of sync with the running process (lost the pid somehow) so thats why systemctl stop didnt work, but the mysqld process was doign what its supposed to | 09:39 |
| *** flaviosr has quit IRC | 09:42 | |
| *** flaviosr has joined #openstack-ansible | 09:45 | |
| gunix | SamYaple: so 4567 is the galera cluster, right? so that started but mysql won't listen untill all nodes are up | 09:47 |
| gunix | SamYaple: you seem to know a lot about galera. i have another question: what if the cluster has 3 nodes and 2 nodes go down. will the 3rd node (still alive) stop writing, since it doesn't have a quorum any more? | 09:47 |
| SamYaple | gunix: 4567 is the default wsrep port for sst, 4568 is for ist, and 4444 is used by xtrabackup. these are all part of wsrep and internal to the mysqld process | 09:48 |
| SamYaple | gunix: that depends, did the other 2 nodes *crash*? if thats the case, then yes | 09:48 |
| SamYaple | if they were cleanly shutdown then the 3 node will continue to write out | 09:48 |
| gunix | SamYaple: oooh so if i just do shutdown -h now on the other nodes, it's considered safe? | 09:56 |
| SamYaple | i suppose that depends on how the clusters get stopped. `mysqladmin shutdown` is the only way i trust to stop it safely | 09:57 |
| SamYaple | everything else has timeouts that will kill it if it doesnt respond in time | 09:57 |
| gunix | STONITH | 09:57 |
| SamYaple | you can do that with galera though. its not going to hurt the data | 09:57 |
| SamYaple | it might stop accepting writes though | 09:58 |
| *** vnogin has joined #openstack-ansible | 10:01 | |
| gunix | do what with galera> | 10:04 |
| gunix | ? | 10:04 |
| SamYaple | shoot the other node in the head | 10:04 |
| gunix | oh | 10:04 |
| SamYaple | im assuming thats what you were going for | 10:04 |
| gunix | yea | 10:04 |
| SamYaple | worst thing that you can do is for a breif period of time get some reads off a cluster out of quorum and possible get stale data | 10:05 |
| gunix | i mean, "stonith" is kind of confusing for me, as a term. does it mean that one node literally shuts down the other node, or just ignores it's existance? | 10:05 |
| SamYaple | but you can't write to it | 10:05 |
| SamYaple | ive always read it as "forget about the other node, this one needs to work now" | 10:06 |
| *** vnogin has quit IRC | 10:06 | |
| SamYaple | its not a great acronym to describe anything useful | 10:06 |
| *** nshetty|BMGR is now known as nshetty | 10:07 | |
| *** markvoelker has joined #openstack-ansible | 10:07 | |
| *** jafeha has joined #openstack-ansible | 10:09 | |
| jafeha | good morning | 10:09 |
| *** taseer1 has joined #openstack-ansible | 10:09 | |
| *** Taseer has quit IRC | 10:10 | |
| *** taseer2 has joined #openstack-ansible | 10:10 | |
| *** vnogin has joined #openstack-ansible | 10:10 | |
| *** kiennt26 has quit IRC | 10:10 | |
| *** taseer1 has quit IRC | 10:14 | |
| evrardjp | taseer2: you're flapping usernames :) | 10:18 |
| *** gkadam_ has joined #openstack-ansible | 10:20 | |
| *** udesale has quit IRC | 10:20 | |
| *** gkadam_ has quit IRC | 10:20 | |
| Neptu_ | evrardjp: hej I saw you are the responsable guy of building https://github.com/openstack/openstack-ansible-os_monasca-agent. Awesome | 10:21 |
| openstackgerrit | Tuan Luong-Anh proposed openstack/openstack-ansible master: Using --option ARGUMENT https://review.openstack.org/516240 | 10:22 |
| evrardjp | Neptu_: I am not ? | 10:22 |
| Neptu_ | evrardjp: I saw you contributing to that repo | 10:23 |
| Neptu_ | ot last contribution | 10:23 |
| Neptu_ | :D | 10:23 |
| evrardjp | yes I did contribute recently to that repo, but not for any functional side | 10:23 |
| gunix | guys, how do i install the openstack clinet and the openrc file on one node using ansible? | 10:24 |
| gunix | using OSA | 10:24 |
| evrardjp | this role is 1 year old, and has only 25 commits, most of them automated... Much work to be done there | 10:24 |
| evrardjp | gunix: there is already one in your utility container gunix | 10:24 |
| evrardjp | woops said your nick twice | 10:24 |
| evrardjp | gunix: connect to this container and everything is available | 10:25 |
| Neptu_ | evrardjp: yes I noticed is not much action there... | 10:25 |
| *** thorst has joined #openstack-ansible | 10:25 | |
| gunix | evrardjp: you can say my nick how many times you want. just be careful, if you say the name more then 3 times, you get a random virus on your windows machine | 10:25 |
| evrardjp | gunix: good I have none. | 10:25 |
| evrardjp | :) | 10:25 |
| gunix | if you don't use windows, you are safe. also, if you use windows, don't say windows 3 times, because nsa will randomly post your nude pics on a porn site | 10:25 |
| Neptu_ | evrardjp: Im trying to make an update of my aoi to include the client once I modify ansible-role-requirements.yml witht he monasca role how i update the system? | 10:27 |
| evrardjp | I have no clue how monasca works Neptu_, I am very sorry | 10:27 |
| evrardjp | but I suppose we need a playbook that target the nodes that uses this role? | 10:27 |
| evrardjp | no clue... | 10:27 |
| gunix | evrardjp: found the container: (openstack) hypervisor list | 10:28 |
| gunix | Unknown Error (HTTP 503) | 10:28 |
| evrardjp | gunix: should be in your infra | 10:28 |
| Neptu_ | evrardjp: I thought you can include it on a main deployment same as the asnible role requirements??? | 10:28 |
| gunix | i guess i should not shutdown all 3 controller nodes :D | 10:28 |
| evrardjp | gunix: shutting down all your controllers is NOT a great idea indeed. | 10:28 |
| evrardjp | :p | 10:28 |
| evrardjp | I have freenode connection issues since the whole morning I will reboot my machine. | 10:29 |
| evrardjp | Neptu_: including it in a-r-r will just ensure the role is properly downloaded/ready for its usage | 10:30 |
| evrardjp | after that, you do whatever you want | 10:30 |
| *** thorst has quit IRC | 10:30 | |
| Neptu_ | evrardjp: ok then I will need to modify the playbooks to be installed on one host or in all | 10:31 |
| Neptu_ | now i understand you | 10:32 |
| gunix | evrardjp: what if i have only 1 controller? instead of 3? can it be shutdown | 10:32 |
| evrardjp | gunix: controllers are the MOST important part of openstack | 10:32 |
| SamYaple | gunix: yes | 10:32 |
| evrardjp | it's your infrastructure | 10:32 |
| gunix | evrardjp: SamYaple: ok, but if i have 3 controllers, i should NOT shut down all 3, only 2 ... and always have 1 online... right | 10:33 |
| gunix | ? | 10:33 |
| evrardjp | you're in degraded mode, which is not great. Keep your infra online :) | 10:33 |
| SamYaple | gunix: well you should only shutdown 1 | 10:33 |
| SamYaple | if you shutdown 2, then some writes will only live on 1 node, and if that crashes or has disk errors, then thats no good | 10:34 |
| Neptu_ | evrardjp: one last thing after I instalation of a aio how i force the roles to be downloaded/ready... do i need to run again the full boostrap? | 10:34 |
| evrardjp | max at a time | 10:34 |
| SamYaple | but you can do 2 without going *down*, i just dont recommendit | 10:34 |
| evrardjp | Neptu_: if you added the role into the a-r-r they are technically already downloaded during the process | 10:34 |
| evrardjp | if you want to modify the a-r-r and only refetch new/updated roles you can use the bootstrap-ansible script | 10:35 |
| Neptu_ | evrardjp: I did modified a-r-r after the install | 10:35 |
| evrardjp | in scripts | 10:35 |
| Neptu_ | looking | 10:35 |
| *** markvoelker has quit IRC | 10:41 | |
| hwoarang | odyssey4me: did you have the change to look at the log collection problems that we were discussing on friday? | 10:42 |
| odyssey4me | hwoarang I only saw the review that went through but was on holiday and haven't read any backscroll that far back. Can you give the highlights of what precipitated the revert review? | 10:44 |
| *** sm806 has quit IRC | 10:44 | |
| hwoarang | odyssey4me: so basically the post-run phase fails because the playbook wasn't available | 10:45 |
| Neptu_ | evrardjp: for aio should you use the boostrap-aio.sh script instead or is same same? | 10:45 |
| hwoarang | odyssey4me: SamYaple suggested "[09:55:24] <SamYaple> you have to declare openstack-ansible-tests here https://github.com/openstack/openstack-ansible-tests/blob/master/zuul.d/jobs.yaml#L68" | 10:45 |
| -openstackstatus- NOTICE: Zuul has been restarted due to an unexpected issue. Please recheck any jobs that were in progress | 10:45 | |
| odyssey4me | hwoarang hmm, I wonder if we might have to move our base job into a trusted repo for that to work :/ | 10:45 |
| hwoarang | odyssey4me: it's not clear why 'run' works but 'post-run' doesn't | 10:45 |
| Neptu_ | evrardjp: I used the ./gate-check-commit.sh commit to setup the enviroment | 10:46 |
| odyssey4me | hwoarang yeah, that's weird | 10:46 |
| *** huxinhui has quit IRC | 10:46 | |
| odyssey4me | but yes, it does make sense to add the tests repo there - lemme push up a patch and do a depends-on test to validate | 10:46 |
| *** nshetty is now known as nshetty|brb | 10:48 | |
| evrardjp | Neptu_: gate check commit is the wrapper for gates | 10:49 |
| gunix | SamYaple: evrardjp: i know ideal is to shutdown only 1 node out of 3, but i am talking about unrecovarable situations. right now it seems that if you do shut down 3 nodes, getting all services back online is confusing and undocumented. all services need to be manually inspected and without experience this can take hours or days... | 10:49 |
| evrardjp | technically you should follow the steps here: https://docs.openstack.org/openstack-ansible/latest/contributor/quickstart-aio.html#building-an-aio | 10:49 |
| evrardjp | gunix: it's not undocumented | 10:50 |
| evrardjp | https://docs.openstack.org/openstack-ansible/latest/admin/index.html | 10:50 |
| drifterza | Ahoy | 10:50 |
| evrardjp | gunix: but if you think something is completely missing shoot :) | 10:51 |
| gunix | evrardjp: that is aio ("all in one"). i have 3 controllers :) | 10:51 |
| evrardjp | gunix: the first link is for Neptu_ :p | 10:51 |
| odyssey4me | evrardjp hwoarang SamYaple GYI, we only clone repositories in the gate which aren't already there - that's how we work around being inside/outside infra | 10:51 |
| odyssey4me | *FYI | 10:51 |
| evrardjp | odyssey4me: I know. | 10:52 |
| evrardjp | however we have a bug | 10:52 |
| drifterza | So I see theres a openstack-ansible-plugins/connection/ssh.py plugin, and I've included it in my ansible.cfg under the connection_plugins variable, weird part is, its still trying to use the normal ssh plugin, any ideas on how to force it to use the ssh.py ? | 10:52 |
| evrardjp | I don't know why you said that, or in which context | 10:52 |
| gunix | evrardjp: i shut down all 3 controllers. galera was crushed (the talk i had earlier with SamYaple which got solved by killing mysql and starting the cluster). however, galera is not the only servie that failed to start. i get multiple errors with openstack client and with horizon :D and now i am confused regarding how i should restart the stack | 10:52 |
| odyssey4me | @evrardjp based on your conversation about the post-run log collection on friday | 10:52 |
| evrardjp | odyssey4me: see https://etherpad.openstack.org/p/osa-zuulv3 last part of the page | 10:52 |
| odyssey4me | you mentioned that you don't know that | 10:52 |
| evrardjp | I am not sure what you mean, maybe I wasn't clear, as usual. | 10:53 |
| Neptu_ | evrardjp: thanks double checking | 10:53 |
| *** nshetty|brb is now known as nshetty | 10:53 | |
| gunix | evrardjp: afaik, restarting all services (nova neutron cinder-api swift-api and so on...) after a cold reboot of everything is undocumented. you need to manually troubleshoot everything and see what failed and restart all services when you figure it out :D | 10:53 |
| Neptu_ | evrardjp: dummy question what is gates? | 10:54 |
| evrardjp | gunix: recovering from disaster should be simple as getting mariadb back in function, rerolling rabbit, and re-running all the playbooks | 10:54 |
| gunix | evrardjp: i guess it's enouth to run the setup_openstack playbook | 10:54 |
| evrardjp | gunix: I think you've found something to add to the documentation | 10:55 |
| odyssey4me | evrardjp oh, so the integrated repo stuff isn't working right... I didn't think it would | 10:55 |
| evrardjp | gunix: after you infra is ok? I'd say yes. | 10:55 |
| gunix | evrardjp: do you have any docus on rerolling rabbitmq? i have no experience in configuring mirrored queues, OSA did it for me | 10:55 |
| evrardjp | odyssey4me: it was working fine. | 10:55 |
| evrardjp | gunix: destroy the containers and start again? | 10:56 |
| odyssey4me | evrardjp I haven't done any work on that, but can look into it - I've been focusing on the roles only and logan- has been doing the integrated role stuff. | 10:56 |
| evrardjp | odyssey4me: what I meant is, we tried this (see my first comment on that page) | 10:56 |
| evrardjp | and it was working. | 10:56 |
| evrardjp | I wonder what changed since that time. Or maybe it's because we are not taking care of the example I am showing, which is some kind of check outs | 10:56 |
| *** pcaruana has joined #openstack-ansible | 10:56 | |
| odyssey4me | evrardjp the etherpad has some links, but it's not explaining the problem... and I don't understand the problem from looking at the links either | 10:57 |
| evrardjp | or maybe it's some variables that aren't there anymore | 10:57 |
| evrardjp | odyssey4me: let me rephrase that on the etherpad then | 10:57 |
| evrardjp | I haven't digged into the problem yet too. | 10:57 |
| evrardjp | gunix: in all cases, we should maybe have a "disaster procedure" | 10:57 |
| evrardjp | Neptu_: that's our automated testing before merging. | 10:58 |
| odyssey4me | but I'll look with more attention in a bit - I suspect I might know what the issue is, but would need some time to look through everything properly and right now the role tests are where I'm focused | 10:58 |
| Neptu_ | evrardjp: thanks | 10:58 |
| *** chhavi has quit IRC | 10:58 | |
| *** drifterza has quit IRC | 11:00 | |
| *** drifterza has joined #openstack-ansible | 11:01 | |
| Neptu_ | evrardjp: awesome the role got downloaded as i tested before so awesome... going dig about where to add the role to be installed but i guess that is a question of a playbook | 11:01 |
| *** dmsimard is now known as dmsimard|afk | 11:05 | |
| evrardjp | odyssey4me: rephrased the issue | 11:05 |
| evrardjp | Neptu_: you can probably create one. | 11:05 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Revert "Revert "Move log collection to 'post' stage"" https://review.openstack.org/516249 | 11:06 |
| *** stuartgr has joined #openstack-ansible | 11:06 | |
| Neptu_ | evrardjp: yep I just need to think about the best way since for procurement reasons people will be reluctant to run my playbook even if its dam simple unless is on the openstack ansible repo | 11:07 |
| *** smatzek has joined #openstack-ansible | 11:07 | |
| Neptu_ | need some thinking about this | 11:07 |
| Neptu_ | evrardjp: one thing I do not understand can u add external conf files for the monasca agent once the role is beeing created? | 11:08 |
| evrardjp | you have a good team that's willing to upstream, that's good ! :p | 11:08 |
| Neptu_ | evrardjp: haha the team is not really mine but yes | 11:08 |
| Neptu_ | those are my first contributions so i need to get confortable on commiting code | 11:09 |
| gunix | evrardjp: SamYaple: https://bpaste.net/show/2ec8eaf8146b ... does this mean that the rabbitmq queues are dead? :D | 11:10 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Revert "Revert "Move log collection to 'post' stage"" https://review.openstack.org/516249 | 11:12 |
| evrardjp | gunix: maybe they are in a vhost | 11:17 |
| *** SmearedBeard has quit IRC | 11:18 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] After-install network configuration https://review.openstack.org/516069 | 11:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: [TEST] Test log collection https://review.openstack.org/516252 | 11:27 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Use latest instead of draft https://review.openstack.org/516253 | 11:31 |
| *** SmearedBeard has joined #openstack-ansible | 11:32 | |
| *** mhayden has quit IRC | 11:35 | |
| *** dave-mccowan has joined #openstack-ansible | 11:35 | |
| *** mhayden has joined #openstack-ansible | 11:36 | |
| gunix | evrardjp: you are right | 11:36 |
| mhayden | morning | 11:36 |
| evrardjp | gunix: or am I? ;) | 11:37 |
| evrardjp | (tell that to my wife) | 11:37 |
| Neptu_ | evrardjp: everything that start with os_ what means?? actual components? | 11:37 |
| Neptu_ | im talking about playbooks | 11:37 |
| *** markvoelker has joined #openstack-ansible | 11:38 | |
| Neptu_ | window show 1 | 11:39 |
| *** bhujay has joined #openstack-ansible | 11:40 | |
| *** chas has joined #openstack-ansible | 11:41 | |
| *** chas has quit IRC | 11:41 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/newton: use Newton sha's for cross repo versions https://review.openstack.org/515207 | 11:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516259 | 11:53 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_gnocchi stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516260 | 11:53 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ironic stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516261 | 11:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516262 | 11:54 |
| *** vnogin has quit IRC | 11:55 | |
| evrardjp | afk lunch | 11:57 |
| *** sm806 has joined #openstack-ansible | 12:00 | |
| *** sm806 has quit IRC | 12:00 | |
| *** taseer2 is now known as Taseer | 12:02 | |
| *** vnogin has joined #openstack-ansible | 12:02 | |
| gunix | evrardjp: your wife will never say you are right. | 12:02 |
| *** drifterza has quit IRC | 12:02 | |
| *** udesale has joined #openstack-ansible | 12:03 | |
| *** drifterza has joined #openstack-ansible | 12:04 | |
| *** thorst has joined #openstack-ansible | 12:09 | |
| *** markvoelker has quit IRC | 12:11 | |
| *** kylek3h_ has quit IRC | 12:17 | |
| *** drifterza has quit IRC | 12:18 | |
| *** thorst_ has joined #openstack-ansible | 12:19 | |
| *** thorst__ has joined #openstack-ansible | 12:20 | |
| *** jamesden_ has joined #openstack-ansible | 12:21 | |
| *** thorst has quit IRC | 12:23 | |
| *** jamesdenton has quit IRC | 12:24 | |
| *** thorst_ has quit IRC | 12:24 | |
| *** drifterza has joined #openstack-ansible | 12:25 | |
| *** markvoelker has joined #openstack-ansible | 12:29 | |
| odyssey4me | if any cores are available for this backport review I'd appreciate it: https://review.openstack.org/#/c/516213/ | 12:32 |
| Neptu_ | evrardjp: i just had some questions here in the office... I need to understand is no way openstack ansible dinamically find the monasca agent role and deploy... I need to have a playbook and hook it somewhere | 12:35 |
| Neptu_ | evrardjp: i mean without much magic from the /etc/ansible/roles folder | 12:36 |
| Neptu_ | as far as we talked playbook is strictly necesary | 12:37 |
| *** gkadam has quit IRC | 12:37 | |
| evrardjp | I don't understand what you're saying Neptu_ | 12:37 |
| Neptu_ | the boostrap in itself will not schedule it for deployment on a host group... | 12:38 |
| admin0 | gunix, so you turned off all 3 controllers and want to recover from it ? | 12:38 |
| admin0 | 1. make sure the mysql database is good and they are in the cluster .. | 12:38 |
| admin0 | if the db is good, everything is good .. else you have to remove the db, redo the galera only install and restore the db to get it back operational | 12:39 |
| admin0 | after that, run the rabbit playbook to fix rabbit | 12:39 |
| admin0 | then you can run the rest of the playbook normally .. you just need to ensure your db is good | 12:39 |
| gunix | admin0: ok wait a sec. i was about to delete the environment :)) | 12:39 |
| admin0 | well, if deleting and redoing is an option ( without client data) , you can take that approach as well | 12:40 |
| admin0 | and yes, if all your nodes are down, or even 1-2 controllers are down, you need to restart all services when you bring it back online .. this is due to some db/rabbit timeout i sse in the configs | 12:41 |
| admin0 | a restart helps | 12:41 |
| admin0 | if rabbit containers give error when redoing the playbooks, you can nuke them and recreate | 12:41 |
| evrardjp | gunix: never delete the environment | 12:41 |
| evrardjp | if you're on prod, just re-use | 12:42 |
| *** shardy is now known as shardy_lunch | 12:42 | |
| evrardjp | we need docs for that | 12:42 |
| admin0 | with a good db, most of the time, just rerun the playbooks and it will fix itself | 12:42 |
| evrardjp | admin0: patches welcome :) | 12:42 |
| gunix | admin0: i am running the rabbitmq playbook now. what next | 12:42 |
| gunix | ? | 12:42 |
| admin0 | first step was for you to verify that your db is OK and galera is in cluster ? | 12:43 |
| *** vnogin has quit IRC | 12:43 | |
| *** vnogin has joined #openstack-ansible | 12:44 | |
| gunix | admin0: yea that was the first thing i checked. i figured without osa XP that galera will have some issues. without galera, login on horizon was down. with galera i have the login, but all pages on horizon return an erro | 12:44 |
| gunix | example: Error: Unable to retrieve hypervisor information. | 12:44 |
| gunix | Error: Unable to get nova services list. | 12:44 |
| gunix | Error: Unable to retrieve hypervisor statistics. | 12:44 |
| admin0 | right :) | 12:44 |
| gunix | admin0: rabbitmq playbook returned no fail | 12:44 |
| admin0 | you have to login to the hypervisor and resetart the services .. but before that, you have to restart the APIs or the containers themselves where the API lives | 12:44 |
| admin0 | restart all your keystone containers | 12:45 |
| gunix | admin0: do i have to manually do this or are there some playbooks that can help ? | 12:45 |
| *** vnogin_ has joined #openstack-ansible | 12:45 | |
| *** vnogin has quit IRC | 12:45 | |
| admin0 | using inventory manageer, you get the container groups .. then you can ansible <groupname> -m shell -a "reboot & " | 12:45 |
| gunix | admin0: keystone reboot done: https://bpaste.net/show/85d91d17acf1 :D | 12:46 |
| admin0 | login to the util container | 12:46 |
| admin0 | openstack endpoint list --debug | 12:46 |
| admin0 | does it work | 12:46 |
| gunix | admin0: regarding <groupname>, i usually figure out the group name but i have no ideea where i can find a list of group names | 12:46 |
| admin0 | /opt/openstack-ansible/scripts/inventory-manage.py -l | 12:47 |
| *** vnogin_ has quit IRC | 12:47 | |
| admin0 | componet is the specific components you can target | 12:47 |
| admin0 | make sure you run ansible from inside /opt/ansible | 12:47 |
| *** vnogin has joined #openstack-ansible | 12:47 | |
| admin0 | err: /opt/openstack-ansible | 12:48 |
| gunix | :D | 12:48 |
| gunix | osa-compute-1 is listed as ceilometer_agent_compute but you can also call it with group compute_hosts ... somehow opening /etc/openstack_deploy/openstack_inventory is easier and provides more info :D | 12:49 |
| admin0 | i am lazy to type this all the time, so i have /usr/bin/im with the line cd /opt/openstack-ansible/ | 12:49 |
| admin0 | ; ./scripts/inventory-manage.py -l | 12:49 |
| admin0 | .. and then i just do im | less :) | 12:49 |
| admin0 | im | grep abc etc | 12:49 |
| gunix | anyway i did "systemctl restart nova*" on compute nodes | 12:49 |
| admin0 | yes, but the nova apis need to rejoin to the db/rabbit | 12:50 |
| gunix | i still get Error: Unable to retrieve hypervisor information. | 12:50 |
| gunix | ok so i also reboot the nova containers | 12:50 |
| gunix | ? | 12:50 |
| admin0 | login to one of the nova containers | 12:50 |
| admin0 | then check the logs | 12:50 |
| admin0 | there do a systemctl restart nova-<bla> and see if it connects fine | 12:50 |
| admin0 | if it does, then you can do a "mass" reboot | 12:50 |
| *** chhavi has joined #openstack-ansible | 12:51 | |
| gunix | i can reboot them with: ansible nova_api* -m shell -a "reboot" | 12:51 |
| gunix | or just services with: ansible nova_api* -m shell -a "systemctl restart nova-*" | 12:51 |
| *** woodard has joined #openstack-ansible | 12:52 | |
| *** vnogin has quit IRC | 12:52 | |
| gunix | still error on hypervizors :D | 12:52 |
| admin0 | if doing reboot, i usually put a & at the end so that my terminal is not stuck | 12:53 |
| admin0 | the error or hypervisors, you need to check the log in the hypervisor itself | 12:53 |
| admin0 | where it fails | 12:53 |
| admin0 | or do a tailf * | grep -i error in the log server :) | 12:53 |
| admin0 | gunix if its just a reboot/shutdown/power failure and your login etc works . you can safely run setup-openstack playbook | 12:54 |
| *** kylek3h has joined #openstack-ansible | 12:55 | |
| gunix | admin0: i started openstack-ansible playbooks/setup-openstack.yml | 12:56 |
| gunix | i will tell you were it crashes | 12:56 |
| admin0 | ok | 12:56 |
| gunix | admin0: i am doing this now to learn how to recover from a crash. i find this info really useful. however i will destroy this environment anyway since it's HA setup and I don't need HA at home and i don't have ressources for HA anyway | 12:57 |
| admin0 | you said you have HA and then again you don't have resources for HA .. thats kind of contradicting :) | 12:58 |
| *** vnogin has joined #openstack-ansible | 12:59 | |
| admin0 | gunix, when you have customers and you have a crash .. like a real crash .. you will learn it automatically :D | 12:59 |
| gunix | admin0: my desktop at home is at 30GB ram and 2GB swap, out of 32GB ram and 10 GB swap. i also can't open any steam games when the test-environment is using all the memory | 13:00 |
| gunix | admin0: since i have deb at home and reboots are required once every 2 months, i will set it up with a light infra so that i can play with openstack and also open a game | 13:00 |
| gunix | i should have only 25 GB used by osa | 13:01 |
| gunix | and no swap | 13:01 |
| admin0 | or just buy instances that can be deleted anytime from aws/google at cheap and then virtualize your whole setup .. | 13:01 |
| admin0 | or lease a cheap dedicated server | 13:02 |
| admin0 | so that you can test not only the basics, but a lot more | 13:02 |
| *** kjw3 has quit IRC | 13:03 | |
| *** lhinds has joined #openstack-ansible | 13:03 | |
| gunix | admin0: if i leave the instance up for a few months, it costs as much as one desktop that i can use for years ... i have i7, 32 GB ram and 3 SSDs at home ... i would rather use the ressources i have :D | 13:04 |
| *** lhinds has quit IRC | 13:05 | |
| admin0 | few months ? i was thinking like 2 days per weekend :D | 13:05 |
| *** jafeha has quit IRC | 13:06 | |
| gunix | admin0: nah, i also play with my home desktop when i am at work. i left ssh on 443 so that i can connect to it. my company failed to provide me with good hardware | 13:07 |
| admin0 | ok | 13:07 |
| lagman | Hi, when trying to deploy openstac-ansible, it fails at "galera-install.yml" on the task "pip_install : Install pip packages" it cant access the container inside the infrastrcture server. can someone shed some light on what do to, | 13:07 |
| *** lhinds has joined #openstack-ansible | 13:07 | |
| gunix | admin0: they said they will by me a lenovo workstation in august ... if i am lucky i get it in december. also, they gave me some servers, but i have no internet on them and granting internet would be a security breach. cool, right? | 13:07 |
| admin0 | lagman, is the haproxy up ? | 13:08 |
| mgariepy | morning everyone | 13:08 |
| admin0 | login to that container and do apt-get update .. does it work ? | 13:08 |
| gunix | lagman: you could try to share the error log, and maybe run the playbook with -vvvv to get debug info. | 13:08 |
| *** drifterza has quit IRC | 13:10 | |
| admin0 | gunix, i am sure you can get creative :) | 13:11 |
| mgariepy | gunix, is haproxy running ? | 13:12 |
| *** drifterza has joined #openstack-ansible | 13:12 | |
| *** bhujay has quit IRC | 13:12 | |
| lagman | gunix: here is the error http://paste.openstack.org/show/624966/ | 13:12 |
| mgariepy | on the haproxy host, are the repo endpoint in OK state ? | 13:13 |
| gunix | admin0: fatal: [osa-cinder-1]: FAILED! => {"attempts": 10, "cache_control": "no-cache", "changed": false, "connection": "close", "content": "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n", "content_type": "text/html", "failed": true, "msg": "Status code was not [200, 300]: HTTP Error 503: Service Unavailable", "redirected": false, | 13:13 |
| gunix | "status": 503, "url": "http://172.29.236.9:8776"} | 13:13 |
| gunix | this is where it failed | 13:13 |
| admin0 | grep what runs in that port and fix the service :D | 13:14 |
| admin0 | or run setup-infra :) | 13:14 |
| admin0 | setup openstack depends on setup infra which depends on setup hosts .. you can grep that port, get the service, if its a service that has failed ( from the logs ) run the playbooks to target only this service to get it back into operational state | 13:15 |
| admin0 | setup infra takes considerably longer time, so trying to avoid that | 13:16 |
| admin0 | when your infra is down, your goal would be to get it up as soon as possible to meet the SLAs | 13:16 |
| gunix | admin0: i ran setup-infrastructure earlier and it didn't return any fails. strange, right? :) i will try to troubleshoot the cinder containers | 13:17 |
| admin0 | assuming this is your production environment, cat the setup-openstack yml file .. check where cinder is .. since cinder failed, run the rest manually one playbook at a time | 13:19 |
| admin0 | when all is done, get back to cinder | 13:19 |
| gunix | lagman: i got that error last week and for it got solved by running the playbooks again | 13:19 |
| gunix | admin0: it connects to 172.29.236.9 and that is the VIP ... :D that error i gave you is useless, sorry. i'll take this step by step since i am really curious how it works in the background. i will check all cinder containers | 13:22 |
| *** drifterza_ has joined #openstack-ansible | 13:22 | |
| *** drifterza has quit IRC | 13:25 | |
| *** lbragstad has quit IRC | 13:25 | |
| admin0 | lagman, check/fix your haproxy | 13:27 |
| *** vnogin has quit IRC | 13:28 | |
| evrardjp | Neptu_: I still don't understand your issue :( | 13:28 |
| *** drifterza_ has quit IRC | 13:28 | |
| evrardjp | we have groups defined, you have the role, it's just a question of you deciding to know on what to apply. I don't know monasca so I can't help you there. But it should be straightforward | 13:28 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/pike: Hide yum-config-manager output https://review.openstack.org/514837 | 13:29 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/pike: Add fedora to run_tests_common.sh https://review.openstack.org/514711 | 13:29 |
| *** vnogin has joined #openstack-ansible | 13:29 | |
| Tahvok | Hey guys | 13:30 |
| *** smatzek has quit IRC | 13:30 | |
| Tahvok | I remember you said that ssh public keys are being copied from compute to compute | 13:30 |
| *** smatzek has joined #openstack-ansible | 13:30 | |
| Tahvok | However I've checked now, and I don't see any of them on my compute hosts | 13:31 |
| *** smatzek has quit IRC | 13:31 | |
| Tahvok | Can you tell me where is the code that should do this? | 13:31 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-apt_package_pinning stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516274 | 13:31 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-ceph_client stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516275 | 13:31 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-galera_client stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516276 | 13:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-galera_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516277 | 13:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-haproxy_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516278 | 13:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_container_create stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516279 | 13:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_hosts stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516280 | 13:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-memcached_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516281 | 13:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-openstack_hosts stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516282 | 13:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-openstack_openrc stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516283 | 13:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-ops stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516284 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-apt_package_pinning stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516274 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_aodh stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516285 | 13:33 |
| *** lbragstad has joined #openstack-ansible | 13:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_barbican stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516286 | 13:33 |
| mgariepy | Tahvok, https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_compute_key_distribute.yml | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-ceph_client stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516275 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516287 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cinder stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516288 | 13:33 |
| mgariepy | Tahvok, https://github.com/openstack/openstack-ansible-os_nova/blob/master/tasks/nova_compute_key_populate.yml | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-galera_client stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516276 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_designate stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516289 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_glance stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516290 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_gnocchi stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516291 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-galera_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516277 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_heat stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516292 | 13:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_horizon stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516293 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-haproxy_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516278 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ironic stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516294 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_keystone stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516295 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_container_create stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516279 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_magnum stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516297 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_molteniron stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516298 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_hosts stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516280 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_monasca stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516299 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_neutron stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516300 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-memcached_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516281 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_nova stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516301 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_octavia stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516302 | 13:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-openstack_hosts stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516282 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_rally stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516303 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516304 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_swift stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516305 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-openstack_openrc stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516283 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tacker stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516306 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tempest stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516307 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-ops stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516284 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_trove stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516308 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_watcher stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516309 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_aodh stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516285 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_zaqar stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516310 | 13:35 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-pip_install stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516311 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_barbican stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516286 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-plugins stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516312 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rabbitmq_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516313 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-repo_build stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516314 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516287 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-repo_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516315 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rsyslog_client stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516316 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cinder stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516288 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rsyslog_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516317 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-hardening stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516318 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_designate stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516289 | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_glance stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516290 | 13:37 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_gnocchi stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516291 | 13:37 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_heat stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516292 | 13:37 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_horizon stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516293 | 13:37 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ironic stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516294 | 13:38 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_keystone stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516295 | 13:38 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_magnum stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516297 | 13:38 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_molteniron stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516298 | 13:38 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_monasca stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516299 | 13:39 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_neutron stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516300 | 13:39 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_nova stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516301 | 13:39 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_octavia stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516302 | 13:39 |
| *** thorst__ has quit IRC | 13:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_rally stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516303 | 13:40 |
| Tahvok | mgariepy: I thought libvirtd was running with root user | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516304 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_swift stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516305 | 13:40 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tacker stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516306 | 13:41 |
| *** woodard has quit IRC | 13:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tempest stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516307 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_trove stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516308 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_watcher stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516309 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_zaqar stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516310 | 13:41 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-pip_install stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516311 | 13:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-plugins stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516312 | 13:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rabbitmq_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516313 | 13:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-repo_build stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516314 | 13:42 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-repo_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516315 | 13:43 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rsyslog_client stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516316 | 13:43 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rsyslog_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516317 | 13:43 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-hardening stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516318 | 13:44 |
| *** jafeha has joined #openstack-ansible | 13:45 | |
| *** shardy_lunch is now known as shardy | 13:46 | |
| odyssey4me | evrardjp hwoarang it looks like SamYaple's suggestion works cross-repo: https://review.openstack.org/#/c/516252/ | 13:46 |
| evrardjp | I think it's because the last issues on zuul were recently ironed out by jeblair | 13:47 |
| evrardjp | oh no talking about something else | 13:47 |
| *** jwitko has joined #openstack-ansible | 13:47 | |
| hwoarang | odyssey4me: yeah that's good news but i am not sure why the upgrade job failed :( | 13:49 |
| hwoarang | i had a look but i wasn't sure if it was caused by this patch | 13:49 |
| odyssey4me | hwoarang that's waiting for https://review.openstack.org/516213 to merge | 13:49 |
| hwoarang | logan-: cloudnull: hello. iirc you do a lot of ceph stuff :) could you tell me how you test the upstream ceph-ansible repo? a vagrant up fails quite fast on centos | 13:49 |
| odyssey4me | it was nice to see both success/failure in the test though to validate that the log collection works regardless | 13:50 |
| hwoarang | odyssey4me: yeah that's great. Thanks for fixing it :) | 13:50 |
| *** Smeared_Beard has joined #openstack-ansible | 13:51 | |
| *** SmearedBeard has quit IRC | 13:53 | |
| odyssey4me | evrardjp the upgrades are broken due to a patch that's merging - nothing to do with the log collection | 13:54 |
| evrardjp | Which patch? | 13:55 |
| evrardjp | how did that patch pass? | 13:55 |
| evrardjp | (probably in tests repo I guess then?) | 13:55 |
| odyssey4me | yes, the patch is merging into the pike branch to fix it: https://review.openstack.org/516213 | 13:55 |
| odyssey4me | has https://review.openstack.org/514850 helped improve the success of the suse jobs? should we port it back to pike hwoarang / evrardjp ? | 13:58 |
| hwoarang | i dont think it improved much... :( | 13:59 |
| evrardjp | not that I have seen. | 13:59 |
| odyssey4me | so until we have logs, all we can do is recheck? | 14:00 |
| hwoarang | i am afraid so | 14:00 |
| hwoarang | i really need to see where it hangs | 14:00 |
| hwoarang | the repos appear to be configured properly from the logs that have been gathered prior to reverting your patch | 14:00 |
| hwoarang | so i dont know. i need to see the lxc log | 14:00 |
| evrardjp | I am blocked on the refactor of all this, due to a zuul bug in the integrated. but right now I have bigger fish to fry too (if that exists) | 14:01 |
| *** dxiri has joined #openstack-ansible | 14:01 | |
| *** smatzek has joined #openstack-ansible | 14:03 | |
| *** smatzek has quit IRC | 14:03 | |
| *** smatzek has joined #openstack-ansible | 14:04 | |
| *** smatzek has quit IRC | 14:04 | |
| *** nshetty has quit IRC | 14:04 | |
| *** smatzek has joined #openstack-ansible | 14:05 | |
| evrardjp | could I get some votes real quick on that? https://review.openstack.org/#/c/516253/ | 14:07 |
| evrardjp | I need to find out what's wrong in the post run role | 14:08 |
| evrardjp | spotz: asettle? ^ | 14:08 |
| *** marst has joined #openstack-ansible | 14:11 | |
| *** lbragstad has quit IRC | 14:13 | |
| *** lbragstad has joined #openstack-ansible | 14:13 | |
| odyssey4me | evrardjp that url doesn't work though | 14:17 |
| evrardjp | yeah. | 14:17 |
| odyssey4me | https://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/ gives a 404 error | 14:17 |
| evrardjp | but we need something to merge for the job to retrigger. | 14:17 |
| *** lbragstad has quit IRC | 14:18 | |
| evrardjp | that's a post job that regenerates the docs and publishes them there | 14:18 |
| evrardjp | the other url doesn't work anyway. | 14:18 |
| evrardjp | and that's the good url | 14:18 |
| odyssey4me | looks fine to me, just add workflow | 14:18 |
| evrardjp | so now, let's follow the logs. | 14:19 |
| evrardjp | thanks odyssey4me | 14:19 |
| cloudnull | mornigns | 14:20 |
| lagman | anyone know why i cant the repo containers webserver on port 8181? why doesnt lxcbr0 port forward to the correct container? | 14:20 |
| cloudnull | **mornings | 14:20 |
| cloudnull | hwoarang: no I don't do much with ceph at the moment | 14:20 |
| cloudnull | we have teams that do | 14:20 |
| cloudnull | but not something i deal with personally | 14:20 |
| cloudnull | lagman: try running lxc-veth-check | 14:21 |
| cloudnull | and see if there are ny broken network connections. | 14:21 |
| lagman | cloudnull i get 2.0.8 | 14:21 |
| odyssey4me | lagman lxcbr0 is for outbound connectivity only, the port forwarding is done by haproxy (or your LB if you have one) | 14:21 |
| *** lbragstad has joined #openstack-ansible | 14:22 | |
| cloudnull | lagman: That's good. If the tool has no output, there's nothing broken from the network perspective | 14:22 |
| cloudnull | ++ what odyssey4me said, the next thing to check is your LB (haproxy?). | 14:23 |
| cloudnull | any cores around mind giving this a push - https://review.openstack.org/#/c/515721/ - stand a long swift is not happy at the moment due to those checks . | 14:24 |
| cloudnull | andymccr: ^ | 14:24 |
| odyssey4me | cloudnull why not just revert the whole half-done serialised upgrade patch? | 14:25 |
| cloudnull | we could. | 14:25 |
| odyssey4me | otherwise we may have to whack a few more moles on the way to success | 14:25 |
| cloudnull | so far its just that part that's a blocker, however I can put up a revert patch | 14:26 |
| odyssey4me | lemme look properly quickly | 14:26 |
| odyssey4me | ja, it looks to me like https://github.com/openstack/openstack-ansible-os_swift/commit/176452bebf7e1ceabd2cf72a89c558c0fc7f1420 is self contained and has not been edited since | 14:27 |
| odyssey4me | I'd say just revert. | 14:27 |
| cloudnull | ++ | 14:27 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_swift master: Revert "Don't run sync in upgrades" https://review.openstack.org/516331 | 14:28 |
| cloudnull | I abandoned my patch | 14:28 |
| odyssey4me | hmm, merge conflict somehow | 14:29 |
| *** weezS has joined #openstack-ansible | 14:30 | |
| *** thorst has joined #openstack-ansible | 14:30 | |
| *** wlmbasson has joined #openstack-ansible | 14:31 | |
| *** thorst has quit IRC | 14:31 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_swift master: Revert "Don't run sync in upgrades" https://review.openstack.org/516331 | 14:31 |
| *** thorst has joined #openstack-ansible | 14:31 | |
| cloudnull | the swift_do_sync condition was added later it seems. | 14:31 |
| cloudnull | fixed the conflict | 14:31 |
| Miouge | Do I need to do something special to add rbd and rados to the cinder venv? http://paste.openstack.org/show/624975/ | 14:31 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-openstack_hosts master: [WIP] Include pip install default repos https://review.openstack.org/515399 | 14:32 |
| *** mrch has quit IRC | 14:32 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_swift stable/pike: Revert "Don't run sync in upgrades" https://review.openstack.org/516333 | 14:33 |
| asettle | evrardjp: what what | 14:33 |
| asettle | what do | 14:33 |
| asettle | how now | 14:33 |
| asettle | brown cow | 14:33 |
| *** galstrom_zzz is now known as galstrom | 14:33 | |
| evrardjp | the lazy fox jumps over ? | 14:34 |
| cloudnull | the quick brown one. | 14:34 |
| *** vishwanathj has joined #openstack-ansible | 14:34 | |
| evrardjp | :) | 14:34 |
| *** mrch has joined #openstack-ansible | 14:35 | |
| asettle | Well I'm glad that went somewhere :p | 14:35 |
| evrardjp | asettle: it's settled now | 14:35 |
| asettle | The way I like it | 14:35 |
| evrardjp | I thought yes, but your name says otherwise | 14:35 |
| asettle | It says 'settle'? | 14:35 |
| evrardjp | alpha privative? | 14:36 |
| evrardjp | let me see if that translates directly | 14:36 |
| evrardjp | yup | 14:37 |
| evrardjp | https://www.wikiwand.com/en/Alpha_privative | 14:37 |
| cloudnull | Miouge: I'm not 100% sure. I see that if rbd is in use it should run the ceph client | 14:37 |
| cloudnull | https://github.com/openstack/openstack-ansible-os_cinder/blob/f7f45e118e524540e54a41a2789b143a3459997f/tasks/main.yml#L79-L89 | 14:37 |
| *** udesale has quit IRC | 14:37 | |
| openstackgerrit | Merged openstack/openstack-ansible master: [Docs] Use latest instead of draft https://review.openstack.org/516253 | 14:37 |
| evrardjp | and wikipedia is THE source of truth as we all know :D | 14:37 |
| cloudnull | however IDK how or if those libs get added into the cinder venv | 14:37 |
| *** Brew has joined #openstack-ansible | 14:38 | |
| spotz | evrardjp: That the fix for the redirect? I see it's got all it's +s | 14:39 |
| evrardjp | spotz: almost everything is fixed now | 14:40 |
| spotz | evrardjp: Cool | 14:40 |
| evrardjp | but we still need to fix the publishing job, then we can probably adapt the links on the published landing page | 14:40 |
| spotz | evrardjp: Is that docs or infra or us? | 14:41 |
| evrardjp | atm, working on the infra side. | 14:42 |
| hwoarang | cloudnull: ok thank you | 14:43 |
| cloudnull | sorry. | 14:43 |
| *** hw_wutianwei has joined #openstack-ansible | 14:43 | |
| cloudnull | are you working with ceph these days? | 14:43 |
| hwoarang | just trying to get suse added to the upstream playbooks | 14:43 |
| Miouge | cloudnull: After some digging I probably have something missing: the cinder-volume container has “rbd_ceph_conf=/etc/ceph/ceph.conf” but is not present in that container. So I am missing more than just the python bindings | 14:44 |
| cloudnull | Miouge: do you have the backend setup in the openstack_user_config.yml ? something like https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.example#L675-L690 | 14:46 |
| Miouge | cloudnull: I do have something similar, I renamed “volumes_hdd” to “volumes” and not much else http://paste.openstack.org/show/624981/ | 14:47 |
| cloudnull | ok. that looks good | 14:48 |
| openstackgerrit | Merged openstack/openstack-ansible stable/pike: Fix warning in upgrade https://review.openstack.org/510419 | 14:49 |
| Miouge | cloudnull: but I don’t have any “ceph_mons” in user_variables.yml based on https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/user_variables.yml#L74 it might be necessary :) | 14:51 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_swift master: Revert "Don't run sync in upgrades" https://review.openstack.org/516331 | 14:52 |
| logan- | o/ reading thru backscroll. evrardjp i did test the depends-on when working on the initial integrated repo stuff but maybe something has changed since then? i can dig into it.. do you have an example patch i can see where its happening? | 14:52 |
| evrardjp | Yes I have all the explanation here. | 14:53 |
| evrardjp | just a second :) | 14:53 |
| evrardjp | https://etherpad.openstack.org/p/osa-zuulv3 | 14:53 |
| evrardjp | last paragraph | 14:53 |
| evrardjp | I am doing a recheck on what was working last week | 14:53 |
| logan- | hwoarang: i dont test ceph-ansible separately from osa, so I can't give you much advice on testing it standalone, sorry | 14:54 |
| evrardjp | to see if it's my other patchset that's not working (because it has a series of commits as checkout) | 14:54 |
| evrardjp | or if it's something in the job that changed | 14:54 |
| logan- | ok | 14:54 |
| evrardjp | hwoarang: andymccr is doing ceph-ansible testing IIRC | 14:54 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_swift stable/pike: Revert "Don't run sync in upgrades" https://review.openstack.org/516333 | 14:54 |
| hwoarang | logan-: ok no problem :) | 14:54 |
| hwoarang | evrardjp: ah that's great :) I hope he is around to ask a few questions | 14:54 |
| cloudnull | thanks odyssey4me! | 14:55 |
| cloudnull | :D | 14:55 |
| cloudnull | beat me to it | 14:55 |
| odyssey4me | :) | 14:55 |
| cloudnull | anyone around that might be able to review https://review.openstack.org/#/c/515995/ | 14:55 |
| evrardjp | reviewing. | 14:56 |
| andymccr | hwoarang: hey - yeah i am testing like an AIO ceph build | 15:00 |
| hwoarang | andymccr: do you test the ceph-ansible standalone at all? | 15:00 |
| andymccr | it's an aio based on ceph-ansible, but using containers (like an OSA AIO really, but just the ceph bits) | 15:00 |
| andymccr | so i do some host prep for containers, and then run ceph-ansible against it | 15:01 |
| hwoarang | do you think that's something I could use to test my suse additions in ceph-ansible? because the upstream vagrant file doesn't work out of the box and there are gazillion options | 15:01 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: docs: add a decision help https://review.openstack.org/504088 | 15:01 |
| hwoarang | plus they also have all the roles embedded in teh repo and they split it in separate repos in a magical way so i really have to test tin ceph-ansible repo as is | 15:02 |
| andymccr | hwoarang: sure - may need to add in some bits for SUSE to work properly (mainly thinking the network interfaces), i have CentOS7 support | 15:02 |
| hwoarang | andymccr: yeah that shouldn't be a problem. Can you share the details? | 15:03 |
| andymccr | https://github.com/rcbops/rpc-ceph - it works in the same way as an OSA aio really, just do ./run_tests.sh | 15:03 |
| andymccr | oh the run_tests will deploy the whole cluster but will fail on setting up monitoring bits later on. you can skip that or maybe i can add a flag to make that skippable | 15:03 |
| hwoarang | andymccr: aha i see ok! | 15:04 |
| hwoarang | thank you | 15:04 |
| *** hw_wutianwei has quit IRC | 15:06 | |
| *** esberglu has joined #openstack-ansible | 15:07 | |
| *** esberglu has quit IRC | 15:07 | |
| *** esberglu has joined #openstack-ansible | 15:07 | |
| andymccr | hwoarang: no problem - let me know if you run into some problems. ithink the biggest issue could be if there are differences in the losetup bits | 15:07 |
| andymccr | but i think SUSE will be ok, for example - trusty wont work though so im not setting that one up because the losetup command is older | 15:08 |
| hwoarang | lets see | 15:08 |
| andymccr | and off the top of my head the required suse network conf files arent there | 15:09 |
| andymccr | but those are easy since i just took teh tasks from osa so they should work once the template files are tehre | 15:09 |
| andymccr | *there | 15:09 |
| hwoarang | yeah | 15:10 |
| *** hw_wutianwei1 has joined #openstack-ansible | 15:12 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/pike: Fix container mac generation and remove extra tasks https://review.openstack.org/516344 | 15:12 |
| openstackgerrit | Alexandra Settle proposed openstack/openstack-ansible master: docs: add a decision help https://review.openstack.org/504088 | 15:14 |
| evrardjp | does anyone see me flapping on irc? My log is polluted by lots of "connection refused" on Freenode servers. | 15:14 |
| admin0 | "flappy evrardjp" | 15:18 |
| admin0 | you are as stable as usal .. not flapping | 15:18 |
| *** hachi has quit IRC | 15:18 | |
| evrardjp | weird. | 15:18 |
| evrardjp | but ok | 15:18 |
| *** phalmos has joined #openstack-ansible | 15:19 | |
| Tahvok | Can anyone suggest me on how to proceed with this issue? https://bugs.launchpad.net/openstack-ansible/+bug/1687594 | 15:26 |
| openstack | Launchpad bug 1687594 in openstack-ansible "no configuration for osa neutron to configure external dns yet" [Wishlist,In progress] - Assigned to Albert Mikaelyan (tahvok) | 15:26 |
| *** phalmos has quit IRC | 15:28 | |
| *** dmsimard|afk is now known as dmsimard | 15:30 | |
| *** redondo-mk has joined #openstack-ansible | 15:31 | |
| *** phalmos has joined #openstack-ansible | 15:35 | |
| *** mrch has quit IRC | 15:36 | |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: Add lxc-container-create tests https://review.openstack.org/513457 | 15:38 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 15:38 |
| cloudnull | Tahvok: are you wanting to use designate? | 15:38 |
| cloudnull | or do you just need to inject those sections into your neutron config? | 15:39 |
| Tahvok | cloudnull: I already use it. Just wanted to backport my fixes back to OSA | 15:39 |
| odyssey4me | Tahvok ok, so you've validated that using a keystone v3 config set doesn't work? | 15:40 |
| Tahvok | odyssey4me: not exactly. You need both keystone v3 and v2 | 15:40 |
| odyssey4me | Tahvok was this tested on Pike, or did you test with Queens too? | 15:41 |
| cloudnull | if you're wanting to get those sections into the neutron.conf file you can PR them here https://github.com/openstack/openstack-ansible-os_neutron/blob/master/templates/neutron.conf.j2 -- if you're just wanting to inject the required bits you can do so w/ config_template. | 15:41 |
| odyssey4me | @cloudnull you should read the bug ;) | 15:41 |
| odyssey4me | (with all comments) | 15:41 |
| * cloudnull reading | 15:41 | |
| cloudnull | ok. | 15:41 |
| evrardjp | Tahvok: that's a problem because keystone v2 in queens is kinda dead. | 15:42 |
| odyssey4me | ^ yup | 15:42 |
| Tahvok | On ocata actually. But I've checked here: https://github.com/openstack/neutron/commits/stable/pike/neutron/plugins/ml2/extensions/dns_integration.py | 15:42 |
| Tahvok | And there was almost no work done since ocata release | 15:43 |
| evrardjp | Tahvok: you should start with hamster first | 15:43 |
| evrardjp | I meant master. | 15:43 |
| Tahvok | you mean queens? | 15:44 |
| evrardjp | yes. | 15:44 |
| Tahvok | Will need to find time for that.. | 15:45 |
| Tahvok | I need to create a backend for designate to test it | 15:45 |
| redondo-mk | Hi. I'm missing some understanding in OSA...OSA creates OpenStack services in it's dedicated LXC containers within virutalenv...basically I'm just wondering if you have OpenStack service installed in virtualenv and updating it just installs it in seperate virtualenv, what is it that you gain with having the whole thing in LXC? I can understand for mariadb, rabbitmq and some other things but just | 15:46 |
| redondo-mk | OpenStack services I'm not sure what's the good reason to have it in LXC...and how can you replace neutron agent container without affecting data plane? | 15:46 |
| evrardjp | redondo-mk: some things are still running on bare metal when it makes more sense | 15:49 |
| evrardjp | redondo-mk: this project also started long ago, before openstack was very virtualenv friendly I guess | 15:49 |
| evrardjp | so we started without virtualenvs at that time | 15:50 |
| evrardjp | redondo-mk: the limit is more and more blurry, with the systemd slices too | 15:50 |
| Tahvok | How do you guys test OSA? Do you have a full bare metal machines, or just configure vms? | 15:50 |
| *** vnogin has quit IRC | 15:50 | |
| evrardjp | redondo-mk: if you want, you can run the service on metal, without lxc. Making possible to run everything on metal is one of my goals in this cycle | 15:51 |
| evrardjp | Tahvok: for a full integrated thing, I run mostly the AIO | 15:51 |
| evrardjp | else I run the role's tests | 15:51 |
| evrardjp | I do everything in vagrant because I am lazy | 15:51 |
| openstackgerrit | kourosh vivan proposed openstack/openstack-ansible-os_octavia master: Remove ignore error and unnecessary when https://review.openstack.org/515342 | 15:52 |
| Tahvok | but taking my case - when you need to setup another machine with bind9 backend | 15:52 |
| Tahvok | What would you do? | 15:52 |
| odyssey4me | Tahvok I just build a 8 vCPU, 8GB RAM, 100GB instance and use that to run an AIO or role tests | 15:52 |
| odyssey4me | oh, just add another container and build it in there | 15:52 |
| odyssey4me | or use another instance | 15:53 |
| Tahvok | And what about network and such? | 15:53 |
| *** vnogin has joined #openstack-ansible | 15:53 | |
| Tahvok | I need to create a network port to test designate. How am I supposed to configure the network? | 15:53 |
| *** jvidal has quit IRC | 15:54 | |
| odyssey4me | easiest is probably just another instance, then use its eth0 interface | 15:54 |
| odyssey4me | if that interface is public, obviously make sure you restrict usage from anything outside of your ranges | 15:54 |
| redondo-mk | @evrardjp, what do you mean by "one of the goals"...that doesn't work out of the box now if you configure it to not deploy LXC? And does OSA favor more Ubuntu OS (because of more deployments in Ubuntu?) then Red Hat based deployments? And how is it with systemd-nspawn containers...is this going to be promoted? I saw some work already being done in this direction... | 15:55 |
| evrardjp | redondo-mk: it worked, but we do not test it on a regular basis, and there may have a few bugs. | 15:55 |
| evrardjp | for example, you may want to skip haproxy if you run everything on the same host. So it's a different code path | 15:56 |
| evrardjp | but the deploy itself would work, it just needs different wiring | 15:56 |
| evrardjp | redondo-mk: that's a lot of questions | 15:56 |
| evrardjp | yes RedHat is already in our distro list. | 15:56 |
| evrardjp | all the big roles can be deployed with red hat | 15:57 |
| evrardjp | or centos | 15:57 |
| evrardjp | for systemd-nspawn, it's in the making too | 15:57 |
| redondo-mk | @evrardjp, yeah...maybe too many questions :) Well...I had better experience with Ubuntu then in CentOS...some things in CentOS failed that didn't in Ubuntu... | 15:58 |
| evrardjp | if you're willing to help in any of those topics, we'll welcome you with open arms. | 15:58 |
| evrardjp | if not, we'll welcome you anyway. | 15:58 |
| evrardjp | :D | 15:58 |
| evrardjp | redondo-mk: what did you try? | 15:58 |
| cloudnull | I'm sure mgariepy and mhayden would appreciate the issue reports ::D | 15:58 |
| evrardjp | and what was your issue? | 15:59 |
| evrardjp | redondo-mk: ^ see what cloudnull said :) | 15:59 |
| gunix | admin0: the cinder application is somehow crashed within the cinder containers. | 16:00 |
| gunix | admin0: i think best option here is to destroy the cinder api containers and redeploy it | 16:01 |
| openstackgerrit | Merged openstack/openstack-ansible master: docs: add a decision help https://review.openstack.org/504088 | 16:01 |
| admin0 | ignore cinder, run the rest of the playbook in steps | 16:01 |
| admin0 | then note down on top of cinder how many are crashed to redo | 16:01 |
| evrardjp | I have to go. See you laterz guys! | 16:02 |
| redondo-mk | @evrardjp, I can't remember at this point...I know when I just wanted to deploy AIO, there were more things that failed using CentOS...most of the issues I had were building wheels in repo lxc container failing...again, I don't have details but when I do any kind of deployment next time, I'll open a bug report if necessary... | 16:02 |
| gunix | admin0: oh yeah, that would be smart | 16:03 |
| admin0 | gunix, i do this kind of fulltime :D | 16:03 |
| redondo-mk | We are doing our upgrade deploying OpenStack services in systemd-nspawn so maybe I might be interested in doing some contribution there... | 16:04 |
| mhayden | redondo-mk: that would interest cloudnull | 16:04 |
| admin0 | redondo-mk, document everything :) .. i am interested to give it a try as well | 16:04 |
| admin0 | redondo-mk, is it a production or a test/dev ? | 16:04 |
| mhayden | redondo-mk: also, i've been running a four-node OSA cloud on CentOS 7 since Newton -- happy to help | 16:04 |
| odyssey4me | redondo-mk CentOS support was new & experimental in ocata and has been maturing since then... mhayden runs a small environment with it on pike | 16:04 |
| logan- | evrardjp odyssey4me: do either of you know which patch removed the legacy gate-check-commit job for integrated? | 16:04 |
| cloudnull | redondo-mk: ++ I have a WIP pr to use systemd-nspawn | 16:05 |
| logan- | trying to search for it now and not having much luck | 16:05 |
| odyssey4me | logan- yeah, I can probably find it - what's up? | 16:05 |
| odyssey4me | logan- https://review.openstack.org/514592 | 16:06 |
| logan- | in the job definition we dropped a printenv with the ZUUL_* vars and dumped them to a file called zuul.env, which gate-check-commit consumes inside sudo to restore the vars | 16:06 |
| cloudnull | recently we've gotten our LXC containers to run within a machinectl backend which gets us close. from a framework perspective | 16:06 |
| *** phalmos_ has joined #openstack-ansible | 16:06 | |
| logan- | it looks like that never got translated to the in repo tests | 16:06 |
| redondo-mk | Well...first we're putting it in test and then production...I'll share more if not everything when things will be happening... | 16:06 |
| odyssey4me | logan- yeah, I thought about that earlier | 16:06 |
| odyssey4me | logan- the thing is that we don't need any of those env vars any more | 16:07 |
| logan- | what should we use here instead? https://github.com/openstack/openstack-ansible/blob/bf1835b919e0c9546c61e216622fdc5c10268918/tests/get-ansible-role-requirements.yml#L56-L57 | 16:07 |
| odyssey4me | the approach I'm working on for the roles is to link anything found in the src directory zuul uses, and then clone anything missing | 16:07 |
| odyssey4me | *but* I realise that for role tests we don't pin ... so that doesn't completely work for the integrated gate | 16:08 |
| odyssey4me | so yes, I think the integrated gate may need some extra things - lemme find the bit it won't be in project-config | 16:08 |
| *** Oku_OS is now known as Oku_OS-away | 16:09 | |
| *** phalmos has quit IRC | 16:09 | |
| odyssey4me | logan- the meat of what was actually run is being removed in https://review.openstack.org/#/c/512635/ | 16:09 |
| logan- | ahh perfect thanks | 16:09 |
| odyssey4me | if we just use environment: '{{ zuul | zuul_legacy_vars }}' | 16:10 |
| odyssey4me | then add a task to dump to the file then we should be good... although we can probably skip the file dump thing | 16:10 |
| cloudnull | redondo-mk: ++ I've gotten the nspawn work running in my lab, if you have issues/questions I'd be happy to help, also I'd love to collaborate more on those things . | 16:11 |
| odyssey4me | logan- if we have the environment we should be able to change the process to not bother sourcing the zuul.env file any more | 16:12 |
| logan- | yeah | 16:12 |
| logan- | well probably not right now | 16:12 |
| logan- | we still run gate-check-commit in sudo | 16:12 |
| odyssey4me | sudo -E ? | 16:13 |
| odyssey4me | although we didn't do that before because we didn't want too much junk | 16:13 |
| odyssey4me | ja, so maybe the same method as previous would be good | 16:13 |
| logan- | yeah we can always refine it later, im thinking lets just get the file back where it was to get depends-on working and then improve from there | 16:14 |
| *** hw_wutianwei1 has quit IRC | 16:14 | |
| odyssey4me | good plan | 16:15 |
| redondo-mk | @cloudnull, that's nice to hear! I need to try it first and then I'll get back to you....how are neutron namespaces created within neutron agent systemd-nspawn containers? | 16:17 |
| *** woodard has joined #openstack-ansible | 16:21 | |
| xgerman_ | cores: Wonder if I can have some input on https://review.openstack.org/#/c/514852/ and https://review.openstack.org/#/c/515110/ | 16:22 |
| *** jamesden_ has quit IRC | 16:22 | |
| *** jamesdenton has joined #openstack-ansible | 16:23 | |
| cloudnull | I had to put the neutron-agents on baremetal, outside of the container however I have it constrained in the neutron cgroup and running within the venv. Neutron is still trying to loading kernel modules when the service starts which is not allowed within the nspawn. I also had issues with direct device access which nspawn does not allow. so the stance I've taken so far is to use containers for everything ephemeral and place applications on | 16:24 |
| cloudnull | baremetal when direct device access is required (neutron-agents, nova-compute, cinder-volumes). | 16:24 |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia master: Fix mgmt network creation tasks https://review.openstack.org/515331 | 16:25 |
| cloudnull | if there's a way to force/allow the loading of kernel modules within nspawn it'd be great to get that factored into the WIP | 16:25 |
| odyssey4me | cloudnull don't we already use delegation to load it on the host instead? | 16:26 |
| cloudnull | in lxc we bind mount libs into the container. | 16:27 |
| cloudnull | and do some magic to allow it to work | 16:27 |
| cloudnull | however that same interaction is blocked within nspawn | 16:27 |
| cloudnull | the use of nspawn is great but its quite a bit more restrictive than lxc when it comes to what is and is not allowed within the container. . | 16:30 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: [DNM] Update aio job for depends-on https://review.openstack.org/516383 | 16:31 |
| odyssey4me | cloudnull ok, but if we were to load the modules on the host would those modules be exposed to the container? | 16:32 |
| Miouge | cloudnull: it turned out that the ceph_client parts has ceph_mons != [] has a condition, so adding ceph_mons to my user_variables.yml helped | 16:34 |
| Miouge | Not sure how I am supposed to set the monitor IPs in ceph_mons beforehand. Can’t this be populated from the inventory? | 16:34 |
| cloudnull | odyssey4me: Yes. but its some of the access to those modules that is restricted. This is a little bit old - https://lists.freedesktop.org/archives/systemd-devel/2014-November/024913.html - but highlights some of the restrictions in terms of device pass-through. | 16:35 |
| cloudnull | there's no unconfiged profile that we would be able to apply | 16:35 |
| *** phalmos has joined #openstack-ansible | 16:36 | |
| cloudnull | odyssey4me: this is what we'd need to figure out to let neutron-agents run within the contianers | 16:37 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/master/playbooks/common-playbooks/neutron.yml#L31-L43 | 16:37 |
| cloudnull | i've not been able to figure something out that does that, but for nspawn | 16:37 |
| cloudnull | I can do the bind mount of course, however I can get unrestricted access to the devices and modeules from within the container. | 16:38 |
| odyssey4me | ja, it probably makes better sense to have them on the host anyway | 16:39 |
| cloudnull | that being said, IMHO, having the agents on the host instead of a container would be better from a stability standpoint. | 16:39 |
| *** phalmos_ has quit IRC | 16:39 | |
| cloudnull | removing / migrating a container wouldn't impact l3 availability | 16:39 |
| cloudnull | also that being said, it may just me being lazy :D | 16:40 |
| cloudnull | something more to figure out. | 16:40 |
| openstackgerrit | German Eichberger proposed openstack/openstack-ansible master: Add Neutron variables shared with Octavia https://review.openstack.org/515110 | 16:40 |
| cloudnull | so redondo-mk if you have a way to make it work, i'd be keenly interested. | 16:40 |
| * cloudnull lunching brb | 16:40 | |
| *** gouthamr has joined #openstack-ansible | 16:40 | |
| *** vnogin has quit IRC | 16:42 | |
| *** markvoelker_ has joined #openstack-ansible | 16:47 | |
| *** markvoelker has quit IRC | 16:49 | |
| *** dxiri has quit IRC | 16:50 | |
| *** markvoelker_ has quit IRC | 16:51 | |
| *** markvoelker has joined #openstack-ansible | 17:00 | |
| *** weezS has quit IRC | 17:03 | |
| jmccrory | odyssey4me are cross-repo depends-on with upstream projects expected to work? like https://review.openstack.org/#/c/515020/ | 17:03 |
| *** markvoelker has quit IRC | 17:05 | |
| *** deadnull has joined #openstack-ansible | 17:10 | |
| *** deadnull has quit IRC | 17:10 | |
| *** prometheanfire has quit IRC | 17:22 | |
| *** markvoelker has joined #openstack-ansible | 17:23 | |
| odyssey4me | jmccrory it should for master, yes - as far as I know | 17:24 |
| odyssey4me | I haven't tested it yet though, there may be bugs - https://review.openstack.org/516213 is supposed to be the thing making it work (that's the pike backport) | 17:25 |
| odyssey4me | I might have missed something though. Ping me if you find something - I'm out for the night. | 17:25 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: Update aio job to use become and fix environment https://review.openstack.org/516383 | 17:26 |
| jmccrory | ok i'll look through that today and let you know. have a good one | 17:26 |
| logan- | cya odyssey4me | 17:26 |
| logan- | btw odyssey4me, i threw out all of the sudo and zuul.env stuff after all: https://review.openstack.org/#/c/516383/2 :P | 17:27 |
| *** markvoelker has quit IRC | 17:28 | |
| *** mbuil has quit IRC | 17:29 | |
| *** markvoelker has joined #openstack-ansible | 17:30 | |
| openstackgerrit | German Eichberger proposed openstack/openstack-ansible-os_octavia master: Improve event streaming with a better Octavia verison https://review.openstack.org/514852 | 17:31 |
| odyssey4me | logan- FYI the reason I took the approach for role tests of always finding anything in the zuul src directory and using it, then always cloning anything missing, is that it's consistent in approach, it enables dev testing and is simple | 17:32 |
| odyssey4me | the integrated repo would need one more thing and that is to do a git checkout for anything that's not in the ZUUL_PROJECTS list (ie anything pinned, but not in depends-on) | 17:33 |
| odyssey4me | I'm happy to work on that later though, once we've got role tests doing well | 17:33 |
| logan- | sure | 17:33 |
| logan- | i don't think we even look at zuul projects list in the integrated repo | 17:34 |
| odyssey4me | nope | 17:34 |
| logan- | it just uses anything that zuul provides | 17:34 |
| *** markvoelker has quit IRC | 17:34 | |
| odyssey4me | I'd like to extend the depends-on support to the repo build too, so I'm keen to extend the dev support into the integrated repo in a subsequent patch anyway | 17:34 |
| odyssey4me | so yeah, I think your approach looks good to get things going again | 17:35 |
| logan- | ok cool | 17:35 |
| *** phalmos_ has joined #openstack-ansible | 17:36 | |
| * odyssey4me goes back to !computering | 17:36 | |
| logan- | yup cya! | 17:37 |
| odyssey4me | before I do that, please keep an eye on https://review.openstack.org/516213 and keep rechecking if it fails opensuse... then to help figure out what's going on with opensuse https://review.openstack.org/516249 needs reviews (check the comments for test patches) | 17:37 |
| * odyssey4me disappears in a puff of smoke | 17:38 | |
| *** phalmos has quit IRC | 17:39 | |
| *** dxiri has joined #openstack-ansible | 17:40 | |
| *** rromans_ has joined #openstack-ansible | 17:43 | |
| spotz | night odyssey4me | 17:52 |
| *** chhavi has quit IRC | 18:01 | |
| *** markvoelker has joined #openstack-ansible | 18:05 | |
| *** weezS has joined #openstack-ansible | 18:05 | |
| *** rromans_ has quit IRC | 18:07 | |
| *** markvoelker has quit IRC | 18:09 | |
| *** markvoelker has joined #openstack-ansible | 18:10 | |
| *** Brew_ has joined #openstack-ansible | 18:11 | |
| *** Brew_ has quit IRC | 18:13 | |
| lagman | i just deployed openstack with ansible, but i cant run any instances, "no available hosts" and there is no hypervisor active, can someone tell me where to look? | 18:14 |
| *** Brew_ has joined #openstack-ansible | 18:16 | |
| *** Brew_ has quit IRC | 18:18 | |
| *** Brew_ has joined #openstack-ansible | 18:18 | |
| *** markvoelker has quit IRC | 18:19 | |
| *** markvoelker has joined #openstack-ansible | 18:19 | |
| *** markvoelker_ has joined #openstack-ansible | 18:20 | |
| *** Brew has quit IRC | 18:20 | |
| *** Brew_ has quit IRC | 18:20 | |
| cloudnull | lagman: is that on queens ? | 18:20 |
| cloudnull | .. master | 18:20 |
| cloudnull | also check if the nova-compute services are running | 18:21 |
| cloudnull | and that the neutron-agents on the compute node are running | 18:21 |
| cloudnull | if so check the logs to see if there's something failing | 18:21 |
| mhayden | have we figured out the source of these OpenSUSE shenanigans yet? | 18:23 |
| mhayden | where the LXC container cache fails to build? | 18:23 |
| lagman | cloudnull its on ocata and yes nova-compute is running | 18:23 |
| *** markvoelker has quit IRC | 18:23 | |
| cloudnull | mhayden: I don't think so | 18:24 |
| cloudnull | lagman: is the neutron-linuxbridge-agent up ? | 18:25 |
| cloudnull | anything in the logs? | 18:25 |
| lagman | cloudnull neutron-linuxbridge-agent is up | 18:26 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_swift stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516305 | 18:27 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_swift stable/pike: Revert "Don't run sync in upgrades" https://review.openstack.org/516333 | 18:27 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_horizon master: Install sahara-dashboard local_settings https://review.openstack.org/511227 | 18:28 |
| cloudnull | lagman: anything in the logs ? | 18:28 |
| lagman | cloudnull on nova-conductor.log it says Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available. | 18:29 |
| lagman | on nova-scheduler.log Successfully synced instances from compute | 18:31 |
| mgariepy | how many compute do you have ? maybe the retry filter has kicked in. | 18:31 |
| cloudnull | do you see a compute node when you do a service list ? | 18:31 |
| lagman | only the one | 18:31 |
| lagman | and yeh ido | 18:32 |
| cloudnull | RE: filters, https://review.openstack.org/#/c/516050/ -- mhayden mgariepy :D | 18:32 |
| lagman | | 21 | nova-compute | KUMPAYA | nova | enabled | up | 2017-10-30T18:32:11.000000 | - | | 18:32 |
| *** markvoelker has joined #openstack-ansible | 18:32 | |
| cloudnull | lagman: maybe the flavor you've chosen to build with is incompatible with the host ? | 18:32 |
| mhayden | oh, NUMA, you say | 18:33 |
| *** markvoelker_ has quit IRC | 18:33 | |
| cloudnull | indeed | 18:33 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516294 | 18:33 |
| mhayden | is there a particular reason they're in that order? can we alphabetize it, cloudnull ? | 18:34 |
| cloudnull | its in the same order from before which i think is the order in which it hits the filter.s | 18:35 |
| mhayden | ah okay, i put a question in the review | 18:35 |
| *** markvoelker has quit IRC | 18:37 | |
| *** newmember has quit IRC | 18:37 | |
| openstackgerrit | Merged openstack/openstack-ansible-haproxy_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516278 | 18:41 |
| *** markvoelker has joined #openstack-ansible | 18:41 | |
| mgariepy | mhayden, I think the order is important. | 18:41 |
| openstackgerrit | Merged openstack/openstack-ansible-apt_package_pinning stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516274 | 18:42 |
| mhayden | mgariepy: alrighty | 18:42 |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516275 | 18:45 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_client stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516276 | 18:45 |
| jmccrory | odyssey4me ok, i think i see what's going on. the depends-on works correctly, but the role is using that particular repo as a developer mode constraint so fetches it separately https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L37-L43 | 18:47 |
| openstackgerrit | Merged openstack/openstack-ansible-memcached_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516281 | 18:47 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_openrc stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516283 | 18:48 |
| SamYaple | cloudnull: having a rabbitmq issue on a 400+ node none-OSA environment. its consuming upto 40GB of ram and then hanging because of rabbitmq limits on memory. the environment in question only has 30 VMs. have you seen this before? | 18:49 |
| SamYaple | i know we did alot of rabbitmq work in OSA early on | 18:49 |
| SamYaple | for these larger environments | 18:49 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516282 | 18:51 |
| *** hachi has joined #openstack-ansible | 18:51 | |
| cloudnull | I've seen nothing like that | 18:52 |
| cloudnull | We did a lot of work on tuning the EVM | 18:53 |
| cloudnull | but most of that was geared at thread workers | 18:53 |
| SamYaple | yea that was the first thing i did when i started seeing the issues (dropped all services threads to 1) | 18:53 |
| cloudnull | is this 400 node, compute nodes? | 18:53 |
| SamYaple | yes | 18:53 |
| cloudnull | maybe something to do with mirroed queues | 18:53 |
| cloudnull | are they using in-memory nodes? | 18:54 |
| SamYaple | im not mirroring the autogenerated queues | 18:54 |
| cloudnull | instead of disc-baked queues | 18:54 |
| SamYaple | they are on disk, not in-memory | 18:54 |
| cloudnull | hum. | 18:54 |
| SamYaple | im thinking maybe leaking queue | 18:54 |
| cloudnull | 40GiB or ram would be a lot in my experience | 18:54 |
| SamYaple | indeed. mine too | 18:54 |
| SamYaple | its actually like 50GB, 40% of 128GB | 18:55 |
| SamYaple | and thats the default limit for high meory usage | 18:55 |
| cloudnull | even our environments that are high node count high traffic don't use that much ram | 18:55 |
| cloudnull | could be a leaking queue, what version of rabbitmq ? | 18:55 |
| SamYaple | 3.5.7 | 18:55 |
| cloudnull | I could ask around to some of the operators to see if they've seen anything like that | 18:56 |
| cloudnull | but me, sadly not. | 18:56 |
| openstackgerrit | Major Hayden proposed openstack/ansible-hardening master: Optionally search for world-writable files https://review.openstack.org/513817 | 18:56 |
| SamYaple | ok cool. not having seen it before is actually new info, and new info is good :) | 18:56 |
| *** drifterza_ has joined #openstack-ansible | 18:59 | |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516317 | 19:01 |
| cloudnull | so a couple of our dedicated archs say to check the management plugin | 19:02 |
| cloudnull | and metric collection | 19:02 |
| cloudnull | they've seen that issue where the metrics collection db was SUPER large and needed to be flushed | 19:02 |
| SamYaple | i see... well as luck would have it.... | 19:03 |
| SamYaple | we use both of those things | 19:03 |
| cloudnull | also that the management plugin in general can lead to some find of a leak resulting in high memory usage. | 19:03 |
| cloudnull | **some kind of ... | 19:03 |
| SamYaple | yea we make heavy usage of it for an api | 19:04 |
| SamYaple | they restarted this before i got on, so i havent been able monitor it while its b0rked. but ill keep an eye on those things | 19:04 |
| SamYaple | that sounds like the issue. thanks :) | 19:04 |
| cloudnull | ++ | 19:04 |
| cloudnull | something to look into | 19:04 |
| *** drifterza_ has quit IRC | 19:05 | |
| cloudnull | when I asked them they were pretty firm, the metric collection DB was culprit which makes me believe they've seen this issue before/a lot | 19:05 |
| cloudnull | maybe something we should look into just having a standard flush interval for, assuming we don't already | 19:06 |
| cloudnull | jmccrory: ^ | 19:06 |
| *** phalmos has joined #openstack-ansible | 19:06 | |
| *** stuartgr has quit IRC | 19:06 | |
| jmccrory | yep, pretty much had to disable the management plugin after reaching a certain number of nodes/VMs | 19:07 |
| jmccrory | there's a few configurable neutron heartbeat intervals to play with too | 19:08 |
| *** phalmos_ has quit IRC | 19:09 | |
| SamYaple | cloudnull: https://github.com/rabbitmq/rabbitmq-management/issues/173 , testing a newer rabbitmq now,seems like the issues been resolved | 19:12 |
| *** shardy has quit IRC | 19:14 | |
| cloudnull | while I have some folks here: https://review.openstack.org/#/c/516002/ -- i'd appreciate some reviews there. | 19:15 |
| cloudnull | I think breaking down the jobs will help test new scenarios and keep our test coverage. | 19:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516293 | 19:16 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 19:16 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova stable/pike: Update filters to support over-commit https://review.openstack.org/516429 | 19:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516261 | 19:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.openstack.org/514698 | 19:25 |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia master: Remove ignore error and unnecessary when https://review.openstack.org/515342 | 19:28 |
| *** markvoelker has quit IRC | 19:34 | |
| *** markvoelker has joined #openstack-ansible | 19:34 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: IMAGES_ALLOW_LOCATION Wrongly Templated As String https://review.openstack.org/515133 | 19:43 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516280 | 19:43 |
| *** Brew has joined #openstack-ansible | 19:48 | |
| *** Brew has quit IRC | 19:53 | |
| *** Brew has joined #openstack-ansible | 19:55 | |
| *** Smeared_Beard has quit IRC | 19:58 | |
| *** kylek3h has quit IRC | 20:00 | |
| *** SmearedBeard has joined #openstack-ansible | 20:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516307 | 20:06 |
| *** oanson has quit IRC | 20:08 | |
| *** kylek3h has joined #openstack-ansible | 20:14 | |
| *** kylek3h_ has joined #openstack-ansible | 20:14 | |
| *** phalmos has quit IRC | 20:16 | |
| *** kylek3h has quit IRC | 20:18 | |
| *** mrtenio has quit IRC | 20:30 | |
| *** vnogin has joined #openstack-ansible | 20:31 | |
| *** markvoelker has quit IRC | 20:35 | |
| *** Brew has quit IRC | 20:37 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Revert "Set OpenStack-Ansible release to 17.0.0.0b1" https://review.openstack.org/514177 | 20:45 |
| openstackgerrit | Merged openstack/openstack-ansible master: Update zuul jobs with the per branch data https://review.openstack.org/515731 | 20:45 |
| openstackgerrit | Merged openstack/openstack-ansible stable/newton: Initial Zuulv3 jobs for Newton https://review.openstack.org/515713 | 20:45 |
| *** Brew has joined #openstack-ansible | 20:45 | |
| *** markvoelker has joined #openstack-ansible | 20:45 | |
| *** hachi has quit IRC | 20:46 | |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/pike: Zuul: add file extension to playbook path https://review.openstack.org/516074 | 20:48 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/ocata: Zuul: add file extension to playbook path https://review.openstack.org/516072 | 20:48 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Zuul: add file extension to playbook path https://review.openstack.org/516068 | 20:48 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/newton: Update tests for linters https://review.openstack.org/514038 | 20:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516288 | 20:48 |
| *** pbandark has quit IRC | 20:49 | |
| *** markvoelker has quit IRC | 20:49 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible master: Fix automatic log copying in Zuul runs https://review.openstack.org/516448 | 20:49 |
| *** markvoelker has joined #openstack-ansible | 20:49 | |
| *** phalmos has joined #openstack-ansible | 20:57 | |
| *** woodard has quit IRC | 21:00 | |
| dmsimard | oi, ara 0.14.5 is out, changelog: https://github.com/openstack/ara/releases/tag/0.14.5 | 21:01 |
| *** xingchao has joined #openstack-ansible | 21:02 | |
| *** markvoelker_ has joined #openstack-ansible | 21:02 | |
| gunix | can you get br-mgmt, br-storage and br-vxlan on the same IP, using OSA? | 21:04 |
| *** markvoelker has quit IRC | 21:04 | |
| *** smatzek has quit IRC | 21:09 | |
| *** Brew has quit IRC | 21:10 | |
| *** smatzek has joined #openstack-ansible | 21:10 | |
| *** esberglu has quit IRC | 21:11 | |
| *** esberglu has joined #openstack-ansible | 21:12 | |
| *** xingchao has quit IRC | 21:14 | |
| *** smatzek has quit IRC | 21:14 | |
| *** pcaruana has quit IRC | 21:15 | |
| *** esberglu has quit IRC | 21:16 | |
| *** kylek3h_ has quit IRC | 21:20 | |
| *** dave-mccowan has quit IRC | 21:20 | |
| *** kylek3h has joined #openstack-ansible | 21:21 | |
| *** esberglu has joined #openstack-ansible | 21:23 | |
| *** kylek3h has quit IRC | 21:25 | |
| *** jwitko_ has joined #openstack-ansible | 21:29 | |
| *** esberglu has quit IRC | 21:31 | |
| *** esberglu has joined #openstack-ansible | 21:32 | |
| *** jwitko has quit IRC | 21:32 | |
| *** esberglu has quit IRC | 21:32 | |
| *** esberglu has joined #openstack-ansible | 21:32 | |
| *** jwitko_ has quit IRC | 21:34 | |
| *** smatzek has joined #openstack-ansible | 21:36 | |
| *** thorst has quit IRC | 21:38 | |
| openstackgerrit | Merged openstack/openstack-ansible-pip_install stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516311 | 21:39 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_client stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516316 | 21:40 |
| *** smatzek has quit IRC | 21:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_aodh stable/ocata: Updated from OpenStack Ansible Tests https://review.openstack.org/514748 | 21:44 |
| *** woodard has joined #openstack-ansible | 21:44 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_freezer master: Updated from global requirements https://review.openstack.org/514258 | 21:46 |
| hwoarang | mhayden: around? :) | 21:47 |
| hwoarang | i have a bug for ansible-hardening | 21:47 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 21:49 |
| cloudnull | gunix: like you want them to all be smashed into one bridge? | 21:50 |
| SamYaple | so im writing a small deb package to install the galera scripts and helper service cloudnull. OSA will still need to figure out getting etcd3 client and server going (i checked the etcd server in OSA, its etcd2) | 21:51 |
| cloudnull | gunix: I only use the one bridge in my lab environment. | 21:52 |
| SamYaple | but that way we can share the work among teams | 21:52 |
| cloudnull | http://paste.openstack.org/show/625012/ | 21:52 |
| *** galstrom is now known as galstrom_zzz | 21:53 | |
| cloudnull | gunix: here's the openstack_user_config.yml i use http://paste.openstack.org/show/625014 which gets it all wired up on that one bridge | 21:53 |
| cloudnull | SamYaple: cool! | 21:54 |
| * cloudnull wonders what it would take to stand-up etcd3 | 21:54 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516259 | 21:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_gnocchi stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516260 | 21:54 |
| SamYaple | yea they really dont have anything for etcd3 outside of docker images. so its a tough one | 21:55 |
| cloudnull | it looks like we'd need to compile it | 21:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-galera_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516277 | 21:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_almanach master: Updated from OpenStack Ansible Tests https://review.openstack.org/514692 | 21:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516262 | 21:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_container_create stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516279 | 21:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_freezer master: Updated from OpenStack Ansible Tests https://review.openstack.org/514694 | 21:55 |
| SamYaple | i suppose if we are doing a new package anyway for the recovery service, then hosting etcd3 in a ppa isnt the end of the world | 21:55 |
| SamYaple | licensing permitting | 21:55 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-ops stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516284 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_aodh stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516285 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_barbican stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516286 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516287 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_monasca master: Updated from OpenStack Ansible Tests https://review.openstack.org/514695 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_monasca-agent master: Updated from OpenStack Ansible Tests https://review.openstack.org/514696 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_designate stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516289 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_monasca-ui master: Updated from OpenStack Ansible Tests https://review.openstack.org/514697 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_glance stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516290 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_gnocchi stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516291 | 21:56 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_heat stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516292 | 21:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_keystone stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516295 | 21:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_magnum stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516297 | 21:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_molteniron stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516298 | 21:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_monasca stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516299 | 21:57 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Revert "Revert "Move log collection to 'post' stage"" https://review.openstack.org/516249 | 21:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_neutron stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516300 | 21:57 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_nova stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516301 | 21:58 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_octavia stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516302 | 21:58 |
| *** thorst has joined #openstack-ansible | 21:58 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_rally stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516303 | 21:58 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516304 | 21:58 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_swift stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516305 | 21:58 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tacker stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516306 | 21:58 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_trove stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516308 | 21:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_watcher stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516309 | 21:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_zaqar stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516310 | 21:59 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Revert "Revert "Move log collection to 'post' stage"" https://review.openstack.org/516249 | 21:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-plugins stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516312 | 21:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-rabbitmq_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516313 | 21:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-repo_build stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516314 | 21:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-repo_server stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516315 | 21:59 |
| cloudnull | hwoarang: mhayden: if you're around, https://review.openstack.org/#/c/516249 needed to be rebased. | 21:59 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/ansible-hardening stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516318 | 22:00 |
| cloudnull | if you guys can re-vote on that one it'd be great to have it start the gate cycle again | 22:00 |
| *** thorst has quit IRC | 22:02 | |
| *** jwitko has joined #openstack-ansible | 22:04 | |
| *** phalmos has quit IRC | 22:05 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone stable/ocata: Bypass web server during service setup https://review.openstack.org/515555 | 22:07 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone stable/newton: Bypass web server during service setup https://review.openstack.org/515556 | 22:07 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_freezer master: Updated from global requirements https://review.openstack.org/514258 | 22:08 |
| *** oanson has joined #openstack-ansible | 22:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_freezer master: Updated from global requirements https://review.openstack.org/514258 | 22:11 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_freezer master: Updated from global requirements https://review.openstack.org/514258 | 22:11 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_nova master: Update test image for nova-lxd https://review.openstack.org/515020 | 22:11 |
| openstackgerrit | Merged openstack/openstack-ansible master: Update aio job to use become and fix environment https://review.openstack.org/516383 | 22:14 |
| *** marst has quit IRC | 22:15 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Remove the AIO scenario & add new scenarios to maintain coverage https://review.openstack.org/516002 | 22:15 |
| *** Miouge has quit IRC | 22:18 | |
| cloudnull | hwoarang: any idea where we might start looking into what causes the lxc container cache created to hang? | 22:18 |
| cloudnull | http://logs.openstack.org/22/514722/1/gate/openstack-ansible-functional-opensuse-423/dfff829/job-output.txt.gz#_2017-10-30_22_04_03_407236 | 22:19 |
| hwoarang | cloudnull: i am waiting for https://review.openstack.org/#/c/516249/ to merge | 22:19 |
| hwoarang | once the logs are there i will look into the lxc-cache-preparation log file that we will collect | 22:19 |
| cloudnull | I've a lab w/ openSUSE leap images anything that you can think of off hand that could be the issue? | 22:20 |
| * cloudnull happy to help test | 22:20 | |
| hwoarang | cloudnull: unfortunately i only see this problem on the gates. i run opensuse jobs locally and in the opnfv CI but i can't reproduce it :( | 22:20 |
| hwoarang | so i dont know... | 22:20 |
| cloudnull | :'( | 22:21 |
| cloudnull | welp if you find something and want me to test, i'd be happy to | 22:21 |
| hwoarang | i will let you know! thank you | 22:21 |
| hwoarang | the patch is on its way in so i hope to have more info tomorrow | 22:21 |
| *** SmearedBeard has quit IRC | 22:32 | |
| *** mwynne has joined #openstack-ansible | 22:42 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_nova master: Update test image for nova-lxd https://review.openstack.org/515020 | 22:44 |
| SamYaple | does OSA have any python3 plans/actions in progress? | 23:10 |
| cloudnull | SamYaple: OSA itself is running out of py3 when its available. however I think there's still work to be done to get the services themselves to be built (wheels, venvs) using py3. | 23:13 |
| *** jwitko has quit IRC | 23:13 | |
| cloudnull | I'm not sure where that was left off. | 23:13 |
| SamYaple | cool | 23:15 |
| SamYaple | im doing a python3 cluster and its.... working... fine | 23:15 |
| SamYaple | just was looking for more experince | 23:15 |
| cloudnull | i think its been touched on, and i think we have py3 covered in the pip_install role? | 23:16 |
| SamYaple | i meant like operational experince, ifanyone was running it. common issues/performance bits that kind of thing | 23:16 |
| cloudnull | ah. | 23:17 |
| cloudnull | none that im aware of | 23:17 |
| cloudnull | at least from my perspective | 23:17 |
| SamYaple | not seeing much evidence of it in openstack as a whole either :) | 23:17 |
| cloudnull | hwoarang: http://paste.openstack.org/show/625019/ -- setting up suse 42.3 poking at things I'm seeing an issue installing "sysstat". | 23:18 |
| cloudnull | are there better repos that we should be using by chance? | 23:18 |
| SamYaple | on a different note.. im pretty psyched about getting all the infra services more or less stateless start/restart in any order | 23:18 |
| SamYaple | im going to work on rabbitmq a bit to see if i can get etcd to lock in such a way we dont lose messages when upgrading (wait for them to flush before restarting) | 23:19 |
| SamYaple | that way we wont even need ordered shutdowns on that | 23:20 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/newton: Zuul: add file extension to playbook path https://review.openstack.org/516070 | 23:28 |
| cloudnull | hwoarang: http://paste.openstack.org/show/625020/ these are the repos on my instance. | 23:28 |
| cloudnull | the base image came from the openstack cloud repo, maybe I should use a different image? | 23:28 |
| *** dave-mccowan has joined #openstack-ansible | 23:29 | |
| cloudnull | SamYaple: that'd be awesome! Not having to worry about ordered shutdowns of rabbitmq is a big win. | 23:29 |
| SamYaple | with etcd you can not worry about ordered *starts*. but potentially lose messages on a too-fast container upgrade and rabbitmq-server restart across the board | 23:30 |
| SamYaple | would be nice to get rid of that hiccup | 23:30 |
| *** lbragstad has quit IRC | 23:32 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516290 | 23:32 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516259 | 23:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_gnocchi stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516260 | 23:34 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara stable/newton: Updated from OpenStack Ansible Tests https://review.openstack.org/516262 | 23:34 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat stable/pike: Updated from OpenStack Ansible Tests https://review.openstack.org/516292 | 23:36 |
| *** thorst has joined #openstack-ansible | 23:39 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_hosts master: [DNM] TEST https://review.openstack.org/516474 | 23:41 |
| *** esberglu_ has joined #openstack-ansible | 23:51 | |
| *** gunix has quit IRC | 23:52 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_hosts master: [DNM] TEST https://review.openstack.org/516474 | 23:53 |
| *** esberglu has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!