| bgmccollum | infra was restarting the zuul runners | 00:27 |
|---|---|---|
| *** nicolasbock has quit IRC | 00:36 | |
| openstackgerrit | Byron McCollum proposed openstack/openstack-ansible-os_swift master: Move swift ring tasks before systemd tasks https://review.openstack.org/592748 | 00:40 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: AIO: Tune down the glance uwsgi processes https://review.openstack.org/591062 | 00:52 |
| openstackgerrit | Merged openstack/openstack-ansible stable/queens: AIO: Tune down the glance uwsgi processes https://review.openstack.org/592347 | 00:52 |
| *** dave-mccowan has joined #openstack-ansible | 00:58 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_cinder master: use include_tasks instead of include https://review.openstack.org/585307 | 01:27 |
| *** aicilamd has joined #openstack-ansible | 01:27 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_cinder master: use include_tasks instead of include https://review.openstack.org/585307 | 01:29 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_cinder master: use include_tasks instead of include https://review.openstack.org/585307 | 01:31 |
| *** aicilamd has quit IRC | 01:34 | |
| *** aicilamd has joined #openstack-ansible | 01:35 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Convert test network setup to use the networkd role https://review.openstack.org/592596 | 01:36 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_zaqar master: use include_tasks instead of include https://review.openstack.org/585301 | 01:36 |
| *** aicilamd has quit IRC | 01:37 | |
| *** aicilamd has joined #openstack-ansible | 01:37 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_rally master: Use include_tasks instead of include https://review.openstack.org/585286 | 01:39 |
| *** ianychoi has joined #openstack-ansible | 01:47 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Convert test network setup to use the networkd role https://review.openstack.org/592596 | 01:53 |
| *** spatel has joined #openstack-ansible | 01:55 | |
| spatel | cloudnull: are you there? | 01:56 |
| spatel | any idea what is going on here? http://paste.openstack.org/show/728265/ | 01:56 |
| spatel | DB connection error but everything else working fine.. DB is up | 01:57 |
| spatel | I am going to look deeper | 01:57 |
| cloudnull | spatel ive seen that where the lb severs the connection | 01:59 |
| spatel | I am using F5 lb | 02:00 |
| spatel | I am testing some stuff and i have shutdown infra-02 node to see how my cluster behaving.. | 02:01 |
| spatel | and i noticed that error so looking into | 02:01 |
| spatel | its interesting... if i shutdown my main galaera node then this api shut go to next server | 02:03 |
| spatel | does this service need restart to initiate new connection... | 02:04 |
| *** aludwar has joined #openstack-ansible | 02:07 | |
| spatel | cloudnull: does haproxy provide connection mirroring or ip_conntrack ? | 02:14 |
| cloudnull | hap has a stick table setup | 02:14 |
| cloudnull | you should be able to do the same / similar with an f5 | 02:15 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Convert test network setup to use the networkd role https://review.openstack.org/592596 | 02:16 |
| spatel | does stick table sync connection? | 02:18 |
| cloudnull | spatel https://cbonte.github.io/haproxy-dconv/1.7/intro.html#3.3.11 | 02:19 |
| cloudnull | it has a form of connecion tracking | 02:20 |
| spatel | hmm! | 02:20 |
| cloudnull | but doesn't sync between hap nodes | 02:20 |
| spatel | in F5 i am seeing when i shutdown my primary MySQL box, all other nodes start throwing mysql connect error | 02:21 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Add SUSE 15 to the nspawn hosts role https://review.openstack.org/592177 | 02:21 |
| spatel | F5 shut transfer their connection technically to new MySQL | 02:22 |
| spatel | let me see | 02:22 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_hosts master: Add SUSE 15 to the nspawn hosts role https://review.openstack.org/592177 | 02:34 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-tests master: Convert test network setup to use the networkd role https://review.openstack.org/592596 | 02:38 |
| cloudnull | spatel so maybe its not promoting the other nodes ? | 02:39 |
| spatel | it does | 02:39 |
| spatel | If i run mysql command and connect to VIP it works | 02:39 |
| spatel | look like F5 not transferring active connection to pool member | 02:39 |
| spatel | Active Connections Never Bleed Off of Node | 02:40 |
| cloudnull | so is it not establishing a new connection? | 02:40 |
| cloudnull | ah i see | 02:40 |
| cloudnull | might need dr. jamesdenton to chime in, im not sure what to do in that regard | 02:41 |
| spatel | If client connected to pool member 1 and if i shutdown pool member 1 then that active connection not bleed off to member 2 | 02:41 |
| spatel | I am surprised how it works in HAProxy | 02:41 |
| spatel | This is not sticky feature issue.. sticky is used in round-robin where client connection stay on same server.. | 02:42 |
| spatel | jamesdenton: can help! | 02:42 |
| spatel | lets see if i can find him in morning or shoot an email :) | 02:43 |
| *** chyka has joined #openstack-ansible | 03:11 | |
| *** sawblade6 has joined #openstack-ansible | 03:20 | |
| *** sawblade6 has quit IRC | 03:24 | |
| *** gkadam_ has joined #openstack-ansible | 03:24 | |
| *** ssbarnea has joined #openstack-ansible | 03:28 | |
| *** sawblade6 has joined #openstack-ansible | 03:28 | |
| *** sawblade6 has quit IRC | 03:36 | |
| *** sawblade6 has joined #openstack-ansible | 03:41 | |
| *** dave-mccowan has quit IRC | 03:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar master: use include_tasks instead of include https://review.openstack.org/585268 | 03:46 |
| *** udesale has joined #openstack-ansible | 03:47 | |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_client master: use include_tasks instead of include https://review.openstack.org/585274 | 03:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_molteniron master: use include_tasks instead of include https://review.openstack.org/585303 | 03:49 |
| openstackgerrit | Merged openstack/openstack-ansible-memcached_server master: use include_tasks instead of include https://review.openstack.org/585270 | 03:49 |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client master: use include_tasks instead of include https://review.openstack.org/585285 | 03:51 |
| *** aicilamd has quit IRC | 04:00 | |
| *** aicilamd has joined #openstack-ansible | 04:01 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_searchlight master: use include_tasks instead of include https://review.openstack.org/585313 | 04:03 |
| *** poopcat has quit IRC | 04:04 | |
| *** aicilamd has quit IRC | 04:05 | |
| *** Bhujay has joined #openstack-ansible | 04:06 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_watcher master: Use include_tasks instead of include https://review.openstack.org/585210 | 04:06 |
| openstackgerrit | Merged openstack/openstack-ansible-os_congress master: use include_tasks instead of include https://review.openstack.org/585266 | 04:06 |
| *** chyka has quit IRC | 04:07 | |
| *** ssbarnea has quit IRC | 04:12 | |
| *** sawblade6 has quit IRC | 04:18 | |
| *** ssbarnea has joined #openstack-ansible | 04:21 | |
| *** mmalchuk has quit IRC | 04:26 | |
| spatel | cloudnull: I have opened bug because its happening everywhere in cluster https://bugs.launchpad.net/openstack-ansible/+bug/1787517 | 04:28 |
| openstack | Launchpad bug 1787517 in openstack-ansible "After primary node shutdown getting MySQL connection error " [Undecided,New] | 04:28 |
| *** gkadam_ has quit IRC | 04:30 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Make the project home link in next line https://review.openstack.org/584586 | 04:31 |
| openstackgerrit | Merged openstack/openstack-ansible-os_almanach master: Add the project source code repository in README https://review.openstack.org/584877 | 04:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Update the description for glance api https://review.openstack.org/584764 | 04:32 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: Update the project home to https https://review.openstack.org/584584 | 04:33 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca master: Add the project source code repository in README https://review.openstack.org/584841 | 04:33 |
| openstackgerrit | Merged openstack/openstack-ansible-os_congress master: Update the project home to https https://review.openstack.org/584585 | 04:34 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Add the project source code repository in README https://review.openstack.org/584222 | 04:34 |
| openstackgerrit | Merged openstack/openstack-ansible-os_panko master: Add the project source code repository in README https://review.openstack.org/584840 | 04:35 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest master: Add the project source code repository in README https://review.openstack.org/584846 | 04:35 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Add the project source code repository in README https://review.openstack.org/584839 | 04:35 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: Add repository into README.rst https://review.openstack.org/584880 | 04:38 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar master: Add the project source code repository in README https://review.openstack.org/584834 | 04:38 |
| openstackgerrit | Merged openstack/openstack-ansible-os_watcher master: Add the project source code repository in README https://review.openstack.org/584833 | 04:39 |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590853 | 04:39 |
| *** sawblade6 has joined #openstack-ansible | 04:40 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: Remove the duplicated word https://review.openstack.org/577455 | 04:40 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-ceph_client master: Added possibility to specify ssh username and port for ceph mons. https://review.openstack.org/583943 | 04:41 |
| openstackgerrit | Merged openstack/openstack-ansible-ops stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590886 | 04:41 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_openrc stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590883 | 04:42 |
| openstackgerrit | Merged openstack/openstack-ansible-os_molteniron stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590941 | 04:42 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create master: Remove the unnecessary space https://review.openstack.org/582785 | 04:43 |
| openstackgerrit | Merged openstack/openstack-ansible-os_searchlight master: remove unnecessary '=' https://review.openstack.org/587294 | 04:43 |
| openstackgerrit | Merged openstack/openstack-ansible-apt_package_pinning stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590850 | 04:43 |
| openstackgerrit | Merged openstack/openstack-ansible-pip_install stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590992 | 04:43 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca-ui master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592600 | 04:43 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_server master: use include_tasks instead of include https://review.openstack.org/585302 | 04:43 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_server stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/591011 | 04:43 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_client stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/591008 | 04:43 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Revert "Revert "Update UPPER_CONSTRAINTS_FILE for stable/rocky"" https://review.openstack.org/592312 | 04:44 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590898 | 04:44 |
| *** sawblade6 has quit IRC | 04:44 | |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590880 | 04:44 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_client stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590856 | 04:45 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar master: Trivial: Fix the pep8 warning https://review.openstack.org/589032 | 04:45 |
| openstackgerrit | Merged openstack/openstack-ansible-haproxy_server stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590862 | 04:45 |
| openstackgerrit | Merged openstack/openstack-ansible-os_molteniron master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592597 | 04:46 |
| openstackgerrit | Merged openstack/openstack-ansible-apt_package_pinning master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592483 | 04:47 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_client master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592620 | 04:47 |
| openstackgerrit | Merged openstack/openstack-ansible-memcached_server stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590871 | 04:49 |
| *** jamesdenton has quit IRC | 04:51 | |
| openstackgerrit | Merged openstack/openstack-ansible-pip_install master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592615 | 04:51 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_server master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592621 | 04:52 |
| openstackgerrit | Merged openstack/openstack-ansible-plugins stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590995 | 04:52 |
| openstackgerrit | Merged openstack/openstack-ansible master: Imported Translations from Zanata https://review.openstack.org/591231 | 04:52 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/591004 | 04:54 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca-agent master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592599 | 04:54 |
| openstackgerrit | Merged openstack/openstack-ansible-os_almanach master: use include_tasks instead of include https://review.openstack.org/585279 | 04:54 |
| openstackgerrit | Merged openstack/openstack-ansible-os_monasca-agent master: Trivial: Fix the pep8 warning https://review.openstack.org/589049 | 04:57 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_container_create stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590865 | 04:57 |
| openstackgerrit | Merged openstack/openstack-ansible-plugins master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592616 | 04:58 |
| openstackgerrit | Merged openstack/ansible-hardening master: Add option to skip sudoers NOPASSWD check https://review.openstack.org/565959 | 04:59 |
| openstackgerrit | Merged openstack/ansible-hardening master: Add release note link in README https://review.openstack.org/579069 | 04:59 |
| openstackgerrit | Merged openstack/ansible-hardening master: Add document information to readme https://review.openstack.org/579321 | 04:59 |
| openstackgerrit | Merged openstack/ansible-hardening master: import zuul job settings from project-config https://review.openstack.org/590080 | 04:59 |
| openstackgerrit | Merged openstack/ansible-hardening master: SUSE: Add support for openSUSE Leap 15 https://review.openstack.org/575381 | 04:59 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_server master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592619 | 04:59 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/rocky: Remove tasks that mangle iptables (checksum, NAT) https://review.openstack.org/592685 | 05:00 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Add NODEPOOL_LXC_IMAGE_PROXY for faster and more reliable images https://review.openstack.org/592752 | 05:00 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/591001 | 05:00 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally master: Use include_tasks instead of include https://review.openstack.org/585286 | 05:01 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally master: Add the project source code repository in README https://review.openstack.org/584845 | 05:01 |
| openstackgerrit | Merged openstack/openstack-ansible-os_almanach stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590889 | 05:01 |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592618 | 05:01 |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590998 | 05:02 |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592617 | 05:04 |
| openstackgerrit | Merged openstack/openstack-ansible-os_panko master: use include_tasks instead of include https://review.openstack.org/585269 | 05:04 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590962 | 05:04 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590974 | 05:06 |
| openstackgerrit | Merged openstack/openstack-ansible-os_congress stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590910 | 05:06 |
| openstackgerrit | Merged openstack/openstack-ansible-os_watcher stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590983 | 05:07 |
| openstackgerrit | Merged openstack/openstack-ansible-tests stable/rocky: Add qdrouterd user to MQ setup and update file https://review.openstack.org/592040 | 05:08 |
| openstackgerrit | Merged openstack/openstack-ansible-os_panko stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590959 | 05:09 |
| openstackgerrit | Merged openstack/openstack-ansible-os_gnocchi stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590920 | 05:09 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590938 | 05:10 |
| openstackgerrit | Merged openstack/openstack-ansible-os_watcher master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592612 | 05:10 |
| openstackgerrit | Merged openstack/openstack-ansible-os_panko master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592604 | 05:10 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590923 | 05:10 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592605 | 05:11 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_aodh stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592789 | 05:12 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_barbican stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592790 | 05:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592595 | 05:12 |
| *** spatel has quit IRC | 05:12 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_searchlight master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592607 | 05:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_searchlight master: Trvial: Fix the pep8 warning https://review.openstack.org/589058 | 05:12 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592791 | 05:12 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cinder stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592792 | 05:12 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cloudkitty stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592793 | 05:12 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_congress stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592794 | 05:12 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_designate stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592798 | 05:12 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_glance stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592802 | 05:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_heat stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592804 | 05:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ironic stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592808 | 05:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_keystone stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592811 | 05:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_magnum stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592813 | 05:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_masakari stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592815 | 05:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_neutron stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592819 | 05:14 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_nova stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592820 | 05:14 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_octavia stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592821 | 05:14 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592822 | 05:14 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_swift stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592823 | 05:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tacker stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592824 | 05:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Trivial: Fix the pep8 warning https://review.openstack.org/589046 | 05:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_trove stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592825 | 05:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_watcher stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592826 | 05:15 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_zun stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592827 | 05:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592589 | 05:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty master: Trivial: Fix the pep8 warning https://review.openstack.org/589036 | 05:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: use include_tasks instead of include https://review.openstack.org/585305 | 05:17 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592509 | 05:18 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592504 | 05:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_watcher master: Trivial: Fix the pep8 warning https://review.openstack.org/589061 | 05:19 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592590 | 05:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_rally master: Trivial: Fix the pep8 warning https://review.openstack.org/589056 | 05:21 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: Remove the unnecessary space https://review.openstack.org/582779 | 05:21 |
| *** mmalchuk has joined #openstack-ansible | 05:22 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592609 | 05:22 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590932 | 05:22 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592593 | 05:22 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: use include_tasks instead of include https://review.openstack.org/585284 | 05:23 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Add the project source code repository in README https://review.openstack.org/584842 | 05:23 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592587 | 05:25 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Remove the unnecessary space https://review.openstack.org/582784 | 05:25 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Trivial: Fix the pep8 warning https://review.openstack.org/589031 | 05:27 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592606 | 05:28 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590989 | 05:31 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592614 | 05:31 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Drop un-used packages from role https://review.openstack.org/591569 | 05:33 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: use include_tasks instead of include https://review.openstack.org/585310 | 05:35 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590929 | 05:38 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592592 | 05:41 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Trivial: Fix the pep8 warning https://review.openstack.org/589045 | 05:47 |
| *** sawblade6 has joined #openstack-ansible | 05:49 | |
| *** sawblade6 has quit IRC | 05:54 | |
| *** Tahvok has quit IRC | 06:06 | |
| *** Bhujay has quit IRC | 06:07 | |
| *** Tahvok has joined #openstack-ansible | 06:11 | |
| *** aludwar has quit IRC | 06:18 | |
| *** aludwar has joined #openstack-ansible | 06:23 | |
| *** DanyC has joined #openstack-ansible | 06:23 | |
| *** mmalchuk_ has joined #openstack-ansible | 06:24 | |
| *** mmalchuk has quit IRC | 06:26 | |
| openstackgerrit | melissaml proposed openstack/openstack-ansible master: Remove the duplicated word https://review.openstack.org/592870 | 06:27 |
| openstackgerrit | melissaml proposed openstack/openstack-ansible-lxc_container_create master: Remove the duplicated word https://review.openstack.org/592874 | 06:30 |
| *** DanyC has quit IRC | 06:45 | |
| *** shyamb has joined #openstack-ansible | 06:45 | |
| *** cshen has joined #openstack-ansible | 06:50 | |
| *** pcaruana has joined #openstack-ansible | 06:54 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: [TEST] bionic https://review.openstack.org/586607 | 06:55 |
| *** sawblade6 has joined #openstack-ansible | 06:59 | |
| *** sawblade6 has quit IRC | 07:03 | |
| evrardjp | today I've got time to figure out the restart of the services. anyone want to tag along for the research ride? | 07:10 |
| evrardjp | :p | 07:10 |
| * hwoarang is still fighting with the strategy plugin for opnfv | 07:11 | |
| evrardjp | hwoarang: you are skipping cinder in opnfv? | 07:11 |
| hwoarang | hmm | 07:12 |
| hwoarang | no | 07:12 |
| hwoarang | but fo some reason we have our own cinder.yml for env.d | 07:13 |
| *** Bhujay has joined #openstack-ansible | 07:16 | |
| *** fghaas has joined #openstack-ansible | 07:22 | |
| *** sawblade6 has joined #openstack-ansible | 07:27 | |
| evrardjp | interesting. Anyway it still needs doing for fixing this | 07:30 |
| *** sawblade6 has quit IRC | 07:31 | |
| *** shardy has joined #openstack-ansible | 07:39 | |
| *** shardy has quit IRC | 07:40 | |
| *** shardy has joined #openstack-ansible | 07:41 | |
| *** cshen has quit IRC | 07:45 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_cloudkitty master: Completed required variables for README.rst https://review.openstack.org/583981 | 07:46 |
| *** shyamb has quit IRC | 07:47 | |
| noonedeadpunk | evrardjp: can you please check these cherry-picks? https://review.openstack.org/#/c/591336/ https://review.openstack.org/#/c/592635/ ? | 07:48 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts master: Allow user-defined extra distro packages https://review.openstack.org/591412 | 07:54 |
| *** cshen has joined #openstack-ansible | 07:58 | |
| *** electrofelix has joined #openstack-ansible | 07:59 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Adds networking-ovn support https://review.openstack.org/584067 | 08:05 |
| *** hamzaachi has joined #openstack-ansible | 08:07 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: tests: Use 'ansible_connection=local' for connecting to localhost https://review.openstack.org/592529 | 08:08 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: strategy: linear: Inspect the delegated host on delegated tasks. https://review.openstack.org/591672 | 08:08 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible master: zuul: Add ceph jobs for distribution installations https://review.openstack.org/591949 | 08:09 |
| *** DanyC has joined #openstack-ansible | 08:09 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-ceph_client master: Added possibility to specify ssh username and port for ceph mons. https://review.openstack.org/583943 | 08:10 |
| noonedeadpunk | evrardjp: ^ can you check it as well? It's pretty usefull (as for me) and light patch. | 08:12 |
| *** sawblade6 has joined #openstack-ansible | 08:15 | |
| *** sawblade6 has quit IRC | 08:20 | |
| *** pdeters has joined #openstack-ansible | 08:21 | |
| *** sawblade6 has joined #openstack-ansible | 08:22 | |
| *** luksky has joined #openstack-ansible | 08:23 | |
| *** pdeters has left #openstack-ansible | 08:23 | |
| *** pdeters has joined #openstack-ansible | 08:24 | |
| evrardjp | noonedeadpunk: done | 08:27 |
| evrardjp | thanks for the patch, very interesting case | 08:28 |
| noonedeadpunk | evrardjp: thanks for reviewing) but now I'm a bit confised about divison inside OSA - whom of cores I may annoy to get it merged (except cloudnull)? | 08:29 |
| *** dante-as has joined #openstack-ansible | 08:30 | |
| *** sawblade6 has quit IRC | 08:30 | |
| dante-as | Hi guys, is any way to create an os_subnet with variables for some subnets that have gateways, some not? I didn't found the proper filter for gateway_ip or no_gateway_ip | 08:32 |
| noonedeadpunk | ok, probably I've found an answer to my question. | 08:34 |
| *** sawblade6 has joined #openstack-ansible | 08:35 | |
| evrardjp | dante-as: | default (omit) | 08:40 |
| evrardjp | | default(omit) * | 08:40 |
| *** sawblade6 has quit IRC | 08:46 | |
| dante-as | for gateway_ip option, I am still getting the gateway IP as next IP after the network if it has omit | 08:47 |
| dante-as | evrardjp: for gateway_ip option, I am still getting the gateway IP as next IP after the network if it has omit | 08:47 |
| *** sawblade6 has joined #openstack-ansible | 08:55 | |
| *** shyamb has joined #openstack-ansible | 08:56 | |
| evrardjp | I am not sure what you meant | 09:00 |
| evrardjp | please paste code somewhere | 09:00 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon stable/rocky: Implements custom theme distribution https://review.openstack.org/591336 | 09:02 |
| noonedeadpunk | evrardjp: returning back to horizon simplification - did you mean that we may still have dict with src path and state of dashboards, or it all should be discovered somehow? | 09:07 |
| noonedeadpunk | as it seems that I've began to complicate things... | 09:07 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Introduce OpenStack Healthchecks https://review.openstack.org/587544 | 09:08 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Introduce OpenStack Healthchecks https://review.openstack.org/587544 | 09:08 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Introduce OpenStack Healthchecks https://review.openstack.org/587544 | 09:11 |
| *** Bhujay has quit IRC | 09:13 | |
| noonedeadpunk | never mind) | 09:13 |
| *** Bhujay has joined #openstack-ansible | 09:15 | |
| *** sawblade6 has quit IRC | 09:17 | |
| evrardjp | noonedeadpunk: you found the solution? | 09:17 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon stable/queens: Implements custom theme distribution https://review.openstack.org/592635 | 09:19 |
| *** agowa338 has joined #openstack-ansible | 09:23 | |
| noonedeadpunk | evrardjp: yep, I just tried to automize everything too much, so I get huge generator, instead of specifying dict with path and state of available panels. So never mind, I'm more thinking aloud now | 09:24 |
| agowa338 | Hi, does anyone know if the os_recordset and os_zone module can query all existing records/zones? I want to delete all entries that are not specified in the playbook. | 09:24 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: fix intermittent failures when deploying base glance images https://review.openstack.org/592696 | 09:24 |
| *** shyambiradar has joined #openstack-ansible | 09:25 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Ensure vm_use_snapshot is defined https://review.openstack.org/592940 | 09:28 |
| *** shyamb has quit IRC | 09:29 | |
| dante-as | evrardjp: where I can paste the code? | 09:29 |
| *** udesale has quit IRC | 09:31 | |
| dante-as | evrardjp: please find it on git: dan-tabarca/playbooks/script.yml | 09:32 |
| dante-as | please let me know if you can see it | 09:33 |
| evrardjp | dante-as: full link? | 09:33 |
| evrardjp | is that github? | 09:33 |
| dante-as | https://github.com/dan-tabarca/playbooks/blob/master/script.yml | 09:34 |
| evrardjp | generally pastebins like paste.openstack.org | 09:34 |
| dante-as | yes, it is git-hub | 09:34 |
| evrardjp | dante-as: use default(omit) vs default(None) | 09:34 |
| evrardjp | so that is will be undefined | 09:34 |
| dante-as | I have tried both of them separatelly | 09:34 |
| evrardjp | I don't see the variable subnet definition | 09:34 |
| *** hakonw has joined #openstack-ansible | 09:34 | |
| evrardjp | so I can't say | 09:34 |
| dante-as | evrardjp: here is it: http://paste.openstack.org/show/728289/ | 09:35 |
| dante-as | please let me know if you can see it | 09:37 |
| *** shyambiradar has quit IRC | 09:43 | |
| *** shyamb has joined #openstack-ansible | 09:43 | |
| evrardjp | I was seeing already this from your github, I don't see your subnets var | 09:44 |
| evrardjp | dante-as | default() purpose is to define something when it's not defined | 09:45 |
| evrardjp | so dns_nameservers is improvable | 09:45 |
| evrardjp | gateway_ip is still wrong for what you're tying to achieve | 09:45 |
| evrardjp | you said you want not gateway ip to be defined in the call, right? | 09:46 |
| dante-as | some of the os_subnets have gateway, some not | 09:46 |
| evrardjp | odyssey4me: so... 1) change cinder role to make sure the start order is correct | 09:46 |
| evrardjp | (switching 3 and 4) | 09:46 |
| dante-as | and I would like to make a general rule for that part | 09:46 |
| evrardjp | and the second step would be to change integrated repo to switch L115 with L116 | 09:47 |
| evrardjp | dante-as: that's exactly what default(omit) is for | 09:47 |
| evrardjp | just don't define it in your items | 09:47 |
| evrardjp | but I don't see your "subnet" var so it's hard to tell | 09:47 |
| evrardjp | I can't help you | 09:48 |
| evrardjp | odyssey4me: would that assumption be correct? | 09:48 |
| odyssey4me | evrardjp: well, https://docs.openstack.org/cinder/latest/upgrade.html#after-maintenance-window says that the reloading order doesn't matter as long as the api is restarted last | 09:49 |
| evrardjp | I missed that | 09:49 |
| evrardjp | Diagonal reading : ) | 09:49 |
| dante-as | evrardjp: here is also the var file: one example with gateway and one without gateway | 09:50 |
| odyssey4me | evrardjp: of course we could instead take the approach that we just ignore the second round of reloads with the assumption that someone will do a minor upgrade at least once | 09:50 |
| dante-as | http://paste.openstack.org/show/728290/ | 09:50 |
| evrardjp | odyssey4me: that's not a good idea :) | 09:51 |
| odyssey4me | while I think that is an approach that can be taken, it's not ideal - because the RPC version will remain old until then. | 09:51 |
| evrardjp | I think | 09:51 |
| evrardjp | yeah | 09:51 |
| evrardjp | it's gonna open many problems too for gating of upgrades | 09:52 |
| odyssey4me | if there's a way that we can do all that fact finding inside the role - then it can all be done there | 09:52 |
| odyssey4me | one thing that has also come up with using the venv tag to figure out if everything's upgraded is that when using limits or a stale fact cache it all falls apart | 09:54 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Ensure vm_use_snapshot is defined https://review.openstack.org/592940 | 09:54 |
| evrardjp | dante-as: ok so, multiple points there. 1) you add too many spaces below your module arguments. 2) you're using item.gateway_ip, but in your vars, that's never defined. 3) You're defaulting to none for gateway_ip, instead of omit 4) with_items has an item which is your list of items. Flatten the list, it's better. | 09:54 |
| evrardjp | 5) remove existing networks you created and start again | 09:55 |
| evrardjp | odyssey4me: fact finding inside the role? | 09:55 |
| evrardjp | can't we just call the command? | 09:55 |
| evrardjp | a command* | 09:56 |
| odyssey4me | evrardjp: I'm talking about all this https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-cinder-install.yml#L48-L92 | 09:57 |
| odyssey4me | and this https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-cinder-install.yml#L118-L119 | 09:58 |
| odyssey4me | and finally https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-cinder-install.yml#L165-L192 | 09:58 |
| odyssey4me | what it comes down to is that the migrations must only be done once all software is at the same version | 09:58 |
| odyssey4me | it'd be really nice to figure out a way to move all that logic into the role | 09:59 |
| evrardjp | well if say that we want to really trigger parts of roles using vars, like the discussion with tempest (and that was mostly denied in the past) | 09:59 |
| evrardjp | we could definitely say one is restart_upgrades | 10:00 |
| evrardjp | and one is restart_install | 10:00 |
| evrardjp | then we can define things there | 10:00 |
| odyssey4me | yeah, implementing some sort of task router seems inevitable | 10:00 |
| evrardjp | the venv tag can be discovered in a different way | 10:00 |
| odyssey4me | anyway, I put up a session at the PTG to discuss all this | 10:00 |
| evrardjp | with setuptools | 10:00 |
| evrardjp | or something in cinder | 10:01 |
| evrardjp | I just think it's harder in roles | 10:01 |
| odyssey4me | yeah, I was kinda thinking that perhaps we should not use the venv tag, but instead should use something that fetches the actual service version, not the OSA version | 10:01 |
| odyssey4me | well, if we're ok with adding a new dep (etcd3, or something similar) then we could very easily do things simpler | 10:02 |
| odyssey4me | roles could just push information into the k:v store and read from it | 10:02 |
| evrardjp | ansible facts are a defacto k:v store | 10:02 |
| *** shyamb has quit IRC | 10:03 | |
| *** shyam89 has joined #openstack-ansible | 10:03 | |
| evrardjp | if we need to have a fact that's global instead of per host, you use the deploy node facts | 10:03 |
| odyssey4me | persistently? | 10:04 |
| evrardjp | so here we can combine: the node facts will discover versions of software | 10:04 |
| *** cshen has quit IRC | 10:04 | |
| evrardjp | so that can be queried live | 10:04 |
| evrardjp | and the deploy node itself can keep track of those | 10:04 |
| evrardjp | deploy node ansible_local facts | 10:04 |
| evrardjp | should I say | 10:05 |
| evrardjp | (not the facts cache) | 10:06 |
| *** ansmith has quit IRC | 10:16 | |
| odyssey4me | I think I know what you mean, although that would mean that we also end up doing slightly nasty things like delegating a fact file creation to the deploy host. | 10:18 |
| evrardjp | odyssey4me: 100% correct | 10:20 |
| evrardjp | we could build a module that does it though | 10:20 |
| evrardjp | action plugin | 10:20 |
| evrardjp | to make sure it's more reliable and follows our processes | 10:20 |
| evrardjp | the advantage of etcd is that it allows concurrent writes | 10:21 |
| evrardjp | where here we are exposing ourselves to race conditions | 10:21 |
| evrardjp | if not careful | 10:22 |
| evrardjp | But that's also why I think we should use plays for this, as this removes all the needs to care about this | 10:22 |
| evrardjp | just mark things as "in need of x" | 10:23 |
| evrardjp | (like we do) | 10:23 |
| evrardjp | and then in a play orchestrate the restart (like we do in os-cinder-install) | 10:23 |
| evrardjp | ship those plays inside the role and you have things still self-contained | 10:23 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: use include_tasks instead of include https://review.openstack.org/585298 | 10:24 |
| *** sawblade6 has joined #openstack-ansible | 10:25 | |
| *** shyam89 has quit IRC | 10:30 | |
| *** sawblade6 has quit IRC | 10:30 | |
| *** ansmith has joined #openstack-ansible | 10:31 | |
| *** udesale has joined #openstack-ansible | 10:31 | |
| odyssey4me | hwoarang jrosser evrardjp have any of you seen http://logs.openstack.org/02/592602/1/check/openstack-ansible-functional-ubuntu-xenial/0d64834/logs/host/nova-compute.service.journal.log.txt.gz | 10:32 |
| odyssey4me | seeing it on os_nova, os_tempest, os_octavia | 10:34 |
| *** nicolasbock has joined #openstack-ansible | 10:35 | |
| evrardjp | odyssey4me: nope I haven't seen that, and I just crushed my node. | 10:36 |
| evrardjp | well | 10:36 |
| evrardjp | I have seen failures like this in the past but they were not important | 10:36 |
| evrardjp | because it was until service restart | 10:36 |
| evrardjp | but that's not like it, as you seem spammed | 10:36 |
| evrardjp | I just deployed nova and haven't seen that | 10:37 |
| evrardjp | is that in tempest run? | 10:37 |
| odyssey4me | that particular patch is os_nova | 10:38 |
| evrardjp | that could also be another reason for timeouts | 10:38 |
| evrardjp | could you compare timestamps with tempest run? | 10:38 |
| odyssey4me | it never gets to tempest, because the wait for nova compute to start fails | 10:40 |
| evrardjp | ok | 10:41 |
| evrardjp | yeah never got that | 10:41 |
| evrardjp | or can't remember | 10:41 |
| *** dxiri has quit IRC | 10:43 | |
| mbuil | has anybody used Horizon from a osa deployment? I am having problems and I think they are related to SSL and port 443 :( | 10:45 |
| odyssey4me | kaiokmo: are you in a position to update https://review.openstack.org/591101 soon as requested? | 10:46 |
| evrardjp | mbuil: many times, but I haven't tried myself recently | 10:47 |
| evrardjp | odyssey4me: maybe kaiokmo is not awaken up :) | 10:48 |
| evrardjp | yet | 10:48 |
| evrardjp | odyssey4me: is that the issue you see? | 10:48 |
| odyssey4me | evrardjp: I think it may be a contributing factor, but is not necessarily the issue. | 10:49 |
| evrardjp | ok | 10:49 |
| evrardjp | well race conditions are hard | 10:49 |
| odyssey4me | yeah, I was thinking of updating the patch, given it's been idle for some time | 10:49 |
| evrardjp | I can merge things if you prefer, as this is better | 10:49 |
| evrardjp | than not having the patch | 10:49 |
| evrardjp | it's not gonna be trivial | 10:49 |
| odyssey4me | ok, so you think we let that one through and do a better one later? | 10:50 |
| evrardjp | haha you know my opinion about "doing a better one later" :p | 10:50 |
| evrardjp | else I wouldn't have put a -1 | 10:50 |
| odyssey4me | yeah, I've not actually looked at how the units are done so I have no idea how it all works | 10:51 |
| evrardjp | well I don't know that part | 10:51 |
| *** shyam89 has joined #openstack-ansible | 10:51 | |
| evrardjp | but I guess on ansible side you may now want to do things conditionally | 10:51 |
| odyssey4me | I'll look at it - just doing a patch for os_trove, given that the tempest plugins have changed structure | 10:51 |
| evrardjp | because nova-api will not need that override | 10:51 |
| evrardjp | so now the calls of the role of systemd is different | 10:52 |
| evrardjp | systemd_service | 10:52 |
| evrardjp | it's not a big deal it's making sure we have two different calls | 10:52 |
| evrardjp | but that is like opening a can of worms | 10:52 |
| evrardjp | so maybe we should remove the systemd_after_targets | 10:53 |
| evrardjp | I am all open for a dicussion on how we'll do things | 10:53 |
| mbuil | evrardjp: where did you have the browser? when connecting to port 80, it automatically changes to https and port 443, however, the server never replies because it does not receive packets. haproxy is configured to forward tcp port 80 to horizon container but not 443. Myabe I am missing something :/ | 10:53 |
| evrardjp | odyssey4me: you might want to know that changing the order in role + play has not triggered a race condition in my system at the first run | 10:54 |
| evrardjp | odyssey4me: doing it a second time | 10:54 |
| odyssey4me | evrardjp: great - sounds good | 10:54 |
| dante-as | evrardjp: I have tried your way, but still not working. I have paid attention to what was defined, and the GW IP is still created for the subnets that do not have gateway_ip inserted in the variable file. | 10:54 |
| odyssey4me | maybe this time the order will stay correct :/ | 10:55 |
| evrardjp | dante-as: please paste your code. Full | 10:55 |
| dante-as | evrardjp: here is the code: http://paste.openstack.org/show/728293/ and here is the vars: http://paste.openstack.org/show/728294/ | 10:55 |
| evrardjp | mbuil: I am connecting to 443 | 10:55 |
| evrardjp | mbuil: haproxy seems misconfigured | 10:56 |
| evrardjp | mbuil: there is a redirect from 80 to 443, but 443 should work | 10:56 |
| evrardjp | check your config | 10:56 |
| dante-as | evrardjp: test1 gets GW, test2 get also GW even if is not mentioned | 10:56 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove master: Update trove tempest plugin path https://review.openstack.org/592994 | 10:56 |
| evrardjp | dante-as: did you remove your test2 network beforehand? | 10:57 |
| dante-as | yes | 10:57 |
| dante-as | these are new networks created | 10:57 |
| evrardjp | I don't know I have no issue on my side | 10:58 |
| dante-as | have you tested it somewhere? | 10:58 |
| evrardjp | look at this for inspiration: https://review.openstack.org/#/c/587544/ | 10:58 |
| dante-as | ok, thanks | 10:59 |
| jrosser | dante-as: does it need no_gateway_ip defining as ‘no’ selectively | 11:02 |
| dante-as | jrosser: I was thinking aboutthis method as well, but how should be implemented? | 11:03 |
| jrosser | Use ‘is defined’ to get a bool for the presence of your gateway ip | 11:03 |
| odyssey4me | evrardjp: did you happen to see https://review.openstack.org/574045 ? I've not looked at it carefully, but it might save some time? | 11:06 |
| *** shyam89 has quit IRC | 11:07 | |
| *** shyam89 has joined #openstack-ansible | 11:07 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_watcher stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592826 | 11:10 |
| evrardjp | odyssey4me: oh that might simplify your nova story : ) | 11:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_congress stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592794 | 11:12 |
| evrardjp | I have to check if systemd_services can be a list | 11:12 |
| evrardjp | if that's the case, that's nice. | 11:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592804 | 11:13 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Ensure nova-compute service start after libvirtd https://review.openstack.org/593001 | 11:13 |
| dante-as | jrosser: I have tried now no_gateway_ip: "{{ 'yes' if item.gateway_ip is defined else default('yes') }}" for the same vars file | 11:13 |
| mbuil | evrardjp: could you show me your haproxy.cfg please? I don't see any way how 443 packets can reach horizon container. I am trying to access Horizon from outside the controller | 11:14 |
| evrardjp | we can probably simplify it | 11:14 |
| dante-as | jrosser: now I am not getting any GW IPs | 11:14 |
| evrardjp | mbuil: let's rather do the other way around | 11:14 |
| dante-as | jrosser: any suggestion? | 11:14 |
| evrardjp | show me your haproxy | 11:14 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Ensure nova-compute service start after libvirtd https://review.openstack.org/593001 | 11:14 |
| dante-as | jrosser: sorry, copy paste did not work properly. this is what I have tried: no_gateway_ip: "{{ 'yes' if item.gateway_ip is defined else 'yes' }}" | 11:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592815 | 11:15 |
| odyssey4me | dante-as: um, what - that condition should always result in yes? | 11:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592793 | 11:15 |
| dante-as | odyssey4me: no. if is a gateway_ip => take the gateway_ip from vars. if not, default yes | 11:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592824 | 11:17 |
| mbuil | evrardjp: https://hastebin.com/isitamiyah.pl | 11:17 |
| odyssey4me | dante-as: I think you might mean to do this: "{{ (item.gateway_ip is defined) | ternary(item.gateway_ip, 'yes') }}" ? | 11:17 |
| odyssey4me | that will populate the var with the gateway ip if it's defined, but put 'yes' into the var instead if it is not | 11:17 |
| *** sawblade6 has joined #openstack-ansible | 11:18 | |
| dante-as | odyssey4me: so what should I write exactly in the vars file for that specific row? | 11:19 |
| evrardjp | mbuil: you mush have an override | 11:19 |
| evrardjp | somwhere | 11:19 |
| dante-as | odyssey4me: gateway_ip: 10.10.11.225, yes ? | 11:19 |
| odyssey4me | dante-as: another way to do it is simply: "{{ item.gateway_ip | default('yes') }}" | 11:19 |
| odyssey4me | dante-as: I have not looked at any of your data, only what you wrote as a conditional which makes no sense | 11:20 |
| mbuil | evrardjp: you have an entry for horizon with port 443? | 11:20 |
| odyssey4me | dante-as: looking at the pastebins now | 11:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592789 | 11:21 |
| odyssey4me | dante-as: dns_nameservers: "{{ item.dns if item.dns is defined else omit }}" is easier to understand if you just do: the same as gateway_ip above... "{{ item.dns | default(omit) }}" | 11:21 |
| dante-as | odyssey4me: for "{{ item.gateway_ip | default('yes') }}", I get an error: "The value '10.10.11.225' is not a valid boolean. Valid booleans include: 0, 'on', 'f', 'false', 1, 'no', 'n', '1', '0', 't', 'y', 'off', 'yes', 'true'" | 11:22 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592811 | 11:22 |
| *** sawblade6 has quit IRC | 11:22 | |
| odyssey4me | dante-as: ok, looking at http://paste.openstack.org/show/728293/ and http://paste.openstack.org/show/728294/, other than my comment above about using the default(omit) I don't see a problem - what's the problem? | 11:23 |
| dante-as | odyssey4me: I have some os_subnets that have GWs, some of them do not have GW | 11:23 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592791 | 11:24 |
| dante-as | odyssey4me: I need to separe them. Now we are trying with the no_gateway_ips since this one is a little bit stronger than gateway_ips | 11:24 |
| *** shardy has quit IRC | 11:24 | |
| *** shardy has joined #openstack-ansible | 11:24 | |
| evrardjp | mbuil: http://paste.ubuntu.com/p/MdJ9Wt3hCh/ | 11:24 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592802 | 11:24 |
| odyssey4me | dante-as: gateway_ip: "{{ item.gateway_ip | default(omit) }}" | 11:25 |
| evrardjp | as you insist seeing it : ) | 11:25 |
| odyssey4me | that should work fine. | 11:25 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592827 | 11:25 |
| dante-as | odyssey4me: I have tried it and for the ones that do not have gws, I am still getting gws | 11:25 |
| dante-as | I am getting always CIDR+1 | 11:25 |
| odyssey4me | dante-as: ok, I think that's neutron doing it - or perhaps the os_subnet module | 11:25 |
| evrardjp | dante-as: show your openstack subnet definition with openstack cli on a pastebin | 11:25 |
| evrardjp | before | 11:26 |
| evrardjp | and after a run | 11:26 |
| *** udesale has quit IRC | 11:26 | |
| odyssey4me | IIRC neutron assumes CIDR+1 if you don't give it a gateway | 11:26 |
| evrardjp | before it should be empty | 11:26 |
| odyssey4me | if you use the cli to create a subnet, you'll likely see the same result | 11:26 |
| evrardjp | that's true | 11:26 |
| evrardjp | yeah the option is no_gateway_ip | 11:27 |
| evrardjp | in the module | 11:27 |
| evrardjp | https://docs.ansible.com/ansible/2.5/modules/os_subnet_module.html#os-subnet-module | 11:27 |
| evrardjp | so add an extra line into each of your subnets | 11:27 |
| evrardjp | if need be | 11:27 |
| evrardjp | and pass that to the module | 11:27 |
| evrardjp | problem solved | 11:27 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592822 | 11:27 |
| odyssey4me | no_gateway_ip: "{{ not (item.gateway is defined) }}" | 11:28 |
| evrardjp | or that | 11:28 |
| odyssey4me | well, that should be no_gateway_ip: "{{ not (item.gateway_ip is defined) }}" | 11:28 |
| evrardjp | gateway_ip | 11:28 |
| dante-as | evrardjp: it's always empty before and afterwards its populated | 11:28 |
| evrardjp | dante-as: please read what we just said | 11:28 |
| evrardjp | and read the documentation | 11:28 |
| mbuil | evrardjp: thanks. We must be missing some config in XCI in order to get that haproxy.cfg. Time to investigate | 11:29 |
| evrardjp | mbuil: not missing -- altering | 11:29 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592792 | 11:29 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592798 | 11:30 |
| odyssey4me | dante-as: in case it wasn't obvious, the final thing should look like http://paste.openstack.org/show/728297/ | 11:30 |
| dante-as | odyssey4me: yes, thanks. now I am trying it | 11:31 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592790 | 11:31 |
| odyssey4me | evrardjp: so the two approaches to ensuring libvirt starts first are up https://review.openstack.org/#/q/topic:bug/1780011+(status:open+OR+status:merged) | 11:32 |
| odyssey4me | both ways will do - one just has a little more duplicated than the other | 11:33 |
| evrardjp | oh you filthy :p | 11:33 |
| evrardjp | I was sure this was gonna show up :) | 11:33 |
| evrardjp | I think jimmy's way is more flexible :p | 11:34 |
| evrardjp | in the meantime, that's good enough for me | 11:34 |
| *** cshen has joined #openstack-ansible | 11:34 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592808 | 11:35 |
| odyssey4me | evrardjp: hmm, so systemd services is a list it seems: https://github.com/openstack/ansible-role-systemd_service/blob/master/defaults/main.yml#L87 | 11:36 |
| evrardjp | odyssey4me: that's great. | 11:36 |
| odyssey4me | so yeah, perhaps doing some funky in-line jinja script would be neater, and provide a speed boost as well | 11:36 |
| evrardjp | Let's do that in another patch | 11:36 |
| evrardjp | yeah | 11:36 |
| odyssey4me | yeah, I'll have a go at it now | 11:36 |
| evrardjp | speed boost + flexibility | 11:36 |
| evrardjp | did you know that jmespath can build dicts and lists? :p | 11:37 |
| odyssey4me | anyone seen jmccrory around lately? | 11:37 |
| evrardjp | (please not try to build lists of dicts in the same thing though) | 11:37 |
| odyssey4me | evrardjp: yep, I was just thinking of doing that instead of the jinja script | 11:37 |
| evrardjp | odyssey4me: not recently | 11:37 |
| evrardjp | odyssey4me: so what you can do is apply a map filter on an existing list of services, then prepare the thing the way you want in a query | 11:38 |
| dante-as | odyssey4me: thank you, it worked | 11:38 |
| odyssey4me | dante-as: no worries, happy to have helped | 11:39 |
| evrardjp | odyssey4me: I am worried he might not come to the PTG | 11:39 |
| odyssey4me | I suspect he will not be there. He's moved employers. | 11:39 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592813 | 11:41 |
| odyssey4me | Well, I think he might have - not sure. I might also be mixing him up with someone else. | 11:43 |
| mnaser | Busy AM today | 11:44 |
| odyssey4me | o/ mnaser | 11:45 |
| mnaser | I know andymccr recently moved to redhat but I dunno if his role is still openstackish | 11:45 |
| mnaser | Hi all | 11:45 |
| *** jamesdenton has joined #openstack-ansible | 11:47 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove master: Update trove tempest plugin path https://review.openstack.org/592994 | 11:47 |
| *** shyam89 has quit IRC | 11:47 | |
| odyssey4me | mnaser: so I see you managed to get cinder merging some patches - now can you remember everything that was needed so that we can do the same for rocky? ;) | 11:48 |
| mnaser | I did most of the backports. It was mostly checksum and the stack of 4 in cinder | 11:49 |
| mnaser | If checksum merged in rocky, it should be good to be rechecked | 11:49 |
| * mnaser hasn’t gotten to office yet | 11:49 | |
| openstackgerrit | Manuel Buil proposed openstack/openstack-ansible-os_neutron master: [WIP] - Provide support for SFC in ODL Fluorine https://review.openstack.org/593031 | 11:51 |
| odyssey4me | looks like that did | 11:51 |
| odyssey4me | rechecking the stack now | 11:52 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592820 | 11:53 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: Updated from OpenStack Ansible Tests https://review.openstack.org/592049 | 11:54 |
| *** shyam89 has joined #openstack-ansible | 11:54 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift stable/rocky: Updated from OpenStack Ansible Tests https://review.openstack.org/592823 | 11:55 |
| noonedeadpunk | mnaser: hi, can you kindly check patch https://review.openstack.org/#/c/583943/ ? | 12:02 |
| mnaser | Thanks odyssey4me | 12:03 |
| *** sawblade6 has joined #openstack-ansible | 12:03 | |
| mnaser | noonedeadpunk: checking | 12:03 |
| mnaser | noonedeadpunk: I’m on mobile so can’t easily commit on gerrit but if someone was working around it in OS | 12:05 |
| mnaser | In some other way this might cause a change in behaviour | 12:05 |
| mnaser | Could you remove it from defaults and instead do become: “{{ ceph_mon_user|default(ansible_user) }}” | 12:06 |
| mnaser | Or something like that so it doesn’t change default behaviour | 12:06 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Fix order for restarts https://review.openstack.org/593032 | 12:07 |
| *** sawblade6 has quit IRC | 12:08 | |
| noonedeadpunk | mnaser: than it won't get to documentation.. probably I should place “{{ default(ansible_user) }}” into defaults/main.yml ? | 12:08 |
| noonedeadpunk | * “{{ ansible_user }}” | 12:09 |
| mnaser | noonedeadpunk: you can add it to the defaults but commented out | 12:09 |
| *** shyamb has joined #openstack-ansible | 12:09 | |
| *** pdeters has quit IRC | 12:10 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_cinder master: Always start cinder API last https://review.openstack.org/593033 | 12:11 |
| evrardjp | odyssey4me: so tried 3 times, never got an issue locally. | 12:11 |
| odyssey4me | mnaser: do you mean? ceph_mon_user: "{{ ansible_user }}" | 12:12 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible master: Fix order for restarts https://review.openstack.org/593032 | 12:12 |
| *** shyam89 has quit IRC | 12:12 | |
| evrardjp | odyssey4me: weirdly it was scheduler to restart, so I doubt it's enough | 12:12 |
| evrardjp | I just don't think I was lucky enough to hit the race condition | 12:13 |
| noonedeadpunk | mnaser: yep, as odyssey4me I was also offering to put ceph_mon_user: "{{ ansible_user }}" | 12:13 |
| *** spatel has joined #openstack-ansible | 12:14 | |
| odyssey4me | evrardjp: the integrated patch refers to the https://docs.openstack.org/cinder/latest/upgrade.html#during-maintenance-window which is the part that's handled in the role | 12:14 |
| odyssey4me | the playbook handles the part that's in https://docs.openstack.org/cinder/latest/upgrade.html#after-maintenance-window | 12:14 |
| evrardjp | maybe in one hour we'll have a rc1 freeze in rocky and that would be great | 12:14 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate master: Updated from OpenStack Ansible Tests https://review.openstack.org/592036 | 12:14 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_cinder master: Always start cinder API last https://review.openstack.org/593033 | 12:17 |
| evrardjp | odyssey4me: that's better ^ | 12:17 |
| *** gkapoor1 has joined #openstack-ansible | 12:17 | |
| *** erolg has joined #openstack-ansible | 12:17 | |
| *** erolg_ has joined #openstack-ansible | 12:17 | |
| *** spatel has quit IRC | 12:18 | |
| *** erolg_ has quit IRC | 12:19 | |
| *** schwicht has joined #openstack-ansible | 12:20 | |
| mnaser | I agree with odyssey4me, noonedeadpunk | 12:21 |
| *** schwicht has quit IRC | 12:25 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.openstack.org/592047 | 12:26 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-ceph_client master: Added possibility to specify ssh username and port for ceph mons. https://review.openstack.org/583943 | 12:26 |
| *** schwicht has joined #openstack-ansible | 12:31 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests https://review.openstack.org/592044 | 12:33 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: tests-ansible-env-prep: Fix plugin, tests and ops testing on Vagrant https://review.openstack.org/592527 | 12:36 |
| hwoarang | logan-: cloudnull: hello. would you mind giving https://review.openstack.org/#/c/591672/ another look please? it does not seem to cause any regressions and also fixes our opnfv tests where the deployment host is not part of the openstack nodes | 12:39 |
| noonedeadpunk | mnaser: patched. | 12:44 |
| *** schwicht has quit IRC | 12:44 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Updated from OpenStack Ansible Tests https://review.openstack.org/592033 | 12:46 |
| noonedeadpunk | and generic ansible question: how can I exctract 'files' key from such structure: registered_item: { results: [ { files: [ {} ] } ] }? I know that's it's possible through registering extra fact, and double iteration. But probably I can realize this through loop control or smth like that? | 12:48 |
| *** jamesdenton has quit IRC | 12:48 | |
| *** aicilamd has joined #openstack-ansible | 12:49 | |
| *** jamesdenton has joined #openstack-ansible | 12:49 | |
| kaiokmo | odyssey4me: hey sorry, just got on. thanks for that patch | 12:49 |
| kaiokmo | I was kinda busy this week. thank you for updating it | 12:50 |
| mnaser | Wonderful. I’ll have a look soon | 12:50 |
| noonedeadpunk | to more accurate - it's full structure http://paste.openstack.org/show/728304/ and i need files.path | 12:50 |
| odyssey4me | noonedeadpunk: you need a list of just the paths? | 12:50 |
| noonedeadpunk | odyssey4me: yep | 12:50 |
| *** aicilamd has quit IRC | 12:51 | |
| odyssey4me | noonedeadpunk: could you tell me why it's in a results set? the find module supports a list of paths, so you don't need to with_items | 12:51 |
| *** aicilamd has joined #openstack-ansible | 12:51 | |
| *** jamesdenton has quit IRC | 12:52 | |
| noonedeadpunk | odyssey4me: I'm using when, not to check every path I have. So than I need an extra variable with kinda generator in it... Probably I'll place a patch with workflow -1 to show the whole idea. | 12:53 |
| odyssey4me | noonedeadpunk: yep, while it's possible to do what you're asking - it may be possible to siplify it so that you do not need to | 12:54 |
| kukacz_ | hello, I'd like to evaluate OSA by running it on top of another openstack. I wonder if there is some template (ansible, heat, terraform, ...) to deploy and preconfigure a set of hosts (ie. instances) for multinode deployment ...? | 12:55 |
| noonedeadpunk | odyssey4me: so path is find should be set as a regular list? | 12:56 |
| *** aicilamd has quit IRC | 12:56 | |
| noonedeadpunk | *paths in find | 12:56 |
| odyssey4me | noonedeadpunk: yep | 12:56 |
| odyssey4me | it can be | 12:56 |
| noonedeadpunk | okey, got it. probably it'll resolve my problem. | 12:57 |
| *** jamesdenton has joined #openstack-ansible | 12:57 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: run_test_common.sh: Ensure python3 is installed https://review.openstack.org/593045 | 12:58 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: run_test_common.sh: Ensure python3 is installed https://review.openstack.org/593045 | 12:59 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Freeze all SHAs for RC1 https://review.openstack.org/590503 | 13:01 |
| odyssey4me | w00t! | 13:02 |
| *** shyamb has quit IRC | 13:02 | |
| *** shyam89 has joined #openstack-ansible | 13:03 | |
| hwoarang | finally! | 13:04 |
| hwoarang | no we need to bump systemd_service ;p | 13:04 |
| hwoarang | and go through hell again | 13:04 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove master: Update trove tempest plugin path https://review.openstack.org/592994 | 13:04 |
| hwoarang | *now | 13:04 |
| odyssey4me | yeah, but at least the integrated repo for rocky is now only testing small sets of changes at once | 13:06 |
| evrardjp | guess what today is release day anyway, so let's apply procedures! | 13:07 |
| *** schwicht has joined #openstack-ansible | 13:07 | |
| hwoarang | the switch to python3 for tox is making my friday afternoon miserable | 13:11 |
| evrardjp | mnaser: shouldn't this https://review.openstack.org/#/c/592688/2 be below every other patch? | 13:12 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: run_test_common.sh: Ensure python3 is installed https://review.openstack.org/593045 | 13:13 |
| mnaser | evrardjp: no i think its okay because it only is needed to fix the backend_defaults patch | 13:14 |
| mnaser | also, evrardjp, you voted on https://review.openstack.org/#/q/I2242725682056491106ab8012ad1fa9d0e7f9ecd | 13:14 |
| odyssey4me | noonedeadpunk: here's an example of using find with multiple paths, then using jmespath to get the right info out easily: https://github.com/openstack/openstack-ansible-ops/blob/master/multi-node-aio/playbooks/save-vms.yml#L91-L104 | 13:14 |
| mnaser | if you can do the rest | 13:14 |
| *** spatel has joined #openstack-ansible | 13:16 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible master: Add qdrouterd role for oslo.messaging backend deployment https://review.openstack.org/570261 | 13:17 |
| spatel | odyssey4me: do you think this is a bug https://bugs.launchpad.net/openstack-ansible/+bug/1787517 | 13:18 |
| openstack | Launchpad bug 1787517 in openstack-ansible "After primary node shutdown getting MySQL connection error " [Undecided,New] | 13:18 |
| odyssey4me | spatel: no idea, busy with something else right now | 13:20 |
| *** kevinshort has joined #openstack-ansible | 13:21 | |
| mnaser | evrardjp: do you feel like changing that same patch so we can backport it or think that's a bit too trivial, i'll leave it up to you -- https://review.openstack.org/#/c/593033/ | 13:21 |
| evrardjp | mnaser: slowly slowly : p | 13:22 |
| mnaser | haha sorry | 13:22 |
| * mnaser is chugging through coffee | 13:22 | |
| *** spatel has quit IRC | 13:22 | |
| *** mmalchuk_ has quit IRC | 13:23 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove the last remnants of get-pip.py https://review.openstack.org/591392 | 13:23 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-os_cinder master: Always start cinder API last https://review.openstack.org/593033 | 13:28 |
| evrardjp | mnaser: done https://review.openstack.org/#/c/593033/ | 13:28 |
| evrardjp | odyssey4me: ^ could you please revote? | 13:28 |
| odyssey4me | sure, done - makes sense | 13:29 |
| evrardjp | the time we merge the tempest backport and use of master, tempest will have released something: https://pypi.org/project/Tempest/#history | 13:30 |
| evrardjp | doing and redoing it's always work | 13:30 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: run_test_common.sh: Ensure python3 is installed https://review.openstack.org/593045 | 13:32 |
| *** electrofelix has quit IRC | 13:36 | |
| *** ppetit has joined #openstack-ansible | 13:41 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/rocky: Rollback tempest master for Rocky https://review.openstack.org/593051 | 13:41 |
| *** kevinshort has quit IRC | 13:43 | |
| *** hamzaachi has quit IRC | 13:44 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592611 | 13:44 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_trove master: Updated from OpenStack Ansible Tests https://review.openstack.org/592051 | 13:44 |
| *** kevinshort has joined #openstack-ansible | 13:45 | |
| *** weshay|ruck has joined #openstack-ansible | 13:46 | |
| mnaser | evrardjp: +A'd the patch | 13:52 |
| *** dxiri has joined #openstack-ansible | 13:56 | |
| *** shyam89 has quit IRC | 13:57 | |
| *** Bhujay has quit IRC | 13:59 | |
| *** aicilamd has joined #openstack-ansible | 14:00 | |
| mnaser | alright so | 14:00 |
| *** aicilamd has quit IRC | 14:01 | |
| evrardjp | I am pushing up a patch for bumping the openstack SHAs and our roles SHA to latest of stable/rocky | 14:01 |
| *** aicilamd has joined #openstack-ansible | 14:01 | |
| evrardjp | I should probably do two separate patches though | 14:01 |
| mnaser | evrardjp: cool | 14:02 |
| mnaser | http://logs.openstack.org/01/593001/2/check/openstack-ansible-functional-ubuntu-xenial/ac815c5/logs/host/nova-compute.service.journal.log.txt.gz | 14:02 |
| mnaser | anyone has any ideas about this? | 14:02 |
| odyssey4me | evrardjp: given the struggle to merge things - perhaps not | 14:02 |
| mnaser | looks like nova-compute never started | 14:02 |
| mnaser | it just spins forever | 14:02 |
| odyssey4me | mnaser: yeah, I've seen that in places too | 14:02 |
| mnaser | ill try to do some investigation if anyone hasnt looked into it | 14:02 |
| odyssey4me | thought one of these two options would be helpful, but it seems not: https://review.openstack.org/#/q/topic:bug/1780011+(status:open+OR+status:merged) | 14:02 |
| mnaser | also, i think i am going to propose removal of all nspawn jobs, for now they have not been stable (at all), ubuntu and opensuse has been broken forever | 14:02 |
| mnaser | at least it'll give us 3x 2 hours of workers back to infra | 14:03 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/rocky: Bump roles to latest stable/rocky https://review.openstack.org/593058 | 14:03 |
| mnaser | and they have been failing forever | 14:03 |
| evrardjp | odyssey4me: taht's true | 14:03 |
| evrardjp | let me add the rest on the same patch then | 14:03 |
| odyssey4me | mnaser: perhaps move them to experimental instead? | 14:03 |
| mnaser | odyssey4me: im thinking that when we want to add support again, the commit that adds it will enable the jobs again | 14:04 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts stable/rocky: Use variable with full URL for RDO repositories https://review.openstack.org/592683 | 14:04 |
| odyssey4me | mnaser: fair enough | 14:04 |
| mnaser | so instead of our check experimental running mostly broken stuff, you can just push a commit with the zuul file edited | 14:04 |
| *** aicilamd has quit IRC | 14:05 | |
| odyssey4me | oh yes, that reminds me - does anyone know where there's a devstack zuul job defined? | 14:06 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: run_test_common.sh: Ensure python3 is installed https://review.openstack.org/593045 | 14:06 |
| *** sawblade6 has joined #openstack-ansible | 14:06 | |
| *** aludwar has quit IRC | 14:07 | |
| odyssey4me | got it https://github.com/openstack-dev/devstack/blob/master/.zuul.yaml#L82 | 14:07 |
| *** aludwar has joined #openstack-ansible | 14:09 | |
| *** cshen has quit IRC | 14:14 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Drop non-voting jobs and increase timeout by 20 minutes https://review.openstack.org/593060 | 14:15 |
| mnaser | odyssey4me, evrardjp and other cores around ^ | 14:16 |
| odyssey4me | mnaser: IIRC we're already at the max timeout | 14:16 |
| mnaser | nope | 14:16 |
| mnaser | max timeout is 3 hours afaik | 14:16 |
| mnaser | we are at 2 | 14:16 |
| mnaser | ceph job is at 3 hours | 14:16 |
| odyssey4me | oh, it's the periodics that are at max: timeout: 10800 | 14:17 |
| mnaser | i only left centos as non voting there because i'm actually making a lot of progress and we're really really close, having at least 1 non voting job would be nice because integrated repo is failing only | 14:17 |
| openstackgerrit | Merged openstack/ansible-role-systemd_mount master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592478 | 14:17 |
| mnaser | also, i'd like to consider disabling `openstack-ansible-functional-opensuse-150` for now, because as long as galera is failing, it pretty much always fails and takes 28 minutes to do so | 14:19 |
| mnaser | ill push a patch and we can discuss there | 14:19 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible stable/rocky: Bump SHAs for rocky release https://review.openstack.org/593058 | 14:19 |
| *** schwicht has quit IRC | 14:20 | |
| odyssey4me | evrardjp: I think we need https://review.openstack.org/#/c/592688/2 merged into rocky's os_cinder as a matter of urgency, and that should be included in the sha bump? | 14:21 |
| *** sawblade6 has quit IRC | 14:21 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Drop opensuse-150 jobs https://review.openstack.org/593061 | 14:21 |
| mnaser | evrardjp, odyssey4me: i'd even include https://review.openstack.org/#/c/592689/2 in the list | 14:21 |
| odyssey4me | and it all deps on https://review.openstack.org/#/c/592687/2 | 14:22 |
| evrardjp | odyssey4me: agreed | 14:22 |
| evrardjp | mnaser: agreed too | 14:23 |
| openstackgerrit | Merged openstack/ansible-role-systemd_networkd master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592479 | 14:23 |
| mnaser | darn | 14:23 |
| mnaser | File './x86_64/openSUSE-release-42.3-2.1.x86_64.rpm' not found on medium 'http://mirror.iad.rax.openstack.org/opensuse/update/leap/42.3/oss/' | 14:23 |
| mnaser | all opensuse-42 jobs are broken i think because of tha | 14:23 |
| evrardjp | long story short, give a little nudge to cinder role sha | 14:23 |
| odyssey4me | mnaser: yep, seems like a bad mirror across providers - I saw it on vexxhost too | 14:24 |
| odyssey4me | or perhaps an incomplete mirror | 14:24 |
| mnaser | odyssey4me: the mirrors operate in an atomic release way | 14:24 |
| mnaser | so they're either all broken or all working | 14:24 |
| mnaser | :P | 14:24 |
| openstackgerrit | Merged openstack/ansible-role-systemd_service master: Use the TESTING_BRANCH env var for constraints https://review.openstack.org/592480 | 14:25 |
| odyssey4me | yep | 14:25 |
| evrardjp | mnaser: what's the source of the mirroring in infra? | 14:25 |
| mnaser | evrardjp: not sure, you can find it in openstack/system-config | 14:26 |
| mnaser | i think it was a bad sync to get us in this state | 14:26 |
| evrardjp | well there is only one way to find out | 14:26 |
| mnaser | https://github.com/openstack-infra/system-config/blob/67b77279f5479d0d5c311a3323e0dffe193c1c9c/modules/openstack_project/files/mirror/opensuse-mirror-update.sh | 14:26 |
| mnaser | idk why we're syncing from .de | 14:27 |
| evrardjp | so they did what's on the repo they are syncing from | 14:27 |
| evrardjp | wait | 14:27 |
| mnaser | https://github.com/openstack-infra/system-config/blob/67b77279f5479d0d5c311a3323e0dffe193c1c9c/modules/openstack_project/files/mirror/opensuse-mirror-update.sh#L94 | 14:27 |
| evrardjp | mnaser: isn't it from http://mirrors.rit.edu/opensuse/update/leap/42.3/oss/x86_64/ ? | 14:27 |
| mnaser | wow | 14:28 |
| mnaser | they're incomplete too | 14:28 |
| mnaser | so their bad sync broke our bad sync | 14:28 |
| evrardjp | I am not sure it's that | 14:29 |
| evrardjp | I am thinking it might be a problem elsewhere | 14:29 |
| evrardjp | oh yeah my bad | 14:29 |
| evrardjp | it was a glitch on my chrome so I didn't fine the release file | 14:29 |
| evrardjp | oooookay | 14:29 |
| evrardjp | so yeah | 14:29 |
| *** shardy has quit IRC | 14:29 | |
| evrardjp | the mirrors on suse are okay, some mirrors are okay, rit mirrors are bad, and we are syncing to it? | 14:30 |
| evrardjp | is infra aware of that? | 14:30 |
| *** spatel has joined #openstack-ansible | 14:32 | |
| spatel | odyssey4me: sorry my client was disconnected and i didn't realize | 14:32 |
| mnaser | yes, there is a discussion in #openstack-infra | 14:32 |
| *** aicilamd has joined #openstack-ansible | 14:32 | |
| spatel | Do you think this is real BUG or there is a fix for this? https://bugs.launchpad.net/openstack-ansible/+bug/1787517 | 14:32 |
| openstack | Launchpad bug 1787517 in openstack-ansible "After primary node shutdown getting MySQL connection error " [Undecided,New] | 14:32 |
| spatel | I am seeing random error like this everywhere DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(104, 'Connection reset by peer'))") [SQL: u'SELECT 1'] | 14:32 |
| *** aicilamd has quit IRC | 14:33 | |
| cloudnull | mornings | 14:33 |
| spatel | cloudnull: morning | 14:33 |
| *** aicilamd has joined #openstack-ansible | 14:33 | |
| openstackgerrit | Merged openstack/openstack-ansible stable/queens: Use ZUUL_SRC_PATH to pass the path to Zuul git sources https://review.openstack.org/591159 | 14:33 |
| *** Leo_m has quit IRC | 14:36 | |
| mnaser | gaha | 14:37 |
| mnaser | all opensuse jobs will break now | 14:37 |
| mnaser | https://mirrors.edge.kernel.org/opensuse/update/leap/42.3/oss/x86_64/ seems updated | 14:37 |
| mnaser | i will propose that we mirrored from kernel.org | 14:37 |
| cloudnull | kukacz_ did you get an answer to your question? | 14:37 |
| *** Leo_m has joined #openstack-ansible | 14:37 | |
| kukacz_ | cloudnull: not yet | 14:37 |
| *** aicilamd has quit IRC | 14:38 | |
| *** Leo_m has quit IRC | 14:42 | |
| cloudnull | kukacz_ we dont have much in the way of a heat template however you can make an all in one on an openstack with relative ease, https://cloudnull.io/2016/08/auto-building-an-openstack-ansible-test-node-using-nova-and-userdata/ this is what i do with a local cluster | 14:44 |
| *** aicilamd has joined #openstack-ansible | 14:45 | |
| mnaser | im thinking of abandoning our changes out of ci | 14:46 |
| mnaser | there's no point in wasting ci cycles on something that will fail | 14:46 |
| *** jamesdenton has quit IRC | 14:46 | |
| cloudnull | the training team at rackspace have put this together too https://github.com/cloud-training/rpc-heat-ansible | 14:47 |
| cloudnull | bgmccollum ^ | 14:47 |
| cloudnull | kukacz_ you might be able to use something like that | 14:47 |
| *** fghaas has quit IRC | 14:47 | |
| cloudnull | mnaser seems reasonable | 14:48 |
| mnaser | let me see if we can get this through infra quickly | 14:48 |
| *** Leo_m has joined #openstack-ansible | 14:49 | |
| *** sawblade6 has joined #openstack-ansible | 14:49 | |
| bgmccollum | kukacz_ cloudnull those heat templates / playbooks haven't had much attention in over 2 years... :( | 14:50 |
| mnaser | cloudnull: btw i was thinking this morning | 14:51 |
| cloudnull | rutro | 14:51 |
| cloudnull | :) | 14:51 |
| mnaser | (shower thoughts)... maybe we need to have a go at seeing if we can optimize our ssh connection plugin | 14:51 |
| mnaser | i feel like a lot of the stuff like container checks/etc happening on every single task is adding a fair amount of overhead | 14:51 |
| mnaser | not sure if we can implement some sort of fact caching in it.. or something along those lines to minimize the amount of stuff we have to fork out too | 14:52 |
| cloudnull | Yea we might be able to optimise a lot of that | 14:52 |
| mnaser | https://github.com/openstack/openstack-ansible-plugins/blob/master/connection/ssh.py#L429-L471 | 14:52 |
| mnaser | for example that being called every single time | 14:52 |
| cloudnull | we still need to circle up and get these patches backported https://review.openstack.org/#/c/582199 | 14:53 |
| mnaser | thats wild | 14:53 |
| cloudnull | that change doesnt have much impact on an aio | 14:54 |
| cloudnull | but its a MASSIVE impact on a 300 node environment | 14:54 |
| *** sawblade6 has quit IRC | 14:54 | |
| *** rpittau has quit IRC | 14:54 | |
| mnaser | yeah i've noticed for big enviormnents its so sloow | 14:55 |
| mnaser | or we can help dw work on mitogen and do away with all this | 14:55 |
| mnaser | :P | 14:55 |
| cloudnull | noted here https://review.openstack.org/#/c/581893/ from rickbox 3x performance in their tests | 14:55 |
| cloudnull | if mitogen is better lets do it | 14:56 |
| cloudnull | though we might be trading 1 known problem for sets of unknown problems | 14:56 |
| mnaser | it has built in delegation to containers and does the whole start-up-a-process once and run | 14:56 |
| mnaser | yeah, but i think it has the advantage of starting up once so all the code around lookups/etc happens right away i guess | 14:57 |
| cloudnull | yes, im all in favor if its better | 14:58 |
| mnaser | for example maybe we can rely on reading the config file to get rootfs instead of forking out to lxc-info | 14:58 |
| cloudnull | local testing on physical hosts has been good | 14:58 |
| *** radez has joined #openstack-ansible | 14:58 | |
| mnaser | for stuff like _container_path_pad | 14:58 |
| radez | hey folks, the tox file in openstack-ansible references a {toxinidir}/tests/common/ directory with some anwible related test scripts in it, where do I find this directory? | 14:59 |
| mnaser | i.e. reading `/var/lib/lxc/<container>/config` | 14:59 |
| mnaser | radez: this would be the openstack-ansible-test srepo | 14:59 |
| mnaser | openstack/openstack-ansible-tests | 14:59 |
| radez | mnaser: awesome, thx much | 14:59 |
| mnaser | radez: no problem | 14:59 |
| mnaser | cloudnull: is it safe to assume that container path across all operating systems will be /var/lib/lxc/<container>/config for lxc ? | 15:00 |
| odyssey4me | evrardjp: could you take a peek at https://review.openstack.org/591672 given you have better knowledge of what happens in there than most | 15:00 |
| cloudnull | mnaser yes for the config | 15:03 |
| mnaser | ok so we can just read that file instead of having to get the pid using lxc-info and then checking the pid root | 15:04 |
| cloudnull | the pid root is used to get the fs path | 15:05 |
| cloudnull | the path /proc/$PID/rootfs provides access to fhe filesystem | 15:05 |
| cloudnull | no matter the backend store | 15:05 |
| dw | cloudnull / mnaser only one pair of hands :) fixed one of the OSA issues today, but discovered brakage for one of my own apps on the current master | 15:06 |
| cloudnull | **provides access to the running file system | 15:06 |
| dw | i feel a very productive weekend coming on, though ;) hopefully will have good news by monday | 15:06 |
| cloudnull | NICE! | 15:06 |
| mnaser | cloudnull: is that different than just running passing with the rootfs url from the config (esp for fetch_file and put_file) | 15:06 |
| cloudnull | yes. | 15:07 |
| mnaser | er, not url, path | 15:07 |
| cloudnull | the store option in lxc could be a block device, like lvm | 15:07 |
| cloudnull | so access to the filesystem wont be at /var/lib/lxc/$name/rootfs | 15:07 |
| mnaser | aaaaah | 15:08 |
| cloudnull | same for BTRFS / machinctl | 15:08 |
| mnaser | okay yes | 15:08 |
| mnaser | i didnt think of that use case | 15:08 |
| openstackgerrit | Merged openstack/openstack-ansible stable/rocky: Eliminate installing pip on host/containers https://review.openstack.org/591331 | 15:08 |
| cloudnull | also that path is just the bits on the disk , with bind mounts the running fs could be totally different | 15:09 |
| cloudnull | so we use /proc/ to get into the containers for both nspawn and lxc | 15:09 |
| mnaser | yep, gotcha, makes total sense | 15:09 |
| mnaser | i didnt think of that context | 15:09 |
| kukacz_ | cloudnull, bgmccollum: thanks for that link. I've got AIO up and running, was just thinking of saving some effort when getting ready for multinode deployment | 15:13 |
| kukacz_ | nevermind, it's not such a big effort. I might try to build some template myself | 15:13 |
| mnaser | cloudnull: thanks for your insights, in that case i dont think there's much we can do other than maybe looking into implementing a library like python-lxc | 15:14 |
| mnaser | to avoid forking out, which feels expensive | 15:14 |
| mnaser | that would involve an extra dependency which is meh | 15:15 |
| mnaser | i think i'd defer messing around with this until seeing if we can come up with something interesting with mitogen | 15:15 |
| johnsom | cloudnull Can you take a quick look at this quota fix? https://review.openstack.org/#/c/571368/ | 15:17 |
| *** spatel has quit IRC | 15:19 | |
| *** DanyC has quit IRC | 15:20 | |
| *** DanyC has joined #openstack-ansible | 15:21 | |
| *** kevinshort has quit IRC | 15:23 | |
| *** cshen has joined #openstack-ansible | 15:25 | |
| *** DanyC has quit IRC | 15:25 | |
| *** kevinshort has joined #openstack-ansible | 15:25 | |
| *** dante-as has quit IRC | 15:31 | |
| jrosser | mnaser: I did a run with overlayfs and the memory saving seems very significant, no swap used. This Is because the page cache gets shared between containers. The job failed but there’s so much brok/moving I want to investigate some local builds with overlayfs. | 15:31 |
| mnaser | jrosser: i think we used to run overlayfs in ci before but there was some instabilities | 15:31 |
| mnaser | if we can get those cleared out, then a big win | 15:31 |
| mnaser | if it's better, i'd be in favour of marking it the default choice | 15:32 |
| mnaser | tbh, i would love if we used btrfs + cow, that would make creating all those new containers so much faster | 15:32 |
| mnaser | cloudnull being the resident expert on this stuff.. is there anything stopping us from that? | 15:32 |
| jrosser | Yeah, I’d like to understand it more, it’s the only approach that does page sharing that is available on lxc/nspawn and all the distorts | 15:32 |
| jrosser | *distros | 15:32 |
| bgmccollum | jrosser can you have a look at https://review.openstack.org/#/c/592748/ | 15:33 |
| *** kaiokmo has quit IRC | 15:34 | |
| mnaser | jrosser: with overlayfs tho we can also do cow right? | 15:36 |
| odyssey4me | with overlayfs we had issues with icmp packets in the containers IIRC | 15:37 |
| odyssey4me | basically pings didn't work for the aio through the l3 agents - but those are on the host now, so that's possibly no longer an issue | 15:38 |
| odyssey4me | we tried with btrfs, but the issue of the container running out of space while building the repo persisted, and while lots was done to try and fix it - we never got to the bottom of it | 15:39 |
| jrosser | the total memory usage was about 6G | 15:39 |
| jrosser | so if it can be made to work in a reliable/performant way is much better than the other backing stores and similar to zfs | 15:40 |
| *** devx has quit IRC | 15:40 | |
| mnaser | oh boy | 15:41 |
| mnaser | i found more checksum stuff. | 15:41 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_neutron master: Drop checksum checks https://review.openstack.org/593088 | 15:44 |
| *** aicilamd has quit IRC | 15:44 | |
| *** aicilamd has joined #openstack-ansible | 15:45 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Drop checksum checks https://review.openstack.org/593090 | 15:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Drop checksum checks https://review.openstack.org/593091 | 15:48 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Re-written dashboards linking. https://review.openstack.org/593092 | 15:49 |
| *** aicilamd has quit IRC | 15:49 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Re-written dashboards linking. https://review.openstack.org/593092 | 15:51 |
| noonedeadpunk | mnaser: just to remind about myself) https://review.openstack.org/#/c/583943/ | 15:52 |
| odyssey4me | mnaser: have you figured out what's going on with nova compute when it refuses to start? | 15:52 |
| *** aicilamd has joined #openstack-ansible | 15:52 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Add iptables logs https://review.openstack.org/593093 | 15:53 |
| mnaser | odyssey4me: sorry, nothing stuck out to me right away | 15:53 |
| mnaser | noonedeadpunk: ill have a look when i got a moment | 15:53 |
| noonedeadpunk | mnaser: yep, sure no hurry) | 15:54 |
| mnaser | odyssey4me: http://logs.openstack.org/01/593001/2/check/openstack-ansible-functional-ubuntu-xenial/ac815c5/logs/host/neutron-linuxbridge-agent.service.journal.log.txt.gz looks like even neutron failed to start | 15:55 |
| noonedeadpunk | odyssey4me: are you about this http://paste.openstack.org/show/728319/ ? | 15:56 |
| *** agowa338 has quit IRC | 15:59 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590935 | 15:59 |
| mnaser | odyssey4me: i have strong doubts that the checksum stuff might still be messing with things | 16:00 |
| odyssey4me | noonedeadpunk: no, I've not seen that | 16:01 |
| odyssey4me | noonedeadpunk: where did you see that? | 16:01 |
| *** cshen has quit IRC | 16:02 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia master: Set quota for octavia service project https://review.openstack.org/571368 | 16:02 |
| noonedeadpunk | odyssey4me: never mind, it's probably some my own problem on the computing node. I have qemu 2.11, but I catch this error. Probably there is some parsing problem with nova or whatever | 16:03 |
| odyssey4me | johnsom: https://review.openstack.org/571368 updated to use delegation and run early | 16:04 |
| *** cshen has joined #openstack-ansible | 16:04 | |
| *** luksky has quit IRC | 16:06 | |
| *** hamzaachi has joined #openstack-ansible | 16:08 | |
| *** cshen has quit IRC | 16:10 | |
| *** sawblade6 has joined #openstack-ansible | 16:11 | |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_zaqar master: Add description section https://review.openstack.org/580866 | 16:16 |
| *** sawblade6 has quit IRC | 16:16 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Implement more axes for an increased test matrix https://review.openstack.org/551791 | 16:19 |
| johnsom | odyssey4me Thanks, super appreciate it | 16:19 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-pip_install master: Ensure get-pip cannot be modified https://review.openstack.org/593097 | 16:19 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Implement more axes for an increased test matrix https://review.openstack.org/551791 | 16:19 |
| evrardjp | odyssey4me: mnaser cloudnull could I have eyes on https://review.openstack.org/593097 | 16:19 |
| evrardjp | that would be nice to get that in fast | 16:19 |
| evrardjp | well | 16:20 |
| evrardjp | no... there is no urgency | 16:20 |
| evrardjp | but it would be nice to not let that hanging | 16:20 |
| odyssey4me | good find that | 16:20 |
| johnsom | Ouch | 16:20 |
| *** jamesdenton has joined #openstack-ansible | 16:20 | |
| johnsom | mnaser Need a second set of eyes on this nova issue? I see it's hosed up the octavia OSA role too | 16:21 |
| evrardjp | johnsom: sssssh | 16:21 |
| odyssey4me | evrardjp: got two review comments there | 16:22 |
| evrardjp | odyssey4me: done | 16:22 |
| evrardjp | I mean answered | 16:22 |
| odyssey4me | evrardjp: there's another note | 16:23 |
| odyssey4me | regarding the mode | 16:23 |
| *** macza has joined #openstack-ansible | 16:23 | |
| evrardjp | oh I missed that | 16:24 |
| evrardjp | I can do "0500" if you all prefer | 16:24 |
| evrardjp | odyssey4me: I think that was fixed, but let's do it anyway | 16:24 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible-pip_install master: Ensure get-pip cannot be modified https://review.openstack.org/593097 | 16:24 |
| evrardjp | I am done for the day | 16:25 |
| evrardjp | ttyl | 16:25 |
| odyssey4me | thx evrardjp - have a great w/end! | 16:26 |
| noonedeadpunk | evrardjp: Have a good days off) btw, I've placed discussed patch for horizon, but it think it's topic for discussion on monday | 16:30 |
| mnaser | johnsom: that'd be nice, i am pretty clueless | 16:31 |
| evrardjp | thanks noonedeadpunk | 16:32 |
| *** gkapoor1 has quit IRC | 16:32 | |
| evrardjp | and everyone else | 16:32 |
| mnaser | openstack-ansible-deploy-aio_lxc-opensuse-423 periodic job has been failing 24x7 | 16:32 |
| mnaser | btw, infra fixed opensuse issue | 16:34 |
| mnaser | we can recheck things | 16:34 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Add iptables logs https://review.openstack.org/593093 | 16:35 |
| *** kaiokmo has joined #openstack-ansible | 16:36 | |
| *** vollman has quit IRC | 16:36 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Implement more axes for an increased test matrix https://review.openstack.org/551791 | 16:38 |
| mnaser | evrardjp, cloudnull: i'd like to propose we squash https://review.openstack.org/#/c/590778/ with https://review.openstack.org/#/c/593060/ | 16:38 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_horizon master: Re-written dashboards linking. https://review.openstack.org/593092 | 16:38 |
| mnaser | that very job would have passed with 20 extra minutes (for ubuntu), other than the ubuntu xenial stuff | 16:38 |
| cloudnull | +1 | 16:38 |
| mnaser | and maybe https://review.openstack.org/#/c/593090/ alongside it | 16:39 |
| mnaser | because it includes other checksum bits we missed | 16:39 |
| mnaser | esp that aio_lxc on xenial passed in 'SUCCESS in 2h 00m 23s' | 16:39 |
| mnaser | i think the 20 minutes is super useful | 16:40 |
| mnaser | that specific job failed on opensuse (which was broken) and ceph | 16:40 |
| mnaser | (which is fixed in that patch) | 16:40 |
| odyssey4me | adding time is a slippery slope | 16:40 |
| odyssey4me | it's always better to save time rather than add time | 16:41 |
| mnaser | odyssey4me: it finished in 2 hours on the dot. i would love to look into ways for us to save time but right now we are struggling to land things | 16:41 |
| mnaser | i am totally totally totally not happy with that but it at least gives us the ability to merge fixes | 16:41 |
| mnaser | given that since aug 10th we havent landed anything because of the ceph issue | 16:42 |
| mnaser | and we've had 12 rechecks since | 16:42 |
| odyssey4me | yeah, I hear you - but then we need to focus on cutting time this cycle - because a 2 hour control plane deployment is way too long | 16:43 |
| mnaser | odyssey4me: i agree with you 100% | 16:44 |
| mnaser | hands down i think this should be a priority in the cycle, hence the mitogen work and looking at optimizing the connection plugin or whatever is involved | 16:44 |
| mnaser | deploying a bunch of python code should not take 2 hours | 16:45 |
| mnaser | devstack goes up in 20 minutes i think. we build wheels which takes 10 minutes. i think 30-40 minutes for us should be what we should aim for. IMHO. | 16:45 |
| bgmccollum | during deployment, are there any openstack inter-service dependencies besides keystone? | 16:46 |
| odyssey4me | yes, but that would mean we need to drop services - because devstack doesn't deploy everything we deploy | 16:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Implement more axes for an increased test matrix https://review.openstack.org/551791 | 16:46 |
| bgmccollum | wondering if openstack-ansible could parallelize the per-service playbooks after keystone is laid down | 16:47 |
| odyssey4me | that's what https://review.openstack.org/551791 is doing - it uses the compute kit, just like devstack | 16:47 |
| odyssey4me | bgmccollum: we've tried that before, and it gets complicated real fast | 16:47 |
| odyssey4me | We end up having to do a bunch of complex bash things to make that happen - and it makes the log output less than useful | 16:48 |
| *** DanyC has joined #openstack-ansible | 16:48 | |
| bgmccollum | yeah, i would expect so...and all kinds of potential race conditions are exposed | 16:48 |
| mnaser | odyssey4me: i think ideally we just have different openstack_user_config.yml files depending on the 'scenario' | 16:48 |
| odyssey4me | mnaser: we already do | 16:49 |
| mnaser | and then also bgmccollum idea is good but we lose a lot of ansible stuff | 16:49 |
| mnaser | ah sorry i meant the complex bash stuff was re: dropping services | 16:49 |
| odyssey4me | no, the complex bash stuff is for parallelising | 16:49 |
| johnsom | mnaser File permission issue: http://logs.openstack.org/46/592046/1/check/openstack-ansible-functional-ubuntu-xenial/39ef5a2/logs/host/syslog.txt.gz#_Aug_15_14_41_24 | 16:50 |
| odyssey4me | to make the deploy faster, we have to fundamentalls change how we deploy | 16:50 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_octavia master: use include_tasks instead of include https://review.openstack.org/585281 | 16:50 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Temporarily use tempest master for all builds and fix configs https://review.openstack.org/590778 | 16:50 |
| johnsom | Failed at step EXEC spawning /openstack/venvs/nova-testing/bin/nova-compute: Permission denied | 16:50 |
| mnaser | odyssey4me, cloudnull, evrardjp: squashed commit ^ | 16:50 |
| mnaser | johnsom: thats so strange.. | 16:50 |
| bgmccollum | put ansible in puppet mode? :P | 16:50 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_octavia master: use include_tasks instead of include https://review.openstack.org/585281 | 16:51 |
| odyssey4me | wow johnsom that's a really good find | 16:51 |
| mnaser | odyssey4me: i totally have thought about finding a way of making our installs do a single transaction | 16:51 |
| mnaser | yeah there's no way i would have caught that | 16:51 |
| mnaser | now why it's doing that... | 16:51 |
| cloudnull | johnsom that should be resolved with https://review.openstack.org/#/c/591463/ | 16:52 |
| bgmccollum | centralized deployment preps hosts / containers, then instructs / orchestrates those hosts to "pull" playbooks and run independently and in parallel...totally flip the script | 16:52 |
| *** gyee has joined #openstack-ansible | 16:52 | |
| bgmccollum | crazy talk ^ | 16:52 |
| johnsom | cloudnull This is the nova code though, it looks like that fix is only for neutron | 16:52 |
| johnsom | Probably need a similar patch on the nova side.... | 16:53 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-repo_build master: Remove the unnecessary space https://review.openstack.org/582787 | 16:53 |
| cloudnull | if you can check the ownership of that path I suspect it's 700 and owned by neutron | 16:53 |
| cloudnull | at least that's what i found | 16:54 |
| mnaser | neutron owns /openstack/venvs/nova-testing/bin/nova-compute? :X | 16:54 |
| johnsom | mnaser +1 ????? | 16:54 |
| *** aicilamd has quit IRC | 16:54 | |
| mnaser | also that patch seems to edit the permissions for /etc/neutron rather so im not sure | 16:55 |
| cloudnull | its recursive | 16:55 |
| *** aicilamd has joined #openstack-ansible | 16:55 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-repo_build master: Remove the unnecessary space https://review.openstack.org/582787 | 16:55 |
| odyssey4me | I think that fix is good, but don't think that's the same issue | 16:56 |
| johnsom | Oh, I see, you are doing /openstack down... Hmm, seems odd/dangerous to do that in the neutron role | 16:56 |
| mnaser | its just doing root:root for /openstack | 16:56 |
| odyssey4me | I'm not seeing recursive there cloudnull | 16:56 |
| johnsom | Same with /etc/sudoers.d .... | 16:57 |
| mnaser | shouldnt repo_build create a venv that has executable stuff already? | 16:57 |
| mnaser | sounds like its creating it without it | 16:57 |
| cloudnull | like i said, check the path. i bet the ownership is wrong. | 16:58 |
| cloudnull | at least that's what I was seeing | 16:58 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-repo_build master: Remove the unnecessary space https://review.openstack.org/582787 | 16:58 |
| johnsom | Is this something that changed recently? Do we need to update all of the service roles for a permission change? | 16:58 |
| cloudnull | its expanding the venv and making sure its owned by the given service user. | 16:59 |
| *** aicilamd has quit IRC | 16:59 | |
| mnaser | its curious that it doesnt happen in the integrated repo | 17:00 |
| cloudnull | johnsom i dont think so. I only susepect the neutron role because it was the first one w/ the smart sources change | 17:01 |
| *** cshen has joined #openstack-ansible | 17:01 | |
| cloudnull | and that permission issue fix i linked to was something I forgot to do | 17:01 |
| * mnaser add this under the "reasons why we need to leverage integrated repo playbooks for role testing" | 17:01 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Implement more axes for an increased test matrix https://review.openstack.org/551791 | 17:02 |
| *** mmalchuk has joined #openstack-ansible | 17:04 | |
| *** aicilamd has joined #openstack-ansible | 17:04 | |
| cloudnull | johnsom if that pr doesn't fix the issue then something else is going on. but i had seen something very similar testing rocky in a largish lab | 17:04 |
| mnaser | odyssey4me: i like that! | 17:05 |
| *** mmalchuk_ has joined #openstack-ansible | 17:05 | |
| cloudnull | nova was not able to start because the nova didn't own the executable the service unit was trying to run | 17:05 |
| cloudnull | ^ that was the issue i saw | 17:06 |
| *** sawblade6 has joined #openstack-ansible | 17:07 | |
| *** mmalchu__ has joined #openstack-ansible | 17:08 | |
| *** mmalchuk has quit IRC | 17:08 | |
| *** mmalchuk_ has quit IRC | 17:09 | |
| *** aicilamd has quit IRC | 17:11 | |
| *** aicilamd has joined #openstack-ansible | 17:12 | |
| cloudnull | odyssey4me https://review.openstack.org/#/c/551791 looks great! | 17:13 |
| mnaser | cloudnull: https://review.openstack.org/#/c/593093/ should help us find any other traces | 17:15 |
| mnaser | of the iptables stuff | 17:16 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Implement more axes for an increased test matrix https://review.openstack.org/551791 | 17:16 |
| mnaser | http://logs.openstack.org/93/593093/2/check/openstack-ansible-functional-opensuse-423/d5c3cac/logs/ you can see it working there ! | 17:16 |
| dw | aside from package manager state, would any parts of the OSA stuff benefit from keeping state in ram on the target across tasks? i've already been looking at ways to speed up apt significantly, but i'm not familiar with the OSA stuff in general | 17:16 |
| odyssey4me | cloudnull: tbh the whole test matrix publishing is a royal pain when trying to do this - it could be a lot simpler if we ditch that | 17:16 |
| cloudnull | container pids would be lovely :) | 17:16 |
| dw | (re: apt, there is no reason to e.g. re-read package lists constantly if they haven't changed since last run) | 17:17 |
| cloudnull | -cc dw mnaser | 17:17 |
| mnaser | dw: nothing comes to mind honestly. yum probably has the same useful need. | 17:17 |
| odyssey4me | we could simply have the compute kit by default, and have an override var for extra services which is used per role | 17:17 |
| cloudnull | odyssey4me +1 from me | 17:17 |
| mnaser | odyssey4me: i think that's the way to go. we deploy the equiv of devstack | 17:17 |
| cloudnull | if that stuff is no longer useful delete it | 17:17 |
| dw | mnaser: indeed.. but it's a fairly big chunk to take on, so one at a time definitely for 'smart' modules :) | 17:18 |
| dw | cloudnull: you get that for free with the connection delegation stuff, so hopefully that'll happen soon now | 17:18 |
| mnaser | dw: no worries. i am a centos user and its quite a bit slower (from i guess yum) | 17:18 |
| odyssey4me | well, if you folks are up for that - then I'll go ahead and simplify it | 17:18 |
| cloudnull | dw cool! | 17:18 |
| mnaser | i wonder what are the default devstack services that they deploy | 17:18 |
| cloudnull | odyssey4me ++ | 17:18 |
| mnaser | in devstack-gate | 17:18 |
| odyssey4me | the general idea being that we have a small set of base scenario's, then pluggable extensions to it to add more networking or whatever (for things like octavia) | 17:19 |
| odyssey4me | mnaser: its compute kit - nothing more | 17:19 |
| mnaser | okay so someone already done the mapping, cool | 17:19 |
| odyssey4me | no horizon, no designate, no swift | 17:19 |
| mnaser | what about cinder? cinder with lvm? | 17:19 |
| odyssey4me | cutting swift out saves us around 15-20 mins already | 17:19 |
| cloudnull | a swift only scenario would make a lot sense | 17:20 |
| mnaser | the idea behind moving the integrated repo to computekit only and then having projects run integrated repo + said project only makes a lot of sense. | 17:20 |
| mnaser | but my concern is if we make a magnum change in group_vars... how'd that be tested | 17:21 |
| odyssey4me | mnaser: yep https://github.com/openstack-dev/devstack/blob/master/.zuul.yaml#L332-L336 | 17:21 |
| cloudnull | well. maybe a swift + basekit for the folks using swift as a glance backend | 17:21 |
| mnaser | cloudnull: we can have all the custom swift jobs we want.. inside os_swift, and that can cross-test with os_glance too.. i guess | 17:21 |
| cloudnull | we could implement something similar to what we have in the ops repo | 17:21 |
| odyssey4me | cloudnull: yep, can obviously do that - but do it in the siwft role | 17:21 |
| odyssey4me | *swift | 17:21 |
| cloudnull | which loads different tests based on what was changed | 17:22 |
| mnaser | because we totally don't need to deploy ceph everytime we make a change to something totally unrelated | 17:22 |
| *** Darcidride has joined #openstack-ansible | 17:22 | |
| mnaser | alright but forgive me, whats the difference between computekit and basekit | 17:22 |
| cloudnull | odyssey4me ++ | 17:22 |
| odyssey4me | well, openstack is rather intertwined | 17:22 |
| odyssey4me | there is no such thing as 'base kit' - that's an OSA thing | 17:23 |
| odyssey4me | base kit in OSA = compute kit in openstack | 17:23 |
| mnaser | ok ack | 17:23 |
| mnaser | so running both a 'basekit' job and a 'full aio' is pretty redundant | 17:23 |
| odyssey4me | so with this I'm also normalising the name to something familiar to openstack in general | 17:23 |
| mnaser | because you're covering some parts twice | 17:23 |
| mnaser | i.e. you're deploying nova twice, etc | 17:23 |
| odyssey4me | yes, the current aio has more coverage | 17:24 |
| mnaser | ok i see, but if they're both voting, the basekit job is just extra potential for breaking things :( | 17:24 |
| odyssey4me | ultimately if we move to using the integrated build for everything, we'd likely have to also use bitrot jobs to keep us honest across all services | 17:24 |
| odyssey4me | because having every patch run every test is just not sustainable | 17:24 |
| cjloader | is cirros image broken in master | 17:24 |
| odyssey4me | cjloader: not as far as we know | 17:25 |
| cjloader | trying to ssh into a cirros image | 17:25 |
| mnaser | shouldnt, what are you seeing cjloader ? | 17:25 |
| odyssey4me | more words? | 17:25 |
| cjloader | in mnaio | 17:25 |
| * mnaser doesnt use that too much, sorry | 17:25 | |
| cjloader | says no route to host | 17:25 |
| cloudnull | sounds like its missing a route :P | 17:26 |
| odyssey4me | typically that's a networking issue of some sort | 17:26 |
| cjloader | MNAIO networking? | 17:26 |
| mnaser | cloudnull coming in with the friday-levels troubleshooting | 17:26 |
| cloudnull | hahaha | 17:26 |
| *** luksky has joined #openstack-ansible | 17:26 | |
| odyssey4me | cjloader: that's like asking how long a piece of string is | 17:26 |
| cjloader | odyssey4me: haha | 17:27 |
| odyssey4me | you'd have to build it and trace the packet path | 17:27 |
| mnaser | dw: another one came to mind, yum_repository / apt_repository | 17:27 |
| cloudnull | cjloader make sure teh security groups are in a good spot | 17:27 |
| mnaser | that might be useful to maintain state and remains relatively static | 17:27 |
| cloudnull | like you've allowed tcp/udp/icmp as needed | 17:27 |
| cloudnull | after that tcpdump is your friend | 17:28 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Implement more axes for an increased test matrix https://review.openstack.org/551791 | 17:29 |
| mnaser | can i get some eyes on https://review.openstack.org/#/c/593093/2 and the parent changes please | 17:29 |
| mnaser | im a little bummed centos is still not passing in integrated | 17:32 |
| mnaser | i thought cinder fixes would make it pass | 17:32 |
| dw | mnaser: those should be quite cheap with the extension already | 17:32 |
| dw | mnaser: i've been looking at this for a few months, and apt/yum are so far probably the most interesting cases i think | 17:33 |
| dw | it's a ton of work to make it work though :) but it'll be done | 17:33 |
| dw | just need to keep the feature count vs. bug count optimized ;) | 17:33 |
| *** sawblade6 has quit IRC | 17:41 | |
| mnaser | dw: yep! i'd love to help but there's quite a few things on hand at the moment | 17:42 |
| bgmccollum | did the suse mirrors get worked out? | 17:43 |
| mnaser | bgmccollum: yes sir | 17:43 |
| bgmccollum | ty | 17:43 |
| cloudnull | dw apt yum zypper it at all possible . | 17:43 |
| cloudnull | our suse work is arguably more stable than the cent work (minus the recent repo issues) | 17:44 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Update permissions on execution path folders https://review.openstack.org/591463 | 17:45 |
| *** aicilamd has quit IRC | 17:46 | |
| *** aicilamd has joined #openstack-ansible | 17:46 | |
| bgmccollum | is `regate` a thing? | 17:51 |
| *** aicilamd has quit IRC | 17:51 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: [WIP] Implement more axes for an increased test matrix https://review.openstack.org/551791 | 17:51 |
| cloudnull | bgmccollum just recheck | 17:51 |
| odyssey4me | bgmccollum: it's "recheck" only | 17:51 |
| cloudnull | we used to have reverify but i think that just runs recheck now | 17:52 |
| odyssey4me | yep | 17:52 |
| odyssey4me | there is no escape | 17:52 |
| bgmccollum | saw references to "regate" in Asterisk's zuul...and was wondering | 17:52 |
| cloudnull | while we're in a delete things mood, anyone want to give https://review.openstack.org/#/q/starredby:cloudnull+status:open,n,z a review? | 17:53 |
| bgmccollum | https://wiki.asterisk.org/wiki/display/AST/Continuous+Integration | 17:53 |
| *** aicilamd has joined #openstack-ansible | 17:53 | |
| bgmccollum | also cool that they are using gerrit / zuul | 17:54 |
| cloudnull | that is cool! | 17:55 |
| cjloader | cloudnull: neutron.plugins.ml2.managers [req-057bd2f2-86f4-4b02-bb92-5871c3c8554c - - - - -] Failed to bind port ce37b1d6-bcc5-4e74-bd01-ef169b7fb25d on host infra1 for vnic_type normal using segments [{'network_id': 'e49e1259-cf69-46b4-ad31-27a716a9ad3b', 'segmentation_id': None, 'physical_network': u'flat', 'id': '4fef11a7-ea7c-4a34-aa0b-ea7e6daf8593', 'network_type': u'flat'}] | 18:00 |
| *** schwicht has joined #openstack-ansible | 18:03 | |
| *** spatel has joined #openstack-ansible | 18:07 | |
| *** Darcidride has quit IRC | 18:08 | |
| mnaser | um | 18:09 |
| mnaser | okay nevermind | 18:09 |
| mnaser | so neutron seems to attach 172.29.249.135 successfully to the mv | 18:10 |
| mnaser | vm* | 18:10 |
| mnaser | but it cannot ssh into it (timeout) | 18:10 |
| *** cshen has quit IRC | 18:11 | |
| odyssey4me | mnaser: role test or integrated? | 18:11 |
| mnaser | odyssey4me: integrated for centos | 18:11 |
| mnaser | seems like its the last thing not working | 18:11 |
| odyssey4me | perhaps a firewall rule or something? not sure if cent has a firewall by default | 18:12 |
| mnaser | hmm | 18:12 |
| mnaser | im wondering if it just has no route at all | 18:12 |
| odyssey4me | or that | 18:12 |
| mnaser | http://logs.openstack.org/78/590778/6/check/openstack-ansible-deploy-aio_metal-centos-7/fdd9d28/logs/openstack/instance-info/host_networkd_status_15-21-38.log.txt.gz | 18:12 |
| mnaser | 172.29.249.135 doesnt look like would be reachable | 18:12 |
| *** aicilamd has quit IRC | 18:13 | |
| mnaser | let me compare to another passing job like ubuntu | 18:13 |
| bgmccollum | 172.29.248.1 (external network | 18:14 |
| *** aicilamd has joined #openstack-ansible | 18:14 | |
| bgmccollum | 172.29.248.1 (external network's gateway) configured on br-vlan? (thats how its worked in the past)... | 18:14 |
| bgmccollum | oh yeah, its there...ignore me | 18:14 |
| mnaser | okay so in that case they both work fine | 18:15 |
| mnaser | http://logs.openstack.org/78/590778/6/check/openstack-ansible-deploy-aio_metal-ubuntu-xenial/05ea89d/logs/openstack/instance-info/host_firewall_info_14-34-13.log.txt.gz | 18:15 |
| mnaser | vs | 18:15 |
| mnaser | http://logs.openstack.org/78/590778/6/check/openstack-ansible-deploy-aio_metal-centos-7/fdd9d28/logs/openstack/instance-info/host_firewall_info_15-21-38.log.txt.gz | 18:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate master: use include_tasks instead of include https://review.openstack.org/585278 | 18:16 |
| mnaser | so iptables rules are ok, routes are tehre | 18:16 |
| mnaser | there* | 18:17 |
| mnaser | the instance is up and seems to get an ip via dhcp according to tempest logs | 18:17 |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-tests master: Fix to setup notify mq service https://review.openstack.org/593127 | 18:17 |
| mnaser | http://logs.openstack.org/78/590778/6/check/openstack-ansible-deploy-aio_metal-centos-7/fdd9d28/logs/openstack/aio1-utility/stestr_results.html | 18:17 |
| mnaser | https://www.irccloud.com/pastebin/I5lkmdgX/ | 18:18 |
| mnaser | hmm | 18:18 |
| mnaser | WARN: failed: route add -net "0.0.0.0/0" gw "192.168.74.1" | 18:18 |
| *** hamzaachi has quit IRC | 18:18 | |
| *** erolg has quit IRC | 18:18 | |
| *** hamzaachi has joined #openstack-ansible | 18:18 | |
| *** aicilamd has quit IRC | 18:19 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_nova master: Setup oslo.messaging extra packages for optional drivers https://review.openstack.org/584450 | 18:22 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Drop opensuse-150 jobs https://review.openstack.org/593061 | 18:23 |
| odyssey4me | cloudnull: http://logs.openstack.org/91/551791/10/check/openstack-ansible-deploy-compute_kit-nspawn-lxb-lvm-source-ubuntu-xenial/47eacc4/job-output.txt.gz#_2018-08-17_18_10_54_799307 is the first reason why nspawn fails in the gate - it's a bit odd, given that lxc works with the same play | 18:23 |
| odyssey4me | I think it might be due to https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/tasks/openstack_hosts_configure_apt.yml#L43 - I think the nspawn containers have a recent file, so it doesn' tupdate the cache | 18:24 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Drop checksum checks https://review.openstack.org/593091 | 18:25 |
| cloudnull | odyssey4me this resolves that https://review.openstack.org/#/c/588430 though the gate is still failing just further along | 18:26 |
| bgmccollum | mnaser i see that same warning on a cirros instance i just booted | 18:26 |
| mnaser | and the instance is bootable and functioning, bgmccollum ? | 18:27 |
| cloudnull | that last pr was fialing on cinder things, which may actually be resolved now ? | 18:27 |
| mnaser | cloudnull: the cinder issue is gone, but it looks like in this case teh smoke test to create a vm and attach a floating ip is failing | 18:27 |
| bgmccollum | mnaser yup just was about to ping / ssh in | 18:27 |
| bgmccollum | *able | 18:27 |
| mnaser | everything works fine, the vm is responding and networking goes up | 18:28 |
| odyssey4me | cloudnull: https://review.openstack.org/593033 should resolve the cinder things | 18:28 |
| mnaser | floating ip gets successfully assigned too | 18:28 |
| mnaser | but it doesnt respond to ssh | 18:28 |
| *** DanyC has quit IRC | 18:29 | |
| *** dxiri has quit IRC | 18:29 | |
| mnaser | http://logs.openstack.org/78/590778/6/check/openstack-ansible-deploy-aio_metal-centos-7/fdd9d28/ if anyone wants to give another pair of eyes | 18:29 |
| *** cshen has joined #openstack-ansible | 18:29 | |
| mnaser | oh man | 18:30 |
| mnaser | i wonder if iptables-save is missing? | 18:30 |
| *** hamzaachi has quit IRC | 18:34 | |
| mnaser | http://logs.openstack.org/78/590778/6/check/openstack-ansible-deploy-aio_metal-centos-7/fdd9d28/logs/openstack/aio1-neutron/neutron-l3-agent.log.txt.gz#_2018-08-17_15_11_12_355 | 18:35 |
| mnaser | i mean neutron thinks it goes up | 18:35 |
| mnaser | oh sh | 18:36 |
| mnaser | you know what | 18:36 |
| bgmccollum | checksum? | 18:36 |
| mnaser | this reminds me, someone mentioned there was a systemd bug | 18:36 |
| mnaser | with networkd | 18:36 |
| mnaser | on centos | 18:36 |
| mnaser | because it uses an older release | 18:36 |
| mnaser | https://bugs.launchpad.net/openstack-ansible/+bug/1778537 | 18:37 |
| openstack | Launchpad bug 1778537 in openstack-ansible "LXC bridge issue with networkd on CentOS 7" [Low,Confirmed] - Assigned to Kevin Carter (kevin-carter) | 18:37 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Add iptables logs https://review.openstack.org/593093 | 18:37 |
| *** aicilamd has joined #openstack-ansible | 18:37 | |
| mnaser | someone linked me a bug on github.com grr | 18:39 |
| jrosser | mnaser: http://logs.openstack.org/78/590778/6/check/openstack-ansible-deploy-aio_metal-centos-7/fdd9d28/logs/openstack/instance-info/links_15-21-38.log.txt.gz | 18:39 |
| odyssey4me | cloudnull: mind if I tweak https://review.openstack.org/#/c/588430/3 a bit? | 18:39 |
| jrosser | eth12 being down isnt going to help | 18:39 |
| mnaser | jrosser: you're right, that must be it | 18:40 |
| * mnaser looks at xenial | 18:40 | |
| jrosser | mnaser: throw on a depends-on this https://review.openstack.org/#/c/590424/ | 18:40 |
| mnaser | http://logs.openstack.org/78/590778/6/check/openstack-ansible-deploy-aio_metal-ubuntu-xenial/05ea89d/logs/openstack/instance-info/links_14-34-13.log.txt.gz yeah it was up in eth12 | 18:40 |
| *** spatel has quit IRC | 18:41 | |
| mnaser | jrosser: on your change, the problem exists too http://logs.openstack.org/24/590424/3/check/openstack-ansible-deploy-aio_metal-centos-7/2a42759/logs/openstack/instance-info/links_09-07-14.log.txt.gz | 18:41 |
| jrosser | ah right i was really concentating on bionic there, it must be a different issue | 18:42 |
| *** spatel has joined #openstack-ansible | 18:42 | |
| *** dxiri has joined #openstack-ansible | 18:42 | |
| mnaser | cloudnull: maybe might have an idea about why that link is down | 18:42 |
| *** gyee has quit IRC | 18:43 | |
| jrosser | however i think the part of my change that makes the whole thing abort very early if the bridges are broken is helpful | 18:43 |
| mnaser | http://git.openstack.org/cgit/openstack/openstack-ansible/tree/tests/roles/bootstrap-host/defaults/main.yml#n198 | 18:44 |
| mnaser | those state change are very ubuntu-y | 18:44 |
| *** ppetit has quit IRC | 18:44 | |
| mnaser | but i think `bootstrap_host_bridges_interfaces` is not even used anymore | 18:45 |
| mnaser | it has zero references | 18:45 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Remove un-used bootstrap variables https://review.openstack.org/593137 | 18:50 |
| *** aicilamd has quit IRC | 18:52 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove the last remnants of get-pip.py https://review.openstack.org/591392 | 18:52 |
| *** cshen has quit IRC | 18:53 | |
| *** mmalchuk has joined #openstack-ansible | 18:57 | |
| mnaser | OH MY GOD | 18:58 |
| mnaser | jrosser: | 18:58 |
| mnaser | # which ip => /usr/sbin/ip | 18:58 |
| *** mmalchu__ has quit IRC | 18:58 | |
| *** cshen has joined #openstack-ansible | 18:59 | |
| mnaser | thanks distros for putting stuff everywhere | 19:00 |
| jrosser | and we hardwire the path somewhere? | 19:01 |
| prometheanfire | mnaser: /bin/ip | 19:01 |
| jrosser | oh bootstrap-host | 19:02 |
| prometheanfire | iirc we changed that at some point (a year or two ago) to match other distros | 19:02 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Ensure package cache is updated for nspawn containers https://review.openstack.org/588430 | 19:02 |
| *** aicilamd has joined #openstack-ansible | 19:02 | |
| mnaser | prometheanfire: right, so because of that, centos doesnt ever go up | 19:02 |
| mnaser | because the paths dont match | 19:03 |
| *** aicilamd has quit IRC | 19:04 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Ensure package cache is updated for nspawn containers https://review.openstack.org/588430 | 19:04 |
| odyssey4me | hooray for standards! | 19:04 |
| *** aicilamd has joined #openstack-ansible | 19:04 | |
| *** aicilamd has quit IRC | 19:05 | |
| *** aicilamd has joined #openstack-ansible | 19:05 | |
| *** hamzaachi has joined #openstack-ansible | 19:08 | |
| prometheanfire | :D | 19:08 |
| mnaser | anyone here has access to a ubuntu and suse machine | 19:08 |
| mnaser | and an verify paths for iptables, ethtool and ip? | 19:08 |
| prometheanfire | mnaser: cmd of 'ip' with a path of '/usr/bin:/bin:/sbin:/usr/sbin' ? | 19:09 |
| *** spatel has quit IRC | 19:09 | |
| mnaser | prometheanfire: systemd doesnt let you use path in its execstarts i believe | 19:09 |
| mnaser | you need to specify full paths afaik | 19:09 |
| prometheanfire | oh, that's a unit | 19:10 |
| mnaser | yeah | 19:10 |
| mnaser | i created a mapping, i got the values for centos | 19:10 |
| mnaser | so just need them for ubuntu and suse | 19:10 |
| prometheanfire | doesn't look like shell or command modules allow specifying path | 19:10 |
| prometheanfire | should switch to puppet | 19:10 |
| mnaser | hah | 19:10 |
| *** mmalchuk_ has joined #openstack-ansible | 19:10 | |
| mnaser | :P | 19:10 |
| odyssey4me | prometheanfire: all ansible paths allow the setting of any env var | 19:11 |
| odyssey4me | using the environment argument | 19:12 |
| odyssey4me | s/paths/tasks/ | 19:12 |
| *** mmalchuk has quit IRC | 19:13 | |
| *** cshen has quit IRC | 19:14 | |
| *** cshen has joined #openstack-ansible | 19:14 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Use operating system specific IP utilities https://review.openstack.org/593146 | 19:16 |
| mnaser | odyssey4me, cloudnull, jrosser: the squashed change has passed! https://review.openstack.org/#/c/590778/7 | 19:16 |
| mnaser | evrardjp: ^ | 19:16 |
| mnaser | look at all that green | 19:17 |
| odyssey4me | yeah, it's been a while | 19:17 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/rocky: Remove the checksum fills https://review.openstack.org/592690 | 19:18 |
| *** Leo_m has quit IRC | 19:18 | |
| mnaser | i am hoping that last change | 19:18 |
| mnaser | centos will pass | 19:18 |
| mnaser | and all of our jobs will be green then | 19:18 |
| *** Leo_m has joined #openstack-ansible | 19:18 | |
| odyssey4me | mnaser: so the only trouble you'll have after that merges is to port some of those changes back to rocky | 19:18 |
| mnaser | odyssey4me: i think all of those are backportable | 19:19 |
| mnaser | except for the change that reverts to using the released tempest | 19:19 |
| mnaser | but that can be done afterwards | 19:19 |
| odyssey4me | in pieces perhaps, but that patch as it stands is not given that parts of it are already in rocky | 19:19 |
| odyssey4me | I guess we'll see | 19:19 |
| mnaser | just having a stable CI is already great | 19:20 |
| mnaser | 1h30m reliably seems ok | 19:21 |
| mnaser | except for the occasional slow vm | 19:21 |
| odyssey4me | ok folks, I'm out - have a great w/end! | 19:25 |
| *** tomtom001 has quit IRC | 19:26 | |
| dw | cloudnull: it depends on the structure of the underlying package manager.. with libapt it seems at least possible to maintain parsed archive state across invocations | 19:27 |
| dw | yum in mostly python AFAIK so in principle if it's doable for apt it should be doubly doable for yum | 19:27 |
| dw | cloudnull: i'm not even sure it's possible with libapt yet :) the documentation is the source code. but the capability to keep state cached until end of run exists for any place it can be exploited | 19:28 |
| mnaser | take care odyssey4me | 19:32 |
| *** aicilamd has quit IRC | 19:33 | |
| *** tailDASHeF has joined #openstack-ansible | 19:33 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: use include_tasks instead of include https://review.openstack.org/585304 | 19:44 |
| *** jamesdenton has quit IRC | 19:46 | |
| *** aicilamd has joined #openstack-ansible | 19:48 | |
| *** nicolasbock has quit IRC | 19:50 | |
| *** aicilamd has quit IRC | 19:52 | |
| *** aicilamd has joined #openstack-ansible | 19:52 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible-os_neutron master: Setup oslo.messaging extra packages for optional drivers https://review.openstack.org/583021 | 19:53 |
| *** nicolasbock has joined #openstack-ansible | 19:55 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: Move swift ring tasks before systemd tasks https://review.openstack.org/592748 | 19:55 |
| cloudnull | mnaser back , reviewing now | 19:56 |
| *** aicilamd has quit IRC | 19:57 | |
| cloudnull | odyssey4me you're always welcome to tweak my patches | 19:57 |
| cloudnull | mnaser still need a suse machine check? | 20:00 |
| cloudnull | for ip it's /bin/ip | 20:00 |
| mnaser | cloudnull: if you want to double check the values in https://review.openstack.org/593146 that would be useful | 20:00 |
| mnaser | okay so my patch is borked | 20:00 |
| openstackgerrit | Byron McCollum proposed openstack/openstack-ansible-os_swift stable/rocky: Move swift ring tasks before systemd tasks https://review.openstack.org/593151 | 20:01 |
| mnaser | cloudnull: ill wait for the centos run to finish and see if it works, and then i will respin with the correct path | 20:01 |
| *** aicilamd has joined #openstack-ansible | 20:01 | |
| *** aicilamd has quit IRC | 20:01 | |
| mnaser | if you have access to the other values that'd be nice/useful (iptables and ethtool) | 20:02 |
| *** aicilamd has joined #openstack-ansible | 20:02 | |
| *** aicilamd has quit IRC | 20:03 | |
| *** aicilamd has joined #openstack-ansible | 20:03 | |
| cloudnull | I commented in the review | 20:04 |
| cloudnull | from a opensuse 423 machine | 20:04 |
| mnaser | thank you very muuch cloudnull | 20:05 |
| cloudnull | that's a good find. | 20:05 |
| * cloudnull would have never thought the path would be different on different distros | 20:06 | |
| cloudnull | for core utils that is | 20:06 |
| bgmccollum | generally speaking, safe to recheck proposal bot stuff? | 20:08 |
| cloudnull | generally speaking, yes | 20:08 |
| cloudnull | i think the repo crisis is over now | 20:08 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/rocky: zuul: Drop distro jobs for Xenial and switch to smoke tests https://review.openstack.org/592686 | 20:10 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Add test for Ubuntu Bionic https://review.openstack.org/591287 | 20:16 |
| *** hamzaachi has quit IRC | 20:17 | |
| *** aicilamd has quit IRC | 20:21 | |
| *** Leo_m has quit IRC | 20:25 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Always start cinder API last https://review.openstack.org/593033 | 20:25 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590904 | 20:25 |
| *** Leo_m_ has joined #openstack-ansible | 20:25 | |
| *** kevinsho_ has joined #openstack-ansible | 20:29 | |
| *** aicilamd has joined #openstack-ansible | 20:30 | |
| *** kevinshort has quit IRC | 20:30 | |
| *** kevinsho_ has quit IRC | 20:31 | |
| *** aicilamd has quit IRC | 20:35 | |
| *** aicilamd has joined #openstack-ansible | 20:38 | |
| *** aicilamd has quit IRC | 20:42 | |
| *** hamzaachi has joined #openstack-ansible | 20:49 | |
| mnaser | oh my | 20:53 |
| mnaser | cloudnull: somehow.. opensuse passed anyways | 20:55 |
| cloudnull | those commands would fail, unless theirs a link. | 20:55 |
| mnaser | in other news... https://usercontent.irccloud-cdn.com/file/enXrsX9g/Screen%20Shot%202018-08-17%20at%204.55.30%20PM.png | 20:56 |
| mnaser | :D | 20:56 |
| cloudnull | its also likley those commands are not 100% needed the ethtool commands especially | 20:56 |
| mnaser | so happy to see that | 20:56 |
| cloudnull | woot! | 20:56 |
| mnaser | ONLY TOOK AN ENTIRE CYCLE | 20:56 |
| mnaser | ok: Runtime: 1:30:14.522215 | 20:57 |
| mnaser | pretty reasonable runtime | 20:57 |
| cloudnull | not terribad | 20:57 |
| mnaser | ceph on xenial takes 97 minutes | 20:57 |
| cloudnull | now to get the same gate passing w/ containers. | 20:57 |
| mnaser | xenial metal takes 67 minutes | 20:58 |
| cloudnull | yea. | 20:58 |
| mnaser | still got a little bit of time to catch up on | 20:58 |
| cloudnull | but its great progress! | 20:58 |
| mnaser | i mean, technically, container job should be like, 10 minutes slower? | 20:58 |
| mnaser | 10-15 minutes to setup the containers | 20:58 |
| mnaser | not more than that.. i think? | 20:58 |
| cloudnull | ++ it shouldnt be much slower | 20:58 |
| mnaser | this is a nice way to end the weekend | 20:59 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Use operating system specific IP utilities https://review.openstack.org/593146 | 20:59 |
| mnaser | cloudnull: updated with suse path ^ | 20:59 |
| mnaser | path*s | 20:59 |
| cloudnull | nicely done btw! | 21:00 |
| cloudnull | great to see cent getting love | 21:00 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Switch CentOS metal jobs to voting https://review.openstack.org/593179 | 21:00 |
| mnaser | cloudnull: we run it everywhere in prod so it will continue to get love :p | 21:00 |
| cloudnull | nice! | 21:00 |
| kysse | the way how openstack handlers erros from e.g. libvirt is super idiotic | 21:02 |
| kysse | might cause accidental deletion of instance root disk etc. | 21:03 |
| mnaser | kysse: what happened? | 21:04 |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_hosts stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590877 | 21:04 |
| kysse | https://github.com/openstack/nova/commit/ee759f62b41a8afd7ac88b4e4a20d31d7959c12f | 21:04 |
| kysse | this commit still exists on master branch | 21:05 |
| kysse | like 'instance fails to start after node reboot, lets remove its root disk and stuff, yey' | 21:07 |
| mnaser | i've never encountered that behaviour | 21:08 |
| kysse | you can reproduce it by blocking connections to neutron and restarting node, or symlinking qemu to devnull or something. | 21:10 |
| *** KeithMnemonic has quit IRC | 21:11 | |
| *** asettle has quit IRC | 21:14 | |
| bgmccollum | mnaser cloudnull -- if you have a moment https://review.openstack.org/#/c/593151/ | 21:16 |
| *** schwicht has quit IRC | 21:17 | |
| *** schwicht has joined #openstack-ansible | 21:19 | |
| *** schwicht has quit IRC | 21:19 | |
| *** madorn has quit IRC | 21:22 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add FS detection in metricbeat for containers https://review.openstack.org/593196 | 21:25 |
| cloudnull | ok gotta run , bbl | 21:27 |
| cloudnull | take care all | 21:27 |
| bgmccollum | adios | 21:27 |
| cloudnull | if folks have a moment https://review.openstack.org/#/q/starredby:cloudnull+status:open,n,z , i'd love some more reviews on those | 21:28 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible master: Ensure package cache is updated for nspawn containers https://review.openstack.org/588430 | 21:29 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590868 | 21:29 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590859 | 21:30 |
| *** hamzaachi has quit IRC | 21:38 | |
| bgmccollum | Seen this "Add GPG key for COPR LXC repo" error a few times, seems to be flaky -- http://logs.openstack.org/48/592048/1/gate/openstack-ansible-functional-centos-7/2af1847/logs/ara-report/result/dcd6cc6c-8bb7-4af5-b8ed-39a589fe43b5/ | 21:41 |
| mnaser | yeah thats something i meant to fix at some point | 21:41 |
| mnaser | http://logs.openstack.org/78/590778/7/gate/openstack-ansible-deploy-aio_metal-opensuse-423/a4388cf/logs/host/libvirt/qemu/instance-00000001.log.txt.gz | 21:41 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Revert "Revert "Work around nested virt issues on OVH test nodes"" https://review.openstack.org/593204 | 21:46 |
| *** hamzaachi has joined #openstack-ansible | 21:48 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Revert "Revert "Run with KVM if we can in tests"" https://review.openstack.org/593205 | 21:50 |
| *** cshen has quit IRC | 21:59 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590917 | 21:59 |
| *** nicolasbock has quit IRC | 21:59 | |
| bgmccollum | headed home... | 22:04 |
| openstackgerrit | Merged openstack/openstack-ansible-os_designate stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590914 | 22:10 |
| *** hamzaachi has quit IRC | 22:18 | |
| *** mmalchuk has joined #openstack-ansible | 22:36 | |
| *** mmalchuk_ has quit IRC | 22:38 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift stable/rocky: Update UPPER_CONSTRAINTS_FILE for stable/rocky https://review.openstack.org/590971 | 22:54 |
| *** macza has quit IRC | 22:56 | |
| *** spatel has joined #openstack-ansible | 23:10 | |
| *** spatel has quit IRC | 23:15 | |
| *** spatel has joined #openstack-ansible | 23:22 | |
| spatel | cloudnull: you there? | 23:23 |
| spatel | I wanted to understand this error "ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(104, 'Connection reset by peer'))") [SQL: u'SELECT 1'] " | 23:24 |
| spatel | do you think this can be ignore? | 23:24 |
| *** luksky has quit IRC | 23:25 | |
| *** schwicht has joined #openstack-ansible | 23:40 | |
| bgmccollum | well that's a bit slow, to the point the async task to check up on it fails -- http://logs.openstack.org/51/593151/1/gate/openstack-ansible-functional-ubuntu-bionic/cff0f41/logs/host/aria2c-image-prestage.log.txt.gz | 23:40 |
| *** sawblade6 has joined #openstack-ansible | 23:48 | |
| *** spatel has quit IRC | 23:52 | |
| *** sawblade6 has quit IRC | 23:53 | |
| openstackgerrit | ziyu proposed openstack/openstack-ansible-os_nova master: Add the project source code repository in README https://review.openstack.org/584224 | 23:54 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
