Monday, 2019-01-21

*** markvoelker has joined #openstack-ansible00:16
*** fatdragon has quit IRC00:42
*** orderchaos has quit IRC00:47
*** markvoelker has quit IRC00:49
*** partlycloudy has joined #openstack-ansible01:09
*** bgmccollum has quit IRC01:21
*** bgmccollum has joined #openstack-ansible01:23
*** DanyC has joined #openstack-ansible01:23
*** DanyC has quit IRC01:28
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: modify playbook inventory to make groups more specific  https://review.openstack.org/63166801:41
*** markvoelker has joined #openstack-ansible01:46
*** bgmccollum has quit IRC01:50
openstackgerritzhongshengping proposed openstack/openstack-ansible-tests master: Dissuade .gitignore references to personal tools  https://review.openstack.org/60922901:51
*** markvoelker has quit IRC02:19
*** mkuf_ has quit IRC02:40
*** markvoelker has joined #openstack-ansible03:16
*** aedc has quit IRC03:24
*** shyamb has joined #openstack-ansible03:39
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Allow skydive keystone service setup from an alternate host  https://review.openstack.org/63199603:39
cloudnullguilhermesp ah that's a bummer.03:40
cloudnullis that something we should address in the role? I would suspect so but maybe the intention was to take it another way? I'm not really sure. -cc odyssey4me03:44
*** markvoelker has quit IRC03:48
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Allow skydive keystone service setup from an alternate host  https://review.openstack.org/63199603:49
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Allow skydive keystone service setup from an alternate host  https://review.openstack.org/63199603:50
*** udesale has joined #openstack-ansible04:00
*** udesale has quit IRC04:00
*** udesale has joined #openstack-ansible04:01
*** shyamb has quit IRC04:28
*** spsurya has joined #openstack-ansible05:41
*** radeks__ has joined #openstack-ansible05:49
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin  https://review.openstack.org/63202105:54
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use the correct heat tests  https://review.openstack.org/63069505:55
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_heat master: [DNM] heat tempest tests finding  https://review.openstack.org/63069405:56
*** markvoelker has joined #openstack-ansible06:16
*** vnogin has joined #openstack-ansible06:27
chandankumarjrosser: odyssey4me Good morning06:31
chandankumarjrosser: odyssey4me  Count Pass Fail Error Skip View06:32
chandankumartempest.scenario.test_server_basic_ops.TestServerBasicOps06:32
chandankumaris always giving ssh timedout06:32
chandankumaronly on centos jobs06:32
chandankumarI am not sure this is the reason http://logs.openstack.org/08/631708/2/check/openstack-ansible-functional-centos-7/957c2a4/logs/openstack/openstack1/neutron/neutron-server.log.txt.gz?level=ERROR06:33
chandankumarplease have a look, thanks :-)06:33
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update skydive is using TLS  https://review.openstack.org/63202706:37
*** xenefix has quit IRC06:42
*** adrianreza has quit IRC06:42
*** mrhillsman has quit IRC06:42
*** corvus has quit IRC06:42
*** johnsom has quit IRC06:42
*** gundalow has quit IRC06:42
*** tacco has quit IRC06:42
*** ebbex has quit IRC06:42
*** johnsom has joined #openstack-ansible06:42
*** mrhillsman has joined #openstack-ansible06:42
*** gundalow has joined #openstack-ansible06:42
*** adrianreza has joined #openstack-ansible06:43
*** shyamb has joined #openstack-ansible06:45
*** mkuf_ has joined #openstack-ansible06:47
*** ebbex has joined #openstack-ansible06:47
*** markvoelker has quit IRC06:48
*** shyam89 has joined #openstack-ansible06:51
*** vnogin has quit IRC06:52
*** mkuf_ has quit IRC06:55
*** shyamb has quit IRC06:55
*** mkuf has joined #openstack-ansible06:55
*** vnogin has joined #openstack-ansible06:56
*** vnogin has quit IRC07:07
*** shyam89 has quit IRC07:08
*** shyam89 has joined #openstack-ansible07:10
*** shyam89 has quit IRC07:14
*** shyam89 has joined #openstack-ansible07:15
*** shyam89 has quit IRC07:21
*** vnogin has joined #openstack-ansible07:23
*** vnogin has quit IRC07:27
*** hamzaachi has joined #openstack-ansible07:32
*** markvoelker has joined #openstack-ansible07:46
*** kopecmartin|off is now known as kopecmartin07:47
*** hamzaachi has quit IRC08:16
*** markvoelker has quit IRC08:19
*** vnogin has joined #openstack-ansible08:23
*** hamzaachi has joined #openstack-ansible08:23
*** vnogin has quit IRC08:24
*** vnogin has joined #openstack-ansible08:25
*** gkadam has joined #openstack-ansible08:34
*** tosky has joined #openstack-ansible08:36
*** flaviosr has joined #openstack-ansible08:37
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: Bump SHAs for master  https://review.openstack.org/63043008:39
*** hamzaachi has quit IRC08:40
*** electrofelix has joined #openstack-ansible08:43
*** rgogunksiy has joined #openstack-ansible08:58
*** hamzaachi has joined #openstack-ansible08:59
*** kyblik_ has joined #openstack-ansible09:00
kyblik_ hello. we're having problems deploying OpenStack Rocky using openstack-ansible in heterogeneous environment (some compute nodes have ubuntu 16.04 while others 18.04). the problem is that venv packages are being built only for 16.04. Is this setup unsupported or am I missing something in OSA configuration? thanks.09:00
*** hamzaachi has quit IRC09:00
*** hamzaachi has joined #openstack-ansible09:00
*** hamzaachi has quit IRC09:06
*** markvoelker has joined #openstack-ansible09:16
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Always generate stackviz irrespective of tests pass or fail  https://review.openstack.org/63196709:16
*** vnogin has quit IRC09:20
ionikyblik_, i don't think mixing the versions are supported but maybe i'm wrong. In the past wasn't supported09:20
*** DanyC has joined #openstack-ansible09:23
*** shardy has joined #openstack-ansible09:24
*** dcdamien has joined #openstack-ansible09:31
*** shyam89 has joined #openstack-ansible09:38
jrosserkyblik_: in that situation i think you need to have a repo server deployed on each OS release, then you should get everything built for both releases09:45
kyblik_jrosser: hm, that seems reasonable. you think that would be enough?09:48
jrosserthere are some etherpads somewhere which give details of how to systematically upgrade a deployment from one OS to another, iirc bringing up an infra node (and hence repo) on the new OS is one of the first steps09:49
*** markvoelker has quit IRC09:49
jrosserthe repo servers synchronise their contents between themselves so you should find that all of them can serve packages/venv for all releases once the build/sync has happened09:50
CeeMacmorning all10:03
*** ArchiFleKs has quit IRC10:07
*** ArchiFleKs has joined #openstack-ansible10:10
CeeMacI'm having some weird (possibly python related) issues with one of my hosts10:11
CeeMacits built the same as the others and is same hardware but for the containers on this host the containers-deploy.yml is failing at the "lxc_container_create : Gather container facts" task10:11
CeeMacwith what looks like a python error10:12
CeeMachttp://paste.openstack.org/show/742971/10:12
CeeMacanyone seen this before?10:12
CeeMacbrb10:12
*** shyam89 has quit IRC10:17
odyssey4meCeeMac yep, what's changed is that new builds of ubuntu xenial no longer have the package 'python' installed - so ansible can't find the python runtime any more10:18
odyssey4meCeeMac this should take care of that though? https://github.com/openstack/openstack-ansible/blob/master/playbooks/openstack-hosts-setup.yml#L22-L3710:18
*** openstackgerrit has quit IRC10:21
*** shyam89 has joined #openstack-ansible10:23
kyblik_jrosser: and what would be the process of upgrading whole environment to ubuntu bionic? execute ./scripts/run-upgrade.sh to upgrade Queens to Rocky (still being on 16.04), then create mysqldump, reinstall backend nodes to 18.04, deploy Rocky and import DB and then install infra nodes ?10:25
chandankumarodyssey4me: Hello10:28
chandankumarodyssey4me: this test http://logs.openstack.org/67/631967/2/check/openstack-ansible-functional-distro_install-centos-7/5d8d3a5/logs/openstack/tempest1/stestr_results.html  tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops is always failing on centos , please have a look10:28
chandankumarodyssey4me: I am not sure why it is happening10:28
chandankumarssh timedout10:29
jrosserkyblik_: i don't really have a pointer to well tested instructions for this - there is an etherpad here with ad-hoc notes from some folks who have used 'do-release-upgrade' https://etherpad.openstack.org/p/osa-rocky-bionic-upgrade10:29
odyssey4mekyblik_ yes, you'd upgrade to rocky first to ensure a uniform version that supports both xenial and bionic - then you'd replace a single control plane host to ensure that the packages for both are built10:29
admin0CeeMac, yes .. apt install python on the host10:30
odyssey4methen rolling replace a second control plane host (leave the third xenial host running until all nodes are converted)10:30
odyssey4mekyblik_ There are some notes here - please add to them: https://etherpad.openstack.org/p/osa-rocky-bionic-upgrade10:30
kyblik_jrosser, odyssey4me: thanks, I'll definitely have a look!10:31
kyblik_odyssey4me: in case I wanted to avoid do-dist-upgrade, do you think that mysqldump+DB import after backend node reinstall is viable?10:31
odyssey4mekyblik_ it won't be necessary because mariadb/galera will pull the data from the other running nodes10:32
kyblik_odyssey4me: ah, so rolling upgrade (reinstall respectively) in case of backend nodes too.10:35
kyblik_thanks10:35
CeeMacback10:35
odyssey4mekyblik_ yep10:35
CeeMacadmin0, its already installed10:35
CeeMachttp://paste.openstack.org/show/743031/10:36
admin0i think its looking for python in the image10:36
admin0container10:37
admin0seen this before .. i had to nuke all containers and had to start again10:37
CeeMacodyssey4me, yeah I didn't get any errors on the openstack-hosts-setup.yml or security-hardening.yml playbooks10:37
CeeMacalthough i did get the LXC cache prep timeout10:37
odyssey4meCeeMac interesting - can you confirm whether the 'python' package is installed on that host?10:37
CeeMacyep ^^10:38
CeeMacadmin0, as in *all* of the containers? I tried destoying these containers and removing them from the inventory, but it came back with the same issue10:38
admin0i think the fault is in the machine image downloaded10:39
CeeMacits worked fine on the other hosts that are running containers10:39
odyssey4mehmm, python is in the package list for the cache prep: https://github.com/openstack/openstack-ansible-lxc_hosts/blob/stable/rocky/vars/ubuntu-16.04.yml#L4810:39
CeeMacnot sure if its related to the lxc-cache prep timeout10:39
CeeMacah, so it could be that10:40
odyssey4meit coule be that the prep just took too long and didn't complete10:40
CeeMaci upped the timout from 120 to 270, might tweak it a bit higher10:40
CeeMacrerunning the playbook didnt try and recreate the cache though10:40
CeeMacmaybe i'll destroy these containers and roll back the lvm snapshot then rerun the playbook10:40
admin0if the fault is in the original cache, all new containers will have the same10:41
CeeMacor is there a way to scrub the cache manually to force it to reprep?10:41
*** openstackgerrit has joined #openstack-ansible10:42
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder stable/rocky: Add possibility to have private cinder volume types  https://review.openstack.org/63206210:42
ioniis there a way to remove a specific host from a group?10:43
admin0quick question: how do you guys select between ovs or linuxbridge ? is it safe to say that you only need ovs if you are doing things like NFV etc ? for general purpose computing, LB satisfies all of the tenant based models10:44
*** DanyC has quit IRC10:44
*** DanyC_ has joined #openstack-ansible10:44
odyssey4meioni I guess that depends - but typically remove it from openstack_user_config and remove it using inventory-manage10:46
*** markvoelker has joined #openstack-ansible10:46
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin  https://review.openstack.org/63202110:48
ioniodyssey4me, i don't want to remove it from everything but for example i have a compute node that ended up as network host10:51
ioniodyssey4me, i guess it should be fine to remove it and re-run setup hosts and it will end up in the right groups10:51
ionino?10:51
odyssey4meioni ok, and have you figured out how that went wrong?10:51
ioniodyssey4me, nope, i think i messed up with the configs and forgot about it, i guess10:52
odyssey4meioni I think you just have to figure out the openstack_user_config or conf.d changes to prevent it being in the right group - make the changes necessary, then run the dynamic inventory script: /opt/openstack-ansible/inventory/dynamic_inventory.py10:53
ioniok10:53
ionithanks10:53
admin0ioni, i have opposite of what you are doing .. in our case, all our compute nodes are also network hosts by design10:54
ioniadmin0, i think we talked about that in the past10:55
admin0yeah10:55
*** asettle has joined #openstack-ansible11:02
admin0if i need to throw away a service, how do I do that ?  deleting the container is not enough right as it will re-create it again .. i want to throw away gnocchi, aodh and ceilometer that i added earlier11:07
admin0if i remove from the inventory and # the service, i have seen it recreate again11:07
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Always generate stackviz irrespective of tests pass or fail  https://review.openstack.org/63196711:08
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Always generate stackviz irrespective of tests pass or fail  https://review.openstack.org/63196711:09
*** shyam89 has quit IRC11:10
chandankumarodyssey4me: I need to access demo user password I am not sure what is the best way to do it here https://review.openstack.org/#/c/632021/2/templates/tempest.conf.j2@18611:11
odyssey4mechandankumar commented in review11:17
odyssey4meadmin0 remove it from openstack_user_config, then use inventory-manage to remove them from the persistant inventory11:17
*** markvoelker has quit IRC11:19
*** udesale has quit IRC11:20
kyblik_jrosser: regarding repo server per OS release - any idea how to actually add those new repo servers?11:21
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin  https://review.openstack.org/63202111:21
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin  https://review.openstack.org/63202111:23
jrosserkyblik_: if you are going to reboot/re-install from scratch one of your infra nodes from xenial->bionic that wil just happen for you, one existing repo server will come back as bionic when you run the setup-... playbooks against the fresh host11:23
kyblik_yeah, although I'm not sure that having one infra node on xenial and another on bionic is viable in long term11:37
jrossersure - but the intention is to roll it all over to bionic?11:38
kyblik_jrosser: well, eventually. we have two groups of compute nodes - one which we wanted to upgrade to bionic+rocky using full reinstall and the other just in-place upgrade from queens to rocky. so the latter group would stay on xenial+rocky for undefined period of time11:40
jrosserthere is enough flexibility to do it however you like :) you can fiddle your openstack_user_config to create a (fourth?) repo container on an entirely new host if thats preferable11:42
jrosserprobably something like splitting repo-infra_hosts out into it's own section in openstack_user_config.yml https://github.com/openstack/openstack-ansible/blob/master/inventory/env.d/pkg_repo.yml11:43
odyssey4mekyblik_ you'd need at least one infra host on bionic and one on xenial for as long as you have a mix of those for your computes11:43
kyblik_jrosser: yeah, thanks, I'll look into it11:45
kyblik_odyssey4me: I was thinking about creating new repo containers based of bionic image but running it on xenial infra host11:46
odyssey4mekyblik_ that is possible, but not simple11:46
jrosserthere is a ton of stuff copied from the host into the containers, apt keys and all sorts11:47
*** shyam89 has joined #openstack-ansible11:48
*** noonedeadpunk has joined #openstack-ansible11:52
noonedeadpunkmorning folks11:56
*** ArchiFleKs has quit IRC12:06
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use tempest_cloud_name in tempestconf  https://review.openstack.org/63170812:07
chandankumararxcruz|ruck: ^^ need your feedback on this12:07
odyssey4mejrosser could you take a look at https://review.openstack.org/624773 some time this week?12:12
jrosserodyssey4me: ah yes - will do12:12
arxcruz|ruckchandankumar: hey, what do you need? i'll wait the zuul result to check, sorry, i'm rucking this week, too many things :)12:14
*** markvoelker has joined #openstack-ansible12:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove unused openstack_user_config file  https://review.openstack.org/63072812:16
odyssey4mearxcruz|ruck what does it mean 'to ruck'?12:16
arxcruz|ruckodyssey4me: i'm handling all the bullshit happening on tripleo-ci :D12:17
arxcruz|ruckodyssey4me: so the other people can focus on the sprint without care about bugs and issues on our infra12:17
odyssey4mearxcruz|ruck so that's 'ruck' as in the rugby term?12:17
chandankumararxcruz|ruck: I am removing this keystone_service_internaluri sepcific vars and making them to use only clouds.yaml for generating tempest.conf otherwise it gives undefined errors12:17
arxcruz|ruckodyssey4me: yup12:17
odyssey4meaha, ok cool thanks12:17
arxcruz|ruckodyssey4me: oh, sorry, australian football12:18
arxcruz|ruckwe have a ruck and rover, one filter the bugs, the other fix the bugs12:18
arxcruz|ruckso the rest of the team can focus on the sprint12:18
odyssey4meoh that's interesting12:18
odyssey4mehttps://en.wikipedia.org/wiki/Follower_(Australian_rules_football)12:20
*** ArchiFleKs has joined #openstack-ansible12:25
*** udesale has joined #openstack-ansible12:26
arxcruz|ruckyup :)12:35
chandankumarodyssey4me: need help here http://logs.openstack.org/67/631967/4/check/openstack-ansible-functional-centos-7/c2a419d/logs/openstack/tempest1/stestr_results.html12:36
chandankumarodyssey4me: scenario smoke test is failing with ssh timedoit12:36
chandankumar*timeout12:36
odyssey4mechandankumar ok, it looks like the fails are centos7 only?12:37
chandankumarodyssey4me: yes12:37
chandankumarodyssey4me: it is failing continously today12:38
chandankumarodyssey4me: from here http://zuul.openstack.org/builds?job_name=openstack-ansible-functional-centos-712:38
chandankumarit is failing frequently12:39
odyssey4mechandankumar maybe something is wrong in https://review.openstack.org/592596 - that was the last network-related change to merge for role tests12:39
CeeMacsee if I'm re-running containers-deploy.yml and want to target a specific host and its containers, I can use the actual hostname in --limit rather than lxc_hosts?12:41
CeeMacand the container groups obvs12:42
odyssey4mechandankumar I think we may need mnaser or cloudnull to help there, given they know that domain of work... I unfortunately do not12:42
chandankumarcloudnull: hello12:43
odyssey4meCeeMac yes, group names of inventory names are fine - if you target containers, be sure to include their hosts so that host facts can be sourced12:43
CeeMaccool thanks odyssey4me12:43
odyssey4meCeeMac I typically use lxc_hosts,<service_group> for limits12:44
chandankumarcloudnull: please have a look at this failure http://logs.openstack.org/67/631967/4/check/openstack-ansible-functional-centos-7/c2a419d/logs/openstack/tempest1/stestr_results.html it is failing consitently, it might appears that this changes caused it https://review.openstack.org/#/c/592596/12:44
odyssey4meeg: lxc_hosts,designate_all12:44
chandankumarcloudnull: thanks :-)12:44
admin0my queens -> rocky fails at this step:   openstack-ansible rabbitmq-install.yml -e 'rabbitmq_upgrade=true'     -- logs   https://pastebin.com/4VVdm610    .. checking if anyone has seen this before12:44
odyssey4meadmin0 yep, https://github.com/openstack/openstack-ansible/commit/2b9a3b191cb39be508bbda5a6cd21b747675626f12:44
CeeMacodyssey4me, yep, got you.  I had done that previously when i re-checked this issue on friday and it polls all of the container hosts (as you'd expect)12:45
admin0do i need to rerun the bootstrap again after making this change ?12:45
admin0or is there a way to bootstarp for this role only12:46
CeeMacwasn't sure if would work to just use the specific hostname,12:46
chandankumarodyssey4me: thanks!12:46
odyssey4meadmin0 you'd need to, yes - because it updates one of the roles12:46
odyssey4meadmin0 or you could just checkout the new SHA in that role12:46
admin0odyssey4me, thanks .. the checkout sounds better12:47
*** ansmith has quit IRC12:49
openstackgerritMerged openstack/ansible-hardening stable/rocky: cleanup: use updated conditionals  https://review.openstack.org/62915812:49
*** markvoelker has quit IRC12:49
admin0odyssey4me, this worked .. this will make its way in 18.1.3 i guess .12:51
odyssey4meadmin0 yep12:52
jamesdentonmornin'12:53
chandankumarodyssey4me: http://logs.openstack.org/21/632021/4/check/openstack-ansible-functional-centos-7/89de395/logs/ara-report/result/a1e0b34c-11ab-4d77-b3e0-746b6acffab6/12:54
chandankumarodyssey4me: with heat temepst.conf patch .name is not working12:54
chandankumarodyssey4me: https://review.openstack.org/#/c/632021/12:54
*** asettle has quit IRC12:58
odyssey4mechandankumar replied in review12:58
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin  https://review.openstack.org/63202112:58
*** shyam89 has quit IRC13:06
*** shyam89 has joined #openstack-ansible13:07
*** strobelight has joined #openstack-ansible13:07
openstackgerritMichael Vollman proposed openstack/openstack-ansible-os_manila master: Basic working os_manila role  https://review.openstack.org/61193013:32
*** priteau has joined #openstack-ansible13:40
*** meeeks has joined #openstack-ansible13:48
*** rgogunksiy has quit IRC13:53
*** sum12 has quit IRC13:56
*** sum12 has joined #openstack-ansible13:56
CeeMacwhats the highest anyone has had to set lxc_cache_prep_timout to ?13:57
*** ansmith has joined #openstack-ansible13:57
jrosserCeeMac: is this on hard drives rather than ssd?13:59
CeeMacsadly, yes14:00
CeeMacthey're larger HHDs on this host than others, but thought they were the same RPM14:00
CeeMac2700 was enough for the other hosts cache prep to complete. just tried this one on 3200 - no dice.14:01
CeeMacfigured i'll try 5400 next for the giggles14:01
meeeksHi, I'm having some troubles deploying openstack with the LVM driver. The playbook fails on TASK [Perform online data migrations] and looking at the logs from the cinder-api container I get the following error: "Error attempting to run shared_targets_online_data_migration: MessagingTimeout: Timed out waiting for a reply to message ID". I've probably missed something but can't find it:P14:02
meeeksCeeMac: I had to increase it to 2400 for it to complete. There was also some errors where it crashed on configuring ca-certificates because the /tmp/user/0 directory didn't exist.14:04
CeeMachi meeeks, thanks, i've past that already, failed at 3200 :(14:05
*** udesale has quit IRC14:05
*** udesale has joined #openstack-ansible14:06
openstackgerritMerged openstack/openstack-ansible stable/rocky: Ensure that control plane hosts are also updated correctly  https://review.openstack.org/62589814:07
meeeksCeeMac: aah i missed that you said that:)14:07
CeeMaceasy done :)14:08
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-ops master: Disable opencontrail build option for skydive  https://review.openstack.org/63209614:14
jrosserCeeMac: this is a relatively trivial patch that you might want to try out https://review.openstack.org/#/c/621509/14:19
*** shyam89 has quit IRC14:19
CeeMacjrosser, looks good.  How? >_>14:21
jrosseron your deploy host, go to /etc/ansible/roles/....14:21
CeeMacyep14:22
jrosserinto the lxc_hosts role, thats where that patch applies14:22
jrosserthen on the gerrit page there is a little drop down top right for "Download"14:22
CeeMacwith you so far14:22
jrosserpress "copy to clipboard" at the end if the "cherry pick" line14:22
jrosserand paste that into your terminal in the lxc_hosts repo14:23
CeeMacgot it14:23
CeeMacok, thanks14:23
jrossercool - interested to see if it helps, it certainly did on an esxi setup with hdd14:23
CeeMacI'll let you kow14:24
CeeMac*know14:24
CeeMacjust rolling back the lvm snapshot again14:24
CeeMacI'll push up the timeout as well, for belts and braces, just incase14:25
cloudnullmornings14:45
jrossero/ hello14:45
*** shardy has quit IRC14:46
*** shardy has joined #openstack-ansible14:46
evrardjpthere is a meeting at 4PM utc for the upgrades SIG in #openstack-meeting-4, should we want to discuss upgrades expectations.14:48
evrardjpcloudnull: it's maybe not a bad idea to bring ppl of your team there? ^14:48
cloudnullI can ask around.14:48
cloudnullits a rare US holiday today, so not a lot of folks around .14:49
* cloudnull will be there 14:49
dcdamieno/14:53
dcdamienhttps://review.openstack.org/#/c/630371/ stalled a little ^^14:54
*** strattao has joined #openstack-ansible14:55
*** dave-mccowan has joined #openstack-ansible14:55
openstackgerritMerged openstack/openstack-ansible-ops master: Disable opencontrail build option for skydive  https://review.openstack.org/63209614:58
*** kyblik_ has quit IRC15:04
chandankumarcloudnull: \o/15:04
cloudnullo/15:05
chandankumarcloudnull: http://logs.openstack.org/08/631708/3/check/openstack-ansible-functional-centos-7/0e4a757/logs/openstack/tempest1/stestr_results.html one of the scenario test is failing continously on centos  7 jobs15:05
chandankumarwith ssh timedout please have a look15:06
cloudnullhttps://pasted.tech/pastes/87a45ba081b475a9772d24ee51465cc756f9b95d - this error ?15:08
dmsimardodyssey4me, cloudnull: I don't have good news for the ceph ansible roles that were deleted: https://github.com/ceph/ceph-ansible/issues/350815:09
chandankumarcloudnull: yes15:09
odyssey4medmsimard no worries, thanks for trying - I believe that cjloader is working on transitioning the ocata/pike branches to use the ceph-ansible repo15:10
cloudnulldmsimard :'(15:10
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_neutron master: Use the new services names for sfc  https://review.openstack.org/62221615:10
dmsimardapparently they were deprecated on galaxy15:10
cloudnullchandankumar is it always just the cinder tests?15:10
dmsimarddo we have a way to tell if there are any roles that we are pulling that are deprecated ?15:11
mgariepycan someone review this one please : https://review.openstack.org/#/c/630996/15:11
dmsimardI haven't worked with the actual galaxy much15:11
odyssey4medmsimard we don't use galaxy for any of our bits15:11
cloudnullmost of the roles we pull come from git using a given sha/checkout15:11
dmsimardrequirements.yml is galaxy though, right ? I guess you use git as source15:11
dmsimardyeah15:11
chandankumarcloudnull: it is not cinder tests , it is tmepest scenario test  tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops15:11
*** dave-mccowan has quit IRC15:12
dmsimardodyssey4me, cloudnull: while I have your attention, are there any jobs running full py3 ? I wanted to do some early testing with the new version of ara but it's failing to install with py215:13
dmsimard^ https://review.openstack.org/#/c/631262/15:13
odyssey4mecloudnull it may have to do with https://review.openstack.org/592596 given that merged recently15:13
*** szaher has quit IRC15:13
odyssey4medmsimard no, not yet unfortunately - I'm behind on my work to transition us in that direction... :/15:13
chandankumarcloudnull: failed at this part https://github.com/openstack/tempest/blob/master/tempest/scenario/test_server_basic_ops.py#L5015:13
dmsimardodyssey4me: that's a fair answer, is it a target for stein ?15:14
odyssey4medmsimard yes, I am hoping to still make it in time for stein's release15:14
cloudnullodyssey4me that could be.15:15
dmsimardodyssey4me: ok, I'll check back later. Good luck :)15:15
cloudnullchandankumar is that basic-ops error happening all the time?15:15
odyssey4mecloudnull for role tests that use that test, yes15:16
odyssey4meand only happening since that merge as far as I can see15:16
odyssey4methere's probably just a route missing or osmething15:16
cloudnullOK. i'll spin a test node to see if we can nail down whats up15:16
chandankumarcloudnull: it is happening today only on centos 7 jobs http://zuul.openstack.org/builds?job_name=openstack-ansible-functional-centos-715:16
chandankumarboth on distro as well as source jobs15:17
cloudnullOh ok. so suse and ubuntu and passing ol ?15:17
cloudnull**ok15:17
odyssey4meoh yes, it's centos only - so it may be worth looking at the extra things we had to do for centos for the AIO tests15:17
*** szaher has joined #openstack-ansible15:17
* cloudnull shakes fist at centos 15:17
cloudnullisn't openstack-ansible-functional-centos-7 testing the integrated gate?15:19
cloudnullI think thats testing the role but with the integrated repo?15:20
*** strattao has quit IRC15:23
*** strattao has joined #openstack-ansible15:26
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use tempest_cloud_name in tempestconf  https://review.openstack.org/63170815:33
odyssey4mecloudnull no it's not - it's using the role test facility15:36
cloudnullok15:36
odyssey4methe move to use the integrated gate is not complete yet15:36
cloudnullOK, just checking before I go down this rabbit hole :)15:37
chandankumarcloudnull: odyssey4me one of my dnm patch against heat it is passing http://logs.openstack.org/94/630694/4/check/openstack-ansible-functional-centos-7/b4542ec/15:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Use the correct heat tests  https://review.openstack.org/63069515:41
*** aedc has joined #openstack-ansible15:42
CeeMacjrosser: I blown away by that patch15:43
CeeMaconly took 10 retries and its rattled straight through!15:43
*** TxGirlGeek has joined #openstack-ansible15:43
jrosserCeeMac: fsync really hurts with hdd. every file from every package installed into the LXC image will be fsync'd without that patch15:49
jrosserCeeMac: it totally destroys any attempt your hdd was making to queue the writes in an intelligent way, and everything ends up in a sad mess of drive seeks15:49
odyssey4meGood find that jrosser :)15:50
jrosseri think you don't really see it so much on ssd/nvme becasue there is no real seek time, but it will still be quicker15:50
CeeMacjrosser: you're not wrong15:52
CeeMacthat was painful before15:53
CeeMacI can't believe how much of a difference that made15:53
CeeMacis that going to be merged, or will it always be an optional patch?15:54
jrosser:) - Excellent - glad it helped15:54
CeeMachelped both the project and my sanity! :D15:54
jrosserodyssey4me: should we push the eatmydata patch back to rocky?15:55
openstackgerritMerged openstack/openstack-ansible-os_designate stable/rocky: Set project that owns managed resources  https://review.openstack.org/63037115:56
odyssey4mejrosser sure, why not15:56
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Use libeatmydata during LXC cache prep  https://review.openstack.org/63211815:56
jrosserCeeMac: ^ there you go - when that merges and it gets picked up on a rocky release tag, it'll be in by default15:56
*** dcdamien has quit IRC15:58
jrosserCeeMac: on previous projects i did tons of building debian packages in pbuilder chroots - learned the value of libeaymydata there where it made a huge difference to the runtimes15:59
*** macza has joined #openstack-ansible15:59
*** udesale has quit IRC16:01
*** bgmccollum has joined #openstack-ansible16:05
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-ops master: Remove stickiness config from traefik backends  https://review.openstack.org/63212216:06
CeeMacnice, thanks jrosser16:06
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest master: Add telemetry distro plugin install for aodh  https://review.openstack.org/63212516:17
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_aodh master: Enable overriding the service setup host python interpreter  https://review.openstack.org/62118316:17
*** TxGirlGeek has quit IRC16:19
FrankZhanghey morning guys, anyone of you have free cycle, please help review this one, thanks! https://review.openstack.org/#/c/617719/16:21
*** radeks_ has joined #openstack-ansible16:32
*** radeks__ has quit IRC16:35
chandankumarodyssey4me: https://review.openstack.org/#/c/632125/1/vars/redhat-7.yml for this one can we get something if ceilometer gnocchi aodh and panko is available then install telemetry-tempest-plugin?16:35
*** radeks_ has quit IRC16:44
*** gkadam has quit IRC16:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zun master: Improve docker API version retrieval  https://review.openstack.org/63213116:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_zun master: Enable overriding the service setup host python interpreter  https://review.openstack.org/62123916:47
*** priteau has quit IRC16:58
*** priteau has joined #openstack-ansible17:00
*** shardy has quit IRC17:15
*** shardy has joined #openstack-ansible17:17
*** errr has quit IRC17:19
*** kopecmartin is now known as kopecmartin|off17:21
*** shardy has quit IRC17:25
*** shardy has joined #openstack-ansible17:25
odyssey4mechandankumar yep, there's no panko var/plugin instrumented yet - so it can e added when that is done17:32
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added tempest.conf for heat_plugin  https://review.openstack.org/63202117:37
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Use the correct heat tests  https://review.openstack.org/63069517:38
*** marst has quit IRC17:51
*** logan- has quit IRC17:52
FrankZhanghttps://review.openstack.org/#/c/617719/ zuul gate all jobs failed, can we only recheck these gating? what comment should I use?17:52
*** errr has joined #openstack-ansible17:53
FrankZhang" functional installed: ----------------------------------------,Error when trying to get requirement for VCS system Command "git config --get-regexp remote\..*\.url" failed with error code 1 in /home/zuul/src/git.openstack.org/openstack/openstack-ansible-os_tempest, falling back to uneditable format,Could not determine repository location of17:53
FrankZhang/home/zuul/src/git.openstack.org/openstack/openstack-ansible-os_tempest,## !! Could not determine repository location,openstack-ansible-os-tempest==18.0.0.0b2.dev138"17:53
*** logan_ has joined #openstack-ansible17:53
FrankZhanghttp://logs.openstack.org/19/617719/2/gate/openstack-ansible-functional-centos-7/a88f6cb/job-output.txt.gz#_2019-01-21_16_34_05_32856917:53
*** logan_ is now known as logan-17:53
*** Darcidride_ has joined #openstack-ansible17:53
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/pike: Fix ceph-ansible role-requirements  https://review.openstack.org/63214217:54
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/pike: Fix ceph-ansible role-requirements  https://review.openstack.org/63214217:57
cjloaderodyssey4me ^17:57
cjloaderlooks like has same inner behavior17:58
cjloaderjust needed clone of the broader repo17:58
*** priteau has quit IRC18:01
*** Darcidride_ has quit IRC18:01
*** Darcidride_ has joined #openstack-ansible18:02
*** Darcidride_ has quit IRC18:04
*** Darcidride_ has joined #openstack-ansible18:04
*** shardy has quit IRC18:05
*** electrofelix has quit IRC18:05
*** Darcidride_ has quit IRC18:06
*** openstackgerrit has quit IRC18:07
*** Darcidride_ has joined #openstack-ansible18:07
*** openstackgerrit has joined #openstack-ansible18:09
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update skydive is using TLS  https://review.openstack.org/63202718:09
*** Darcidride_ has quit IRC18:09
*** Darcidride_ has joined #openstack-ansible18:10
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Allow skydive keystone service setup from an alternate host  https://review.openstack.org/63199618:10
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update skydive to use TLS  https://review.openstack.org/63202718:10
prometheanfirecloudnull: tls-1.2 for skydive?18:12
*** Darcidride_ has quit IRC18:12
cloudnullindeed.18:12
*** Darcidride_ has joined #openstack-ansible18:12
prometheanfiregut18:12
*** Darcidride_ has quit IRC18:14
*** Darcidride_ has joined #openstack-ansible18:15
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/pike: Fix ceph-ansible role-requirements  https://review.openstack.org/63214218:15
*** markvoelker has joined #openstack-ansible18:16
jrossercloudnull: on skydive does it look18:17
jrosserLike each of your hosts has a bogus switch hung off it with the name of the host?18:17
cloudnullyes18:22
jrosserOk interesting, I sent them a copy of my topology json, it maybe failing to ignore the hosts own lldp18:23
cloudnullhttps://i.snag.gy/gf6N0X.jpg18:24
jrosserPruning those extras off would really tidy it up18:24
cloudnull100%18:25
cloudnullits fugly in some of my envs18:25
jrosserYes that looks much like mine18:25
jrossercloudnull: I’m here now https://pasteboard.co/HXuWND9.png18:26
jrosserThe patch for nxos lldp has made it much more sensible18:26
cloudnullhum ill have to see that patch makes things better for me18:30
*** radeks has joined #openstack-ansible18:32
jrossercloudnull: it’ll join things up better with nxos switches, but won’t change anything otherwise18:34
*** DanyC_ has quit IRC18:40
*** DanyC has joined #openstack-ansible18:40
cloudnullhttps://i.snag.gy/iXw62k.jpg18:41
cloudnullthis is a fully exploded environment18:41
cjloaderCan I get some reviews https://review.openstack.org/#/c/632142/?18:41
nsmedstransformers, robots in disguise18:41
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Update skydive to use TLS  https://review.openstack.org/63202718:43
*** DanyC has quit IRC18:45
*** radeks has quit IRC18:47
*** markvoelker has quit IRC18:50
*** radeks has joined #openstack-ansible18:54
*** vnogin has joined #openstack-ansible18:57
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/pike: Fix ceph-ansible role-requirements  https://review.openstack.org/63214219:03
openstackgerritMerged openstack/openstack-ansible-ops master: Remove stickiness config from traefik backends  https://review.openstack.org/63212219:06
*** strattao has quit IRC19:16
-openstackstatus- NOTICE: The error causing post failures on jobs has been corrected. It is safe to recheck these jobs.19:16
*** strattao has joined #openstack-ansible19:23
openstackgerritMerged openstack/openstack-ansible-ops master: Allow skydive keystone service setup from an alternate host  https://review.openstack.org/63199619:24
openstackgerritJames Denton proposed openstack/openstack-ansible-os_neutron master: [WIP] Deploy Vector Packet Processing (VPP) Platform for Neutron  https://review.openstack.org/63164419:30
*** aedc has quit IRC19:35
*** markvoelker has joined #openstack-ansible19:46
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63215519:53
openstackgerritMerged openstack/openstack-ansible-ops master: Update skydive to use TLS  https://review.openstack.org/63202719:55
*** vnogin has quit IRC20:06
*** hamzaachi has joined #openstack-ansible20:07
*** DanyC has joined #openstack-ansible20:10
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/pike: Fix ceph-ansible role-requirements  https://review.openstack.org/63214220:17
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/pike: Fix ceph-ansible role-requirements  https://review.openstack.org/63214220:19
*** markvoelker has quit IRC20:20
*** macza has quit IRC20:43
*** macza has joined #openstack-ansible20:45
*** priteau has joined #openstack-ansible20:51
*** aedc has joined #openstack-ansible20:51
*** DanyC has quit IRC21:02
*** DanyC has joined #openstack-ansible21:02
*** priteau has quit IRC21:04
*** DanyC has quit IRC21:07
*** markvoelker has joined #openstack-ansible21:17
*** ansmith has quit IRC21:36
*** strattao has quit IRC21:48
*** markvoelker has quit IRC21:49
*** strattao has joined #openstack-ansible21:51
*** Darcidride_ has quit IRC21:55
*** strobelight has quit IRC21:56
*** DanyC has joined #openstack-ansible22:05
*** markvoelker has joined #openstack-ansible22:08
*** strattao has quit IRC22:08
*** spsurya has quit IRC22:09
*** radeks has quit IRC22:21
*** priteau has joined #openstack-ansible22:33
cjloaderneed another review for https://review.openstack.org/#/c/632142/ to merge22:34
openstackgerritCam J. Loader proposed openstack/openstack-ansible stable/ocata: Fix ceph-ansible role-requirements  https://review.openstack.org/63218222:48
*** macza has quit IRC23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!