Wednesday, 2017-11-22

odyssey4meso that task needs to be in the barbican role00:01
odyssey4meI'm suggesting that it's added to your existing patch00:02
odyssey4meobviously the keystone var ref needs to be adjusted to be barbican related00:02
odyssey4meit suits the existing patch considering that it's applicable to enabling centos deployment for barbican :)00:03
SmearedBeardyeah, I added and rerun, failed at the same part.. gonna destroy container and try again00:05
odyssey4meyeah, it'll only activate if the venv changed00:08
odyssey4meyou don't have to destroy the container - you can just remove the venv and the venv tarball, but destroying the container is probably simpler00:09
openstackgerritMerged openstack/openstack-ansible stable/newton: Retry apt cache update  https://review.openstack.org/52180700:16
SmearedBeardWahoo odyssey4me that all worked, ill update my patch as well00:17
odyssey4meSmearedBeard excellent :)00:17
odyssey4methanks for persisting!00:17
SmearedBeardthank you for all the help and guidance00:18
odyssey4meI suppose the lesson learned from all that is to lead with the actual symptom first, then discuss what you've looked at... we spent a good while thinking it was a repo build issue there. ;)00:19
odyssey4mehaha, I was just wondering whether we have bug report guidance... and: https://docs.openstack.org/openstack-ansible/latest/contributor/contribute.html#reporting-bugs00:21
openstackgerritAaron Mehar proposed openstack/openstack-ansible-os_barbican master: Add CentOS 7 deployment support  https://review.openstack.org/52177200:22
odyssey4methe trouble with having documentation is that it reaches the stage where figuring out how to lay it out in a useful and intuitive way becomes hard00:23
SmearedBeardyeah, I find troubleshooting is a real skill to have and learn that many people neglect :)00:23
odyssey4meyeah, finding the right audience is one of the hardest parts of doing docs00:24
SmearedBeardyeah, this is my first contribution to something and it I found myself going from page to page, site to site to look at guides, and bugs etc to make sure I was doing it right00:24
odyssey4meon the one hand you have people who can't seem to understand how to work through the steps, but on the other the steps are not always obvious00:25
odyssey4meon the one hand we want to make this a system where you really have a lot of choice - you can implement just about any architecture00:26
odyssey4mebut on the other hand, with that choice comes a matrix of possibilities which we can't possibly test the entire matrix for... so things fall through the cracks and we have to rely on humans to report issues00:27
odyssey4meadding more test paths adds more tests to maintain... and unfortunately the number of people who can triage the issue, much less submit a suggested fix, is limited00:28
odyssey4meso thank you for stepping up - you had an itch, and you figured out how to scratch it and shared it :)00:29
SmearedBeardI minimal and common use testing is fine IMO, and anything beyond can be done perhaps on a case-by-case or individually..00:30
SmearedBeardno worries, that is what opensource is all about :)00:30
SmearedBeardit's past midnight, Im off to bed00:30
*** chyka_ has quit IRC00:31
*** chyka has joined #openstack-ansible00:31
odyssey4meyeah, ultimately as a project we try to help out where we can - but we also have to manage our own tech debt and realise that it's those that contribute materially that drive the way the project goes00:31
odyssey4meit is past midnight, but my patch just worked... holy moley00:32
odyssey4mehaha, I'm rambling00:33
odyssey4mehave a good night's rest SmearedBeard - chat again tomorrow... thanks again00:33
*** chyka has quit IRC00:36
*** SmearedBeard has quit IRC00:39
openstackgerritMerged openstack/openstack-ansible stable/pike: Make pw-token-gen.py more random  https://review.openstack.org/52071400:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/pike: [DOCS] Correct secret uuid in Ceph example  https://review.openstack.org/52203900:49
*** Chealion has quit IRC00:57
*** Chealion has joined #openstack-ansible00:58
*** hw_wutianwei has joined #openstack-ansible01:02
*** nshetty has joined #openstack-ansible01:09
*** huxinhui_ has joined #openstack-ansible01:15
*** tnarg has quit IRC01:19
*** hw_wutianwei has quit IRC01:30
*** nshetty_ has joined #openstack-ansible01:44
*** nshetty has quit IRC01:46
*** kukacz has quit IRC02:00
*** kukacz_ has joined #openstack-ansible02:01
*** chyka has joined #openstack-ansible02:02
*** hw_wutianwei has joined #openstack-ansible02:06
*** chyka has quit IRC02:07
*** nshetty_ has quit IRC02:20
cloudnullevenings02:21
openstackgerritMerged openstack/openstack-ansible stable/pike: [DOCS] Correct secret uuid in Ceph example  https://review.openstack.org/52203902:26
*** woodard has quit IRC02:29
*** woodard has joined #openstack-ansible02:30
*** mrtenio-afk has quit IRC02:56
*** mrtenio-afk has joined #openstack-ansible02:56
openstackgerritCuong Nguyen proposed openstack/openstack-ansible-os_nova master: Use group_names to check a host belongs to group  https://review.openstack.org/52094402:58
*** albertcard1 is now known as poopcat02:58
*** cuongnv has joined #openstack-ansible03:00
cuongnvcloudnull: can you pls help to review this https://review.openstack.org/#/c/520989?03:00
cuongnvodyssey4me: ^ thanks for your review and recheck03:01
*** poopcat has quit IRC03:04
*** poopcat has joined #openstack-ansible03:04
openstackgerritCuong Nguyen proposed openstack/openstack-ansible-os_glance master: Use glance_services to get group name  https://review.openstack.org/52098903:12
cloudnullcuongnv: looking03:15
*** m|y|k has joined #openstack-ansible03:15
cloudnullcuongnv: where is glance_services being defined?03:16
cuongnvcloudnull: defaults/main.yml03:16
cloudnullah i see it now.03:21
cloudnullnicely done!03:21
cuongnvcloudnull: thanks03:26
*** m|y|k has quit IRC03:59
*** m|y|k has joined #openstack-ansible04:00
*** john51 has quit IRC04:02
*** john51 has joined #openstack-ansible04:02
*** nshetty has joined #openstack-ansible04:41
*** pmannidi has quit IRC04:45
*** poopcat has quit IRC05:02
*** udesale has joined #openstack-ansible05:06
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Add dynamic table for our tested Scenarios  https://review.openstack.org/52029405:07
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova stable/ocata: Update filters to support over-commit  https://review.openstack.org/52211305:10
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Update all SHAs for master  https://review.openstack.org/51752005:11
*** hybridpollo has quit IRC05:34
*** pmannidi has joined #openstack-ansible05:34
*** chyka has joined #openstack-ansible05:47
*** lihi has quit IRC05:51
*** oanson_ has joined #openstack-ansible05:52
*** oanson has quit IRC05:52
*** oanson_ is now known as oanson05:52
*** chyka has quit IRC05:52
*** lihi has joined #openstack-ansible05:53
*** markvoelker has quit IRC05:53
*** udesale has quit IRC06:00
*** udesale has joined #openstack-ansible06:00
*** taseer1 has joined #openstack-ansible06:00
*** Taseer has quit IRC06:01
*** taseer2 has joined #openstack-ansible06:01
*** taseer1 has quit IRC06:05
*** gouthamr has quit IRC06:10
*** indistylo has joined #openstack-ansible06:22
*** indistylo has quit IRC06:26
*** openstackgerrit has quit IRC06:33
*** hw_wutianwei1 has joined #openstack-ansible06:45
*** hw_wutianwei has quit IRC06:48
*** hw_wutianwei1 is now known as hw_wutianwei06:48
*** SmearedBeard has joined #openstack-ansible06:52
*** markvoelker has joined #openstack-ansible06:54
*** arbrandes has joined #openstack-ansible07:01
*** arbrandes1 has quit IRC07:03
*** SmearedBeard has quit IRC07:04
*** PaulusNL has joined #openstack-ansible07:15
*** nshetty is now known as nshetty|lunch07:22
*** openstackgerrit has joined #openstack-ansible07:25
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Converge neutron agents onto Baremetal  https://review.openstack.org/45445007:25
*** jvidal has joined #openstack-ansible07:30
*** pmannidi has quit IRC07:37
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova stable/pike: Set the nova-lxd lib to use PIKE  https://review.openstack.org/52215507:41
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-os_nova stable/pike: Update filters to support over-commit  https://review.openstack.org/51642907:42
*** threestrands has quit IRC07:47
*** pcaruana has joined #openstack-ansible07:49
*** mbuil has joined #openstack-ansible07:49
*** mrch has joined #openstack-ansible07:51
prometheanfirego to bed cloudnull07:54
openstackgerritMerged openstack/openstack-ansible-plugins stable/ocata: Add missing domain parameter to _get_group_roles  https://review.openstack.org/52160007:56
openstackgerritMerged openstack/openstack-ansible-ceph_client stable/ocata: Remove one layer of includes  https://review.openstack.org/52156807:58
*** SmearedBeard has joined #openstack-ansible07:58
*** shardy has joined #openstack-ansible07:59
*** SmearedBeard has quit IRC08:05
*** chyka has joined #openstack-ansible08:19
PaulusNLgoodmorning all, I've got a new openstack-ansible test environment in a personal VM environment. I don't know why but the only thing that is not working is Horizon webinterface on the extermal vip address08:19
PaulusNL(this would also be eventually be a start for out production)08:20
PaulusNLhow can I troubleshoot this best?08:20
*** gkadam has joined #openstack-ansible08:20
PaulusNLif I connect to the utility-container I can see users with the command openstack user list for example08:20
*** chyka has quit IRC08:23
*** holmsten has joined #openstack-ansible08:31
*** nshetty|lunch is now known as nshetty08:38
evrardjpgood morning everyone08:42
evrardjpPaulusNL: so external IP and internal IP are different?08:42
evrardjpif yes, did you check whether haproxy binds on this external IP, port 443?08:43
*** sxc731 has joined #openstack-ansible08:43
PaulusNLevrardjp: thank you for your quick response08:46
PaulusNLyes they are different but maybe I need to redo this with the same ip08:46
PaulusNLand I will check my HAproxy08:47
evrardjpkeep those different08:49
evrardjpit's easier08:49
PaulusNLok08:49
PaulusNLI will do this08:49
evrardjpthis way you can isolate traffic too.08:49
PaulusNLok08:49
evrardjpwas it the same IP?08:49
PaulusNLnope08:49
PaulusNLtwo different08:49
PaulusNLinternal and external08:50
PaulusNLbut after I rebooted my deployment server (that also includes a few containers) that also has the utility_container is not started automatically08:50
PaulusNLso I think I have more problems with this test enviroment08:51
evrardjpI don't know why you have so much trouble.08:54
evrardjpmaybe if you show your openstack user config and your user_*.yml files, we could help you. It shouldn't be that hard on clean machines.08:55
evrardjpIs your deploy server an infra node?08:56
PaulusNLok08:56
PaulusNLjep08:56
PaulusNLshall I paste it08:56
evrardjplxc-ls -f ?08:56
*** udesale has quit IRC08:57
PaulusNLevrardjp: did a pm08:58
evrardjpI see08:58
PaulusNLI also don't know why I'm running in so much trouble but I also did not get time for doing more about this installation08:58
evrardjpit runs on boot08:58
evrardjpyou just need to wait a little after a reboot08:58
PaulusNLfrom today I have time from my boss :)08:58
PaulusNLok08:58
evrardjpit takes a while to load all those containers when you have a single machine doing everything08:59
PaulusNLok clear08:59
PaulusNLneed to talk to someone... I will leave it for a while like this :)09:00
PaulusNLthank again evrardjp09:00
PaulusNLI can also paste the config I used09:01
PaulusNLif that is helpfull?09:01
evrardjpgenerally it is, we can understand the context a little more09:01
evrardjpplease paste a link here for everyone to help you with09:01
PaulusNLok09:01
evrardjpjust remove what you consider confidential (public ips/names)09:02
PaulusNLthat is for this no problem... it's my personall test env09:02
PaulusNLpm with the link to the config09:03
evrardjpcould you write it here? This way anyone can help?09:04
evrardjp I have too much on my plate09:04
PaulusNLyes I can09:04
PaulusNLhttp://paste.openstack.org/show/627045/09:05
PaulusNLthanks evrardjp09:06
evrardjpodyssey4me: hwoarang are you aware of a releasenotes problem?09:10
evrardjpor linters, or tempest...09:12
*** pbandark has joined #openstack-ansible09:14
*** taseer2 is now known as Taseer09:23
hwoarangevrardjp: good morning. no clue09:40
*** pester has joined #openstack-ansible09:48
*** fxpester has quit IRC09:49
* bndzor breaks his ceph cluster on purpose to remove raid60, switching to raid 010:01
openstackgerritMerged openstack/openstack-ansible-tests stable/ocata: Retry apt cache update  https://review.openstack.org/52180310:03
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/pike: Remove one layer of includes  https://review.openstack.org/52156610:03
evrardjphwoarang: thanks :)10:04
openstackgerritMerged openstack/openstack-ansible-os_gnocchi stable/ocata: Deprecate auth_plugin option  https://review.openstack.org/52181610:05
openstackgerritMerged openstack/openstack-ansible-os_aodh stable/ocata: Change auth config to keystone v3 according to openstack ocata aodh docs[1], we modified auth config under the "keystone_authtoken" and "service_credentials" groups. [1] https://docs.openstack.org/project-install-guide/telemetry-alarming/ \ ocata/install-  https://review.openstack.org/52183210:13
*** cuongnv has quit IRC10:13
*** electrofelix has joined #openstack-ansible10:25
*** hw_wutianwei has quit IRC10:28
*** stuartgr has joined #openstack-ansible10:30
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_ceilometer stable/newton: Initial OSA zuul v3 role jobs  https://review.openstack.org/51399910:50
andymccrrandom question - does anybody remember where we setup UCA in mitaka?11:01
*** jwitko has quit IRC11:05
odyssey4meandymccr we didn't11:12
andymccrahh ok11:12
odyssey4mewe only started using UCA in newton11:12
andymccrwell that would explain it :)11:14
andymccri wonder if default ubuntu backports comes with the libec packages for swift11:14
andymccrmust do since we install them as part of mitaka11:14
odyssey4meandymccr for trusty it was in backports, yes11:15
odyssey4meevrardjp I know that infra has been modifying the tests/builds for releasenotes and docs over the last week or so, and it has resulted in breakage. Over the w/end it was totally busted.11:20
openstackgerritMerged openstack/openstack-ansible-os_barbican master: Add CentOS 7 deployment support  https://review.openstack.org/52177211:22
*** sxc731 has quit IRC11:22
evrardjpodyssey4me: I've pinged Andreas11:23
evrardjpI've seen the issue appearing after a patch of his11:23
odyssey4mehwoarang it'd be nice to round out the barbican role with suse support too ;)11:23
odyssey4meevrardjp I see a comment in #openstack-infra saying that releasenotes are working again - the comment was at 06:33 this morning11:25
hwoarangodyssey4me: sure :)11:33
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi stable/newton: Use numpy 1.11.1  https://review.openstack.org/52222411:56
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi stable/newton: Use numpy version specified in upper constraints  https://review.openstack.org/52222411:57
hwoarangodyssey4me: do you know why we require postgresql-devel libraries in various os_* roles?11:59
odyssey4mehwoarang no idea, at some point I think they may have become a build requirement... but I don't know11:59
hwoarangright12:00
*** ianychoi has quit IRC12:00
*** ianychoi has joined #openstack-ansible12:00
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi stable/ocata: Use numpy version specified in upper constraints  https://review.openstack.org/52222612:01
*** huxinhui_ has quit IRC12:04
*** shardy has quit IRC12:04
*** shardy has joined #openstack-ansible12:06
openstackgerritMerged openstack/openstack-ansible-tests master: Use package module to install lvm2  https://review.openstack.org/52195312:10
*** shardy has quit IRC12:15
*** shardy has joined #openstack-ansible12:16
openstackgerritMerged openstack/openstack-ansible-os_nova stable/ocata: Update filters to support over-commit  https://review.openstack.org/52211312:18
*** kukacz_ is now known as kukacz12:20
*** shardy has quit IRC12:23
*** SmearedBeard has joined #openstack-ansible12:26
*** shardy has joined #openstack-ansible12:28
*** pester has quit IRC12:36
*** fxpester has joined #openstack-ansible12:36
*** mrch has quit IRC12:38
odyssey4mehwoarang evrardjp could you take a peek at https://review.openstack.org/#/q/Id9636b1160f84d069e1e231996f19773c81402fd asap please12:40
evrardjpit rings me a bell that if we remove that, we have a 1.11.x that gives problem12:42
evrardjphttps://etherpad.openstack.org/p/osa-telemetry-stack-fixing12:42
FoXMaNhello12:43
odyssey4mehttps://github.com/openstack/openstack-ansible-os_gnocchi/commit/7fa88e5dec2f2828e3e0f44353932188eb9888ce is where it got added12:43
odyssey4meit happens to only be panndas that needs constraining, because it's not in u-c12:43
evrardjphttps://github.com/openstack/openstack-ansible-os_gnocchi/commit/7fa88e5dec2f2828e3e0f44353932188eb9888ce12:43
evrardjpok12:43
evrardjplet's try this12:44
evrardjpgokhan: FYI ^12:44
evrardjpFoXMaN: hello12:44
FoXMaNany advice on how to debug slow horizon/api responses? i narrowed down problem to nova-api-os-compute container which takes 20-55 seconds to respond the request12:45
FoXMaNmachines running containers have 6 cores and 32GB of ram. No sign of i/o related problems12:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi stable/ocata: Use numpy version specified in upper constraints  https://review.openstack.org/52222612:45
FoXMaNbut everything is slow12:45
evrardjpFoXMaN: nothing in the logs? dns resolving is fine?12:46
odyssey4meevrardjp hwoarang updated the ocata one to reflect that the version is the same in u-c12:46
*** jvidal has quit IRC12:46
odyssey4methe newton one was set to that version by me when trying to make the gates work again, so I think the newton one is good12:46
evrardjpdone12:47
odyssey4methanks12:47
evrardjpgokhan: please note the change ^12:47
FoXMaNevrardjp: i haven't seen anything relevant. except horizon logs stating that request cannot be sent (due to haproxy timeout i belive)12:48
gokhanhello odyssey4me evrardjp12:48
evrardjpFoXMaN: did you try to see in a cli debug mode, to find which call is generally wrong?12:48
evrardjpFoXMaN: could you check also your haproxy 's backends state?12:48
odyssey4megokhan howdy - I took the version spec for numpy out of the gnocchi role as it's already covered by upper-constraints12:48
evrardjpFoXMaN: is that a new installation or something old?12:49
flaper87Hey folks, quick q re zuul: how are the jobs referenced in your zuul definition? I noticed you have openstack-ansible-linters in some repos but I don't see any requirement on openstack-ansible-tests12:49
flaper87is that done automagically by zuul?12:49
odyssey4meflaper87 requirement?12:49
FoXMaNevrardjp: this is 2 months old installation. 20 compute hosts12:49
evrardjpflaper87: https://github.com/openstack/openstack-ansible-tests/blob/master/zuul.d/jobs.yaml#L17 ?12:50
FoXMaNevrardjp: by cli debug mode you mean client debug or service ?12:50
gokhanodyssey4me, yep you are right. ın fact pandas is enough12:50
flaper87lemme rephrase12:50
FoXMaNevrardjp: haproxy times out after 50 seconds of waiting for backend response so it returns gateway timeout12:50
evrardjpclient debug... to see all the calls, finding which one is slow12:50
flaper87When I patch for https://github.com/openstack/openstack-ansible-plugins is posted12:50
flaper87it triggers these jobs: https://github.com/openstack/openstack-ansible-plugins/blob/master/zuul.d/project.yaml#L18-L3012:51
odyssey4meflaper87 yup12:51
flaper87but, openstack-ansible-linters is not defined in that repo but openstack-ansible-tests12:51
odyssey4mecorrect12:51
flaper87how does zuul find the definition for those jobs?12:51
evrardjpyes, we all share a funny loving global scope12:51
evrardjpflaper87: zuul is kinda awesome and scary, that's your answer. It knows ALL the jobs.12:52
flaper87evrardjp: ok, so, it's done by zuul12:52
flaper87coolio12:52
odyssey4meflaper87 it loads jobs into a single namespace from all repositories in an order that's specified in project-config... and all that has to happen is that the repo holding the tests needs to be before the repo using the tests, so we pushes our tests repo high up: https://review.openstack.org/51267812:53
*** yifei has quit IRC12:53
flaper87a-ha, that's the piece I was missing12:54
flaper87gotcha, thanks for the details12:54
odyssey4mesure, no problem :)12:54
FoXMaNevrardjp: client debug indicates that  GET http://172.23.0.20:8774/v2.1/f2321282f3824f99868879b6afe3368f/servers/detail?all_tenants=True took 20 of 22 total request time12:54
evrardjpI definitely don't know what this means like that.12:55
odyssey4meevrardjp mind giving this a nudge? https://review.openstack.org/52192312:55
evrardjpFoXMaN: so it's just nova_api_os_compute that is slow?12:56
evrardjpodyssey4me: that's so ugly.12:57
gokhanevrardjp, https://review.openstack.org/#/c/521818/  what is reason failure. I can not understand clearly. error is ERROR:   functional: commands failed12:57
evrardjpodyssey4me: you're doing me a lot of pain recently.12:57
*** cjloader has joined #openstack-ansible12:57
evrardjpBut ok, voted.12:57
odyssey4meevrardjp how so?12:57
FoXMaNevrardjp: yes12:58
evrardjpso many ways to fix the linting, just skipping it ISN'T the solution :p12:58
evrardjpbut you know it is a backport, and we need to advance.12:58
odyssey4meevrardjp oh sure, but better to get the gates moving sometimes than to get all creative while the patches pile up12:59
evrardjpvote is there.12:59
odyssey4mein most cases these test implementations are so horribly hacky they frighten me12:59
evrardjpgokhan: if you check on the log13:00
odyssey4metrying to make them more elegant starts feeling a bit like this: https://media.giphy.com/media/wotqBCVmRVn2w/giphy.gif13:00
evrardjpyou can see on this line the reason of the failure http://logs.openstack.org/18/521818/1/gate/openstack-ansible-functional-centos-7/f83d560/job-output.txt.gz#_2017-11-22_09_24_19_59548513:00
evrardjpodyssey4me: lol13:01
*** pradiprwt has joined #openstack-ansible13:04
gokhanevrardjp ok thanks. And also on nova-console and aodh haproxy config gives http 503 error . Service is unavailable beacause of 'http-check expect status 401'. If we remove this , services are working succesfully.13:05
evrardjpgokhan: you can change that in haproxy config13:05
evrardjpgokhan: https://github.com/openstack/openstack-ansible/blob/master/group_vars/haproxy_all/haproxy.yml13:05
pradiprwtHi Everyone, How to disable audh, ceilometer projects in installation ?13:06
evrardjpit's not by default pradiprwt13:06
pradiprwtevrardjp: I remember last time I have disabled it by deleting few files "rm /etc/openstack_deploy/conf.d/{ceilometer,gnocchi,aodh}.yml" but now files names are different something like audh.yml.aio13:08
pradiprwtdo I need to delete the .aio file ?13:09
odyssey4mepradiprwt the .aio files are not loaded by default, only .yml files13:11
odyssey4meso to enable, you rename to .yml13:11
*** cjloader has quit IRC13:12
pradiprwtodyssey4me : I want to disable it, default audh and ceilometer enabled13:13
odyssey4meno, by default it is not enabled13:13
hwoarangevrardjp: can we get this in https://review.openstack.org/#/c/521790/ so we dont miss breakages on pike? thank you13:13
pradiprwtWhile istallation I can see the containers with these names13:14
odyssey4mepradiprwt the default aio build renames these file in order to enable the services: https://github.com/openstack/openstack-ansible/blob/master/tests/bootstrap-aio.yml#L29-L3813:14
odyssey4mepradiprwt then you must be using a *really* old build, or you've put in config yourself to enable them13:14
evrardjphwoarang: I like this dynamism we have with the jobs. We should continue that way if a problem occurs.13:15
hwoarang:)13:15
odyssey4meevrardjp yeah, it's nice13:15
evrardjpwhile still fixing/hoping it shouldn't happen.13:15
evrardjpI am wondering what we can do for copr though.13:15
pradiprwtodyssey4me: got it , I am installing multi-node with HA, I think I need to rename the files13:16
gokhanevrardjp aodh-api haproxy config is fixed on master. But on pike and ocata it must also be fixed because of this line for pike https://github.com/openstack/openstack-ansible/blob/stable/pike/group_vars/all/haproxy.yml#L21513:16
evrardjpbecause right now copr deserves to be called ncopr13:16
openstackgerritMerged openstack/openstack-ansible-os_gnocchi stable/newton: Use numpy version specified in upper constraints  https://review.openstack.org/52222413:16
evrardjpncapr*13:16
hwoarangevrardjp: it's very convenient we can tweak jobs from a single repo13:16
evrardjpyeah13:16
odyssey4meevrardjp yeah, we should implement an infra mirror for it13:16
evrardjpthat we manage13:16
hwoarangcorp stuff is annoying indeed :/13:16
evrardjpnot sure we can13:16
odyssey4mewhy not? either mirror or reverse proxy13:17
odyssey4mewe did a for mariadb, and a reverse proxy for images.linuxcontainers13:17
evrardjpreverse proxy maybe but mirror ... couldn't there be a problem with the licensing?13:17
odyssey4meI dunno. rev proxy is fine then13:17
odyssey4meI'd just like to see less centos fails13:18
evrardjptell me about it!13:18
evrardjp:p13:18
evrardjpor having a fast centos.13:18
evrardjpwe are one month too early to ask for Xmas.13:18
evrardjplet's continue with what we have!13:19
gokhanevrardjp, also for nova-console it must be fixed. https://github.com/openstack/openstack-ansible/blob/master/group_vars/haproxy_all/haproxy.yml#L192 . this line must be removed. İf you confirm, ı can send patch for it.13:19
evrardjpgokhan: I can't confirm ..., because ppl don't seem to complain about it... I'd say we need to properly triage this if that's a bug13:20
evrardjpcould you file a bug?13:20
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/newton: Update os_gnocchi role SHA for numpy version fix  https://review.openstack.org/52224013:25
FoXMaNevrardjp: after restarting container hosts,memcached did not start inside the containers. So we had no memcached alive13:26
evrardjpthat sounds bad :p13:27
odyssey4meevrardjp hwoarang could you please review https://review.openstack.org/522240 asap13:28
evrardjpit was already done!13:28
odyssey4methanks13:28
*** pradiprwt has quit IRC13:29
odyssey4methe ocata one can wait for the next role sha bump as it's the same version13:29
*** yifei has joined #openstack-ansible13:29
*** yifei has quit IRC13:33
*** hw_wutianwei has joined #openstack-ansible13:36
openstackgerritMerged openstack/openstack-ansible stable/pike: Retry apt cache update  https://review.openstack.org/52179913:38
*** dave-mccowan has joined #openstack-ansible13:41
hwoarangevrardjp: is this outdated https://specs.openstack.org/openstack/openstack-ansible-specs/ or am i looking at the wrong thing?13:42
odyssey4mehwoarang yeah, for some reason the specs repo isn't publishing properly13:42
odyssey4meI haven't had a chance to look into it.13:42
hwoarangoh ;/13:42
hwoarangok13:42
odyssey4meit appears to only be ours, and I see other service specs are being published13:43
odyssey4meif you're able to look into it I'd appreciate that!13:43
evrardjpyeah we don't publish13:44
*** dave-mcc_ has joined #openstack-ansible13:44
odyssey4mewe are supposed to13:44
evrardjpI don't think it matters.13:44
evrardjpdoes it?13:44
odyssey4meit matters - either we publish, or that content must be removed13:44
evrardjpI mean: are we really supposed to?13:44
odyssey4mebecause right now it looks like we aren't doing anything with specs13:45
odyssey4meyes, we are supposed to - we've been publishing them since kilo when we created that repo13:45
*** dave-mccowan has quit IRC13:45
hwoarangit's convenient to know what specs are there i think13:46
evrardjpdoing it in the past doesn't mean we are forced to by governance13:46
hwoarangwell it's misleading if you keep outdated info there :)13:46
*** SerenaFeng has joined #openstack-ansible13:46
odyssey4meno, we're not forced to by governance13:46
evrardjphwoarang: I'd rather make that part of our documentation.13:46
*** SerenaFeng has quit IRC13:46
odyssey4mebut we *chose* to implement the specs process13:46
odyssey4meand if we have specs, we should publish them just like we do all our other docs13:47
odyssey4methe published specs are indexed better by search engines than the git source13:48
evrardjpI am all in for open design. How the sausage is made doesn't matter to me.13:48
evrardjpI am thinking of removing blueprints too.13:49
*** yifei has joined #openstack-ansible13:50
odyssey4mesure, if you like13:51
*** SerenaFeng has joined #openstack-ansible13:51
evrardjpbut I agree this spec thing isn't clear.13:51
odyssey4mepersonally I find that specs are useful as a record of design decisions including the background that led up to them13:51
odyssey4mereviews alone don't cover that, and there isn't really any other facility that provides something similar at this stage13:52
odyssey4meif we choose to implement something else to replace it that makes better sense for our workflow, then I'm all in13:52
odyssey4meIt could perhaps be useful to rather publish them in our docs so that they're in-context.13:53
openstackgerritMerged openstack/openstack-ansible-os_tempest master: Fix ansible-lint test failures  https://review.openstack.org/52202913:58
gokhanok evrardjp I will file a bug13:58
evrardjpodyssey4me: I don't mean get rid of the specs. The specs are interesting. I think it should be closer to the project, part of our documentation maybe. Will talk with Docs PTL what he thinks about that.14:01
evrardjpI've updated the wiki to point to our specs too with the cgit.14:01
openstackgerritLogan V proposed openstack/openstack-ansible-specs master: Fix linter issue  https://review.openstack.org/52225714:01
openstackgerritMerged openstack/openstack-ansible-os_neutron stable/pike: Resolve ansible-lint test failures  https://review.openstack.org/52192314:02
logan-i think that might fix the specs publishing issue ^14:08
evrardjplogan-: if this is the only cause, that was a quick fix :)14:11
logan-i don't understand how that got merged14:12
logan-so who knows14:12
logan-lol14:12
openstackgerritMerged openstack/openstack-ansible-os_gnocchi stable/ocata: Use numpy version specified in upper constraints  https://review.openstack.org/52222614:14
hwoaranghmm something regressed really bad on centos https://review.openstack.org/518611 opensuse finishes in less than 1h but centos timesout in tempest :/14:14
logan-actually it doesn't look like the ci job runs tox at all, so doc8 isn't running there14:14
logan-i guess it probably isn't the issue then14:14
evrardjpmaybe someone should file a bug14:15
evrardjpthis way we register the issue and we can properly triage it, this way ppl can say what they think about the criticallity14:15
evrardjp-l14:15
openstackgerritMerged openstack/openstack-ansible-os_nova master: Use group_names to check a host belongs to group  https://review.openstack.org/52094414:16
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_rally stable/ocata: Initial Zuul v3 role jobs  https://review.openstack.org/51429214:16
openstackgerritMerged openstack/openstack-ansible-specs master: Fix linter issue  https://review.openstack.org/52225714:18
openstackgerritMerged openstack/openstack-ansible-tests stable/pike: zuul.d: jobs.yaml: Restore voting status of openSUSE jobs  https://review.openstack.org/52179014:20
openstackgerritMerged openstack/openstack-ansible-os_rally stable/ocata: Updated from OpenStack Ansible Tests  https://review.openstack.org/52013914:21
*** PaulusNL has quit IRC14:24
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_rally stable/ocata: Initial Zuul v3 role jobs  https://review.openstack.org/51429214:26
odyssey4melogan- evrardjp I suspect that the job is incomplete, because I didn't see a job go into post for the publishing14:26
odyssey4methe job definition I mean14:26
logan-ah14:26
logan-yeah14:26
odyssey4meit's odd though, because when I last checked we were defined the same way as other specs repositories14:27
odyssey4meadmittedly that was a long time ago though, and I wasn't looking very hard because all the other zuul v3 stuff was happening14:27
evrardjpfile a bug, triage it, fix it depending on the urgency :p14:30
evrardjpit's a process :)14:30
*** SerenaFeng has quit IRC14:31
openstackgerritMerged openstack/openstack-ansible-os_aodh stable/ocata: Deprecate auth_plugin option  https://review.openstack.org/52181814:31
*** esberglu has joined #openstack-ansible14:34
openstackgerritMerged openstack/openstack-ansible stable/pike: Update all SHAs for 16.0.4  https://review.openstack.org/51882914:36
gun1xi got approval for hardware, to create a POC14:37
gun1xi need to get the list of hardware now. was thinking of: 2 firewalls/routers, 2 switches, 3 controller nodes, 3 compute+ceph nodes14:37
gun1xi still need to figure the best vendor for network devices14:38
*** nshetty has quit IRC14:45
cloudnullmornings14:45
*** portante has quit IRC14:49
cloudnullgun1x: We use a lot of F5 of Cisco and/or Arista. Most of the nics we use are Intel (some mellanox) and we've recently be testing Q'Logic (I've heard it looks good so far).14:55
cloudnull** F5, Cisco and/or Arista...14:56
gun1xcloudnull: all nics 10GB i guess14:56
cloudnullyup.14:56
gun1xand classic 4 nics / node? 2 bonds, 1 bond mgmt + storage, 1 bond vxlan + vlan ?14:56
cloudnullwe'll use the onboard nics if we need 1GB14:56
gun1xwhy would you "need" lesser power14:57
gun1x?14:57
cloudnulldepends on the setup.14:58
cloudnulland the port density14:58
jrosser25G nics are the very similar price to 10G these days and same for switches14:58
cloudnull^14:58
gun1xnice14:58
evrardjpDid anyone already get issues with http://docs.ansible.com/ansible/latest/git_module.html  that gets fixed by removing the depth: argument?14:58
evrardjpIt's not the first time14:59
evrardjpfor me14:59
evrardjpudner ansible 2.114:59
gun1xare you running compute and ceph OSD on the same nodes? and ceph mon on the controllers?14:59
evrardjpand good morning gun1x jrosser cloudnull14:59
odyssey4meevrardjp you're seeing issues if depth is set?14:59
evrardjpyes14:59
gun1xgood morning, it's 5 pm in eastern europe :D14:59
odyssey4memight it be an old git version?14:59
evrardjpno, git 2.15:00
evrardjpnot 1.9 something15:00
odyssey4meoh, interesting - I've not seen it, but I can totally believe nonsense happening with ansible 2.1.x15:00
evrardjpso it should be totally alright15:00
odyssey4meansible 2.2 did so many things better - 2.1 was a bit of a fail15:00
evrardjpit's quite transient15:00
evrardjpthe worse thing15:00
cloudnullo/ evrardjp15:00
evrardjpis that this module is probably calling module.warn which doesn't work (let me check on that one)15:01
odyssey4meevrardjp perhaps we should just remove all uses of git depth for newton then?15:01
evrardjpcloudnull: how was the bug triage yesterday?15:01
evrardjpN and O I'd say.15:01
odyssey4meit'll just slow it down horribly, which is not good15:01
evrardjpI've seen this happen in O too.15:01
evrardjpO2. Hahah.15:01
cloudnullevrardjp: it was the best !15:01
evrardjpcloudnull: great!15:01
evrardjpI have seen no logs of it.15:02
cloudnullwe triaged like you wouldn't believe15:02
evrardjpThat's right!15:02
cloudnullkinda like a secret santa15:02
evrardjpLD15:02
evrardjp:D15:02
* cloudnull forgot he was supposed to run the triage15:02
* cloudnull is a bad person15:02
evrardjpMaybe we should organise one this week? Next week is community meeting15:02
cloudnullIDK, i'm off tomorrow and friday15:03
evrardjpwith the zuul, summit, holidays, meetings, it was quite a few weeks.15:03
cloudnullI suspect most of the US will be too15:03
evrardjpok15:03
cloudnullbut I'd be happy to join in should we have a meeting15:03
evrardjplet's do it next week, another day of the week, probably thursday15:03
*** portante has joined #openstack-ansible15:03
cloudnullgun1x: re:ceph - I believe we deploy using stand alone ceph cluster to keep the separation of concerns however others may have thoughts on / real data on what works.15:05
* cloudnull is no ceph expert15:05
*** cjloader has joined #openstack-ansible15:06
andymccri think it depends on your use-case gun1x - e.g. if you dont have massive workloads and/or you have limited hardware you "could" put osds on compute. im not a huge fan of that approach personally.15:07
andymccrmon hosts could go on infra, but again if you have large load they'll probably need to be split off at some point15:07
gun1xandymccr: for a Proof of Concept i think it should be enough15:08
andymccrgun1x: yeah definitely, for a PoC it'd work fine15:09
gun1xandymccr: i am looking through https://github.com/ceph/ceph-ansible/blob/master/group_vars/all.yml.sample and can't find the variables to configure object storage to run on hdd and block storage to run on ssd15:10
gun1xlogan- said this is possible but i can't remember wher eto find the docu on how to write the playbook15:11
andymccrgun1x: that's not really a ceph-ansible thing, thats how you configure your drives and pools15:11
gun1xandymccr: so that is not configure via ansible and is done via ceph commands?15:12
*** gouthamr has joined #openstack-ansible15:12
andymccrgun1x: yeah take a look at this article for example: https://www.sebastien-han.fr/blog/2014/08/25/ceph-mix-sata-and-ssd-within-the-same-box/15:12
andymccryou'd then create the 2 pools and for your cinder backend you'd use the pool with only SSDs, and for radosGW you'd use the sata pool name15:13
gun1xandymccr: doesn't OSA automatically create the pools?15:13
jrosserimho you might want to do ceph yourself using ceph-ansible15:14
andymccrgun1x: i think we create a pool to get it set-up as part of an AIO15:14
jrosseryou will then learn all about it rather than fight ceph-ansible indirectly through OSA15:14
jrosserans in luminous it will auto allocate SSD and HDD into the relevant pools for you, no need to fiddle the crush map15:15
andymccrjrosser: ahh sweet. did not know that had been added to luminous - how does it know which is ssd vs sata?15:15
gun1xjrosser: AFAIK support for ceph was added in pike in OSA15:16
jrosseri think "spinning media" is a flag that can be read from a drive15:16
andymccrjrosser: oh that is pretty awesome15:16
andymccrhttps://blog-fromsomedude.rhcloud.com/2017/05/16/Luminous-series-CRUSH-devices-class/15:17
andymccrgives some details gun1x - pretty decent read15:17
jrossergun1x: yeah i know, just sharing that even though thats there we've chosen to separate them15:17
jrosserbut it's up to you :) felt like too much to tangle together i guess in our environment15:18
andymccrjrosser: i think its nice in some ways - e.g. the inventory integration, but from a learning perspective i'd agree, deploy ceph yourself with ceph-ansible and give that a go15:18
gun1xok i will go through the documentation15:21
evrardjpoh interesting, we are not the only ones: https://github.com/openstack/os-faults/blob/master/os_faults/ansible/executor.py#L19-L2215:22
gun1xthank you for the advice, andymccr jrosser15:26
jrosserandymccr: cat /sys/block/sda/queue/rotational15:33
jrosseralso lsblk -d -o name,rota15:34
*** marst has joined #openstack-ansible15:36
*** mbuil has quit IRC15:51
*** mbuil has joined #openstack-ansible15:52
logan-for converged vs non converged osds, ive not had great experiences when the servers have a lot of drives. there is too much contention between vms and the osds for cpu and ram. but converged has worked great for me on blades where i am only running 1 or 2 osds per node15:52
*** holmsten has quit IRC15:56
*** yifei has quit IRC15:57
*** michelv has joined #openstack-ansible16:08
odyssey4meI'm clearly being dumb here. Is there a condition I can use to test if a host exists in the inventory.16:17
odyssey4meI'm using the openstack dynamic inventory, and need to only do something against a host if it's already there.... if not, then the creation will happen and life moves on swimmingly.16:18
odyssey4mejrosser logan- evrardjp thoughts?16:18
logan-if hostvars[host] is defined16:19
logan-if host in groups['all']16:19
logan-one of those maybe?16:19
odyssey4meI suppose I could use... when: "'server-name' in groups['all']"16:19
odyssey4meheh, same-same16:19
odyssey4methanks16:19
logan-if host in hostvars.keys()16:20
odyssey4meI think I like that one best16:21
evrardjptechnically the best would be if member of groups['all']16:22
odyssey4meoh? ok16:22
evrardjpbut most likely if host in hostvars.keys() would work16:22
cloudnullodyssey4me: looking for a known host in inventory or a host of a particular group?16:24
odyssey4mein the inventory as a whole16:26
openstackgerritLogan V proposed openstack/openstack-ansible master: [WIP] Implement iptables management  https://review.openstack.org/51209916:33
odyssey4meyeah, this works nicely: when: "'server-name' in groups['all']"16:35
*** germs has joined #openstack-ansible16:41
SmearedBeardhey all.. quick question I understand this to divide the number of cores by 2 to the nearest whole number, but not sure what that 1 means at the end ;[ansible_processor_vcpus|default(2) // 2, 1;16:44
SmearedBeardif that a minimum acceptable value?16:44
odyssey4meSmearedBeard paste the whole line16:44
SmearedBeardheat_api_threads: "{{ [[ansible_processor_vcpus|default(2) // 2, 1] | max, heat_api_threads_max] | min }}"16:45
odyssey4meIIRC it takes the smallest value between half the cores (an integer result) and the max processes16:45
logan-hwoarang: is the build hanging at the end of the run still happening? im guessing you didnt find the cause yet. i think i might have just stumbled on something16:45
odyssey4meyeah, so the array is [ half cores, 1 ] | max -> ie the largest number16:45
*** hw_wutianwei has quit IRC16:46
odyssey4methen that is used in another array - [ that largest number, the max threads ] | min16:46
odyssey4meit's there to prevent masses of processes from being created in very hihgh end systems like POWER where there are often 128+ cores in a system16:46
hwoaranglogan-: yeah it does. i have a strange feeling that ansible can't kill processes properly or something and this https://review.openstack.org/#/c/520615/ seems to fix it16:46
hwoaranglogan-: as you see the opensuse job failed but because of tempest not because of timeout. the playbook finishes correctly16:47
hwoarangi have no clue...16:47
logan-yep i was just suspecting dstat isnt being killled16:47
logan-http://osa-ci.objects-us-dfw-1.cloud.lstn.net/516448/10/927/logs/console.log <- last 3 lines of output here16:47
logan-log collcetion function didn't run in that build and you can see jenkins alert about the hung file descriptors16:48
hwoarangyeah but i would expect that parent should kill everything upon exit16:48
*** woodard has quit IRC16:48
logan-instead of running "dstat <args> &" maybe we should run it in nohup or something to decouple it from the gate-check-commit descriptors16:48
hwoarangyeah :/16:48
hwoarangsneaky bug16:48
odyssey4melogan- hwoarang I noticed when working on the tests to verify api uptime during upgrades that there are problems trying to kill processes from ansible tasks... I think it might have to do with something along the lines of one thing can't kill another subshell's process or something like that16:49
logan-yup i see the only other thing we're launching in a similar fashion is some stuff in the rolling upgrade tests16:49
odyssey4meyup, and that's where I saw the issue16:49
odyssey4methe one in the integrated repo still isn't working right as far as I know16:50
SmearedBeardodyssey4me that makes sense and what I understood, I was making notes and released I had 16 threads, but my processor is an i7 so the I should have been assigned 4 threads..16:50
odyssey4methe keystone one is working though16:50
odyssey4meSmearedBeard I don't think it's cores though, it's vCPU's... so it's likely cores + hyperthreading16:50
odyssey4meyou may have 8 cores, but the host will think it's 16 cpu's... and therefore ansible will too16:51
logan-ok thanks ill take a look at that16:51
*** savvas has quit IRC16:51
*** savvas_ has joined #openstack-ansible16:51
SmearedBeardodyssey4me it's an i7 4 core, so 8 with hyperthreading - I am running grep -c processor /proc/cpuinfo and getting 8..16:52
odyssey4meSmearedBeard oh, that's interesting16:52
odyssey4meSmearedBeard if you do this what's shown: ansible -m debug -a var=ansible_processor_vcpus <host>16:53
openstackgerritLogan V proposed openstack/openstack-ansible master: Fix automatic log copying in Zuul runs  https://review.openstack.org/51644817:02
*** michelv has quit IRC17:07
*** chyka has joined #openstack-ansible17:07
mbuilcan somebody confirm me that we can use "with_first_found" syntax with an "include" statement like here ==> https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/main.yml#L73?17:08
mbuilIt is not working for me :(17:08
*** chyka has joined #openstack-ansible17:10
cloudnullmbuil: yup that should work.17:12
cloudnullwhat issue are you seeing ?17:12
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_rally stable/ocata: Initial Zuul v3 role jobs  https://review.openstack.org/51429217:13
mbuilcloudnull: ok. I don't see anything being executed from the include playbook. However, if I include it doing (in this case neutron_plugin_type = ml2.opendaylight) - include: opendaylght_config.yml, then I can see the playbook being executed17:15
cloudnulland the task file is in the providers directory ?17:16
cloudnullproviders/opendaylght_config.yml17:16
mbuilcloudnull: https://hastebin.com/bucurecove.md. The Include provider specific config(s) is supposed to execute that part, but you can observe that nothing is executed and it jumps to the next line17:16
mbuilcloudnull: yes, it is there ==> https://github.com/openstack/openstack-ansible-os_neutron/tree/master/tasks/providers17:17
cloudnullah so it's literally not running this file https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/opendaylight_config.yml17:18
cloudnullI thought I saw a patch for that fly by the other day17:18
* cloudnull looking17:18
*** pcaruana has quit IRC17:18
mbuilcloudnull: exactly. Perhaps it gets the name incorrectly... I am investigating too17:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest stable/pike: Fix ansible-lint test failures  https://review.openstack.org/52230417:22
mbuilcloudnull: I found the issue, my mistake! There was a typo in the neutron_plugin_type17:23
mbuilcloudnull: it works now :)17:24
cloudnullnice!17:24
mbuilcloudnull: thanks!17:24
* cloudnull is glad it wasn't something we broke :D17:24
odyssey4methose with_first_found tasks do result in some obscure errors17:24
*** gkadam has quit IRC17:25
odyssey4meI've wasted time on one of those at least once, going down rabbit holes of trying to figure out why ansible's looking in the wrong places... eventually to realise that the error was far, far simpler.17:26
odyssey4mejmccrory evrardjp how's that inventory conversion script going?17:26
odyssey4mealso, in other news, I'm kinda warming to the idea of having our 'openstack-ansible' become something a little less of a bash-hack and quite a bit more of a real client17:28
andymccrwith_first_found is the devil17:28
odyssey4meit could allow us to have commands like 'openstack-ansible bootstrap roles' and perhaps replace the inventory-manage script with 'openstack-ansible inventory ...'17:28
evrardjpodyssey4me: good to hear :)17:29
evrardjpodyssey4me: the click framework is quite great on those topics, very simple to use, and to extend17:29
evrardjpI didn't work on the inventory conversion for a while, but it was working for me17:30
odyssey4mebut the click framework is not what openstack uses unfortunately, so if we use it we become a snowflake17:30
evrardjpyes, but it's widely used in the python community, so I think we are good.17:30
odyssey4meI wonder why the openstack client didn't use it.17:30
evrardjphttps://github.com/pallets17:31
odyssey4meI vaguely remember a ML discussion about it years ago.17:31
evrardjpI think we are independent enough17:31
evrardjpI didn't want to learn something more complex than something I already knew how to do :p17:31
evrardjpwe are using jinja anyway so we are already using part of pallets work? :)17:32
odyssey4meyeah, I'm good with that17:32
evrardjpanyway17:32
odyssey4menot sure if we'd be violating some form of governance if we used it17:32
evrardjpI'd say that if we do we'd be happy to change things later.17:33
evrardjpbut I think the prio is not on that17:33
evrardjpthe prio right now is on ansible 2.417:33
evrardjpbringing the same features as what we have, but with ansible 2.417:33
bndzorsoon(TM)17:34
evrardjpso using our wrapper with an empty inventory file to be able to load userspace group vars / host vars + the docs that goes with it.17:34
evrardjpthat and the usual burning fires.17:34
odyssey4meI'm pretty close to making the repo server entirely optional in a deployment, and also making it optionally a cache for wheels instead of having to build them.17:36
odyssey4meI should have some more patches up soon for that.17:36
*** savvas has joined #openstack-ansible17:40
*** savvas_ has quit IRC17:40
cloudnull^ prometheanfire - re: using click, does that have any implications in requirements?17:41
bndzorfml, the moment i need the console and it says 503 Service Unavailable17:44
evrardjpodyssey4me: cloudnull could you vote on this? it works now: https://review.openstack.org/#/c/514292/17:45
evrardjpbndzor: sorry to hear that :p17:45
evrardjplooks like you were the only one to have that recently17:45
evrardjpI heard that today too17:45
evrardjpwonder what changed17:46
cloudnullbndzor: https://i.imgflip.com/1ztyzp.jpg17:46
bndzor:(17:46
odyssey4meevrardjp awesome to see https://review.openstack.org/#/c/514292/6 working - happy to merge it?17:48
evrardjpyeah17:48
evrardjpwill fix rally everywhere, this way I want to show a benchmark team that deploying on metal rocks.17:48
evrardjpI am listening too much metal too, doesn't help.17:49
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_rally master: Constraint ansible installation  https://review.openstack.org/52231417:51
bndzorhmm17:53
bndzorwhat keeps overwriting my /etc/resolv.conf17:53
bndzordpkg -P cloudinit, still happening.. dhcd ?17:53
bndzordhcpd*?17:53
*** dmsimard is now known as dmsimard|afk17:53
prometheanfirecloudnull: what's using click?17:53
evrardjpme17:53
evrardjpI am using it alot!17:54
prometheanfirein what, where?17:54
odyssey4mebndzor usually resolvconf will do that based on network interface config17:54
evrardjpin osa, and almost all the cli tooling I write17:54
evrardjpprometheanfire: why would that be a problem?17:54
cloudnullprometheanfire: its a py lib https://pypi.python.org/pypi/click17:54
prometheanfireit shouldn't17:54
evrardjpk17:54
prometheanfireit'd be a problem for co-installability17:54
evrardjpwhat do you mean?17:55
prometheanfireI don't think osa is listed in projects.txt17:55
prometheanfirelooking17:55
odyssey4meyes17:55
cloudnullwe participate in requirements so if we add this lib, its added everywhere. right?17:55
odyssey4mebut that will only be an issue if we expect it to be in the global requirements17:55
prometheanfirehttps://github.com/openstack/requirements/blob/master/projects.txt#L12717:55
odyssey4meif we have it as a requirement through some other means (like we do with ansible), then it won't be affected by that17:55
prometheanfireit should be added, we track the deps of projects that are in reqs17:56
evrardjpnot sure what you meant there odyssey4me?17:56
odyssey4me*but* if we plan to build a package and publish it to pypi, then we'd have to have it in requirements.txt and therefore we'd have to have it in global requirements17:56
prometheanfireit's installed through ansible, not reqs.txt?17:56
evrardjpprometheanfire: at the moment nothing is listed in reqs. and click isn't installed through ansible17:57
odyssey4meevrardjp any package in requirements.txt or test-requirements.txt has to be registered in global requirements in the requirements repo - there's a process and set of requirements for that17:57
odyssey4meif we get it in, then it'll get put into u-c and all those nice things17:57
evrardjpok, and would that be a problem?17:57
prometheanfirewell, I don't think there'd be a problem with adding it to the reqs repo17:57
evrardjpok17:58
evrardjpthen we are good.17:58
odyssey4me*but* one of the criteria is that there isn't something in the list that already does what it does17:58
prometheanfiretrue, is there?17:58
odyssey4meand so we'd have to justify why we aren't using the things the openstackclient is using17:58
evrardjpbecause we are not openstackclient17:59
evrardjp:p17:59
prometheanfirethat's not a valid reason18:00
prometheanfirethe best angle to take would be to find something click can do that the other libs can't18:00
prometheanfireI also don't see tests in https://github.com/pallets/click18:00
evrardjpbut there is no lib that does that?18:00
evrardjphttps://github.com/pallets/click/tree/master/tests ?18:01
cloudnullwe use click in monitorstack https://github.com/openstack/monitorstack/blob/master/requirements.txt#L118:01
prometheanfiremaybe https://github.com/pallets/click/blob/master/click/testing.py18:01
cloudnullbut its not part of g-r-r18:01
prometheanfirelol18:01
evrardjphttps://github.com/pallets/click/blob/master/tox.ini#L1218:01
prometheanfireya, missed that entire folder18:01
odyssey4mecloudnull yeah, but I don't think monitorstack is subscribed to the global requirements management18:01
cloudnullit is not18:01
evrardjpbut when you think about it, this tooling could be completely optional18:02
evrardjpand completely separated18:02
odyssey4meI'm not saying that click can't be used, but I am just saying that don't think of it as a given that you will be able to. Your review to put it into g-r will need to justify why your lib is better to use for what we're doing.18:02
odyssey4meif the reasons are good, I find that the requirements team (hey prometheanfire!) are usually happy to roll with it18:03
prometheanfireyep, beyond that it looks good18:03
evrardjpwell click allows me to define in a very clean way with just a few decorators a context for my cli functions.18:04
evrardjpAlso convenient way to validate arguments without the need of argparse18:04
prometheanfiredon't tell me here, tell me in the review :P18:05
evrardjpI mean18:05
evrardjpyeah.18:05
evrardjp "I am just lazy" should be a good thing, right?18:05
prometheanfirein that case you should make everyone lazy, and switch out lib_x to for click18:06
prometheanfirein all of openstack :P18:06
odyssey4meso, I just asked mordred about it - to his knowledge click wasn't considered for OSC... cliff was used perhaps because it was the best around at the time that met the needs of OSC... and it seems that click wasn't around then18:07
*** mbuil has quit IRC18:07
odyssey4meosc started in 2012, click released 1.0 in 2014 it seems18:07
evrardjpoh yeah that would make sense18:07
odyssey4mearmed with all that info, I think you have a proposal ;)18:08
evrardjpwell I will remember all this conversation :)18:09
prometheanfireso, since we have another lib that does it all already it's gonna be a hard sell18:11
prometheanfireatm the only reason to allow it in reqs would be if cliff was fully switched out for click18:11
odyssey4mewell, that could be an interesting project ;)18:12
prometheanfireyep18:12
prometheanfirebasically, find a feature that's NEEDED that click provides that cliff doesn't and it will be easier18:13
evrardjpdecorators!18:13
evrardjpI mean proper decorators18:13
evrardjpdamn let's ignore this whole thing18:14
prometheanfirelol18:14
evrardjpI will just do like monitorstack18:14
prometheanfireiirc, the requirements bot may not like that18:15
evrardjp?18:15
evrardjpmonitorstack is just a different project, so we can do something like that, no?18:15
odyssey4meyeah, if we add a requirement to requirements.txt that isn't in g-t then we'll fail the requirements checks18:15
prometheanfirehaving something that's not tracked18:15
odyssey4mewe can, but then we have to unsubscribe from the g-r process and go back to managing it ourselves18:16
prometheanfiremonitorstack isn't in projects.txt18:16
evrardjpno I mean having a different project18:16
evrardjpyeah18:16
evrardjpopenstack-ansible-cli18:16
evrardjpboom18:16
prometheanfireoh, that's possible18:16
evrardjpodyssey4me: are the linters fixed? I thought everything was fixed now18:17
prometheanfireit may come back to bite you if click (or any other lib not tracked) starts to not be co-installable with the rest of openstack (lib versions supported)18:17
evrardjpodyssey4me: ignore this. me short memory.18:17
*** m|y|k has quit IRC18:17
odyssey4meprometheanfire for us, today, that's less of an issue - but it would become an issue if we ever got packaged by the distributions18:18
odyssey4methere's the other factor of us effectively splitting from the community direction if we did it, which doesn't sit well18:18
odyssey4meisn't it amazing how a simple technology choice can be the cause of so much trouble ;)18:19
prometheanfirenot totally true, it'd be installed in the global namespace, iirc nova-compute is run there as well18:19
prometheanfire:D18:19
prometheanfireif it was installed in a safe place I wouldn't care as much18:21
prometheanfireisn't ansible itself installed in a venv?18:21
odyssey4meand I guess we can never expect our users to do that ;)18:21
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/pike: Fix linters  https://review.openstack.org/52232218:21
odyssey4meyes, our tooling puts it into a venv and most of our users, as far as we know, use that tooling18:21
odyssey4mebut it is plausible that someone may not18:22
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/pike: Remove one layer of includes  https://review.openstack.org/52156618:22
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/pike: Remove one layer of includes  https://review.openstack.org/52156618:22
prometheanfireya, that makes it hard18:22
prometheanfirehow much work would it be to use cliff instead?18:22
*** TxGirlGeek has joined #openstack-ansible18:23
openstackgerritMerged openstack/openstack-ansible-os_rally stable/ocata: Initial Zuul v3 role jobs  https://review.openstack.org/51429218:23
prometheanfirebecause I think that's what I'd tell any other project :(18:24
evrardjpI checked, cliff isn't that easy to review and read. So I'd say it would involve writing quite a series of code. I'd rather write everything using argparse instead, and multiply the amount of code we have by 3.18:24
prometheanfireyou can bug tonyb and dirk about it as well18:24
prometheanfiremaybe message -dev about fully switching to it?18:24
evrardjpNah it's fine we already spent too much time discussing it.18:24
evrardjplol no18:25
odyssey4meif we have the right reasons, and it doesn't interfere, and it's co-installable, then it might still be accepted into g-r given the right justification18:25
evrardjpI'd rather re-write with argparse than start the politics fight. I'll get results faster.18:25
evrardjpcould anyone vote on https://review.openstack.org/#/c/522322/1 ?18:26
prometheanfireya, the ease of use can be an argument, I'm just not sure it's enough :(18:26
odyssey4medone18:26
evrardjpcloser to python community :)18:26
evrardjpthanks18:26
evrardjpzaqar passes on my machine!18:27
evrardjpwoot18:27
evrardjpcloudnull: could you give me a nice fast vote? https://review.openstack.org/#/c/52232218:27
odyssey4mewooho18:28
bndzorif i want to deploy more compute nodes, is it enough to run os-nova ?18:28
prometheanfiredynamic-inventory needs to be run right?18:29
prometheanfirehttps://docs.openstack.org/openstack-ansible/latest/admin/maintenance-tasks/scale-environment.html18:29
odyssey4mebndzor nova and neutron18:29
bndzorodyssey4me: cool, in that order ?18:30
bndzorah cool, thanks for that link prometheanfire18:30
odyssey4meit appears so https://github.com/openstack/openstack-ansible/blob/master/playbooks/setup-openstack.yml#L20-L2118:30
odyssey4meopenstack-ansible os-nova-install.yml os-neutron-install.yml18:31
prometheanfireodyssey4me: does dynamic-inventory not need to be run anymore?18:31
odyssey4meprometheanfire ansible runs it, and it's still there right now - but we are trying to move away from it for ansible 2.418:33
odyssey4meunless I've misunderstood your question?18:33
prometheanfireoh, neat18:34
evrardjpyeah then ansible tower!18:34
prometheanfirethe docs say to run it manually18:34
prometheanfirelol18:34
odyssey4mehuh? where?18:34
odyssey4meas far as I know it does say so as a diagnostic or verification only18:35
prometheanfireoh18:35
evrardjpyeah ansible runs it automatically?18:35
odyssey4meoh I see in the page you linked18:35
odyssey4methat's a pretty strange sequence to advise18:35
odyssey4methose warnings are all pre-newton18:36
prometheanfireyarp18:36
odyssey4meoh, I see what's being done there18:37
evrardjp?18:37
odyssey4methat's an interesting way of catering to a control plane host's containers18:37
odyssey4meso basically you don't have to type out all the container names in your limit18:37
odyssey4methat tag to update the hosts is very old too18:38
odyssey4mewow, it would be so nice to have an army of operators to keep these docs up to date18:38
odyssey4meok, I'm out for the night - cheers all!18:40
prometheanfirenn18:40
evrardjpgood night odyssey4me!18:40
bndzorah btw, i have a strange question guys. So in my old env, im running xen. Is there anyway that if i dd and create an image of one of my old machines, can push it into openstack somehow ?18:42
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-ceph_client stable/ocata: Fix linters  https://review.openstack.org/52234618:43
bndzoror, i guess i could setup one vm, attach a temporary volume, dd it back there, then detach it, and create a new image and attach that as the volume18:46
*** hachi_ has joined #openstack-ansible18:47
*** viktor_ has joined #openstack-ansible18:48
prometheanfireshould have asked for reviews on https://review.openstack.org/52169618:48
openstackgerritMerged openstack/openstack-ansible-ceph_client stable/pike: Fix linters  https://review.openstack.org/52232218:48
*** shardy has quit IRC18:51
openstackgerritMerged openstack/openstack-ansible stable/newton: Update os_gnocchi role SHA for numpy version fix  https://review.openstack.org/52224018:53
*** poopcat has joined #openstack-ansible19:02
*** albertcard1 has joined #openstack-ansible19:08
*** poopcat has quit IRC19:08
cloudnullbndzor: I wrote a tool to something like that a while back19:16
cloudnullhttps://github.com/cloudnull/InstanceSync19:16
bndzornice!19:16
cloudnullits really just fancy rsync19:16
*** cjloader has quit IRC19:17
cloudnullits likely a bit old and will need to be updated for more modern instance19:17
cloudnullhowever it could help you get the job done.19:17
bndzorim doing a test now with dd first to see if it even works19:18
cloudnull Aug 20, 2012 ... its been a minute since i looked at this19:19
cloudnulldd should work just fine.19:19
cloudnullyou might need to install cloud-init19:19
cloudnullthen dd19:19
*** electrofelix has quit IRC19:19
cloudnullif cloud-init wasn't already there.19:20
bndzorhehe19:21
bndzorlets see what happens without19:21
bndzorthe thing is, im also moving over a legacy environment into os19:21
bndzori dont have time to reinstall and do everything fancy right now so this will have to do19:22
bndzorive done the most ghetto crazy setup ever with a ethernet bridge between the two environments etc19:22
bndzorbut hey, it works :)19:22
*** stuartgr has quit IRC19:22
logan-bndzor: i don't know if you have any windows instances, but i helped a client move windows vms from xen to openstack recently and wrote some stuff on it https://logan.protiumit.com/2017/04/13/xen-to-kvm.html19:22
bndzoryuck windows!19:23
bndzorodyssey4me: i can confirm its not enough to run thoose two playbooks19:23
bndzorfatal: [appserver7]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.57' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127}19:23
bndzorfatal: [appserver4]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.54' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127}19:23
bndzorfatal: [appserver6]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.56' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127}19:23
bndzorfatal: [appserver5]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.55' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127}19:23
bndzorfatal: [appserver3]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Warning: Permanently added '172.29.236.53' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: /usr/bin/python: not found\n", "module_stdout": "", "msg": "MODULE FAILURE", "rc": 127}19:23
bndzoril just run setup-everything later and head out to grab some food19:24
cloudnulllogan-: nice!19:24
* cloudnull has not had to do anything windows related (professionally) in a very long time.19:25
bndzori only use windows to play some games19:25
bndzorwhen ever that happens..19:25
bndzorbeen working so hard latley that i am expecting a calls from chineese sewing factoriess coming soon19:26
cloudnulllol19:26
cloudnulllogan-: that driver issue looks like a tun of fun19:28
bndzoronly windows can have a message that tells you a system disk could not be found booting from it19:29
bndzor:-D19:29
logan-cloudnull: yeah windows cloud image prep is a blast. its not limited to xen conversions either... i think i get mild ptsd anytime someone says "just slipstream these drivers"19:30
logan-cause you just know its gonna be a week of fighting bsod19:30
bndzorthat feeling when you get ~150mbit/s on a 20ge uplink and the other side has the same :(19:31
logan-haha19:31
cloudnullha!19:31
bndzornever, ever, ovh again19:31
cloudnulloye logan- do you guys sell those discontinued servers out right?19:36
cloudnullLike If i wanted to buy a couple and colo?19:36
logan-yes19:36
bndzorif you want cheap ass servers check out kimsufi.. but its very lowend and ovh ;(19:36
cloudnullI'm contemplating moving all of my things off "cloud" and into dedicated gear19:37
cloudnullI've got a couple whitebox servers at home .19:38
SmearedBeardand soyoustart, it's the middle brand19:38
cloudnullbut I'm thinking about getting a half rack19:38
SmearedBeardso I found that trove does not support CentOS 7 either :( so ill be testing this this evening and sending a patch up for this as well19:39
*** hachi_ has quit IRC19:40
cloudnull++19:40
cloudnullsadly a couple of our roles have haven't had the same multi-os love as others.19:41
logan-cloudnull seems to be a popular thing lately. lots of ppl doing colo term deal with either old hardware bundled into the deal or we lease new hardware to fill the space.19:41
cloudnullI want to say all of them support opensuse.19:41
cloudnull^ thanks hwoarang :)19:41
SmearedBeardcloudnull meh not big deal, it's even enough to figure it and add19:41
SmearedBeardand they do not.. barbican and trove have not got opensuse19:41
cloudnullah.19:42
hwoarangyeah not all of them19:43
hwoarangi am doing barbican these days so stay tuned19:43
cloudnulllogan-: are the prices for purchase on the https://www.limestonenetworks.com ?19:43
SmearedBeardcool.. I did start looking at Barbican but opensuse package names looked weird, so I went back my testing my deployment19:44
*** hachi_ has joined #openstack-ansible19:44
bgmccollum_If I add a provider_network after completing a deployment, i know i need to re-run the neutron playbook, but I think I need to run something else so the corresponding interface gets created in the neutron_agents container...yes?19:44
cloudnullbgmccollum_: lxc-container-create19:44
hwoarangSmearedBeard: normally they are pretty similar to Centos. and the package list appears to be a subset of those listed in os_horizon19:44
bgmccollum_do i need to destroy before? or it'll fix it up?19:44
bgmccollum_cloudnull thx19:45
cloudnullyou can chain the two together `openstack-ansible lxc-container-create.yml os-neutron-install.yml --limit neutron_all`19:45
*** bgmccollum_ is now known as bgmccollum19:45
bgmccollumperfect19:45
cloudnullnothing to do before hand19:45
cloudnullit'll add the interfaces into the container config and restart it19:45
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: scripts: scripts-library.sh: Use pgrep -f to find the dstat process  https://review.openstack.org/52061519:45
SmearedBeardcloudnull ill take a look and and test once vagrant is done running for trove/centos19:46
cloudnullsweet!19:46
logan-cloudnull: no not really. Most of the stuff on the site is bare metal lease not colo. Ping me sometime what you want specs wise. We have been basically giving away e3v1 and dual 5600s in most of the deals lately and also free months on longer term deals19:48
SmearedBeardcloudnull speaking of --limit.. my deploy failed, ansible has the ability to restart from the trove play right?19:48
logan-cloudnull: all that old hw doesn’t do much good sitting in the stock room so if we can get someone into colo on it, we make it cheap :p19:49
*** hachi_ has quit IRC19:50
*** hachi_ has joined #openstack-ansible19:53
cloudnullSmearedBeard: yes.19:56
cloudnullyou can just rerun the os-trove-install playbook19:56
cloudnullwithout having to do all of the setup-openstack plays19:56
cloudnullassuming it failed within the trove role.19:56
*** esberglu has quit IRC20:00
*** esberglu has joined #openstack-ansible20:00
*** esberglu has quit IRC20:00
SmearedBeardcloudnull yeah, failed at ' Gather variables for each operating system'.. what about things after trove? octavia is LBaaSv2 right?20:01
*** esberglu has joined #openstack-ansible20:01
cloudnullthat's likely a fact issue or can be related to the operating system variable file not existing20:02
cloudnullhttps://github.com/openstack/openstack-ansible-os_neutron/tree/master/vars20:02
cloudnullit tries to do a lookup of the variable file by name using facts.20:03
cloudnullso if the facts are bad / missing or if the variable file is not found it'll throw that error20:03
cloudnullas for octavia, yes that is lbaasv220:03
cloudnull^ xgerman_20:03
cloudnull^ has been working on that role.20:03
SmearedBeardcloudnull I fixed the issue and just testing before sending a patch, I just wanted to avoid runnig setup-openstack.yml again, as it's long20:05
cloudnull++20:05
*** esberglu has quit IRC20:05
gun1xlogan-: andymccr: https://www.sebastien-han.fr/blog/2014/11/27/ceph-recover-osds-after-ssd-journal-failure/20:06
gun1xdid you ever try anything like this to recover a journal?20:06
bndzor# ceph health20:12
bndzorHEALTH_OK20:12
* bndzor goes back to beeing not productive20:13
cloudnullhahaha20:13
*** hybridpollo has joined #openstack-ansible20:15
*** gouthamr has quit IRC20:20
*** albertcard1 is now known as poopcat20:21
*** gouthamr has joined #openstack-ansible20:22
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_rally master: Constraint ansible installation  https://review.openstack.org/52231420:23
andymccrgun1x: not recently, im not sure if there is a better process now20:23
gun1xandymccr: ok so that actually works20:23
andymccrgun1x: if only the journal has failed hten should do yeah. assuming you set the right values there - that command uses like 20GB journals so if youre using something diff it wont20:25
andymccrand device names are correct etc :)20:25
andymccrbut that article is pretty old, so perhaps they have improved that20:25
gun1xandymccr: thanks!20:26
*** hachi_ has quit IRC20:26
cloudnullwhere's leseb when you need him :P20:27
*** hachi_ has joined #openstack-ansible20:28
*** hachi_ has quit IRC20:28
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_zaqar stable/ocata: Initial Zuul v3 role jobs  https://review.openstack.org/51429920:28
*** hamza has joined #openstack-ansible20:29
evrardjpcloudnull: in france20:30
evrardjp:p20:30
evrardjpcloudnull: andymccr because you seem to be there: wanna vote on this one? https://review.openstack.org/#/c/522346/20:31
evrardjpit's about ceph!20:31
cloudnulldone20:32
evrardjpat the same time you could also vote for more ceph things: https://review.openstack.org/#/c/521566/20:32
logan-gun1x: nope. havent had a situation where ive ever needed to recover a journal20:32
bndzorknock in wood20:34
hamzaHello,20:35
hamzacan you please tell me what is the expected time for this to get merged ? https://review.openstack.org/#/c/477017   i mean the idea of isolating using systemd-nspawn20:36
bndzorthats a large commit20:39
bndzorbut i like that idea20:39
*** pbandark has quit IRC20:40
openstackgerritMerged openstack/openstack-ansible-ceph_client stable/ocata: Fix linters  https://review.openstack.org/52234620:43
cloudnullhamza: I need to continue working on that PR20:44
cloudnullif you have thoughts on it I'd really appreciate the feedback20:44
cloudnullit needs a rebase, and I think I can cut out quite a bit of the PR with recent changes being added in.20:45
* cloudnull just needs more hours in the day20:45
bndzorhmm20:46
bndzorhow do i detach a volume ?20:46
cloudnullhamza: quite a few of the parts of that patch have landed in our existing lxc setup20:46
cloudnullso we can do things like have contianer images managed by machinectl20:46
bndzorah, nevermind, had to power down20:48
cloudnullI just wrote a blog post on using machinectl to preseed images on a deployment https://cloudnull.io/2017/11/openstack-ansible-machinectl-image-management/20:48
cloudnull^ shameless plug20:48
cloudnullalso covering the backend storage options https://cloudnull.io/2017/11/storage-setup-for-machinectl/20:49
cloudnullso we can get really close to nspawn now. and hopfully in the very near future we can actually just use nspawn20:50
cloudnullat least as an option.20:50
cloudnullI hope that helps hamza?20:50
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-tests stable/ocata: Depth argument is prome to failure under ansible 2.1  https://review.openstack.org/52238620:53
*** TxGirlGeek has quit IRC20:56
*** viktor_ has quit IRC20:59
*** gouthamr has quit IRC21:03
hamzaThanks cloudnull it helps, i was not aware of those blog posts. Actually, i have been using LXD with btrfs filesystem  and macvlan for networking that can be also used with systemd-nspawn,21:10
cloudnull++ that's where I want to take the nspawn work21:11
hamzawhich adds good features like snapshotting, migration... also macvlan is better in terms of performance21:11
cloudnullthe pr as it stands is using macvlan and machinectl (btrfs).21:12
cloudnullI have that setup running my home cloud on one of the controllers. so far so good.21:12
hamzai did not get any problem using them with LXD, We have Ocata LXC deployment works fine, but i think with nspawn OSA could be better21:14
cloudnullI tend to agree. :D21:15
*** gouthamr has joined #openstack-ansible21:15
cloudnullhamza: https://cloudnull.io/2017/06/nspawning-openstack-ansible/ - also might be of some use from a perspective point of view.21:19
cloudnullits at least what I've been working toward21:19
bndzorrunning setup-everything.yml takes 7 hard years :/21:22
*** xingchao has joined #openstack-ansible21:23
bndzorhmm, i belive ceph reports incorrect numbers in horizon21:26
bndzorLocal Disk Usage21:26
bndzorUsed 280GB of 218.3TB21:26
bndzornow, the 218TB, i got half of that, but it counts each nodes as one because it shows up as local.21:26
cloudnullsetup-eveything can take a while.21:26
cloudnullsounds like you have plenty of space :D21:26
bndzoradding 5 more compute nodes, so i guess its going to count 100*7tb21:26
bndzorwell not so extremly much, 3 storage nodes21:27
bndzor40tb each21:27
cloudnull40TiB ~= 218TiB :P21:27
bndzor40 would be the correct number per node, so 12021:28
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-tests stable/ocata: Depth argument is prome to failure for ansible<2.3  https://review.openstack.org/52238621:28
bndzor/dev/sdb2  208896 93767828622 93767619727 43.7T unknown21:28
bndzorbtw, is anyone running hadoop here? Would you say its ok to run inside nova or should i go for ironic?21:31
hamzathank you cloudnull, you did a goood job. it is well explained :)21:34
cloudnullwe have folks that run hadoop workloads in VMs.21:34
cloudnullit seems to work for them21:34
cloudnullhamza: thanks! if you have any feedback on the PR please let me know21:35
cloudnullbndzor: but I've never run it for any reason other than to play around.21:35
cloudnullso idk if and where the ugly parts are21:36
bndzorah.. im just worried about if i loose lets say 10-15% using it on a compute node or not21:36
bndzorcause i got like 15 of machines for that21:36
bndzorso 10% would be a bit21:36
bndzorand the number, i got from reading some old benchmarks21:37
*** dave-mcc_ has quit IRC21:39
*** threestrands has joined #openstack-ansible21:45
*** threestrands has quit IRC21:45
*** threestrands has joined #openstack-ansible21:45
*** hamza has quit IRC21:51
bndzorlogan-: i think the ceph storage is a bit buggy when displaying in nova -> http://upload.boras.org/upload/files/5a15f8365f61f5a15f8365f6bd.png21:56
*** xingchao has quit IRC21:57
bndzorerr horizon. i need sleep21:59
gun1xbndzor: nice infrastructure.21:59
gun1xwhere's the bug?22:00
bndzorso, im using ceph as storage22:00
bndzorthe total storage size is 109.2TB22:00
bndzorbut it calculates every machine (because it shows up as local)22:00
gun1xoh22:01
gun1xis this pike or queens?22:01
bndzorpike22:01
gun1xthis is bad news22:01
gun1xthere must be some patch in upstream for horizon for this22:02
gun1xprobably 1 line of vode22:02
gun1x*code22:02
gun1xdoes this look ok via openstack api?22:02
bndzorhow would i check that ?22:02
gun1xshould be hypervizor list and after that hypervizor show name22:03
gun1xif you don't know what i am talking about, find the utility container on your controller node and load the rc file with environment variables (should be in home folder of root user) and type "openstack"22:03
bndzordoesnt show any diskspace with hypervisor list --long22:04
bndzor+----+----------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+22:04
bndzor| ID | Hypervisor Hostname        | Hypervisor Type | Host IP       | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB |22:04
bndzor+----+----------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+22:04
bndzor|  5 | appserver2.openstack.local | QEMU            | 172.29.236.52 | up    |          0 |    48 |           2048 |    515883 |22:04
bndzoretc22:04
gun1xbpaste :(22:04
bndzorlooks fine here, get a wider screen! :D22:04
gun1xwhat about hypervisor show spindles22:04
gun1xsorry wrong paste22:05
gun1xwhat about hypervisor show appserver2.openstack.local22:05
gun1xdamn thank god i didn't have something strange in my clipboard22:05
bndzorhaha22:05
bndzordisk_available_least | 11142522:06
bndzorfree_disk_gb         | 11177922:06
*** xingchao has joined #openstack-ansible22:06
bndzorlocal_gb             | 11177922:06
gun1xso this is right22:06
bndzoryup22:06
bndzorchecked another node and its correct there also22:07
gun1xif this doesn't get fixed till i get to prod with this, i'll have to dust off my django skills22:07
bndzorso its only cosmetic22:07
gun1xhttps://www.youtube.com/watch?v=aHaGipVcIJ422:08
gun1xwatching this atm22:08
bndzorfirst thing i saw was redhat so i hit ctrl + w quickly22:09
gun1xomg man :D22:10
*** Dinesh_Bhor has quit IRC22:11
bndzor:)22:12
gun1xbndzor: is this the deploy you were trying to do a few days ago, bndzor ?22:12
bndzoryeah22:13
bndzorits working fine now22:13
bndzortried breaking ceph, adding back ceph nodes, adding in more compute nodes etc, all seems working.. small small things but solved easy22:14
gun1xhow did you deploy ceph?22:14
gun1xwith OSA ?22:14
bndzorhttps://docs.openstack.org/project-deploy-guide/openstack-ansible/latest/app-config-prod-ceph.html22:14
gun1xbndzor: ok, so not with ceph-ansible. everything from top to bottom with openstack-ansible22:15
bndzoryep! :)22:15
bndzorthe only thing i dont like about it is that its slower than a train in india, but atleast it works22:16
gun1xdid you set osd_objectstore: bluestore22:16
gun1xin user variables?22:16
bndzoryep22:16
gun1x"atleast it works" :D22:16
gun1xhow much time did it take to build everything?22:16
bndzor4 hours22:17
bndzorwell, maybe 322:17
bndzorhard to keep track of time hehe22:17
bgmccollumdid you change the number of forks ansible uses? That can speed it up depending on the number of nodes you are deploying to...22:18
bgmccollumi think the default is like 522:18
gun1xbndzor: dude, i find 4 hours to be really ok.22:18
bndzorgun1x: my first openstack hack install was bashscripted with ssh and apt packages, ~10 minutes22:18
bndzorbut, i didnt like to have my own hacks for that22:19
gun1x...22:19
gun1xbndzor: i have a request, i hope you have the time :D22:19
bndzorsup22:19
gun1xcan you please show me df -h on the OSD nodes?22:19
bndzorsure sec22:20
gun1xand also, are the OSD nodes separate from computes?22:20
gun1xor on top of them?22:20
bndzorthey are running on the controllers22:20
bndzoror wait22:20
bndzorosd = storage machines22:20
gun1xyes.22:20
bndzoryeah they run on the storage nodes22:21
gun1xshare your playbook if you can, i'll figure from there22:21
bndzorso i have right now 3 controllers, 7 compute nodes and 3 storage nodes22:21
bndzorFilesystem      Size  Used Avail Use% Mounted on22:21
bndzorudev             32G     0   32G   0% /dev22:21
bndzortmpfs           6.3G   34M  6.3G   1% /run22:21
bndzor/dev/sda1       147G   11G  129G   8% /22:21
bndzortmpfs            32G     0   32G   0% /dev/shm22:21
bndzortmpfs           5.0M     0  5.0M   0% /run/lock22:21
bndzortmpfs            32G     0   32G   0% /sys/fs/cgroup22:21
bndzor/dev/sdb1        94M  5.4M   89M   6% /var/lib/ceph/osd/ceph-322:21
bndzortmpfs           6.3G     0  6.3G   0% /run/user/100022:21
gun1xi am curious because i will need to do the same thing if i really get the hardware i was promised today ..22:21
bndzorthats how it looks like22:21
gun1xdude BPASTE22:21
bndzorwhats bpaste22:21
gun1xhttps://bpaste.net22:22
gun1xa tool that prevents you from spaming irc22:22
bndzorhttps://bpaste.net/show/be635d155b2122:22
bndzorits not 1992 anymore, nobody is gonna get flooded from 10 lines22:22
gun1x:D22:22
bndzorwell, thats how it looks like22:22
gun1xsome channels have rulles about this22:22
bndzorye, but they are stuck in 1992 and still live in the basement @ mommys22:23
gun1xso yea, i will have to do something similar and i always scare about doing anything (even a poc) wihtout testing at home before. i don't have hardware tot est this at hoem :D22:23
bndzorye well its very straight forward22:23
gun1xcan you share your playbook? and lsblk on OSD hosts?22:23
gun1xi mean, how come only one storage device for ceph?22:23
bndzorits litteraly the same one you see on the link i gave you22:23
bndzori have 3 storage nodes22:24
bndzorso, that paste x 322:24
gun1xthat's df -h, not lsblk22:24
bndzorwell, i guess the ceph-X22:24
gun1xlsblk shows all storage devices22:24
gun1xdf -h shows only filesystems22:24
bndzorah my bad22:24
bndzorhttps://bpaste.net/show/b744428e1ea622:24
*** xingchao has quit IRC22:24
bndzorSo, sdb is a raid 0 on each host22:24
gun1xraid 0?22:25
bndzorfirst install was with sdb beeing a raid 1, but since ceph has the redudancy, i dont need to waste space, so i took down the machine, destroyed the raid and did a raid022:25
gun1xlike ... you have 20 TB disks? what?22:25
bndzorthat machine has 40tb in total22:25
gun1xyea, raid 0 is 2 disks stripe22:25
bndzori used to have 20 as it was raided, but removed that since it would waste22:25
bndzornow il just let ceph take care of it heh22:25
gun1xso disks have to be 20TB disks to get 40 TB22:26
bndzorno ?22:26
gun1xbndzor: btw raids are not really supported by ceph, i guess you know that. they would rather see the raid controllers in JBOD mode and let ceph juggle multiple disks22:26
bndzori have like 8 drives x 6 something22:26
bndzorthis is a hardware raid, so its all gucci22:26
bndzorall works fine22:27
bndzor:)22:27
gun1xjust saying22:27
bndzoryup22:27
gun1xso yea, did OSA figure by its own that sda1 is OS and did it use sdb to create sdb1 without manual intervention?22:27
bndzoryeah22:28
gun1xnice22:28
gun1xi guess it uses all discs by default, excluding os disk22:28
bndzoronly, when i took down the machine to remove the raid1 and make it a raid0 i ran into issues22:28
bndzorso i ran the playbook, it failed, ran it again, it worked.22:28
bndzordid the same thing on next storage, same thing22:28
gun1xso this means you have no journals22:28
bndzorthen you need to remove the old osd:s22:28
bndzorsince it creates new22:29
bndzor  services:22:29
bndzor    mon: 3 daemons, quorum controller1-ceph-mon-container-22820959,controller3-ceph-mon-container-98c90950,controller2-ceph-mon-container-cb90beb922:29
bndzor    mgr: controller1-ceph-mon-container-22820959(active), standbys: controller2-ceph-mon-container-cb90beb9, controller3-ceph-mon-container-98c9095022:29
bndzor    osd: 3 osds: 3 up, 3 in22:29
gun1xi see there is journal_size: 10240 within the recommanded user variables... however it doesn't let you specify a device for journal22:30
gun1xmy bad, it has osd_scenario: collocated too22:32
gun1xwith ceph-ansible you need to specify dedicated_devices: for journal22:34
gun1xand devices: for OSDs22:34
gun1xi wonder if this works on OSA22:34
bndzoroh, no idea22:35
bndzorbut im sure logan- knows22:35
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Modify LXC container create to JIT create  https://review.openstack.org/49097622:42
bndzoroh well, gonna head home, bbl22:43
gun1xbndzor: i found it within files on local system, searching on github now to find it there too22:43
gun1xbndzor: https://github.com/ceph/ceph-ansible/blob/master/roles/ceph-osd/defaults/main.yml#L118-L17022:45
*** marst has quit IRC22:46
*** pmannidi has joined #openstack-ansible23:22
*** lbragstad has quit IRC23:23
*** gouthamr has quit IRC23:34
*** cloudtoko has joined #openstack-ansible23:34
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible master: Modify LXC container create to JIT create  https://review.openstack.org/49097623:34
*** cloudtoko has quit IRC23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!