*** macz_ has quit IRC | 00:02 | |
*** gyee has quit IRC | 00:03 | |
*** ianychoi has quit IRC | 00:03 | |
*** lemko has quit IRC | 00:03 | |
*** grabes has quit IRC | 00:03 | |
*** jpward has quit IRC | 00:03 | |
*** dwilde has quit IRC | 00:03 | |
*** cloudnull has quit IRC | 00:03 | |
*** ThiagoCMC has quit IRC | 00:03 | |
*** openstackgerrit has quit IRC | 00:03 | |
*** irclogbot_1 has quit IRC | 00:03 | |
*** antonym has quit IRC | 00:03 | |
*** spotz has quit IRC | 00:03 | |
*** hindret has quit IRC | 00:03 | |
*** rh-jelabarre has quit IRC | 00:03 | |
*** bl0m1 has quit IRC | 00:03 | |
*** guilhermesp has quit IRC | 00:03 | |
*** jungleboyj has quit IRC | 00:03 | |
*** fyx has quit IRC | 00:03 | |
*** persia has quit IRC | 00:03 | |
*** mubix has quit IRC | 00:03 | |
*** melwitt has quit IRC | 00:03 | |
*** snadge has quit IRC | 00:03 | |
*** Carcer has quit IRC | 00:03 | |
*** fridtjof[m] has quit IRC | 00:03 | |
*** Adri2000 has quit IRC | 00:03 | |
*** rh-jelabarre has joined #openstack-ansible | 00:04 | |
*** hindret has joined #openstack-ansible | 00:05 | |
*** snadge has joined #openstack-ansible | 00:05 | |
*** bl0m1 has joined #openstack-ansible | 00:05 | |
*** guilhermesp has joined #openstack-ansible | 00:05 | |
*** jungleboyj has joined #openstack-ansible | 00:05 | |
*** Carcer has joined #openstack-ansible | 00:05 | |
*** fyx has joined #openstack-ansible | 00:05 | |
*** persia has joined #openstack-ansible | 00:05 | |
*** mubix has joined #openstack-ansible | 00:05 | |
*** melwitt has joined #openstack-ansible | 00:05 | |
*** fridtjof[m] has joined #openstack-ansible | 00:05 | |
*** Adri2000 has joined #openstack-ansible | 00:05 | |
*** dwilde has joined #openstack-ansible | 00:06 | |
*** gyee has joined #openstack-ansible | 00:06 | |
*** cloudnull has joined #openstack-ansible | 00:06 | |
*** ThiagoCMC has joined #openstack-ansible | 00:06 | |
*** ianychoi has joined #openstack-ansible | 00:06 | |
*** lemko has joined #openstack-ansible | 00:06 | |
*** grabes has joined #openstack-ansible | 00:06 | |
*** spotz has joined #openstack-ansible | 00:06 | |
*** jpward has joined #openstack-ansible | 00:06 | |
*** irclogbot_1 has joined #openstack-ansible | 00:06 | |
*** antonym has joined #openstack-ansible | 00:06 | |
*** ioni has quit IRC | 00:06 | |
*** masterpe has quit IRC | 00:07 | |
*** bl0m1 has quit IRC | 00:07 | |
*** fridtjof[m] has quit IRC | 00:07 | |
*** csmart has quit IRC | 00:07 | |
*** tosky has quit IRC | 00:07 | |
*** cyberpear has quit IRC | 00:09 | |
*** guilhermesp has quit IRC | 00:09 | |
*** jungleboyj has quit IRC | 00:09 | |
*** fyx has quit IRC | 00:09 | |
*** jungleboyj has joined #openstack-ansible | 00:09 | |
*** guilhermesp has joined #openstack-ansible | 00:10 | |
*** irclogbot_1 has quit IRC | 00:10 | |
*** fyx has joined #openstack-ansible | 00:11 | |
*** irclogbot_0 has joined #openstack-ansible | 00:11 | |
*** cyberpear has joined #openstack-ansible | 00:11 | |
*** spatel has joined #openstack-ansible | 00:12 | |
*** bl0m1 has joined #openstack-ansible | 00:17 | |
*** spatel has quit IRC | 00:17 | |
*** csmart has joined #openstack-ansible | 00:33 | |
*** ioni has joined #openstack-ansible | 00:42 | |
*** ioni has quit IRC | 00:51 | |
*** csmart has quit IRC | 00:51 | |
*** csmart has joined #openstack-ansible | 01:03 | |
*** fridtjof[m] has joined #openstack-ansible | 01:27 | |
*** ioni has joined #openstack-ansible | 01:27 | |
*** masterpe has joined #openstack-ansible | 01:27 | |
*** cloudnull has quit IRC | 01:49 | |
*** maharg101 has joined #openstack-ansible | 02:16 | |
*** spatel has joined #openstack-ansible | 02:17 | |
*** maharg101 has quit IRC | 02:21 | |
*** openstackgerrit has joined #openstack-ansible | 02:34 | |
openstackgerrit | Satish Patel proposed openstack/openstack-ansible-os_senlin master: This is typo error causing issue to re-install senlin https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/770878 | 02:34 |
---|---|---|
*** cloudnull has joined #openstack-ansible | 02:35 | |
*** ianychoi has quit IRC | 02:42 | |
*** cloudnull has quit IRC | 02:46 | |
*** jbadiapa has quit IRC | 03:16 | |
*** cloudnull has joined #openstack-ansible | 03:21 | |
*** cloudnull has quit IRC | 03:53 | |
*** cloudnull has joined #openstack-ansible | 03:55 | |
*** cloudnull has quit IRC | 04:02 | |
*** cloudnull has joined #openstack-ansible | 04:05 | |
*** gyee has quit IRC | 04:12 | |
*** maharg101 has joined #openstack-ansible | 04:17 | |
*** maharg101 has quit IRC | 04:22 | |
*** ianychoi has joined #openstack-ansible | 04:29 | |
*** dave-mccowan has quit IRC | 04:40 | |
*** evrardjp has quit IRC | 05:33 | |
*** evrardjp has joined #openstack-ansible | 05:35 | |
*** pto has joined #openstack-ansible | 06:05 | |
*** pto has quit IRC | 06:06 | |
*** pto has joined #openstack-ansible | 06:07 | |
*** pto_ has joined #openstack-ansible | 06:15 | |
*** pto has quit IRC | 06:18 | |
*** maharg101 has joined #openstack-ansible | 06:18 | |
*** pto_ has quit IRC | 06:20 | |
*** pto has joined #openstack-ansible | 06:20 | |
*** maharg101 has quit IRC | 06:23 | |
*** miloa has joined #openstack-ansible | 06:38 | |
*** spatel has quit IRC | 07:03 | |
*** pcaruana has joined #openstack-ansible | 07:29 | |
*** jbadiapa has joined #openstack-ansible | 07:34 | |
*** jbadiapa has quit IRC | 07:39 | |
*** rpittau|afk is now known as rpittau | 08:09 | |
*** andrewbonney has joined #openstack-ansible | 08:12 | |
*** MickyMan77 has quit IRC | 08:33 | |
*** tosky has joined #openstack-ansible | 08:42 | |
*** maharg101 has joined #openstack-ansible | 08:53 | |
*** spatel has joined #openstack-ansible | 09:16 | |
*** spatel has quit IRC | 09:21 | |
ebbex | 21:07 < jrosser> well this was the most obvious and smallest change to make https://review.opendev.org/q/topic:%22osa-new-pip%22+(status:open%20OR%20status:merged) | 09:32 |
jrosser | ebbex: morning! | 09:37 |
ebbex | Apologies, catching up in the chat, checking out links. Apparently pasted in the buffer. | 09:38 |
ebbex | Though I remeber looking at this https://review.opendev.org/c/openstack/openstack-ansible-repo_build/+/348775 for some reason. | 09:40 |
ebbex | I'm getting nowhere with the https://review.opendev.org/c/openstack/openstack-ansible-ops/+/741997 stuff, i can see the uri request go into kibana/nodejs, out to elasticsearch, then disappear. And then timeout. | 09:43 |
*** macz_ has joined #openstack-ansible | 10:07 | |
*** macz_ has quit IRC | 10:12 | |
kleini | How is it possible to configure a common cinder volume backend host name for different cinder volume instances in different cinder volume containers on infra hosts? I use Ceph as backend and all cinder volumes should be able to deal with all Ceph volumes. | 10:12 |
*** bbezak has joined #openstack-ansible | 10:56 | |
admin0 | kleini, your question is not clear | 10:57 |
admin0 | you have different pools you want to expose ? | 10:57 |
admin0 | because the volume backend name is same for all | 10:57 |
kleini | Unfortunately the host name is part of the cinder volume instance and that goes into the host value of the volume. This hostname contains the container IDs | 11:02 |
kleini | http://paste.openstack.org/show/801656/ <- see the os-vol-host-attr:host value | 11:04 |
kleini | so the controller2-cinder-volumes-container-640d1209@rbd instance will not feel responsible for the volume | 11:05 |
admin0 | hmm.. never looked into this . | 11:09 |
admin0 | so if the original container is down, it stops to work ? | 11:09 |
*** MrClayPole has left #openstack-ansible | 11:10 | |
jrosser | kleini: there is a trail for this that starts here https://github.com/openstack/openstack-ansible-os_cinder/commit/c148d77e29af6faebc1c9b012ae08aed447cd179 | 11:24 |
kleini | thanks jrosser. that looks promising but first I need to fix my not working additional compute | 11:27 |
admin0 | jrosser, how do i use this ? just cherry-pick it ? | 11:28 |
jrosser | i don't understand, sorry | 11:30 |
admin0 | sorry .. i understand it incrorrectly | 11:30 |
kleini | admin0: you need to check, whether the code of that commit is in your /etc/ansible/roles/os_cinder role on the deployment host. if not, you need to consider to cherry pick that | 11:31 |
kleini | depends on where that change is released and backported | 11:32 |
admin0 | my latest one with ceph is on 21.2.0 .. its not there | 11:35 |
admin0 | 21.2.1 sorry | 11:35 |
admin0 | i guess i can just copy this file in the folder and rerun it | 11:36 |
CeeMac | on a semi-related note | 11:36 |
CeeMac | is there a way to configure osa to deploy cinder-volume in HA without using ceph? | 11:36 |
CeeMac | i don't mind active/passive if there are restrictions | 11:36 |
*** bbezak has left #openstack-ansible | 11:37 | |
jrosser | kleini: http://paste.openstack.org/show/801658/ | 11:37 |
jrosser | what gets put in os-vol-host-attr:host sure is confusing though | 11:38 |
kleini | doesn't matter if all volume instances feel responsible for these Ceph volumes | 11:42 |
kleini | jrosser: there is an additional step in the scripts/add-compute.sh that is not in the documentation for ussuri for scaling the cloud. will try to provide a fix | 11:44 |
kleini | playbooks/openstack-hosts-setup.yml | 11:45 |
*** macz_ has joined #openstack-ansible | 11:58 | |
*** macz_ has quit IRC | 12:03 | |
*** jamesdenton has quit IRC | 12:07 | |
*** jamesdenton has joined #openstack-ansible | 12:07 | |
*** mgariepy has quit IRC | 12:14 | |
*** spatel has joined #openstack-ansible | 12:17 | |
*** spatel has quit IRC | 12:21 | |
*** pto has quit IRC | 12:53 | |
*** mgariepy has joined #openstack-ansible | 13:02 | |
*** jbadiapa has joined #openstack-ansible | 13:06 | |
kleini | I still have no luck with my additional compute node. Either placement or Nova scheduler is not considering this new compute node. Node is listed properly in resource providers, compute services and in cell_v2. I need to take care of different tasks now. Maybe somebody has an idea, what can cause this. Otherwise I have to debug placement and Nova scheduler. | 13:20 |
*** spatel has joined #openstack-ansible | 13:48 | |
spatel | noonedeadpunk: how did you verify this patch? - https://review.opendev.org/c/openstack/senlin/+/749874 | 13:50 |
spatel | its not working for me. | 13:50 |
noonedeadpunk | what do you mean under not working? | 13:51 |
spatel | I have dedicated HAProxy box and in that scenario senlin try to talk to external interface of HAproxy and not able to communicate because of asymmetric routing | 13:52 |
spatel | so i use your patch to tell interface=internal but still its using public interface for keystone. (i can see that in tcpdump) | 13:53 |
spatel | look like senlin acting like normal end user which mostly use external/public endpoint to talk to services. | 13:54 |
spatel | If you look at other services they have dedicated section for neutron/nova to define endpoint but in senlin it try to discover itself just like end users. | 13:55 |
spatel | noonedeadpunk: ^^ | 13:56 |
spatel | I am going to play in lab to collect more data to understand what is going. | 13:57 |
spatel | by the way i found my issue related senlin re-install :) patch is here - https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/770878 | 13:57 |
noonedeadpunk | hm.... I think I targeted my patch to auth only through private... | 13:58 |
noonedeadpunk | But never checked what endpoints it gets when asks for them | 13:58 |
spatel | I have noticed in tcpdump it use internal interface to collect all endpoint but then try to use public endpoint to talk to services | 13:58 |
noonedeadpunk | so yeah, I could have missed smth | 13:59 |
noonedeadpunk | yeah, I see, doh... | 13:59 |
noonedeadpunk | Well, I was testing just different thing :( | 13:59 |
spatel | last night i was talking to erik about that in senlin channel if you see the history | 14:00 |
noonedeadpunk | yeah, see it | 14:01 |
spatel | This is race condition where if you deploy Haproxy on dedicated box then you will hit this bug :) | 14:02 |
noonedeadpunk | have you tried to adjust sdk? | 14:02 |
admin0 | kleini, jrosser .. is it safe to use https://github.com/openstack/openstack-ansible-os_cinder/blob/c148d77e29af6faebc1c9b012ae08aed447cd179/templates/cinder.conf.j2 for new builds ? nothing else required to do ? | 14:02 |
spatel | Yes i did but no impact | 14:02 |
noonedeadpunk | I see | 14:02 |
spatel | i will try again with fresh lab and see | 14:02 |
*** jbadiapa has quit IRC | 14:03 | |
noonedeadpunk | well, I guess it needs checking out senlin code again to see where it goes wrong, and honestly having no time at the moment for this.... | 14:03 |
noonedeadpunk | But abolutely sure it's smth that needs fixing | 14:03 |
noonedeadpunk | as it's weird they don't have that for so long | 14:03 |
spatel | noonedeadpunk: no big deal :) | 14:05 |
*** jbadiapa has joined #openstack-ansible | 14:05 | |
spatel | this is only impact if someone deploy haproxy on dedicated box otherwise everything should work fine. | 14:05 |
noonedeadpunk | well, talking to public endpoint might be a big deal actually | 14:06 |
noonedeadpunk | that means that things are exposed | 14:06 |
jrosser | senlin is on our to-do list and the senlin service has no route at all to the public endpoint | 14:09 |
jrosser | *in our deployment | 14:09 |
spatel | +1 | 14:14 |
*** miloa has quit IRC | 14:15 | |
spatel | noonedeadpunk: can we get review/vote on this patch? - https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/770878 | 14:16 |
openstackgerrit | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_senlin master: DNM - test patch for senlin tempest testing https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/754045 | 14:25 |
*** dave-mccowan has joined #openstack-ansible | 14:27 | |
kleini | admin0: I don't know. I just discovered, that two cinder volume instances can not take care of others instance created volumes | 14:42 |
admin0 | kleini, thanks for the update .. | 14:43 |
kleini | I can test that in a staging environment but need to take care now of other tasks | 14:44 |
andrewbonney | spatel: Thanks for sharing your neutron policy yesterday. We've worked around our issue as follows if it's useful: http://paste.openstack.org/show/801662/ | 14:46 |
spatel | andrewbonney: beautiful, so i can add that snippet in my user_variable.yml right? | 14:47 |
andrewbonney | Yeah, obviously just swapping in anything you've changed from the defaults in your own policy file | 14:48 |
*** rpittau is now known as rpittau|afk | 14:48 | |
spatel | awesome! i will try it out. we should add that snippet in example config of OSA :) | 14:48 |
noonedeadpunk | will V bump ever merge.... | 14:52 |
*** pcaruana has quit IRC | 14:56 | |
openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Use PCI_SLOT_NAME cut instead of ls -ld cut more precisely https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/769700 | 15:05 |
*** pcaruana has joined #openstack-ansible | 15:06 | |
CeeMac | noonedeadpunk: hi | 15:29 |
noonedeadpunk | o/ | 15:29 |
CeeMac | noonedeadpunk: we're trying to work out if there's been any work on cinder-volume HA outwith ceph and so far it doesn't look like it. | 15:30 |
CeeMac | have you much experience with that side of things? I know you've been looking at pacemaker for masakari | 15:30 |
*** pcaruana has quit IRC | 15:31 | |
noonedeadpunk | nah, I was using only ceph and nfs as cinder-volume backends... | 15:31 |
noonedeadpunk | and for both of them you don't need anything fancy | 15:32 |
CeeMac | ah, ok thanks | 15:32 |
CeeMac | just trying to work through some operational processes and understand cinder a bit more with someone who has joined my team | 15:32 |
CeeMac | we're just covering off the volume management aspect, and the impact of the cinder-volume service being unavailable during a storage node reboot or maintenance event | 15:33 |
CeeMac | and wondering how big an issue (or not) not having HA for cinder-volume is | 15:34 |
CeeMac | i saw active-active was introduced for ceph in stein | 15:34 |
noonedeadpunk | if you have shared filesystem, I guess you can probably just set the same `volume_backend_name` across cinder-volumes | 15:34 |
CeeMac | also trying to work out how best to implement active/passive with external iscsi | 15:35 |
noonedeadpunk | which will make cinder-volume to work in active/passive i guess | 15:35 |
CeeMac | trying to work out where the backend_host fits in | 15:35 |
CeeMac | well, we have the same backend name across the cinder nodes, but the host name is obviously unique | 15:36 |
noonedeadpunk | but actually I see nothing wrong in active/passive since in DB you have volumes owned by the same name, so in case of failure of one service, another one will be able to pick up operations for the volume | 15:36 |
CeeMac | so host1@backend#pool, host2@backend#pool etc | 15:36 |
noonedeadpunk | `volume_backend_name` will set that host1 to smth unique | 15:37 |
CeeMac | is that the same as the backen_host then, or something different | 15:37 |
CeeMac | its as simple as that, no need for pacemaker on cinder nodes? | 15:37 |
CeeMac | the cinder docs are a little confusing/misleading | 15:38 |
noonedeadpunk | so with active/passive on ceph I had smth like that http://paste.openstack.org/show/801670/ | 15:38 |
noonedeadpunk | oh, sorry, not `volume_backend_name` but `backend_host` | 15:39 |
CeeMac | ah, ok, good, that makes more sense then! | 15:39 |
CeeMac | so you would have had backend_host=rbd | 15:39 |
noonedeadpunk | yep, exactly | 15:39 |
noonedeadpunk | I think the maint point here is that storage should be available acroos cinder-volume instances | 15:40 |
CeeMac | yeah | 15:40 |
CeeMac | we have that | 15:40 |
CeeMac | it was just the unique service names that was throwing me | 15:41 |
noonedeadpunk | yeah | 15:41 |
noonedeadpunk | an cinder-manage has cli that allows to migrate volume to the new name in DB | 15:41 |
CeeMac | so then, if we had the backend_host configured right for active/passive, and the backend type set right to group the backens | 15:42 |
CeeMac | everything is pretty resilient | 15:42 |
noonedeadpunk | Yeah, I guess so | 15:42 |
CeeMac | thats the 'cinder-manage volume update_host' then? | 15:42 |
noonedeadpunk | yeah, exactly | 15:43 |
CeeMac | cool, i'll lab it up and see what breaks :) | 15:52 |
CeeMac | thanks noonedeadpunk | 15:52 |
CeeMac | if i added that backend_host in cinder.conf and restarted cinder-volume would it automatically create the new volume service with the new name? | 15:53 |
CeeMac | s/would/should | 15:54 |
*** mgariepy has quit IRC | 15:55 | |
*** MrClayPole has joined #openstack-ansible | 16:02 | |
CeeMac | noonedeadpunk: quick follow up question, how do the storage nodes which is active and which is passive for each backend | 16:07 |
*** macz_ has joined #openstack-ansible | 16:09 | |
noonedeadpunk | yes, it should create new service automatically | 16:10 |
noonedeadpunk | well, in perfect condition, iirc they're using zookeeper for this | 16:11 |
noonedeadpunk | for coordination | 16:11 |
noonedeadpunk | But they still have some source node name or smth while messaging.. Tbh I'm not sure here. Maybe it even does not matter which cinder-volume will perform an acton since from DB and volume side that should not matter much | 16:12 |
*** mgariepy has joined #openstack-ansible | 16:42 | |
*** maharg101 has quit IRC | 17:24 | |
*** klamath_atx has quit IRC | 17:33 | |
*** dave-mccowan has quit IRC | 17:38 | |
CeeMac | yeah i was trying to work out if we need some kind of cluster implementation or not | 17:53 |
CeeMac | the cinder HA documentation mentions pacemaker but I don't see any examples for cinder-volume active/passive | 17:53 |
CeeMac | thanks again noonedeadpunk | 18:00 |
*** klamath_atx has joined #openstack-ansible | 18:02 | |
noonedeadpunk | I have no idea why pacemaker should be used... | 18:02 |
noonedeadpunk | i believe there might be scenarious... but dunno | 18:07 |
*** gyee has joined #openstack-ansible | 18:14 | |
CeeMac | yeah its a mystery | 18:15 |
CeeMac | i'll try it out on monday in my lab and see if I can work it through | 18:17 |
*** andrewbonney has quit IRC | 19:00 | |
*** spatel has quit IRC | 19:03 | |
openstackgerrit | Merged openstack/openstack-ansible stable/victoria: Bump Victoria for the release https://review.opendev.org/c/openstack/openstack-ansible/+/770073 | 19:46 |
noonedeadpunk | thanks god | 19:49 |
noonedeadpunk | I thought that would never happen.... | 19:50 |
jrosser | \o/ awesome | 19:52 |
*** jbadiapa has quit IRC | 22:09 | |
*** logan- has quit IRC | 22:40 | |
*** ioni has quit IRC | 22:40 | |
*** logan- has joined #openstack-ansible | 22:53 | |
*** mwhahaha has quit IRC | 22:57 | |
*** mwhahaha has joined #openstack-ansible | 22:57 | |
*** simondodsley has quit IRC | 22:58 | |
*** simondodsley has joined #openstack-ansible | 22:59 | |
*** rh-jelabarre has quit IRC | 22:59 | |
*** rh-jelabarre has joined #openstack-ansible | 22:59 | |
*** ioni has joined #openstack-ansible | 23:00 | |
*** bl0m1 has quit IRC | 23:01 | |
*** jrosser has quit IRC | 23:01 | |
*** cyberpear has quit IRC | 23:01 | |
*** bl0m1 has joined #openstack-ansible | 23:02 | |
*** cyberpear has joined #openstack-ansible | 23:03 | |
*** jrosser has joined #openstack-ansible | 23:03 | |
*** logan- has quit IRC | 23:04 | |
*** logan- has joined #openstack-ansible | 23:18 | |
*** kleini has quit IRC | 23:20 | |
*** johanssone has quit IRC | 23:20 | |
*** crazzy has quit IRC | 23:20 | |
*** crazzy has joined #openstack-ansible | 23:21 | |
*** johanssone has joined #openstack-ansible | 23:24 | |
*** dwilde has quit IRC | 23:49 | |
*** dwilde has joined #openstack-ansible | 23:50 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!