| *** rcernin has quit IRC | 00:17 | |
| *** rcernin has joined #openstack-containers | 00:17 | |
| flwang | brtknr: :) thank you | 00:37 |
|---|---|---|
| *** vesper11 has quit IRC | 03:45 | |
| *** vesper11 has joined #openstack-containers | 03:47 | |
| *** ricolin has joined #openstack-containers | 03:54 | |
| *** ykarel has joined #openstack-containers | 04:52 | |
| *** udesale has joined #openstack-containers | 05:27 | |
| *** namrata has joined #openstack-containers | 07:04 | |
| *** rcernin has quit IRC | 07:09 | |
| *** CosmicSound4 has joined #openstack-containers | 07:14 | |
| *** CosmicSound has quit IRC | 07:15 | |
| *** lpetrut has joined #openstack-containers | 07:16 | |
| *** CosmicSound4 has quit IRC | 07:19 | |
| *** CosmicSound has joined #openstack-containers | 07:24 | |
| *** CosmicSound is now known as cosmicsound | 07:35 | |
| *** cosmicsound has quit IRC | 07:49 | |
| *** cosmicsound has joined #openstack-containers | 07:54 | |
| brtknr | flwang: any luck? | 07:59 |
| *** elenalindq has joined #openstack-containers | 08:00 | |
| namrata | Hi,brtknr this issue was discussed on friday and you asked for kubectl get pods --all-namepsaceshttp://paste.openstack.org/show/786643/The issue which we are facing "When we spin up a magnum Kubernetes cluster.We see all the pods in pending state http://paste.openstack.org/show/786532/ as which means that the cluster has been unable to select a | 08:05 |
| namrata | minion node for them After removing NoSchedule taint from the minion nodes manually we get it working The solution we got online for pods in pending. https://ask.openstack.org/en/question/120442/magnum-kubernetes-noschedule-taint/ the reference link says to add [trust]cluster_user_trust = true to magnum.conf but there is an OSA variable which | 08:05 |
| namrata | sets magnum_cluster_user_trust to true. And this variable is set to true by defaulthttps://opendev.org/openstack/openstack-ansible-os_magnum/commit/aceaa3e5f174af1e40060440c025f49cd1875e0b and we are not overriding it.Can I get help how to get it working without manually untainting nodes." | 08:05 |
| namrata | can you help me with this | 08:05 |
| namrata | brtknr http://paste.openstack.org/show/786643/ | 08:23 |
| *** ykarel is now known as ykarel|lunch | 08:33 | |
| *** namrata has quit IRC | 08:59 | |
| *** ricolin has quit IRC | 09:16 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum master: Bump up prometheus operator chart to 8.2.2 https://review.opendev.org/695443 | 09:30 |
| *** namrata has joined #openstack-containers | 09:33 | |
| *** ykarel|lunch is now known as ykarel | 09:37 | |
| *** namrata has quit IRC | 09:47 | |
| *** namrata has joined #openstack-containers | 10:00 | |
| brtknr | namrata: can you show me the output of `kubectl get all -A` and the content of /etc/sysconfig/heat-params | 10:11 |
| brtknr | i can help you but a lot of the issues you are reporting have been fixed in stein/train/master so again, i will reiterate running a more recent release of magnum... it has a very loose coupling with other openstack services so it is not going to break things... we are running magnum "train 9.1.0" release with openstack "rocky" without any issues... | 10:13 |
| brtknr | i can help you but a lot of the issues you are reporting have been fixed in stein/train/master so again, i will reiterate running a more recent release of magnum... it has a very loose coupling with other openstack services so it is UNLIKELY to break things... we are running magnum "train 9.1.0" release with openstack "rocky" without any issues... | 10:14 |
| brtknr | ...after having lots of issues running the latest version of kubernetes on rocky | 10:17 |
| namrata | brtknr `kubectl get all -A` http://paste.openstack.org/show/786651/ and content of /etc/sysconfig/heat-params on minion http://paste.openstack.org/show/786652/ | 10:18 |
| namrata | brtknr thanks for the suggestion we are discussing on our plan to upgrade our regions but until then magnum rocky not working is affecting many things. | 10:21 |
| brtknr | How do you deploy openstack? Openstack ansible? | 10:23 |
| brtknr | namrata: | 10:23 |
| namrata | yes openstack ansibe | 10:23 |
| namrata | *ansible | 10:24 |
| brtknr | namrata: ask in #openstack-ansible channel how to upgrade magnum train, that will be the quickest way to fix your problem if you need up to date kubernetes | 10:25 |
| brtknr | If you are going to wait for a new rocky release, you will be waiting for a long time as none of the maintainers are using magnum rocky in prod | 10:26 |
| brtknr | we are running stein/train on all customer sites now | 10:26 |
| brtknr | e.g I don’t know how long it takes with OSA, but with kolla, upgrading magnum to train takes 10 mins max if using upstream image | 10:28 |
| namrata | brtknr i underdstand your point but these are the issues of broken images as for me v 1.10.11-1 works but other like v 1.14.8 and v1.13.12 dosent work | 10:30 |
| brtknr | I suggest using the default kube_tag for rocky as only those have the guarantee of being supported. Kubernetes has fast dev cycle, we try to keep up to support latest kube_tag.. as a result, magnum releases are more tightly coupled with kubernetes version than openstack releases | 10:33 |
| brtknr | If you don’t want to run latest magnum, I suggest using something like kubespray which will allow you to run the latest version of k8s | 10:35 |
| namrata | brtknr thanks for the information but clearly default kube_tag for rocky is exposed to billion laughs which is not ideal. | 10:36 |
| brtknr | namrata: looking at your paste, i do not see openstack-cloud-controller-manager daemonset running... | 10:42 |
| brtknr | i'd suggest trying with cloud_provider_enabled=false label but I cant think of anything else to suggest beyond that | 10:42 |
| brtknr | elenalindq: ^ | 10:47 |
| elenalindq | brtknr Roger, that. | 10:47 |
| brtknr | namrata: elenalindq: I just asked the OSA folks about upgrading magnum to train on openstack deployed using OSA, they say it can be done: | 10:49 |
| brtknr | brtknr | noonedeadpunk: we are running magnum train with openstack rocky deployed using kolla-ansible... was | 10:49 |
| brtknr | | wondering if the same could be done with OSA | 10:49 |
| brtknr | noonedeadpunk | yeah, you can define some exact version of magnum or even install it from your own repo | 10:49 |
| brtknr | brtknr | magnum has loose dependency with other openstack services.... magnum train using heat templates from | 10:49 |
| brtknr | | queens | 10:49 |
| brtknr | noonedeadpunk | In that case it may theoretically work... | 10:50 |
| *** pcaruana has joined #openstack-containers | 11:14 | |
| *** namrata has quit IRC | 11:18 | |
| *** namrata has joined #openstack-containers | 11:25 | |
| namrata | brtknr setting `cloud_provider_enabled=false` gives me a failed cluster with status reason `"cinder" volume driver needs "cloud_provider_enabled" label to be true or unset.` | 11:27 |
| namrata | so its not working | 11:27 |
| brtknr | namrata: so try setting volume size to 0? | 11:28 |
| brtknr | its because you have volume driver set to "cinder" | 11:29 |
| brtknr | https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/heat/k8s_fedora_template_def.py#L100 | 11:29 |
| *** openstack has joined #openstack-containers | 11:53 | |
| *** ChanServ sets mode: +o openstack | 11:53 | |
| brtknr | namrata: not docker volume size, the volume_driver | 12:10 |
| brtknr | namrata: not docker volume size, the volume_driver in cluster template | 12:10 |
| namrata | okay | 12:11 |
| *** dave-mccowan has joined #openstack-containers | 12:30 | |
| *** dave-mccowan has quit IRC | 12:35 | |
| namrata | brtknr I get a CREATE_COMPLETE cluster but no pods http://paste.openstack.org/show/786658/ | 12:37 |
| *** namrata has quit IRC | 12:40 | |
| *** namrata has joined #openstack-containers | 12:41 | |
| *** lpetrut has joined #openstack-containers | 12:53 | |
| *** goldyfruit has quit IRC | 13:11 | |
| *** goldyfruit has joined #openstack-containers | 13:14 | |
| *** goldyfruit has quit IRC | 13:19 | |
| *** KeithMnemonic has joined #openstack-containers | 13:54 | |
| *** ykarel_ is now known as ykarel | 14:27 | |
| *** KeithMnemonic has quit IRC | 14:33 | |
| *** goldyfruit has joined #openstack-containers | 14:38 | |
| *** spiette has joined #openstack-containers | 14:53 | |
| *** goldyfruit has quit IRC | 14:53 | |
| *** goldyfruit has joined #openstack-containers | 14:55 | |
| *** goldyfruit_ has joined #openstack-containers | 14:59 | |
| *** ykarel_ has joined #openstack-containers | 15:01 | |
| *** goldyfruit has quit IRC | 15:01 | |
| *** ykarel has quit IRC | 15:04 | |
| *** namrata has quit IRC | 15:19 | |
| *** namrata has joined #openstack-containers | 15:45 | |
| *** ykarel_ is now known as ykarel|away | 15:49 | |
| *** namrata has quit IRC | 16:04 | |
| *** openstackstatus has joined #openstack-containers | 16:14 | |
| *** ChanServ sets mode: +v openstackstatus | 16:14 | |
| *** elenalindq has quit IRC | 16:27 | |
| *** goldyfruit___ has joined #openstack-containers | 16:33 | |
| *** goldyfruit_ has quit IRC | 16:36 | |
| *** elenalindq has joined #openstack-containers | 16:42 | |
| *** lpetrut has quit IRC | 16:51 | |
| *** goldyfruit_ has joined #openstack-containers | 16:55 | |
| *** goldyfruit___ has quit IRC | 16:57 | |
| *** udesale has quit IRC | 16:57 | |
| *** FlorianFa has quit IRC | 17:29 | |
| andrein | Hello, I've just upgraded our openstack deployment from stein to train and I think I (partially) broke Magnum. I'm getting the following error when updating cluster templates: | 18:20 |
| andrein | https://www.irccloud.com/pastebin/TQ1bAz6A/ | 18:20 |
| andrein | looking at the history, it looks like the /hidden attribute is non-removable since ~10 months ago, could this actually be a horizon bug? | 18:23 |
| *** ykarel|away has quit IRC | 18:42 | |
| *** colin- has joined #openstack-containers | 20:24 | |
| flwang | andrein: still round? the hidden issue is a known one | 20:37 |
| flwang | andrein: can you login into your db can check the value of the 'hidden' for templates table? | 20:38 |
| flwang | brtknr: no, no luck for the api issue, i'm still working on that | 20:40 |
| andrein | flwang: checking now | 20:40 |
| andrein | flwang: all my cluster templates show "hidden: 0" | 20:43 |
| flwang | ok, then could you pls clarify what's your current probnlem? | 20:44 |
| flwang | can't update template? | 20:45 |
| andrein | yes | 20:45 |
| andrein | it used to work on stein, though | 20:45 |
| flwang | what's your current version of magnum client? | 20:46 |
| andrein | I'm using the horizon dashboard, was just installing magnum client to try it this way | 20:47 |
| andrein | it works with python-magnumclient==2.15.0 | 20:48 |
| flwang | andrein: just to be clear, so now the CLI works for you, but the dashboard still doesn't work? | 20:50 |
| andrein | yes, that's correct | 20:50 |
| flwang | andrein: did you set the microversion in local_settings.py for horizon? | 20:52 |
| andrein | that's a good question, I don't think so. I deployed this with kolla-ansible. | 20:53 |
| andrein | I'll check the horizon container in a sec | 20:53 |
| andrein | what am I looking for in local_settings.py? | 20:55 |
| flwang | OPENSTACK_API_VERSIONS | 20:59 |
| andrein | OPENSTACK_API_VERSIONS = {"identity": 3} | 21:01 |
| andrein | that's all I have there | 21:01 |
| *** elenalindq has quit IRC | 21:02 | |
| andrein | flwang: the service catalog has "http://172.16.0.250:9511/v1" listed as the magnum endpoint | 21:03 |
| flwang | OPENSTACK_API_VERSIONS = {"identity": 3, "container-infra": "latest"} try this | 21:04 |
| andrein | flwang: this might be complicated to set up, that variable is not overridable in kolla-ansible | 21:08 |
| flwang | andrein: ok, then i will leave it to you :) that's the thing you need to check and test | 21:10 |
| andrein | flwang: actually, it's overridable, deploying it now | 21:12 |
| andrein | flwang: still no luck, same error. The response from horizon is "HTTPResponse instance has no attribute 'headers'", status code 500. no longer see anything in the magnum logs. | 21:22 |
| andrein | so I think it's not getting past horizon | 21:22 |
| flwang | do you mean the 500 error is in horizon or magnum? | 21:23 |
| andrein | horizon throws 500, magnum used to throw 400 (as I posted above) | 21:24 |
| andrein | now I don't see anything at all in magnum's log | 21:24 |
| flwang | ok, did you run 'python manage.py collectstatic' and then compress ? | 21:24 |
| flwang | you shouldn't need this, but with the new change, you shouldn't see any error caused by the new config | 21:25 |
| flwang | i'm not familiar with your horizon deployment way, but at least you need to restart the horizon | 21:26 |
| andrein | I deployed a new container with the new setting | 21:26 |
| andrein | using kolla-ansible | 21:27 |
| andrein | I think I have a few stacktraces in horizon's log now | 21:27 |
| flwang | then you can check your horizon log see why | 21:27 |
| flwang | if you can get correct response with cli, that means magnum server is working as expected at least | 21:28 |
| andrein | flwang: here's the exception, I can't make much sense of it unfortunately https://www.irccloud.com/pastebin/R9qpjZWz/ | 21:29 |
| *** rcernin has joined #openstack-containers | 21:31 | |
| flwang | andrein: can you see the magnum client version of your horizon env? | 21:31 |
| andrein | flwang: the horizon container also has python-magnumclient=2.15.0 | 21:32 |
| flwang | andrein: sorry, i don't have any clue with the exception, you may have to add some break points to understand why | 21:35 |
| *** pcaruana has quit IRC | 21:38 | |
| andrein | flwang: ok, figured it out. there are actually no changes, my elasticsearch cluster was just slow. magnum still throws a 400 response with: Client-side error: '/hidden' is a mandatory attribute | 21:44 |
| brtknr | flwang: the api issue doesn’t exist on master branch, maybe you missed a backport to your special stein branch | 22:18 |
| flwang | brtknr: good point, i will take another look | 22:29 |
| flwang | brtknr: btw, you mentioned that you're using stable/train right? | 22:29 |
| flwang | did you see any issue related to node group? | 22:29 |
| *** KeithMnemonic has joined #openstack-containers | 22:41 | |
| *** cosmicsound has quit IRC | 23:45 | |
| *** cosmicsound has joined #openstack-containers | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!