| *** goldyfruit_ has joined #openstack-containers | 00:06 | |
| *** sapd1_x has joined #openstack-containers | 00:19 | |
| *** sapd1_x has quit IRC | 01:50 | |
| jakeyip | FYI: we just upgraded keystone to Rocky today and ran into problems with magnum. The symptoms are 503 in nova from occm. This is due to occm using a cached queens keystone token that is for some reason no longer valid in rocky keystone | 02:48 |
|---|---|---|
| jakeyip | restarting minion generates a new token and fixes the problem. for clusters we don't control we are hoping to ride this out since occm asks for a new token every 6 hrs. | 02:49 |
| flwang | jakeyip: what do you mean 'restart minion generates new token'? | 03:44 |
| *** sapd1_x has joined #openstack-containers | 03:45 | |
| jakeyip | flwang: the error was something like this on the minion `Sep 18 01:47:52 kubedemo-qvk4skhj24o4-minion-0 runc[3646]: W0918 01:47:52.525502 3657 kubelet_node_status.go:974] Failed to set some node status fields: failed to get node address from cloud provider: The service is currently unable to handle the request due to a temporary overloading or maintenance. This is a temporary condition. Try again later.` | 03:45 |
| flwang | but there is no token on the minion, but the credentials of the trustee user | 03:47 |
| flwang | so i don't understand why you need to restart the minion node | 03:47 |
| flwang | did you try to restart the kubelet? | 03:47 |
| flwang | or recreate the occm pod? | 03:48 |
| *** ricolin has joined #openstack-containers | 03:54 | |
| *** ramishra has joined #openstack-containers | 04:01 | |
| *** sapd1_x has quit IRC | 04:16 | |
| *** ykarel has joined #openstack-containers | 04:24 | |
| *** janki has joined #openstack-containers | 04:34 | |
| *** udesale has joined #openstack-containers | 04:45 | |
| *** pcaruana has joined #openstack-containers | 04:46 | |
| *** ramishra has quit IRC | 05:39 | |
| jakeyip | Restarting minion is probably an overkill just the process will be enough. We were struggling to understand what and just restarted a demo cluster to be sure. | 05:41 |
| jakeyip | * a minion on a demo cluster that wasn't doing anything | 05:42 |
| *** ttsiouts has joined #openstack-containers | 06:00 | |
| *** ttsiouts has quit IRC | 06:04 | |
| *** rcernin has quit IRC | 06:57 | |
| brtknr | flwang: any idea why the function api tests are failing? I don’t understand the error in the logs | 06:59 |
| *** trident has quit IRC | 07:08 | |
| *** trident has joined #openstack-containers | 07:19 | |
| *** ivve has joined #openstack-containers | 07:32 | |
| *** ykarel is now known as ykarel|lunch | 07:44 | |
| *** rtt-rtp has joined #openstack-containers | 07:45 | |
| *** ttsiouts has joined #openstack-containers | 08:16 | |
| jakeyip | erm is meeting today? | 08:36 |
| *** KeithMnemonic1 has joined #openstack-containers | 08:55 | |
| *** KeithMnemonic has quit IRC | 08:55 | |
| *** KeithMnemonic1 has quit IRC | 08:55 | |
| *** KeithMnemonic1 has joined #openstack-containers | 08:56 | |
| *** flwang1 has joined #openstack-containers | 08:56 | |
| *** pcaruana has quit IRC | 08:57 | |
| flwang1 | strigazi: ping | 08:57 |
| flwang1 | meeting? | 08:57 |
| *** ramishra has joined #openstack-containers | 08:59 | |
| *** pcaruana has joined #openstack-containers | 09:01 | |
| jakeyip | I am here, is there going to be a meeting? | 09:07 |
| flwang1 | jakeyip: not sure, seems strigazi is not around | 09:08 |
| flwang1 | jakeyip: anything you want to discuss? | 09:08 |
| jakeyip | wondering what keystone vesion are you running flwang1 | 09:09 |
| flwang1 | jakeyip: not sure if i'm allowed to mention that info | 09:10 |
| flwang1 | what i can tell is it's an old version | 09:10 |
| jakeyip | ok, just wondering if there was anything we could have done differently for the keystone upgrade | 09:11 |
| flwang1 | jakeyip: ah, ok, but i missed your last message about why you have to restart the node | 09:13 |
| jakeyip | maybe it's just something different in how keystone handle trust tokens between queens and rocky but doesn't matter | 09:13 |
| jakeyip | we didn't have to restart node, restarting a process was probably enough | 09:14 |
| flwang1 | jakeyip: ok, i see. anyhow, thanks for sharing | 09:18 |
| flwang1 | what's your user feedback about magnum internally? | 09:19 |
| jakeyip | k8s is something everyone says they want... but pick up has been slow | 09:21 |
| jakeyip | probably need more training materials | 09:21 |
| jakeyip | what about your side? | 09:23 |
| *** ramishra_ has joined #openstack-containers | 09:25 | |
| *** ramishra has quit IRC | 09:26 | |
| flwang1 | we do have some customers are using it for production | 09:26 |
| flwang1 | but overall, similar cases | 09:26 |
| *** ramishra has joined #openstack-containers | 09:26 | |
| flwang1 | everyone like it, but it needs bigger changes to use k8s into prod | 09:27 |
| *** ramishra_ has quit IRC | 09:30 | |
| jakeyip | yes I think a 'long running k8s cluster' needs to be a reality first. one that can survive upgrades and growing and shrinking number of nodes. | 09:34 |
| flwang1 | jakeyip: yep, that's true | 09:41 |
| jakeyip | let's call it a day since no one is here... I'm think I'm heading off | 09:49 |
| *** sapd1_x has joined #openstack-containers | 10:00 | |
| *** openstackgerrit has quit IRC | 10:06 | |
| *** ttsiouts has quit IRC | 10:12 | |
| *** ttsiouts has joined #openstack-containers | 10:13 | |
| *** ttsiouts has quit IRC | 10:17 | |
| *** sapd1_x has quit IRC | 10:18 | |
| *** ramishra has quit IRC | 10:20 | |
| *** ykarel|lunch is now known as ykarel | 10:30 | |
| *** ttsiouts has joined #openstack-containers | 10:55 | |
| *** udesale has quit IRC | 11:01 | |
| *** udesale has joined #openstack-containers | 11:02 | |
| *** openstackgerrit has joined #openstack-containers | 11:10 | |
| openstackgerrit | Vishal Manchanda proposed openstack/magnum-ui master: Update the constraints url https://review.opendev.org/682856 | 11:10 |
| *** pcaruana has quit IRC | 11:19 | |
| *** dave-mccowan has joined #openstack-containers | 11:21 | |
| *** pcaruana has joined #openstack-containers | 11:28 | |
| *** rtt-rtp has quit IRC | 11:34 | |
| *** udesale has quit IRC | 11:44 | |
| *** janki has quit IRC | 12:07 | |
| *** goldyfruit_ has quit IRC | 12:15 | |
| *** janki has joined #openstack-containers | 12:52 | |
| *** janki has quit IRC | 12:53 | |
| *** ramishra has joined #openstack-containers | 13:01 | |
| *** ricolin_ has joined #openstack-containers | 13:08 | |
| *** ricolin has quit IRC | 13:11 | |
| *** ricolin_ is now known as ricolin | 13:12 | |
| *** ttsiouts has quit IRC | 13:19 | |
| *** ttsiouts has joined #openstack-containers | 13:20 | |
| *** ttsiouts has quit IRC | 13:20 | |
| *** ttsiouts has joined #openstack-containers | 13:21 | |
| *** goldyfruit_ has joined #openstack-containers | 13:30 | |
| openstackgerrit | Vishal Manchanda proposed openstack/magnum-ui master: [WIP] Generate PDF documentation https://review.opendev.org/682897 | 13:35 |
| openstackgerrit | Vishal Manchanda proposed openstack/magnum-ui master: Generate PDF documentation https://review.opendev.org/682897 | 13:54 |
| *** ykarel is now known as ykarel|afk | 14:04 | |
| *** openstackgerrit has quit IRC | 14:21 | |
| *** ykarel|afk is now known as ykarel | 14:54 | |
| *** zul has joined #openstack-containers | 15:14 | |
| *** ivve has quit IRC | 15:18 | |
| *** goldyfruit_ has quit IRC | 15:20 | |
| *** ttsiouts has quit IRC | 15:36 | |
| *** trident has quit IRC | 15:41 | |
| *** ykarel is now known as ykarel|away | 15:50 | |
| *** trident has joined #openstack-containers | 15:53 | |
| *** ramishra has quit IRC | 15:55 | |
| *** henriqueof has quit IRC | 16:02 | |
| *** ykarel|away has quit IRC | 16:05 | |
| *** goldyfruit_ has joined #openstack-containers | 16:28 | |
| *** ivve has joined #openstack-containers | 16:32 | |
| *** jmlowe has quit IRC | 16:39 | |
| *** mgariepy has quit IRC | 17:16 | |
| *** mgariepy has joined #openstack-containers | 17:22 | |
| *** ricolin has quit IRC | 17:27 | |
| guilhermesp | hello. I just applied this patch manually in my deployment ( master ) https://review.opendev.org/#/c/621734/ but seems that I still experiencing this http://paste.openstack.org/show/777441/ | 17:38 |
| guilhermesp | is there anyting else out of this patch that needs to be done? | 17:38 |
| *** jmlowe has joined #openstack-containers | 17:44 | |
| brtknr | guilhermesp: can you mention that in the review? also specify your deployment method? | 17:51 |
| brtknr | im testing that PS on devstack and works fine... | 17:51 |
| guilhermesp | sure. I will comment | 17:52 |
| brtknr | guilhermesp: what is your deployment method? | 17:53 |
| *** jmlowe has quit IRC | 17:53 | |
| guilhermesp | we are deploying using openstack-ansible in a multinode environment. We get the source code and installs it in a virtualenv | 17:53 |
| guilhermesp | just commented brtknr | 17:59 |
| brtknr | guilhermesp: ah okay i am not familiar with that method. would you mind sharing the command you used to created the cluster/cluster template? | 18:01 |
| guilhermesp | actually I'm creating using horizon gui brtknr but I can surely send you the cluster -template details in a minute | 18:04 |
| guilhermesp | brtknr: http://paste.openstack.org/show/777444/ and then I just create a cluster based on that template with 1 master and 1 node | 18:06 |
| guilhermesp | the flavors has 0 root disk btw | 18:07 |
| brtknr | guilhermesp: can you try with a flavor with non-zero root disk? | 18:08 |
| guilhermesp | ah I'm afraid I can, we are doing this in a production env and we can't right now create flavors with non-zero root disk =/ | 18:14 |
| guilhermesp | I can't* | 18:15 |
| brtknr | guilhermesp: oh! | 18:16 |
| brtknr | guilhermesp: this looks relevant: https://bugs.launchpad.net/bugs/1837054 | 18:17 |
| openstack | Launchpad bug 1825020 in OpenStack Compute (nova) stein "duplicate for #1837054 resize of instance, that was booted from volume, trigger image size check then fails resize" [High,Fix committed] - Assigned to Matt Riedemann (mriedem) | 18:17 |
| guilhermesp | ah thanks brtknr we are running 19.0.2... I'm gonna double check heat + magnum and see if everything is in place | 18:26 |
| guilhermesp | I will keep you informed and will update the patch set comment when I have some more news | 18:26 |
| *** goldyfruit___ has joined #openstack-containers | 18:38 | |
| *** goldyfruit_ has quit IRC | 18:41 | |
| *** flwang1 has quit IRC | 18:46 | |
| *** jmlowe has joined #openstack-containers | 19:03 | |
| *** jmlowe has quit IRC | 19:42 | |
| *** pcaruana has quit IRC | 20:10 | |
| *** jmlowe has joined #openstack-containers | 20:12 | |
| *** goldyfruit___ has quit IRC | 20:44 | |
| *** goldyfruit___ has joined #openstack-containers | 20:56 | |
| *** goldyfruit_ has joined #openstack-containers | 21:22 | |
| *** goldyfruit___ has quit IRC | 21:25 | |
| *** goldyfruit_ has quit IRC | 22:15 | |
| *** goldyfruit_ has joined #openstack-containers | 22:40 | |
| *** rcernin has joined #openstack-containers | 23:16 | |
| *** goldyfruit_ has quit IRC | 23:24 | |
| *** goldyfruit_ has joined #openstack-containers | 23:29 | |
| *** ivve has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!