| *** xinliang has joined #openstack-containers | 00:32 | |
| flwang | cosmicsound: we(Catalyst Cloud) are using Ubuntu, but I know some other companies are using CentOS | 01:06 |
|---|---|---|
| *** spsurya has joined #openstack-containers | 01:28 | |
| cosmicsound | I tried now ubuntu | 01:35 |
| cosmicsound | And my magnum keeps on failing for now | 01:36 |
| cosmicsound | 2019-12-02 01:22:38Z [kubernetes-ilbbpahqslad.kube_minions]: CREATE_FAILED CREATE aborted (Task create from ResourceGroup "kube_minions" Stack "kubernetes-ilbbpahqslad" [abee1847-8156-45a2-989a-22985ac9f0d1] Timed out) | 01:36 |
| cosmicsound | This is not much to debug on | 01:36 |
| cosmicsound | would like to give centos a try | 01:36 |
| cosmicsound | any reasons why your company choosed ubuntu over centos? | 01:37 |
| cosmicsound | i know the community is bigger in terms of cloud enablement @ ubuntu | 01:37 |
| *** dave-mccowan has joined #openstack-containers | 01:45 | |
| flwang | i think it's because we are already on ubuntu before we got openstack | 01:52 |
| *** ramishra has joined #openstack-containers | 03:36 | |
| *** ricolin has joined #openstack-containers | 03:47 | |
| *** ykarel has joined #openstack-containers | 04:20 | |
| *** udesale has joined #openstack-containers | 04:37 | |
| *** dave-mccowan has quit IRC | 04:40 | |
| openstackgerrit | Simon Merrick proposed openstack/magnum-ui master: Add ui for resizing clusters https://review.opendev.org/696814 | 04:43 |
| *** xinliang has quit IRC | 04:44 | |
| *** ramishra has quit IRC | 05:20 | |
| *** ramishra has joined #openstack-containers | 05:39 | |
| *** elenalindq has joined #openstack-containers | 05:45 | |
| *** pcaruana has joined #openstack-containers | 06:32 | |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| *** ykarel|lunch is now known as ykarel | 08:20 | |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: [k8s] Update metrics-server https://review.opendev.org/637939 | 08:40 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: [k8s] Add heapster_enabled label https://review.opendev.org/696832 | 08:40 |
| openstackgerrit | Diogo Guerra proposed openstack/magnum master: [k8s] Deprecate heapster https://review.opendev.org/696833 | 08:40 |
| dioguerra | brtknr: I Added the heapster_enabled label and changed the default k8s metrics provider to the metrics-server. Altho in different commits... | 08:43 |
| dioguerra | My devstack is bugging again... so i wanted to postpone. But since i did this a while ago, i just pushed upstream | 08:44 |
| *** trident has quit IRC | 09:07 | |
| *** trident has joined #openstack-containers | 09:09 | |
| brtknr | dioguerra: thank you :) | 09:39 |
| andrein | cosmicsound: your issue sounds similar to what I experienced a while back. do you have "enable_cluster_user_trust: True" in globals.yml? | 09:45 |
| *** elenalindq has quit IRC | 09:48 | |
| andrein | hi flwang | 10:07 |
| *** ivve has joined #openstack-containers | 10:08 | |
| *** lpetrut has joined #openstack-containers | 10:13 | |
| andrein | hi brtknr | 10:28 |
| openstackgerrit | Andrei Nistor proposed openstack/magnum master: Change k8s-keystone-auth docker repo https://review.opendev.org/696706 | 10:37 |
| brtknr | hi andrein | 10:45 |
| brtknr | strigazi: are we okay to merge this? https://review.opendev.org/#/c/695709/ | 10:57 |
| brtknr | strigazi: also this: https://review.opendev.org/#/c/695709/3 | 10:57 |
| *** ivve has quit IRC | 11:26 | |
| andrein | brtknr: wanted to ask for your feedback on the release notes on the k8s-keystone-auth repo change | 11:54 |
| brtknr | andrein: sure whats up | 11:59 |
| andrein | https://review.opendev.org/696706 | 12:00 |
| andrein | brtknr: ^^ | 12:00 |
| brtknr | andrein: LGTM :) | 12:00 |
| andrein | thanks! | 12:01 |
| *** ivve has joined #openstack-containers | 12:19 | |
| *** udesale has quit IRC | 12:36 | |
| *** udesale has joined #openstack-containers | 12:37 | |
| trident | brtknr: Do you have any suggestions on the road forward regarding https://review.opendev.org/#/c/696253/ ? We probably have to revert that patch from stable/rocky. Is there after that any way to get the working code (for example my suggestion of moving the label loop to a separate part later in the process) merged? I tried to apply the patch I think is related to the differences in behaviour from rocky to stein ( | 12:48 |
| trident | https://github.com/openstack/magnum/commit/2ab874a5be951a6eba4f9d4f54c106bc0c53d9b1#diff-75661c66ec3a574f251543009c12d86f ) but it didn't apply cleanly and I have not yet investigated why. | 12:48 |
| openstackgerrit | Andrei Nistor proposed openstack/magnum master: Change k8s-keystone-auth docker repo https://review.opendev.org/696706 | 13:05 |
| openstackgerrit | Andrei Nistor proposed openstack/magnum master: Change k8s-keystone-auth docker repo https://review.opendev.org/696706 | 13:09 |
| *** dave-mccowan has joined #openstack-containers | 13:15 | |
| *** dave-mccowan has quit IRC | 13:21 | |
| brtknr | trident: i doubt that patch is relevant to be honest | 13:27 |
| trident | The reason I thought so was that it seem to change the order stuff is done by removing the cloud-init scripts to set up flannel which are the only parts that are run after enable-services-master.sh. So in fact it seems like enable-services-master.sh is the last script being run after that patch is applied. If I interpret correctly? | 13:30 |
| trident | brtknr: If I am correct, in stein it's quite okay that enable-services-master.sh blocks in that loop while waiting for flannel to be set up, which happens in another context which will not be blocked by enable-services-master.sh. | 13:31 |
| brtknr | trident: i think the way that things are deployed in stein is different to rocky | 13:35 |
| brtknr | in rocky, it is a SoftwareConfig | 13:36 |
| brtknr | whereas in stein+ it is deployed as a software deployment | 13:36 |
| brtknr | trident: can you try with this modification: https://seashells.io/v/yNAf5uuU | 13:40 |
| brtknr | and see if that works for you? | 13:40 |
| trident | brtknr: Will try, thanks! | 13:41 |
| trident | brtknr: Will take 15-20 minutes or so. | 13:43 |
| brtknr | trident: also have you tried spawning the cluser with calico plugin ? | 13:45 |
| trident | brtknr: I have not tried with calico plugin. The modification you suggested above seem to work perfectly though. Both with tags 1.14.8 and 1.15.5. | 13:56 |
| brtknr | trident: excellent | 13:58 |
| trident | brtknr: Is that a patch we could get merged into stable/rocky? | 13:59 |
| brtknr | trident: lets see, its not something we do often | 14:01 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: fix: Deploy enable_service last (rocky only) https://review.opendev.org/696884 | 14:01 |
| brtknr | trident: ^ | 14:03 |
| brtknr | lets see what strigazi and flwang have to say about this, i have never seen a patch merge directly to a stable branch | 14:03 |
| brtknr | but in this case, the fix is only relevant to rocky | 14:04 |
| trident | brtknr: I actually read up a bit on it, and according to policy it seems to be okay at least: "Whether the fix is already on master and all consequent stable branches: a change must be a backport of a change already merged onto master, unless the change simply does not make sense on master." | 14:05 |
| brtknr | trident: ah, thanks for checking that | 14:07 |
| trident | brtknr: That's by the way from https://docs.openstack.org/project-team-guide/stable-branches.html | 14:08 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: fix: Deploy enable_service last (rocky only) https://review.opendev.org/696884 | 14:08 |
| brtknr | trident: could you comment on the patch saying this addresses your issue | 14:09 |
| brtknr | once this gets merged, we can cut a new release | 14:09 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/train: bug: Use configured heat-container-agent tag https://review.opendev.org/696887 | 14:13 |
| trident | brtknr: Absolutely, will comment! | 14:15 |
| openstackgerrit | Andrei Nistor proposed openstack/magnum master: Change k8s-keystone-auth docker repo https://review.opendev.org/696706 | 14:15 |
| brtknr | andrein: you also need to update the k8s_keystone_tag in kubecluster.yaml | 14:17 |
| *** ykarel is now known as ykarel|afk | 14:17 | |
| brtknr | and please update the commit message saying you have bumped up the version for ussuri | 14:17 |
| trident | brtknr: Commented. Thanks very much for working with us on this! | 14:20 |
| brtknr | trident: happy to help but if you want to run 1.16.x, there is no other option but to run magnum train | 14:21 |
| brtknr | unless you are happy to backport everything from master | 14:21 |
| openstackgerrit | Andrei Nistor proposed openstack/magnum master: Change k8s-keystone-auth docker repo https://review.opendev.org/696706 | 14:22 |
| andrein | brtknr: updated the tag, sorry I missed that | 14:22 |
| brtknr | andrein: thanks, looking good now | 14:24 |
| trident | brtknr: Yeah, we are aware of that and upgrading going forward is planned for the whole environments - just don't know exactly when we will get there. With a whole bunch of environments and most of them under different compliance requirements it's not always a simple a quick process unfortunately. | 14:30 |
| brtknr | trident: i am glad you are able to support 1.15.5 though... | 14:31 |
| brtknr | i didn't think rocky had the required changes to support this | 14:31 |
| trident | brtknr: With the heat_container_agent_tag=train-stable label it seem to work as far as I have tested at least. | 14:32 |
| *** munimeha1 has joined #openstack-containers | 14:35 | |
| brtknr | trident: if you could test with calico plugin too and report back, that would be great, i am struggling to get my devstack environment up today for some reason | 14:38 |
| brtknr | trident: we want to make sure we havent inadvertly broken something else as a result of this change | 14:38 |
| trident | brtknr: I'll try that as well! | 14:39 |
| brtknr | trident: thanks | 14:39 |
| *** spiette has joined #openstack-containers | 15:18 | |
| *** ykarel|afk is now known as ykarel|away | 15:54 | |
| *** ykarel|away has quit IRC | 16:02 | |
| trident | brtknr: Okay, lets see. With calico plugin both 1.14.8 and 1.15.5 fails. They both go to CREATE_COMPLETE but in in 1.15.5 no calico-node-* gets spun up, only calico-kube-controllers and the rest of the pods hangs in ContainerCreating. However, without https://review.opendev.org/696884 it fails in seemingly the exact same way (which makes sense, as all three scripts enable-services-master is moved after in that patch starts with "if [ | 16:03 |
| trident | "$NETWORK_DRIVER" != "flannel" ]; then exit 0". | 16:03 |
| trident | brtknr: Without https://review.opendev.org/#/c/696253/ on the other hand I don't even get that far. Most pods are just hanging in status Pending as nodes are tainted. | 16:04 |
| trident | brtknr: So my assessment is that those issues doesn't have anything to do with the fixes in question but is a separate issue that should probably be addressed separately? | 16:05 |
| brtknr | trident: thanks for reporting back, i think i know why this is breaking | 16:06 |
| brtknr | trident: are you using fedora atomic 29? | 16:08 |
| trident | brtknr: Yes. | 16:09 |
| *** elenalindq has joined #openstack-containers | 16:21 | |
| *** ivve has quit IRC | 16:26 | |
| *** ykarel|away has joined #openstack-containers | 16:35 | |
| *** lbragsta_ has joined #openstack-containers | 16:36 | |
| *** udesale has quit IRC | 16:41 | |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: fix: Deploy enable_service last (rocky only) https://review.opendev.org/696884 | 16:41 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: k8s: Clear cni configuration https://review.opendev.org/696915 | 16:41 |
| openstackgerrit | Bharat Kunwar proposed openstack/magnum stable/rocky: k8s_fedora: Set rp_filter=1 for calico https://review.opendev.org/696916 | 16:41 |
| brtknr | trident: please try again with the tree above | 16:41 |
| brtknr | that ought to work | 16:41 |
| trident | Thanks, will try! | 16:43 |
| *** trident has quit IRC | 16:51 | |
| *** trident has joined #openstack-containers | 16:53 | |
| *** df_sbr has joined #openstack-containers | 16:54 | |
| *** KeithMnemonic has joined #openstack-containers | 17:18 | |
| *** ykarel has joined #openstack-containers | 17:32 | |
| trident | brtknr: Hmm.. This time the master didn't even get the kubectl command installed. So obviously it blocks on the loop trying to do the kubectl patch. | 17:34 |
| *** ykarel|away has quit IRC | 17:34 | |
| trident | brtknr: It seems to be writing stuff to /srv/magnum/kubernetes/install-kubernetes.sh including the atomic install commands. Then that script is never run. | 17:42 |
| *** lbragsta_ has quit IRC | 17:44 | |
| *** ivve has joined #openstack-containers | 18:11 | |
| *** ricolin has quit IRC | 18:22 | |
| *** spsurya has quit IRC | 18:25 | |
| *** jmlowe has quit IRC | 18:39 | |
| *** jmlowe has joined #openstack-containers | 18:56 | |
| *** lpetrut has quit IRC | 19:00 | |
| *** ykarel has quit IRC | 19:07 | |
| *** jmlowe has quit IRC | 19:48 | |
| *** andrein_ has joined #openstack-containers | 19:54 | |
| *** kklimonda_ has joined #openstack-containers | 19:55 | |
| *** jrosser_ has joined #openstack-containers | 19:55 | |
| *** dioguerr1 has joined #openstack-containers | 19:57 | |
| *** pcaruana has quit IRC | 20:02 | |
| *** dioguerra has quit IRC | 20:02 | |
| *** andrein has quit IRC | 20:02 | |
| *** Jeffrey4l_ has quit IRC | 20:02 | |
| *** johanssone has quit IRC | 20:02 | |
| *** jrosser has quit IRC | 20:02 | |
| *** kklimonda has quit IRC | 20:02 | |
| *** andrein_ is now known as andrein | 20:02 | |
| *** kklimonda_ is now known as kklimonda | 20:02 | |
| *** jrosser_ is now known as jrosser | 20:02 | |
| *** Jeffrey4l has joined #openstack-containers | 20:02 | |
| *** johanssone has joined #openstack-containers | 20:03 | |
| *** pcaruana has joined #openstack-containers | 20:05 | |
| *** jmlowe has joined #openstack-containers | 20:14 | |
| *** munimeha1 has quit IRC | 20:20 | |
| *** rcernin has joined #openstack-containers | 21:57 | |
| *** rcernin has quit IRC | 21:59 | |
| *** elenalindq has quit IRC | 22:01 | |
| *** pcaruana has quit IRC | 22:11 | |
| *** rcernin has joined #openstack-containers | 22:57 | |
| *** dioguerr1 has quit IRC | 23:06 | |
| *** dioguerra has joined #openstack-containers | 23:06 | |
| brtknr | flwang: Really sorry I still haven’t tested your patch because I’ve been failing to get a working devstack environment up and running | 23:15 |
| flwang | brtknr: that's ok :) | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!