| *** hongbin has quit IRC | 00:16 | |
| *** jaewook_oh has joined #openstack-containers | 01:15 | |
| *** openstackgerrit has joined #openstack-containers | 01:36 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/magnum master: Fixing gate failing due to bad AMQP virtual_host https://review.openstack.org/610740 | 01:36 |
|---|---|---|
| eandersson | Confirmed that the above fixes the currently broken gate fungi strigazi | 01:37 |
| * fungi is not entirely sure why he was mentioned there | 01:41 | |
| eandersson | Ops, meant to poke flwang1 | 01:42 |
| eandersson | My bad fungi | 01:42 |
| fungi | np | 01:47 |
| *** hongbin has joined #openstack-containers | 02:09 | |
| *** jaewook_oh has quit IRC | 02:44 | |
| *** jaewook_oh has joined #openstack-containers | 02:53 | |
| *** ramishra has joined #openstack-containers | 03:02 | |
| *** dave-mccowan has quit IRC | 03:08 | |
| *** hongbin has quit IRC | 03:48 | |
| *** Bhujay has joined #openstack-containers | 03:51 | |
| *** Bhujay has quit IRC | 03:52 | |
| *** lpetrut has joined #openstack-containers | 03:52 | |
| *** Bhujay has joined #openstack-containers | 03:52 | |
| *** Bhujay has quit IRC | 04:06 | |
| *** ykarel has joined #openstack-containers | 04:22 | |
| openstackgerrit | Erik Olof Gunnar Andersson proposed openstack/python-magnumclient master: Fixing broken tempest test https://review.openstack.org/610876 | 04:30 |
| *** lpetrut has quit IRC | 04:34 | |
| openstackgerrit | yatin proposed openstack/magnum master: Update heat-container-agent version tag https://review.openstack.org/609579 | 04:50 |
| *** janki has joined #openstack-containers | 05:47 | |
| strigazi | mordred: I was off the other day, eandersson proposed a fix. Thanks for pinging. Was this broken for many days? | 06:02 |
| *** spiette has quit IRC | 07:00 | |
| *** rcernin has quit IRC | 07:01 | |
| *** spiette has joined #openstack-containers | 07:03 | |
| *** lpetrut has joined #openstack-containers | 07:20 | |
| *** lpetrut has quit IRC | 07:25 | |
| *** ttsiouts has joined #openstack-containers | 07:30 | |
| *** mattgo has joined #openstack-containers | 07:31 | |
| brtknr | strigazi: max_connections no longer an issue but now i have a new failure mode: http://paste.openstack.org/show/732220/ | 07:50 |
| strigazi | brtknr: https://review.openstack.org/#/c/610740/ | 07:55 |
| brtknr | full conductor log: http://paste.openstack.org/show/732221/ | 07:55 |
| brtknr | Cool! When will it get merged? | 08:02 |
| strigazi | we just way the ci | 08:03 |
| strigazi | it is almost done | 08:03 |
| openstackgerrit | Merged openstack/magnum master: Fixing gate failing due to bad AMQP virtual_host https://review.openstack.org/610740 | 08:05 |
| strigazi | brtknr: ^^ | 08:05 |
| brtknr | Great thank you :D | 08:10 |
| brtknr | Although max_connections defaulting to low value appears to be a lingering issue on CentOS | 08:11 |
| brtknr | Heat opens around 370 connections with mariadb during ./stack.sh | 08:11 |
| brtknr | and unless a high limit for open_files_limit(=1024 in my case which matches ulimit -n) is passed as a config to mariadb, magnum installation fails | 08:13 |
| brtknr | only on Centos, i did not see this with Ubuntu for whatever reason | 08:14 |
| * brtknr also finds it interesting that he did not see the AMQP issue on Ubuntu devstack, only on CentOS. | 08:25 | |
| *** ykarel is now known as ykarel|mtg | 08:51 | |
| *** ykarel|mtg is now known as ykarel | 09:03 | |
| *** salmankhan has joined #openstack-containers | 09:10 | |
| *** Dobroslaw has joined #openstack-containers | 09:16 | |
| *** flwang1 has joined #openstack-containers | 09:48 | |
| flwang1 | strigazi: ping | 09:48 |
| strigazi | flwang1: o/ | 09:50 |
| flwang1 | strigazi: do you have time for a quick sync? | 09:50 |
| strigazi | flwang1: yes, but before a question, do you know how to pass separated parameters in gnu make? (I have spent like 30mins and I'm stuck) :) | 09:51 |
| flwang1 | like 'make xxx' ? | 09:52 |
| flwang1 | or 'make aaa bbb'? | 09:52 |
| strigazi | make foo FLAGS='-X a -X b | 09:52 |
| strigazi | make foo FLAGS='-X a -X b' | 09:52 |
| strigazi | anyway, tell me | 09:54 |
| flwang1 | no idea, sorry | 09:54 |
| flwang1 | 1. are you aware of the broken gate? | 09:54 |
| strigazi | it is fixed now | 09:54 |
| flwang1 | 2. what's the docker_storage_driver you're using in prod? | 09:54 |
| strigazi | Merged openstack/magnum master: Fixing gate failing due to bad AMQP virtual_host https://review.openstack.org/610740 this patch | 09:54 |
| strigazi | overlay2 only | 09:55 |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Fix prometheus monitoring https://review.openstack.org/609203 | 09:55 |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Allow cluster template being renamed https://review.openstack.org/609174 | 09:56 |
| flwang1 | ok, interesting | 09:56 |
| strigazi | isn't this merged already by someone else? Fix prometheus monitoring https://review.openstack.org/609203 | 09:56 |
| strigazi | it is not https://review.openstack.org/#/c/600905/ | 09:57 |
| flwang1 | ah, yes, same issue with different fix | 09:58 |
| *** ykarel has quit IRC | 09:59 | |
| flwang1 | i don't mind abandoning my one | 10:00 |
| strigazi | I had +2 at some point, lost track afterwards | 10:01 |
| flwang1 | ok, then | 10:01 |
| flwang1 | we can merge that one | 10:01 |
| *** ttsiouts has quit IRC | 10:08 | |
| *** ykarel has joined #openstack-containers | 10:08 | |
| *** ykarel is now known as ykarel|lunch | 10:09 | |
| *** ttsiouts has joined #openstack-containers | 10:29 | |
| *** ykarel_ has joined #openstack-containers | 10:48 | |
| *** ykarel|lunch has quit IRC | 10:50 | |
| *** serlex has joined #openstack-containers | 10:51 | |
| openstackgerrit | Feilong Wang proposed openstack/magnum master: Cluster delete failed due to heat return stack could not be found Process is as follows: 1. delete one magnum cluster 2. poll_and_check function periodic query stack status by heatclient 3. heatclient return "The Stack *** could not be found" exception 4. https://review.openstack.org/599925 | 10:55 |
| *** ykarel_ is now known as ykarel | 10:56 | |
| *** ttsiouts has quit IRC | 11:02 | |
| *** ttsiouts has joined #openstack-containers | 11:03 | |
| *** ttsiouts has quit IRC | 11:07 | |
| *** dave-mccowan has joined #openstack-containers | 11:09 | |
| *** ttsiouts has joined #openstack-containers | 11:17 | |
| *** ttsiouts has quit IRC | 11:31 | |
| *** ttsiouts has joined #openstack-containers | 11:32 | |
| *** ttsiouts has quit IRC | 11:34 | |
| *** ttsiouts has joined #openstack-containers | 11:35 | |
| *** mattgo has quit IRC | 11:35 | |
| *** ttsiouts has quit IRC | 11:37 | |
| *** ttsiouts has joined #openstack-containers | 11:38 | |
| tobias-urdin | strigazi: i think i've found an annoying bug, flanneld systemd service depends on flannel-docker-bridge but we dont perform a daemon-reload here | 11:39 |
| tobias-urdin | https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/common/templates/kubernetes/fragments/flannel-service.sh#L81 | 11:39 |
| tobias-urdin | so it fails to find that service | 11:39 |
| tobias-urdin | or something along those lines, kubernetes minion does not work right away, need to manually fix some processes, or perhaps this is fixed in the upcoming 7.0.2 release for rocky? | 11:41 |
| tobias-urdin | i'll do some more testing, see if i can reproduce | 11:45 |
| *** ttsiouts has quit IRC | 11:46 | |
| *** ttsiouts has joined #openstack-containers | 11:47 | |
| *** ttsiouts has quit IRC | 11:47 | |
| *** ttsiouts has joined #openstack-containers | 11:47 | |
| strigazi | tobias-urdin it will be "fixed" | 11:50 |
| strigazi | tobias-urdin: https://review.openstack.org/#/c/597150/ | 11:50 |
| strigazi | you can give this patcha a go ^^ | 11:50 |
| *** ttsiouts has quit IRC | 11:50 | |
| *** ttsiouts has joined #openstack-containers | 11:51 | |
| *** mattgo has joined #openstack-containers | 11:51 | |
| tobias-urdin | strigazi: ack, seems like a large one, assuming that wont be backported since it changes the usage of flannel i'll see what i can do to get current stable/rocky working | 11:55 |
| *** ttsiouts has quit IRC | 11:55 | |
| *** ttsiouts has joined #openstack-containers | 12:00 | |
| strigazi | tobias-urdin: we will add in 7.1.0, it improves stability of flannel and aligns with calico | 12:01 |
| tobias-urdin | ack | 12:02 |
| tobias-urdin | strigazi: also seeing this: Auth plugin and its options for service user must be provided in [keystone_auth] section. Using values from [keystone_authtoken] section is deprecated.: MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url | 12:02 |
| tobias-urdin | https://github.com/openstack/magnum/blob/stable/rocky/magnum/common/keystone.py#L47 | 12:02 |
| tobias-urdin | a lot of deployments (and tooling) changed from the deprecated auth_uri to www_authenticate_uri in the rocky release | 12:03 |
| tobias-urdin | perhaps that's the issue, i assume this only reads auth_uri which is not there anymore and doesn't read www_authenticate_uri and fallback to auth_uri? | 12:03 |
| strigazi | it is not that, the started work to refactor authtoken left unfinished and needs to be cleaned. We can clean it and adopt www_authenticate_uri | 12:04 |
| * strigazi will be back | 12:04 | |
| tobias-urdin | facepalm, yeah saw now that it said auth_url and not auth_uri | 12:09 |
| *** ttsiouts has quit IRC | 12:32 | |
| mordred | strigazi: the eandersson patch fixed it, yay! yeah, it was broken for a bit - but there were other breaks going on so I didn't notice immediately | 12:57 |
| mordred | strigazi: we had some "fun" related to OOM errors breaking things | 12:58 |
| * mordred loves fun | 12:58 | |
| *** mattgo has quit IRC | 12:59 | |
| *** jaewook_oh has quit IRC | 13:23 | |
| *** ykarel is now known as ykarel|afk | 13:50 | |
| tobias-urdin | strigazi: after some digging seems like the issue is flanneld related, it doesn't start on first attempt | 13:55 |
| tobias-urdin | http://paste.openstack.org/show/732246/ | 13:55 |
| tobias-urdin | see the end in that paste, i need to run that for it to work | 13:55 |
| tobias-urdin | this is on the minion | 13:55 |
| *** janki has quit IRC | 13:58 | |
| strigazi | tobias-urdin: in f27 seems to work | 14:03 |
| tobias-urdin | ok, so i should probably just use f27 for now | 14:04 |
| *** hongbin has joined #openstack-containers | 14:05 | |
| *** mattgo has joined #openstack-containers | 14:06 | |
| *** munimeha1 has joined #openstack-containers | 14:19 | |
| tobias-urdin | strigazi: sorry for the noice, remember i messed around with this ealier but had to wipe all glance images so must have got the wrong one.. | 14:26 |
| *** ykarel|afk has quit IRC | 14:29 | |
| *** ykarel|afk has joined #openstack-containers | 14:30 | |
| strigazi | https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 or latest f27 | 14:33 |
| strigazi | tobias-urdin: ^^ | 14:34 |
| *** mattgo has quit IRC | 14:35 | |
| *** ykarel|afk has quit IRC | 14:48 | |
| *** ykarel|afk has joined #openstack-containers | 14:48 | |
| tobias-urdin | strigazi: k8s worked on first attempt with f27 | 14:49 |
| brtknr | tobias-urdin: what are you using to deploy your openstack? | 14:51 |
| brtknr | tobias-urdin: devstack or some other tool? | 14:51 |
| tobias-urdin | puppet openstack modules | 14:51 |
| tobias-urdin | on centos + rdo | 14:51 |
| brtknr | tobias-urdin: did you need to change your mariadb config to increase number of open files? LimitNOFILE=infinity | 14:52 |
| tobias-urdin | no | 14:53 |
| brtknr | what do you get back when run # show variables like "open_files_limit"; under your mysql console | 14:54 |
| tobias-urdin | | open_files_limit | 16364 | | 15:00 |
| *** ramishra has quit IRC | 15:01 | |
| *** kaiokmo has quit IRC | 15:08 | |
| tobias-urdin | strigazi: found a bug in f27 the /etc/sysconfig/docker file contains --live-restore which causes swarm-mode to fail | 15:37 |
| tobias-urdin | magnum-start-swarm-manager[2069]: Error response from daemon: --live-restore daemon configuration is incompatible with swarm mode | 15:37 |
| tobias-urdin | so need to sed out that option and start/restart docker then run swarm-manager/swarm-worker systemd unit files | 15:38 |
| strigazi | tobias-urdin: strange, I never saw this, in 100s of clusters | 15:40 |
| tobias-urdin | i took the latest fedora atomic 27 which was released 201804-something | 15:40 |
| openstackgerrit | weizj proposed openstack/magnum-tempest-plugin master: Update the hacking to latst https://review.openstack.org/611021 | 15:41 |
| tobias-urdin | i'll spawn up fedora atomic 27 and 28 and see what that file contains by default, the drivers dir has no mentioned of a live-restore | 15:41 |
| openstackgerrit | weizj proposed openstack/magnum master: Update the hacking to latst https://review.openstack.org/611024 | 15:45 |
| tobias-urdin | strigazi: | 15:49 |
| tobias-urdin | [fedora@f27 ~]$ cat /etc/sysconfig/docker | grep OPTIONS | 15:49 |
| tobias-urdin | OPTIONS='--selinux-enabled --log-driver=journald --live-restore' | 15:49 |
| tobias-urdin | [fedora@f28 ~]$ cat /etc/sysconfig/docker | grep OPTIONS | 15:49 |
| tobias-urdin | OPTIONS='--selinux-enabled --log-driver=journald --live-restore' | 15:49 |
| tobias-urdin | both seems to have --live-restore | 15:49 |
| strigazi | I have this: OPTIONS='-H fd:// -H tcp://0.0.0.0:2375 --tlsverify --tlscacert=/etc/docker/ca.crt --tlskey=/etc/docker/server.key --tlscert=/etc/docker/server.crt --selinux-enabled --log-driver=journald' | 15:50 |
| tobias-urdin | that was on two clean fedora atomic 27/28, i checked the logic in the swarm_fedora_atomic_v2 driver and it just appends to the OPTIONS | 15:51 |
| tobias-urdin | here https://github.com/openstack/magnum/blob/stable/rocky/magnum/drivers/common/templates/swarm/fragments/add-docker-daemon-options.sh#L13 | 15:52 |
| strigazi | tobias-urdin: http://git.openstack.org/cgit/openstack/magnum/tree/magnum/drivers/common/templates/swarm/fragments/add-docker-daemon-options.sh?h=stable/rocky | 15:52 |
| tobias-urdin | yeah, so if the templates are spawned with --live-restore they will fail, i haven't verified if the docker version of f28 throws that as well | 15:52 |
| tobias-urdin | by that i mean; this "Error response from daemon: --live-restore daemon configuration is incompatible with swarm mode" | 15:53 |
| strigazi | can you run this script to see what it does? I think it removes it | 15:53 |
| tobias-urdin | i did try that, copied that script and did against a test file, it just appended the options, didn't remove any existing in OPTIONS | 15:54 |
| tobias-urdin | s/appended/prepended/g | 15:54 |
| strigazi | did you try this image: https://download.fedoraproject.org/pub/alt/atomic/stable/Fedora-Atomic-27-20180212.2/CloudImages/x86_64/images/Fedora-Atomic-27-20180212.2.x86_64.qcow2 | 15:54 |
| tobias-urdin | no im using the later one | 15:55 |
| tobias-urdin | https://ftp.icm.edu.pl/pub/Linux/dist/fedora-alt/atomic/stable/Fedora-Atomic-27-20180419.0/CloudImages/x86_64/images/Fedora-Atomic-27-20180419.0.x86_64.qcow2 | 15:55 |
| tobias-urdin | shouldn't we support both? | 15:55 |
| strigazi | yes, I just wanted to understand why it works for us | 15:56 |
| tobias-urdin | give me a sec, i'll spawn a instance with that image and check the /etc/sysconfig/docker file | 15:56 |
| tobias-urdin | [fedora@f27-old ~]$ cat /etc/sysconfig/docker | grep OPTIONS | 15:59 |
| tobias-urdin | OPTIONS='--selinux-enabled --log-driver=journald' | 15:59 |
| tobias-urdin | strigazi: confirmed it's a image issue, so we should probably add a sed that removes --live-restore | 16:00 |
| tobias-urdin | do you want me to write a fix, test it and push a patch? | 16:02 |
| tobias-urdin | should be super simple, one-line sed to that shell script | 16:02 |
| tobias-urdin | i'll be afk for a while, will be back in 1 hour | 16:03 |
| brtknr | strigazi: magnum finally working via devstack :D | 16:06 |
| brtknr | strigazi: so many hurdles, you wont believe! | 16:06 |
| *** ykarel|afk is now known as ykarel | 16:18 | |
| strigazi | brtknr: I'm using ubuntu for the past 6 months for devstack | 16:35 |
| brtknr | I have no doubt, but lots of things that arent documented anywhere | 16:36 |
| brtknr | strigazi: like configuring number of open files | 16:36 |
| brtknr | and default security group policy | 16:36 |
| brtknr | etc. | 16:36 |
| *** ykarel is now known as ykarel|away | 16:37 | |
| brtknr | i mean,they are documented but its not obvious that is the problem | 16:37 |
| *** ykarel|away has quit IRC | 16:46 | |
| *** munimeha1 has quit IRC | 17:10 | |
| openstackgerrit | Tobias Urdin proposed openstack/magnum master: [swarm-mode] Remove --live-restore from Docker daemon options https://review.openstack.org/611097 | 17:12 |
| tobias-urdin | strigazi: ^ also tested, swarm-mode on f27 2018-04 built image now works (probably f28 as well) | 17:13 |
| *** PagliaccisCloud has joined #openstack-containers | 17:18 | |
| *** kaiokmo has joined #openstack-containers | 17:18 | |
| openstackgerrit | Tobias Urdin proposed openstack/magnum master: [swarm-mode] Remove --live-restore from Docker daemon options https://review.openstack.org/611097 | 17:22 |
| colby_ | Hey Guys. We had a user try and attach a secret directly to the loadbalancer listener without using a contiainer. So it errored out and now its stuck in the pending_update and cant be deleted. How can I delete the load balancer now? | 17:22 |
| colby_ | Im on pike btw | 17:22 |
| colby_ | woops sorry guys wrong channel | 17:56 |
| *** salmankhan has quit IRC | 18:02 | |
| *** flwang1 has quit IRC | 18:44 | |
| *** dave-mccowan has quit IRC | 19:07 | |
| *** dave-mccowan has joined #openstack-containers | 19:07 | |
| *** flwang has joined #openstack-containers | 21:00 | |
| flwang | o/ | 21:00 |
| colin- | hello | 21:00 |
| eandersson | o/ | 21:00 |
| strigazi | #starmeeting containers | 21:00 |
| strigazi | #startmeeting containers | 21:01 |
| openstack | Meeting started Tue Oct 16 21:01:02 2018 UTC and is due to finish in 60 minutes. The chair is strigazi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:01 |
| *** openstack changes topic to " (Meeting topic: containers)" | 21:01 | |
| openstack | The meeting name has been set to 'containers' | 21:01 |
| imdigitaljim | o/ | 21:01 |
| strigazi | #topic Roll Call | 21:01 |
| *** openstack changes topic to "Roll Call (Meeting topic: containers)" | 21:01 | |
| cbrumm__ | o/ | 21:01 |
| imdigitaljim | o/ | 21:01 |
| strigazi | o/ | 21:01 |
| eandersson | \o/ | 21:01 |
| flwang | \o/ | 21:02 |
| strigazi | #topic Announcements | 21:02 |
| *** openstack changes topic to "Announcements (Meeting topic: containers)" | 21:02 | |
| strigazi | thanks eandersson for fixing the magnum gate https://review.openstack.org/#/c/610740/ | 21:03 |
| eandersson | anytime | 21:03 |
| strigazi | Do you know why it used work or why ot broke? | 21:03 |
| strigazi | Do you know why it used work or why it broke? | 21:03 |
| eandersson | I was looking at some recent changes with oslo.messaging that affects the defaults | 21:03 |
| eandersson | Which I think caused it with 9.0.1 | 21:04 |
| flwang | that's bad | 21:04 |
| strigazi | o.m version bump then, thanks | 21:04 |
| eandersson | https://github.com/openstack/requirements/commit/29f8a80a71111ea59141c1aeab94bae9cf3f30a9#diff-0bdd949ed8a7fdd4f95240bd951779c8 | 21:05 |
| flwang | introducing changes in base components and breaking everybody | 21:05 |
| strigazi | flwang: it is better than it used to be I think :) | 21:05 |
| *** munimeha1 has joined #openstack-containers | 21:05 | |
| flwang | better for what? | 21:05 |
| strigazi | less breaks/ time unit | 21:05 |
| flwang | haha | 21:06 |
| *** canori02 has joined #openstack-containers | 21:06 | |
| eandersson | Yea - it's usually 3rd party libraries now that break things | 21:06 |
| eandersson | like evenlet | 21:06 |
| canori02 | O/ | 21:06 |
| flwang | it's true, but i think it's because its losing contributors :D | 21:06 |
| strigazi | canori02: o/ | 21:07 |
| strigazi | #topic Stories/Tasks | 21:07 |
| *** openstack changes topic to "Stories/Tasks (Meeting topic: containers)" | 21:07 | |
| strigazi | I don't think I have anything from last week to discuss, flwang do you want to discuss the CT rename? | 21:08 |
| imdigitaljim | flwang: i found a solution to making a single LB scenario | 21:09 |
| imdigitaljim | huge cut in resource usage is nice :) | 21:09 |
| strigazi | imdigitaljim: use a single LB for many services? | 21:09 |
| imdigitaljim | for etcd/api | 21:10 |
| strigazi | oh, right | 21:10 |
| strigazi | imdigitaljim: is there a patch? | 21:10 |
| imdigitaljim | i think our plan is to submit a v2 driver that is parity with the first one's features | 21:10 |
| imdigitaljim | and huge improvements all around | 21:10 |
| imdigitaljim | cluster speed/feature rich/compatibility | 21:10 |
| flwang | imdigitaljim: cool, we can discuss the single lb later | 21:11 |
| imdigitaljim | yeah definitely | 21:11 |
| imdigitaljim | i can push up a WIP for you guys soon if you want to see where things are | 21:12 |
| flwang | strigazi: yep, i'd like to know you guys opinions about CT rename | 21:12 |
| flwang | imdigitaljim: go for it | 21:12 |
| flwang | as you know, Catalyst Cloud just released magnum on our prod | 21:12 |
| cbrumm__ | Nice! | 21:12 |
| cbrumm__ | congrats | 21:12 |
| flwang | and we're starting to get the pain of change/update public templates | 21:12 |
| flwang | cbrumm__: hah, thanks | 21:13 |
| colin- | that's great, how did the deployment go? | 21:13 |
| flwang | colin-: we got some issues, but so far, most of the things are ok | 21:13 |
| flwang | a big lesson learn is don't use docker_storage_size with overlay/overlay2 | 21:14 |
| imdigitaljim | we are considering a redesign (part of the v2) | 21:14 |
| imdigitaljim | that decouples that entirely | 21:14 |
| flwang | and magnum/heat-container-agent does have the multi region bug | 21:14 |
| flwang | imdigitaljim: nice | 21:15 |
| colin- | that sounds rough, glad it worked out though | 21:15 |
| eandersson | what is the issue with overlay? | 21:15 |
| imdigitaljim | so you dont have to worry about kube-tag or other quick changing data | 21:15 |
| imdigitaljim | ^ | 21:15 |
| imdigitaljim | +1 eandersson | 21:15 |
| imdigitaljim | what is the issues with overlay2? | 21:15 |
| flwang | imdigitaljim: that's interesting | 21:15 |
| flwang | with overlay2 and docker_volume_size(external volume for docker), we have seen some issue with docker, which cause pods can't be created | 21:16 |
| flwang | it's not always repeatable | 21:16 |
| flwang | we just realized it yesterday with strigazi's help, still need testing though | 21:16 |
| flwang | but just FYI | 21:16 |
| flwang | btw, our storage backend is ceph | 21:17 |
| imdigitaljim | cgroup issue? | 21:17 |
| *** kaiokmo has quit IRC | 21:17 | |
| eandersson | Interesting - feel free to share more when you have had time to dig deeper | 21:17 |
| flwang | we're using cgroupfs already | 21:17 |
| colin- | which cgroup driver? just curious | 21:17 |
| colin- | ok | 21:17 |
| *** canori02 has quit IRC | 21:17 | |
| flwang | I still don't know the root cause, may need help from storage expert or docker expert | 21:18 |
| *** canori02 has joined #openstack-containers | 21:18 | |
| flwang | but i will keep you guys posted if there is any progress | 21:18 |
| eandersson | Sounds good - thanks | 21:18 |
| tobias-urdin | flwang: sorry for shiming in, regarding the mentioned multi-region bug in heat-container-agent is that addressed or is there a bug tracked somewhere with more info? | 21:18 |
| tobias-urdin | we are planning to deploy magnum in another region soon | 21:18 |
| *** kaiokmo has joined #openstack-containers | 21:19 | |
| tobias-urdin | eandersson: thanks for the review | 21:19 |
| eandersson | anytime | 21:19 |
| strigazi | tobias-urdin: it is addressed, one moment | 21:20 |
| *** canori02 has quit IRC | 21:21 | |
| strigazi | tobias-urdin: https://review.openstack.org/#/c/584215/ | 21:21 |
| strigazi | tobias-urdin: included here: https://hub.docker.com/r/openstackmagnum/heat-container-agent/tags/ rocky-stable | 21:21 |
| flwang | it has been fixed | 21:21 |
| strigazi | tobias-urdin: this patch is pending: https://review.openstack.org/#/c/609579/ | 21:22 |
| flwang | tobias-urdin: you also need you update your existing deployment | 21:22 |
| flwang | i mean Mangum | 21:23 |
| flwang | for existing cluster, should be fine | 21:23 |
| strigazi | flwang: and heat to queens, right? | 21:23 |
| flwang | strigazi: yes, at least Queens | 21:23 |
| flwang | I fixed it in Rocky, and I think i have backported all necessary changes into queens | 21:23 |
| *** serlex has left #openstack-containers | 21:24 | |
| tobias-urdin | ok thanks! so if I understand correctly when 609579 is merged and backported to stable/rocky we are all good? | 21:24 |
| strigazi | tobias-urdin: yes | 21:24 |
| tobias-urdin | got it, thanks | 21:24 |
| *** canori02 has joined #openstack-containers | 21:24 | |
| flwang | all good for Magnum | 21:25 |
| flwang | tobias-urdin: also need to upgrade Heat | 21:26 |
| tobias-urdin | ack | 21:27 |
| flwang | tobias-urdin: me and strigazi will have a topic at Berlin to share what we know ;) | 21:28 |
| *** canori02 has quit IRC | 21:30 | |
| tobias-urdin | flwang: cool, i'll keep an close eye out for the video on that, unfortunately won't attend berlin :( | 21:31 |
| flwang | no problem | 21:32 |
| flwang | strigazi: i'm good now | 21:32 |
| flwang | strigazi: imdigitaljim: anything else you want to discuss? | 21:32 |
| imdigitaljim | just throwing it out, were working on inplace upgrades currently | 21:33 |
| imdigitaljim | strigazi: has some work thats very close | 21:33 |
| imdigitaljim | to what we started with | 21:33 |
| imdigitaljim | we're adding new mechanisms but i should also be adding it to the v2 driver | 21:33 |
| imdigitaljim | but id like to maybe discuss some basic approaches next time | 21:34 |
| strigazi | imdigitaljim: will you push upstream something for v1? | 21:34 |
| imdigitaljim | probably not for v1 since its too far back from the updates | 21:34 |
| imdigitaljim | it perhaps can be backported | 21:34 |
| imdigitaljim | we *are* keeping features of the v1 driver introduced on the v2 | 21:35 |
| imdigitaljim | so when were done with v2 it should be a minimum everything v1 has | 21:36 |
| imdigitaljim | people can still choose v1 for their needs/legacy but because its a driver they can also choose v2 | 21:36 |
| imdigitaljim | :P | 21:36 |
| strigazi | I think we should have a spec or another document for these changes | 21:37 |
| strigazi | makes sense? | 21:37 |
| imdigitaljim | im sure i can come up with something :) | 21:37 |
| strigazi | maybe a spec to sync with nodegroups | 21:38 |
| imdigitaljim | yeah i plan to watch your nodegroups changes and improve/port that | 21:38 |
| imdigitaljim | (were not using it so i might need you to test it) | 21:38 |
| strigazi | sure | 21:39 |
| imdigitaljim | i will still be pushing a few for common | 21:39 |
| imdigitaljim | that can help any/all drivers | 21:39 |
| imdigitaljim | flwang: btw congrats on the prod deploy! | 21:40 |
| flwang | imdigitaljim: thanks, it wouldn't happen without you guys help | 21:41 |
| flwang | strigazi: +1 for spec for v2 | 21:41 |
| flwang | I assume it a big change, so better to have a spec | 21:41 |
| flwang | or at least doing the code and spec in parallel, so that reviewer can understand the design correctly | 21:41 |
| imdigitaljim | yeah totally understandable | 21:41 |
| imdigitaljim | some of the goals of the v2 was improved readability as well | 21:42 |
| flwang | imdigitaljim: quick question, does the v2 involve any API change? | 21:42 |
| imdigitaljim | no | 21:42 |
| flwang | nice | 21:42 |
| imdigitaljim | core magnum is mostly untouched | 21:42 |
| flwang | it's great | 21:43 |
| imdigitaljim | there are a some constraints heat >= queens/rocky?, octavia (if used) >= rocky, kubernetes >= v1.11.X | 21:44 |
| imdigitaljim | i think heat can be pike if not using octavia though | 21:45 |
| *** munimeha1 has quit IRC | 21:45 | |
| imdigitaljim | octavia >= queens* | 21:45 |
| strigazi | queens is a reasonable requirement | 21:46 |
| *** mattgo has joined #openstack-containers | 21:48 | |
| strigazi | Anything else for the meeting? | 21:49 |
| flwang | strigazi: i'm good | 21:49 |
| strigazi | others? | 21:50 |
| tobias-urdin | the issue we talked about earlier today is fixed in https://review.openstack.org/#/c/611097/ | 21:50 |
| tobias-urdin | nothing fancy :) | 21:51 |
| strigazi | thanks +2 | 21:51 |
| tobias-urdin | should be safe for backports as well | 21:51 |
| strigazi | thanks everyone for joining the meeting, see you | 21:52 |
| strigazi | #endmeeting | 21:53 |
| *** openstack changes topic to "OpenStack Containers Team" | 21:53 | |
| openstack | Meeting ended Tue Oct 16 21:53:10 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:53 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-10-16-21.01.html | 21:53 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-10-16-21.01.txt | 21:53 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/containers/2018/containers.2018-10-16-21.01.log.html | 21:53 |
| strigazi | tobias-urdin: we will backport it, for sure | 21:53 |
| * strigazi is going offline | 21:53 | |
| flwang | strigazi: o/ | 21:54 |
| *** mattgo has quit IRC | 22:07 | |
| *** mattgo has joined #openstack-containers | 22:09 | |
| *** mattgo has quit IRC | 22:19 | |
| *** rcernin has joined #openstack-containers | 22:49 | |
| *** hongbin has quit IRC | 23:07 | |
| *** dave-mccowan has quit IRC | 23:24 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!