08:00:08 <licanwei> #startmeeting Watcher
08:00:09 <openstack> Meeting started Wed Oct  9 08:00:08 2019 UTC and is due to finish in 60 minutes.  The chair is licanwei. Information about MeetBot at http://wiki.debian.org/MeetBot.
08:00:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
08:00:12 <openstack> The meeting name has been set to 'watcher'
08:00:14 <Dantalion> hello
08:00:39 <licanwei> Dantalion: hi
08:01:26 <chenke> Hello
08:01:41 <licanwei> #topic Announcements
08:02:16 <licanwei> Train will be released next week
08:02:39 <Dantalion> Anything that remains to be done?
08:02:44 <chenke> Time is very close.
08:03:03 <licanwei> yes
08:04:20 <licanwei> Watcher is ready for the new release
08:04:47 <Dantalion> That's good to hear :)
08:05:35 <chenke> Great.
08:07:04 <licanwei> #topic Review Action Items
08:08:14 <licanwei> https://review.opendev.org/#/c/686110/ Fix issues on stable/rocky needs review
08:08:35 <licanwei> It's for Rocky
08:09:00 <Dantalion> Just so im sure Rocky -> Stein -> Train ? right
08:09:15 <licanwei> yes
08:09:17 <chenke> Ye. This friend submits a bug report  https://bugs.launchpad.net/watcher/+bug/1847434
08:09:17 <openstack> Launchpad bug 1847434 in watcher "Missing Metrics of an Instance stops audit execution" [Undecided,New]
08:09:53 <Dantalion> I can look at it this friday I will have time to work on Watcher again
08:10:21 <Dantalion> I remember An old problem with eventlet and threading, we switched threadpool to resolve it, rocky must still use eventlet
08:10:26 <licanwei> hi soumplis, are you here?
08:10:35 <soumplis> yes here
08:10:39 <soumplis> hi all
08:10:57 <Dantalion> hi :)
08:11:05 <licanwei> soumplis plans to join Watcher team
08:11:16 <chenke> It sounds like you will continue to work on watcher for a long time？
08:11:29 <chenke> Hi soumplis. Welcome.
08:11:48 <chenke> Dantalion right?
08:12:15 <Dantalion> That is my plan yes and welcome soumplis Watcher could use the help
08:12:26 <licanwei> Dantalion: Good news!
08:12:54 <licanwei> https://review.opendev.org/#/c/686888/ Use Horizon project template for django jobs needs review
08:13:18 <licanwei> It's for Watcher dashboard
08:13:26 <soumplis> Our team (currently present skatsaounis also) is investing on Watcher and we plan to use it on all our datacenters as our infrastructure expands. Currently one DC with ~140 nova-compute nodes
08:15:03 <chenke> How many virtual machines will be on each node
08:15:17 <licanwei> soumplis: for your case, I suggest you use the latest Train
08:16:45 <licanwei> #topic discussions
08:17:06 <licanwei> Dantalion: your turn~
08:17:35 <soumplis> @chenke We do not really know as apart from traditional IaaS service we have a high demand for kubernetes clusters, so we encourage our users to utilize Openstack to deploy their own kubernetes clusters (using juju). So we expect high variation on the workloads and having Watcher in place seems to be of paramount importance
08:18:20 <chenke> soumplis Thanks for your explain.
08:18:21 <Dantalion> A few weeks ago we send my thesis to a select few via email, I think now is the time to send it on the OpenStack mailing list
08:18:23 <soumplis> @licanwei Our strategy is to go with the previous of the latest stable released, so now that Train will be out, we will go for Stein.
08:19:07 <licanwei> Dantalion: yes, It's great!
08:20:24 <licanwei> soumplis: thanks
08:20:32 <chenke> Dantalion This is a great opportunity to promote our watcher.
08:21:33 <Dantalion> I think so to I hope many OpenStack users will read it, I will send it this friday
08:21:53 <licanwei> Helm chart for Watcher and accompanying docker.io image
08:22:06 <licanwei> Dantalion: It's a good idea
08:23:14 <licanwei> Dantalion: CERN plan to use Helm chart?
08:23:26 <Dantalion> Yes, next discussion?
08:23:56 <Dantalion> Someone has done most of the work creating a basic helm chart for me
08:24:22 <Dantalion> I only need to give it a good look and setup LOCI to push the docker.io images than Watcher will have a helm chart :)
08:24:42 <licanwei> I'm new to Helm chart~
08:25:57 <licanwei> We need to add Watcher chart to Openstack Helm project
08:26:01 <Dantalion> It will allow OpenStack users to deploy Watcher on kubernetes. You can configure many things in the chart such as the number of appliers and have load balancers in front.
08:26:16 <Dantalion> This is all pretty new to me to so it will take some time to understand the entire chart.
08:26:59 <licanwei> We can create a bp for this work
08:28:05 <Dantalion> Yes I think it makes sense to track all these activities even though they will be patches for openstack-helm-charts and loci
08:30:13 <licanwei> agree~
08:30:55 <Dantalion> I have one final note about this bug: https://bugs.launchpad.net/watcher/+bug/1847091
08:30:55 <openstack> Launchpad bug 1847091 in watcher "Actions do not start executing" [Undecided,New]
08:31:12 <Dantalion> It says: "I am running the latest stable/rocky from github on Ubuntu 18.04"
08:31:24 <Dantalion> I think this is caused by still using eventlet
08:32:15 <Dantalion> I will check friday
08:32:16 <soumplis> Do you want me to check it somehow ?
08:32:33 <licanwei> Dantalion: Taskflow uses eventlet?
08:32:55 <Dantalion> Because I can't remember if the applier used eventlet or the decision-engine did
08:33:41 <licanwei> I'll check the patch about eventlet after meeting
08:33:54 <licanwei> maybe you are right
08:34:11 <chenke> Do you think this failure is caused by eventlet?
08:35:05 <Dantalion> I remember we needed to switch from eventlet to different threadpool because it was giving us deadlock / livelock problems with unit testing. But it could be that it was not limited to unit testing but we simply did not encounter it elsewhere
08:36:46 <licanwei> soumplis: https://review.opendev.org/#/c/611275/
08:37:15 <licanwei> soumplis: there are some changes about apllier work engine
08:38:01 <chenke> Dantalion we can put this as an important task in the next U version.
08:39:04 <licanwei> soumplis: You can merge this commit and test again
08:39:48 <soumplis> This patch seems to apply to Python 3, yet we use python2.7. So maybe we'll have to change the python version also.
08:39:56 <soumplis> I'll do some quick testing
08:43:17 <licanwei> soumplis: https://review.opendev.org/#/c/611275/7/watcher/applier/workflow_engine/default.py
08:43:51 <licanwei> soumplis: add  executor='greenthreaded' to engines.load
08:45:15 <licanwei> any questions?
08:46:06 <licanwei> I'll close the meeting if no more questions
08:46:14 <Dantalion> no more questions
08:46:57 <licanwei> bye~
08:47:04 <Dantalion> until next time
08:47:19 <chenke> thanks all.
08:47:22 <chenke> bye
08:47:25 <licanwei> #endmeeting