| *** sapd1_x has joined #openstack-self-healing | 00:05 | |
| *** sapd1_x has quit IRC | 00:35 | |
| *** mrunge has quit IRC | 01:07 | |
| *** mrunge has joined #openstack-self-healing | 03:43 | |
| *** ricolin has joined #openstack-self-healing | 03:58 | |
| *** witek has joined #openstack-self-healing | 06:57 | |
| *** rakhmerov has joined #openstack-self-healing | 08:01 | |
| aspiers | morning! anyone got topics they want to discuss today? | 09:04 |
|---|---|---|
| witek | hi aspiers, I could shortly report from the billing initiative meeting | 09:05 |
| aspiers | sure | 09:05 |
| aspiers | #startmeeting self-healing | 09:05 |
| openstack | Meeting started Wed Jun 5 09:05:20 2019 UTC and is due to finish in 60 minutes. The chair is aspiers. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:05 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:05 |
| *** openstack changes topic to " (Meeting topic: self-healing)" | 09:05 | |
| openstack | The meeting name has been set to 'self_healing' | 09:05 |
| aspiers | #topic report from billing initiative meeting | 09:05 |
| *** openstack changes topic to "report from billing initiative meeting (Meeting topic: self-healing)" | 09:05 | |
| aspiers | go for it :) | 09:05 |
| witek | :) | 09:05 |
| witek | Public Cloud SIG has started the "Billing Initiative" | 09:06 |
| witek | I have attended the last two meetings | 09:06 |
| witek | http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-05-23-14.00.log.html | 09:06 |
| witek | http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-05-28-14.05.log.html | 09:06 |
| witek | the discussions so far focused on collecting and storing the required measurements | 09:07 |
| witek | I was promoting the idea of instrumenting the code of OpenStack services | 09:08 |
| aspiers | OK | 09:08 |
| witek | I think it's in a long term the best approach both for billing and monitoring in general | 09:08 |
| witek | guys have not been very enthusiastic about it though | 09:09 |
| aspiers | Does that tie in with what we discussed in Denver with Dirk / Ben / others about exposing internal service metrics via endpoints? | 09:09 |
| witek | yes, that's the same approach | 09:10 |
| aspiers | OK | 09:10 |
| aspiers | I just had a crazy idea for the future :) | 09:10 |
| aspiers | Billing could offer refunds if monitoring sees that users are impacted by outages | 09:11 |
| aspiers | that kind of ties billing with self-healing | 09:11 |
| witek | nice use case :) | 09:11 |
| witek | anyway, they pointed out that the implementation will take long time and were not optimistic if it can be finished at all | 09:13 |
| witek | Mohamed Nasser suggested writing Prometheus exporters instead | 09:13 |
| aspiers | OK | 09:13 |
| witek | I think, instrumenting the code is not much effort, it just requires coordination with other projects | 09:15 |
| aspiers | Yeah | 09:15 |
| witek | and will pay off in long term | 09:15 |
| aspiers | Maybe write a spec? | 09:15 |
| witek | because instrumentation will live with the code | 09:15 |
| witek | and is able to collect more data than `black box` monitoring with exporter | 09:15 |
| aspiers | Yeah | 09:16 |
| aspiers | A spec would be a good way to help people understand your vision | 09:16 |
| aspiers | You could give some example code to show how it would work | 09:16 |
| witek | yes, I think I should prioritize it on my list | 09:17 |
| witek | that's all from me I guess | 09:17 |
| aspiers | Not sure where the best place for the spec is | 09:18 |
| witek | if we want to provide oslo library, probably there, but I'm not sure if we need one | 09:18 |
| witek | self-healing could be alternatively | 09:19 |
| aspiers | yup | 09:20 |
| aspiers | well you can draft the spec and submit it somewhere | 09:20 |
| aspiers | if it's the wrong place it's easy to move :) | 09:20 |
| witek | correct | 09:20 |
| aspiers | alright, thanks for reporting about that | 09:23 |
| aspiers | #topic AOB | 09:23 |
| *** openstack changes topic to "AOB (Meeting topic: self-healing)" | 09:23 | |
| aspiers | I don't have any updates except I still need to send a report about Denver :-( | 09:23 |
| aspiers | anything else from your side? | 09:23 |
| witek | ideas for session proposals for Shanghai? | 09:24 |
| aspiers | oh, good point | 09:24 |
| aspiers | we should try the same one we submitted last time with Ifat | 09:25 |
| witek | yes, that might work | 09:25 |
| witek | I don't have anything else | 09:27 |
| aspiers | nor me | 09:27 |
| aspiers | when's the deadline for Shanghai? | 09:27 |
| aspiers | July? | 09:27 |
| witek | early July | 09:27 |
| aspiers | OK | 09:27 |
| aspiers | We have time :) | 09:27 |
| aspiers | Maybe we can talk to the new Vitrage PTL about it | 09:27 |
| witek | right | 09:27 |
| aspiers | Cool | 09:28 |
| witek | next two weeks I'm in vacation | 09:28 |
| aspiers | I won't have time this week either | 09:28 |
| aspiers | Do you still have the text of the submission? | 09:28 |
| witek | yes, will find it | 09:28 |
| aspiers | Maybe best to forward to him sooner, so he has time to think about it while you are away | 09:28 |
| aspiers | Cool, thanks | 09:28 |
| aspiers | Alright, thanks again and catch you soon! | 09:29 |
| witek | thanks, bye | 09:29 |
| aspiers | o/ | 09:29 |
| aspiers | #endmeeting | 09:29 |
| *** openstack changes topic to "https://wiki.openstack.org/wiki/Self_healing_SIG | https://storyboard.openstack.org/#!/project/openstack/self-healing-sig" | 09:29 | |
| openstack | Meeting ended Wed Jun 5 09:29:37 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:29 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-09.05.html | 09:29 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-09.05.txt | 09:29 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-09.05.log.html | 09:29 |
| *** ricolin has quit IRC | 10:47 | |
| *** tojuvone has quit IRC | 11:03 | |
| *** tojuvone has joined #openstack-self-healing | 11:04 | |
| *** ricolin has joined #openstack-self-healing | 12:03 | |
| *** sapd1_x has joined #openstack-self-healing | 13:14 | |
| joadavis | re "instrumenting the code is not much effort, it just requires coordination with other projects" - There have been some similar discussion in the Telemetry side about having each service instrument their code and provide notifications. So some projects are already on-board with the idea. | 14:32 |
| aspiers | joadavis: cool | 14:33 |
| *** akhil_jain has joined #openstack-self-healing | 14:50 | |
| *** sapd1_x has quit IRC | 14:59 | |
| *** witek has quit IRC | 15:44 | |
| aspiers | might be a few mins late for the meeting | 16:25 |
| aspiers | if anybody has anything to discuss, feel free to start without me | 16:26 |
| *** ekcs has joined #openstack-self-healing | 16:55 | |
| aspiers | hey ekcs | 17:03 |
| ekcs | hello! | 17:04 |
| ekcs | how’s it been? | 17:04 |
| aspiers | blegh :) | 17:05 |
| aspiers | busy | 17:05 |
| aspiers | how about you? | 17:05 |
| ekcs | haha yea I get it. | 17:05 |
| aspiers | oh, I guess I owe you a pic of my new trackpad | 17:06 |
| ekcs | it’s been pretty good. busy as well. got pulled into a lot more internal work. at least it’s fun work. but does stretch my time that’s for sure. | 17:06 |
| ekcs | yea would love to see how it’s working out! | 17:06 |
| aspiers | nice | 17:06 |
| aspiers | I'm mostly working on nova at the moment | 17:06 |
| aspiers | anyway, do we have anything to discuss today? | 17:07 |
| * aspiers logs into storyboard | 17:07 | |
| ekcs | not too much from me. I’ve made some progress on documenting the monasca-tacker-congress integration, but slowly. | 17:08 |
| aspiers | oh, just remembered a couple of things | 17:08 |
| aspiers | might as well minute them | 17:08 |
| aspiers | #startmeeting self-healing | 17:08 |
| openstack | Meeting started Wed Jun 5 17:08:21 2019 UTC and is due to finish in 60 minutes. The chair is aspiers. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:08 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:08 |
| *** openstack changes topic to " (Meeting topic: self-healing)" | 17:08 | |
| openstack | The meeting name has been set to 'self_healing' | 17:08 |
| aspiers | So this morning witek mentioned some ongoing discussions around billing, and the idea that instrumenting service code in order to provide metrics might work better than black-box monitoring for that | 17:09 |
| aspiers | which ties in with https://storyboard.openstack.org/#!/story/2005632 | 17:09 |
| aspiers | #topic exporting metrics from services | 17:09 |
| *** openstack changes topic to "exporting metrics from services (Meeting topic: self-healing)" | 17:09 | |
| aspiers | BTW we seem to have a duplicate story in storyboard for this I think? | 17:10 |
| aspiers | https://storyboard.openstack.org/#!/story/2005640 | 17:10 |
| aspiers | seem to remember some weirdness with StoryBoard when we were submitting stories recently | 17:10 |
| ekcs | oh weird. yea I may have created a duplicate because of the weirdness. | 17:11 |
| ekcs | I guess we should delete one? | 17:11 |
| aspiers | yeah, https://storyboard.openstack.org/#!/story/2005632 has one fewer task | 17:11 |
| ekcs | ok I’ll delete that one. | 17:11 |
| aspiers | thanks | 17:11 |
| aspiers | not much more to say on that right now except link to this morning's minutes | 17:12 |
| aspiers | #link http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-09.05.html this morning's minutes | 17:12 |
| aspiers | #topic heat + octavia + aodh | 17:13 |
| *** openstack changes topic to "heat + octavia + aodh (Meeting topic: self-healing)" | 17:13 | |
| ekcs | great. yea I read up on the morning meeting. sounds like there isn’t great support just yet, but great thing that witek is working on it. | 17:13 |
| aspiers | so this popped up on the mailing list: | 17:13 |
| aspiers | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006582.html demo of app auto-healing via heat+octavia+aodh | 17:13 |
| aspiers | Didn't get a response though | 17:13 |
| aspiers | We can either keep chasing or try to document at least a skeleton for it ourselves | 17:14 |
| aspiers | #action aspiers to create a story for documenting that use case | 17:14 |
| ekcs | got it. yea first step maybe simply to link to that video in a skeletal doc. I can take a stab at that. | 17:15 |
| aspiers | I'll finish that after the meeting | 17:16 |
| aspiers | I mean, finish creating the story | 17:16 |
| aspiers | That would be awesome if you could kick it off | 17:16 |
| ekcs | yup. | 17:16 |
| aspiers | We can totally merge a skeleton and flesh it out later | 17:16 |
| aspiers | Main thing is promoting the discoverability / awareness | 17:16 |
| aspiers | If people are aware and they need more details, they'll probably ask for them | 17:16 |
| aspiers | #topic automated testing | 17:17 |
| *** openstack changes topic to "automated testing (Meeting topic: self-healing)" | 17:17 | |
| ekcs | sounds good | 17:17 |
| aspiers | This old chestnut :) | 17:17 |
| aspiers | So we *may* have an intern doing a masters thesis on this topic | 17:17 |
| aspiers | in which case we could expect to see some progress | 17:17 |
| aspiers | but nothing guaranteed yet | 17:17 |
| aspiers | fingers crossed! | 17:17 |
| ekcs | oh very nice! I also see that ricolin started some basic tempest setup. | 17:18 |
| aspiers | Yup. IIRC it's still marked WIP so not sure if he needs any help with that | 17:18 |
| ricolin | aspiers, ekcs, yes, it's working already but I'm more working on how to make the test scenario test more stable | 17:19 |
| aspiers | ricolin: cool! | 17:19 |
| aspiers | #link https://storyboard.openstack.org/#!/story/2005830 New story for documenting Heat+Octavia+Aodh | 17:19 |
| aspiers | ricolin: Let us know if you need any help | 17:19 |
| ekcs | awesomeness | 17:20 |
| aspiers | I think that was all I had for now | 17:20 |
| aspiers | #topic AOB | 17:20 |
| ricolin | the self-healing scenario is very unstable in https://review.opendev.org/656070 try to figure out why | 17:20 |
| *** openstack changes topic to "AOB (Meeting topic: self-healing)" | 17:20 | |
| aspiers | ah OK | 17:20 |
| aspiers | anything else? | 17:20 |
| * aspiers takes a look at that review | 17:20 | |
| ekcs | ricolin: are these similar to tests already being run on heat repos? | 17:21 |
| aspiers | heat_tempest_plugin.common.exceptions.TimeoutException: Request timed out | 17:21 |
| aspiers | Details: Stack SelfHealingTest-243821469/c9e222f4-e0f0-4cbf-ba58-dea30d2d6a08 failed to reach UPDATE_COMPLETE status within the required time (1200 s). | 17:21 |
| aspiers | #topic heat self-healing tests | 17:22 |
| *** openstack changes topic to "heat self-healing tests (Meeting topic: self-healing)" | 17:22 | |
| ekcs | knowing what’s new exsting heat tests may help us diagnose. | 17:22 |
| aspiers | true | 17:22 |
| ricolin | the time out is when the healing process didn't start in any reason | 17:23 |
| aspiers | OK | 17:23 |
| aspiers | that's beyond my familiarity right now | 17:24 |
| ricolin | Heat should play better role during entire process and help to make sure all component works well | 17:24 |
| ricolin | and reduce the unstable cases | 17:24 |
| aspiers | do you know why it didn't start? | 17:25 |
| ricolin | I think I got some idea | 17:25 |
| ricolin | but since next week is part of my wedding ceremony, I won't be that available before 6/15 | 17:26 |
| aspiers | Ah! No problem, enjoy! :-D | 17:26 |
| ekcs | oh wow congrats! | 17:26 |
| ricolin | and the rest part happen in 11/17 so it's going to be a very long years for me!lol | 17:27 |
| ricolin | ekcs, aspiers thx! | 17:27 |
| aspiers | haha | 17:27 |
| aspiers | alright | 17:27 |
| aspiers | anything else anyone want to discuss? | 17:27 |
| ricolin | aspiers, in short, I think that test case fail because Heat didn't make sure the Mistral workflow is up and running stable before we assume next step | 17:27 |
| aspiers | ahah, I see | 17:28 |
| ricolin | I will look into that and hope I can bring some good knews | 17:28 |
| ricolin | knews/news | 17:28 |
| aspiers | perfect | 17:28 |
| ekcs | great! | 17:28 |
| ricolin | Once that test is stable, the rest gate job setting will be easy | 17:28 |
| ricolin | since all required patch is already there | 17:29 |
| aspiers | nice | 17:29 |
| aspiers | I guess we need a short doc explaining it too | 17:29 |
| ekcs | not a discussion topic per se, but I’ve been wavering in my personal priority between identifying and supporting new use cases vs documenting existing use cases. I think I settled on documenting existing as higher priority at this stage of the sig. | 17:31 |
| aspiers | personally I think either is fine | 17:31 |
| aspiers | Whatever you are more excited about ;) | 17:31 |
| ekcs | = ) | 17:32 |
| aspiers | Any small contributions are a lot better than nothing :) | 17:32 |
| ekcs | yup | 17:32 |
| aspiers | We're all busy with other stuff, so IMO there's no problem at all with being selective and time-boxing SIG work | 17:33 |
| aspiers | Alright, sounds like we're done for today? | 17:34 |
| ekcs | yup | 17:34 |
| aspiers | cool | 17:34 |
| aspiers | thanks, and catch you soon! | 17:34 |
| ekcs | yup later guys! have a great week! | 17:35 |
| aspiers | o/ | 17:35 |
| aspiers | #endmeeting | 17:35 |
| *** openstack changes topic to "https://wiki.openstack.org/wiki/Self_healing_SIG | https://storyboard.openstack.org/#!/project/openstack/self-healing-sig" | 17:35 | |
| openstack | Meeting ended Wed Jun 5 17:35:30 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:35 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-17.08.html | 17:35 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-17.08.txt | 17:35 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-17.08.log.html | 17:35 |
| *** akhil_jain has quit IRC | 17:40 | |
| *** ricolin has quit IRC | 18:41 | |
| *** joadavis has quit IRC | 22:05 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!