*** sapd1_x has joined #openstack-self-healing | 00:05 | |
*** sapd1_x has quit IRC | 00:35 | |
*** mrunge has quit IRC | 01:07 | |
*** mrunge has joined #openstack-self-healing | 03:43 | |
*** ricolin has joined #openstack-self-healing | 03:58 | |
*** witek has joined #openstack-self-healing | 06:57 | |
*** rakhmerov has joined #openstack-self-healing | 08:01 | |
aspiers | morning! anyone got topics they want to discuss today? | 09:04 |
---|---|---|
witek | hi aspiers, I could shortly report from the billing initiative meeting | 09:05 |
aspiers | sure | 09:05 |
aspiers | #startmeeting self-healing | 09:05 |
openstack | Meeting started Wed Jun 5 09:05:20 2019 UTC and is due to finish in 60 minutes. The chair is aspiers. Information about MeetBot at http://wiki.debian.org/MeetBot. | 09:05 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 09:05 |
*** openstack changes topic to " (Meeting topic: self-healing)" | 09:05 | |
openstack | The meeting name has been set to 'self_healing' | 09:05 |
aspiers | #topic report from billing initiative meeting | 09:05 |
*** openstack changes topic to "report from billing initiative meeting (Meeting topic: self-healing)" | 09:05 | |
aspiers | go for it :) | 09:05 |
witek | :) | 09:05 |
witek | Public Cloud SIG has started the "Billing Initiative" | 09:06 |
witek | I have attended the last two meetings | 09:06 |
witek | http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-05-23-14.00.log.html | 09:06 |
witek | http://eavesdrop.openstack.org/meetings/publiccloud_wg/2019/publiccloud_wg.2019-05-28-14.05.log.html | 09:06 |
witek | the discussions so far focused on collecting and storing the required measurements | 09:07 |
witek | I was promoting the idea of instrumenting the code of OpenStack services | 09:08 |
aspiers | OK | 09:08 |
witek | I think it's in a long term the best approach both for billing and monitoring in general | 09:08 |
witek | guys have not been very enthusiastic about it though | 09:09 |
aspiers | Does that tie in with what we discussed in Denver with Dirk / Ben / others about exposing internal service metrics via endpoints? | 09:09 |
witek | yes, that's the same approach | 09:10 |
aspiers | OK | 09:10 |
aspiers | I just had a crazy idea for the future :) | 09:10 |
aspiers | Billing could offer refunds if monitoring sees that users are impacted by outages | 09:11 |
aspiers | that kind of ties billing with self-healing | 09:11 |
witek | nice use case :) | 09:11 |
witek | anyway, they pointed out that the implementation will take long time and were not optimistic if it can be finished at all | 09:13 |
witek | Mohamed Nasser suggested writing Prometheus exporters instead | 09:13 |
aspiers | OK | 09:13 |
witek | I think, instrumenting the code is not much effort, it just requires coordination with other projects | 09:15 |
aspiers | Yeah | 09:15 |
witek | and will pay off in long term | 09:15 |
aspiers | Maybe write a spec? | 09:15 |
witek | because instrumentation will live with the code | 09:15 |
witek | and is able to collect more data than `black box` monitoring with exporter | 09:15 |
aspiers | Yeah | 09:16 |
aspiers | A spec would be a good way to help people understand your vision | 09:16 |
aspiers | You could give some example code to show how it would work | 09:16 |
witek | yes, I think I should prioritize it on my list | 09:17 |
witek | that's all from me I guess | 09:17 |
aspiers | Not sure where the best place for the spec is | 09:18 |
witek | if we want to provide oslo library, probably there, but I'm not sure if we need one | 09:18 |
witek | self-healing could be alternatively | 09:19 |
aspiers | yup | 09:20 |
aspiers | well you can draft the spec and submit it somewhere | 09:20 |
aspiers | if it's the wrong place it's easy to move :) | 09:20 |
witek | correct | 09:20 |
aspiers | alright, thanks for reporting about that | 09:23 |
aspiers | #topic AOB | 09:23 |
*** openstack changes topic to "AOB (Meeting topic: self-healing)" | 09:23 | |
aspiers | I don't have any updates except I still need to send a report about Denver :-( | 09:23 |
aspiers | anything else from your side? | 09:23 |
witek | ideas for session proposals for Shanghai? | 09:24 |
aspiers | oh, good point | 09:24 |
aspiers | we should try the same one we submitted last time with Ifat | 09:25 |
witek | yes, that might work | 09:25 |
witek | I don't have anything else | 09:27 |
aspiers | nor me | 09:27 |
aspiers | when's the deadline for Shanghai? | 09:27 |
aspiers | July? | 09:27 |
witek | early July | 09:27 |
aspiers | OK | 09:27 |
aspiers | We have time :) | 09:27 |
aspiers | Maybe we can talk to the new Vitrage PTL about it | 09:27 |
witek | right | 09:27 |
aspiers | Cool | 09:28 |
witek | next two weeks I'm in vacation | 09:28 |
aspiers | I won't have time this week either | 09:28 |
aspiers | Do you still have the text of the submission? | 09:28 |
witek | yes, will find it | 09:28 |
aspiers | Maybe best to forward to him sooner, so he has time to think about it while you are away | 09:28 |
aspiers | Cool, thanks | 09:28 |
aspiers | Alright, thanks again and catch you soon! | 09:29 |
witek | thanks, bye | 09:29 |
aspiers | o/ | 09:29 |
aspiers | #endmeeting | 09:29 |
*** openstack changes topic to "https://wiki.openstack.org/wiki/Self_healing_SIG | https://storyboard.openstack.org/#!/project/openstack/self-healing-sig" | 09:29 | |
openstack | Meeting ended Wed Jun 5 09:29:37 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 09:29 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-09.05.html | 09:29 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-09.05.txt | 09:29 |
openstack | Log: http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-09.05.log.html | 09:29 |
*** ricolin has quit IRC | 10:47 | |
*** tojuvone has quit IRC | 11:03 | |
*** tojuvone has joined #openstack-self-healing | 11:04 | |
*** ricolin has joined #openstack-self-healing | 12:03 | |
*** sapd1_x has joined #openstack-self-healing | 13:14 | |
joadavis | re "instrumenting the code is not much effort, it just requires coordination with other projects" - There have been some similar discussion in the Telemetry side about having each service instrument their code and provide notifications. So some projects are already on-board with the idea. | 14:32 |
aspiers | joadavis: cool | 14:33 |
*** akhil_jain has joined #openstack-self-healing | 14:50 | |
*** sapd1_x has quit IRC | 14:59 | |
*** witek has quit IRC | 15:44 | |
aspiers | might be a few mins late for the meeting | 16:25 |
aspiers | if anybody has anything to discuss, feel free to start without me | 16:26 |
*** ekcs has joined #openstack-self-healing | 16:55 | |
aspiers | hey ekcs | 17:03 |
ekcs | hello! | 17:04 |
ekcs | how’s it been? | 17:04 |
aspiers | blegh :) | 17:05 |
aspiers | busy | 17:05 |
aspiers | how about you? | 17:05 |
ekcs | haha yea I get it. | 17:05 |
aspiers | oh, I guess I owe you a pic of my new trackpad | 17:06 |
ekcs | it’s been pretty good. busy as well. got pulled into a lot more internal work. at least it’s fun work. but does stretch my time that’s for sure. | 17:06 |
ekcs | yea would love to see how it’s working out! | 17:06 |
aspiers | nice | 17:06 |
aspiers | I'm mostly working on nova at the moment | 17:06 |
aspiers | anyway, do we have anything to discuss today? | 17:07 |
* aspiers logs into storyboard | 17:07 | |
ekcs | not too much from me. I’ve made some progress on documenting the monasca-tacker-congress integration, but slowly. | 17:08 |
aspiers | oh, just remembered a couple of things | 17:08 |
aspiers | might as well minute them | 17:08 |
aspiers | #startmeeting self-healing | 17:08 |
openstack | Meeting started Wed Jun 5 17:08:21 2019 UTC and is due to finish in 60 minutes. The chair is aspiers. Information about MeetBot at http://wiki.debian.org/MeetBot. | 17:08 |
openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 17:08 |
*** openstack changes topic to " (Meeting topic: self-healing)" | 17:08 | |
openstack | The meeting name has been set to 'self_healing' | 17:08 |
aspiers | So this morning witek mentioned some ongoing discussions around billing, and the idea that instrumenting service code in order to provide metrics might work better than black-box monitoring for that | 17:09 |
aspiers | which ties in with https://storyboard.openstack.org/#!/story/2005632 | 17:09 |
aspiers | #topic exporting metrics from services | 17:09 |
*** openstack changes topic to "exporting metrics from services (Meeting topic: self-healing)" | 17:09 | |
aspiers | BTW we seem to have a duplicate story in storyboard for this I think? | 17:10 |
aspiers | https://storyboard.openstack.org/#!/story/2005640 | 17:10 |
aspiers | seem to remember some weirdness with StoryBoard when we were submitting stories recently | 17:10 |
ekcs | oh weird. yea I may have created a duplicate because of the weirdness. | 17:11 |
ekcs | I guess we should delete one? | 17:11 |
aspiers | yeah, https://storyboard.openstack.org/#!/story/2005632 has one fewer task | 17:11 |
ekcs | ok I’ll delete that one. | 17:11 |
aspiers | thanks | 17:11 |
aspiers | not much more to say on that right now except link to this morning's minutes | 17:12 |
aspiers | #link http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-09.05.html this morning's minutes | 17:12 |
aspiers | #topic heat + octavia + aodh | 17:13 |
*** openstack changes topic to "heat + octavia + aodh (Meeting topic: self-healing)" | 17:13 | |
ekcs | great. yea I read up on the morning meeting. sounds like there isn’t great support just yet, but great thing that witek is working on it. | 17:13 |
aspiers | so this popped up on the mailing list: | 17:13 |
aspiers | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006582.html demo of app auto-healing via heat+octavia+aodh | 17:13 |
aspiers | Didn't get a response though | 17:13 |
aspiers | We can either keep chasing or try to document at least a skeleton for it ourselves | 17:14 |
aspiers | #action aspiers to create a story for documenting that use case | 17:14 |
ekcs | got it. yea first step maybe simply to link to that video in a skeletal doc. I can take a stab at that. | 17:15 |
aspiers | I'll finish that after the meeting | 17:16 |
aspiers | I mean, finish creating the story | 17:16 |
aspiers | That would be awesome if you could kick it off | 17:16 |
ekcs | yup. | 17:16 |
aspiers | We can totally merge a skeleton and flesh it out later | 17:16 |
aspiers | Main thing is promoting the discoverability / awareness | 17:16 |
aspiers | If people are aware and they need more details, they'll probably ask for them | 17:16 |
aspiers | #topic automated testing | 17:17 |
*** openstack changes topic to "automated testing (Meeting topic: self-healing)" | 17:17 | |
ekcs | sounds good | 17:17 |
aspiers | This old chestnut :) | 17:17 |
aspiers | So we *may* have an intern doing a masters thesis on this topic | 17:17 |
aspiers | in which case we could expect to see some progress | 17:17 |
aspiers | but nothing guaranteed yet | 17:17 |
aspiers | fingers crossed! | 17:17 |
ekcs | oh very nice! I also see that ricolin started some basic tempest setup. | 17:18 |
aspiers | Yup. IIRC it's still marked WIP so not sure if he needs any help with that | 17:18 |
ricolin | aspiers, ekcs, yes, it's working already but I'm more working on how to make the test scenario test more stable | 17:19 |
aspiers | ricolin: cool! | 17:19 |
aspiers | #link https://storyboard.openstack.org/#!/story/2005830 New story for documenting Heat+Octavia+Aodh | 17:19 |
aspiers | ricolin: Let us know if you need any help | 17:19 |
ekcs | awesomeness | 17:20 |
aspiers | I think that was all I had for now | 17:20 |
aspiers | #topic AOB | 17:20 |
ricolin | the self-healing scenario is very unstable in https://review.opendev.org/656070 try to figure out why | 17:20 |
*** openstack changes topic to "AOB (Meeting topic: self-healing)" | 17:20 | |
aspiers | ah OK | 17:20 |
aspiers | anything else? | 17:20 |
* aspiers takes a look at that review | 17:20 | |
ekcs | ricolin: are these similar to tests already being run on heat repos? | 17:21 |
aspiers | heat_tempest_plugin.common.exceptions.TimeoutException: Request timed out | 17:21 |
aspiers | Details: Stack SelfHealingTest-243821469/c9e222f4-e0f0-4cbf-ba58-dea30d2d6a08 failed to reach UPDATE_COMPLETE status within the required time (1200 s). | 17:21 |
aspiers | #topic heat self-healing tests | 17:22 |
*** openstack changes topic to "heat self-healing tests (Meeting topic: self-healing)" | 17:22 | |
ekcs | knowing what’s new exsting heat tests may help us diagnose. | 17:22 |
aspiers | true | 17:22 |
ricolin | the time out is when the healing process didn't start in any reason | 17:23 |
aspiers | OK | 17:23 |
aspiers | that's beyond my familiarity right now | 17:24 |
ricolin | Heat should play better role during entire process and help to make sure all component works well | 17:24 |
ricolin | and reduce the unstable cases | 17:24 |
aspiers | do you know why it didn't start? | 17:25 |
ricolin | I think I got some idea | 17:25 |
ricolin | but since next week is part of my wedding ceremony, I won't be that available before 6/15 | 17:26 |
aspiers | Ah! No problem, enjoy! :-D | 17:26 |
ekcs | oh wow congrats! | 17:26 |
ricolin | and the rest part happen in 11/17 so it's going to be a very long years for me!lol | 17:27 |
ricolin | ekcs, aspiers thx! | 17:27 |
aspiers | haha | 17:27 |
aspiers | alright | 17:27 |
aspiers | anything else anyone want to discuss? | 17:27 |
ricolin | aspiers, in short, I think that test case fail because Heat didn't make sure the Mistral workflow is up and running stable before we assume next step | 17:27 |
aspiers | ahah, I see | 17:28 |
ricolin | I will look into that and hope I can bring some good knews | 17:28 |
ricolin | knews/news | 17:28 |
aspiers | perfect | 17:28 |
ekcs | great! | 17:28 |
ricolin | Once that test is stable, the rest gate job setting will be easy | 17:28 |
ricolin | since all required patch is already there | 17:29 |
aspiers | nice | 17:29 |
aspiers | I guess we need a short doc explaining it too | 17:29 |
ekcs | not a discussion topic per se, but I’ve been wavering in my personal priority between identifying and supporting new use cases vs documenting existing use cases. I think I settled on documenting existing as higher priority at this stage of the sig. | 17:31 |
aspiers | personally I think either is fine | 17:31 |
aspiers | Whatever you are more excited about ;) | 17:31 |
ekcs | = ) | 17:32 |
aspiers | Any small contributions are a lot better than nothing :) | 17:32 |
ekcs | yup | 17:32 |
aspiers | We're all busy with other stuff, so IMO there's no problem at all with being selective and time-boxing SIG work | 17:33 |
aspiers | Alright, sounds like we're done for today? | 17:34 |
ekcs | yup | 17:34 |
aspiers | cool | 17:34 |
aspiers | thanks, and catch you soon! | 17:34 |
ekcs | yup later guys! have a great week! | 17:35 |
aspiers | o/ | 17:35 |
aspiers | #endmeeting | 17:35 |
*** openstack changes topic to "https://wiki.openstack.org/wiki/Self_healing_SIG | https://storyboard.openstack.org/#!/project/openstack/self-healing-sig" | 17:35 | |
openstack | Meeting ended Wed Jun 5 17:35:30 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:35 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-17.08.html | 17:35 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-17.08.txt | 17:35 |
openstack | Log: http://eavesdrop.openstack.org/meetings/self_healing/2019/self_healing.2019-06-05-17.08.log.html | 17:35 |
*** akhil_jain has quit IRC | 17:40 | |
*** ricolin has quit IRC | 18:41 | |
*** joadavis has quit IRC | 22:05 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!