| *** jmasud has joined #openstack-meeting | 00:08 | |
| *** rakhmerov has quit IRC | 00:13 | |
| *** yasufum has joined #openstack-meeting | 00:30 | |
| *** rfolco|ruck has joined #openstack-meeting | 00:31 | |
| *** rfolco|ruck has quit IRC | 00:36 | |
| *** rfolco|ruck has joined #openstack-meeting | 00:42 | |
| *** rfolco|ruck has quit IRC | 00:47 | |
| *** zigo has quit IRC | 00:53 | |
| *** moguimar has quit IRC | 01:13 | |
| *** Liang__ has joined #openstack-meeting | 01:23 | |
| *** gyee has quit IRC | 01:33 | |
| *** zigo has joined #openstack-meeting | 02:27 | |
| *** johnsom has quit IRC | 02:27 | |
| *** baojg has quit IRC | 02:27 | |
| *** haleyb has quit IRC | 02:27 | |
| *** rbudden has quit IRC | 02:31 | |
| *** rfolco|ruck has joined #openstack-meeting | 02:32 | |
| *** johnsom has joined #openstack-meeting | 02:33 | |
| *** baojg has joined #openstack-meeting | 02:33 | |
| *** Liang__ has quit IRC | 02:34 | |
| *** rfolco|ruck has quit IRC | 02:37 | |
| *** Liang__ has joined #openstack-meeting | 02:37 | |
| *** tetsuro has quit IRC | 03:08 | |
| *** tetsuro has joined #openstack-meeting | 03:10 | |
| *** yamamoto has joined #openstack-meeting | 03:11 | |
| *** andrebeltrami has quit IRC | 03:37 | |
| *** Liang__ has quit IRC | 03:38 | |
| *** tetsuro has quit IRC | 03:42 | |
| *** tetsuro has joined #openstack-meeting | 03:43 | |
| *** ociuhandu has joined #openstack-meeting | 03:47 | |
| *** tetsuro has quit IRC | 03:48 | |
| *** ociuhandu has quit IRC | 03:52 | |
| *** tetsuro has joined #openstack-meeting | 03:55 | |
| *** manpreet has quit IRC | 04:24 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #openstack-meeting | 04:33 | |
| *** manpreet has joined #openstack-meeting | 05:02 | |
| *** vishalmanchanda has joined #openstack-meeting | 05:19 | |
| *** ianychoi_ is now known as ianychoi | 06:16 | |
| *** apetrich has joined #openstack-meeting | 06:26 | |
| *** ralonsoh has joined #openstack-meeting | 06:29 | |
| *** e0ne has joined #openstack-meeting | 06:41 | |
| *** e0ne has quit IRC | 06:45 | |
| *** dklyle has quit IRC | 06:47 | |
| *** apetrich has quit IRC | 06:56 | |
| *** apetrich has joined #openstack-meeting | 07:14 | |
| *** tosky has joined #openstack-meeting | 07:19 | |
| *** slaweq has joined #openstack-meeting | 07:27 | |
| *** rcernin has quit IRC | 07:45 | |
| *** ociuhandu has joined #openstack-meeting | 07:47 | |
| *** priteau has joined #openstack-meeting | 07:50 | |
| *** masahito has joined #openstack-meeting | 08:06 | |
| *** hyunsikyang has joined #openstack-meeting | 08:09 | |
| *** belmoreira has joined #openstack-meeting | 08:12 | |
| *** moguimar has joined #openstack-meeting | 08:12 | |
| *** ociuhandu has quit IRC | 08:13 | |
| *** ociuhandu has joined #openstack-meeting | 08:13 | |
| *** ociuhandu has quit IRC | 08:20 | |
| *** ociuhandu has joined #openstack-meeting | 08:25 | |
| *** Lucas_Gray has joined #openstack-meeting | 08:31 | |
| *** ociuhandu has quit IRC | 08:45 | |
| *** masahito has quit IRC | 08:55 | |
| *** priteau has quit IRC | 08:56 | |
| *** strigazi has joined #openstack-meeting | 09:06 | |
| *** tetsuro has quit IRC | 09:16 | |
| *** lpetrut has joined #openstack-meeting | 09:16 | |
| *** moguimar has quit IRC | 09:58 | |
| *** yamamoto has quit IRC | 09:59 | |
| *** Liang__ has joined #openstack-meeting | 10:03 | |
| *** Lucas_Gray has quit IRC | 10:07 | |
| *** slaweq has quit IRC | 10:17 | |
| *** yamamoto has joined #openstack-meeting | 10:19 | |
| *** moguimar has joined #openstack-meeting | 10:25 | |
| *** yamamoto has quit IRC | 10:26 | |
| *** slaweq has joined #openstack-meeting | 10:27 | |
| *** ociuhandu has joined #openstack-meeting | 10:29 | |
| *** Liang__ has quit IRC | 10:30 | |
| *** rfolco|ruck has joined #openstack-meeting | 10:31 | |
| *** yamamoto has joined #openstack-meeting | 10:34 | |
| *** rfolco|ruck has quit IRC | 10:37 | |
| *** ociuhandu has quit IRC | 10:38 | |
| *** yasufum has quit IRC | 10:38 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** Lucas_Gray has joined #openstack-meeting | 10:53 | |
| *** oneswig has joined #openstack-meeting | 11:03 | |
| oneswig | #startmeeting scientific-sig | 11:03 |
|---|---|---|
| openstack | Meeting started Wed Sep 23 11:03:55 2020 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:03 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:03 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:03 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:03 |
| oneswig | apologies for lateness | 11:04 |
| *** ociuhandu has joined #openstack-meeting | 11:05 | |
| belmoreira | o/ | 11:10 |
| belmoreira | hi oneswig | 11:11 |
| oneswig | Hi belmoreira, how's things? | 11:11 |
| oneswig | priteau was mentioning the discussion in the large scale sig this morning | 11:11 |
| belmoreira | good, and with you? | 11:12 |
| oneswig | I have been negligent in my attention to the Scientific SIG :-( | 11:12 |
| oneswig | Otherwise things are going well I'd say | 11:12 |
| belmoreira | yes, we had the large scale sig meeting this morning and the discussion for the next goals and PTG | 11:12 |
| *** e0ne has joined #openstack-meeting | 11:12 | |
| belmoreira | good to hear that | 11:13 |
| oneswig | What are the pain points you are having with scaling? | 11:14 |
| *** yamamoto has joined #openstack-meeting | 11:15 | |
| belmoreira | well, I think we hit everything possible to hit during the last years... for now until we can physically add more nodes we should be good | 11:17 |
| oneswig | Pierre mentioned you were describing limitations with cells because of not helping network scaling - is that new or something you've been fighting from the beginning? | 11:20 |
| belmoreira | it's more related with neutron scalability, not nova or cells design | 11:20 |
| belmoreira | last year we split the infrastructure in 2 regions. Currently we have 3 regions | 11:21 |
| *** raildo has joined #openstack-meeting | 11:25 | |
| *** yamamoto has quit IRC | 11:25 | |
| belmoreira | I think it will be interesting to share it | 11:26 |
| oneswig | How does storage link between the 3 regions, do you have to share storage? | 11:27 |
| janders | hi oneswig belmoreira | 11:36 |
| janders | belmoreira do you have multiple cells per region, do I get that right? | 11:36 |
| *** b1airo has joined #openstack-meeting | 11:36 | |
| oneswig | Hi janders, good to see you | 11:36 |
| janders | oneswig good to see you too. Apologies for joining late - team meeting clash. | 11:36 |
| b1airo | g'evening | 11:36 |
| janders | hey b1airo! | 11:37 |
| b1airo | my late excuse is more about beer... | 11:37 |
| janders | b1airo important! :) | 11:37 |
| oneswig | Hi b1airo, very important. Back home? | 11:38 |
| oneswig | #chair b1airo | 11:38 |
| openstack | Current chairs: b1airo oneswig | 11:38 |
| b1airo | oh totally janders , certainly higher priority than meetings anyways :-P | 11:38 |
| janders | I look forward to times when we can combine both again | 11:38 |
| janders | as it should be | 11:38 |
| b1airo | no oneswig , still up north hanging out with the NIWA crew | 11:38 |
| belmoreira | janders yes, we have multiple cells | 11:39 |
| b1airo | how many now belmoreira ? | 11:39 |
| belmoreira | in 3 regions we have more than 70 cells in total | 11:39 |
| b1airo | *whistle* | 11:39 |
| janders | belmoreira do any of your cells span regions, or is each cell contained within one region? | 11:40 |
| belmoreira | each cell has a maximum of 250 nodes | 11:40 |
| belmoreira | janders cells are per region | 11:40 |
| janders | belmoreira which aspect of scalability do cells help with the most in your experience? | 11:40 |
| b1airo | are you still following the same, err, "disposable" cell controllers model? :-) | 11:40 |
| belmoreira | each cell has it's own rabbit infrastructure | 11:41 |
| *** rfolco|ruck has joined #openstack-meeting | 11:41 | |
| b1airo | (i vaguely recall you are running your cell controllers within the prod cloud itself... 🐢) | 11:41 |
| belmoreira | and it's a good failure domain, in case of issues things are contained | 11:41 |
| belmoreira | b1airo :) yes, all our controller plane runs inside the cloud itself (inception) | 11:42 |
| janders | belmoreira nice! :) | 11:43 |
| b1airo | agree on that - spanning regions (a user facing construct) across cells (a backend scalability and failure domain concern) seems like a questionable idea | 11:43 |
| janders | belmoreira does this architecture pose a challenge in case of a need of a full-system shutdown? | 11:43 |
| belmoreira | you mean a shutdown in the data centre :) | 11:44 |
| janders | belmoreira I really like it, just wonder what extra measures are needed to prevent losing the "starer motor" | 11:44 |
| janders | belmoreira yeah | 11:44 |
| belmoreira | yes, sure... if that happens we need to understand what needs to be available first | 11:45 |
| b1airo | i guess maybe the api top-level needs to come up first, followed by compute "cell0" (i guess that must be a thing in this architecture? | 11:46 |
| belmoreira | but is not a big issue, because instance start doesn't need the control plane | 11:46 |
| janders | belmoreira right! | 11:46 |
| janders | belmoreira do you have dedicated compute nodes for infra services, so that they are separate from user workloads and easy to identify? | 11:47 |
| belmoreira | b1airo yes, if we really need APIs from the beginning, but if a disaster happens APIs availability will be probably the last | 11:47 |
| b1airo | ha, good point! so "cell0" is really just select instance startup directly on compute nodes? | 11:47 |
| janders | do the infra instances have networking statically configured? | 11:48 |
| janders | (cause I suppose DHCP services may not be available yet) | 11:48 |
| b1airo | was coming to the networking question too :-) | 11:49 |
| janders | or is the inception arch cell-specific, with neutron being independent of this? | 11:49 |
| belmoreira | b1airo in a case of a disaster we will probably force instance start per compute node | 11:49 |
| belmoreira | and worry to have the DBs up | 11:49 |
| belmoreira | janders we use DHCP, but it's a separate infrastructure... yes, it needs to be up | 11:51 |
| janders | belmoreira makes sense | 11:51 |
| belmoreira | janders users and infra instances share the same infrastructure | 11:52 |
| janders | belmoreira no noisy neighbour issues? | 11:52 |
| belmoreira | only compute instances have their dedicated cells/regions | 11:52 |
| belmoreira | janders yes, sometimes... we usually live migrate noisy neighbours to less busy compute nodes | 11:53 |
| *** martial_ has joined #openstack-meeting | 11:53 | |
| martial_ | Late (still getting kids ready) | 11:54 |
| oneswig | Hi martial_, morning | 11:54 |
| janders | belmoreira it's awesome to hear about your architecture and your experiences with it, thanks for sharing! | 11:55 |
| oneswig | janders: how's things with you? | 11:55 |
| janders | oneswig good, thank you for asking! :) | 11:55 |
| belmoreira | janders np | 11:56 |
| janders | something I've been looking most recently that might be useful for the SIG is potentially introducing NVMe-aware cleaning to Ironic | 11:56 |
| janders | (think trim/discard/... functionality) | 11:56 |
| oneswig | janders: I've seen key rotation used for SATA SSDs, does that also apply here? | 11:57 |
| oneswig | The discard idea is good though! | 11:57 |
| janders | yeah some of the "secure" deletion options leverage manipulating crypto keys | 11:58 |
| janders | what's supported really varies but I hope we can find enough common ground | 11:58 |
| janders | how are things at your end oneswig? What are you guys up to these days? | 11:59 |
| oneswig | janders: too much to describe in our final minute :-( | 11:59 |
| oneswig | I think finally we are back to having rather too much fun. | 11:59 |
| janders | oneswig true! poor timing on my behalf. Next time! | 11:59 |
| oneswig | Until next time. I promise to come better prepared. | 12:00 |
| janders | have a good one all | 12:00 |
| oneswig | time to close | 12:00 |
| oneswig | #endmeeting | 12:00 |
| janders | and thanks again for sharing super interesting stuff belmoreira | 12:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 12:00 | |
| oneswig | thanks all | 12:00 |
| openstack | Meeting ended Wed Sep 23 12:00:38 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-09-23-11.03.html | 12:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-09-23-11.03.txt | 12:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-09-23-11.03.log.html | 12:00 |
| *** tosky_ has joined #openstack-meeting | 12:02 | |
| *** tosky is now known as Guest73219 | 12:03 | |
| *** tosky_ is now known as tosky | 12:03 | |
| *** yamamoto has joined #openstack-meeting | 12:03 | |
| *** Guest73219 has quit IRC | 12:05 | |
| *** oneswig has quit IRC | 12:12 | |
| *** yamamoto has quit IRC | 12:16 | |
| *** redrobot has joined #openstack-meeting | 12:23 | |
| *** priteau has joined #openstack-meeting | 12:35 | |
| *** eharney has joined #openstack-meeting | 12:37 | |
| *** baojg has quit IRC | 12:41 | |
| *** baojg has joined #openstack-meeting | 12:41 | |
| *** yamamoto has joined #openstack-meeting | 12:54 | |
| *** yamamoto has quit IRC | 12:55 | |
| *** yamamoto has joined #openstack-meeting | 12:55 | |
| *** masahito has joined #openstack-meeting | 13:03 | |
| *** yamamoto has quit IRC | 13:10 | |
| *** eharney has quit IRC | 13:21 | |
| *** andrebeltrami has joined #openstack-meeting | 13:23 | |
| *** tosky_ has joined #openstack-meeting | 13:26 | |
| *** tosky is now known as Guest13336 | 13:26 | |
| *** tosky_ is now known as tosky | 13:26 | |
| *** Guest13336 has quit IRC | 13:28 | |
| *** tosky_ has joined #openstack-meeting | 13:29 | |
| *** ociuhandu_ has joined #openstack-meeting | 13:31 | |
| *** tosky is now known as Guest61020 | 13:31 | |
| *** tosky_ is now known as tosky | 13:31 | |
| *** Guest61020 has quit IRC | 13:32 | |
| *** ociuhandu has quit IRC | 13:35 | |
| *** ZhuXiaoYu has joined #openstack-meeting | 13:40 | |
| *** rbudden has joined #openstack-meeting | 13:40 | |
| *** TrevorV has joined #openstack-meeting | 13:40 | |
| *** ociuhandu_ has quit IRC | 13:41 | |
| *** ociuhandu has joined #openstack-meeting | 13:42 | |
| *** masahito has quit IRC | 13:49 | |
| *** yamamoto has joined #openstack-meeting | 13:50 | |
| *** eharney has joined #openstack-meeting | 13:55 | |
| *** Liang__ has joined #openstack-meeting | 13:56 | |
| *** Liang__ is now known as LiangFang | 14:00 | |
| *** liuyulong has joined #openstack-meeting | 14:00 | |
| liuyulong | Hello | 14:00 |
| liuyulong | #startmeeting neutron_l3 | 14:00 |
| openstack | Meeting started Wed Sep 23 14:00:47 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:00 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:00 |
| slaweq | hi | 14:01 |
| *** yamamoto has quit IRC | 14:02 | |
| liuyulong | hi | 14:02 |
| liuyulong | No announcements from me today, so maybe we can directly goto the Bug section to cut the meeting time. | 14:03 |
| ralonsoh | hi | 14:04 |
| liuyulong | OK, no objection, : ) | 14:04 |
| liuyulong | #topic Bugs | 14:04 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:05 | |
| liuyulong | ralonsoh, hi | 14:05 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017254.html | 14:05 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017432.html | 14:05 |
| liuyulong | These are the bug lists from our deputy. | 14:05 |
| liuyulong | First one | 14:05 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1895950 | 14:06 |
| openstack | Launchpad bug 1895950 in neutron "keepalived can't perform failover if the l3 agent is down" [Medium,Won't fix] | 14:06 |
| slaweq | I don't understand why You marked it as won't fix | 14:06 |
| liuyulong | I replied to this last week, the L3 agent should be alive during the HA router state change. | 14:07 |
| slaweq | IMO if we will move bringing interfaces to be up to neutron-state-change-monitor process it should works | 14:07 |
| liuyulong | After the patch https://review.opendev.org/#/c/707406/ | 14:07 |
| slaweq | but that is regression introduced by this patch | 14:07 |
| slaweq | isn't it? | 14:07 |
| slaweq | small but IMHO still regression | 14:08 |
| *** thgcorrea has joined #openstack-meeting | 14:08 | |
| liuyulong | Because the running state-agent process can not do that work if you do not re-spawn it. | 14:09 |
| slaweq | isn't it respawned if You restart L3 agent? | 14:09 |
| ralonsoh | no | 14:10 |
| liuyulong | I'm not sure, but from my experience, the state change process will run as it is. | 14:10 |
| ralonsoh | if the keepalived-state-change process is running, is not rebooted | 14:11 |
| ralonsoh | but if reload_cfg if enabled, then we'll send SIGHUP | 14:11 |
| ralonsoh | (reload_cfg is false when restarting l3 agent) | 14:12 |
| ralonsoh | so no, we don't restart it | 14:12 |
| slaweq | ok, so maybe we can add bringing interfaces to be up/down to the state-change process and keep it in l3 agent for 1 cycle | 14:12 |
| liuyulong | It reloads the config options, not the python process. | 14:12 |
| slaweq | later remove it from the l3 agent | 14:12 |
| slaweq | or maybe 2 cycles | 14:12 |
| slaweq | and add e.g. release note about that | 14:12 |
| slaweq | idk | 14:12 |
| *** martial_ has quit IRC | 14:12 | |
| ralonsoh | one question: if the l3 agent is down, how this host will become master? | 14:13 |
| slaweq | keepalived can still be running | 14:13 |
| slaweq | and it can failover | 14:13 |
| slaweq | but l3 agent will not bring interfaces up on new master node | 14:13 |
| ralonsoh | yeah, that was my question | 14:13 |
| liuyulong | The DB state updating still needs L3 agent alive. | 14:13 |
| ralonsoh | ^ | 14:13 |
| slaweq | I know that | 14:13 |
| slaweq | but still IMO would be better to have working dataplane even in case when L3 agent is down for some reason | 14:14 |
| liuyulong | Actually L3 agent must run during HA router failover, it is designed by this. (not me, but it is) : ) | 14:14 |
| slaweq | liuyulong: before Your patch even? | 14:15 |
| liuyulong | No, I mean HA state change workflow has something related to L3 agent. It needs L3 agent to do some work. | 14:15 |
| liuyulong | Not the gateway, but something like RA, DB state, config state and so on. | 14:16 |
| liuyulong | But, it's fine to add the gateway UP action to the state-change process. | 14:17 |
| liuyulong | I'm fine with it. | 14:17 |
| slaweq | ok, lets keep this bug as won't fix for now | 14:17 |
| slaweq | and maybe check/update docs to be clear about that there | 14:18 |
| *** apetrich has quit IRC | 14:19 | |
| liuyulong | Sorry, bad connection | 14:21 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1894843 | 14:21 |
| openstack | Launchpad bug 1894843 in neutron "[dvr_snat] Router update deletes rfp interface from qrouter even when VM port is present on this host" [Medium,New] | 14:21 |
| liuyulong | I have no idea why set "dvr_snat" on every hypervisor? Should it be "dvr"? | 14:23 |
| ralonsoh | dvr_snat should be only on network controllers | 14:24 |
| slaweq | we are using dvr_snat e.g. in our gates | 14:24 |
| liuyulong | L3 agent in "dvr_snat" with mixed compute service does not work fine from my personal experiences. | 14:24 |
| slaweq | and that possible can cause some failures in dvr multinode jobs maybe | 14:24 |
| slaweq | (idk for sure but just guessing) | 14:24 |
| liuyulong | IMO, this should be documented well, users should not deploy their cloud like this. | 14:25 |
| liuyulong | IMO, there are no much agent mode check for "dvr_snat" during the router processing. | 14:26 |
| liuyulong | We have consensus that the "dvr_snat" is for those centralized network node (functions) which can not be distributed. | 14:27 |
| liuyulong | So, my advice for this bug/user is to change the config options. | 14:29 |
| ralonsoh | agree | 14:29 |
| liuyulong | The final cloud deployment should be in two scenario: | 14:30 |
| liuyulong | 1. their compute nodes have ability to external network (internet), so the compute node set the L3 agent mode to "dvr". | 14:30 |
| liuyulong | 2. compute node can not reach the Internet, set the agent mode to "dvr_no_external" | 14:31 |
| liuyulong | 3. centralized network nodes should be run dedicated physical hosts, and the L3 agent mode is "dvr_snat". | 14:32 |
| liuyulong | OK, no more bugs from me | 14:35 |
| *** b1airo has quit IRC | 14:35 | |
| liuyulong | OK, let's move on | 14:36 |
| liuyulong | #topic On demand agenda | 14:36 |
| *** openstack changes topic to "On demand agenda (Meeting topic: neutron_l3)" | 14:36 | |
| ralonsoh | nothing from me | 14:38 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1895972 | 14:38 |
| openstack | liuyulong: Error: Could not gather data from Launchpad for bug #1895972 (https://launchpad.net/bugs/1895972). The error has been logged | 14:38 |
| liuyulong | Another gap is filling... Congrats! | 14:39 |
| ralonsoh | this feature is ongoing but yes! | 14:39 |
| liuyulong | There are C works, so it is one example of fullstack development process for OVN feature. | 14:41 |
| liuyulong | Python works are not started. | 14:42 |
| liuyulong | #link https://review.opendev.org/#/c/738551/ | 14:42 |
| liuyulong | slaweq, hi, I've replied the comments. | 14:43 |
| liuyulong | I've tested it from my local devstack environment for a while. | 14:43 |
| ralonsoh | and what is happening with https://review.opendev.org/#/c/731446 | 14:43 |
| ralonsoh | ? | 14:43 |
| ralonsoh | superseded by yours, I think so | 14:44 |
| liuyulong | I cannot say I covered every cases, but those I noticed and experienced. | 14:44 |
| liuyulong | ralonsoh, yep, it has 2 closes bugs. | 14:44 |
| slaweq | liuyulong: ok, I will check that | 14:46 |
| liuyulong | But with some deep thinking, after these flows refactor or rediect (some works else), IMO the entire flow structure may have a chance to redesign in someday. | 14:47 |
| liuyulong | It could be a long story. Just forget it. : ) | 14:48 |
| ZhuXiaoYu | Oh, I wonder why https://review.opendev.org/#/c/731446 is not approved too | 14:48 |
| *** dklyle has joined #openstack-meeting | 14:48 | |
| ZhuXiaoYu | would you give an explanation? | 14:50 |
| ZhuXiaoYu | I will tell Li YaJie later | 14:50 |
| liuyulong | Please take look at the inline comments in gerrit, and the meeting LOG here. : ) | 14:51 |
| liuyulong | OK, no more talks from me now. | 14:51 |
| liuyulong | I will left 1 or 2 mins here. | 14:51 |
| ZhuXiaoYu | https://review.opendev.org/#/c/743661/ | 14:52 |
| ZhuXiaoYu | my patch for ecmp | 14:52 |
| ZhuXiaoYu | I really hope it can be 'merged' | 14:53 |
| liuyulong | It's feature freeze now, IMO, it should be moved to next dev cycle. | 14:55 |
| liuyulong | Wait... | 14:55 |
| ZhuXiaoYu | ..so when is the next dev cycle? | 14:57 |
| liuyulong | #link http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-09-15-14.00.log.html#l-13 | 14:57 |
| liuyulong | #link https://launchpad.net/neutron/+milestone/victoria-3 | 14:57 |
| liuyulong | If this was not in the V-3 list, it will not be merged for now. | 14:57 |
| liuyulong | Sorry, I cannot open the launchpad.net for now. | 14:58 |
| slaweq | ZhuXiaoYu: yes, we are in the RC-1 week now | 14:58 |
| slaweq | so we can merge this patch after rc-1 will be released and we will have stable/victoria branch created already | 14:58 |
| ZhuXiaoYu | got it, really thx for tell me that, it's helpful | 14:59 |
| liuyulong | I will start another round review of the spec https://review.opendev.org/#/c/729532 this week. | 14:59 |
| liuyulong | Time is up. | 15:00 |
| liuyulong | Thank you guys. | 15:00 |
| liuyulong | Bye | 15:00 |
| ZhuXiaoYu | Bye | 15:00 |
| slaweq | thx | 15:00 |
| liuyulong | #endmeeting | 15:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:00 | |
| openstack | Meeting ended Wed Sep 23 15:00:26 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-09-23-14.00.html | 15:00 |
| slaweq | bye | 15:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-09-23-14.00.txt | 15:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-09-23-14.00.log.html | 15:00 |
| *** ZhuXiaoYu has quit IRC | 15:02 | |
| *** LiangFang has quit IRC | 15:04 | |
| *** liuyulong has quit IRC | 15:20 | |
| *** baojg has quit IRC | 15:35 | |
| *** baojg has joined #openstack-meeting | 15:36 | |
| *** baojg has quit IRC | 15:37 | |
| *** baojg has joined #openstack-meeting | 15:37 | |
| *** yasufum has joined #openstack-meeting | 15:37 | |
| *** eharney has quit IRC | 15:43 | |
| *** lpetrut has quit IRC | 15:45 | |
| *** gyee has joined #openstack-meeting | 15:45 | |
| *** Trevor_V has joined #openstack-meeting | 15:48 | |
| *** TrevorV has quit IRC | 15:48 | |
| *** ociuhandu has quit IRC | 15:49 | |
| *** Trevor_V has quit IRC | 15:56 | |
| *** belmoreira has quit IRC | 15:57 | |
| *** tosky has quit IRC | 15:59 | |
| *** belmoreira has joined #openstack-meeting | 15:59 | |
| *** belmoreira has quit IRC | 16:00 | |
| *** ociuhandu has joined #openstack-meeting | 16:01 | |
| *** ociuhandu has quit IRC | 16:07 | |
| *** baojg has quit IRC | 16:12 | |
| *** baojg has joined #openstack-meeting | 16:12 | |
| *** Lucas_Gray has quit IRC | 16:13 | |
| *** apetrich has joined #openstack-meeting | 16:21 | |
| *** ralonsoh has quit IRC | 16:28 | |
| *** haleyb has joined #openstack-meeting | 16:36 | |
| *** eharney has joined #openstack-meeting | 16:43 | |
| *** yasufum has quit IRC | 16:43 | |
| *** etp has quit IRC | 16:49 | |
| *** etp has joined #openstack-meeting | 16:52 | |
| *** etp has quit IRC | 17:00 | |
| *** priteau has quit IRC | 17:02 | |
| *** etp has joined #openstack-meeting | 17:02 | |
| *** hemna has quit IRC | 17:36 | |
| *** zaneb has quit IRC | 17:37 | |
| *** zaneb has joined #openstack-meeting | 17:38 | |
| *** hemna has joined #openstack-meeting | 17:47 | |
| *** e0ne has quit IRC | 17:56 | |
| *** yamamoto has joined #openstack-meeting | 18:00 | |
| *** yamamoto has quit IRC | 18:04 | |
| *** eharney has quit IRC | 18:22 | |
| *** hemna has quit IRC | 18:32 | |
| *** baojg has quit IRC | 18:36 | |
| *** baojg has joined #openstack-meeting | 18:37 | |
| *** TrevorV has joined #openstack-meeting | 18:38 | |
| *** hemna has joined #openstack-meeting | 18:42 | |
| *** baojg has quit IRC | 18:57 | |
| *** baojg has joined #openstack-meeting | 18:58 | |
| *** tosky has joined #openstack-meeting | 19:15 | |
| *** slaweq has quit IRC | 19:25 | |
| *** adrianc_ has quit IRC | 19:35 | |
| *** adrianc has joined #openstack-meeting | 19:35 | |
| *** eharney has joined #openstack-meeting | 19:50 | |
| *** eharney has quit IRC | 19:55 | |
| *** yamamoto has joined #openstack-meeting | 20:01 | |
| *** yamamoto has quit IRC | 20:05 | |
| *** slaweq has joined #openstack-meeting | 20:14 | |
| *** slaweq has quit IRC | 20:16 | |
| *** slaweq has joined #openstack-meeting | 20:17 | |
| *** apetrich has quit IRC | 20:32 | |
| *** vishalmanchanda has quit IRC | 20:38 | |
| *** hemna has quit IRC | 20:43 | |
| *** baojg has quit IRC | 20:44 | |
| *** viks____ has quit IRC | 20:49 | |
| *** hillpd has quit IRC | 20:49 | |
| *** phrobb has quit IRC | 20:49 | |
| *** vkmc has quit IRC | 20:49 | |
| *** luyao has quit IRC | 20:49 | |
| *** whoami-rajat__ has quit IRC | 20:50 | |
| *** sri_ has quit IRC | 20:50 | |
| *** gagehugo has quit IRC | 20:50 | |
| *** rpittau|afk has quit IRC | 20:50 | |
| *** xyang has quit IRC | 20:50 | |
| *** gmann has quit IRC | 20:50 | |
| *** yonglihe has quit IRC | 20:50 | |
| *** dougwig has quit IRC | 20:50 | |
| *** thgcorrea has quit IRC | 20:50 | |
| *** andrebeltrami has quit IRC | 20:50 | |
| *** knikolla has quit IRC | 20:50 | |
| *** whoami-rajat__ has joined #openstack-meeting | 20:51 | |
| *** vkmc has joined #openstack-meeting | 20:51 | |
| *** hillpd has joined #openstack-meeting | 20:51 | |
| *** phrobb has joined #openstack-meeting | 20:51 | |
| *** nicolasbock has quit IRC | 20:51 | |
| *** arne_wiebalck has quit IRC | 20:51 | |
| *** jamespage has quit IRC | 20:51 | |
| *** PrinzElvis has quit IRC | 20:51 | |
| *** erbarr has quit IRC | 20:51 | |
| *** andrebeltrami has joined #openstack-meeting | 20:52 | |
| *** sri_ has joined #openstack-meeting | 20:52 | |
| *** bcm has quit IRC | 20:52 | |
| *** clayg has quit IRC | 20:52 | |
| *** luyao has joined #openstack-meeting | 20:52 | |
| *** knikolla has joined #openstack-meeting | 20:52 | |
| *** nicolasbock has joined #openstack-meeting | 20:52 | |
| *** hemna has joined #openstack-meeting | 20:52 | |
| *** bcm has joined #openstack-meeting | 20:52 | |
| *** dougwig has joined #openstack-meeting | 20:52 | |
| *** PrinzElvis has joined #openstack-meeting | 20:52 | |
| *** thgcorrea has joined #openstack-meeting | 20:52 | |
| *** xyang has joined #openstack-meeting | 20:52 | |
| *** arne_wiebalck has joined #openstack-meeting | 20:53 | |
| *** erbarr has joined #openstack-meeting | 20:53 | |
| *** clayg has joined #openstack-meeting | 20:53 | |
| *** gagehugo has joined #openstack-meeting | 20:53 | |
| *** rpittau|afk has joined #openstack-meeting | 20:53 | |
| *** yonglihe has joined #openstack-meeting | 20:53 | |
| *** gmann has joined #openstack-meeting | 20:53 | |
| *** jamespage has joined #openstack-meeting | 20:53 | |
| *** bbowen has quit IRC | 20:57 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Sep 23 21:00:25 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| kota_ | hello | 21:00 |
| *** patchbot has joined #openstack-meeting | 21:01 | |
| timburke | maybe it's just you and me | 21:02 |
| kota_ | oh yeah | 21:02 |
| timburke | agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| *** tdasilva has joined #openstack-meeting | 21:02 | |
| timburke | main thing i wanted to mention was some follow up on the state of our gate | 21:03 |
| kota_ | okay | 21:03 |
| clayg | o/ | 21:03 |
| timburke | #topic busted gates | 21:03 |
| *** openstack changes topic to "busted gates (Meeting topic: swift)" | 21:03 | |
| kota_ | oh, it looks bunch of branches were broken... | 21:04 |
| timburke | i *think* swift's ussuri gate is fixed now -- at any rate, i stopped seeing emails about docs failure | 21:04 |
| timburke | assuming it's moving, i'll try to get some stable releases out for ussuri and train this week | 21:05 |
| timburke | swift client's gate is better now! the fix landed after the deadline to branch for victoria, though, so i might need to reach out to the stable team to sort out how best to fix that one | 21:06 |
| timburke | (the fix involved some requirements changes, so i worry a little that a simple backport may not be great) | 21:06 |
| kota_ | i see | 21:06 |
| timburke | i discovered pyeclib's gate was broken after seeing p 744623 | 21:07 |
| patchbot | https://review.opendev.org/#/c/744623/ - pyeclib - [goal] Migrate testing to ubuntu focal (ABANDONED) - 4 patch sets | 21:07 |
| timburke | p 753472 fixed it, but disabled the two jobs we had to test against tip-of-master libec | 21:08 |
| patchbot | https://review.opendev.org/#/c/753472/ - pyeclib - Fix gate (MERGED) - 1 patch set | 21:08 |
| clayg | focal is gunna be so great - i'm sure I'll try upgrading to it at some point | 21:08 |
| kota_ | clayg! | 21:09 |
| clayg | kota_: i snuck in 😁 | 21:09 |
| timburke | at some point we should dig into how those fail, but they're both such low-volume repos that i'm fairly certain they still work well together | 21:10 |
| timburke | while i was looking at pyeclib, i also pushed in p 753421 to test against py38 on focal and py36 on centos8 | 21:10 |
| patchbot | https://review.opendev.org/#/c/753421/ - pyeclib - Update gate jobs (MERGED) - 4 patch sets | 21:10 |
| kota_ | libec-pyeclib-unit said `/bin/bash: line 17: tox: command not found` :( | 21:11 |
| kota_ | at p 753472 | 21:11 |
| patchbot | https://review.opendev.org/#/c/753472/ - pyeclib - Fix gate (MERGED) - 1 patch set | 21:11 |
| kota_ | no p 744623 | 21:11 |
| patchbot | https://review.opendev.org/#/c/744623/ - pyeclib - [goal] Migrate testing to ubuntu focal (ABANDONED) - 4 patch sets | 21:11 |
| timburke | i love how snappy pyeclib's jobs are -- at 2-4 mins per job, i feel like we can add more target platforms all day long! | 21:12 |
| kota_ | sounds good | 21:12 |
| timburke | but all of this reminded me that i should check on the state of libec's gate; will report back next week | 21:14 |
| timburke | that's all i've got for the gate stuff; any questions or comments? | 21:14 |
| kota_ | nothing so far. thanks for your effort to keep the gate to work. | 21:15 |
| *** eharney has joined #openstack-meeting | 21:16 | |
| clayg | timburke: 👏 | 21:16 |
| timburke | all right, i've just got one other topic on my mind lately | 21:16 |
| timburke | #topic hung proxy servers | 21:16 |
| *** openstack changes topic to "hung proxy servers (Meeting topic: swift)" | 21:16 | |
| timburke | there have been two distinct issues that came up recently are somewhat related | 21:17 |
| timburke | one is https://bugs.launchpad.net/swift/+bug/1895739 | 21:18 |
| openstack | Launchpad bug 1895739 in OpenStack Object Storage (swift) "Proxy server sometimes deadlocks while logging client disconnect" [Undecided,In progress] | 21:18 |
| *** mlavalle has joined #openstack-meeting | 21:20 | |
| timburke | the nitty-gritty is in the bug, but the summary is that while we're down in logging, garbage collection may cause us to try to grab the same (non-reentrant) lock twice in the same (green)thread | 21:20 |
| timburke | the other is https://github.com/eventlet/eventlet/pull/498 | 21:21 |
| timburke | where eventlet sees that there's a fd read to read, but then doesn't wake anyone up to read it | 21:21 |
| timburke | good news is that the second one is already merged (and tagged!) following https://github.com/eventlet/eventlet/pull/645 -- thanks for cleaning it up clayg! | 21:23 |
| clayg | tight poll loop keeps asking for the same fd, and it says it's ready - but it just keeps polling | 21:23 |
| timburke | the first one has a patch at p 752593 | 21:24 |
| patchbot | https://review.opendev.org/#/c/752593/ - swift - Replace threading._active_limbo_lock with a re-ent... - 3 patch sets | 21:24 |
| timburke | i think both of these issues can affect other services, it's just acutely bad on proxies | 21:24 |
| timburke | as much as anything, i just wanted to raise awareness in case anyone else sees similar issues, and maybe see if i could get someone to look at the swift patch ;-) | 21:25 |
| clayg | does lp bug #1895739 only effect py3? | 21:28 |
| openstack | Launchpad bug 1895739 in OpenStack Object Storage (swift) "Proxy server sometimes deadlocks while logging client disconnect" [Undecided,In progress] https://launchpad.net/bugs/1895739 | 21:28 |
| timburke | i've only *observed it* on py3 -- and i'm not sure why :-( | 21:28 |
| timburke | looking at py2's code, it seems like it *could* happen there, too... but again, i've not actually seen it | 21:28 |
| timburke | maybe there was some change in GC algo? | 21:29 |
| clayg | what kind of lock *is* _active_limbo_lock in cpython? does eventlet patch it by default? | 21:29 |
| timburke | i still haven't found a good way to reliably reproduce the problem, either :-( | 21:29 |
| clayg | 😢 | 21:29 |
| timburke | clayg, so in cpython it's a pretty low-level lock -- uses https://docs.python.org/2/library/thread.html#thread.allocate_lock as i recall | 21:31 |
| timburke | eventlet *does* patch it; it gets replaces with a Semaphore | 21:32 |
| clayg | neato! | 21:32 |
| timburke | which seems like a reasonable replacement given the semantics | 21:33 |
| timburke | i tried to go over some of the weirdness that leads to this in the bug -- it's not really clear to me whether we're to blame, eventlet's to blame, or cpython's to blame :-/ | 21:35 |
| timburke | swapping out for our own reentrant lock seems like the most-reasonable approach, though, especially since it's already getting patched | 21:35 |
| timburke | clayg, since you've already put some effort into thining about eventlet and our PipeMutex, mind takinga look this week? | 21:36 |
| clayg | i'm sure it's fine - but without a repro it's hard to say exactly | 21:37 |
| *** baojg has joined #openstack-meeting | 21:37 | |
| timburke | all right, that's all i've got planned | 21:38 |
| timburke | #topic open discussion | 21:38 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:38 | |
| timburke | what else should we talk about this week? | 21:38 |
| clayg | are we still stalled out on pyeclib? | 21:38 |
| timburke | pyeclib's good now, afaik -- maybe you're thinking of p 738959 though? | 21:39 |
| patchbot | https://review.opendev.org/#/c/738959/ - liberasurecode - Be willing to write fragments with legacy crc - 2 patch sets | 21:39 |
| timburke | i still haven't circled back on it -- i'm coming around to wanting to at least treat set-to-the-empty-string the same as unset, but beyond that i'm not sure | 21:41 |
| timburke | i think my main question is: which falsey values should we look for? | 21:44 |
| timburke | kota_, clayg any thoughts there? keeping in mind that the check'll have to be written in C | 21:46 |
| clayg | i like 0 and 1 for true and false in C | 21:47 |
| kota_ | clayg: agree. plus empty value seems False. | 21:47 |
| clayg | anyone have any idea why making a request that uses acl's results in the env getting copied? p 752770 | 21:48 |
| patchbot | https://review.opendev.org/#/c/752770/ - swift - Log error processing manifest as ServerError - 1 patch set | 21:48 |
| timburke | ok, i'll code that up this week | 21:48 |
| clayg | we end up loosing the storage policy index from the req.environ as well | 21:48 |
| *** rfolco|ruck has quit IRC | 21:55 | |
| *** yonglihe has quit IRC | 21:56 | |
| *** yonglihe has joined #openstack-meeting | 21:56 | |
| timburke | i have no idea. sorry. went looking | 21:57 |
| timburke | i'll see about digging into it more on the patch, though | 21:57 |
| timburke | all right, i think that'll do it | 21:58 |
| timburke | thank you all for coming, and thank you for working on swift! | 21:58 |
| timburke | #endmeeting | 21:58 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:58 | |
| openstack | Meeting ended Wed Sep 23 21:58:27 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:58 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-09-23-21.00.html | 21:58 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-09-23-21.00.txt | 21:58 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-09-23-21.00.log.html | 21:58 |
| *** patchbot has left #openstack-meeting | 21:58 | |
| *** yamamoto has joined #openstack-meeting | 22:01 | |
| *** phrobb has quit IRC | 22:01 | |
| *** phrobb has joined #openstack-meeting | 22:03 | |
| *** rbudden has quit IRC | 22:04 | |
| *** rbudden has joined #openstack-meeting | 22:05 | |
| *** yamamoto has quit IRC | 22:06 | |
| *** yamamoto has joined #openstack-meeting | 22:12 | |
| *** slaweq has quit IRC | 22:13 | |
| *** bbowen has joined #openstack-meeting | 22:13 | |
| *** eharney has quit IRC | 22:19 | |
| *** TrevorV has quit IRC | 22:22 | |
| *** rcernin has joined #openstack-meeting | 22:46 | |
| *** PrinzElvis has quit IRC | 22:50 | |
| *** knikolla has quit IRC | 22:50 | |
| *** sri_ has quit IRC | 22:50 | |
| *** thgcorrea has quit IRC | 22:50 | |
| *** whoami-rajat__ has quit IRC | 22:50 | |
| *** walshh_ has quit IRC | 22:50 | |
| *** andrebeltrami has quit IRC | 22:50 | |
| *** dougwig has quit IRC | 22:50 | |
| *** bcm has quit IRC | 22:50 | |
| *** phrobb has quit IRC | 22:50 | |
| *** erbarr has quit IRC | 22:50 | |
| *** jungleboyj has quit IRC | 22:50 | |
| *** yonglihe has quit IRC | 22:50 | |
| *** arne_wiebalck has quit IRC | 22:50 | |
| *** johnsom has quit IRC | 22:50 | |
| *** xyang has quit IRC | 22:50 | |
| *** clayg has quit IRC | 22:50 | |
| *** manpreet has quit IRC | 22:50 | |
| *** masayukig has quit IRC | 22:50 | |
| *** aprice has quit IRC | 22:50 | |
| *** mnaser has quit IRC | 22:50 | |
| *** nicolasbock has quit IRC | 22:51 | |
| *** vkmc has quit IRC | 22:51 | |
| *** whoami-rajat__ has joined #openstack-meeting | 22:51 | |
| *** erbarr has joined #openstack-meeting | 22:51 | |
| *** yonglihe has joined #openstack-meeting | 22:51 | |
| *** bcm has joined #openstack-meeting | 22:51 | |
| *** arne_wiebalck has joined #openstack-meeting | 22:51 | |
| *** knikolla has joined #openstack-meeting | 22:51 | |
| *** xyang has joined #openstack-meeting | 22:51 | |
| *** walshh_ has joined #openstack-meeting | 22:51 | |
| *** sri_ has joined #openstack-meeting | 22:51 | |
| *** andrebeltrami has joined #openstack-meeting | 22:51 | |
| *** PrinzElvis has joined #openstack-meeting | 22:51 | |
| *** dougwig has joined #openstack-meeting | 22:51 | |
| *** jungleboyj has joined #openstack-meeting | 22:51 | |
| *** clayg has joined #openstack-meeting | 22:52 | |
| *** seongsoocho has quit IRC | 22:52 | |
| *** aprice has joined #openstack-meeting | 22:52 | |
| *** seongsoocho has joined #openstack-meeting | 22:52 | |
| *** masayukig has joined #openstack-meeting | 22:52 | |
| *** phrobb has joined #openstack-meeting | 22:53 | |
| *** mnaser has joined #openstack-meeting | 22:54 | |
| *** rpittau|afk has quit IRC | 22:54 | |
| *** nicolasbock has joined #openstack-meeting | 22:54 | |
| *** thgcorrea has joined #openstack-meeting | 22:55 | |
| *** johnsom has joined #openstack-meeting | 22:55 | |
| *** manpreet has joined #openstack-meeting | 22:56 | |
| *** rpittau|afk has joined #openstack-meeting | 22:56 | |
| *** vkmc has joined #openstack-meeting | 22:56 | |
| *** whoami-rajat__ has quit IRC | 22:57 | |
| *** bcm has quit IRC | 22:57 | |
| *** whoami-rajat__ has joined #openstack-meeting | 23:00 | |
| *** bcm has joined #openstack-meeting | 23:01 | |
| *** moguimar has quit IRC | 23:03 | |
| *** rcernin has quit IRC | 23:34 | |
| *** rcernin has joined #openstack-meeting | 23:35 | |
| *** etp has quit IRC | 23:35 | |
| *** etp has joined #openstack-meeting | 23:36 | |
| *** rbudden has quit IRC | 23:39 | |
| *** tosky has quit IRC | 23:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!