| *** d34dh0r53 has quit IRC | 00:16 | |
| *** d34dh0r53 has joined #openstack-meeting | 00:25 | |
| *** d34dh0r53 has quit IRC | 00:25 | |
| *** d34dh0r53 has joined #openstack-meeting | 00:28 | |
| *** d34dh0r53 has quit IRC | 00:30 | |
| *** martial has quit IRC | 00:31 | |
| *** maohongbo has quit IRC | 00:34 | |
| *** d34dh0r53 has joined #openstack-meeting | 00:35 | |
| *** d34dh0r53 has quit IRC | 00:36 | |
| *** d34dh0r53 has joined #openstack-meeting | 00:39 | |
| *** d34dh0r53 has quit IRC | 00:39 | |
| *** d34dh0r53 has joined #openstack-meeting | 00:43 | |
| *** d34dh0r53 has quit IRC | 00:44 | |
| *** d34dh0r53 has joined #openstack-meeting | 00:52 | |
| *** d34dh0r53 has quit IRC | 00:53 | |
| *** ricolin_ has joined #openstack-meeting | 00:57 | |
| *** ricolin_ has quit IRC | 00:58 | |
| *** d34dh0r53 has joined #openstack-meeting | 00:59 | |
| *** d34dh0r53 has quit IRC | 00:59 | |
| *** d34dh0r53 has joined #openstack-meeting | 01:01 | |
| *** d34dh0r53 has quit IRC | 01:04 | |
| *** d34dh0r53 has joined #openstack-meeting | 01:14 | |
| *** d34dh0r53 has quit IRC | 01:14 | |
| *** liuyulong has quit IRC | 01:23 | |
| *** d34dh0r53 has joined #openstack-meeting | 01:28 | |
| *** rfolcOUT has quit IRC | 01:28 | |
| *** d34dh0r53 has quit IRC | 01:28 | |
| *** d34dh0r53 has joined #openstack-meeting | 01:33 | |
| *** d34dh0r53 has quit IRC | 01:34 | |
| *** d34dh0r53 has joined #openstack-meeting | 01:37 | |
| *** d34dh0r53 has quit IRC | 01:38 | |
| *** d34dh0r53 has joined #openstack-meeting | 01:41 | |
| *** d34dh0r53 has quit IRC | 01:42 | |
| *** dtrainor has quit IRC | 01:46 | |
| *** maohongbo has joined #openstack-meeting | 01:46 | |
| *** d34dh0r53 has joined #openstack-meeting | 01:52 | |
| *** d34dh0r53 has quit IRC | 01:52 | |
| *** d34dh0r53 has joined #openstack-meeting | 01:56 | |
| *** d34dh0r53 has quit IRC | 01:57 | |
| *** d34dh0r53 has joined #openstack-meeting | 02:02 | |
| *** d34dh0r53 has quit IRC | 02:03 | |
| *** d34dh0r53 has joined #openstack-meeting | 02:05 | |
| *** d34dh0r53 has quit IRC | 02:05 | |
| *** d34dh0r53 has joined #openstack-meeting | 02:09 | |
| *** d34dh0r53 has quit IRC | 02:10 | |
| *** d34dh0r53 has joined #openstack-meeting | 02:19 | |
| *** d34dh0r53 has quit IRC | 02:19 | |
| *** Lucas_Gray has quit IRC | 02:21 | |
| *** d34dh0r53 has joined #openstack-meeting | 02:21 | |
| *** d34dh0r53 has quit IRC | 02:22 | |
| *** d34dh0r53 has joined #openstack-meeting | 02:38 | |
| *** d34dh0r53 has quit IRC | 02:39 | |
| *** mmethot has quit IRC | 02:43 | |
| *** d34dh0r53 has joined #openstack-meeting | 02:45 | |
| *** d34dh0r53 has quit IRC | 02:45 | |
| *** maohongbo has quit IRC | 02:45 | |
| *** maohongbo has joined #openstack-meeting | 02:47 | |
| *** rfolcOUT has joined #openstack-meeting | 02:48 | |
| *** maohongbo1 has joined #openstack-meeting | 03:04 | |
| *** maohongbo has quit IRC | 03:05 | |
| *** maohongbo1 is now known as maohongbo | 03:05 | |
| *** d34dh0r53 has joined #openstack-meeting | 03:09 | |
| *** d34dh0r53 has quit IRC | 03:09 | |
| *** ykatabam has quit IRC | 03:14 | |
| *** d34dh0r53 has joined #openstack-meeting | 03:19 | |
| *** d34dh0r53 has quit IRC | 03:19 | |
| *** maohongbo has quit IRC | 03:22 | |
| *** maohongbo has joined #openstack-meeting | 03:23 | |
| *** d34dh0r53 has joined #openstack-meeting | 03:28 | |
| *** d34dh0r53 has quit IRC | 03:28 | |
| *** ykatabam has joined #openstack-meeting | 03:31 | |
| *** psachin has joined #openstack-meeting | 03:33 | |
| *** mmethot has joined #openstack-meeting | 03:34 | |
| *** d34dh0r53 has joined #openstack-meeting | 03:41 | |
| *** d34dh0r53 has quit IRC | 03:41 | |
| *** d34dh0r53 has joined #openstack-meeting | 03:48 | |
| *** maohongbo1 has joined #openstack-meeting | 04:03 | |
| *** maohongbo has quit IRC | 04:04 | |
| *** maohongbo1 is now known as maohongbo | 04:04 | |
| *** d34dh0r53 has quit IRC | 04:40 | |
| *** d34dh0r53 has joined #openstack-meeting | 04:43 | |
| *** d34dh0r53 has quit IRC | 04:44 | |
| *** d34dh0r53 has joined #openstack-meeting | 04:45 | |
| *** d34dh0r53 has quit IRC | 04:45 | |
| *** d34dh0r53 has joined #openstack-meeting | 04:47 | |
| *** d34dh0r53 has quit IRC | 04:48 | |
| *** whoami-rajat__ has joined #openstack-meeting | 04:52 | |
| *** d34dh0r53 has joined #openstack-meeting | 04:53 | |
| *** d34dh0r53 has quit IRC | 04:53 | |
| *** d34dh0r53 has joined #openstack-meeting | 04:54 | |
| *** d34dh0r53 has quit IRC | 04:56 | |
| *** gyee has quit IRC | 04:57 | |
| *** d34dh0r53 has joined #openstack-meeting | 05:01 | |
| *** links has joined #openstack-meeting | 05:15 | |
| *** tetsuro has joined #openstack-meeting | 05:41 | |
| *** yaawang has quit IRC | 06:12 | |
| *** yaawang has joined #openstack-meeting | 06:12 | |
| *** hyunsikyang__ has joined #openstack-meeting | 06:22 | |
| *** hyunsikyang has quit IRC | 06:26 | |
| *** jawad_axd has joined #openstack-meeting | 06:49 | |
| *** jawad_ax_ has joined #openstack-meeting | 06:52 | |
| *** jawad_axd has quit IRC | 06:54 | |
| *** pcaruana has joined #openstack-meeting | 07:38 | |
| *** simon-AS559 has joined #openstack-meeting | 07:38 | |
| *** ociuhandu has joined #openstack-meeting | 07:46 | |
| *** ociuhandu has quit IRC | 07:58 | |
| *** e0ne has joined #openstack-meeting | 07:59 | |
| *** tesseract has joined #openstack-meeting | 08:06 | |
| *** e0ne has quit IRC | 08:08 | |
| *** e0ne has joined #openstack-meeting | 08:09 | |
| *** mahatic has quit IRC | 08:09 | |
| *** e0ne has quit IRC | 08:23 | |
| *** simon-AS5591 has joined #openstack-meeting | 08:25 | |
| *** simon-AS559 has quit IRC | 08:28 | |
| *** apetrich has joined #openstack-meeting | 08:31 | |
| *** e0ne has joined #openstack-meeting | 08:33 | |
| *** rsimai has joined #openstack-meeting | 08:40 | |
| *** maohongbo has quit IRC | 08:42 | |
| *** dmacpher has joined #openstack-meeting | 08:43 | |
| *** maohongbo has joined #openstack-meeting | 08:43 | |
| *** e0ne has quit IRC | 08:46 | |
| *** brinzhang_ has quit IRC | 08:46 | |
| *** dmacpher_ has joined #openstack-meeting | 08:47 | |
| *** dmacpher has quit IRC | 08:51 | |
| *** ralonsoh has joined #openstack-meeting | 08:52 | |
| *** jraju__ has joined #openstack-meeting | 09:03 | |
| *** links has quit IRC | 09:04 | |
| *** diablo_rojo has joined #openstack-meeting | 09:07 | |
| *** simon-AS559 has joined #openstack-meeting | 09:08 | |
| *** simon-AS5591 has quit IRC | 09:11 | |
| *** brinzhang has joined #openstack-meeting | 09:12 | |
| *** brinzhang_ has joined #openstack-meeting | 09:14 | |
| *** brinzhang_ has quit IRC | 09:16 | |
| *** brinzhang_ has joined #openstack-meeting | 09:16 | |
| *** brinzhang has quit IRC | 09:18 | |
| *** rpittau|afk is now known as rpittau | 09:25 | |
| *** brinzhang has joined #openstack-meeting | 09:31 | |
| *** brinzhang_ has quit IRC | 09:34 | |
| *** brinzhang has quit IRC | 09:36 | |
| *** maohongbo1 has joined #openstack-meeting | 09:38 | |
| *** maohongbo has quit IRC | 09:40 | |
| *** maohongbo1 is now known as maohongbo | 09:40 | |
| *** e0ne has joined #openstack-meeting | 09:50 | |
| *** simon-AS559 has quit IRC | 10:01 | |
| *** ociuhandu has joined #openstack-meeting | 10:02 | |
| *** simon-AS559 has joined #openstack-meeting | 10:05 | |
| *** ociuhandu has quit IRC | 10:19 | |
| *** links has joined #openstack-meeting | 10:22 | |
| *** jraju__ has quit IRC | 10:23 | |
| *** geguileo has joined #openstack-meeting | 10:28 | |
| *** e0ne has quit IRC | 10:29 | |
| *** ykatabam has quit IRC | 10:44 | |
| *** dmacpher__ has joined #openstack-meeting | 10:46 | |
| *** dmacpher_ has quit IRC | 10:48 | |
| *** ykatabam has joined #openstack-meeting | 10:49 | |
| *** e0ne has joined #openstack-meeting | 10:57 | |
| *** ociuhandu has joined #openstack-meeting | 10:59 | |
| *** ociuhandu has quit IRC | 11:18 | |
| *** ociuhandu has joined #openstack-meeting | 11:29 | |
| *** ociuhandu has quit IRC | 11:34 | |
| *** vishakha has quit IRC | 11:35 | |
| *** diablo_rojo has quit IRC | 11:38 | |
| *** simon-AS559 has quit IRC | 11:45 | |
| *** simon-AS559 has joined #openstack-meeting | 11:46 | |
| *** rfolcOUT is now known as rfolco | 12:07 | |
| *** dviroel has joined #openstack-meeting | 12:07 | |
| *** jawad_ax_ has quit IRC | 12:38 | |
| *** whoami-rajat__ has quit IRC | 12:45 | |
| *** liuyulong has joined #openstack-meeting | 12:52 | |
| *** enriquetaso has joined #openstack-meeting | 12:54 | |
| *** jawad_axd has joined #openstack-meeting | 12:55 | |
| *** liuyulong has joined #openstack-meeting | 12:55 | |
| *** liuyulong has quit IRC | 12:56 | |
| *** anastzhyr has joined #openstack-meeting | 12:56 | |
| *** liuyulong has joined #openstack-meeting | 12:58 | |
| *** jawad_ax_ has joined #openstack-meeting | 12:59 | |
| *** liuyulong has quit IRC | 12:59 | |
| *** jawad_axd has quit IRC | 12:59 | |
| *** jawad_ax_ has quit IRC | 13:03 | |
| *** jawad_axd has joined #openstack-meeting | 13:04 | |
| *** liuyulong has joined #openstack-meeting | 13:08 | |
| *** liuyulong_ has joined #openstack-meeting | 13:08 | |
| *** jawad_axd has quit IRC | 13:09 | |
| *** ociuhandu has joined #openstack-meeting | 13:09 | |
| *** jawad_axd has joined #openstack-meeting | 13:09 | |
| *** jawad_ax_ has joined #openstack-meeting | 13:12 | |
| *** jawad_axd has quit IRC | 13:14 | |
| *** jawad_axd has joined #openstack-meeting | 13:16 | |
| *** jawad_ax_ has quit IRC | 13:16 | |
| *** ociuhandu has quit IRC | 13:17 | |
| *** jawad_axd has quit IRC | 13:21 | |
| *** whoami-rajat__ has joined #openstack-meeting | 13:27 | |
| *** ociuhandu has joined #openstack-meeting | 13:38 | |
| *** ociuhandu has quit IRC | 13:42 | |
| *** jawad_axd has joined #openstack-meeting | 13:45 | |
| *** dmacpher__ has quit IRC | 13:45 | |
| *** dmacpher__ has joined #openstack-meeting | 13:46 | |
| *** jawad_axd has quit IRC | 13:49 | |
| *** Liang__ has joined #openstack-meeting | 13:53 | |
| *** jawad_axd has joined #openstack-meeting | 13:56 | |
| *** Liang__ is now known as LiangFang | 13:58 | |
| *** jawad_axd has quit IRC | 14:00 | |
| liuyulong | 1 | 14:00 |
|---|---|---|
| liuyulong_ | 1 | 14:00 |
| liuyulong | #startmeeting neutron_l3 | 14:01 |
| openstack | Meeting started Wed Jan 8 14:01:12 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:01 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:01 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:01 |
| liuyulong | #chair liuyulong_ | 14:01 |
| openstack | Current chairs: liuyulong liuyulong_ | 14:01 |
| liuyulong | Happy new year everyone! | 14:02 |
| *** thgcorrea has joined #openstack-meeting | 14:02 | |
| *** andrebeltrami has joined #openstack-meeting | 14:02 | |
| *** sfernand has joined #openstack-meeting | 14:02 | |
| liuyulong | #topic Announcements | 14:03 |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 14:03 | |
| liuyulong | #link https://launchpad.net/neutron/+milestone/ussuri-2 | 14:03 |
| liuyulong | Expected: 2020-02-12 | 14:04 |
| liuyulong | There will be about 10 days holidays for Chinese New Year this month. | 14:05 |
| liuyulong | Someone may not online, so time is running out... | 14:06 |
| haleyb | hi | 14:07 |
| liuyulong | hi | 14:07 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1858419 | 14:08 |
| openstack | Launchpad bug 1858419 in neutron "Docs needed for tunables at large scale" [Undecided,Confirmed] | 14:08 |
| liuyulong | Slawek asked me something in mail about this large scale cloud. | 14:08 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1858419/comments/1 | 14:08 |
| liuyulong | Allow me to say something here | 14:09 |
| liuyulong | This could be a really long story. | 14:09 |
| liuyulong | Config option tunning may have a lot choices. | 14:10 |
| liuyulong | But neutron itself still have some architecture defect, which may not be resolved by configuration. | 14:10 |
| slaweq | hi | 14:10 |
| slaweq | sorry for being late | 14:10 |
| liuyulong | As you may see in the comment #1, we did some local works for neutron itself. | 14:11 |
| slaweq | liuyulong: I know that we can't solve everything by config options | 14:11 |
| liuyulong | (Some of them was talked during Shanghai PTG.) | 14:11 |
| slaweq | but it's rather more about identyfing options which are crucial for large scale and to add some note for some options that e.g. "setting this to high/load value may have impact on large scale because it will make huge load on rabbitmq" (it's just an example for non existing option now :)) | 14:12 |
| liuyulong | Yes, we can start in such way. | 14:14 |
| liuyulong | Anyway, I will share some config tunning running in our cloud deployment. | 14:14 |
| slaweq | liuyulong: thx a lot | 14:15 |
| liuyulong | OK, let's move on. | 14:15 |
| liuyulong | #topic Bugs | 14:15 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:15 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-January/011831.html | 14:16 |
| liuyulong | And this I guess: | 14:16 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011766.html | 14:16 |
| liuyulong | May be also this: | 14:17 |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011751.html | 14:17 |
| liuyulong | OK, first one: | 14:17 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1858086 | 14:17 |
| openstack | Launchpad bug 1858086 in neutron "qrouter's local link route cannot be restored " [Medium,Confirmed] | 14:17 |
| liuyulong | This should be an API leak for the user input check. | 14:18 |
| liuyulong | We should not allow user to add some route destination CIDR which overlaps the subnet. | 14:19 |
| liuyulong | There are too many potential risks for DVR related traffic. | 14:19 |
| haleyb | yes, i thought i was reading that wrong but how can you add a route to a local subnet via a non-local IP ? | 14:19 |
| liuyulong | It is router route-add action? | 14:21 |
| *** jawad_axd has joined #openstack-meeting | 14:21 | |
| liuyulong | Not the subnet static route, right? | 14:22 |
| slaweq | it's "extra-route" but I'm not sure what action is called on server side for it | 14:24 |
| slaweq | on client's side You do "neutron router-update --extra-route" | 14:24 |
| *** jawad_axd has quit IRC | 14:26 | |
| liuyulong | Yes, "openstack router set --route destination=<subnet>,gateway=<ip-address>]" | 14:26 |
| liuyulong | Such overlap should not be allowed. | 14:28 |
| liuyulong | This is obvious, when you add an IP address to your host, the system will add a default on-link route for it. | 14:30 |
| liuyulong | That means "this subnet is directly accessible.", change it does not make any sense in most scenario. | 14:32 |
| liuyulong | But by the way, the bug reporter said neutron does not recover that route automatically. | 14:33 |
| haleyb | i would tend to agree, actually surprised it didn't throw an exception when adding it | 14:33 |
| liuyulong | This can be another view of the bug, since neutron does not handle such on-link route in the qrouter namespace when it is directly accessible. | 14:33 |
| liuyulong | So, I think it's OK to terminate it at the very beginning of API. | 14:35 |
| slaweq | sounds good for me | 14:35 |
| liuyulong | OK, next one. | 14:35 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1857422 | 14:36 |
| openstack | Launchpad bug 1857422 in neutron "neutron-keepalived-state-change and keeplived cannot be cleanup for those routers which is deleted during l3-agent died" [Undecided,New] | 14:36 |
| *** links has quit IRC | 14:37 | |
| *** ociuhandu has joined #openstack-meeting | 14:38 | |
| liuyulong | Firstly, because the L3-agent is dead, so the "delete RPC" will not be processed, this could be a reason why the processed remained. | 14:38 |
| haleyb | if i'm remembering correctly, the l3-agent should clean-up the namespace(s) at the end of it's sync, but is it just not cleaning keepalived stuff because it didn't know that the associated router was ha ? | 14:40 |
| liuyulong | But we did encounter similar phenomena in our own deployment when L3 agent is alive. The "neutron-keepalived-state-change" and "radvd" processes sometimes remain when routers were deleted. | 14:40 |
| haleyb | is this the same thing? | 14:42 |
| liuyulong | haleyb, I'm not sure, maybe the user's L3-agent is just dead too long time to re-process the delete RPC. | 14:42 |
| *** eharney has quit IRC | 14:42 | |
| *** ociuhandu has quit IRC | 14:43 | |
| liuyulong | haleyb, no, just some similar phenomena. | 14:43 |
| haleyb | right, if for example it didn't get the RPC, that's when the resources get orphaned? | 14:43 |
| *** ociuhandu has joined #openstack-meeting | 14:44 | |
| liuyulong | Yes, according to the "reproduction steps" in the bug description. | 14:44 |
| haleyb | i guess it seems like a valid bug | 14:46 |
| liuyulong | If we need to cover this situation, the L3-agent may need a persistent cache to distinguish which router was delete during the down time. And then starts the delete procedure for the stale routers. | 14:47 |
| *** ociuhandu has quit IRC | 14:47 | |
| *** enriquetaso has quit IRC | 14:49 | |
| *** whoami-rajat__ has quit IRC | 14:49 | |
| liuyulong | And I still have questions, the router namespace, meta-proxy and radvd process will remain too? Or just neutron-keepalived-state-change and keeplived ? | 14:50 |
| haleyb | at the end of sync, the l3-agent should have cleaned the router namespace | 14:50 |
| haleyb | initial sync at startup that is | 14:51 |
| liuyulong | And +1 to Miguel's comment, if this is not seen in the production environment, then it is contrived. : ) https://bugs.launchpad.net/neutron/+bug/1857422/comments/2 | 14:53 |
| openstack | Launchpad bug 1857422 in neutron "neutron-keepalived-state-change and keeplived cannot be cleanup for those routers which is deleted during l3-agent died" [Undecided,New] | 14:53 |
| liuyulong | Last one: | 14:54 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1856839 | 14:54 |
| openstack | Launchpad bug 1856839 in neutron "[L3] router processing time increase if there are large set ports" [Medium,In progress] - Assigned to LIU Yulong (dragon889) | 14:54 |
| liuyulong | Code is here: https://review.opendev.org/701077 | 14:55 |
| *** whoami-rajat has joined #openstack-meeting | 14:55 | |
| liuyulong | It is an optimization for large scale cloud. : ) | 14:55 |
| slaweq | I would also like to ask You for review https://review.opendev.org/#/c/700011/ if You will have some time | 14:57 |
| liuyulong | We are running out of time, maybe you can leave the comment in the gerrit. | 14:58 |
| liuyulong | Alright, let's end here. | 14:59 |
| liuyulong | #endmeeting | 14:59 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 14:59 | |
| openstack | Meeting ended Wed Jan 8 14:59:26 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-01-08-14.01.html | 14:59 |
| slaweq | thx | 14:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-01-08-14.01.txt | 14:59 |
| slaweq | o/ | 14:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-01-08-14.01.log.html | 14:59 |
| *** ayoung has joined #openstack-meeting | 15:00 | |
| *** LiangFang has quit IRC | 15:00 | |
| *** rcernin has quit IRC | 15:01 | |
| *** ykatabam has quit IRC | 15:04 | |
| *** priteau has joined #openstack-meeting | 15:06 | |
| *** liuyulong has quit IRC | 15:18 | |
| *** liuyulong_ has quit IRC | 15:18 | |
| *** ociuhandu has joined #openstack-meeting | 15:18 | |
| *** ociuhandu has quit IRC | 15:23 | |
| *** whoami-rajat has quit IRC | 15:24 | |
| *** artom has quit IRC | 15:26 | |
| *** eharney has joined #openstack-meeting | 15:28 | |
| *** ociuhandu has joined #openstack-meeting | 15:42 | |
| *** ociuhandu has quit IRC | 15:50 | |
| *** ociuhandu has joined #openstack-meeting | 15:51 | |
| *** ociuhandu has quit IRC | 15:58 | |
| *** artom has joined #openstack-meeting | 16:01 | |
| *** whoami-rajat has joined #openstack-meeting | 16:03 | |
| *** whoami-rajat has quit IRC | 16:09 | |
| *** dmacpher__ has quit IRC | 16:11 | |
| *** dmacpher__ has joined #openstack-meeting | 16:11 | |
| *** mattw4 has joined #openstack-meeting | 16:17 | |
| *** enriquetaso has joined #openstack-meeting | 16:28 | |
| *** gyee has joined #openstack-meeting | 16:45 | |
| *** thgcorrea has quit IRC | 17:02 | |
| *** ociuhandu has joined #openstack-meeting | 17:03 | |
| *** rsimai is now known as rsimai_away | 17:05 | |
| *** artom has quit IRC | 17:06 | |
| *** ociuhandu has quit IRC | 17:08 | |
| *** psachin has quit IRC | 17:12 | |
| *** simon-AS559 has left #openstack-meeting | 17:12 | |
| *** ociuhandu has joined #openstack-meeting | 17:26 | |
| *** ociuhandu has quit IRC | 17:27 | |
| *** ociuhandu has joined #openstack-meeting | 17:27 | |
| *** tbarron_ is now known as tbarron | 17:30 | |
| *** ociuhandu has quit IRC | 17:31 | |
| *** ociuhandu has joined #openstack-meeting | 17:34 | |
| *** jawad_axd has joined #openstack-meeting | 17:40 | |
| *** ociuhandu has quit IRC | 17:41 | |
| *** jawad_axd has quit IRC | 17:44 | |
| *** ociuhandu has joined #openstack-meeting | 17:44 | |
| *** ociuhandu has quit IRC | 17:49 | |
| *** rpittau is now known as rpittau|afk | 17:51 | |
| *** artom has joined #openstack-meeting | 17:58 | |
| *** jawad_axd has joined #openstack-meeting | 18:00 | |
| *** priteau has quit IRC | 18:01 | |
| *** andrebeltrami has quit IRC | 18:02 | |
| *** jawad_axd has quit IRC | 18:05 | |
| *** e0ne has quit IRC | 18:07 | |
| *** ociuhandu has joined #openstack-meeting | 18:13 | |
| *** tesseract has quit IRC | 18:15 | |
| *** ociuhandu has quit IRC | 18:17 | |
| *** jiaopengju has quit IRC | 18:25 | |
| *** jiaopengju has joined #openstack-meeting | 18:25 | |
| *** anastzhyr has quit IRC | 18:25 | |
| *** pcaruana has quit IRC | 18:27 | |
| *** dmacpher_ has joined #openstack-meeting | 18:50 | |
| *** dmacpher__ has quit IRC | 18:53 | |
| *** enriquetaso has quit IRC | 18:56 | |
| *** jawad_axd has joined #openstack-meeting | 19:02 | |
| *** jawad_axd has quit IRC | 19:07 | |
| *** bnemec has quit IRC | 19:47 | |
| *** bnemec has joined #openstack-meeting | 19:52 | |
| *** eharney has quit IRC | 20:01 | |
| *** e0ne has joined #openstack-meeting | 20:14 | |
| *** eharney has joined #openstack-meeting | 20:15 | |
| *** e0ne_ has joined #openstack-meeting | 20:28 | |
| *** e0ne has quit IRC | 20:28 | |
| *** eharney has quit IRC | 20:38 | |
| *** patchbot has joined #openstack-meeting | 20:58 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Jan 8 21:00:06 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| kota_ | o/ | 21:00 |
| seongsoocho | o/ | 21:00 |
| rledisez | hi o/ | 21:00 |
| *** mattoliverau has joined #openstack-meeting | 21:01 | |
| *** zaitcev has joined #openstack-meeting | 21:01 | |
| timburke | i'm guessing mattoliverau's around, too ;-) | 21:01 |
| timburke | agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:01 |
| mattoliverau | o/ | 21:01 |
| timburke | first up, i just had a question i wanted to pose to people | 21:02 |
| tdasilva | o/ | 21:02 |
| timburke | #topic probe tests and non-standard configs | 21:02 |
| *** openstack changes topic to "probe tests and non-standard configs (Meeting topic: swift)" | 21:02 | |
| timburke | actually, i take that back | 21:02 |
| timburke | *first up*, welcome back everyone! it's been a bit :-) | 21:03 |
| timburke | hope everyone's end-of-year breaks were good | 21:03 |
| zaitcev | I do not understand why *PROBE* tests need to be "tolerant". They already assume the structure of the cluster, and so the cluster must be artificial. This is in a stark difference with *functional* tests, which can and do operate on production clusters. | 21:03 |
| mattoliverau | Yeah, happy new year everyone! | 21:03 |
| timburke | zaitcev, fair enough. my main thinking was that i'd like to get my dev environment closer to the real thing | 21:04 |
| timburke | so, back to my original question: how much divergence from "standard" SAIO configs can we reasonably expect for probe tests? | 21:05 |
| timburke | this came up in the context of https://review.opendev.org/700818 | 21:06 |
| zaitcev | I would imagine not much. Although of course an LB appears to be immaterial, remember that they percolate because of that url_base thing. | 21:06 |
| patchbot | patch 700818 - swift - Deprecate per-service auto_create_account_prefix (MERGED) - 6 patch sets | 21:06 |
| timburke | where i wanted to try out the new config option and see some existing tests pass | 21:06 |
| timburke | that spun out https://review.opendev.org/701075 (because the reconciler *really* wants auto_create_account_prefix to be "." | 21:08 |
| patchbot | patch 701075 - swift - probe-tests: Get reconciler test passing - 2 patch sets | 21:08 |
| clayg | to the extent it's *reasonable* to make probe tests more tolerant of various configs we want to support I think it's a nobel endeavor | 21:08 |
| zaitcev | OK, look. I'm okay with anything like this, as long as the source of the tests does not get too complicated. If you post something to address the non-standard prefix and it's just a bunch of parameters passed in configs, I would be happy to review. | 21:08 |
| clayg | zaitcev: 👍 | 21:08 |
| timburke | and https://review.opendev.org/701076 where we were assuming that *no constraints at all* were set, at least for py3 | 21:08 |
| patchbot | patch 701076 - swift - probe-tests: Avoid a DuplicateSectionError on py3 - 1 patch set | 21:08 |
| clayg | yeah that one surprised me a little bit - probably worth loosing up that a bit | 21:09 |
| timburke | but then i got thinking about it more broadly, in particular while trying to get some SSL support for my dev environment... and i'm finding that it's maybe gonna be painful to make that somewhere that i can still run probe tests :-( | 21:10 |
| *** ociuhandu has joined #openstack-meeting | 21:10 | |
| clayg | "painful to make that somewhere" ? | 21:11 |
| zaitcev | I read it as "make that _work_ somewhere etc." | 21:12 |
| timburke | it seems like it will be painful to make my SSL-enabled dev environment be a place that i can still run probe tests | 21:12 |
| zaitcev | I accidentally the whole Internet | 21:12 |
| timburke | anyway, as much as anything, i just wanted to seed the idea a bit, get you guys thinking about it. i feel like we'll probably argue about it over beers in vancouver ;-) | 21:13 |
| timburke | #topic bulk upload regression | 21:13 |
| *** openstack changes topic to "bulk upload regression (Meeting topic: swift)" | 21:13 | |
| timburke | #link https://launchpad.net/bugs/1857546 | 21:14 |
| openstack | Launchpad bug 1857546 in OpenStack Object Storage (swift) "Auto-extract looses X-Delete-At and X-Delete-After headers" [High,Confirmed] | 21:14 |
| clayg | YEAH!!! Only... I think the argument is more about "value & priority" vs "should probtests be well designed and flexible" | 21:14 |
| mattoliverau | Sorry I'm a bit slow, but could we start using conf.d in saio and then probe tests can drop .conf and remove then as they restart the cluster? | 21:14 |
| clayg | oh yeah... that bulk regression 🤮 | 21:14 |
| rledisez | that one comes from one of our customer (he was faster than us to report the bug :D) | 21:15 |
| timburke | this came in over the break, wanted to make people aware of it. sorry, that one was my bad | 21:15 |
| clayg | I really *liked* the refactor tho, it's something about copied requests and what belongs in the environ? Is there a compromise that's not just "oops, tried to make it better... didn't on one axis - REVERT" | 21:15 |
| timburke | #link https://github.com/openstack/swift/commit/6f00d42 | 21:15 |
| timburke | cause ^^^ | 21:15 |
| rledisez | i'll get sure to review the PR. I guess reverting is just to probe the test, right? | 21:15 |
| rledisez | *to prove | 21:16 |
| timburke | #link https://review.opendev.org/#/c/700652/ | 21:16 |
| patchbot | patch 700652 - swift - Revert "bulk: Use make_subrequest to make subreque... - 1 patch set | 21:16 |
| timburke | fix ^^^ | 21:16 |
| rledisez | should some headers be copied (which one then?) or just copy them all? | 21:16 |
| timburke | clayg, so it's definitely *not* a complete revert -- the bug i was trying to close is (should be?) still closed | 21:17 |
| clayg | ORLY!? | 21:17 |
| timburke | maybe i should have separated it into two commits... oh well | 21:18 |
| timburke | rledisez, yeah, it seemed like the way we decide which headers to store (config option in the object server) kinda means that we *have to* copy all headers.. | 21:18 |
| timburke | at which point the sanitization we get from make_subrequest does more harm than good | 21:19 |
| clayg | timburke: so some places you added new_env['swift.proxy_access_log_made'] = False to keep the original bug closed 🤔 | 21:19 |
| timburke | yup | 21:19 |
| timburke | i still think logging subrequests must be a good idea | 21:19 |
| clayg | yes 👍 | 21:20 |
| rledisez | totally | 21:20 |
| clayg | ok, well I guess we can probably merge the "revert" and just have two fixed bugs... I thought it was either/or somehow | 21:20 |
| timburke | anyway, i think we can have more discussions about it on the review -- just wanted to raise awareness and mention that i intend to backport it once we've got the fix on master | 21:20 |
| rledisez | That behavior was actually unspecified. I'm thinking it could be a specific option instead of just relying on "some headers" to pass, and some not to | 21:21 |
| rledisez | like X-Bulk-Delete-After ? | 21:21 |
| *** sfernand has quit IRC | 21:22 | |
| timburke | *shrug* i'm not necessarily opposed | 21:22 |
| timburke | but first i want to fix your customer :-) | 21:22 |
| timburke | #topic SHA-1 | 21:22 |
| *** openstack changes topic to "SHA-1 (Meeting topic: swift)" | 21:22 | |
| clayg | rledisez: bulk has a body we can add new options there | 21:23 |
| timburke | so some researchers have demonstrated an improved chosen-prefix attack on SHA-1! | 21:23 |
| timburke | #link https://sha-mbles.github.io/ | 21:23 |
| timburke | my memory is that we currently only use SHA-1 for tempurls and formpost | 21:24 |
| timburke | oh, and admin access to /info probably... | 21:24 |
| timburke | fortunately, tempurl now has a config option to specify the allowed digests | 21:25 |
| timburke | #link https://github.com/openstack/swift/commit/5a4d3bdfc | 21:25 |
| timburke | formpost does not yet | 21:25 |
| timburke | #link https://bugs.launchpad.net/swift/+bug/1794601 | 21:25 |
| openstack | Launchpad bug 1794601 in OpenStack Object Storage (swift) "Formpost middleware should support stronger hash functions" [Undecided,New] | 21:25 |
| zaitcev | so, what now? SHA-256? | 21:26 |
| timburke | we might want to get on that bug sooner rather than later | 21:26 |
| timburke | zaitcev, that was my thinking. SHA-256 or SHA-512 | 21:26 |
| clayg | bits for days! | 21:26 |
| clayg | By renting a GPU cluster online, the entire chosen-prefix collision attack on SHA-1 costed us about 75k USD | 21:27 |
| timburke | and we should think about starting the deprecation/removal process for SHA-1 signatures. we've kinda known about the need for that for a while | 21:27 |
| clayg | I would easily remove formpost from swift for that kind of money 💰 | 21:27 |
| timburke | clayg, i think we use it for uploading tarballs ;-) | 21:28 |
| clayg | BOO | 21:28 |
| timburke | :P | 21:28 |
| clayg | i guess we'll fix it then | 21:28 |
| timburke | it shouldn't be too hard to do | 21:28 |
| rledisez | I'm not sure how it impact tempurl. generating collision does not mean you know what you're looking for (aka. you don't know the secret, or you don't need colisions). so yes, we must deprecate SHA1, just because, but there is no emergency i think | 21:28 |
| clayg | @timburke by "deprecate" you mean like "log warning" => "refused to start" if you've configured these things to use signatures we're not happy with? | 21:29 |
| *** rfolco has quit IRC | 21:29 | |
| clayg | do you ever need to support more than one kind of signature at a time? Like a migration path sort of plan? | 21:29 |
| timburke | rledisez, 👍 good to get a public cloud operator's perspective | 21:29 |
| timburke | clayg, log warning => start ignoring sha-1 if you've got it listed, was my thinking | 21:30 |
| clayg | deprecating stuff is always a good idea, we should deprecate auto_create_account_prefix | 21:30 |
| timburke | clayg, tempurl *today* should support multiple signatures | 21:30 |
| *** ralonsoh has quit IRC | 21:30 | |
| clayg | sweet so prior art even | 21:30 |
| clayg | hrmm... I wonder how we make sure it get's done "eventually" | 21:31 |
| timburke | of course, this also means swiftclient should learn how to do the new stuff ;-) | 21:31 |
| rledisez | hum, if you ignore it, you just break all already generated url, right? you can't do that, it's up to the operator to decide to remove it | 21:31 |
| clayg | maybe someone will come click that little follow button or make it 🔥 or something | 21:31 |
| *** armax has joined #openstack-meeting | 21:32 | |
| clayg | timburke: thanks for bringing this one up! | 21:32 |
| timburke | rledisez, i was hoping to get away with having a fairly long window, maybe even have a step of logging warnings *per-request* when handling sha-1 sigs... actual ignoring could be a ways off | 21:33 |
| timburke | that about does it for big topics i wanted to bring up | 21:33 |
| timburke | on to updates! | 21:33 |
| timburke | #topic versioning | 21:33 |
| *** openstack changes topic to "versioning (Meeting topic: swift)" | 21:33 | |
| clayg | 😎 | 21:34 |
| timburke | clayg, tdasilva i feel like there's been a lot of progress lately :-D | 21:34 |
| clayg | we are locking shit DOWN! | 21:34 |
| clayg | I need to re-look at this one https://review.opendev.org/#/c/698139/ | 21:35 |
| patchbot | patch 698139 - swift - Skip container-sync when versioning enabled - 9 patch sets | 21:35 |
| clayg | which is the only dangling "probably should squash" (or at least fast follow merge) before it's on master | 21:35 |
| clayg | rledisez: you may have to rub some braincells together about container-sync and versioning - for now they're not going to work together, but we won't *break* either - so I guess we were hoping that would be "enough" | 21:36 |
| clayg | this is still the meat -> https://review.opendev.org/#/c/682382/ | 21:36 |
| patchbot | patch 682382 - swift - New Object Versioning mode - 70 patch sets | 21:36 |
| clayg | and the money -> https://review.opendev.org/#/c/673682/ | 21:36 |
| patchbot | patch 673682 - swift - s3api: Implement object versioning API - 44 patch sets | 21:36 |
| timburke | how quickly should we try to get the client support from https://review.opendev.org/691877 merged after the swift patch lands? | 21:37 |
| patchbot | patch 691877 - python-swiftclient - object versioning features - 8 patch sets | 21:37 |
| timburke | how are you feeling about the client support? | 21:37 |
| clayg | of course if you have swiftclient with versions support (p 691877) you don't need to mess with that aws s3api list-object-versions non-sense | 21:37 |
| rledisez | clayg: versioning at source or at destination will not work with container-sync? | 21:37 |
| clayg | timburke: thanks for reminding me! I should work on `swift delete --all` 👍 | 21:37 |
| clayg | rledisez: at *source* was the primary concern, I'm not sure I've tested at dest | 21:38 |
| timburke | rledisez, source, i believe. clients won't be able to set both x-versions-enabled and x-container-sync-to or something like that | 21:38 |
| clayg | might be worth saying something to that effect ont he container-sync + versionsing patch | 21:38 |
| tdasilva | rledisez: I believe dest too, in that we won't allow enabling versioning if sync header is there | 21:38 |
| *** ociuhandu has quit IRC | 21:39 | |
| timburke | do we know how well a VW-enabled container serves as a sync destination? | 21:39 |
| clayg | rledisez: you might have to refresh us how container-sync works | 21:39 |
| clayg | timburke: could be a interesting data point - I don't think it's something we test - althougth we do have some probetests for container-sync | 21:39 |
| rledisez | clayg: a process scan the sqlite containers, iterate through the "new rows" (based on some sync point), and PUT/DELETE the objects to the remote endpoint. currently I think it works with versionning at destination. I don't see why it wouldn't work actually, but I didn't tested | 21:41 |
| timburke | clayg, so how soon do we expect these to land on master? and what can/should people be doing to help out? | 21:42 |
| clayg | I'd like to make a decision about "can we land new versioning on master w/o container-sync" because... we don't really have any plans to make that situation any better | 21:43 |
| clayg | if the answer is "yes" but it needs to fail/error in expected and obvious ways... then I'm sure we'll have that tied up sometime next week | 21:44 |
| clayg | we were doing some release stuff, but now I'm available to finish up whatever we need to do to land upstream | 21:44 |
| clayg | has anyone tried out the API?! | 21:44 |
| timburke | (besides me, clayg, and tdasilva ;-) | 21:45 |
| clayg | nothing of any significance has changed in AGES - it's just been spit and polish | 21:45 |
| clayg | we're certainly past the point of making significant redesign sort of conversations - so if you haven't looked at it yet, it's probably more just FYI at this point | 21:45 |
| tdasilva | viks was asking about testing the s3 api last night, which is great | 21:46 |
| clayg | we'll merge it when it's done | 21:46 |
| clayg | 🤗 viks___ | 21:46 |
| timburke | 👍 | 21:46 |
| timburke | #topic slo ranged reads | 21:46 |
| *** openstack changes topic to "slo ranged reads (Meeting topic: swift)" | 21:46 | |
| timburke | just a quick update on the patch | 21:46 |
| timburke | #link https://review.opendev.org/697739/ | 21:46 |
| patchbot | patch 697739 - swift - Have slo tell the object-server that it wants whol... - 6 patch sets | 21:46 |
| timburke | i tried out having symlink and dlo use the same header as slo, worked like a charm! | 21:47 |
| timburke | i think that cleans up the status codes getting logged at the object layer nicely, too | 21:47 |
| timburke | it'll be interesting getting opinions from other people on it | 21:48 |
| timburke | #topic quoted etags | 21:48 |
| *** openstack changes topic to "quoted etags (Meeting topic: swift)" | 21:48 | |
| timburke | #link https://review.opendev.org/700056 | 21:48 |
| patchbot | patch 700056 - swift - Middleware that allows a user to have quoted Etags - 5 patch sets | 21:48 |
| clayg | timburke: ¡excelente! | 21:48 |
| timburke | patch has tests and docs now! | 21:48 |
| timburke | thanks for proposing the new middleware, rledisez! i'm liking that direction more and more | 21:48 |
| clayg | nice! | 21:49 |
| timburke | i think the only reservations i have with it now come down to naming (both for the config option and the cotnainer-metadata header) | 21:50 |
| tdasilva | my hesitation with that patch is the need for a brand new middleware just for quoting etags. It feels like it should be an option on/off | 21:50 |
| clayg | @rledisez what do you think about a global cluster wide option as well? sounds like legacy use-cases could still turn it off per account/container with the explicit metadata | 21:50 |
| timburke | clayg, i already added that :-) defaults to off | 21:50 |
| clayg | timburke: well then let's merge that shit! | 21:51 |
| rledisez | tdasilva: i don't like the middleware neither, but a global option does not give the flexibility if per account/container. without that flexibility, we will breaks so many apps :( | 21:51 |
| timburke | tdasilva, idk -- i kinda like the separation of concerns. *especially* given the granularity rledisez built in | 21:52 |
| timburke | now, to take another look at torgomatic's https://review.opendev.org/#/c/504472/ ... | 21:53 |
| patchbot | patch 504472 - swift - Shorten typical proxy pipeline. - 4 patch sets | 21:53 |
| clayg | i don't mind having a small single purpose middleware - compared to what we do in bulk | 21:53 |
| clayg | oh and even for clusters where "oh I definately want that on everywhere" - they may someday find one account/container that wants to use some old app that blows up with quoted etags - so I love having the explicit metadata options per collection | 21:54 |
| rledisez | what I'm mostly concerned about middleware (except that paste does not really seem maintained) is the side effect of a bad ordered pipeline | 21:54 |
| clayg | yeah, with the defaults should we think about making it auto-inserted? | 21:55 |
| rledisez | timburke: about the naming of the header, I trust your english better than mine ;) | 21:55 |
| timburke | rledisez, for *sure*. i think i've come across like 4 or 5 different bug reports with super-strange (and not necessarily overlapping!) errors that came down to "pipeline was bad" | 21:55 |
| clayg | actually - that doesn't really help us... even if you auto-insert it you have to configure it | 21:55 |
| rledisez | i'll just notify the only user of this feature to prepare before we upgrade | 21:56 |
| clayg | pipelines suck.. | 21:56 |
| timburke | clayg, well, auto-inserted allows users to do self-service | 21:56 |
| clayg | timburke: oh that's nice! | 21:56 |
| timburke | rledisez, i'd also be ok with looking for an old header name for you. that's not a big deal | 21:56 |
| timburke | *shrug* | 21:57 |
| timburke | anyway, that's all i've got | 21:57 |
| clayg | timburke: althought auto-insert migth not be really what I want... it be more like "auto-ORDERED" pipelines or osmething | 21:57 |
| timburke | #topic open discussion | 21:57 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:57 | |
| timburke | clayg, auto-ordered pipelines would be *amazing* | 21:57 |
| timburke | but maybe also crazy hard, i think | 21:57 |
| rledisez | timburke: let's not polute the code, i'll maintain a small patch internaly until i can drop it | 21:57 |
| timburke | and it'd require that we actually specify middleware dependencies | 21:58 |
| timburke | maybe we could put some attributes on the filter_factories? start there? | 21:59 |
| timburke | idk | 21:59 |
| timburke | it'd be cool, though! | 21:59 |
| mattoliverau | I had some code that did that.. just to play years ago. | 21:59 |
| rledisez | timburke: for sure. even dropping the notion of pipeline by default and just have flags "enable_bulk", "enable_etag_quoter", …. keep the pipeline for advanced users | 22:00 |
| mattoliverau | it might still be up in gerrit somewhere. Probably need alot of work | 22:00 |
| rledisez | I think it was kinda like torgomatic's patch | 22:00 |
| clayg | i gotta bounce | 22:00 |
| timburke | all right, looks like we're out of time | 22:00 |
| timburke | thank you all for coming, and thank you for working on swift! | 22:00 |
| clayg | rledisez: can you drop a comment/review on it - i didn't really underestand it I don't think | 22:00 |
| timburke | #endmeeting | 22:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:00 | |
| openstack | Meeting ended Wed Jan 8 22:00:54 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-01-08-21.00.html | 22:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-01-08-21.00.txt | 22:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-01-08-21.00.log.html | 22:00 |
| *** zaitcev has left #openstack-meeting | 22:01 | |
| *** jawad_axd has joined #openstack-meeting | 22:06 | |
| *** eharney has joined #openstack-meeting | 22:08 | |
| *** patchbot has left #openstack-meeting | 22:08 | |
| *** jawad_axd has quit IRC | 22:10 | |
| *** slaweq has quit IRC | 22:15 | |
| *** e0ne_ has quit IRC | 22:31 | |
| *** armax has quit IRC | 22:33 | |
| *** ykatabam has joined #openstack-meeting | 22:47 | |
| *** jawad_axd has joined #openstack-meeting | 23:08 | |
| *** slaweq has joined #openstack-meeting | 23:11 | |
| *** jawad_axd has quit IRC | 23:12 | |
| *** dviroel has quit IRC | 23:12 | |
| *** rcernin has joined #openstack-meeting | 23:14 | |
| *** slaweq has quit IRC | 23:16 | |
| *** armax has joined #openstack-meeting | 23:19 | |
| *** eharney has quit IRC | 23:19 | |
| *** jawad_axd has joined #openstack-meeting | 23:29 | |
| *** jawad_axd has quit IRC | 23:33 | |
| *** tetsuro_ has joined #openstack-meeting | 23:50 | |
| *** tetsuro has quit IRC | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!