Wednesday, 2019-07-31

*** yamamoto has quit IRC00:04
*** slaweq has joined #openstack-meeting00:11
*** slaweq has quit IRC00:16
*** bobh has quit IRC00:44
*** diablo_rojo has joined #openstack-meeting00:53
*** tetsuro has joined #openstack-meeting00:55
*** ricolin has joined #openstack-meeting01:01
*** enriquetaso has joined #openstack-meeting01:10
*** zhangchi has joined #openstack-meeting01:17
zhangchi#startmeeting tricircle01:18
openstackMeeting started Wed Jul 31 01:18:15 2019 UTC and is due to finish in 60 minutes.  The chair is zhangchi. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.01:18
*** openstack changes topic to " (Meeting topic: tricircle)"01:18
openstackThe meeting name has been set to 'tricircle'01:18
zhangchiMorning all01:18
*** SergeyLukjanov has quit IRC01:20
zhangchiWelcome back to tricircle meeting01:21
*** SergeyLukjanov has joined #openstack-meeting01:22
*** ricolin has quit IRC01:31
*** enriquetaso has quit IRC01:32
*** zhangchi has quit IRC01:32
*** jamesmcarthur has joined #openstack-meeting01:35
*** zhangchi has joined #openstack-meeting01:46
*** openstack changes topic to "OpenStack Meetings ||"01:49
openstackMeeting ended Wed Jul 31 01:49:15 2019 UTC.  Information about MeetBot at . (v 0.1.4)01:49
openstackMinutes (text):
*** zhangchi has quit IRC01:51
*** ykatabam has quit IRC01:54
*** yamamoto has joined #openstack-meeting01:54
*** diablo_rojo has quit IRC01:59
*** yamamoto has quit IRC01:59
*** slaweq has joined #openstack-meeting02:11
*** hjwon has quit IRC02:13
*** slaweq has quit IRC02:16
*** tetsuro has quit IRC02:16
*** baojg has joined #openstack-meeting02:36
*** jamesmcarthur has quit IRC02:38
*** jamesmcarthur has joined #openstack-meeting02:45
*** armax has quit IRC02:48
*** ykatabam has joined #openstack-meeting02:48
*** njohnston has quit IRC02:51
*** tetsuro has joined #openstack-meeting02:52
*** rcernin has quit IRC02:57
*** rcernin has joined #openstack-meeting03:13
*** jamesmcarthur has quit IRC03:22
*** jamesmcarthur has joined #openstack-meeting03:23
*** jamesmcarthur has quit IRC03:27
*** tetsuro has quit IRC03:28
*** psachin has joined #openstack-meeting03:37
*** tetsuro has joined #openstack-meeting03:46
*** cheng1 has quit IRC03:52
*** cheng1 has joined #openstack-meeting03:52
*** jamesmcarthur has joined #openstack-meeting03:53
*** imsurit has joined #openstack-meeting04:00
*** jamesmcarthur has quit IRC04:00
*** slaweq has joined #openstack-meeting04:11
*** slaweq has quit IRC04:16
*** whoami-rajat has joined #openstack-meeting04:19
*** apetrich has quit IRC04:20
*** imsurit has quit IRC04:25
*** imsurit has joined #openstack-meeting04:26
*** Luzi has joined #openstack-meeting04:30
*** jamesmcarthur has joined #openstack-meeting04:30
*** jamesmcarthur has quit IRC04:38
*** nitinuikey has joined #openstack-meeting04:46
*** nitinuikey has quit IRC04:54
*** vishalmanchanda has joined #openstack-meeting05:13
*** radeks has joined #openstack-meeting05:17
*** gyee has quit IRC05:26
*** jamesmcarthur has joined #openstack-meeting05:34
*** nitinuikey has joined #openstack-meeting05:37
*** jamesmcarthur has quit IRC05:39
*** baojg has quit IRC05:47
*** links has joined #openstack-meeting05:58
*** jraju__ has joined #openstack-meeting06:02
*** links has quit IRC06:03
*** ykatabam has quit IRC06:05
*** ykatabam has joined #openstack-meeting06:07
*** jamesmcarthur has joined #openstack-meeting06:08
*** slaweq has joined #openstack-meeting06:11
*** rubasov has quit IRC06:12
*** belmoreira has joined #openstack-meeting06:13
*** nitinuikey has quit IRC06:13
*** baojg has joined #openstack-meeting06:14
*** jamesmcarthur has quit IRC06:15
*** slaweq has quit IRC06:16
*** artom has quit IRC06:32
*** artom has joined #openstack-meeting06:33
*** jamesmcarthur has joined #openstack-meeting06:57
*** slaweq has joined #openstack-meeting06:58
*** ykatabam has quit IRC07:02
*** rcernin has quit IRC07:02
*** jamesmcarthur has quit IRC07:03
*** takamatsu has quit IRC07:17
*** brault has joined #openstack-meeting07:19
*** brault has quit IRC07:23
*** apetrich has joined #openstack-meeting07:26
*** rubasov has joined #openstack-meeting07:26
*** tesseract has joined #openstack-meeting07:27
*** belmoreira has quit IRC07:28
*** tssurya has joined #openstack-meeting07:31
*** pcaruana has quit IRC07:35
*** ociuhandu has joined #openstack-meeting07:35
*** igordc has joined #openstack-meeting07:37
*** igordc has quit IRC07:38
*** ociuhandu has quit IRC07:40
*** ociuhandu has joined #openstack-meeting07:44
*** ociuhandu has quit IRC07:44
*** e0ne has joined #openstack-meeting07:58
*** belmoreira has joined #openstack-meeting08:01
*** electrofelix has joined #openstack-meeting08:03
*** lpetrut has joined #openstack-meeting08:04
*** e0ne has quit IRC08:07
*** peschk_l has joined #openstack-meeting08:11
*** tetsuro has quit IRC08:11
*** ttsiouts has joined #openstack-meeting08:11
*** pcaruana has joined #openstack-meeting08:13
*** iyamahat has quit IRC08:14
*** jgriffith has quit IRC08:14
*** ociuhandu has joined #openstack-meeting08:15
*** jgriffith has joined #openstack-meeting08:16
*** imsurit has quit IRC08:16
*** tetsuro has joined #openstack-meeting08:17
*** ttsiouts has quit IRC08:21
*** ttsiouts has joined #openstack-meeting08:22
*** cuseinovic has joined #openstack-meeting08:25
*** ttsiouts has quit IRC08:27
*** takamatsu has joined #openstack-meeting08:32
*** cheng1 has quit IRC08:36
*** ttsiouts has joined #openstack-meeting08:36
*** cheng1 has joined #openstack-meeting08:37
*** ricolin has joined #openstack-meeting08:40
*** takamatsu_ has joined #openstack-meeting08:48
*** [yann] has joined #openstack-meeting08:48
*** takamatsu has quit IRC08:49
*** cheng1 has quit IRC08:55
*** cheng1 has joined #openstack-meeting08:58
*** panda has quit IRC08:58
*** panda has joined #openstack-meeting08:59
*** belmoreira has quit IRC09:03
*** ttsiouts has quit IRC09:05
*** priteau has joined #openstack-meeting09:05
*** tetsuro has quit IRC09:06
*** e0ne has joined #openstack-meeting09:06
*** cheng1 has quit IRC09:07
*** ttsiouts has joined #openstack-meeting09:09
*** cheng1 has joined #openstack-meeting09:11
*** ociuhandu has quit IRC09:12
*** ociuhandu has joined #openstack-meeting09:12
*** ralonsoh has joined #openstack-meeting09:13
*** rfolco|ruck has joined #openstack-meeting09:17
*** imsurit has joined #openstack-meeting09:20
*** nfakhir has joined #openstack-meeting09:20
*** ykatabam has joined #openstack-meeting09:22
*** kopecmartin|off is now known as kopecmartin09:23
*** ttsiouts has quit IRC09:23
*** ttsiouts has joined #openstack-meeting09:24
*** ttsiouts has quit IRC09:28
*** takamatsu_ has quit IRC09:30
*** takamatsu has joined #openstack-meeting09:37
*** [yann] has quit IRC09:37
*** cuseinovic has quit IRC09:44
*** ttsiouts has joined #openstack-meeting09:45
*** ttsiouts has quit IRC10:00
*** ttsiouts has joined #openstack-meeting10:01
*** apetrich has quit IRC10:05
*** ttsiouts has quit IRC10:06
*** takamatsu has quit IRC10:08
*** takamatsu has joined #openstack-meeting10:24
*** lpetrut has quit IRC10:24
*** bbowen has joined #openstack-meeting10:34
*** pcaruana has quit IRC10:45
*** belmoreira has joined #openstack-meeting10:50
*** oneswig has joined #openstack-meeting10:50
*** noggin143 has joined #openstack-meeting10:55
*** witek has joined #openstack-meeting10:59
*** dh3 has joined #openstack-meeting10:59
oneswig#startmeeting scientific-sig11:00
openstackMeeting started Wed Jul 31 11:00:22 2019 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.11:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"11:00
openstackThe meeting name has been set to 'scientific_sig'11:00
oneswig#link Agenda for today
*** ricolin_ has joined #openstack-meeting11:01
*** Lucas_Gray has joined #openstack-meeting11:01
oneswigHi all11:01
oneswigTwo quick points to cover first.11:02
oneswig#topic OpenStack user survey11:02
*** openstack changes topic to "OpenStack user survey (Meeting topic: scientific-sig)"11:02
oneswig#link add your scientific openstack cloud here and be counted
oneswigIt's true, we've yet to do ours...11:02
*** ricolin has quit IRC11:03
oneswig#topic Open Infra days Rome / Milan11:03
*** openstack changes topic to "Open Infra days Rome / Milan (Meeting topic: scientific-sig)"11:03
oneswigTwo infra days in one11:03
oneswig#link CFP closes today (not sure what time)
oneswigwhat better accompaniment to the delights of Italy than the delights of open infrastructure :-)11:04
oneswigOk, let's move to today's main event11:05
oneswig#topic Monitoring for Chargeback and Accounting11:05
*** openstack changes topic to "Monitoring for Chargeback and Accounting (Meeting topic: scientific-sig)"11:05
oneswigpriteau: thanks for doing this investigation and coming along to talk about it.11:06
priteauHappy to share :)11:06
oneswig#link CloudKitty+Monasca for accounting and chargeback
*** njohnston has joined #openstack-meeting11:06
oneswig^^ That's Pierre's first findings11:06
witeknice summary11:07
oneswigHi witek, thanks for joining11:07
peschk_lI've read it this morning, we'd be glad to help with this :)11:07
priteauNice to see that you joined peschk_l. I am still fairly new to CloudKitty so hopefully I didn't write anything completely absurd.11:08
witekpriteau: have you tried editing `monasca_field_definitions.yaml` to include additional metadata?11:08
oneswigWe've been looking for integrated options for accounting for usage and this is the first contender investigated11:09
priteauwitek: I did, but didn't manage yet to get the flavor info in there. There should be a way to do it though, because I see actual values for `created_at`, `launched_at`, `host`…11:09
priteauI also discovered in the process that Monasca has a restriction of 16 metadata values per metric11:10
verdurinImpressive work. I noted the caveat about account not being taken of instance flavour for Ceilosca. That's a shame.11:11
priteauverdurin: It should be possible to pass flavor data by changing the configuration, just haven't found the perfect incantation yet.11:12
priteauI will update the post once I get it to work.11:13
verdurinGreat, thanks.11:13
oneswigI think the most interesting piece is the final section and the different ways that usage data can be collected11:13
witekpriteau: I assume you've seen that example already:
priteauGnocchi stores this metadata by default, so it would be good to have feature parity when using Monasca11:13
belmoreirapriteau: nice post. Instance flavor would be a great addition. Did you look into volume types?11:14
priteauwitek: Yes, I tried importing monasca_field_definitions.yaml from master, and also adapting meter definitions in Ceilometer. I am sure it must be something minor missing, just need some more debugging to figure it out.11:15
priteaubelmoreira: Not yet looked at charging for volumes, and the cloud I am using doesn't actually run Cinder (it's a bare-metal cloud). But you can check peschk_l's post about it:
oneswigThe model of polling for usage is subject to rounding errors and coarse granularity.  Can you cover the options for using notification events?11:16
dh3presumably when flavor type is in there, Cloudkitty can charge differently for different flavors? I'm thinking dedicated vs overcommitted CPUs11:17
peschk_lnote that the configuration format of cloudkitty has changed since this post, the reference is available here:
peschk_ldh3: yes, that's a pretty standard usecase11:18
priteaudh3: Yes, that's the idea. Check the official CloudKitty documentation for an example:
dh3OK, I see, is there a way to scale based on flavor name (e.g. "all m1.* flavors cost 8 times what o1.* flavors cost") or are the flavor costs all set explicitly?11:19
priteauoneswig: Ceilometer is the main option for capturing and publishing notification events, as it already supports many OpenStack services. However, it looks like we may have other options in the future, such as Monasca capturing events directly:
witekyes, the the pipeline for persisting notifications is missing the collecting agent only11:20
noggin143FYI, there is also an initiative going on in the public cloud working group looking at alternatives to ceilometer -
oneswigpriteau: thanks, I was wondering about that.11:20
peschk_ldh3: you need to explicitly set the price of each flavor. But globing would be a nice addition11:20
priteaudh3: I don't think you can do this with the hashmap rating module, but there is also a pyscript one which should allow complete customisation11:21
dh3peschk_l priteau thanks!11:21
oneswignoggin143: thanks - it would be good to connect to that effort.  I added Pierre's blog to their etherpad11:21
witekthey're meeting Thursdays, bi-weekly11:22
priteaudh3: But presumably you want to charge differently for each m1.* flavor, so you would have to enter multiple costs explicitly anyway11:22
peschk_lI beliieve they've a meeting tomorrow at 14h UTC11:23
priteaunoggin143: thanks for the reminder about the public cloud SIG effort, I will try to join their meetings11:23
*** ttsiouts has joined #openstack-meeting11:23
oneswigdo you know if there has been further discussion beyond the content of the etherpad?11:23
noggin143there was a discussion at the Denver summit if I remember rightly.11:24
peschk_l(dh3 CK has a meeting friday at 15h UTC, if you want to ask usquestions)11:24
*** cheng1 has quit IRC11:25
oneswigunfortunately eavesdrop does not collect meeting logs for their channel11:25
noggin143there was also a good presentation at Denver comparing the different agents -
oneswigI recognise that guy :-)11:26
priteauLog of their last meeting:
*** cheng1 has joined #openstack-meeting11:27
witekthere have been several discussion following the Denver PTG/Summit11:27
oneswigpriteau: there was also the option of having the Monasca Agent draw data from prometheus openstack exporters, what's the latest there?11:28
priteauoneswig: I got this alternative solution to work using a recent version of prometheus-openstack-exporter, which gathers server status. I extended the exporter to report the flavor ID and was able to charge based on this information.11:30
priteauI need to figure out what limitations there are compared to using Ceilometer and notifications, will report in another blog post soon.11:31
oneswigThanks priteau, I think that would be useful information.11:32
oneswignoggin143: does CERN collect usage data using APEL?11:32
peschk_lone thing with the prometheus openstack exporters is that they do not provide many chageable metrics11:32
peschk_lthey're very convenient for monitoring, but do not provide enough information for rating11:33
noggin143CERN exports usage data to APEL using a combination of data sources and cASO11:33
priteaupeschk_l: now has metrics such as `openstack_nova_server_status`, which I've used for rating with CloudKitty.11:34
peschk_lprtieau: good to know, thanks11:34
noggin143we tried ceilometer for nearly 2 years but there were many issues so belmoreira wrote a sensor to pull the data from libvirt11:34
noggin143INFN are also interested in this area (Christina Duma)11:36
oneswignoggin143: do they use a similar setup to CERN?11:36
priteaunoggin143: did you have issues with Ceilometer extracting the data or its metrics storage backend? (originally MongoDB, then Gnocchi)11:37
*** raildo has joined #openstack-meeting11:38
belmoreirapriteau we only used MongoDB (Gnocchi was still not available). Extracting data as nearly impossible11:39
belmoreiracurrently we only keep ceilometer for the notifications11:40
priteauI had the same experience on Chameleon. We had to purge the MongoDB database regularly if we wanted queries to return a result.11:40
noggin143For INFN, they did not have a good solution and were looking for advice11:40
priteauoneswig: I just found that the publiccloud meeting logs are actually at
*** raub has joined #openstack-meeting11:41
priteauThey still use publiccloud_wg instead of publiccloud_sig for their meeting name.11:42
noggin143ceilometer at scale also created very high keystone load11:42
oneswigah ok, thanks priteau11:42
oneswignoggin143: I think we've been using the caso collector, or something similar to it, for parts of the IRIS project.11:43
oneswigDid anyone have more to add on this, or suggestions for other areas to explore?11:45
priteauSomething nice about CloudKitty is that it's very easy to extend. I wrote a proof of concept collector that uses the Nova os-simple-tenant-usage API to charge for historical Nova usage.11:46
peschk_lWell, we'd love some feedback about people using CK with monasca, as there are only few existing integrations. priteau 's article was nice for that11:48
peschk_lpriteau: is the code available somewhere ?11:49
*** noggin143 has quit IRC11:49
*** ociuhandu has quit IRC11:49
priteaupeschk_l: I'll clean it up a bit and share it11:49
witekone thing I'd like to move forward in future is to instrument the OpenStack services itself11:50
witekso that they can provide valuable measurements about their state (and usage) without the need of external black-box monitoring11:51
oneswigwitek: sounds like really useful insights.11:51
oneswigI am sure there are a lot of "golden signals" that are buried deep within these services.11:51
priteauThis would be great for monitoring large OpenStack deployments. You can monitor things externally, like API response time, but having internal insight would be even better.11:52
dh3witek: that sounds good, we do some ad-hoc monitoring of e.g. rabbitmq queue sizes, would be nice to have that instrumented "out of the box"11:52
oneswigWe are nearing time to close.  Final points for discussion?11:54
oneswig#topic AOB11:55
*** openstack changes topic to "AOB (Meeting topic: scientific-sig)"11:55
dh3just a quick call for anyone with experience of Spark/Hail talking to Ceph rgw/S3 - got a user getting unexpected Content-Length errors which I can't reproduce - would like to talk to someone else who has done this11:56
oneswigMight be one for verdurin?11:57
verdurindh3: we have some interest here, but no-one running it yet11:58
dh3verdurin: ah OK, thanks anyway11:58
oneswigdh3: how has that rolling ceph upgrade gone?11:58
dh3oneswig: complete :)11:58
oneswigI'm watching one deep-scrub before moving to Nautilus, hopefully this afternoon11:58
dh3(now we are battling large omap objects and orphaned bits and pieces...........)11:58
oneswigdh3 - that's a story for another day, surely11:59
oneswigAre you going to CERN Ceph day in September?11:59
dh3no but my colleague Matthew is11:59
dh3he will probably have war stories to share :)12:00
oneswigGreat - hopefully I'll see him there.12:00
*** dviroel has joined #openstack-meeting12:00
oneswigOK we are out of time12:00
verdurinThanks, bye.12:00
oneswigthanks priteau and everyone for joining12:00
*** openstack changes topic to "OpenStack Meetings ||"12:00
openstackMeeting ended Wed Jul 31 12:00:36 2019 UTC.  Information about MeetBot at . (v 0.1.4)12:00
openstackMinutes (text):
*** ykatabam has left #openstack-meeting12:00
priteauGood bye12:00
*** oneswig has quit IRC12:00
peschk_lthanks oneswig12:01
priteauThanks witek and peschk_l for joining, I'll keep you posted of my progress.12:01
witekthanks priteau, good work12:02
*** links has joined #openstack-meeting12:07
*** jraju__ has quit IRC12:07
*** takamatsu has quit IRC12:10
*** pcaruana has joined #openstack-meeting12:10
peschk_lthanks priteau12:14
*** carloss has joined #openstack-meeting12:19
*** ociuhandu has joined #openstack-meeting12:22
*** takamatsu has joined #openstack-meeting12:31
*** priteau has quit IRC12:39
*** psachin has quit IRC12:41
*** bobh has joined #openstack-meeting12:43
*** apetrich has joined #openstack-meeting12:43
*** apetrich has quit IRC12:44
*** apetrich has joined #openstack-meeting12:45
*** jamesmcarthur has joined #openstack-meeting12:46
*** bobh has quit IRC12:47
*** lpetrut has joined #openstack-meeting12:50
*** ricolin_ is now known as ricolin12:50
*** Luzi has quit IRC12:54
*** dh3 has quit IRC12:59
*** lpetrut has quit IRC12:59
*** lpetrut has joined #openstack-meeting12:59
*** brtknr has left #openstack-meeting13:16
*** tesseract has quit IRC13:20
*** enriquetaso has joined #openstack-meeting13:22
*** tesseract has joined #openstack-meeting13:24
*** lseki has joined #openstack-meeting13:25
*** ttsiouts has quit IRC13:25
*** ttsiouts has joined #openstack-meeting13:26
*** bbowen has quit IRC13:26
*** mriedem has joined #openstack-meeting13:27
*** bbowen has joined #openstack-meeting13:27
*** bbowen has quit IRC13:29
*** bbowen has joined #openstack-meeting13:30
*** ttsiouts has quit IRC13:31
*** Lucas_Gray has quit IRC13:33
*** Lucas_Gray has joined #openstack-meeting13:38
*** ociuhandu has quit IRC13:39
*** ociuhandu has joined #openstack-meeting13:39
*** ttsiouts has joined #openstack-meeting13:41
*** belmoreira has quit IRC13:43
*** belmoreira has joined #openstack-meeting13:45
*** liuyulong has joined #openstack-meeting13:46
*** eharney has joined #openstack-meeting13:55
*** jamesmcarthur has quit IRC13:55
*** belmoreira has quit IRC13:56
liuyulong#startmeeting neutron_l314:00
openstackMeeting started Wed Jul 31 14:00:59 2019 UTC and is due to finish in 60 minutes.  The chair is liuyulong. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.14:01
*** openstack changes topic to " (Meeting topic: neutron_l3)"14:01
openstackThe meeting name has been set to 'neutron_l3'14:01
liuyulongToday will be a quick short meeting, IMO.14:02
liuyulong#topic Announcements14:02
*** openstack changes topic to "Announcements (Meeting topic: neutron_l3)"14:02
liuyulongJust added my name here ^14:03
slaweqgreat, we will finally meet in person liuyulong :)14:03
liuyulong#chair haleyb14:03
openstackCurrent chairs: haleyb liuyulong14:03
*** Lucas_Gray has quit IRC14:04
liuyulongI have no more announcements. IMO, we have reminded all of them yesterday in team meeting.14:05
liuyulongOK, let's move on.14:06
liuyulong#topic Bugs14:06
*** openstack changes topic to "Bugs (Meeting topic: neutron_l3)"14:06
liuyulongBoden Russell was our bug deputy last week, thanks.14:06
liuyulongAnd again, I will skip all the bugs which were fixed or the related patches are getting merged now.14:07
liuyulongFirst one:14:07
openstackLaunchpad bug 1837635 in neutron "HA router state change from "standby" to "master" should be delayed" [Undecided,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez)14:07
liuyulongFor the fix, IMO, it looks good to me, but I'd like to see more deep test results.14:08
liuyulongI just added two scenarios here:
*** jraju__ has joined #openstack-meeting14:09
*** links has quit IRC14:09
liuyulongSometimes, the actual results of the running program may be different from what you expected.14:10
openstackLaunchpad bug 1834308 in neutron "[DVR][DB] too many slow query during agent restart" [Medium,In progress] - Assigned to LIU Yulong (dragon889)14:10
liuyulongI submitted the patch set yesterday.14:11
liuyulongIt is here:
liuyulongA pep8 failure...14:11
liuyulongAll the DB query in this patch has the highest frequency of call when restart ovs-agent.14:12
haleybdo you have numbers on the improvement?14:13
liuyulongAnd it is time-consuming, when your 'ports' table is getting larger and lager. These query will have a worse results.14:14
*** iyamahat has joined #openstack-meeting14:14
liuyulong40 nodes of ovs-agent restart will call these DB query about 300K+ times.14:14
*** belmoreira has joined #openstack-meeting14:15
liuyulongAnd these query costs about 0.1s+ seconds logged by our mariadb cluster.14:17
haleyb.1s per query?14:17
*** brinzhang_ has quit IRC14:17
liuyulongYes, one of them is about 0.5s+. Let me link it in the gerrit.14:18
*** brinzhang_ has joined #openstack-meeting14:18
*** brinzhang_ has quit IRC14:18
*** brinzhang_ has joined #openstack-meeting14:18
liuyulongThis one.14:19
liuyulonghaleyb, the results is when the ports table has about 10-20K records.14:20
*** brinzhang has quit IRC14:21
haleybso _get_ports_query() is really slow14:23
liuyulongThe scale of resource is about: 17000+ VMs, 3000+ DVR routers, 3000+ network, 3000+ subnets and 3000+ security groups; 40 security group rules for each security group.14:25
liuyulongAfter this change, the ovs-agent restart time has a very significant improvement, it's about 40-50mins to 15mins.14:26
njohnstonI wonder if it would be further optimized by adding an index specifically on the Port.device_owner field.  I'll comment on the change.14:26
tidwellrhi, btw14:27
haleybliuyulong: that's quite an improvement, even if 15mins is still a long time :)14:27
*** AlanClark has joined #openstack-meeting14:27
liuyulong40 - 50 mins, I can't believe it once, but indeed it is.14:27
*** takamatsu has quit IRC14:28
liuyulongrpc_loop 1 it will scan the ports and process it. 40-50mins.......14:28
haleybliuyulong: it looks like you have lots of reviewers now14:30
liuyulongMore detail about our test deployment is: 3 neutron-server with about (172 workers), its 3 nodes DB and 3 nodes MQ, are all in dedicated server.14:31
liuyulongYes, neutron has its own DB and MQ.14:31
*** trident has quit IRC14:32
*** jraju__ has quit IRC14:34
liuyulongLast one:14:35
openstackLaunchpad bug 1838431 in neutron "[scale issue] ovs-agent port processing time increases linearly and eventually timeouts" [Undecided,New]14:35
liuyulongMore like a L2 issue...14:35
*** trident has joined #openstack-meeting14:36
slaweqliuyulong: this one looks like related to already known problem with "remote_security_group" rules in SG14:37
slaweqthere was bug reported for it already IIRC14:37
liuyulongThe test have not get successfully yet.14:37
liuyulongI will test it again today.14:38
liuyulongOne more interesting thing is that we disable the DHCP for this test. No DHCP agent in this test. I can image if DHCP is enabled the vif-plug-timeout may get more...14:40
liuyulongThat's all bugs from me.14:41
liuyulongany other bugs that need the team to pay attention?14:41
haleybthere was one miguel filed yesterday14:41
slaweqliuyulong: I can't find any bug reported for Your last issue but please check in L34714:41
slaweqnjohnston: raised this problem on last PTG14:41
liuyulongslaweq, OK, great14:41
openstackLaunchpad bug 1838449 in neutron "Router migrations failing in the gate" [Medium,Confirmed] - Assigned to Miguel Lavalle (minsel)14:42
slaweqliuyulong: please do Your test without security group rules which reference to remote_group_id14:42
slaweqthan it should be much, much faster14:42
haleybliuyulong: that was the only other bug i had, was going to try and reproduce locally today for miguel14:44
slaweqhaleyb: yes, this one hurts us quite lot in CI jobs14:45
liuyulongslaweq, actually I refactor may test to 27 tenants yesterday. It looks better now.14:45
liuyulonghaleyb, thanks for bring up this, seems Miguel has found the issue code.14:46
slaweqliuyulong: because if You have more tenants, there is less IPs (ports) using same security group probably and thus it's faster14:46
liuyulongslaweq, yes, and I'm trying to add more security group for each tenant, or network.14:47
*** trident has quit IRC14:47
liuyulongFor one tenant and one default security group, it is a disaster.14:48
liuyulongIMO, every one try to test this will be very easy to encounter this problem.14:49
liuyulongnjohnston's PTG summary looks very similar to this.14:49
slaweqliuyulong: yes, we had this issue too14:49
*** trident has joined #openstack-meeting14:50
liuyulongAnd maybe some security group DB query also need some optimizing work.14:51
liuyulongOK, next topic14:51
liuyulong#topic Routed Networks14:51
*** openstack changes topic to "Routed Networks (Meeting topic: neutron_l3)"14:51
*** belmoreira has quit IRC14:51
liuyulongmlavalle, tidwellr, wwriverrat: your turn now.14:51
tidwellrif mlavalle and wwriverrat don't have anything, we can talk briefly floating IP's for routed networks14:53
tidwellr and the POC code
njohnstonI don't see mlavalle online14:55
tidwellrif it isn't obvious by my nagging folks to take a look at these, this has turned into my pet project :)14:55
tidwellrI've spun up a little lab where I've tested the POC code, it seems to work nicely and it's not terribly invasive. What I'm interested in is feedback about the approach in the spec14:57
liuyulongtidwellr, thank you for replying to my question in the patch sets.15:00
liuyulongOK, let's end the meeting.15:00
*** openstack changes topic to "OpenStack Meetings ||"15:00
openstackMeeting ended Wed Jul 31 15:00:17 2019 UTC.  Information about MeetBot at . (v 0.1.4)15:00
openstackMinutes (text):
*** trident has quit IRC15:00
liuyulongThank you guys for attending.15:00
*** armax has joined #openstack-meeting15:02
*** trident has joined #openstack-meeting15:03
*** belmoreira has joined #openstack-meeting15:03
*** rosmaita has joined #openstack-meeting15:06
*** zhengMa has joined #openstack-meeting15:07
*** trident has quit IRC15:08
*** yamahata has quit IRC15:15
*** iyamahat has quit IRC15:15
*** trident has joined #openstack-meeting15:16
*** jamesmcarthur has joined #openstack-meeting15:18
*** diablo_rojo has joined #openstack-meeting15:19
*** trident has quit IRC15:24
*** bbowen has quit IRC15:26
*** trident has joined #openstack-meeting15:27
*** tosky has joined #openstack-meeting15:27
*** lpetrut has quit IRC15:34
*** trident has quit IRC15:35
*** trident has joined #openstack-meeting15:38
*** ttsiouts has quit IRC15:42
*** ttsiouts has joined #openstack-meeting15:42
*** jamesmcarthur has quit IRC15:44
*** tssurya has quit IRC15:46
*** ttsiouts has quit IRC15:47
*** e0ne has quit IRC15:49
*** e0ne has joined #openstack-meeting15:50
*** belmoreira has quit IRC15:50
*** gyee has joined #openstack-meeting15:52
*** woojay has joined #openstack-meeting15:53
*** sfernand has joined #openstack-meeting15:53
*** sfernand has left #openstack-meeting15:54
*** zaneb has quit IRC15:56
*** efried is now known as efried_rollin15:57
*** witek has quit IRC15:59
*** ociuhandu has quit IRC15:59
*** sfernand has joined #openstack-meeting16:00
jungleboyj#startmeeting Cinder16:00
openstackMeeting started Wed Jul 31 16:00:18 2019 UTC and is due to finish in 60 minutes.  The chair is jungleboyj. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Cinder)"16:00
openstackThe meeting name has been set to 'cinder'16:00
jungleboyjcourtesy ping:  jungleboyj whoami-rajat rajinir lseki carloss pots woojay erlon geguileo eharney rosmaita enriquetaso e0ne smcginnis davidsha walshh_ xyang hemna _hemna tosky16:00
_pewp_jungleboyj \( ・_・)16:00
*** thgcorrea has joined #openstack-meeting16:01
jungleboyjCinderers Assemble!16:02
jungleboyjGive people one more minute here.16:03
*** andrewar_ has joined #openstack-meeting16:04
rosmaitai don't see how anyone would dare to be late after hearing "Cinderers Assemble!"16:04
jungleboyjrosmaita:  :-)16:04
jungleboyjSecretly I needed more coffee.  I am the late one.16:04
jungleboyj#topic announcements16:05
*** openstack changes topic to "announcements (Meeting topic: Cinder)"16:05
jungleboyjI have submitted a request to be part of the Shanghai PTG.16:05
jungleboyjI know that rosmaita and I are going to be there.16:05
jungleboyjWell, I don't have travel approved yet, but the Lenovo offices are hosting OUI, so they better get me there.16:06
* e0ne didn't get approval yet16:06
jungleboyjAnyway, I estimated our attendance at 20 people, half what we have had in the past, and asked for 2 days for technical discussion and .25 days for onboarding.16:06
jungleboyjHaven't heard any confirmation on the request yet.16:07
jungleboyjSo, that is the plan there.16:07
jungleboyjWe have the mid-cycle coming up in Raleigh in less than a month now:
jungleboyjWe have 4 people confirmed.16:08
jungleboyj4 or 5 remote attendees.16:08
jungleboyjI am hoping to have zoom to use to host the meeting this time.16:08
jungleboyj*fingers crossed*16:08
jungleboyjPlease add topics to the etherpad.16:09
jungleboyjI will work on getting topics organized before the meeting.16:09
whoami-rajatjungleboyj: hangout was good last time :)16:09
jungleboyjWe will have hangouts if nothing else.  :-)16:10
*** zaneb has joined #openstack-meeting16:10
jungleboyjThat is all I had for announcements16:10
jungleboyj#topic Milestone-2 follow-up16:10
*** openstack changes topic to "Milestone-2 follow-up (Meeting topic: Cinder)"16:10
jungleboyjI think we got the possible drivers and specs merged.16:11
jungleboyjThank you to all who helped make that happen.16:11
jungleboyjhemna_:  and rosmaita  The last minute help was greatly appreciated.16:11
whoami-rajatThanks rosmaita16:12
rosmaitade nada16:12
jungleboyjSo, I wanted to ask about the spec for updating the scheduler to use the DB.16:12
jungleboyje0ne: Wants to delay that to the Unicorn release so he can focus on the generic backup work.16:12
jungleboyjThat sounds like a good plan to me.16:13
e0nejungleboyj: thanks for mentioning this16:13
eharneymore time for design thought on that sounds like it would be beneficial anyway, so, cool16:13
jungleboyjYeah ... I am fine with that.16:13
jungleboyje0ne: Thanks for being realistic about what you can get done.16:13
*** lpetrut has joined #openstack-meeting16:14
e0nejungleboyj: thanks for supporting me. I appreciate your help16:14
jungleboyje0ne: Any time.16:14
jungleboyjSo, was there anything else that people wanted to bring up about Milestone-2?16:14
*** lpetrut has quit IRC16:15
*** lpetrut has joined #openstack-meeting16:15
jungleboyj#topic  Leverage hw accelerator in image compression16:15
*** openstack changes topic to "Leverage hw accelerator in image compression (Meeting topic: Cinder)"16:15
*** gyee has quit IRC16:16
jungleboyjzhengMa:  Hey.16:16
*** trident has quit IRC16:16
*** gyee has joined #openstack-meeting16:17
zhengMaHappy for spec merged16:17
jungleboyjYes.  Good news.16:17
zhengMaAnd we have implemented according to the spec16:17
*** baojg has quit IRC16:17
jungleboyjGood.  Looks like you patch is currently failing the check.16:17
zhengMa#spec link
zhengMayes, because of the unit test..16:18
jungleboyjYou are also missing an update to cinder/opts.py16:18
zhengMawe have not implemented yet :(16:18
zhengMatox right?16:19
jungleboyjtox -e genopts16:19
zhengMasorry, my mistake16:19
jungleboyjCheck the pep8 check for details.16:19
zhengMaOk, got it16:20
jungleboyjOtherwise, it does appear that you have a UT issue.16:20
jungleboyjzhengMa:  Is this just a request for reviews?16:21
zhengMayes, the newly added container_format property DOES impact some unit test16:21
*** trident has joined #openstack-meeting16:21
jungleboyjIf so, noted.  If you can get the items I have noted fixed we will take a look.16:21
zhengMajungleboyj: yes, we want some comments so we can quickly fix our code:)16:22
jungleboyjOk.  Will do.  Start by getting the check passing please.16:22
zhengMaOk, thank you!16:22
eharneydoes it implement the non-hardware compression path?16:22
*** electrofelix has quit IRC16:22
zhengMaeharney: yes, we implemented a gzip class for software compression16:23
zhengMajungleboyj: got it, thank you!16:23
jungleboyjCool.  Any other questions?16:23
* jungleboyj marks hemna_ tardy16:23
rosmaitazhengMa: those failures look like a test fixture problem16:24
_pewp_jungleboyj (/ .□.) ︵╰(゜Д゜)╯︵ /(.□. )16:24
jungleboyjrosmaita:  ++16:24
jungleboyj#topic question about the cross-project GPG encryption effort16:25
*** openstack changes topic to "question about the cross-project GPG encryption effort (Meeting topic: Cinder)"16:25
zhengMarosmaita: yes, I have just read the log. And we're going to fix it16:25
rosmaitawe have an approved spec for train, "image encryption and decryption"16:25
rosmaitait says that "This spec requires the implementation of appropriate encryption/decryption functionality in a global library shared between the components involved in image encryption workflows (Nova, Cinder, OSC), like os-brick"16:25
rosmaitai was reviewing the glance spec for this, and that one says about the global library, "We determined to use os-brick".16:25
rosmaitawhich is a bit stronger statement16:25
rosmaitaso this is partly an awareness notice16:26
rosmaitabut also, i'd think we'd want a spec outlining the proposed changes in os-brick?16:26
rosmaitahow do we usually handle that?16:26
hemna_rosmaita:do we have a cinder spec?16:27
jungleboyjI don't think we have required a separate spec for os-brick in the past.16:27
jungleboyjI could be wrong though.16:27
rosmaitahemna_: we have just the general spec16:27
rosmaitai just want to make sure everyone is on board with using os-brick for this16:28
jungleboyjYes, we did a lot of discussion around this and everyone reached agreement on os-brick rather than trying to create some other library.16:29
smcginnisIt would be good if the spec detailed what changes are needed in os-brick to support this since cinder and glance are going to both dependent on those changes.16:29
rosmaitai've been attending the encryption team pop-up meeting and can put it on the agenda16:30
rosmaitacan ask for a revision to the spec16:30
jungleboyjrosmaita: ++16:30
jungleboyjSounds like a good way to proceed.16:30
rosmaitaok, that's all from me, then16:30
jungleboyjrosmaita: Thank you!16:31
jungleboyj#topic py3.7 review underway16:31
*** openstack changes topic to "py3.7 review underway (Meeting topic: Cinder)"16:31
jungleboyjSo, for awareness I have started the py37 review:16:31
*** zbitter has joined #openstack-meeting16:32
jungleboyjSince I am going through all the CIs I am taking general notes in the etherpad as well.16:32
*** zaneb has quit IRC16:32
jungleboyjHave a fear that the results of my investigation will become a long mid-cycle discussion.16:32
rosmaitagood thing we have an otherwise light agenda16:33
jungleboyjRegardless, I am going to keep working through the review and will present the results here next week before starting to push up 'unsupported' patches.16:33
jungleboyjrosmaita:  Yeah.  For better or worse.16:33
jungleboyjAnyway, that was all I had on that.  Was for awareness more than anything and you can watch the results develop in that etherpad.16:34
smcginnisDo we need to restate the motivation and importance of this effort for anyone?16:34
*** ociuhandu has joined #openstack-meeting16:35
eharneyi'm pretty sure a few drivers have dependencies that don't state py3.7 support, that might be something to scan while looking through the drivers16:35
hemna_rosmaita:brick seems like the right place for shared code for stuff related to volumes16:35
jungleboyjeharney: Guidance on how to stop that if they are running py37?16:35
eharneyjungleboyj: not sure what you mean16:36
jungleboyjNot sure what you are saying maybe.16:36
*** igordc has joined #openstack-meeting16:36
jungleboyjDo you mean there are a few drivers that don't support 3.7 because of dependencies that aren't ready?16:36
eharneyi mean we can probably identify some drivers that are likely to have issues with this, if we want to16:37
hemna_if we identify drivers that can't py3.7, we can just mark them unsupported for the release and let them fix their stuffs16:37
jungleboyjOk.  If you know of ones that are going to have issues putting them into the etherpad would be helpful.16:37
*** lpetrut has quit IRC16:37
jungleboyjhemna_: Based on the stance we have made, that would seem to be waht we need to do.16:38
*** jamesmcarthur has joined #openstack-meeting16:38
jungleboyjAnything else on that discussion.16:38
jungleboyj#topic open discussion16:39
*** openstack changes topic to "open discussion (Meeting topic: Cinder)"16:39
*** panda has quit IRC16:39
jungleboyjAny other topics for discussion today?16:40
*** panda has joined #openstack-meeting16:41
jungleboyjGoing once.16:42
jungleboyjGoing twice ..16:42
jungleboyjOk.  I guess that is all we have for today.16:43
smcginnisThanks jungleboyj16:43
whoami-rajatjungleboyj: Thanks!16:43
jungleboyjThanks everyone for joining.  Talk to you all next week!16:43
*** openstack changes topic to "OpenStack Meetings ||"16:44
openstackMeeting ended Wed Jul 31 16:44:00 2019 UTC.  Information about MeetBot at . (v 0.1.4)16:44
openstackMinutes (text):
*** woojay has left #openstack-meeting16:44
*** rosmaita has left #openstack-meeting16:45
*** diablo_rojo has quit IRC16:46
*** diablo_rojo has joined #openstack-meeting16:46
*** takamatsu has joined #openstack-meeting16:47
*** trident has quit IRC16:48
*** zhengMa has quit IRC16:48
*** diablo_rojo has quit IRC16:48
*** lpetrut has joined #openstack-meeting16:48
*** tosky has left #openstack-meeting16:48
*** diablo_rojo has joined #openstack-meeting16:49
*** ricolin has quit IRC16:50
*** trident has joined #openstack-meeting16:51
*** ociuhandu has quit IRC16:52
*** jgriffith has quit IRC16:57
*** takamatsu has quit IRC16:59
*** andrewar_ has left #openstack-meeting16:59
*** jamesmcarthur has quit IRC17:00
*** lpetrut has quit IRC17:01
*** diablo_rojo_ has joined #openstack-meeting17:01
*** ekcs has joined #openstack-meeting17:04
*** diablo_rojo has quit IRC17:04
*** diablo_rojo_ has quit IRC17:10
*** diablo_rojo has joined #openstack-meeting17:11
*** e0ne has quit IRC17:18
*** ociuhandu has joined #openstack-meeting17:21
*** ociuhandu has quit IRC17:26
*** igordc has quit IRC17:32
*** imsurit has quit IRC17:40
*** yamahata has joined #openstack-meeting17:44
*** iyamahat has joined #openstack-meeting17:45
*** ralonsoh has quit IRC17:57
*** tesseract has quit IRC18:05
*** radeks has quit IRC18:11
*** altlogbot_0 has quit IRC18:19
*** altlogbot_3 has joined #openstack-meeting18:20
*** radeks has joined #openstack-meeting18:22
*** bbowen has joined #openstack-meeting18:25
*** njohnston has quit IRC18:35
*** igordc has joined #openstack-meeting18:36
*** e0ne has joined #openstack-meeting18:37
*** takamatsu has joined #openstack-meeting18:43
*** sfernand has quit IRC18:53
*** radeks_ has joined #openstack-meeting18:56
*** radeks has quit IRC18:58
*** thgcorrea has quit IRC19:01
*** e0ne has quit IRC19:02
*** whoami-rajat has quit IRC19:08
*** e0ne has joined #openstack-meeting19:12
SotK#startmeeting storyboard19:13
openstackMeeting started Wed Jul 31 19:13:53 2019 UTC and is due to finish in 60 minutes.  The chair is SotK. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.19:13
*** openstack changes topic to " (Meeting topic: storyboard)"19:13
openstackThe meeting name has been set to 'storyboard'19:13
SotK#link Agenda19:14
SotKor something approximating one anyway19:14
SotKI don't think there's much to discuss this week, so this could be short19:14
fungisounds good19:15
SotK#topic Migration Updates19:15
*** openstack changes topic to "Migration Updates (Meeting topic: storyboard)"19:15
SotK#link!/board/45 Progress Board19:15
SotKthe status/progress board is up to date now I believe19:16
SotKI don't know that there are any other updates here though19:17
fungiyeah, i'm unaware of any19:18
SotK#topic PTG + Forum19:18
*** openstack changes topic to "PTG + Forum (Meeting topic: storyboard)"19:18
SotKwe should start thinking about what we want to do at the PTG + Forum19:19
SotKlast PTG there was some discussion of having onboarding sessions for contributors and users19:19
SotKI don't think I'll be there but I'm happy to call in like we've done in the past assuming that its possible and at a reasonable time19:21
fungiyes, do we have a brainstorming etherpad chosen yet?19:21
fungii don't know how reasonable shanghai times will be for you, and also am somewhat skeptical that we'd manage decent voice quality from behind the great firewall, but it's worth a try19:22
SotKit'd need to be mid-afternoon in Shanghai time at the earliest really I think19:23
SotKI don't think we have an etherpad yet, I'll try to remember to make one in the near future unless someone beats me to it19:23
fungii can just make one now19:24
*** e0ne has quit IRC19:24
SotKthanks :)19:25
*** njohnston has joined #openstack-meeting19:25
SotK#topic In Progress Work19:26
*** openstack changes topic to "In Progress Work (Meeting topic: storyboard)"19:26
SotK#link!/board/1 Board for tracking active development19:27
SotKwe updated and populated that board the other week, to give a rough overview  of what's currently being worked on and what our priorities are19:27
SotK#link!/board/115 Board for notable lists19:28
*** njohnston has left #openstack-meeting19:28
SotKwe also made that board which contains stories sorted by some interesting tags, to get a view of the triaged work we have available19:29
SotKI've still not got round to doing the reviews I keep promising, I'll continue to try to get to that in the next couple of days19:30
*** bbowen has quit IRC19:31
SotKsimilarly the attachments work is still waiting on me finding some free time and energy, hopefully I should get to that in the next couple of weeks19:31
SotKanything else to mention here?19:32
*** radeks_ has quit IRC19:32
fungii think that about covers it19:33
fungii was hoping to run some more slow query analysis this week19:34
fungibut haven't found time yet19:34
fungithe week is still somewhat young, at least19:34
diablo_rojo_phonSorry I'm late.19:34
diablo_rojo_phonWent for a run that lasted longer than it should have.19:34
fungiwe can take a few to let you catch up, though it's been brief19:34
SotKheh, np :)19:34
* diablo_rojo_phon reada19:35
*** liuyulong has quit IRC19:35
diablo_rojo_phonCaught up.19:36
diablo_rojo_phonI don't think I have anything to add to what you said aside from encouraging everyone to think of how we want to do onboarding. At the ptg? Or trying to do a forum session?19:37
SotKmaybe contributors at the ptg and users at the forum? idk19:38
diablo_rojo_phonThat would make a lot of sense lol19:39
diablo_rojo_phonI can probably draft a forum proposal for the user onboarding in the etherpad19:39
SotKsounds good19:40
*** kopecmartin is now known as kopecmartin|off19:41
diablo_rojo_phonAs for ptg..I can put in a request for space. I know the person with the request survey ;)19:41
fungido we think there would be more use for forum interaction than ptg this time?19:42
fungior vice versa?19:42
diablo_rojo_phonI don't think I'd have any reason for ptg things aside from onboarding contributors19:43
* SotK agrees19:45
fungiyeah, if we don't expect to have critical mass of folks on hand in shanghai to hack on sb development or documenting or something then that makes sense19:46
*** takamatsu has quit IRC19:47
*** efried_rollin is now known as efried19:47
*** eharney has quit IRC19:48
fungicorvus: tarballs?19:49
fungier, wrong channel19:49
fungieverybody loves a tarball19:49
diablo_rojo_phonSotK: there is travel support..if codethink will give you the time..19:50
diablo_rojo_phonIt's open through the 8th.19:50
fungiwe can even provide tarballs19:50
SotKits a long way to travel though, and I don't have sufficient holidays spare really19:52
* mordred hands SotK a tarball19:53
SotKactually, speaking of tarballs, we should fix
fungiwe probably ought to move to anyway (which was the actual context of the contextless "tarballs!" interjection earlier)19:55
*** e0ne has joined #openstack-meeting19:55
SotKyeah, that sounds like a good idea19:56
fungibut yes, i thought i'd cleaned that tree up once already19:56
fungimaybe we're continuing to upload unpacked content there?19:56
fungii'll double-check dates19:56
diablo_rojo_phonSotK: yeah definitely don't take vacation for it, but if they would let you work from Shanghai you might be able to get TSP.19:57
fungi-rwxrwxr-x 1 jenkins jenkins 1224 Mar 15 19:30 /srv/static/tarballs/storyboard-webclient/index.html19:57
fungiwe've merged content as recently as june for that repo, so it's no longer happening (i think)19:58
fungiwill clean it up now19:58
* SotK doesn't think that would be likely19:59
SotKThanks fungi19:59
fungicleaned up19:59
SotKalright, we're out of time, thanks for coming folks20:00
*** openstack changes topic to "OpenStack Meetings ||"20:00
openstackMeeting ended Wed Jul 31 20:00:20 2019 UTC.  Information about MeetBot at . (v 0.1.4)20:00
openstackMinutes (text):
diablo_rojo_phonThanks SotK!20:00
fungithanks SotK!20:01
*** e0ne has quit IRC20:12
*** eharney has joined #openstack-meeting20:35
*** AlanClark has quit IRC20:35
*** witek has joined #openstack-meeting20:36
*** trident has quit IRC20:44
*** mriedem has quit IRC20:44
*** mriedem has joined #openstack-meeting20:45
*** trident has joined #openstack-meeting20:46
*** mriedem has quit IRC20:54
*** witek has quit IRC20:55
*** mriedem has joined #openstack-meeting20:55
*** bobh has joined #openstack-meeting21:00
timburke#startmeeting swift21:00
openstackMeeting started Wed Jul 31 21:00:13 2019 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
*** tdasilva has joined #openstack-meeting21:00
timburkewho's here for the swift meeting?21:00
*** patchbot has joined #openstack-meeting21:01
timburkeagenda hasn't changed too much21:01
timburke#topic 404s from handoffs21:02
*** openstack changes topic to "404s from handoffs (Meeting topic: swift)"21:02
timburkei mentioned last week and asked that people start to think about it and form some opinions21:02
patchbotpatch 672186 - swift - Ignore 404s from handoffs for objects when calcula... - 7 patch sets21:02
timburkerledisez helpfully suggested that i make the new behavior configurable21:03
rledisez*for few versions, to help transition21:03
timburke...but as i tried to write up an explanation for why you'd enable the new option... i'm having some doubts...21:03
timburkeregardless, i think we've got a bit of time -- clayg pointed out that it really ought to have some unit tests to cover the part of the change that only affects replicated21:04
claygoh dear, what did I do :'(21:05
timburkebut i'll try to get that done sooner rather than later21:05
*** bobh has quit IRC21:05
claygoh, right - I don't like that knob at all21:05
claygI don't see any comments on the review that explain the change, so I'm guessing there was a discussion in irc that I missed?21:06
timburkerledisez, what do we really expect operators to *do* with the option? ok, we upgraded -- crap, there's a bunch more 503s... twiddle the knob and make them 404s?21:06
claygrledisez: am I correct that the primary concern is some graph that tracks 5XX errors going up under load/failure/rebalance?21:06
rledisezI think it should basically say "the new behavior might breaks your clients because the error code changed. to keep the old behavior while you upgrade your clients, set foo = false. the old behavior will be removed in version 2.Y"21:07
rledisezthe point is not about the metrics. i just won't get my bonus this year ;)21:07
timburkewhat remediation would allow them to turn it back *off*21:07
claygbut I want rledisez to get his bonus!  he buys me beers sometimes!21:08
claygI don't think a client that was "correctly" handling a 404 with a retry would be surprised by needing to retry a 50321:08
claygmaybe there's a more subtle status code change I'm not considering with proper reference - "might break clients" is a good reason *not* to change it all, but I'm not sure I understand the risk?21:09
timburkeyeah... idk -- clients have to be able to handle 5xx, even now...21:09
rledisezI know some internal users that retry on 404 because I taught them about eventual consistency. I'm not sure what will happen on 50321:09
rledisez(not saying they are right to retry, but that's an other discussion)21:10
timburkerledisez, what sdk do they use? or is it pretty hand-rolled?21:10
rledisezmostly homemade for what I know21:11
rledisezlanguages are Go, Perl and Java21:11
timburkeyeah... that does make it tricky...21:11
notmynamerledisez: but I think we all tought the clients incorrectly! a 404 should only maybe retry. most of the time, 404 should be the right answer if it's what's given. swift was doing the wrong thing by giving 404 instead of 50321:12
timburkekota_, mattoliverau: have you had a chance to think about the issue?21:13
rlediseznotmyname: i totally agree with that21:13
notmynamebut if they have been taught to retry on 404 (and we always said retry on 5xx), and we change 404->503, aren't they even more likely to retry if there's a failure?21:13
kota_hmm... interesting21:13
notmynameso even if clients today have been retrying on 404, they will still work after the 404->503 change (because they better already be handling 5xx anyway)21:13
mattoliverauI won't lie, I forgot to look21:13
claygwell, we certainly could have been more specific when we want to say "i couldn't find your object, but also might be relevant I couldn't talk to anyone that might have been authoratative"21:13
timburkefwiw, i'd written up to try to describe the issue21:13
openstackLaunchpad bug 1837819 in OpenStack Object Storage (swift) "Overloaded object primaries cause 404s on GET" [Medium,New] - Assigned to Tim Burke (1-tim-z)21:13
claygwe want the client to retry, and a 5XX to me is a more RESTful way to indicate to *most* clients what they should do next...21:14
rledisezas I said, I'm don't think they should retry, but the most common I case I see is people doing an upload, and checking it's there right after (even if they got a 2xx). I tried to explain "trust swift", but no, they don't trust21:14
kota_IMHO, we could retry on both cases. if the users knows swift error statement absolutely retry with 503, not sure on 404.21:15
timburkerledisez, we recently had a customer doing the same thing, but looking for the object in *listings* 🤦21:15
notmynamerledisez: yeah. unfortunately I haven't been able to figure out how to fix users yet ;-)21:15
kota_imo, it's not 5xx, just 50321:15
kota_because we should not retry on 500 Internal Server Error, that wouldn't be fixed in the near future.21:16
mattoliverauThe users surely just need a sleep(10) or setting :p21:16
rledisezmaybe i'm just too cautious, because I don't think to enable that flag in my clusters. i'll do a proper communication before upgrading. but i'm thinking of somebody who would want to rollback quickly21:17
claygrledisez: yeah... if someone was already running on the edge and they didn't get enough communication before the upgrade they might be really confused/nervous about what the new status code is telling them21:17
clayg"everything was working FINE!!!" - yeah... no... it wasn't.21:17
timburkethis is highlighting for me that this probably ought to have an UpgradeImpact regardless of whether we keep the config option21:18
kota_sounds reasonable21:19
clayghonestly this change shouldn't even be as contreversal as p 667235 (in that case the proxy could really be doing an inline retry)21:20
patchbot - swift - Don't handle object without container (MERGED) - 1 patch set21:20
claygwell, I don't want the config option - but won't -2 it or anything w/o it - but it might be the ONLY CONFIG OPTION EVAR that I set a reminder to make sure we deprecate it in the next release 😉21:21
timburke...and *that* makes me wonder if maybe the guy that wrote the bug shouldn't have been the one clicking +A...21:21
timburke(well, insofar as that guy was *me*)21:21
claygtimburke: you're PTL you can +A whenver you want ;)21:22
notmynamelol, not it!21:23
timburkewe've got some precedent for adding known-terrible-idea config options:
timburkei'll keep thinking on it, but i'm kinda leaning toward clayg's position personally21:24
timburkelet's keep moving21:26
timburke#topic py321:26
*** openstack changes topic to "py3 (Meeting topic: swift)"21:26
kota_config opt_out/in might be terrible21:26
timburkenot much to report21:26
kota_i'm imagine s3acl...21:26
mattoliverauI need to look closer at it, and sorry I didn't. Like kota_ says, 503 should be retry. But its the contract changing that is the impact. And one of our dev and Ops wants an escape clause, for a release, I think I'm ok with that.21:26
mattoliverauanyway, move on :)21:26
timburkeugh, yeah... s3_acl...21:27
timburke landed! hopefully zaitcev's cluster will be happier now :-)21:27
patchbotpatch 672610 - swift - py3: fix non-ascii metadata handling in account-se... (MERGED) - 2 patch sets21:27
timburkeas did -- there may be a bit of a long tail of patches like that :-/21:28
patchbotpatch 672803 - swift - py3: Fix title-casing in HeaderKeyDict (MERGED) - 3 patch sets21:28
timburkeit'll be nice to get so i can run probe tests locally again :-P21:30
patchbotpatch 671333 - swift - py3: (mostly) port probe tests - 2 patch sets21:30
timburkethat's about it21:30
timburke#topic lots of small files21:30
*** openstack changes topic to "lots of small files (Meeting topic: swift)"21:30
timburkerledisez, kota_ i haven't seen too much lately on the branch (but that's ok)21:31
kota_ya, sorry21:31
timburkelike i said, that's ok! no need to apologize21:31
claygtdasilva: is going to be all up in the losf pretty soon!21:31
rledisezyeah. alecuyer is off so don't expect a lot from OVH for few weeks21:31
timburkei know people here at swiftstack have been getting increasingly interested -- i think tdasilva has been taking a look recently?21:31
*** zaitcev has joined #openstack-meeting21:32
kota_nice, tdasilva!!!21:32
timburkefwiw, i feel like this ought to be the next big item of work that we're all focused on21:32
*** tdasilva has quit IRC21:33
kota_oh, tdasilva has left.21:33
timburkebah. i was just about to ask him if there was anything he'd like to bring up about it... oh well21:34
timburke#topic sharding21:34
*** openstack changes topic to "sharding (Meeting topic: swift)"21:34
timburkemattoliverau, i know you've got a few patches up now -- anything we ought to be doing besides reviewing them?21:35
mattoliverauNah just reviewing them and point out the obvious flaws and edge cases :)21:35
timburke#topic symlinks and versioning21:36
*** openstack changes topic to "symlinks and versioning (Meeting topic: swift)"21:36
mattoliverauI last patch is a complete POC were we send the ranges from the scanner via UPDATE to do a reverse rollback stratergy. Not sure I like it, but was an idea I had.21:36
mattoliverauthat's all21:36
claygon the hardlinks I was looking at some comments this morning, and adding some more tests for behaviors that we want to better specify - I think we're still not 100% clear on how hardlinks to manifests/symlinks should look21:37
claygI think timburke had the most experience/insight - so i'm hoping to get his feedback on some of the new tests I drafted21:37
*** e0ne has joined #openstack-meeting21:37
timburkei'll be sure to take a look :-)21:38
timburkeany other blockers for you, or places that you need more input?21:38
*** zaitcev has quit IRC21:39
claygi also managed to get up the s3 versioning patch at the end of the chain, p 67368221:39
patchbot - swift - s3api: Implement versioning status API - 1 patch set21:39
claygone thing that starts to shine through on that one is how much it just assumes it knows how versioning works and does what it needs to implement the aws api21:39
claygI think in a more perfect world we'd have looked at the s3 versioning features and added them to versioned writes - then s3api is just doing *translation* from aws apis to swift apis21:40
*** takamatsu has joined #openstack-meeting21:40
claygbut since we don't have spellings for ... e.g. "copy version_id XYZ"  we just "do it" in s3api21:40
claygbut I don't think moving forward with something that works really prevents us in any meaningful way from doing that work later (except that maybe we'd have less motivation to do so)21:41
claygOTOH, I'm not sure much moving to symlink versionsing is really going to throw off clients that sort of had to learn how stack & history versoining worked already so they could do things like "restore version X" or "delete version Y"21:42
clayghad we had an API for that all along it would make less of a difference to clients when we decide to change the underlying implementation21:43
timburkemakes sense21:43
clayganyway, is what is... something I'll be a little more aware of as we flesh more of the s3api matrix down the road...21:44
claygI don't think i'm blocked right now and I feel like i'm making progress21:44
claygall feedback is appreciated!21:44
timburke#topic shanghai21:45
*** openstack changes topic to "shanghai (Meeting topic: swift)"21:45
timburkei know i dropped it from the agenda, but i've been looking at the order in which i'll have to do things and i thought i ought to share21:45
timburkekeeping in mind that it's a bit US-centric, but may be more-or-less applicable for you, too mattoliverau, kota_, and rledisez21:46
kota_something like VISA?21:46
mattoliverauits probably pretty similar I suspect. Ie get a letter, get visa, etc.21:47
timburkelooks like to get the visa (, $140) i need the invitation letter (, and to get *that* i need to register (, $161 after the contributor discount)21:49
timburkeas far as i can tell, airfare and hotel can be done at any point along there21:50
*** zaitcev has joined #openstack-meeting21:50
kota_that registration with early bird will close around... 14 (maybe?) Aug.21:50
rledisezkota_: I was looking for that information earlier. do you have a link?21:50
kota_I'm not sure the contribution discount will keep the $16121:51
timburke says "Summit registration is open - get your tickets before prices increase on August 14 at 11:59pm PT! "21:51
kota_same URL21:51
rledisezperfect. thx. how did i miss that :)21:51
kota_no information about the standard price after early bird.21:51
timburkethat's about it. i did send something to the mailing list to call out our etherpad ( -- it was a bit behind when most projects did theirs21:53
timburkewe'll see who else puts their name on :-D21:54
timburkethat's all i've got21:54
timburke#topic open discussion21:54
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:55
claygi gotta bounce, ya'll be good21:55
timburkeanyone have anything eles to bring up in the last five minutes?21:55
timburkeall right. thank you all for coming, and thank you for working on swift!21:57
*** openstack changes topic to "OpenStack Meetings ||"21:57
openstackMeeting ended Wed Jul 31 21:57:08 2019 UTC.  Information about MeetBot at . (v 0.1.4)21:57
openstackMinutes (text):
*** njohnston has joined #openstack-meeting21:59
*** raildo has quit IRC22:00
*** zaitcev has left #openstack-meeting22:02
*** slaweq has quit IRC22:04
*** rcernin has joined #openstack-meeting22:05
*** carloss has quit IRC22:19
*** patchbot has quit IRC22:36
*** mriedem has quit IRC22:43
*** panda has quit IRC23:03
*** panda has joined #openstack-meeting23:04
*** notmyname has quit IRC23:05
*** njohnston has quit IRC23:09
*** slaweq has joined #openstack-meeting23:11
*** slaweq has quit IRC23:15
*** jamesmcarthur has joined #openstack-meeting23:22
*** e0ne has quit IRC23:33
*** eharney has quit IRC23:45
*** notmyname has joined #openstack-meeting23:52
*** notmyname has quit IRC23:53
*** notmyname has joined #openstack-meeting23:53
*** takamatsu has quit IRC23:59

Generated by 2.15.3 by Marius Gedminas - find it at!