Wednesday, 2019-10-09

*** markvoelker has quit IRC00:05
*** markvoelker has joined #openstack-meeting00:07
*** mattw4 has quit IRC00:07
*** markvoelker has quit IRC00:14
*** markvoelker has joined #openstack-meeting00:19
*** diablo_rojo has joined #openstack-meeting00:37
*** bbowen has quit IRC00:39
*** bbowen has joined #openstack-meeting00:39
*** njohnston_ has joined #openstack-meeting01:09
*** njohnsto_ has joined #openstack-meeting01:10
*** njohnston_ has quit IRC01:14
*** njohnsto_ has quit IRC01:18
*** ayoung has quit IRC01:19
*** ayoung has joined #openstack-meeting01:20
*** igordc has quit IRC01:24
*** davee___ has joined #openstack-meeting01:26
*** davee__ has quit IRC01:26
*** rbudden has quit IRC01:49
*** rbudden has joined #openstack-meeting01:50
*** rbudden has joined #openstack-meeting01:51
*** rbudden has joined #openstack-meeting01:52
*** brinzhang has joined #openstack-meeting01:53
*** brinzhang_ has joined #openstack-meeting01:57
*** brinzhang has quit IRC02:00
*** macz has joined #openstack-meeting02:02
*** whoami-rajat has joined #openstack-meeting02:07
*** macz has quit IRC02:07
*** brinzhang has joined #openstack-meeting02:09
*** apetrich has quit IRC02:10
*** psachin has joined #openstack-meeting02:11
*** brinzhang_ has quit IRC02:12
*** ayoung has quit IRC02:19
*** ayoung has joined #openstack-meeting02:19
*** ricolin has joined #openstack-meeting02:21
*** brinzhang_ has joined #openstack-meeting02:39
*** brinzhang has quit IRC02:42
*** brinzhang has joined #openstack-meeting02:42
*** brinzhang_ has quit IRC02:43
*** brinzhang has quit IRC02:44
*** brinzhang has joined #openstack-meeting02:45
*** brinzhang_ has joined #openstack-meeting02:52
*** brinzhang has quit IRC02:55
*** markvoelker has quit IRC03:09
*** macz has joined #openstack-meeting03:09
*** markvoelker has joined #openstack-meeting03:10
*** macz has quit IRC03:14
*** psachin has quit IRC03:17
*** psachin has joined #openstack-meeting03:18
*** markvoelker has quit IRC03:23
*** markvoelker has joined #openstack-meeting03:24
*** dviroel has quit IRC03:25
*** diablo_rojo has quit IRC03:40
*** yaawang has quit IRC04:12
*** yaawang has joined #openstack-meeting04:12
*** ociuhandu has joined #openstack-meeting04:16
*** ociuhandu has quit IRC04:20
*** markvoelker has quit IRC04:35
*** markvoelker has joined #openstack-meeting04:35
*** brinzhang has joined #openstack-meeting04:48
*** brinzhang_ has quit IRC04:51
*** luyao has quit IRC04:52
*** ayoung has quit IRC04:55
*** ayoung has joined #openstack-meeting04:55
*** brinzhang_ has joined #openstack-meeting05:00
*** brinzhang has quit IRC05:03
*** ayoung has quit IRC05:12
*** ayoung has joined #openstack-meeting05:13
*** boxiang has joined #openstack-meeting05:15
*** zhubx has quit IRC05:16
*** yamamoto has quit IRC05:25
*** yamamoto has joined #openstack-meeting05:30
*** brinzhang has joined #openstack-meeting05:38
*** brinzhang_ has quit IRC05:40
*** links has joined #openstack-meeting05:52
*** kopecmartin|off is now known as kopecmartin06:07
*** slaweq has joined #openstack-meeting06:08
*** jbadiapa has joined #openstack-meeting06:29
*** brinzhang_ has joined #openstack-meeting07:00
*** brinzhang has quit IRC07:03
*** pcaruana has joined #openstack-meeting07:10
*** tesseract has joined #openstack-meeting07:13
*** verdurin has quit IRC07:16
*** verdurin has joined #openstack-meeting07:18
*** apetrich has joined #openstack-meeting07:24
*** ayoung has quit IRC07:24
*** ayoung has joined #openstack-meeting07:25
*** yamamoto has quit IRC07:30
*** tssurya has joined #openstack-meeting07:32
*** brinzhang_ has quit IRC07:32
*** brinzhang_ has joined #openstack-meeting07:33
*** yamamoto has joined #openstack-meeting07:33
*** yamamoto has quit IRC07:34
*** yamamoto has joined #openstack-meeting07:34
*** rpittau|afk is now known as rpittau07:35
*** slaweq has quit IRC07:42
*** slaweq has joined #openstack-meeting07:49
*** ayoung has quit IRC07:52
*** ayoung has joined #openstack-meeting07:54
*** ralonsoh has joined #openstack-meeting07:58
*** brinzhang has joined #openstack-meeting08:01
*** brinzhang has quit IRC08:01
*** brinzhang has joined #openstack-meeting08:01
*** brinzhang_ has quit IRC08:03
*** yaawang has quit IRC08:08
*** yaawang has joined #openstack-meeting08:08
*** ociuhandu has joined #openstack-meeting08:20
*** ociuhandu has quit IRC08:20
*** brinzhang_ has joined #openstack-meeting08:20
*** ociuhandu has joined #openstack-meeting08:21
*** ociuhandu has quit IRC08:23
*** ociuhandu has joined #openstack-meeting08:23
*** brinzhang has quit IRC08:23
*** ociuhandu has quit IRC08:34
*** e0ne has joined #openstack-meeting08:39
*** boxiang has quit IRC08:47
*** boxiang has joined #openstack-meeting08:47
*** epoojad1 has joined #openstack-meeting08:50
*** brinzhang has joined #openstack-meeting09:02
*** ayoung has quit IRC09:04
*** brinzhang_ has quit IRC09:05
*** ayoung has joined #openstack-meeting09:07
*** ttsiouts has joined #openstack-meeting09:13
*** brinzhang_ has joined #openstack-meeting09:15
*** brinzhang has quit IRC09:19
*** brinzhang has joined #openstack-meeting09:19
*** ociuhandu has joined #openstack-meeting09:23
*** priteau has joined #openstack-meeting09:26
*** yamamoto has quit IRC09:30
*** brinzhang has quit IRC09:33
*** brinzhang has joined #openstack-meeting09:33
*** brinzhang has quit IRC09:39
*** brinzhang has joined #openstack-meeting09:39
*** moguimar has quit IRC09:54
*** moguimar has joined #openstack-meeting09:56
*** brinzhang has quit IRC10:06
*** yamamoto has joined #openstack-meeting10:07
*** brinzhang has joined #openstack-meeting10:07
*** ttsiouts has quit IRC10:11
*** boxiang has quit IRC10:12
*** boxiang has joined #openstack-meeting10:12
*** jamesmcarthur has joined #openstack-meeting10:15
*** jamesmcarthur has quit IRC10:19
*** ociuhandu has quit IRC10:23
*** ociuhandu has joined #openstack-meeting10:25
*** ykatabam has quit IRC10:31
*** brinzhang has quit IRC10:33
*** brinzhang has joined #openstack-meeting10:34
*** ayoung has quit IRC10:38
*** lpetrut has joined #openstack-meeting10:39
*** ayoung has joined #openstack-meeting10:39
*** tesseract has quit IRC10:44
*** tesseract has joined #openstack-meeting10:46
*** rcernin has quit IRC10:48
*** yamamoto has quit IRC10:50
*** dviroel_ has joined #openstack-meeting10:50
*** dh3 has joined #openstack-meeting10:53
*** brinzhang has quit IRC10:57
*** brinzhang_ has quit IRC10:57
*** brinzhang_ has joined #openstack-meeting10:58
*** brinzhang has joined #openstack-meeting10:58
*** yamamoto has joined #openstack-meeting10:59
*** oneswig has joined #openstack-meeting10:59
*** yamamoto has quit IRC10:59
*** yamamoto has joined #openstack-meeting11:00
oneswig#startmeeting scientific-sig11:00
openstackMeeting started Wed Oct  9 11:00:32 2019 UTC and is due to finish in 60 minutes.  The chair is oneswig. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.11:00
*** openstack changes topic to " (Meeting topic: scientific-sig)"11:00
openstackThe meeting name has been set to 'scientific_sig'11:00
oneswig#link agenda for today
oneswig(such as it is)11:01
oneswigHi dh3, how's things?11:02
dh3busy :)  no change there!11:02
oneswigSaw an interesting development this morning11:02
oneswig#link SUSE drops their OpenStack product
*** janders has joined #openstack-meeting11:03
dh3Suse has always been a bit niche (IME) and OpenStack is niche so maybe they're only dropping a tiny bit of business11:03
oneswigHi verdurin, afternoon11:03
janderssorry for being late11:04
jandersSuse dropping OpenStack?11:04
oneswigdh3: does seem like it.  I'm mostly worried about where it leaves their core contributors11:04
*** yamamoto has quit IRC11:04
dh3mmm that's a point11:04
oneswigjanders: yes it seems like it11:04
oneswigI wonder what alternatives they'll be transitioning customers to, doesn't say in the post...11:05
jandersgood question however I have to say I dont think I've ever met anyone running SuSE Cloud...11:06
oneswigactually, not sure I have either.11:07
oneswigThey do appear to have taken a wrong turn in deployment tooling in recent years, not sure what they ended up with.11:07
jandersI did consider them for a couple projects to be fair, but the tooling was always like.... wait.... what?!?11:08
jandersactive passive databases, chef config management etc11:08
oneswigWell, interesting times.11:09
jandersit does look like OpenStack is past the top of the curve11:09
jandersthose who are using it right are having a blast11:09
jandersthe others realise maybe it's not the way11:09
oneswigjanders: there's certainly a lot less hype and more practical usage it seems.11:10
jandersshame likes of IBM seem to be slowing down development of things such as gpfs-openstack integration etc11:10
jandersgood they wrote the container friendly version before that happened11:10
jandersits a shame cause it's a killer11:10
janderswith this I've got faster storage in VMs than quite a few supercomputers11:11
*** ociuhandu has quit IRC11:11
jandersand that's without baremetal/sriov11:12
oneswigI wonder what will come of work like this for secure computing frameworks:
jandersyet people are still undecided whether to develop it further... :/11:12
dh3we've had users stand up gpfs on openstack instances (without needing sysadmin help!) as part of a k8s layer, that must say something11:12
oneswigdh3: that's impressive and surprising11:12
oneswigmight also be a statement on your users11:13
oneswigIs your k8s an OpenShift deployment or do you roll your own?11:13
dh3some of them do jump in with all 3 feet :)11:14
jandersyeah that is the justification I hear from IBM when they say OpenStack integration will not be developed further. Diverting resources to k8s.11:14
dh3our k8s is DIY at the moment but we are pushing towards Rancher to get the nice UI layer11:14
oneswigdh3: kubespray or more fundamentally DIY?11:14
*** dh3 has quit IRC11:15
oneswighmm seems we've lost dh311:15
jandersGPFS is quite flexible - can be relatively simple or quite complex11:15
*** ociuhandu has joined #openstack-meeting11:15
jandersour first attempt of deploying GPFS-EC ended up destroying all the OSes it was supposed to run on11:16
*** ociuhandu has quit IRC11:16
jandersthe magic script was filtering out the software raid but not the member drives - and formatted them :D11:16
*** dh3 has joined #openstack-meeting11:16
*** ociuhandu has joined #openstack-meeting11:16
dh3(dunno what happened there)11:16
dh3AFAIK people are building on kubespray11:16
jandersprobably we were the first site to deploy with swraid thats why11:16
oneswigjanders: oops...11:17
jandersgot the hotfix overnight and the second deploy was all good11:17
jandersnow it's doing 120/50 GB/s read/write on six servers and six clients11:18
jandersgpfs backed cinder is getting close to 20GB/s in VMs11:19
oneswigjanders: are you using SR-IOV or ASAP2 for storage I/O to VMs?11:19
oneswigor is it all block?11:19
jandersthat's the best part11:19
jandershypervisors connect to GPFS over HDR100/HDR200 (depending which ones)11:19
dh3janders: do you have any write ups, blog posts, etc?11:20
jandersVM networking is stock standard11:20
jandersno - but happy to chat if you're interested11:20
dh3potentially yes. we haven't used gpfs (on the "systems supported" side) for years. but always happy to look around. I'll drop you an email, thanks11:21
janderswe could make it quite a bit faster especially on the write side but we traded that off for redundancy11:21
*** yamamoto has joined #openstack-meeting11:22
janderslosing a server doesn't hurt it, could probably run without two but that's not a requirement so haven't tested that scenario11:22
jandersessentially it's ceph architecture with HPC filesystem performance11:23
jandersand minimal changes to the OpenStack11:23
*** yamamoto has quit IRC11:23
oneswigjanders: have you seen roadmaps for that?11:23
jandersfor what exactly?11:23
oneswigOngoing development for GPFS+OpenStack11:24
*** tssurya has quit IRC11:24
jandersI'm being told they pulled resources from this and only maintain it but do not develop it11:24
oneswigAre there constraints on the version you can run?11:24
jandersI've got it going with RH-OSP1311:25
jandersbut I think it would run with latest, too11:25
jandersI currently have cinder and glance integrated, I tested nova, too and it worked fine11:26
dh3do you mount gpfs as (say) /var/lib/nova/instances on the hypervisor then let everything run as normal?11:27
jandersfor nova, yes11:27
jandersfor cinder, no11:27
jandersthough with nova I only turned it on for testing, I have 1.5TB of NVMe in each compute node11:29
jandersGPFS can do 70k IOPS per client, that NVMe can do 10x that11:29
jandersso right now it's cinder for capacity/throughput and ephemeral for IOPS11:30
*** raildo has joined #openstack-meeting11:30
oneswigA good balance of options11:30
verdurinjanders: I'll contact you about this too, if I may11:30
oneswig(for the user that knows how to exploit that)11:30
janderssure - no worries, happy to chat more11:31
dh3similar to us but the compute is SSD not NVMe11:31
*** ttsiouts has joined #openstack-meeting11:32
jandersit is a very interesting direction cause there's a lot of push for performance hence walking away from VMs to containers and baremetal11:32
oneswigjanders: what kind of workloads are you supporting with this cloud?11:32
jandersand with something like that if it's storage performance they want, VMs suddenly become good enough again11:32
jandersit's for the cybersecurity research system. The workloads are still being identified/decided.11:33
jandersGPFS was designed to stand up to ML workloads that were killing our older HPC storage11:33
jandersit's essentially a smaller more efficiently balanced version of our BeeGFS design11:33
*** yamamoto has joined #openstack-meeting11:33
jandersif I had PCIe4 these could do 40GB/s read per node11:33
*** yamamoto has quit IRC11:33
jandersbut unfortunately I don't11:34
*** yamamoto has joined #openstack-meeting11:34
*** ykatabam has joined #openstack-meeting11:35
jandersdo you guys have any experience tuning GPFS clients for good IOPS numbers?11:35
*** janders has quit IRC11:36
*** davidsha has joined #openstack-meeting11:36
oneswigsorry, not here jamespage11:36
oneswigjanders I mean - lost again?11:36
*** janders has joined #openstack-meeting11:38
*** yamamoto has quit IRC11:38
oneswigwelcome back janders :-)11:39
jandersthanks! :)  network hiccups11:39
verdurinjanders: our cluster nodes are all GPFS but we haven't needed to do any advanced tuning with them11:39
jandersdo you remember what IOPS are you getting on clients?11:39
verdurinNot offhand, but I can find out.11:40
jandersthat would be quite interesting11:40
janderswhat's the storage backend on your cluster?11:40
verdurinThis is mainly EDR to spinning disk.11:40
verdurinSome FDR.11:41
jandersJBODs or array based?11:41
verdurinDDN arrays.11:41
verdurinLatest iteration will have a small pool of SSD.11:42
janderswhat's the capacity?11:42
verdurin~7PB usable.11:43
jandersours is ~250TB11:44
verdurinHence we're not desperately keen on capacity-based licensing...11:44
jandersbut all-NVME11:44
jandersthen EC is not a good idea - cheaper to buy more kit11:45
verdurinoneswig: your earlier point about contributors is an important one. Lots of people stepping down from PTL-type roles of late, I see.11:45
*** baojg has quit IRC11:46
oneswigYes that has been a trend.11:46
*** ykatabam has quit IRC11:47
oneswigWe've been looking recently at the condition of Tungsten Fabric integration with Kolla-Ansible.  It's pretty good, in that it's less than a year behind, but doesn't appear to be advancing beyond that point.  I'm still investigating.11:48
oneswigIt appears Tungsten has some invasive requirements for installing widgets in the containers of other services.11:49
*** apetrich has quit IRC11:50
oneswigjanders: in a vague attempt to follow the agenda, there was a question on the SIG Slack about usage accounting.  What do you use for this, if anything?11:50
jandersnothing at the moment11:51
jandersour User Services guys interview users to identify how much resources they really need (as opposed to what they think they need or they would look like) and set the quotas accordingly11:51
jandersfrom there it's assumed it's a solved problem11:51
jandersnot very accurate but kinda works for now11:52
jandersbetter than giving users what they ask for on a shared system I suppose11:52
jandershope to have a better answer few months down the track :)11:53
*** ileixe_ has joined #openstack-meeting11:53
oneswigThanks janders, good to know11:53
jandersgiven the User Services guys are nice to us we are nice to them and give them a simple yaml interface for managing projects, memberships and quotas11:54
jandersfrom there ansible sets it all, they don't need to know OpenStack commands11:54
dh3"simple" + "yaml"... :/  our service desk get to set quotas using Cloudforms (it was only marginally quicker to set up than writing our own interface)11:56
oneswigNearly at time - anything for AOB?11:57
verdurinoneswig: billing (or rather costing) keeps coming up for us, and we've relied on rough heuristics for now.11:57
*** brinzhang has quit IRC11:57
*** janders25 has joined #openstack-meeting11:58
oneswigverdurin: noted.  At this end, I'm hoping for priteau to update the study he did on CloudKitty earlier this summer.11:58
*** brinzhang has joined #openstack-meeting11:58
janders25################################  sample-five:    name: sample-five    enabled: true    description: asdf    quota:      instances: 3      cores: 20      ram: 25600      volumes: 6      gigabytes: 5      snapshots: 3      floating_ips: 8    members:      - user123    networking:      create_default: true      create_router:11:58
*** epoojad1 has quit IRC11:59
janders25oops formatting died but I wanted to show that yaml can be simple to work with11:59
janders25my networking is really bad today too11:59
dh3I know that but some people are CLI-avoidant!11:59
priteaujanders25: If they were not nice to you, the interface would be XML?12:00
dh3(hard enough getting them to edit the LSF users file)12:00
janders25true! :)12:00
oneswigAnd for APEL users, at some point we hope to complete the loop with data submission from OpenStack infrastructure12:00
*** brinzhang has quit IRC12:00
oneswighi priteau - ears burning :-) ?12:00
oneswigAh, we are out of time12:00
janders25priteau: yes! xml.... you nailed it, I was editing pacemaker configs today... argh!12:00
*** brinzhang has joined #openstack-meeting12:00
oneswigxml gives a vintage feel nowadays12:00
*** openstack changes topic to "OpenStack Meetings ||"12:01
janders25thank you all12:01
openstackMeeting ended Wed Oct  9 12:00:59 2019 UTC.  Information about MeetBot at . (v 0.1.4)12:01
oneswigThanks all12:01
openstackMinutes (text):
dh3thanks, ttfn12:01
*** janders25 has quit IRC12:01
*** oneswig has quit IRC12:01
*** janders has quit IRC12:01
*** apetrich has joined #openstack-meeting12:03
*** dh3 has quit IRC12:04
*** yamamoto has joined #openstack-meeting12:07
*** moguimar has quit IRC12:11
*** ayoung has quit IRC12:12
*** ayoung has joined #openstack-meeting12:15
*** ociuhandu has quit IRC12:15
*** ociuhandu has joined #openstack-meeting12:15
*** apetrich has quit IRC12:18
*** ociuhandu has quit IRC12:20
*** ttsiouts has quit IRC12:24
*** ttsiouts has joined #openstack-meeting12:25
*** ttsiouts_ has joined #openstack-meeting12:28
*** ttsiouts has quit IRC12:28
*** psachin has quit IRC12:31
*** ociuhandu has joined #openstack-meeting12:33
*** ayoung has quit IRC12:38
*** bcafarel has quit IRC12:39
*** bcafarel has joined #openstack-meeting12:40
*** ayoung has joined #openstack-meeting12:41
*** mriedem has joined #openstack-meeting12:44
*** brinzhang has quit IRC12:45
*** brinzhang has joined #openstack-meeting12:46
*** brinzhang_ has quit IRC12:47
*** brinzhang_ has joined #openstack-meeting12:47
*** brinzhang_ has quit IRC12:48
*** brinzhang_ has joined #openstack-meeting12:48
*** brinzhang_ has quit IRC12:50
*** brinzhang_ has joined #openstack-meeting12:50
*** yamamoto has quit IRC12:51
*** brinzhang_ has quit IRC12:52
*** brinzhang_ has joined #openstack-meeting12:52
*** yamamoto has joined #openstack-meeting12:53
*** yamamoto has quit IRC12:53
*** yamamoto has joined #openstack-meeting12:53
*** brinzhang_ has quit IRC12:54
*** brinzhang_ has joined #openstack-meeting12:54
*** ociuhandu has quit IRC12:54
*** ociuhandu has joined #openstack-meeting12:55
*** ociuhandu has quit IRC12:56
*** ociuhandu has joined #openstack-meeting12:56
*** yamamoto has quit IRC12:57
*** dviroel_ is now known as dviroel12:58
*** eharney has joined #openstack-meeting13:01
*** apetrich has joined #openstack-meeting13:05
*** slaweq has quit IRC13:07
*** enriquetaso has joined #openstack-meeting13:09
*** slaweq has joined #openstack-meeting13:10
*** markvoelker has quit IRC13:10
*** markvoelker has joined #openstack-meeting13:11
*** moguimar has joined #openstack-meeting13:11
*** markvoelker has quit IRC13:11
*** markvoelker has joined #openstack-meeting13:11
*** ayoung has quit IRC13:12
*** slaweq_ has joined #openstack-meeting13:13
*** ayoung has joined #openstack-meeting13:14
*** slaweq has quit IRC13:14
*** brinzhang has quit IRC13:17
*** brinzhang has joined #openstack-meeting13:18
*** brinzhang has quit IRC13:19
*** brinzhang has joined #openstack-meeting13:20
*** brinzhang has quit IRC13:20
*** AlanClark has joined #openstack-meeting13:21
*** brinzhang has joined #openstack-meeting13:21
*** lseki has joined #openstack-meeting13:25
*** ayoung has quit IRC13:26
*** brinzhang has quit IRC13:26
*** brinzhang has joined #openstack-meeting13:27
*** brinzhang_ has quit IRC13:27
*** brinzhang_ has joined #openstack-meeting13:27
*** ayoung has joined #openstack-meeting13:27
*** jawad_axd has joined #openstack-meeting13:28
*** bnemec has quit IRC13:29
*** slaweq_ is now known as slaweq13:29
*** bnemec has joined #openstack-meeting13:31
*** pcaruana has quit IRC13:31
*** pcaruana has joined #openstack-meeting13:33
*** moguimar has joined #openstack-meeting13:40
*** ianychoi has quit IRC13:41
*** rbudden has joined #openstack-meeting13:47
*** rf0lc0 has joined #openstack-meeting13:51
*** AlanClark has quit IRC13:51
*** jawad_axd has quit IRC13:51
*** jawad_axd has joined #openstack-meeting13:52
*** klindgren_ has joined #openstack-meeting13:52
*** brinzhang has quit IRC13:54
*** mahatic has quit IRC13:56
*** tonyb has quit IRC13:56
*** johanssone has quit IRC13:56
*** brinzhang_ has quit IRC13:56
*** rfolco has quit IRC13:56
*** gibi has quit IRC13:56
*** SotK has quit IRC13:56
*** rbudden has quit IRC13:56
*** apetrich has quit IRC13:56
*** klindgren has quit IRC13:56
*** mjturek has quit IRC13:56
*** brinzhang has joined #openstack-meeting13:56
*** brinzhang_ has joined #openstack-meeting13:56
*** SotK has joined #openstack-meeting13:56
*** mjturek has joined #openstack-meeting13:56
*** brinzhang has quit IRC13:56
*** gibi has joined #openstack-meeting13:56
*** brinzhang has joined #openstack-meeting13:56
*** jawad_axd has quit IRC13:57
*** brinzhang has quit IRC13:57
*** johanssone has joined #openstack-meeting13:58
*** brinzhang has joined #openstack-meeting13:58
*** ociuhandu has quit IRC13:59
*** apetrich has joined #openstack-meeting14:00
*** rf0lc0 is now known as rfolco14:02
*** davee__ has joined #openstack-meeting14:05
*** davee___ has quit IRC14:05
*** slaweq has quit IRC14:07
*** slaweq has joined #openstack-meeting14:08
*** lpetrut has quit IRC14:09
*** abishop_ has joined #openstack-meeting14:16
*** abishop has quit IRC14:17
*** apetrich has quit IRC14:18
*** jamesmcarthur has joined #openstack-meeting14:19
*** asbishop has joined #openstack-meeting14:22
*** abishop_ has quit IRC14:24
*** apetrich has joined #openstack-meeting14:27
*** ileixe_ has quit IRC14:28
*** asbishop has quit IRC14:30
*** ayoung has quit IRC14:36
*** ayoung has joined #openstack-meeting14:38
*** artom has joined #openstack-meeting14:42
*** ayoung has quit IRC14:49
*** ayoung has joined #openstack-meeting14:51
*** jawad_axd has joined #openstack-meeting14:53
*** macz has joined #openstack-meeting15:02
*** yamamoto has joined #openstack-meeting15:02
*** tosky has joined #openstack-meeting15:03
*** diablo_rojo has joined #openstack-meeting15:06
*** ayoung has quit IRC15:12
*** njohnston_ has joined #openstack-meeting15:13
*** ayoung has joined #openstack-meeting15:13
*** yamamoto has quit IRC15:14
*** links has quit IRC15:15
*** njohnston_ has quit IRC15:16
*** ociuhandu has joined #openstack-meeting15:19
*** lpetrut has joined #openstack-meeting15:21
*** ociuhandu has quit IRC15:23
*** slaweq has quit IRC15:26
*** slaweq has joined #openstack-meeting15:27
*** slaweq has quit IRC15:28
*** ociuhandu has joined #openstack-meeting15:30
*** ttsiouts_ has quit IRC15:33
*** ttsiouts has joined #openstack-meeting15:33
*** slaweq has joined #openstack-meeting15:35
*** lpetrut has quit IRC15:35
*** ttsiouts has quit IRC15:37
*** brinzhang has quit IRC15:42
*** brinzhang_ has quit IRC15:42
*** brinzhang has joined #openstack-meeting15:43
*** brinzhang_ has joined #openstack-meeting15:43
*** slaweq has quit IRC15:44
*** brinzhang_ has quit IRC15:44
*** abishop has joined #openstack-meeting15:44
*** ircuser-1 has joined #openstack-meeting15:44
*** brinzhang_ has joined #openstack-meeting15:45
*** rpittau is now known as rpittau|afk15:45
*** rosmaita has joined #openstack-meeting15:52
*** toabctl has quit IRC15:53
*** woojay has joined #openstack-meeting15:57
jungleboyj#startmeeting Cinder16:00
openstackMeeting started Wed Oct  9 16:00:20 2019 UTC and is due to finish in 60 minutes.  The chair is jungleboyj. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
*** openstack changes topic to " (Meeting topic: Cinder)"16:00
openstackThe meeting name has been set to 'cinder'16:00
jungleboyjCourtesy ping:  jungleboyj whoami-rajat rajinir lseki carloss pots woojay erlon geguileo eharney rosmaita enriquetaso e0ne smcginnis davidsha walshh_ xyang hemna _hemna tosky sfernand16:00
geguileohi! o/16:00
*** thgcorrea has joined #openstack-meeting16:01
*** ayoung has quit IRC16:01
*** andrebeltrami has joined #openstack-meeting16:02
_pewp_jungleboyj ( *՞ਊ՞*)ノ16:02
jungleboyjLook at all the people.16:02
*** ayoung has joined #openstack-meeting16:02
rosmaitabig turnout16:02
jungleboyjFigures given there is a light agenda.  :-)16:03
jungleboyjAnyway, lets get started.16:03
jungleboyj#topic announcements16:04
*** openstack changes topic to "announcements (Meeting topic: Cinder)"16:04
jungleboyjLooks like the announcements are yours rosmaita ....16:04
rosmaitafirst, RC-2 is available -- please test!16:04
jungleboyjrosmaita: ++16:04
rosmaitathe "final RC" is whatever has been released on Friday (11 October)16:05
rosmaitaso if we have any critical bug fixes, we need to find, fix, and merge by Friday16:05
smcginnisWhich day?16:05
rosmaitaotherwise, if there's anything found after that, it's up to the release team (not us) whether it can be included in the release16:05
jungleboyjsmcginnis:  Oh no he didn't!16:06
smcginnis:D :D16:06
jungleboyjrosmaita:  Save yourself from the torture I got in the past.16:06
* jungleboyj whispers just say Thursday16:06
rosmaitaoh, ok, Thursday (10 Oct)16:07
smcginnisGood save. :P16:07
jungleboyjsmcginnis:  He is smarter than I am.16:07
rosmaitaanyway, i don't see any bugs filed or patches proposed for stable/train, and we are running out of time, so i think RC-2 will be our final RC16:07
jungleboyjGood.  Nice that we could get it done in two RCs.  Thanks for the great leadership there rosmaita16:08
rosmaitaother announcement is the running reminder to please include topics on the PTG planning etherpad16:08
jungleboyjrosmaita:  ++  If we are going all the way to Shanghai it should be productive.16:08
rosmaitaand i guess final announcement is Train coordinated release is one week from today!16:08
*** ayoung has quit IRC16:09
jungleboyjWhooo whooo, chugga chugga ...16:09
rosmaitathat's all from me16:09
jungleboyjrosmaita:  Awesome.  Thank you.16:09
*** ayoung has joined #openstack-meeting16:10
jungleboyjI only had one other topic for this week.16:10
jungleboyj#topic Discuss Matt's proposal to remove legacy attach code16:10
*** openstack changes topic to "Discuss Matt's proposal to remove legacy attach code (Meeting topic: Cinder)"16:10
jungleboyjIt looks like the last note in the thread is here:16:11
e0necan we really drop APIs with microversions and before cinder API v4?16:11
jungleboyjWant to make sure that we get the people who understand this code involved.16:11
*** igordc has joined #openstack-meeting16:12
smcginnisYeah, I think we had realized we can't ever really drop code.16:12
jungleboyje0ne:  Good question.16:12
smcginnisEven with a new microversion that excludes it, we still need the code there for the lower MVs.16:12
geguileowe don't drop the API code, they just stop using it16:13
smcginnisNova can choose not to call the old code anymore of course.16:13
e0nesmcginnis: +116:13
geguileoright now they may call both of them from the same host16:13
geguileodepending on the instance and how/when the volumes where attached16:13
jungleboyjBut this isn't a question of removing it from Cinder, it is removing it from Nova.  Right?16:13
smcginnisWe should figure out a good way to track things we do want to drop in case we ever do decide to do a v4 release and consolidate all mv changes into a new base version.16:14
geguileobut to remove it from 'nova they need to migrate the BDM information to Cinder16:14
jungleboyjsmcginnis:  ++16:14
*** ayoung has quit IRC16:14
jungleboyjgeguileo:  So do we have a response to Matt's last note?16:17
*** ayoung has joined #openstack-meeting16:17
geguileoI don't have one yet (been sick in bed Mon and Tues)16:18
jungleboyjgeguileo:  Ugh.  :-(  Sorry to hear that.  Feel better.16:18
*** ociuhandu_ has joined #openstack-meeting16:18
geguileoI'm better now, thanks16:19
jungleboyjgeguileo:  If you are planning to respond though, I think we are good.  Just wanted to make sure that that wasn't forgotten.16:19
jungleboyjSo, I think that is all I had on that topic.16:20
jungleboyj#topic Open Discussion16:20
*** openstack changes topic to "Open Discussion (Meeting topic: Cinder)"16:20
geguileojungleboyj: I'm not entirely sure that it will work  :-(16:20
rosmaitayeah, let's revisit this at next week's meeting16:20
geguileojungleboyj: in my opinion I would create a new microversion and accept the PUT method of the attachment to accept the connection information16:20
geguileorosmaita: sounds good16:21
jungleboyjrosmaita: ++16:21
jungleboyjRespond to Matt, see what he says and we can further discuss based on that.16:21
rosmaitageguileo: that seems like a good idea, though, we make a call so that nova can pass us the "old" info and that way it won't get lost16:22
*** ociuhandu has quit IRC16:22
*** jamesmcarthur has quit IRC16:22
rosmaitai mean, we don't make a call, we make a new call available for nova to use16:22
geguileorosmaita: we add functionality to the old attachments call16:23
*** brinzhang has quit IRC16:23
*** brinzhang has joined #openstack-meeting16:23
jungleboyjIf we can't ever get rid of it though ... not the end of the world to help Nova get beyond it.16:23
jungleboyjThe Greater Good16:23
*** brinzhang has quit IRC16:24
jungleboyjOk.  Any other topics for today?16:25
*** brinzhang has joined #openstack-meeting16:25
whoami-rajatjungleboyj: just wanted to enquire if this is the right thing to do[1] since somehow the online migrations are already running on grenade jobs on gate (still looking into it)16:25
davee__support it until a proper announcement with plenty of lead time for deprecation for users to update their environment appropriately16:25
*** ociuhandu_ has quit IRC16:25
*** brinzhang has quit IRC16:26
*** brinzhang has joined #openstack-meeting16:26
jungleboyjwhoami-rajat:  Good question.16:27
jungleboyjwhoami-rajat:  I am not very familiar with Grenade.16:27
jungleboyjeharney:  ^^^16:27
toskyit's not much about grenade, I'd say16:27
smcginnisI thought online migrations just ran at startup.16:27
toskygrenade deploys using devstack, then it updates the code using the same configuration and some scripts which follows the suggested upgrade procedures16:28
*** brinzhang has quit IRC16:28
toskythe question is more whether the online migrations needs to be run automatically or not; the documentation seems to hint that they need to be executed manually16:28
*** brinzhang has joined #openstack-meeting16:28
toskyor at least that's how my unexperienced eye reads them :) (looking for the link...)16:29
*** jamesmcarthur has joined #openstack-meeting16:30
whoami-rajatthis man page [1] states 'This command should be run after upgrading the database schema.' which is exactly what i did in my patch16:30
jungleboyjwhoami-rajat:  But you think that it is already getting run even without your patch in place?16:32
smcginnisWould be odd that we would need explicit handling for Cinder. Should be the same for all services.16:33
smcginnisAt least I would hope.16:33
whoami-rajatsmcginnis: if it runs after startup, is nova following a different approach to it[1]?16:33
smcginniswhoami-rajat: Oh? So you're just doing the same thing nova is doing?16:33
toskythat's the idea of whoami-rajat's patch, the question is: is it the right way? Is the documentation unclear?16:34
smcginnisThat looks like it should be fine then.16:34
jungleboyjsmcginnis:  ++16:35
davee__smcginnis pulls pin on grenade and tosses to whoami-rajat ...16:35
*** jamesmcarthur has quit IRC16:35
whoami-rajatjungleboyj:  yes, initially my patch failed due to a dependency on previous online migration, but when grenade updated master, it passed16:35
jungleboyjHe he16:35
jungleboyjwhoami-rajat:  Ok, than I think that is probably the direction we want to go.16:36
smcginnisUnfortunately I think dulek and scottda were the ones that knew all of the DB migration stuff well.16:36
geguileowell, we had a problem with online migration....16:36
jungleboyjsmcginnis:  Unfortunately.  I used to know it to some extent in the past but have been totally disconnected on the online stuff.16:37
geguileowhere we didn't follow the basics of it16:37
geguileoand required services to be up and running16:37
smcginnisWhich is weird, because I would expect "online" migrations to require that anyway.16:37
geguileowhich is a no-no for online data migrations16:37
smcginnisBut yeah, I think it would be useful if all of the expectations and things were better documented somewhere.16:37
* jungleboyj is so confused16:38
jungleboyjSo what does Online mean?16:38
smcginnisAny maybe they are, but I don't know where.16:38
geguileosmcginnis: I think I added it in a patch16:38
toskybut then we may want to recheck how to make sure that granade (or any upgrade test) catches any issue linked to *not* executing the online migrations16:38
davee__I t would be a reasonable assumption that any patch that uses a modified DB schema should also perform a DB Migration as part of that patch16:39
*** jamesmcarthur has joined #openstack-meeting16:39
toskydoes it mean that the online migration code is executed automatically?16:40
* tosky confused16:40
geguileotosky: no, I think it is executed as part of any upgrade16:41
geguileothe whole upgrade thing is quite complex16:41
smcginnis"online-migrations" seems to be a bit of a misnomer.16:42
jungleboyjSo what makes them online vs any other DB upgrade?16:42
jungleboyjsmcginnis:  ++16:42
geguileobut in sumary it's something like this16:42
geguileoyou make some changes in the DB that require either new info or info that you can get from other parts of the DB16:42
geguileothen you add online migration code to your component16:42
geguileothat runs every time you load the object and see it doesn't have the new data16:42
geguileoso you load, check, if old, then change, and save16:42
geguileothat way you are postponing the changes until the objects are used16:43
geguileoand spread the load16:43
geguileothen, for the next release you have an online migration code on the manager16:43
geguileothat does that serially, without requiring the services running (in case this is an FFU and not a rolling upgrade)16:43
geguileoand on that same release you can remove the online migration from your service code so it doesn't get checked on the loads16:44
geguileobecause the cinder-manage online migration code will be run before the new services are started16:44
davee__geguileo gets it  +216:45
smcginnisSo these can be removed now on master, right?
geguileomaybe you cannot remove it on the next (N+1) but instead in N+216:45
geguileoI'm not entirely sure about that one16:45
*** bnemec has quit IRC16:45
geguileobut when you run the cinder-manage online migrations, the amount of data to migrate should be considerably smaller16:45
geguileoand therefore faster16:46
smcginnisThe code comment you added indicated N+1. I think we still state we only do single hop upgrades, but I suppose it would be a small performance impact to keep things around an extra release just in case.16:46
geguileosmcginnis: you can remove the cinder-manage code in N+116:46
geguileosmcginnis: not sure about the service online migration code though, in case you are running rolling upgrades16:47
geguileoI would have to think about it to be sure16:47
smcginnisAh, I think I get it.16:47
*** bnemec has joined #openstack-meeting16:47
davee__always a good idea to include a versioning field in the db schema aand base upgrade logic on that version number so you know what needs to be done to end up in a stable operational state16:47
smcginnisThat's basically how it works.16:47
jungleboyjgeguileo:  ++ Yeah, that is helpful.16:48
smcginnisAt least for the schema. Not the data that these appear to address.16:48
*** slaweq has joined #openstack-meeting16:48
toskywhen everything is confirmed, could the documentation at be updated then to explain that it's not strictly needed, but it just speeds up everything?16:49
whoami-rajatgeguileo: so the online migration running is related to upgrade process and not cinder right? it might be a part of the volbak playbook or the upgrade process grenade follows ?16:49
tosky(or whethever is the expected behavior)16:49
geguileowhoami-rajat: the cinder-manage online migration is part of the upgrade16:50
geguileowhoami-rajat: the online migrations should also be part of the cinder service code16:50
*** AlanClark has joined #openstack-meeting16:51
smcginnisThat's part of the confusion - there are two different "online migration" steps.16:52
jungleboyjThere is the part that runs online as part of the service and the part that runs later during upgrade?16:52
whoami-rajatgeguileo:  so it should be run manually during an upgrade or is automatically associated with the startup of services?16:53
geguileojungleboyj: yup, great summary :-)16:53
*** panda is now known as panda|off16:53
geguileowhoami-rajat: it should be handled by the upgrade mechanism16:53
geguileowhoami-rajat: whatever that is16:53
*** markvoelker has quit IRC16:53
jungleboyjgeguileo:  Ah, so I am getting it.  So that is why people got in trouble is they had originally written it to assume the service was up but then when they moved it to upgrade and the service wasn't up, it blew up.16:53
geguileojungleboyj: yup16:53
geguileojungleboyj: it'll only be up if we are doing rolling upgrades16:54
geguileobut if you do a forward upgrade from N to N+3, then you bring them down16:54
*** markvoelker has joined #openstack-meeting16:54
geguileoand you run db-sync and online upgrade on each of the intermediary releases16:54
jungleboyjgeguileo:  Ok.  Good to know.16:54
jungleboyjThis makes more sense now.16:55
davidshaJust want to get this in quickly before the meeting ends, I think this bug may have gone under the radar:
openstackLaunchpad bug 1843431 in os-brick "NVMeOF connector fails to disconnect a volume with the latest nvme-cli" [Undecided,New]16:55
smcginnisOh great. Any code ready for that yet?16:56
jungleboyjdavidsha:  Thanks for raising.  Look like we have someone fixing it.16:56
davidshaThis patch is it?
smcginnisAh, just didn't get linked to the bug report.16:57
*** mriedem is now known as mriedem_afk16:58
smcginnisOne other quick thing before we end - we had a few late driver submissions in train.16:58
smcginnisI've put them under a better topic to help track them for ussuri.16:58
*** gyee has joined #openstack-meeting16:58
smcginnisAny other ones, please update the topic to ussuri-drivers to help us keep track of what's out there.16:58
jungleboyjsmcginnis:  Thanks!16:59
*** ekcs has joined #openstack-meeting17:00
jungleboyjOur time is up.  Thank you everyone.17:00
*** openstack changes topic to "OpenStack Meetings ||"17:00
openstackMeeting ended Wed Oct  9 17:00:38 2019 UTC.  Information about MeetBot at . (v 0.1.4)17:00
openstackMinutes (text):
geguileothanks everyone!17:00
*** davidsha has quit IRC17:01
*** rosmaita has left #openstack-meeting17:01
*** woojay has left #openstack-meeting17:02
*** jawad_ax_ has joined #openstack-meeting17:02
*** e0ne has quit IRC17:02
*** jawad_axd has quit IRC17:02
*** slaweq has quit IRC17:02
*** tosky has left #openstack-meeting17:12
*** bbowen has quit IRC17:17
*** priteau has quit IRC17:20
*** ekcs has quit IRC17:29
*** bbowen has joined #openstack-meeting17:31
*** ekcs has joined #openstack-meeting17:34
*** ayoung has quit IRC17:37
*** ayoung has joined #openstack-meeting17:38
*** ekcs has quit IRC17:40
*** ricolin has quit IRC17:41
*** davee__ has quit IRC17:44
*** igordc has quit IRC17:59
*** ociuhandu has joined #openstack-meeting18:07
*** brinzhang has quit IRC18:10
*** brinzhang_ has quit IRC18:10
*** brinzhang has joined #openstack-meeting18:11
*** brinzhang_ has joined #openstack-meeting18:11
*** ayoung has quit IRC18:12
*** ayoung has joined #openstack-meeting18:13
*** ralonsoh has quit IRC18:15
*** ekcs has joined #openstack-meeting18:17
*** mriedem_afk is now known as mriedem18:18
*** slaweq has joined #openstack-meeting18:19
*** slaweq has quit IRC18:25
*** e0ne has joined #openstack-meeting18:25
*** raildo has quit IRC18:27
*** ekcs has quit IRC18:28
*** slaweq has joined #openstack-meeting18:29
*** bbowen has quit IRC18:36
*** slaweq has quit IRC18:39
*** ekcs has joined #openstack-meeting18:41
*** jamesmcarthur has quit IRC18:45
*** andrebeltrami has quit IRC18:51
*** e0ne has quit IRC19:00
*** pcaruana has quit IRC19:07
*** e0ne has joined #openstack-meeting19:08
*** mgagne has quit IRC19:11
*** ekcs has quit IRC19:12
*** yamamoto has joined #openstack-meeting19:12
*** bbowen has joined #openstack-meeting19:15
*** yamamoto has quit IRC19:17
*** slaweq has joined #openstack-meeting19:20
*** thgcorrea has quit IRC19:20
*** tesseract has quit IRC19:20
*** ociuhandu has quit IRC19:26
*** zaneb has quit IRC19:33
*** zbitter has joined #openstack-meeting19:33
*** kopecmartin is now known as kopecmartin|off19:35
*** ekcs has joined #openstack-meeting19:43
*** ekcs has quit IRC19:55
*** munimeha1 has joined #openstack-meeting19:55
*** e0ne has quit IRC19:56
*** ekcs has joined #openstack-meeting19:56
*** slaweq has quit IRC20:06
*** ayoung has quit IRC20:06
*** ayoung has joined #openstack-meeting20:08
*** slaweq has joined #openstack-meeting20:11
*** artom has quit IRC20:13
*** slaweq has quit IRC20:16
*** pcaruana has joined #openstack-meeting20:33
*** ociuhandu has joined #openstack-meeting20:37
*** ociuhandu has quit IRC20:39
*** ekcs has quit IRC20:40
*** pcaruana has quit IRC20:40
*** lpetrut has joined #openstack-meeting20:41
*** ekcs has joined #openstack-meeting20:47
*** lpetrut has quit IRC20:52
timburke#startmeeting swift21:00
openstackMeeting started Wed Oct  9 21:00:02 2019 UTC and is due to finish in 60 minutes.  The chair is timburke. Information about MeetBot at
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.21:00
*** openstack changes topic to " (Meeting topic: swift)"21:00
openstackThe meeting name has been set to 'swift'21:00
*** alecuyer has joined #openstack-meeting21:00
timburkewho's here for the swift meeting?21:00
*** zaitcev has joined #openstack-meeting21:00
timburkei know clayg's under the weather today, so i'm assuming tdasilva or i will be talking about versioning later21:01
timburke#topic Shanghai21:01
*** openstack changes topic to "Shanghai (Meeting topic: swift)"21:01
timburkea few different things to bring up21:02
timburkefirst, as always, there's the etherpad for topics21:02
timburkebut we've also been getting more details about logistics21:02
timburkenotably, looks like21:02
timburke* we've got three days' worth of time dedicated to swift21:02
timburke* no coffee in the meeting rooms21:02
timburke* we *can* move furniture this time ;-)21:02
timburke* and sounds like there's going to be a pretty hard cut-off at 4:30, so we may need to investigate other places to continue conversations before dinner21:03
mattoliverauNo coffee!21:03
timburkei know, right? and they expect work to get done... ;-)21:03
kota_perhaps, tea exists, instead?21:03
rledisezmattoliverau: but the discussions can move to a pub from 4:30, you got to choose between coffee or beer ;)21:04
alecuyerdoesn't look like it from the link :/21:04
mattoliveraulol, that's a good point21:04
timburkenope :-(21:04
timburke"Unfortunately, the venue does not allow ANY food or drink in any of the rooms."21:04
kota_*not allow ANY food or drink in any of the rooms.*21:04
mattoliverauSo there may be coffee outside21:04
mattoliverauso more hallway track time21:04
timburkeand we can maybe organize some breakfast get-togethers or something21:05
timburkeand finally, i remember being sad that i didn't think to do this in denver, so i proposed an ops feedback session; looks like we haven't had one of those since Boston21:05
timburkeand it was accepted!21:05
mattoliverauoh cool, nice one21:05
timburkeso we'll see who we can meet there :-)21:06
*** whoami-rajat has quit IRC21:06
timburkeany other comments/questions on Shanghai? any further information i should try to run down for people?21:07
*** ayoung has quit IRC21:07
timburkei ought to book the team photo; do we have any preferences on timing? morning, afternoon, ...21:08
alecuyerI don't mind either time21:08
*** ayoung has joined #openstack-meeting21:08
rledisezgiven the timezone, I think I would prefer during the night, so whatever…21:08
timburkemaybe i'll aim for right before or right after lunch21:09
timburkenext up21:09
timburke#topic train release21:09
*** openstack changes topic to "train release (Meeting topic: swift)"21:09
mattoliverauright before so you don't have to rush back.21:09
timburkejust a quick follow-up from last week21:09
kota_i don't have strict limitation. I'll leave Saturday morning so I can make me available until Friday21:09
timburkewe have a 2.23.0! \o/21:09
timburkeand a stable/train branch21:09
timburkethanks again for all your hard work -- best release yet :-)21:10
*** AlanClark has quit IRC21:10
timburkeon to updates!21:10
zaitcevMake sure not to wear any signage on your shirt, most especially not in Chinese ideograms that you cannot read. Unbeknownst to you, they may refer to Taiwan, Tibet, HK, or Uigurs.21:10
timburkegood call21:11
timburke#topic versioning21:11
*** openstack changes topic to "versioning (Meeting topic: swift)"21:11
timburkei haven't seen a tdasilva yet...21:11
timburkebut we've been investigating this null-namespace idea... and bumping into some complications with C code (perhaps unsurprisingly)21:12
zaitcevI'm surprised, actually. You can have an empty string "" in C easily, so...21:13
*** lpetrut has joined #openstack-meeting21:13
timburkebut a string like "foo\0bar"...21:13
timburkeanyway, still trying to figure out how best to move forward with it, but i don't think we've been able to rule it out entirely yet. if NUL doesn't work out, we may try to claw back "\x01", similar to how we claimed the leading "." namespace in accounts21:15
timburkestill need to wire up versioning to use the reserved namespace, and layer s3api on top of that. we should have something of a solid proof-of-concept in time for discussions in Shanghai, though21:16
rledisezjust saying as I heard it from a colleague today, why not using / as it is already forbidden in container names. the reason might be the enormous impact on codebase21:16
rledisezI don't know if it was considered21:17
timburkeinteresting thought. i think we'd have a hard time distinguishing container requests from object requests, though... hm...21:18
rledisezyeah, that's what I though too21:18
timburkeadd an x-backend-no-really-this-is-my-container-name header? ;-)21:19
rlediseztimburke: nah, let's fork sqlite instead ;)21:19
timburkeanyway, i don't think we're stuck on anything at the moment, just a matter of needing to try a few more things and get a better feel for how we ought to move next21:20
timburke#topic lots of small files21:20
*** openstack changes topic to "lots of small files (Meeting topic: swift)"21:20
timburkealecuyer, hopefully my changes to were agreeable21:21
alecuyerYes, thanks a lot for your help with this one!21:21
*** ayoung has quit IRC21:21
timburkehappy to! and there's also a fresh merge from master21:21
alecuyernext, I'll need to rebase this one  , which is much smaller21:22
timburkeoh nice, yeah21:23
alecuyerand then I'd like to think about how to handle leveldb failure (again I'd like to stress that we're never seeing leveldb self destructing, but occasionally fsck will remove a .ldb file and the db goes bad)21:23
timburkeand i think i spotted a problem in the quarantine logic:
alecuyeroh I missed it! will check it, thanks timburke21:24
*** ayoung has joined #openstack-meeting21:24
alecuyercurrently a leveldb failure is handled "out of band" . I wonder if that should happen within swift (rebuild the db if it's corrupt)21:24
alecuyerWe don't expect swift to repair a filesystem, but in this case I'm not sure..21:24
*** ekcs has quit IRC21:25
timburkedo we have a feeling for roughly how long it takes to rebuild the db?21:25
rledisezI think for a 6TB drive it's about 12 hours if i remember well21:25
alecuyerFor a disk full of small files yes that's about right (but that's what we're expecting for LOSF right)21:26
* timburke whistles21:26
rledisezif there is fewer bigger files it would be way faster of course21:26
timburkethere should be multiple dbs per drive, right? would that be for rebuilding *all* dbs on a drive?21:27
alecuyerthere is one db per drive/policy21:27
timburkeoh, i was just mis-remembering21:28
rledisezin the example I gave there is only one policy per drive, so one db21:28
timburkemakes sense21:28
*** rfolco has quit IRC21:28
rledisezjsut to put things in regards, it's way faster than xfsrepair in the same condition21:28
alecuyer(about 70 million "files")21:29
mattoliverauI guess an auditor could fire off an async concurrent job to start a rebuild if detected.. but 12 hours means you'd want to be sure.21:29
timburkeyeah, my initial thought was that maybe the auditor (or maybe even replicator/reconstructor) could handle this... but that might make for some really lumpy cycle times...21:30
*** lpetrut has quit IRC21:30
alecuyercurrently we watch it through the index server python parent process21:30
mattoliverauyeah, it might be considered a different type of job21:30
mattoliverauSo it could enqueue something maybe. that'll either alert operators or maybe even fire something off.21:31
*** radeks_ has quit IRC21:31
alecuyeryes one could have the choice of manual trigger vs fully automated. So something to think about21:32
mattoliverauits a thought. Self healing would be cool :)21:32
mattoliverauAdd to the ideas list :)21:32
timburke👍 is there anything else we should be looking at/thinking about?21:33
rledisezjust thinking, if the repair does not start, the data are mostly unreachable, right? so any good reasons to not fire the repair?21:33
rledisezjust a note of our deployment: we tried to extend our deployment of pure golang leveldb but we had to rollback, some strange errors alecuyer is investigating21:33
alecuyertimburke: no I think that's it from me21:34
timburkecool. i'll think more about the db file stuff...21:35
timburke#topic sharding21:35
*** openstack changes topic to "sharding (Meeting topic: swift)"21:35
timburkemattoliverau took a look at the stat-latching patch!21:35
timburkethanks :-D21:35
mattoliverauI haven't really done as much on this as I'd like.. kinda been distracted.21:36
mattoliverauoh yeah I did look at that21:36
mattoliverauI like it. Will test it some more locally in an SAIO21:36
mattoliverauTry and do that today while I have no idea what I'm doing workwise21:36
timburkesomething something, changing strategic direction something something?21:37
mattoliverauyeah. Well I was lucky and wasn't layed off, but given a new role outside of openstack21:37
*** ekcs has joined #openstack-meeting21:37
mattoliverauin SES of all places.21:37
mattoliverauBut no idea what that really means until I meet with my new manager today sometime.21:38
timburketrue. thanks for still showing up here this week! i hope you can keep working on swift, but i know you'll keep doing good work whatever you're working on21:38
mattoliverauI hope to still be given time to work on an upstream project... but it might mean no more travel to openstack events supported by my company.21:39
timburkethat's all i had21:39
timburke#topic open discussion21:40
*** openstack changes topic to "open discussion (Meeting topic: swift)"21:40
timburkeanything else we ought to discuss?21:40
timburkeall right, let's let mattoliverau and kota_ get breakfast, and alecuyer get to bed ;-)21:43
timburkethank you all for coming, and thank you for working on swift!21:43
alecuyerthhanks all21:43
*** openstack changes topic to "OpenStack Meetings ||"21:43
openstackMeeting ended Wed Oct  9 21:43:24 2019 UTC.  Information about MeetBot at . (v 0.1.4)21:43
openstackMinutes (text):
*** ayoung has quit IRC21:43
*** ayoung has joined #openstack-meeting21:44
*** alecuyer has left #openstack-meeting21:44
*** diablo_rojo has quit IRC21:47
*** eharney has quit IRC21:49
*** ekcs has quit IRC22:05
*** slaweq has joined #openstack-meeting22:11
*** ekcs has joined #openstack-meeting22:12
*** ykatabam has joined #openstack-meeting22:13
*** rcernin has joined #openstack-meeting22:14
*** slaweq has quit IRC22:15
*** zaitcev has left #openstack-meeting22:19
*** slaweq has joined #openstack-meeting22:24
*** markvoelker has quit IRC22:28
*** slaweq has quit IRC22:29
*** mriedem has quit IRC22:30
*** ociuhandu has joined #openstack-meeting22:30
*** ociuhandu has quit IRC22:35
*** jamesmcarthur has joined #openstack-meeting22:42
*** dviroel has quit IRC22:59
*** macz has quit IRC23:02
*** yamamoto has joined #openstack-meeting23:10
*** ekcs has quit IRC23:18
*** yamamoto has quit IRC23:27
*** gyee has quit IRC23:27
*** yamamoto has joined #openstack-meeting23:30
*** munimeha1 has quit IRC23:37
*** ekcs has joined #openstack-meeting23:43
*** ekcs has quit IRC23:43
*** ekcs has joined #openstack-meeting23:43
*** jamesmcarthur has quit IRC23:43
*** dklyle has quit IRC23:47
*** david-lyle has joined #openstack-meeting23:47
*** markvoelker has joined #openstack-meeting23:48
*** slaweq has joined #openstack-meeting23:56
*** brinzhang_ has quit IRC23:57
*** brinzhang has quit IRC23:58
*** brinzhang has joined #openstack-meeting23:58
*** brinzhang_ has joined #openstack-meeting23:58
*** markvoelker has quit IRC23:59

Generated by 2.15.3 by Marius Gedminas - find it at!