| corvus | oh neat | 00:00 |
|---|---|---|
| corvus | apparently gerrit can create repos by pushing to them | 00:00 |
| clarkb | ooh | 00:00 |
| corvus | but gitea doesn't know about them | 00:00 |
| fungi | according to man ssh, "PreferredAuthentications can be used to change the default order" | 00:01 |
| corvus | i guess we'll see what happens when my script gets around to creating the repos that gerrit is pushing now | 00:01 |
| fungi | if it becomes necessary | 00:01 |
| openstackgerrit | Merged openstack-infra/infra-specs master: Move project-hosting spec to completed https://review.openstack.org/622609 | 00:02 |
| openstackgerrit | Merged openstack-infra/infra-specs master: Add opendev Gerrit spec https://review.openstack.org/623033 | 00:02 |
| clarkb | The doc update should merge soon too | 00:03 |
| fungi | infra-root: reminder, we'll want https://review.openstack.org/627018 merged before the next gerrit restart | 00:03 |
| mordred | clarkb: also, I think we were saying this to fungi earlier (and by we, I mean corvus) - but we should also walk y'all through all the various k8s things - maybe screencast walkthrough style - before going live with this bad boy | 00:04 |
| corvus | http://38.108.68.66/openstack/steth is one that gerrit replicated to before my script created it; so we should check back on that later | 00:04 |
| mordred | cool | 00:04 |
| corvus | yeah, i totally voluteered mordred and i for that without asking :) | 00:04 |
| corvus | but i figured mordred is exicted about this and likes to talk, so it was a safe bet | 00:05 |
| mordred | I have learned many things about kubectl that are non-terrible | 00:05 |
| clarkb | mordred: ya that would be good though I feel like I have a high enough level understanding of it that I'd work my way through kubectl docs and be fine | 00:05 |
| openstackgerrit | Merged openstack-infra/infra-specs master: Overhaul instructions in README.rst for clarity https://review.openstack.org/623211 | 00:05 |
| fungi | i'll appreciate the show-n-tell | 00:06 |
| clarkb | we should also make sure we size things appropriately as much as we can figure upfront | 00:06 |
| clarkb | (isnce the atomic based k8s had a sad due to undersizing) | 00:06 |
| mordred | clarkb: yah - seeing as how I knew nothing a couple of weeks ago and now feel comfortable debugging stuff, I'm not worried about anybody - but yeah, show-n-tell should be nice | 00:06 |
| mordred | yah. although we're not running this on atomic anymore | 00:06 |
| clarkb | ya its xenial ansible dpeloyed | 00:07 |
| mordred | also - things like "replace a node" and "resize a node" are all things we should do before it goes live | 00:07 |
| corvus | one of the things we can do during the show-and-tell is bootstrap the whole thing from nothing with everyone watching. and drop nodes and watch ceph heal. | 00:07 |
| mordred | ++ | 00:07 |
| clarkb | mordred: ++ and upgrade k8s | 00:07 |
| mordred | yah. although we don't currently have a candidate k8s version to upgrade to | 00:08 |
| mordred | maybe we could deploy it on 1.11.4 - and then upgrade to .5 | 00:08 |
| mordred | just for the exercise | 00:08 |
| clarkb | ++ | 00:08 |
| clarkb | we should also do a braindump on the networking. which driver did you choose? | 00:08 |
| mordred | I made no choices - the k8s-on-openstack author chose weave | 00:09 |
| mordred | but yes - we should DEFINITELY talk about networking - especually about the octavia interface | 00:09 |
| corvus | my script is at the beginning of the openstack/ alphabet, and gerrit is at the end | 00:10 |
| corvus | so they're decidedly out of sync still | 00:10 |
| corvus | openstack/c vs openstack/x | 00:10 |
| mordred | http://38.108.68.66/explore/repos is fun though | 00:11 |
| clarkb | mordred: is weave open source? their website ient clear one that | 00:11 |
| clarkb | there us a trial version of something | 00:11 |
| corvus | http://38.108.68.66/openstack-attic/akanda has populated | 00:12 |
| mordred | yeah. it's one of the two 'default' options | 00:12 |
| mordred | akanda repo looks great! | 00:12 |
| corvus | http://38.108.68.66/openstack-attic/akanda seems to show the kilo branch by default | 00:13 |
| mordred | I agree with that | 00:14 |
| clarkb | looks like weave net us open but weave cloud is not | 00:14 |
| clarkb | in any case should be fine for us to use | 00:14 |
| *** jamesmcarthur has joined #openstack-infra | 00:14 | |
| corvus | there is a default branch setting in the ui | 00:15 |
| mordred | clarkb: yeah. I wouldn't mind learning a bit more and potentially looking at calico and ipv6 ... but for now I think weave is working fine | 00:15 |
| mordred | corvus: the default branch is set to stable/kilo in the db | 00:16 |
| corvus | it seems to be set to kilo in akanda | 00:16 |
| *** eernst has quit IRC | 00:16 | |
| mordred | corvus: maybe it's getting set by the force push - maybe gerrit decided to force push stable/kilo last? | 00:16 |
| corvus | the only non-master values are on akanda and akanda-appliance-builder so far | 00:17 |
| mordred | corvus: hrm. database-api is fine - it's on lthe ... yeah | 00:17 |
| mordred | did the default branch for akanda get set to non-master in gerrit? | 00:17 |
| corvus | but maybe those are the only repos that existed in the db during the push? | 00:18 |
| mordred | hrm. extra weird | 00:18 |
| corvus | i guess we'll see when things swing back around | 00:18 |
| mordred | yeah | 00:18 |
| clarkb | head isnt pushed so i dont think gerrit can communicate that info | 00:19 |
| clarkb | this is one reason removing master is "bad" | 00:19 |
| *** jamesmcarthur has quit IRC | 00:21 | |
| mordred | yeah - but that makes the default_branch field being set even weirder | 00:22 |
| corvus | melange just got updated as well | 00:22 |
| mordred | corvus: yes, I agree | 00:23 |
| corvus | so when everything is settled, let's set the defaults back to master, then see if gerrit's force-pushing updates them again | 00:23 |
| corvus | i'm going to afk for a bit while these things run. i'll check back in later | 00:24 |
| *** tosky has quit IRC | 00:25 | |
| clarkb | hrm weave pods can be evicted by k8s if the node runs out of resources. We may need or want some simple monitoring for base functionality too | 00:27 |
| clarkb | (thats likely longer term need) | 00:27 |
| openstackgerrit | Merged openstack-infra/system-config master: Add more \\ to launchpad-bug Gerrit tracking-id https://review.openstack.org/627018 | 00:29 |
| clarkb | melange is a nameI havent heard in a while | 00:30 |
| clarkb | the git spice must flow | 00:30 |
| *** hwoarang has quit IRC | 00:31 | |
| mordred | corvus: ++ to checking after the first pass | 00:32 |
| mordred | I'd put money on it being something with the initial push | 00:32 |
| *** hwoarang has joined #openstack-infra | 00:32 | |
| mordred | (and might be a bug we can file if we can figure it out) | 00:33 |
| *** whoami-rajat has quit IRC | 00:34 | |
| *** xek has quit IRC | 00:42 | |
| *** xek has joined #openstack-infra | 00:43 | |
| *** rh-jelabarre has quit IRC | 00:44 | |
| *** xek has quit IRC | 00:47 | |
| *** bobh has joined #openstack-infra | 00:51 | |
| *** bobh has quit IRC | 00:56 | |
| *** rascasoft has joined #openstack-infra | 00:57 | |
| *** rascasoft has quit IRC | 01:06 | |
| corvus | okay, http://38.108.68.66/openstack/steth has been created. gerrit pushed to it a long time ago, but there's no content there | 01:08 |
| corvus | so we'd probably either need another push, or to tell gitea to resync from disk in order fo it to notice content | 01:08 |
| corvus | it's still chugging; i'll wait until it's done before i do anything like that | 01:08 |
| openstackgerrit | James E. Blair proposed openstack-infra/system-config master: Add gitea k8s resource definitions and playbook https://review.openstack.org/626759 | 01:12 |
| corvus | that's the result of today's work | 01:12 |
| clarkb | heh that ansible reads yaml into the playbook then writes yaml back to k8s :) | 01:13 |
| clarkb | puppet suffered from a lot of this too. lack of passthrough serialized data | 01:14 |
| corvus | clarkb: which bit are you looking at? | 01:14 |
| clarkb | corvus: the top of the playbook with the from yaml stuff | 01:15 |
| corvus | clarkb: ah, ansible does not suffer from that problem :) | 01:16 |
| corvus | clarkb: you can totally pass through data unaltered; see the examples at https://docs.ansible.com/ansible/latest/modules/k8s_module.html#examples | 01:16 |
| clarkb | is the from yaml unnecessary then? | 01:16 |
| corvus | clarkb: specifically the third from the bottom there ("Create a Deployment by reading the definition from a local file") | 01:16 |
| corvus | clarkb: it's there so that we can jinja-interpolate things into the yaml | 01:16 |
| corvus | clarkb: specifically for the secret.yaml file | 01:17 |
| *** bobh has joined #openstack-infra | 01:17 | |
| clarkb | ya but namespace doesnt seem to be a template? | 01:17 |
| corvus | clarkb: i just kept the pattern for the other files, but we can switch to direct passthrough for the ones that don't use it. | 01:17 |
| clarkb | its valid k8s config as is | 01:17 |
| clarkb | ah | 01:17 |
| *** jamesmcarthur has joined #openstack-infra | 01:17 | |
| corvus | i thought "always assume k8s files are jinja-templatable" was a neat idea, but i'm not married to it | 01:18 |
| clarkb | I think I prefer that since its more accurate to what we want (eg dont need templating just apply directly) | 01:18 |
| corvus | yeah, the downside is we may end up with a deployment with "{{ foobar }}" in it before we realize we need to turn on jinja | 01:18 |
| corvus | though, i guess logically, i should use .j2 extensions to make it clear | 01:19 |
| corvus | Playbook run took 0 days, 1 hours, 27 minutes, 52 seconds | 01:19 |
| corvus | that's all the repos created in the db | 01:19 |
| corvus | pushes are still happening | 01:19 |
| corvus | we're on openstack-infra/g there | 01:20 |
| *** bobh has quit IRC | 01:21 | |
| *** hwoarang has quit IRC | 01:21 | |
| *** hwoarang has joined #openstack-infra | 01:22 | |
| *** jamesmcarthur has quit IRC | 01:23 | |
| clarkb | corvus: thinking out loud here we probably want all the k8s yaml to live in separate files so that you can use them without ansible easily too? the configmap config seems to be the exception to that rule in the playbook | 01:24 |
| *** rascasoft has joined #openstack-infra | 01:33 | |
| *** rascasoft has quit IRC | 01:43 | |
| *** bobh has joined #openstack-infra | 01:49 | |
| *** bobh has quit IRC | 01:54 | |
| *** hwoarang has quit IRC | 02:04 | |
| *** hwoarang has joined #openstack-infra | 02:08 | |
| *** hamerins has joined #openstack-infra | 02:09 | |
| *** tinwood has quit IRC | 02:10 | |
| *** markvoelker has joined #openstack-infra | 02:10 | |
| *** rascasoft has joined #openstack-infra | 02:11 | |
| *** tinwood has joined #openstack-infra | 02:11 | |
| *** hamerins has quit IRC | 02:11 | |
| *** hamerins has joined #openstack-infra | 02:12 | |
| *** rascasoft has quit IRC | 02:18 | |
| *** jamesmcarthur has joined #openstack-infra | 02:19 | |
| *** hamerins has quit IRC | 02:20 | |
| *** jamesmcarthur has quit IRC | 02:24 | |
| *** _alastor_ has quit IRC | 02:28 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:29 | |
| *** hamerins has joined #openstack-infra | 02:30 | |
| *** hamerins has quit IRC | 02:31 | |
| *** rfolco has quit IRC | 02:34 | |
| *** rfolco has joined #openstack-infra | 02:35 | |
| *** hamerins has joined #openstack-infra | 02:39 | |
| *** mriedem_away has quit IRC | 02:58 | |
| *** jamesmcarthur has joined #openstack-infra | 03:20 | |
| *** jamesmcarthur has quit IRC | 03:25 | |
| *** armax has quit IRC | 03:26 | |
| *** gouthamr has quit IRC | 03:36 | |
| *** gyee has quit IRC | 03:37 | |
| *** dmellado has quit IRC | 03:38 | |
| *** stevebaker has quit IRC | 03:38 | |
| *** dulek has quit IRC | 03:38 | |
| *** bhavikdbavishi has quit IRC | 03:38 | |
| *** armax has joined #openstack-infra | 03:48 | |
| *** gouthamr has joined #openstack-infra | 03:49 | |
| *** dmellado has joined #openstack-infra | 03:50 | |
| *** stevebaker has joined #openstack-infra | 03:51 | |
| *** jamesmcarthur has joined #openstack-infra | 03:51 | |
| *** dulek has joined #openstack-infra | 03:54 | |
| *** hamerins has quit IRC | 03:55 | |
| *** jamesmcarthur has quit IRC | 03:56 | |
| *** hamerins has joined #openstack-infra | 03:58 | |
| *** hamerins has quit IRC | 03:58 | |
| *** bhavikdbavishi has joined #openstack-infra | 03:59 | |
| *** armax has quit IRC | 04:01 | |
| *** stevebaker has quit IRC | 04:16 | |
| *** dmellado has quit IRC | 04:17 | |
| *** dulek has quit IRC | 04:17 | |
| *** gouthamr has quit IRC | 04:17 | |
| *** hwoarang has quit IRC | 04:21 | |
| *** hwoarang has joined #openstack-infra | 04:23 | |
| *** dmellado has joined #openstack-infra | 04:27 | |
| *** dulek has joined #openstack-infra | 04:27 | |
| *** gouthamr has joined #openstack-infra | 04:28 | |
| *** bobh has joined #openstack-infra | 04:28 | |
| *** stevebaker has joined #openstack-infra | 04:29 | |
| *** bobh has quit IRC | 04:44 | |
| *** jamesmcarthur has joined #openstack-infra | 04:52 | |
| *** jamesmcarthur has quit IRC | 04:54 | |
| *** jamesmcarthur has joined #openstack-infra | 04:54 | |
| *** jamesmcarthur has quit IRC | 04:57 | |
| *** jamesmcarthur has joined #openstack-infra | 04:58 | |
| *** jamesmcarthur has quit IRC | 05:17 | |
| *** jamesmcarthur has joined #openstack-infra | 05:17 | |
| *** jamesmcarthur has quit IRC | 05:48 | |
| *** bhavikdbavishi has quit IRC | 05:50 | |
| *** jamesmcarthur has joined #openstack-infra | 05:51 | |
| *** jamesmcarthur has quit IRC | 06:04 | |
| *** jamesmcarthur has joined #openstack-infra | 06:06 | |
| *** ykarel|away has joined #openstack-infra | 06:09 | |
| *** jamesmcarthur has quit IRC | 06:11 | |
| *** e0ne has joined #openstack-infra | 06:22 | |
| *** e0ne has quit IRC | 07:05 | |
| *** ccamacho has quit IRC | 07:05 | |
| *** ccamacho has joined #openstack-infra | 07:06 | |
| *** e0ne has joined #openstack-infra | 07:10 | |
| *** spa-87 has joined #openstack-infra | 07:14 | |
| *** e0ne has quit IRC | 07:43 | |
| *** ykarel|away has quit IRC | 07:46 | |
| *** agopi has joined #openstack-infra | 08:02 | |
| *** yboaron_ has joined #openstack-infra | 08:16 | |
| *** ykarel|away has joined #openstack-infra | 08:18 | |
| *** jtomasek has joined #openstack-infra | 08:21 | |
| *** spa-87 has quit IRC | 08:24 | |
| *** jtomasek has quit IRC | 08:28 | |
| *** hwoarang has quit IRC | 08:45 | |
| *** hwoarang has joined #openstack-infra | 08:51 | |
| *** hwoarang has quit IRC | 09:02 | |
| *** ykarel|away has quit IRC | 09:10 | |
| *** spa-87 has joined #openstack-infra | 09:12 | |
| *** ykarel|away has joined #openstack-infra | 09:15 | |
| *** hwoarang has joined #openstack-infra | 09:21 | |
| *** d0ugal has quit IRC | 09:26 | |
| *** agopi has quit IRC | 09:26 | |
| *** wolverineav has joined #openstack-infra | 09:31 | |
| *** jbadiapa has quit IRC | 09:33 | |
| *** wolverineav has quit IRC | 09:36 | |
| *** agopi has joined #openstack-infra | 09:48 | |
| *** hwoarang has quit IRC | 09:49 | |
| *** hwoarang has joined #openstack-infra | 09:53 | |
| *** agopi has quit IRC | 09:56 | |
| *** ykarel|away has quit IRC | 10:12 | |
| *** xek has joined #openstack-infra | 10:13 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Move postgres to tmpfs https://review.openstack.org/627035 | 10:14 |
| *** yboaron_ has quit IRC | 10:21 | |
| *** evrardjp has quit IRC | 10:27 | |
| *** evrardjp_ has joined #openstack-infra | 10:27 | |
| *** agopi has joined #openstack-infra | 10:40 | |
| *** pbourke has quit IRC | 10:42 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:42 | |
| *** pbourke has joined #openstack-infra | 10:44 | |
| *** bhavikdbavishi has quit IRC | 10:57 | |
| *** dkehn has quit IRC | 11:16 | |
| *** jaosorior has joined #openstack-infra | 11:18 | |
| *** agopi has quit IRC | 11:37 | |
| *** agopi has joined #openstack-infra | 11:38 | |
| *** ykarel|away has joined #openstack-infra | 11:41 | |
| *** ykarel|away has quit IRC | 11:49 | |
| *** dayou has quit IRC | 11:53 | |
| *** dayou has joined #openstack-infra | 11:57 | |
| *** agopi has quit IRC | 11:59 | |
| *** ykarel|away has joined #openstack-infra | 12:05 | |
| *** jamesmcarthur has joined #openstack-infra | 12:07 | |
| *** jamesmcarthur has quit IRC | 12:11 | |
| *** ykarel|away has quit IRC | 12:29 | |
| *** ykarel|away has joined #openstack-infra | 12:30 | |
| *** agopi has joined #openstack-infra | 12:31 | |
| *** tobiash has quit IRC | 13:19 | |
| *** tobiash has joined #openstack-infra | 13:23 | |
| *** agopi has quit IRC | 13:28 | |
| *** zul has quit IRC | 13:42 | |
| *** agopi has joined #openstack-infra | 13:46 | |
| *** markvoelker has quit IRC | 13:46 | |
| *** markvoelker has joined #openstack-infra | 13:46 | |
| *** e0ne has joined #openstack-infra | 13:54 | |
| *** agopi has quit IRC | 13:56 | |
| *** dkehn has joined #openstack-infra | 14:08 | |
| AJaeger | corvus: looking at your gitea instance: On git.openstack.org we've hidden the retired repos, your gitea instance showed them earlier today. But now I cannot reach it, I get a "500 Internal Server Error" | 14:09 |
| openstackgerrit | melissaml proposed openstack-infra/infra-specs master: remove the repetition words https://review.openstack.org/627039 | 14:12 |
| openstackgerrit | melissaml proposed openstack-infra/infra-specs master: remove the repetition words in doc https://review.openstack.org/627040 | 14:19 |
| *** bobh has joined #openstack-infra | 14:22 | |
| *** tosky has joined #openstack-infra | 14:22 | |
| fungi | AJaeger: that reminds me, i've recently come to realize that hiding retired repos does make it harder for people coming later to find the references we've left in their readme files indicating they're no longer maintained or have moved elsewhere | 14:33 |
| fungi | in particular, someone reached out to osf staff trying to find out what had happened to the openstack salt formulas. the readme files in them clearly stated they had moved to the saltstack github org, but we aren't actually putting those readme files anywhere search engines will index | 14:34 |
| fungi | i don't know whether we should be doing something different, mostly just pointing out the lack of discoverability we've engineered into it which seems to run counter to other aspects of the retirement process | 14:35 |
| mordred | AJaeger: I agree - I also cannot reach the new service ... I don't see anything wrong at the k8s level - it might be the octavia load balancer (which is a bit of a black box compared to everything else) | 14:36 |
| AJaeger | fungi: yes, indeed that is unfortunate. On the other hand, we have 542 retired repos, so filtering them out might be worth while. Definitley not a blocker for moving forward - but something to look at... | 14:39 |
| AJaeger | mordred: thanks for checking - no urgency from my side ;) | 14:40 |
| *** bobh has quit IRC | 14:40 | |
| *** bobh has joined #openstack-infra | 14:43 | |
| *** stevebaker has quit IRC | 14:51 | |
| *** dmellado has quit IRC | 14:51 | |
| *** stevebaker has joined #openstack-infra | 14:51 | |
| *** bobh has quit IRC | 14:52 | |
| *** dmellado has joined #openstack-infra | 14:53 | |
| *** hwoarang has quit IRC | 14:55 | |
| *** e0ne has quit IRC | 14:55 | |
| *** hwoarang has joined #openstack-infra | 14:57 | |
| *** e0ne has joined #openstack-infra | 14:59 | |
| *** bobh has joined #openstack-infra | 15:04 | |
| *** bobh has quit IRC | 15:04 | |
| corvus | we've nearly filled our cephfs: 10.110.59.103:6790,10.109.161.206:6790,10.103.118.117:6790:/ 239G 227G 12G 95% /data | 15:09 |
| corvus | oh, i think we *have* filled it | 15:10 |
| corvus | [Macaron] PANIC: session(start): open /data/gitea/sessions/8/d/8dfbcd26c1f08dbe: no space left on device | 15:10 |
| fungi | hah, nice | 15:11 |
| fungi | i guess the replication got far enough to eat it all? | 15:11 |
| corvus | yeah. plus we're indexing the repos, and i think they said indexes take 5x the repo space? i'm getting numbers now. | 15:12 |
| fungi | oof | 15:13 |
| fungi | raw repos on review.o.o take 12gb according to du | 15:13 |
| fungi | so call that 60gb minimum? | 15:14 |
| corvus | drwx------ 1 git git 868M Dec 22 08:47 repos.bleve | 15:15 |
| corvus | hrm, i must be missing something. | 15:15 |
| *** Vadmacs has joined #openstack-infra | 15:23 | |
| *** ykarel|away has quit IRC | 15:25 | |
| corvus | du reports 4.1G used by repos. and 1.2G by gitea (indexes, sessions). | 15:25 |
| corvus | that's all there is on the filesystem. | 15:26 |
| * corvus learns more about cephfs | 15:26 | |
| *** dhill_ has quit IRC | 15:43 | |
| corvus | mordred, fungi: i think the current issue with availability is a ceph problem. if i strace the gitea process and hit the url, i see it get the request and attempt to do work. but i think there's a hung IO call on that machine: https://etherpad.openstack.org/p/OYJJLz0xlx | 15:51 |
| corvus | (i don't think the load balancer is implicated) | 15:51 |
| fungi | got it | 15:51 |
| corvus | oh yeah, there's a "ceph_fsync" in that call stack | 15:52 |
| *** e0ne has quit IRC | 15:52 | |
| *** e0ne has joined #openstack-infra | 16:09 | |
| openstackgerrit | melissaml proposed openstack-infra/infra-specs master: remove the repetition words in docs https://review.openstack.org/627040 | 16:14 |
| *** tosky has quit IRC | 16:19 | |
| *** bhavikdbavishi has joined #openstack-infra | 16:24 | |
| *** e0ne has quit IRC | 16:25 | |
| openstackgerrit | melissaml proposed openstack-infra/infra-specs master: remove the repetition words in docs https://review.openstack.org/627040 | 16:25 |
| corvus | apparently cephfs clients can hang indefinitely waiting to write data out when the filesystem is full (but should proceed when that is corrected) | 16:28 |
| corvus | so these 2 problems may be related | 16:29 |
| corvus | i do not understand how 5G used fills up 227G of space. | 16:29 |
| *** eernst has joined #openstack-infra | 16:36 | |
| *** rkukura has quit IRC | 16:41 | |
| *** bhavikdbavishi has quit IRC | 16:52 | |
| *** markvoelker has quit IRC | 16:52 | |
| *** markvoelker has joined #openstack-infra | 16:53 | |
| *** markvoelker has quit IRC | 16:57 | |
| *** armax has joined #openstack-infra | 16:59 | |
| *** eernst has quit IRC | 17:02 | |
| *** armax has quit IRC | 17:04 | |
| *** hwoarang has quit IRC | 17:15 | |
| *** hwoarang has joined #openstack-infra | 17:16 | |
| *** spa-87 has quit IRC | 17:29 | |
| corvus | okay, my theory is that ceph's bluestore driver is using 64k minimum allocation size (which is apparently its default on hdds) rather than 4k. we have lots of small files, so that exacerbates the space used. the math almost works out -- our number of objects*3*64k is 245GiB; we're using 227GiB. i don't know if the estimated usage being greater than the actual means that is wrong, or if i'm just | 17:36 |
| corvus | missing something. | 17:36 |
| corvus | at any rate, it's in the ballpark, and it's the only explanation i've come up with for the discrepancy. | 17:36 |
| corvus | i'd like to confirm the min_alloc_size, but it doesn't show up in our config file, so it must be using one of the implied defaults. but i don't know how to verify what the value is. | 17:37 |
| corvus | and it's time for me to afk | 17:37 |
| persia | 245GiB is 228GB by my calculations. Are you sure both are measured in GiB? | 17:38 |
| corvus | persia: fairly; ceph seems to consistently use GiB | 17:39 |
| persia | Also, I fail at units. 228GiB approximates 245GB. | 17:39 |
| fungi | seems like a reasonable theory, yes | 17:45 |
| *** bobh has joined #openstack-infra | 17:57 | |
| *** jamesmcarthur has joined #openstack-infra | 18:19 | |
| *** bobh has quit IRC | 18:23 | |
| *** bobh has joined #openstack-infra | 18:28 | |
| *** markvoelker has joined #openstack-infra | 18:29 | |
| *** bhavikdbavishi has joined #openstack-infra | 18:30 | |
| *** bobh has quit IRC | 18:38 | |
| *** bobh has joined #openstack-infra | 18:41 | |
| *** dave-mccowan has joined #openstack-infra | 18:42 | |
| *** ociuhandu has joined #openstack-infra | 18:52 | |
| *** ociuhandu has quit IRC | 18:52 | |
| *** bobh has quit IRC | 18:56 | |
| *** spa-87 has joined #openstack-infra | 19:01 | |
| *** bhavikdbavishi has quit IRC | 19:12 | |
| *** jamesmcarthur has quit IRC | 19:24 | |
| *** spa-87 has quit IRC | 20:12 | |
| *** jamesmcarthur has joined #openstack-infra | 20:32 | |
| *** e0ne has joined #openstack-infra | 20:40 | |
| *** dave-mccowan has quit IRC | 20:55 | |
| *** Vadmacs has quit IRC | 21:06 | |
| *** markvoelker has quit IRC | 21:30 | |
| *** markvoelker has joined #openstack-infra | 21:30 | |
| *** slaweq has joined #openstack-infra | 21:33 | |
| *** e0ne has quit IRC | 21:33 | |
| *** markvoelker has quit IRC | 21:35 | |
| *** pcaruana has joined #openstack-infra | 21:53 | |
| *** jaosorior has quit IRC | 22:07 | |
| *** slaweq has quit IRC | 22:13 | |
| *** jamesmcarthur has quit IRC | 22:22 | |
| *** gary_perkins has quit IRC | 22:25 | |
| *** jamesmcarthur has joined #openstack-infra | 22:26 | |
| *** gary_perkins has joined #openstack-infra | 22:28 | |
| *** gary_perkins has quit IRC | 22:37 | |
| *** gary_perkins has joined #openstack-infra | 22:44 | |
| *** gary_perkins has quit IRC | 22:51 | |
| corvus | okay, i feel like there must have been a better way to do this, but i think i have the answer i was looking for. | 22:53 |
| corvus | i deleted the k8s deployment and replicaset objects for the ceph osd1 pod with --cascade=false, so the pod was left running | 22:53 |
| *** gary_perkins has joined #openstack-infra | 22:53 | |
| corvus | then i deleted the osd1 pod. without the deployment and replicasets, the pod was not restarted | 22:54 |
| *** pcaruana has quit IRC | 22:55 | |
| corvus | i then made a copy of the osd1 directory (this directory is part of ceph which basically just has some metadata about where the actual storage is, along with some symlinks to the underlying storage device) and placed that copy on the ceph tools pod, after making sure the ceph tools pod was running on the k8s node that osd1 used to run on | 22:55 |
| corvus | all of that was so that i can run "ceph-kvstore-tool" which, when run against a bluestore, wants the bluestore to be un-mounted. | 22:55 |
| corvus | i ran "ceph-kvstore-tool bluestore-kv ./osd1/ get S min_alloc_size" | 22:55 |
| corvus | and it output: | 22:56 |
| corvus | (S, min_alloc_size) | 22:56 |
| corvus | 00000000 00 00 01 00 00 00 00 00 |........| | 22:56 |
| corvus | which, when i compensate for endianness seems to give me the number 65536 | 22:56 |
| corvus | (i believe it's a uint64_t) | 22:56 |
| corvus | and now back to afking | 22:57 |
| *** gary_perkins has quit IRC | 23:12 | |
| *** xek has quit IRC | 23:23 | |
| *** markvoelker has joined #openstack-infra | 23:31 | |
| *** Adri2000 has quit IRC | 23:47 | |
| *** Adri2000 has joined #openstack-infra | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!