| *** ykarel|away is now known as ykarel | 04:46 | |
| *** bhagyashris_ is now known as bhagyashris | 05:39 | |
| *** rpittau|afk is now known as rpittau | 07:23 | |
| *** jpena|off is now known as jpena | 07:32 | |
| zbr | apparently IncompleteRead with mirror.bhs1.ovh.opendev.org on https://zuul.opendev.org/t/openstack/build/efc4a37164974cce98b128e203871abc | 10:45 |
|---|---|---|
| opendevreview | Merged openstack/project-config master: afsdocs_secret-tox-docs-site: Zuul 4.6.0 fix https://review.opendev.org/c/openstack/project-config/+/804170 | 11:12 |
| opendevreview | Andreas Jaeger proposed openstack/project-config master: Adjust secrets for developer.o.o https://review.opendev.org/c/openstack/project-config/+/804226 | 11:14 |
| *** rpittau is now known as rpittau|afk | 11:28 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** rlandy is now known as rlandy|ruck | 11:47 | |
| jcapitao | ykarel: ack, let me check | 12:27 |
| *** jpena|lunch is now known as jpena|off | 12:28 | |
| ykarel | jcapitao, wrong channel :) | 12:34 |
| jcapitao | yup sorry folks ! | 12:35 |
| zbr | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3AIncompleteRead | 12:39 |
| zbr | this tells me that ovh-bhs1 mirror is borked | 12:39 |
| zul | got a question, why hasnt this merged yet? https://review.opendev.org/c/starlingx/config/+/802512/5 | 13:15 |
| *** ykarel is now known as ykarel|away | 13:23 | |
| *** abhishekk is now known as akekane|home | 13:44 | |
| *** akekane|home is now known as abhishekk | 13:44 | |
| fungi | zbr: looks like there's a segfault from apache at 10:18:55 utc in that server's dmesg today | 13:55 |
| zbr | lots of brokeb builds caused by this, also it would work considering how easy will be to identify these when logstash is gone | 13:55 |
| fungi | aha, here's why: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=68218&rra_id=all | 13:56 |
| fungi | ran out of room for apache's cache | 13:56 |
| fungi | we may be caching too much, or not expiring things from the cache readily enough | 13:56 |
| fungi | zbr: it will be harder to identify, but think how much worse off the project will be if the few people we have maintaining our code review and ci systems stop doing that to build and take care of a new elastic-recheck system | 13:57 |
| fungi | or we could stop mirroring things, i guess | 13:58 |
| fungi | i mean, there's lots of stuff we *could* stop running instead of elastic-recheck | 13:58 |
| fungi | there's an htcacheclean process on that mirror which has been trying to complete since july 21 | 13:59 |
| fungi | i've restarted apache completely just to make sure it's sane for the moment | 14:00 |
| fungi | i think aiming to keep the apache cache at or below 70% was a bad choice, i'll see if i can convince the other sysadmins to drop it back to 50% again | 14:01 |
| fungi | it looks like we can easily dump 30gb of data into that cache in a matter of a few hours, much faster than the daily htcacheclean we run, but also it looks like htcacheclean itself is unable to keep up | 14:02 |
| zbr | this sounds bit weird,maybe runnning more often than every day? | 14:04 |
| fungi | oh, sorry it's hourly not daily | 14:04 |
| fungi | starts at the top of the hour, so another kicked off at 14:00 utc | 14:04 |
| fungi | though i don't know how long they take to complete, looks like the one which was still running from july was hung and doing nothing but was still holding the lock | 14:05 |
| fungi | i'll monitor this and see if i can tell roughly how long the new runs take | 14:05 |
| fungi | zul: it's a little hard to tell with everything going on there... it has a bunch of not-yet-merged parent changes though. are you asking why all of those haven't merged yet? | 14:10 |
| fungi | zul: it looks like the earliest ancestor of that change which hasn't merged yet is 802504 | 14:12 |
| fungi | zul: 802504 got approved at 13:38 and zuul enqueued it into the gate pipeline, but then 5 minutes later at 13:43 you pushed a new patchset for it so its earlier patchset was dequeued from the gate pipeline and the new revision tested in check (where it failed) | 14:13 |
| fungi | (times in utc) | 14:13 |
| zul | fungi: fack ok | 14:16 |
| fungi | if those changes don't actually depend on one another, you could push revisions which don't have the others as ancestors and are just relative to the current state of the target branch | 14:17 |
| fungi | when chains of changes get really long like that, they can become unwieldy to manage and keep current with one another | 14:17 |
| fungi | the 14:00z htcacheclean on the bhs1 mirror did complete its run in under an hour (just finished in the last few minutes) | 14:51 |
| *** jpena|off is now known as jpena | 14:57 | |
| clarkb | fungi: the reason for the larger cache size is docker basically | 15:12 |
| clarkb | fungi: just a few sets of images are enough to fill the cache up | 15:12 |
| clarkb | if we are too conservative with our cache size we'll end up not caching a ton of stuff aiui | 15:12 |
| clarkb | it is definitely a balancing act between having as much cache as possible so that we actually cache useful data and not overfilling the cache preventing cleanup from keeping up | 15:14 |
| *** sshnaidm is now known as sshnaidm|afk | 15:35 | |
| *** jpena is now known as jpena|off | 15:42 | |
| fungi | yeah, we basically ate the open 30% of that cache volume in a matter of ~2 hours | 15:57 |
| fungi | according to the graph | 15:57 |
| *** ykarel is now known as ykarel|away | 16:33 | |
| *** rlandy|ruck is now known as rlandy|ruck|afk | 16:51 | |
| zul | fungi: any idea why this one is being held up? https://review.opendev.org/c/starlingx/ansible-playbooks/+/802381 | 17:12 |
| fungi | zul: its parent hasn't been approved yet | 17:14 |
| fungi | https://review.opendev.org/795405 | 17:14 |
| zul | how do you find the parent? | 17:15 |
| zul | doh never mind | 17:16 |
| fungi | zul: the "relation chain" list in the upper right of the change view, or by looking at the parent commit in the change details (underneath where it lists the repository and branch). the former will take you to the relevant change in review, the latter shows you a git commit view | 17:16 |
| zul | yeah i just noticed :) | 17:17 |
| fungi | it's not the best designed ui, i get lost all the time | 17:17 |
| fungi | for example, i can never remember which order the relation chain is listed in (it's newest first, oldest or closest to the branch tip last) | 17:18 |
| zul | yeah its starting to get to me :0 | 17:18 |
| opendevreview | Clark Boylan proposed openstack/project-config master: Disable airship citylcoud nodepool provider https://review.opendev.org/c/openstack/project-config/+/804281 | 17:19 |
| zul | one last question how do you remove something from the chain? | 17:19 |
| fungi | easiest way is to `git rebase -i` and then delete the patch you don't want as a parent from the list of commits to be applied | 17:19 |
| zul | thanks | 17:20 |
| fungi | though be forewarned that if you've done a `git remote update` or similar, that may rebase whatever commits you're keeping onto a newer branch state too | 17:20 |
| fungi | i use https://pypi.org/project/git-restack/ as a convenience subcommand to avoid that | 17:21 |
| clarkb | I have a long standing bug upstream that they should consider a subway map representation instead (however I'm sure it is much more difficult to do right) | 17:21 |
| fungi | it is basically rebase -i but on whatever commit is common between your current branch and the remote | 17:21 |
| opendevreview | Radosław Piliszek proposed openstack/project-config master: Allow kolla cores to toggle kolla wipstate https://review.opendev.org/c/openstack/project-config/+/804283 | 17:29 |
| opendevreview | Merged openstack/project-config master: Disable airship citylcoud nodepool provider https://review.opendev.org/c/openstack/project-config/+/804281 | 17:39 |
| Roamer` | FTR, this can be done manually - running `git log` and then `git rebase -i a0b1cc3ff3`, specifying a commit ID *earlier* than the one that needs to be removed or modified or whatever... but yeah, there are tools that automate that | 17:39 |
| Roamer` | (I personally always keep local branches that are exactly the same as remote branches, so `git rebase master` or `git rebase stable/wallaby` is easy) | 17:40 |
| Roamer` | `-i`, of course, pfffth | 17:41 |
| fungi | yep, totally agree | 17:41 |
| fungi | i used to do exactly what you describe, but git restack is essentially a shortcut for that | 17:41 |
| clarkb | I do a lot of git rebase -i HEAD~X to reorder/squash etc. I find writing a commit at the end of the stack to squash back is the easiest way to curate long stacks too | 17:41 |
| opendevreview | Merged openstack/project-config master: Allow kolla cores to toggle kolla wipstate https://review.opendev.org/c/openstack/project-config/+/804283 | 17:56 |
| *** rlandy|ruck|afk is now known as rlandy|ruck | 18:59 | |
| *** ssamal is now known as ssamal|afk | 19:29 | |
| *** ssamal|afk is now known as ssamal | 22:25 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!