| opendevreview | Martin Kopec proposed openstack/tempest master: Make plugin sanity check job voting again https://review.opendev.org/c/openstack/tempest/+/912372 | 10:09 |
|---|---|---|
| opendevreview | Clark Boylan proposed openstack/devstack master: Add eBPF bcc tool profiling https://review.opendev.org/c/openstack/devstack/+/912474 | 16:21 |
| clarkb | dansmith: ^ I discovered these tools over the weekend. I suspect they may end up being super useful for devstack debugging and profiling | 16:22 |
| clarkb | I'm starting with a couple of simple ones, but there are a bucnh of tools that I think may be relevant to us so that change will probably grow new functioanlity over time if it doesn't merge | 16:22 |
| dansmith | cool, will look in a bit | 16:22 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Add eBPF bcc tool profiling https://review.opendev.org/c/openstack/devstack/+/912474 | 16:52 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Add eBPF bcc tool profiling https://review.opendev.org/c/openstack/devstack/+/912474 | 17:40 |
| clarkb | the assumption that a service maps exactly to one process is a bit annoying when you want a bunch of little processes under a common enablement flag, but we can make this work | 17:41 |
| dansmith | yeah, looks interesting.. by choosing swapin you're looking for things that are using lots of memory and causing thrashing themselves? | 18:05 |
| dansmith | I assume if there's just lots of memory pressure you'll see multiple things needing to read from swap and it won't necessarily be any specific culprit | 18:05 |
| dansmith | but if there is one hot spot that'd be useful | 18:06 |
| clarkb | dansmith: I think swapin is going to show us things that are fetching memory out of swap a lot. It is possible those things might not use a ton of memory | 18:06 |
| clarkb | and ya it might not tell us anything useful in the end but we won't know until we profile it | 18:06 |
| clarkb | there are also disk/file/block io profiling tools in there that I want to use | 18:06 |
| clarkb | https://github.com/iovisor/bcc/tree/master/tools shows all the tools and example usage | 18:07 |
| clarkb | Once I haev this generally working we can add a bunch of those tools that we think are useful then see which end up being helpful | 18:07 |
| clarkb | filesslower, runqlat, kvmexit (though this likely won't work wtih qemu), and ustat/pythonstat, may be good ones to look at next | 18:08 |
| clarkb | maybe slabratetop | 18:09 |
| clarkb | really there are so many interesting things in here it may take some time to figure out where we get the most insight for devstack | 18:09 |
| dansmith | yeah I skimmed | 18:10 |
| opendevreview | Douglas Mendizábal proposed openstack/tempest master: Enable Secure RBAC in Keystone https://review.opendev.org/c/openstack/tempest/+/912489 | 19:05 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Add eBPF bcc tool profiling https://review.opendev.org/c/openstack/devstack/+/912474 | 21:04 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Add eBPF bcc tool profiling https://review.opendev.org/c/openstack/devstack/+/912474 | 21:30 |
| opendevreview | Clark Boylan proposed openstack/devstack master: Add eBPF bcc tool profiling https://review.opendev.org/c/openstack/devstack/+/912474 | 22:06 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!