| opendevreview | Merged openstack/devstack stable/train: Mark our source trees as safe for git to use as other users https://review.opendev.org/c/openstack/devstack/+/838556 | 03:19 |
|---|---|---|
| opendevreview | Benny Kopilov proposed openstack/tempest master: Fix remote_client param description https://review.opendev.org/c/openstack/tempest/+/838812 | 04:38 |
| *** pojadhav is now known as pojadhav|ruck | 04:55 | |
| opendevreview | ZhouYanbing proposed openstack/devstack master: modify the sample value of LOGDAYS https://review.opendev.org/c/openstack/devstack/+/838829 | 07:02 |
| *** jpena|off is now known as jpena | 07:11 | |
| *** pojadhav|ruck is now known as pojadhav|lunch | 08:07 | |
| *** pojadhav|lunch is now known as pojadhav|ruck | 09:21 | |
| opendevreview | Eduardo Olivares proposed openstack/tempest master: Validate network downtime during live migration https://review.opendev.org/c/openstack/tempest/+/828686 | 09:23 |
| opendevreview | Merged openstack/devstack stable/rocky: Mark our source trees as safe for git to use as other users https://review.opendev.org/c/openstack/devstack/+/838681 | 11:09 |
| *** pojadhav|ruck is now known as pojadhav|brb | 11:22 | |
| opendevreview | Eduardo Olivares proposed openstack/tempest master: Validate network downtime during live migration https://review.opendev.org/c/openstack/tempest/+/828686 | 11:53 |
| *** pojadhav|brb is now known as pojadhav|ruck | 11:54 | |
| *** bhagyashris_ is now known as bhagyashris | 11:55 | |
| opendevreview | ribaudr proposed openstack/tempest master: [WIP] Test tempest for Nova and Manila https://review.opendev.org/c/openstack/tempest/+/838743 | 14:59 |
| *** pojadhav|ruck is now known as pojadhav|out | 15:59 | |
| *** jpena is now known as jpena|off | 16:15 | |
| dansmith | gmann: thinking about something like this, at least as a baseline: https://termbin.com/rls0 | 17:58 |
| gmann | dansmith: looks good but comparison is with previous run on same patch? or how you collect? | 18:00 |
| dansmith | that comparison is between a full-py3 and a nova-ceph-multistore of the same patch | 18:00 |
| dansmith | the two urls there are what it's comparing | 18:00 |
| dansmith | but you can just link to any two performance.jsons | 18:01 |
| dansmith | should also add some shortcuts like using gerrit review,patch ids or something | 18:01 |
| dansmith | it's very interesting to me that we did 50% more neutron DB queries on nova-ceph-multistore vs full-py3 | 18:01 |
| dansmith | and 50% more placement queries too, which seems weird | 18:02 |
| dansmith | also, meant to ask earlier, but it seems like we're mounting neutron on the root of the tls proxy, instead of under /network | 18:03 |
| gmann | +1, that is great data. I am juts thinking if we should take tempest-full-py3 always as base or some static data we collect today and that we compare in every job | 18:03 |
| dansmith | so all the neutron queries show as v2.0 because they're literally http://host/v2.0/subnets | 18:03 |
| dansmith | gmann: yeah, so I was thinking maybe if I could figure out what the url is to a weekly periodic job, I could compare $current against that | 18:04 |
| gmann | yeah that is much better. just wondering if periodic job is something high in query. but at least that is good start. | 18:07 |
| dansmith | what I mean is, | 18:09 |
| dansmith | for a given patch, for each job on that patch, if I could find the url to a recent periodic, then I could compare the $current_patch[job] to $latest_periodic[job] | 18:09 |
| clarkb | dansmith: you can query builds by pipeline and grab the last two in periodic then compare those via the zuul api | 18:09 |
| dansmith | to have some idea if the current patch is regressing against the current | 18:10 |
| clarkb | let me see if I can get a url for that | 18:10 |
| dansmith | clarkb: ack, I can probably figure that out from dash.py's use of zuul data | 18:10 |
| dansmith | clarkb: better would be if I can ask graphite what the 7-day average for $job is for a given metric value | 18:11 |
| clarkb | ya would be elasticsearch I think but same difference | 18:11 |
| dansmith | but "value from the last periodic of this" would give me at least a point in time comparison between unmerged and merged | 18:11 |
| clarkb | grafana would graph it out of elasticsearch | 18:11 |
| dansmith | yeah, $backend I mean | 18:11 |
| clarkb | https://zuul.opendev.org/api/tenant/openstack/builds?job_name=tempest-all&project=openstack%2Ftempest&pipeline=periodic&skip=0&limit=5 | 18:13 |
| clarkb | something like that for the zuul api | 18:13 |
| gmann | dansmith: ok, sounds good for checking if that patch is causing regression but in case that is missed to notice then after some time $latest_periodic will have same perf and then we will be saying all good on later patches | 18:13 |
| dansmith | clarkb: yeah sweet | 18:14 |
| clarkb | that includes the log url for each build and you can construct the path to the stats file from that | 18:14 |
| gmann | along with this I was thinking if we can tell project you have regressed from start of cycle and it is not fixed yet. | 18:14 |
| dansmith | gmann: yeah, but after 7 days we've lost all our history so it'll be hard to do that for longer cycles | 18:14 |
| dansmith | gmann: yeah, that would be great, but we need some other persistence for that I think | 18:15 |
| gmann | dansmith: I am thinking to store the current data in file in devstack and then compare that. | 18:15 |
| gmann | we can update that if a valid hike from any project like new feature/tests | 18:15 |
| dansmith | gmann: yep, we could do something like that, at least initially | 18:15 |
| clarkb | dansmith: re 7 days that is why I suggested dpawlik store it in a different index that doesn't expire so quickly | 18:15 |
| dansmith | clarkb: ah | 18:15 |
| clarkb | dansmith: I think the data set will be significantly smaller that way and you can do say a 3 month rotation on it | 18:15 |
| dansmith | yeah | 18:16 |
| dansmith | gmann: so if we did files, we would need something like data/$project/$job.json yeah? | 18:16 |
| dansmith | gmann: we could start with just nova or something and see how that goes for a few weeks | 18:16 |
| gmann | dansmith: like 1. static file comparison to know overall regression 2. $latest_periodic comparision to know which patch is causing it if team are active to notice that | 18:17 |
| gmann | dansmith: +1. | 18:17 |
| dansmith | yeah | 18:17 |
| dansmith | gmann: do we have the job name in a variable during a run? | 18:40 |
| gmann | dansmith: not sure, may be clarkb knows | 18:40 |
| dansmith | okay that'll be pretty important :P | 18:40 |
| clarkb | if you look in the logs for the job there is the inventory file and that should have the non secret vars | 18:40 |
| clarkb | https://zuul.opendev.org/t/openstack/build/5ab74c10274d49bca26fb8b3eadd5b4d/log/zuul-info/inventory.yaml#290 would be {{ zuul.job }} in ansible | 18:41 |
| dansmith | ah nice | 18:42 |
| dansmith | I've never looked in this directory, but..suuper handy :) | 18:42 |
| sean-k-mooney | hum i proably shoudl make zuul.job to my molecue senarior in my ansible repo | 19:34 |
| sean-k-mooney | clarkb: dansmith by the way for devstack change shoudl i file bugs for everythign im fixing in general or is it ok to just submit patches | 19:35 |
| sean-k-mooney | i was never quite sure how strict we were with that for devstack | 19:35 |
| sean-k-mooney | im going to fix the package ordering issue in https://review.opendev.org/c/openstack/devstack/+/838043 now but just wondering if i shoudl file a bug or not | 19:36 |
| opendevreview | sean mooney proposed openstack/devstack master: install mod_ssl on centos 9 stream by default https://review.opendev.org/c/openstack/devstack/+/838043 | 19:41 |
| dansmith | sean-k-mooney: I dunno, I defer to gmann, et al | 19:44 |
| sean-k-mooney | ok well that should pass check this time so if i respin i can file one if people want | 19:46 |
| sean-k-mooney | i think im going to finish there for today. | 19:46 |
| sean-k-mooney | clarkb: i might take another look at the global venv patch on monday | 19:47 |
| opendevreview | Dan Smith proposed openstack/devstack master: WIP: Test static perfdata comparisons https://review.opendev.org/c/openstack/devstack/+/838947 | 20:00 |
| dansmith | gmann: quick hacky attempt ^ | 20:00 |
| dansmith | the nova patch to generate the data is still running, but I'll point it at this when it's done to test | 20:01 |
| dansmith | not sure the project/job hierarchy is necessary, but seems like it's probably safest to avoid potential collisions, but I'll take advice on that | 20:01 |
| opendevreview | Dan Smith proposed openstack/devstack master: WIP: Test static perfdata comparisons https://review.opendev.org/c/openstack/devstack/+/838947 | 20:10 |
| clarkb | sean-k-mooney: thanks. I'm definitely getting sucked into other stuff again now | 20:20 |
| clarkb | dansmith: I'm revieiwing https://review.opendev.org/c/openstack/devstack/+/837139 and it seems we don't do apache logs by default because we don't seta default for debuntu/suse or red hat file locations? | 21:08 |
| clarkb | am I reading that correctly or have I missed where we set a non empty list value? | 21:08 |
| dansmith | not sure what you mean | 21:08 |
| clarkb | dansmith: https://review.opendev.org/c/openstack/devstack/+/837139/20/roles/capture-performance-data/tasks/main.yaml#12 lines 12-15 there dfeault to [] for those values and the role doesn't seem to set that value | 21:09 |
| dansmith | if there are logs, then it does.. that's how we're seeing the api request data | 21:09 |
| dansmith | like here https://85682c22f75746955b38-998a6cfec97762292b03290ad6103366.ssl.cf5.rackcdn.com/837139/20/check/nova-ceph-multistore/d3c4ea2/controller/logs/performance.json | 21:10 |
| clarkb | hrm what sets those vars then? Is it from other devstack stuff? a fact maybe? | 21:10 |
| dansmith | under root['api'] | 21:10 |
| dansmith | yeah those are set by the collect-apache-logs thing | 21:10 |
| dansmith | which is why I rearranged my thing after that role runs | 21:10 |
| clarkb | aha yup its in there. Got it | 21:10 |
| dansmith | https://review.opendev.org/c/openstack/devstack/+/837139/20/playbooks/post.yaml | 21:11 |
| dansmith | apache-logs-conf that is | 21:11 |
| clarkb | dansmith: ok I've approved it | 21:26 |
| clarkb | probably the only thing we really need to look out for is mysql having a sad collecting the perf data? | 21:26 |
| gmann | sean-k-mooney: bug is not needed as such as you have changes up. if it need more changes and more testing then yes bug will be good to track the work | 21:36 |
| gmann | dansmith: ack, will check it | 21:41 |
| dansmith | clarkb: having a sad? | 21:51 |
| dansmith | clarkb: I compared some runtimes and it doesn't appear to impact much that I can tell | 21:51 |
| dansmith | it's also not clear to me that it really logs every select either, not sure why | 21:51 |
| dansmith | like maybe if it's cached or comes purely from an index or something | 21:52 |
| clarkb | dansmith: ya I don't expect problems but mysql often surprises me | 21:52 |
| dansmith | yeah I initially defaulted it to off because I was worried about it, but.. so far haven't noticed | 21:52 |
| opendevreview | Dan Smith proposed openstack/devstack master: WIP: Test static perfdata comparisons https://review.opendev.org/c/openstack/devstack/+/838947 | 21:59 |
| opendevreview | Merged openstack/devstack master: Gather performance data after tempest https://review.opendev.org/c/openstack/devstack/+/837139 | 23:06 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!