| sean-k-mooney | we were previously building the release notes without using the tox jobs defs form the proejcts | 03:56 |
|---|---|---|
| sean-k-mooney | so they may be missing or broken on some stable branches | 03:56 |
| sean-k-mooney | i dboubt that the case for ironic bug i know watcher-dashbaord didnt install reno in its docs requiremnt until a week or two ago | 03:57 |
| sean-k-mooney | https://github.com/openstack/watcher-dashboard/commit/6d60804fdfb462bb2781ad020de981d594416f23 | 03:57 |
| sean-k-mooney | that was not previosly requried for the jobs to work but we have not backprote that to all of stable | 03:58 |
| sean-k-mooney | with that said nova was also broken by exist vs exists https://opendev.org/openstack/openstack-zuul-jobs/commit/e3089488531677ca6ed629db94aaeb47c88bac7e | 04:00 |
| sean-k-mooney | so ya htere were a couple of issue with tha role. | 04:01 |
| opendevreview | Merged openstack/openstack-zuul-jobs master: Remove bionic nodeset from unit and functional test jobs https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/958668 | 06:08 |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Remove lower constrains for opensearch-py https://review.opendev.org/c/openstack/ci-log-processing/+/993627 | 06:45 |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Bump OpenSearch and OpenSearch Dashboards to 3.5 https://review.opendev.org/c/openstack/ci-log-processing/+/993618 | 07:17 |
| opendevreview | Merged openstack/ci-log-processing master: Remove lower constrains for opensearch-py https://review.opendev.org/c/openstack/ci-log-processing/+/993627 | 07:27 |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Update Python to 3.12 https://review.opendev.org/c/openstack/ci-log-processing/+/993716 | 07:44 |
| *** elodilles_pto is now known as elodilles | 07:50 | |
| opendevreview | Merged openstack/ci-log-processing master: Bump OpenSearch and OpenSearch Dashboards to 3.5 https://review.opendev.org/c/openstack/ci-log-processing/+/993618 | 07:54 |
| *** benj_5 is now known as benj_ | 08:10 | |
| stephenfin | fungi: Understood. I need to figure out how to actually test the thing next time. Reverting was the best thing to do alright | 08:29 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Remove packaging configuration https://review.opendev.org/c/openstack/project-config/+/993628 | 09:27 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Apply ruff https://review.opendev.org/c/openstack/project-config/+/993629 | 09:27 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: Bump Python versions when creating a new release https://review.opendev.org/c/openstack/project-config/+/993630 | 09:27 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: trivial: Fix shellcheck issues https://review.opendev.org/c/openstack/project-config/+/993631 | 09:27 |
| opendevreview | Stephen Finucane proposed openstack/project-config master: trivial: Add PEP 723 headers for Python scripts https://review.opendev.org/c/openstack/project-config/+/993632 | 09:27 |
| opendevreview | Merged openstack/ci-log-processing master: Update Python to 3.12 https://review.opendev.org/c/openstack/ci-log-processing/+/993716 | 11:18 |
| dtantsur | Hey folks! I vaguely remember that we cache cirros images on the testing nodes. Is it so? Where are they cached? | 11:38 |
| frickler | dtantsur: it is to, but I need to dig the details myself, give me a moment | 11:50 |
| frickler | dtantsur: this should be the list including paths on the node images we produce https://opendev.org/opendev/zuul-providers/src/branch/master/dib-elements/cache-devstack/source-repository-images | 11:52 |
| dtantsur | Thank you frickler! | 11:53 |
| opendevreview | Merged openstack/project-config master: [cinder] Make Review-Priority sticky https://review.opendev.org/c/openstack/project-config/+/993177 | 14:46 |
| opendevreview | Monty Taylor proposed openstack/project-config master: Add more repos for drizzle website and sysadmin automation https://review.opendev.org/c/openstack/project-config/+/993214 | 15:17 |
| frickler | #status log deleted the feature/policy-refactor branch from glance which was no longer needed | 16:05 |
| opendevstatus | frickler: finished logging | 16:05 |
| fungi | for those who missed it, mnasiadka is now co-charing the tact sig with me (thanks!!!): https://governance.openstack.org/tc/reference/sigs/ | 16:49 |
| mnasiadka | yay | 16:52 |
| mnasiadka | fungi: looking at the repo list - I was kind of interested where ci-log-processing is used, I probably have zero knowledge what logs land there and if we use that for anything | 16:53 |
| fungi | dpawlik can probably explain it | 16:53 |
| fungi | it's a bolted-on service primarily managed by him, to slurp build logs out of zuul and feed them into opensearch | 16:54 |
| clarkb | it was the replacement for the elasticsearch cluster the infra team ran when we decided continuing to maintain that was not tenable with the level of help we had | 16:58 |
| mnasiadka | Well, I’m pretty sure the mission is not just to store the logs in OpenSearch for the fun of it | 17:01 |
| clarkb | yes the idea was and continues to be that openstack developers can use it to do investigations into why gate jobs are failing | 17:03 |
| clarkb | and then use that info to ensure bugs actually get fixed and so on | 17:03 |
| clarkb | the old elastic-recheck tool went as far as having a bunch of bug signatures encoded so that it could provide a high level dashboard of where problems are and what percentage of failuers can be claissified | 17:04 |
| fungi | i got dpawlik to document the basics at https://governance.openstack.org/tc/reference/sigs/tact-sig.html#opensearch | 17:04 |
| fungi | (note it's not managed by the opendev sysadmins, and so not in our configuration management, but we allocated some cloud resources for the server) | 17:05 |
| clarkb | The problems we had were that the volume of data is quite high (to the point where terabytes of storage was only good for like a week of data and that was after we filtered out debug logs and only looked at info and above). Consistent problems with project teams not understanding basics of logging (eg things that identify failures should be more than just a debug log level | 17:06 |
| clarkb | and logs need timestamps and so on). Problems running elstaicsearch itself. It is hard to operate without losing data nevermind doing upgrades | 17:06 |
| clarkb | and eventually the infra team decided that they didn't have time to keep caring for it and the opensearch folks at amazon were willing to give us some credits if someone would care for it there and dpawlik stood up to do that | 17:06 |
| fungi | aws donating dedicated opensearch-as-a-service resources helps alleviate most of that workload, and so dpawlik is basically just manging the ingestion glue | 17:07 |
| mnasiadka | I’m happy that somebody is managing that glue, I’m rather asking if there is anybody that is using it (e.g. something like the old elastic -recheck tool) | 17:10 |
| clarkb | Its a system I am extremely proud to have helped build (I had the original idea back at the san diego summit, was that folsom?). I think it or something like it works really well for large scale sofwtare development of complicated software that is prone to race conditions and integration struggles | 17:11 |
| clarkb | but I also think that with a QA team on life support and the infra team needing to focus more on foundational tools like gerrit and zuul and so on it was the right decision to deprioritize it | 17:11 |
| clarkb | mnasiadka: ya its a good question. | 17:12 |
| clarkb | I don't use it when I'm asked to debug gate failures for openstack. | 17:12 |
| clarkb | I think most of the value is in tracking broad trends and prioritizing bugs that impact the gate as a whole. It is useful when debugging individual test jobs, but not as much so I don't tend to use it these days | 17:14 |
| mnasiadka | I think it’s nice we have that data, but we probably don’t use it in even similar way that elastic-recheck did, which could be a useful trend analysis saying stop rechecking, fix your code because it already happened 17 times this month | 17:16 |
| fungi | yes, part of the reason for just allocating some cloud resources to it and handing it off is that there were definitely people who complained when we took the old service down, and so we were able to satisfy those complaints while allowing them to take on the responsibility for its replacement | 17:16 |
| fungi | at this point i'd say the only person who could reliably answer the question of its usefulness is dpawlik | 17:17 |
| fungi | and i hope that he wouldn't continue to maintain it if he determined that there were no users | 17:17 |
| fungi | at the time, it was apparently integral to the operation of red hat's downstream ci/cd operations related to tripleo, but it's possible their reliance on it shifted when tripleo was eventually wound down | 17:19 |
| frickler | well it is used from time to time by qa and nova ppl at least, for example to track this special case of cirros related issues | 17:29 |
| frickler | https://opensearch.logs.openstack.org/_dashboards/app/data-explorer/discover/#?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-14d,to:now))&_a=(discover:(columns:!(hosts_region,project,build_patchset,build_change),interval:d,isDirty:!t,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_q=(filters:!(),query:(language:kuery,query:'message:%22K | 17:29 |
| frickler | ernel%20panic%20-%20not%20syncing:%20Attempted%20to%20kill%20init!!%22')) | 17:29 |
| frickler | but the urls are nasty, one day I'll get back to my url-shortener idea ;) | 17:30 |
| opendevreview | Merged openstack/project-config master: Add more repos for drizzle website and sysadmin automation https://review.opendev.org/c/openstack/project-config/+/993214 | 17:37 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Drop iotronic repos from zuul and IRC https://review.opendev.org/c/openstack/project-config/+/993820 | 18:03 |
| opendevreview | Merged openstack/project-config master: Drop iotronic repos from zuul and IRC https://review.opendev.org/c/openstack/project-config/+/993820 | 19:19 |
| opendevreview | Dr. Jens Harbott proposed openstack/openstack-zuul-jobs master: Job cleanup for outdated node types https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/993669 | 20:45 |
| opendevreview | Dr. Jens Harbott proposed openstack/project-config master: Cleanup some projects' zuul definitions https://review.opendev.org/c/openstack/project-config/+/993840 | 20:54 |
| opendevreview | Dr. Jens Harbott proposed openstack/openstack-zuul-jobs master: Job cleanup for outdated node types https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/993669 | 21:00 |
| clarkb | I went ahead an single core approved 993840 as it seems straightforward. | 21:01 |
| clarkb | frickler: and for 993669 I left a note but +2'd I think we can merge it as is if you'd prefer to not update per my comment | 21:05 |
| frickler | clarkb: thx for these, I have more cleanups in the pipeline, so I'd be fine to merge both as they are now and do more cleanup later | 21:09 |
| clarkb | that works for me | 21:10 |
| clarkb | I'll let you decide if you want to single core approv 993669 then? | 21:11 |
| opendevreview | Merged openstack/project-config master: Cleanup some projects' zuul definitions https://review.opendev.org/c/openstack/project-config/+/993840 | 21:11 |
| frickler | done, thx, will continue tomorrow | 21:13 |
| opendevreview | Merged openstack/openstack-zuul-jobs master: Job cleanup for outdated node types https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/993669 | 21:25 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!