| *** rlandy|ruck|afk is now known as rlandy|ruck | 01:24 | |
| *** ministry is now known as __ministry | 02:26 | |
| *** frenzy_friday is now known as frenzyfriday|sick | 04:17 | |
| *** fzzf1 is now known as fzzf | 05:41 | |
| *** gibi_pto_back_thu is now known as gibi | 07:56 | |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: DNM Initial project commit https://review.opendev.org/c/openstack/ci-log-processing/+/815604 | 09:46 |
|---|---|---|
| *** rlandy is now known as rlandy|ruck | 10:32 | |
| *** lbragstad4 is now known as lbragstad | 10:33 | |
| *** lbragstad1 is now known as lbragstad | 10:43 | |
| *** lbragstad7 is now known as lbragstad | 11:07 | |
| *** dviroel|rover|out is now known as dviroel|rover | 11:12 | |
| *** dpawlik1 is now known as dpawlik | 12:16 | |
| *** jcapitao is now known as jcapitao_lunch | 12:53 | |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: DNM Initial project commit https://review.opendev.org/c/openstack/ci-log-processing/+/815604 | 12:57 |
| amoralej | may i get reviews on https://review.opendev.org/c/openstack/project-config/+/811442/ and https://review.opendev.org/c/openstack/project-config/+/816465 when you have a chance? | 13:19 |
| *** jcapitao_lunch is now known as jcapitao | 14:00 | |
| EmilienM | hello everyone, do you know if there is something done in the nodepool instances to get rid of the crontab which runs apt-get unattended updates on Ubuntu? | 14:33 |
| fungi | oh, is that installed? i wouldn't have expected ubuntu-minimal elements to incorporate the unattended-upgrades package | 14:36 |
| fungi | i'll check our image build logs | 14:37 |
| opendevreview | daniel.pawlik proposed openstack/ci-log-processing master: Initial project commit https://review.opendev.org/c/openstack/ci-log-processing/+/815604 | 14:37 |
| fungi | EmilienM: i don't see it included in our ubuntu-focal node builds: https://nb01.opendev.org/ubuntu-focal-0000119527.log | 14:38 |
| fungi | EmilienM: do you have an example job log or something showing it present? | 14:38 |
| EmilienM | fungi: no it's not in opendev infra, it's in openlab, and I was wondering if you had a workaround | 14:39 |
| fungi | maybe it's getting pulled in as a dependency of some other package at job runtime | 14:39 |
| EmilienM | but I see that you don't install that deb so you don't hit the issue. | 14:39 |
| fungi | EmilienM: oh, yeah i would say the workaround is to build minimal images and then install your dependencies at job runtime | 14:39 |
| fungi | i didn't even realize openlab was still operating | 14:39 |
| fungi | haven't heard from them in years, not since mrhillsman moved on | 14:40 |
| EmilienM | fungi: gophercloud CI is running there now | 14:40 |
| EmilienM | fungi: BTW thx for your answer on the thread | 14:40 |
| fungi | you're welcome! | 14:40 |
| EmilienM | once we figure out if our existing contributors are fine with Gerrit & that there is no pushback from the opendev folks, I think we'll move forward | 14:41 |
| dpawlik | fungi, clarkb,: hey, finally I propose change ready for review https://review.opendev.org/c/openstack/ci-log-processing/+/815604 . Take a look when you have few min. Thans | 14:59 |
| dpawlik | melwitt: hey, the first step for moving logs to the Opensearch is ready to review. tl;dr logscraper will push logs to log2gearman services instead of post job "submit logs", that was doing that. PS available here ^^ | 15:03 |
| *** dviroel|rover is now known as dviroel|lunch|appt|afk | 15:03 | |
| melwitt | dpawlik: nice! will take a look | 15:07 |
| clarkb | EmilienM: fungi: there is also a disable flag in apt config somewhere too iirc | 15:17 |
| fungi | yeah, it can be turned off | 15:19 |
| fungi | though that doesn't help if you're turing it off during job runtime because of races with jobs running and trying to install packages at the same time unattended-upgrades has the package database locked | 15:20 |
| clarkb | ya definitely best to update the images | 15:22 |
| clarkb | dpawlik: feel free to discuss the opensearch stuff here. That way others can help if I'm not around | 16:16 |
| fungi | clarkb: https://review.opendev.org/816465 is going to fail to build images until we have centos stream 9 packages mirrored, right? | 16:27 |
| fungi | (since our nodepool builders rely on one of our mirrors to retrieve packages) | 16:27 |
| clarkb | fungi: looking at that I think we may only overload the mirrors for debuntu? Note the DIB_DISTRIBUTION_MIRROR vars set on the debuntu builds | 16:29 |
| fungi | ahh, okay | 16:29 |
| clarkb | do we still build stretch? we should clean that out if so I think | 16:29 |
| afaranha | fungi, clarkb quick question, regarding yesterday talk about zuul fips jobs. If we implement the unittest-fips and tox-fips on ozj, we need to copy the playbooks and roles as well, correct? for example, for unittest https://opendev.org/zuul/zuul-jobs/src/branch/master/zuul.d/general-jobs.yaml#L10-L18 we would need to copy playbooks/unittests/{pre,post}.yaml over to ozj. | 17:10 |
| afaranha | In this case it's better to create the fips version of unittest and tox on zj | 17:10 |
| clarkb | afaranha: you don't need to copy the roles. They are available. You would need to copy the playbooks | 17:11 |
| afaranha | clarkb, I'll leave it on zj then, better than have duplicated files, okay? | 17:14 |
| clarkb | sure | 17:19 |
| *** dviroel|lunch|appt|afk is now known as dviroel|rover | 17:20 | |
| fungi | yeah, that's what i was saying yesterday about leaving the roles where they are, you just need to duplicate job definitions (and their associated playbooks) | 17:21 |
| afaranha | ack | 17:42 |
| dpawlik | fungi: as I mention clarkb in priv message, opensearch service will be soon deployed, logscraper is in review state. I will do a PS with ansible role tomorrow and I guess next step will be add new vms to Openstack infrastructure that will have services like log2gearman, subunit2sql, logscraper | 17:47 |
| fungi | exciting! i hope it goes smoothly | 17:47 |
| dpawlik | should we "port" or copy the roles responsible for that services to ci-log-processing repo or it should be as it is? | 17:48 |
| fungi | none of those services were deployed with ansible roles, they'd need to be written from scratch i expect | 17:49 |
| clarkb | what is log2gearman? | 17:49 |
| *** jpena|off is now known as jpena | 17:49 | |
| dpawlik | sorry, gearman-client and gearman-worker | 17:50 |
| clarkb | ya that stuff was hidden in a puppet module. I think putting it into a repo that can be maintained (eg ci-log-processing) would be best | 17:50 |
| dpawlik | or if the hosts does not use all the resources the services can be added there | 17:50 |
| clarkb | I guess it isn't clear to me how much of the old stuff you are intending to keep (I thought fungi had mentioned not using gearman a few times since it is fairly legacy, but it does work so not the end of the world to keep using it) | 17:51 |
| amoralej | fungi, thanks for your reviews, anyway i'll also investigate to add centos-stream9 to the mirrors so that we can use it later | 17:51 |
| dpawlik | clarkb: I was thinking that service like subunit2sql is still required and IIUC https://docs.opendev.org/opendev/system-config/latest/logstash.html gearman client is required | 17:53 |
| clarkb | dpawlik: fungi: but ya in general I think the tools should be tools first and then deployed with config management rather than config management that happens to deploy tools | 17:53 |
| clarkb | dpawlik: well we said we're turning off subunit2sql and no one has figured out a replacement for it | 17:53 |
| dpawlik | clarkb: I can change the logscraper that will push the messages to the logstash directly, it should not be a problem | 17:53 |
| clarkb | dpawlik: similar to the ELK stuff it needs a giant database that needs care and feeding. Updating just the log processing part is only part of the problem | 17:53 |
| dpawlik | clarkb: ah, that is something I miss | 17:54 |
| clarkb | I was under the impression we were turning that off. Largely beacuse the openstack health service isn't used anymore (it has been broken for months and no one has asked about fixing it) | 17:54 |
| clarkb | gmann: fungi: ^ if I've missed something re subunit2sql and openstack health let me know | 17:54 |
| dpawlik | clarkb: in that case, I need to do small redesign of logscraper script | 17:54 |
| fungi | i assumed if people wanted a service similar to openstack-health there would be volunteers to set it up and run it | 17:55 |
| fungi | but yeah, we're still planning to turn it off (along with the old logstash et cetera) | 17:55 |
| clarkb | ok so ya I guess if dpawlik wants to keep it alive that is fine but you'd need to find a very large sql database for it (I think the current one is 500GB or 1TB or something) | 17:56 |
| fungi | the service is not in a maintainable state from a deployment and care and feeding standpoint | 17:56 |
| fungi | pretty sure it's currently a 500gb trove instance | 17:56 |
| clarkb | dpawlik: basically for now my recommendation is to focus only on the ELK related stuff | 17:56 |
| clarkb | dpawlik: that is what we have a replacement backend for | 17:56 |
| fungi | yeah, one thing at a time | 17:56 |
| gmann | clarkb: agree on turning it off but let me put it in QA meeting agenda also in case anyone would like to maintain it. | 17:57 |
| dpawlik | clarkb: so from my side, I don't use subunit2sql service, but Opensearch would be awesome to have will all the logs like it is right now. So the logscraper script was pushing the logs directly to gearman services that were sending data to the logstash | 17:57 |
| clarkb | dpawlik: got it, in that case keeping gearman for now and replacing it if it makes sense later is fine too. I just wanted ot make sure I understood what was being discussed | 17:58 |
| dpawlik | clarkb: cool | 17:59 |
| clarkb | to be extra clear the setup will be something like "Infra provided VMs without additional management to run gearman, scraper and log pusher" -> "AWS resources that run logstash and opensearch" | 17:59 |
| dpawlik | yup, that is ok for me | 18:00 |
| clarkb | ok cool I think I'm up to speed then | 18:00 |
| fungi | yeah, basically let us know what ram/cpu count and base distro image you need provisioned | 18:00 |
| clarkb | also I think mnaser_ indicated willingness to host stuff so we can/should spin that up in vexxhost likely | 18:01 |
| dpawlik | reed is it You that will maintain Opensearch too, right? (but from the AWS side) | 18:01 |
| fungi | dpawlik: that's probably not the reed you're thinking it is | 18:01 |
| fungi | (the reed in here is an emeritus community member from the early days of openstack) | 18:02 |
| clarkb | I think the other reed was going to spin things up and provide some tooling but may not be doing long term maintenance. Basically we're relying on aws tooling to take on the bulk of that | 18:02 |
| dpawlik | fungi: oh :D | 18:02 |
| dpawlik | I need to ask him if he can join irc or matrix... hope not slack | 18:03 |
| fungi | yeah, the matrix oftc bridge should work for joining this channel | 18:04 |
| fungi | if that's preferable to straight up irc | 18:04 |
| dpawlik | fungi: about system resources I would like to make few calculations base on RDO infra before | 18:04 |
| fungi | take your time | 18:04 |
| fungi | just letting you know we can boot stuff up when you're ready | 18:05 |
| dpawlik | fungi: or maybe we can start with some small VM and resize flavor is easy, going back is problematic | 18:05 |
| *** amoralej is now known as amoralej|off | 18:05 | |
| fungi | i'm not sure how easy it is to resize instances in vexxhost, that's not something i've tried there (i know it's basically impossible in rackspace these days, but that's not vexxhost) | 18:05 |
| dpawlik | fungi: we resize some instances on vexxhost, but I don't remember what was the root disk backend... | 18:07 |
| fungi | sure, we can certainly try then | 18:07 |
| dpawlik | fungi, clarkb: cool. thanks for clarifying, I need to go for today. Will ping you tomorrow. Have a nice day | 18:08 |
| *** jpena is now known as jpena|off | 18:10 | |
| fungi | you too | 18:17 |
| clarkb | stephenfin: its a good thing we removed the pbr dogfooding of the pep517 stuff to start as it appears to not be working on a test bindep change: https://review.opendev.org/c/opendev/bindep/+/816741 I think making these changes as small steps like this is a very good idea now :) | 19:01 |
| clarkb | I probably won't be able to dig any further today, but will try and look when I can | 19:01 |
| opendevreview | Merged openstack/project-config master: kolla-cli: enter retirement https://review.opendev.org/c/openstack/project-config/+/814597 | 20:23 |
| *** dviroel|rover is now known as dviroel|out | 22:13 | |
| *** rlandy|ruck is now known as rlandy|ruck|afk | 22:29 | |
| opendevreview | Merged openstack/project-config master: Add support for CentOS Stream 9 in nodepool elements https://review.opendev.org/c/openstack/project-config/+/811442 | 23:58 |
| opendevreview | Merged openstack/project-config master: Add centos-9-stream nodepool image https://review.opendev.org/c/openstack/project-config/+/816465 | 23:59 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!