Tuesday, 2022-02-15

opendevreviewIan Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file  https://review.opendev.org/c/zuul/zuul-jobs/+/82881800:06
opendevreviewIan Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file  https://review.opendev.org/c/zuul/zuul-jobs/+/82881800:14
opendevreviewIan Wienand proposed opendev/system-config master: [WIP] Base work for exporting encrypted logs  https://review.opendev.org/c/opendev/system-config/+/82881000:18
clarkbok meeting agenda has been sent00:26
mordredfungi: awesome.00:34
mordredfungi: also - considering the wiki is like the second oldest service ... it's kind of mind blowing that it's alive to the extent that it is with zero config mgmt :)00:35
mordred(eavesdrop being the first)00:35
mordredalthough IIRC, we started on a different wiki and migrated - so it's not really the second oldest service00:36
fungiyeah, it was moinmoin previously00:36
fungithough just about everything on eavesdrop has been replaced with different software over time as well00:36
mordredit has indeed00:37
mordredand oh yeah - I remember the moinmoin now00:37
opendevreviewIan Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file  https://review.opendev.org/c/zuul/zuul-jobs/+/82881800:49
opendevreviewIan Wienand proposed opendev/system-config master: Base work for exporting encrypted logs  https://review.opendev.org/c/opendev/system-config/+/82881001:46
opendevreviewIan Wienand proposed opendev/system-config master: run-production-playbook: return encrypted logs  https://review.opendev.org/c/opendev/system-config/+/82914701:46
opendevreviewIan Wienand proposed zuul/zuul-jobs master: encrypt-file : role to encrypt a file  https://review.opendev.org/c/zuul/zuul-jobs/+/82881801:51
ianw^ fungi/clarkb: that stack is ready ... interested to see what you think of it.  it took a *lot* of tweaking, but the final result is pretty straight forward03:14
opendevreviewMerged opendev/system-config master: Stop mirroring Fedora 34  https://review.opendev.org/c/opendev/system-config/+/82911503:53
fungi#status log Updated OpenID provider for the refstack.openstack.org service from openstackid.org to id.openinfra.dev04:29
opendevstatusfungi: finished logging04:29
*** pojadhav|afk is now known as pojadhav04:41
opendevreviewMerged opendev/system-config master: Switch refstack's IDP to OpenInfraID  https://review.opendev.org/c/opendev/system-config/+/82406504:52
*** ysandeep|out is now known as ysandeep05:30
*** amoralej|off is now known as amoralej07:00
*** ysandeep is now known as ysandeep|lunch08:02
*** jpena|off is now known as jpena08:34
dpawlikclarkb: hey, sure08:41
*** ysandeep|lunch is now known as ysandeep09:12
amusilgtema: Hi, could you please take a look at https://review.opendev.org/c/openstack/openstacksdk/+/76193310:14
*** pojadhav- is now known as pojadhav10:24
*** pojadhav is now known as pojadhav|afk10:24
dtantsurFYI folks, it was brought to my attention that "gitweb" link on our gerrit pages leads to "Not Found"10:44
*** rlandy|out is now known as rlandy|ruck11:16
*** dviroel|out is now known as dviroel11:19
*** pojadhav|afk is now known as pojadhav12:31
*** ysandeep is now known as ysandeep|brb12:40
*** ykarel_ is now known as ykarel12:49
*** ysandeep|brb is now known as ysandeep13:10
*** amoralej is now known as amoralej|lunch13:20
fungidtantsur: yeah, we're in the process of pointing those to gitea but have run into a bug in gerrit which we're trying to get fixed. the commit id link should take you to the gitiles view for now until we've got the gitea patch in13:48
dtantsurack, thanks13:48
fungidtantsur: https://review.opendev.org/825339 is the config update we'll land as soon as we have a patched gerrit13:49
fungithe depends-on there is the patch in the gerrit community's gerrit we're testing across different gerrits13:50
lourotfungi o/ I believe since we have landed https://review.opendev.org/c/openstack/project-config/+/825089 , https://github.com/openstack/charm-nova-compute-nvidia-vgpu should have been created. Do you have some insights about what could have gone wrong? Thanks!13:58
*** amoralej|lunch is now known as amoralej14:02
fungilourot: looks like the maintain-github-openstack-mirror should be taking care of creating repositories in github, and it last succeeded 2021-12-07 according to https://zuul.opendev.org/t/openstack/builds?job_name=maintain-github-openstack-mirror&skip=0&limit=10014:03
fungilooks like the repo listing mechanism in it bitrotted around some structureal change in zuul vars: https://zuul.opendev.org/t/openstack/build/69feba2382c5419685a7a807e64736a8/log/job-output.txt#381-38914:04
lourotfungi, thanks for the pointers! do you have a timeline as to when this could get fixed?14:18
fungilourot: no clue, someone will need to look into how the job needs to be adjusted. if that someone is me, it won't be until this afternoon14:18
fungier, that's vague, sorry. won't be for at least a few more hours due to meetings14:19
fungii can probably take a closer look at it after today's opendev meeting ends around 20 utc14:20
lourotthanks a lot, that's a more granular/precise timeline than what I expected actually :D14:20
fungii strive for precision ;)14:29
opendevreviewDmitriy Rabotyagov proposed openstack/project-config master: Move missed repos under openstack-ansible-roles ACL  https://review.opendev.org/c/openstack/project-config/+/82927814:48
*** pojadhav is now known as pojadhav|dinner15:04
*** ysandeep is now known as ysandeep|out15:25
clarkbdpawlik: hey, so I'm trying to get up to speed on where the ELK stuff with opensearch is in order to plan the removal of the old services15:36
clarkbI think we are very close to being able to do that? What do you think are the remaining TODOs before we can tell people to use the new location and stop using the old one?15:37
*** dviroel is now known as dviroel|lunch15:37
dpawlikclarkb: right. So today I rebuild logscraper + logearman container images and all seems to be fine but.. Probably the logstash service is down. I don't have access to listen cloud formation projects in AWS. I send an email right now to Reed to check it15:41
clarkbthanks. I think we also probably want to start asking people to use the new system to try it out?15:42
clarkbpeople like melwitt and dansmith and gmann are probably good early users if they are interested15:42
dpawlikclarkb: besides that, index-pattern seems to visible to readonly user, so it's good. I was thinking to change the kibana login page to show user credentials to login, when session will be gone or maybe add to the Openstack page somewhere a link that is automatically injecting header with basic credentials so user does not need to login 15:43
dpawlikclarkb: let's wait 1 day or 215:44
dpawlikclarkb: until logstash is not up and running. Hope it will be done soon 15:44
dpawlikor maybe Reed will set proper access to my account so I can check it15:44
fungithe other reed ;)15:45
dpawliksorry reed :)15:46
clarkbdpawlik: sounds like a plan. Thanks for the update15:46
dpawlikclarkb: it was working ok, before my PTO, but I compare it with old elasticsearch metrics and seems that some of them were not available in opensearch. I did not have time before pto to check whats going on. When I take random missing build on my local instance it seems that issue was on remote host where the logs are stored (I spotted an issue on15:48
dpawlikOVH host)15:48
dpawlikclarkb: but as I said, I did not check it fully. Maybe it is an issue with gearman (logscraper catch that build and process it)15:49
clarkbok, if the old system was able to pull the logs the new one should be able to as well. There is no authentication for log access for example15:50
clarkbbasically no extra privileges are involved15:50
dpawlikclarkb: I have the Opensearch opendev as prio. Will notify you and fungi what is the current progress15:50
fungidpawlik: i thought rt had set you up with aws admin access before you were off15:52
dpawlikclarkb: so in tl;dr: logstash seems to be overloaded (or is down right now); autologin for kibana will be an awesome feature; replace kibana login page with readonly user credentials info; create new visualization/dashboard + port them to yaml and store in ci-log-processing repo15:52
dpawlikfungi: he fix errors that I have for opensearch, but I did not check cloud formation service 15:53
dpawlikfungi: I was not aware how it is running, until I did not check the documentation :) https://review.opendev.org/c/openstack/ci-log-processing/+/826405/4/opensearch-config/deploy_opensearch.sh15:53
fungiahh, cool15:54
*** amoralej is now known as amoralej|off16:08
*** pojadhav|dinner is now known as pojadhav|afk16:16
*** pojadhav|afk is now known as pojadhav16:16
jrosserwould it be ok to save a copy of the contents of the `zuul` variable into the zuul-info directory? (https://zuul-ci.org/docs/zuul/latest/job-content.html#zuul-variables)16:17
clarkbjrosser: we do, it is in the inventory file16:18
fungithe zuul vars are exposed as ansible vars, so are included i that faux "inventory" view16:18
fungi(it's not the exact ansible inventory ansible uses i the builds, just an approximation, but the vars should be representative)16:19
jrosserok thats great, thanks16:19
fungii can't seem to type "in" today16:19
jrosserpart two would be, is that anywhere on the disk i can read with include_vars?16:19
clarkbonly as the file we log I think. Not sure if you can include_vars an inventory file safely16:21
*** marios is now known as marios|run16:21
clarkbyou could write it out explicitly yourself though and then include that later without the inventory stuff16:21
jrosseri had started hacking up an osa specific thing for the embedded ansible to be able to read a copy of the `zuul` yaml16:23
jrosserbut it felt like that might also be more widely applicable for anything with an embedded ansible16:23
opendevreviewClark Boylan proposed opendev/system-config master: Haproxy http checks for Gitea  https://review.opendev.org/c/opendev/system-config/+/82914116:30
*** ysandeep|out is now known as ysandeep16:33
*** dviroel|lunch is now known as dviroel16:52
*** marios|run is now known as marios16:54
*** ysandeep is now known as ysandeep|out17:11
clarkbthat was fun, rebooted to pick up updates and turns out simpledrm driver completely breaks booting on amd gpu devices?17:16
fungium, wow17:29
fungii guess it's "too simple"?17:29
clarkbI think they just don't know what is going on yet :/17:36
*** jpena is now known as jpena|off17:38
clarkbfungi: the haproxy change is still failing. The issue now is that we haven't told haproxy how to verify the ssl certs. In testing we don't really want to verify the ssl certs because it is all self signed (and that file is on another server os hard to add it to the local trust change). But in production it could be beneficial. Do you have an opinion on splitting testing from17:57
fungiclarkb: the https check needs to point to a ca file (or set verify none or ssl-server-verify none globally)17:57
clarkbproduction here (and hoping we don't instantly down all gitea when this change lands) or should we just not verify for now in both places?17:57
clarkbha yup17:57
fungiseems we were both looking it over at the same exact moment17:57
fungii agree in production it could be nice if invalid certs caused members to be excluded from the available pool, but in practice that's probably not super necessary (we didn't have it before anyway)17:58
fungii'd go with the simple solution for now17:59
clarkbright the tcp checks didn't have verification either17:59
opendevreviewClark Boylan proposed opendev/system-config master: Haproxy http checks for Gitea  https://review.opendev.org/c/opendev/system-config/+/82914118:00
fungiin reality we have some belts (periodic le jobs) and braces (certcheck warning us a month ahead of expirations), so i'm not super worried one of the gitea servers is going to spontaneously have a bad cert. odds are they'll all have it or none will18:00
fungiso i doubt it's worth the added complexity for something we can't have proper test coverage of anyway18:01
fungior, i guess we could have test coverage, but that would be even more complex still18:01
fungii imagine it would entail having test and prod versions of ca certs to verify against18:03
clarkbyes, we'd have to slurp the cert from the gitea host and then splat it out onto the haproxy host. Then bind mound that and set the ca-file and restart haproxy after the deployment18:03
fungiand somehow plumbing the gitea server cert in the job so that it's also the ca18:03
clarkbin prod we'd use /etc/ssl/certs/ca-certificates.crt18:03
fungiright that18:03
fungialso, complex health checks can represent a liability if it means more that can go wrong with the checking itself, unnecessarily evacuating the entire pool and taking the service completely offline18:04
clarkbit is doable and if anyone feels strongly about that we can add it in. However, I don't think the lack of verification is a regression compared to the old tcp checks so may make a good followon18:05
fungiyep, i concur18:05
clarkbDo we know if https://mirror.bhs1.ovh.opendev.org/wheel/centos-8-x86_64/ can be safely cleaned up? I suspect that the stream nodes are using the stream mirror properly due to the issueswe had when things were shared previously. I don't think the wheels are a big storage consumer but noticed we've still got centos-8 there18:15
clarkbfungi: I've updated the gerrit gitea changes upstream to what I'm hoping are pretty close to final iterations. Might be worthwhile respinning a hold on your change to test that?18:17
fungiyeah, i can put one in now18:54
fungiclarkb: do we need to add the other change as a depends-on in 825339?18:56
clarkbfungi: I don't think so. It seems to have worked somehow18:57
fungi(for the gitiles patch)18:57
clarkbit won't hurt though18:57
clarkbhttps://gerrit-review.googlesource.com/c/plugins/gitiles/+/330361 is the url18:57
fungiyeah, that's the one i just found18:58
fungiit depends-on the other change, so i can simply update the depends-on in ours to that18:58
fungiassuming we have plugins/gitiles in our projects list18:59
fungi(we do, right?)18:59
* fungi checks18:59
fungiyeah, it's in the tenant18:59
clarkbwe should18:59
opendevreviewJeremy Stanley proposed opendev/system-config master: Use Gitea for Gerrit's code browser URLs  https://review.opendev.org/c/opendev/system-config/+/82533919:01
opendevreviewJeremy Stanley proposed opendev/system-config master: DNM: Fail our Gerrit testing for an autohold  https://review.opendev.org/c/opendev/system-config/+/82539619:01
fungiautohold has been set for 825396,419:01
fungiand i've dropped the old autohold19:03
fungihuh, gertty complains with a syntax error when querying for a topic which starts with a digit19:27
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Replace kpartx with qemu-nbd in extract-image  https://review.opendev.org/c/openstack/diskimage-builder/+/82861720:12
opendevreviewSteve Baker proposed openstack/diskimage-builder master: Move grub-install to the end, and skip for partition images  https://review.opendev.org/c/openstack/diskimage-builder/+/82697620:12
opendevreviewJeremy Stanley proposed openstack/project-config master: Read tenant config with care in github_manager  https://review.opendev.org/c/openstack/project-config/+/82940220:27
fungilourot: ^ that should fix the github repo creation job20:27
fungiturned out to be a simple fix20:27
ianwNeilHanlon/clarkb: re: 828435 we'll want to update several matches in https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements first I think20:47
clarkbianw: ah thanks for catching that20:54
ianwwe could probably add that into the dib boot tests21:00
ianwit does expand the list of things that can go wrong in that job though, and that list is already fairly long21:00
clarkbI think we can take this one step at a time. Address what we know needs addressing, land it and see if the images build and go from there21:01
clarkbglean is another thing that may not work21:01
clarkb(I don't know if glean will recognize it is on a rhel machine)21:01
NeilHanlonianw: same change, or a different one/21:03
ianwi'd probably stack it before that change as a new one.  if you can run a build with those elements all the better21:04
opendevreviewMerged openstack/project-config master: Read tenant config with care in github_manager  https://review.opendev.org/c/openstack/project-config/+/82940221:04
opendevreviewNeil Hanlon proposed openstack/project-config master: Add Rocky Linux to nodepool elements tooling  https://review.opendev.org/c/openstack/project-config/+/82940521:05
opendevreviewNeil Hanlon proposed openstack/project-config master: Add rockylinux-8 to nodepool configuration  https://review.opendev.org/c/openstack/project-config/+/82843521:06
opendevreviewNeil Hanlon proposed openstack/project-config master: Add Rocky Linux to nodepool elements tooling  https://review.opendev.org/c/openstack/project-config/+/82940521:09
*** dviroel is now known as dviroel|out21:35
opendevreviewNeil Hanlon proposed openstack/project-config master: Add Rocky Linux to nodepool elements tooling  https://review.opendev.org/c/openstack/project-config/+/82940522:31
*** rlandy|ruck is now known as rlandy|ruck|bbl22:46
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Update platform support to describe stable testing  https://review.opendev.org/c/openstack/diskimage-builder/+/41820423:20
opendevreviewIan Wienand proposed openstack/diskimage-builder master: Update platform support to describe stable testing  https://review.opendev.org/c/openstack/diskimage-builder/+/41820423:40
*** rlandy|ruck|bbl is now known as rlandy|ruck23:46

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!