Tuesday, 2023-11-14

*** shaolin__ is now known as shaolin_10:00
*** dtantsur_ is now known as dtantsur14:51
JayFI think serach is broken on the mailing list?16:05
JayFLooks like new items are not being indexed16:05
JayFreproducer: search "eventlet", search newest first16:05
JayFalternatively, my browser may have just failed to execute the JS until I did it a third time (WTF?)16:05
tonybJayF: so it is or is not something I should look at?16:07
JayFPEBKAC/PICNIC/err: id10t/take your pick16:07
tonybJayF: I'm happy to go with something more like "I hate computers".  Which you will hear me mumble many, many times each day :)16:09
fungiJayF: or it might be slow and then caches results? i went to the main openstack-discuss archive page, entered "eventlet" without quotes in the search field, and got back 52 pages of results16:15
JayFThis was literally the drop down field didn't trigger the re-sort until I did it the second time16:15
JayFwhich I don't blame the website for until I see it happen more than once16:15
JayFbecause firefox just tends to do weird stuff at times these days, and I compile my own which is a multiplier for that16:16
fungioh, got it16:16
fungii did a similar search for "rabbit" which returned 575 results and the top scoring result was from ~4 years ago. i changed the sort drop-down to latest first and it immediately re-sorted the result for me, bubbling a result from yesterday to the top of the page16:18
fungi(also firefox)16:18
fungibut yeah, i wouldn't be surprised if ff is inconsistently squirrely16:18
fungithe embedded search engine also might briefly lock when it's indexing additions, not really sure what the underlying architecture there is like16:21
clarkbI've just updated the meeting agenda and will send it out momentarily16:40
opendevreviewTony Breeds proposed opendev/zone-opendev.org master: Add DNS records for mirror02.ord.rax  https://review.opendev.org/c/opendev/zone-opendev.org/+/90092216:41
opendevreviewTony Breeds proposed opendev/system-config master: Add new mirror02.ord.rax to inventory  https://review.opendev.org/c/opendev/system-config/+/90092316:45
opendevreviewTony Breeds proposed opendev/zone-opendev.org master: Add DNS records for mirror02.ord.rax  https://review.opendev.org/c/opendev/zone-opendev.org/+/90092216:50
clarkbtonyb: some notes on ^16:55
clarkbtonyb: oh you also need to bump the serial16:55
clarkbI use `date +%s` values16:55
clarkbalso we tick over to 1700000000 today16:56
clarkbI think16:56
tonybThanks.  I caught the whitespace and serial issues.16:56
fungii was just about to leave the same comments. saved me the trouble!16:57
tonybI assumed the LE stuff came later.16:57
clarkbtonyb: no we do the LE stuff upfront and that allows the ansible deployment to do everything at once16:57
clarkbit is the big upside to having LE confirmation in DNS, we don't need to do multiple passes in deployment16:57
tonybI assume I do *NOT* want to update the mirror.ord CNAME until we have verified the new server is good16:59
clarkbthat will be the last step to put the new server into production17:00
tonybGot it.17:01
SvenKieskehas anybody else trouble reaching review.opendev.org? global down detectors think it's just me17:02
clarkbSvenKieske: no issues from here17:03
SvenKieskeokay it's just really slow here17:03
clarkbreally slow -> that could be a symptom of ipv6 -> ipv4 fallback17:03
clarkbif your ipv6 path isn't working17:03
fungior a choked peering point between two backbone providers somewhere in the path from your isp to vexxhost17:04
SvenKieskeI actually seem to have some packet loss (with icmp) to review.opendev.org, but on both ipv4 and ipv6, but more on ipv6, interesting17:05
SvenKieskeI guess that's a hint from the universe to stop working for today17:05
tonybI spent so long teaching vim to always use 4 spaces instead of a tab, I can't seem to make it do the right thing for the zone file ... please hold17:10
clarkbtonyb: I use augroup on file extension matches to set file type specific rules17:13
* tonyb tries that17:14
clarkbI actually have it set up to only do special rules for certain file types and then don't have special rules for dns zone files17:14
tonybAhh I should probably try that.17:17
clarkblooks like my vim recognizes the bind files as filetype bindzone. I think that means you can use something like: https://paste.opendev.org/show/bgwBX1vDlFjHkw9r9gj8/17:22
opendevreviewTony Breeds proposed opendev/zone-opendev.org master: Add DNS records for mirror02.ord.rax  https://review.opendev.org/c/opendev/zone-opendev.org/+/90092217:23
clarkbI have a sudden urge to update my vimrc :)17:23
tonybYou're welcome?17:24
clarkbtonyb: some copy pasta updates needed (inline comments for that) otherwise lgtm17:28
fungii'm lazy, so i just '/    ' and then 4s<^V><tab>17:29
opendevreviewTony Breeds proposed opendev/zone-opendev.org master: Add DNS records for mirror02.ord.rax  https://review.opendev.org/c/opendev/zone-opendev.org/+/90092217:31
tonybfungi: That'd work.17:54
opendevreviewMerged opendev/zone-opendev.org master: Add DNS records for mirror02.ord.rax  https://review.opendev.org/c/opendev/zone-opendev.org/+/90092219:11
artomHeya, so we're trying to test huge pages in the gate, and need the PDPE1GB CPU flag. We're using the 'nested-virt-ubuntu-jammy' label for our nodeset, and as luck would have it, whenever we get a VM from Vexxhost for that label, we have that flag. Great!19:58
fungii take it there's no way to emulate or mock that19:59
artomLooking at the project-config however, it looks as though OVH _also_ provide that label, so I'd like to check whether their VMs also have that flag. I haven't come up with anything smarter than just rechecking https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/900824 until it happens to grab an OVH VM, but so far no luck.19:59
artomfungi, the value is specifically in the integration tests on real hardware19:59
artomSo I'm wondering - is there a way to "force" a specific nodepool provider - or maybe the question I should be asking, are OVH _actually_ still providing that label? Like, maybe the flavor doesn't exist on their cloud, or we have no more quota or something...20:00
fungiartom: not really, you can push multiple do-not-merge changes with zuul config adjusted to just the jobs you're trying with, but we don't have a way to limit jobs to specific donor providers. i'll take a quick look at the config for ovh20:02
artomIt just seems awefully sus that even after half a dozen rechecks, we're always landing on Vexx20:03
fungii agree that's unexpected20:03
clarkbartom: you can see https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl04.opendev.org.yaml is configured to provide the flavor in ovh20:03
clarkband it is the same flavor as all the other nodes in ovh20:03
tonyb[tony@thor ~]$ date --date='now + 130mins' +%s20:04
clarkbI suspect that what may be happeing here is that vexxhost doesn't provide normal nodes anymore so it is able to schedule the special labels quickly20:04
clarkbwhereas ovh is also dealing with all the normal requests20:04
clarkbin any case you can just check the cpu flags on any job running on ovh as the flavors are the same according to the config there20:05
clarkbdon't huge pages also imply numa and if the underlying cloud isn't doing numa we can't run them?20:05
artomclarkb: aha, interesting. Yeah, that was going to be my question - if every OVN label is the same flavor, than any old OVH node will do.20:05
artomOVS, OVN, OVH...20:05
JayFis it OVerwhelming?20:06
fungiyep, i agree. it looks like ovh would in theory provide those nodes, it's probably just swamped with orders of magnitude more requests for standard labels20:06
artomOverly so20:06
artomThese puns deserve an OVation20:10
opendevreviewMerged opendev/system-config master: Add new mirror02.ord.rax to inventory  https://review.opendev.org/c/opendev/system-config/+/90092320:48
clarkblooks like the le job failed for ^21:19
clarkboh! I know why :)21:20
clarkbtonyb: so bridge:/var/log/ansible/letsencrypt.yaml.log.2023-11-14T21:13:10 is where the logs for that went. I'm fairly positive teh reason it failed is we didn't add the ansible host vars for the LE cert info21:21
clarkbtonyb: you may also need to add an ansible handler for the cert21:21
clarkbsorry I should've known that was missing in the inventory addition21:21
fungii missed it too :(21:22
tonybAhh I see.  I should have done a more complete check.22:14
tonybSo is it just a matter of adding those handlers? or now that the job failed is the cleanup recovery more complex?22:14
opendevreviewTony Breeds proposed opendev/system-config master: Add letsencrypt_certs for mirror02.ord  https://review.opendev.org/c/opendev/system-config/+/90095322:40
opendevreviewTony Breeds proposed openstack/project-config master: Add tonyb to global IRC admins  https://review.opendev.org/c/openstack/project-config/+/90095422:53
opendevreviewTony Breeds proposed opendev/system-config master: Add tonyb to statusbot nicks  https://review.opendev.org/c/opendev/system-config/+/90095522:55
opendevreviewMerged openstack/project-config master: Add tonyb to global IRC admins  https://review.opendev.org/c/openstack/project-config/+/90095423:14
ianwtonyb: should be fine to just run again and will deploy23:31
ianwi've had in my todo list "lint against forgetting handlers/rework handler" for ... a long time :)23:32
tonybianw: If you give me an idea how to do it, I'd be happy to implement it23:33
ianwopenstack-infra/system-config/playbooks/roles/letsencrypt-create-certs/handlers has a note from 2019 about how listen: wasn't working.  i think that perhaps i was thinking at the time it could be made simpler where every cert doesn't need it's own handler23:42
ianwso tasks/acme.yaml could notify: something more generic, instead of "notify: 'letsencrypt updated {{ item.key }}'" which means every cert needs it's own handler23:43
ianwbecause most of the handlers are the same restarting apache.  one complication would be it would have to maybe default to apache, but you'd set a variable to override it if you want to do something else on cert renewal ...23:43
ianwto keep it more or less as is, though, i think it just requires a script that walks inventory and pulls out the letsencrypt_certs: variables.  then check each variable entry has a handler in tplaybooks/roles/letsencrypt-create-certs/handlers/main.yaml 23:47
tonybianw: Ah okay.  Definately more in the "rework" category ;P23:47
ianw"just" ... probably not too much craziness for a a python script 23:48
ianwparse the handler file into a dict, walk all the inventory files, if you see a "letsencrypt_certs" dict, for each key, make sure handler exists23:48
ianwi can't think of any way to do it less bespoke than that, though23:49
ianwit certainly is a trap we've fallen into more than once, so probably justified effort :)23:49
clarkbI rechecked 900953 in order to gather more data about whether or not that failure is on our end or quays. cc corvus there might be a problem with registry.zuul-ci.org fwiw see https://zuul.opendev.org/t/openstack/build/e045e455ecf44f819b0d72cefba7797d23:55
corvusdidn't we abandon that?23:56
tonybIIUC quay has been having problems today.23:56

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!