Friday, 2022-08-12

opendevreviewIan Wienand proposed zuul/zuul-jobs master: test-requirements: bump to Ansible 2.8
opendevreviewIan Wienand proposed zuul/zuul-jobs master: linters: use Ansible 2.8 for ansible-lint
opendevreviewMerged opendev/system-config master: Use rackspace mirror to sync centos stream repos
*** dviroel|afk is now known as dviroel00:26
ianw^ one that deploys i'll do a manual sync to ensure sanity00:35
opendevreviewMerged opendev/system-config master: Add IRC logging for #openstack-latinamerica
*** rlandy|bbl is now known as rlandy|out01:11
Clark[m]I think precreating the xapian dir fixed the issue based on the screenshots01:15
Clark[m]The archives page has listings now01:15
fungilooks like infra-prod-base failed on deploying 85292301:46
fungichecking logs01:46
fungilooks like it may be tripping over the ua enablement task for the storyboard server?01:48
fungiit's running `ua enable esm-infra` which is resulting in an rc of 101:49
*** ysandeep|out is now known as ysandeep01:53
fungithis check should be passing:
fungicausing it to be skipped i mean01:55
fungii see the expected string in `ua status` output on the server01:56
fungimaybe it was something transient?01:56
fungii'll try to reenqueue it01:57
fungidid it again. i've temporarily added the sb server to the emergency disable list and reenqueued again02:21
opendevreviewOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml
opendevreviewIan Wienand proposed zuul/zuul-jobs master: linters: add ansible-lint job with Ansible 2.8
opendevreviewIan Wienand proposed zuul/zuul-jobs master: zuul_azure_storage_upload: rename
opendevreviewMerged openstack/project-config master: Normalize projects.yaml
*** ysandeep is now known as ysandeep|breakfast03:14
*** ysandeep|breakfast is now known as ysandeep|afk03:35
opendevreviewIan Wienand proposed zuul/zuul-jobs master: zuul_azure_storage_upload: rename
opendevreviewIan Wienand proposed zuul/zuul-jobs master: linters: add ansible-lint job with Ansible 2.8
opendevreviewIan Wienand proposed zuul/zuul-jobs master: linters: add ansible-lint job with Ansible 2.8
*** ysandeep|afk is now known as ysandeep05:35
ianw#status log resynced centos-stream mirror from RAX06:34
opendevstatusianw: finished logging06:34
opendevreviewMerged opendev/system-config master: system-config-run-borg-backup: add to gate
opendevreviewMerged zuul/zuul-jobs master: Use RDO wallaby repo to install openvswitch in CS9
opendevreviewMerged zuul/zuul-jobs master: Add cli running example for s3 and swift log uploaders
*** ysandeep is now known as ysandeep|lunch07:31
*** ysandeep|lunch is now known as ysandeep08:34
*** ysandeep is now known as ysandeep|bbl10:05
*** ysandeep|bbl is now known as ysandeep11:16
fungi#status log Restarted meetbot container to pick up recently deployed channel addition in its config12:46
opendevstatusfungi: finished logging12:46
*** ysandeep is now known as ysandeep|afk12:55
*** ysandeep|afk is now known as ysandeep13:52
*** ysandeep is now known as ysandeep|out14:04
*** Guest139 is now known as diablo_rojo_phone14:10
*** dviroel is now known as dviroel|out14:31
*** marios is now known as marios|out15:00
fungieventual feedback on our warehouse feature request to chunk up the lines in the pep 691 json responses seems to be that proxies should adapt to pypi, not the other way around:
priteauHello. This is probably a stupid question, but do you know why has not run gate jobs when W+1 was added two days ago and why check jobs are not running after the rebase I just did?16:09
priteauThere is no unmerged parent change.16:11
priteauCould it be stuck due to the depends-on?16:11
fungipriteau: zuul doesn't know about stackhpc/ansible-role-os-images16:11
fungiyou'll want to add it there16:12
priteauAh thanks :)16:12
opendevreviewPierre Riteau proposed openstack/project-config master: Add StackHPC Ansible roles used by openstack/kayobe
fungitesting the currently held mm3 test node...16:31
fungii signed up with my normal address, which is the default owner for the openstack-discuss ml, and was able to manage it after fishing the address verification url out of the outgoing exim queue16:32
fungii used that to bulk subscribe a separate address to that ml, and set it as pre-verified in order to simplify testing16:33
fungii then injected a message over raw tcp socket for the list from a third address, confirmed it was held for moderation, released it from moderation, and found an outgoing copy to the list subscriber16:34
funginext, i similarly injected another message to the list from the subscribed address, and confirmed that it went through without getting held for moderation16:35
fungiso basic list functionality seems to be there16:35
fungiarchival doesn't seem to be working yet, or is delayed perhaps16:38
fungihyperkitty claims there are have been no posts to the list, when it should have two16:38
dpanechHi, could someone add me as the 1st member of the cores group of this repo that I created yesterday?
fungidpanech: sure, just a moment16:39
fungidpanech: i've added you to the starlingx-jenkins-pipelines-core group16:42
dpanechthank you!16:42
fungiany time!16:43
dpanechfungi: I think the storyboard didn't get created for this project, can you help?16:55
fungidpanech: yes, sorry we've got a config deployment hiccup there, i'll get it added momentarily. thanks for the reminder16:56
Clark[m]fungi: anything in the logs for hyperkitty? Maybe we still have permissions issues16:56
fungiClark[m]: i haven't checked that far yet, just managed to perform rudimentary testing so far16:57
Clark[m]fungi: it just occurred to me that you have have tested on the older of two holds. The newest one was put in place at the end of my day yesterday and should've fixed hyperkitty. I'll dig that one out to confirm in a couple minutes 17:10
fungithat would 'splain it if so17:11
fungii pulled the ipa out of scrollback but may have missed a newer one mentioned17:11
fungii should be able to repeat the same exercises on another node pretty easily17:11
fungidpanech: the redeploy to update the sb projects is queued up behind some other builds but should hopefully run within the hour17:13
dpanechok thank you17:13
fungii'll give you a heads up once it's there17:14
clarkbfungi: produced these screenshots so I suspect hyperkitty may actually function there17:22
clarkbthat was the result of precreating the xapian index dir so that it can be chowned by the container entry point17:22
fungigot it. i'll retest on that node in a bit. thanks!17:26
clarkbwow pypi's respones to that line break issue is "we won't change it because we don't change anything for users" its frustrating because adding linebreaks doesn't violate any specs or protocols its purely to make things better for users17:35
clarkbon top of that we're doing this to make their lives easier17:36
clarkbmaybe we should stop proxying entirely since their position seems tobe that proxying isn't a valid use case17:37
fungiagreed, we already "solved" it for our proxy (for the most part)17:37
clarkbuntil they add a package with more than 10MB of data then we'll hvae to bump our limits again.17:37
clarkbbut ya I think its mostly annoyin gthat they won't consider a different still completely valid json representation to make it easier on users17:37
fungiat least we mostly know about it now17:38
clarkbbecaues the current version is already spec compliant. Well they both are...17:38
fungii think newlines (or any "unnecessary" whitespace) are considered a waste of bandwidth these days17:39
fungieven though compression likely addresses that17:39
clarkbits also inconsequential in the face of 10MB of sha text17:39
clarkbI'm going to write a response indicating the reason we proxy in the first place is to reduce load on pypi. We'r etrying to be good community citizens but they are making it difficult and that adding newlines is just as valid as not according to json17:40
clarkbresponse posted. I tried to be constructive17:48
fungidpanech:!/project/starlingx/jenkins-pipelines is showing up for me now19:22
fungilooks like it already has a story from you though, so i guess you spotted it before i checked again19:23
clarkbfungi: assuming your testing on the newer node shows hyperkitty is working I think the last thing is sortoing out if we want different configs for our lists and how to configure the "special" ones that are private or broadcast only?19:51
dpanechfungi: its all good, thanks19:52
clarkbinfra-root the last patchset to the mm3 chnage added a restriction for django admin access to localhost only. I just tested it and it seems to work. Do we think that is desireable? I don't expect we'll ever really need it as we'll interact with mailman instead so figured walling it off was a good idea20:00
fungiclarkb: yes, the django localhost restriction sounds great to me21:21
clarkbgreat, doing an ssh port forward and hitting it from my browser worked fine and accessing it directly failed. I think we're set on that then21:22
fungias for the list configs, i think what we're going to actually want to test is importing mm2 configuration for the existing ones of various sorts and confirming they still work the way we intended (private archives stay private, hidden lists stay hidden, completely moderated lists are still completely moderated, closed subscription lists still don't allow random people to subscribe, and so on)21:23
clarkbI see, then we can build out configs for new lists using ansible or just have their owners configure them before they see use21:23
fungii think the importer is supposed to be pretty thorough at this point, but if we find corner cases we'll want to address those during migration21:23
fungiand yeah, adding new lists is probably fine to do the way we always have (let the list owner configure it before they announce it's ready to use)21:24
fungiexisting lists we'll want to import the configs for21:24
clarkbthe other thing I need to look at is refining the backup exclusions. I suspect we can avoid backing up the xapian content and just make mailman regenerate that if necessary21:25
fungiyeah probably. we'll want to backup the archives/configs/database mainly21:26
clarkbya right now just about everything shoul dbe backed up in the existing change. More about not backing up stuff that will caus eproblems if backed up frequently like xapian indexes21:27
fungiclarkb: testing on and hyperkitty still shows 0 posts in the openstack-discuss archive when there should be 221:42
clarkbfungi: which list?21:43
fungiit definitely tried to deliver both posts to the list's test subscriber though, i can see those sitting in exim's queue outbound21:43
clarkbI agree nothing in the archive svia the web there21:43
fungi1oMcK0-000ADY-Iw and 1oMcOW-000AF9-7l got forwarded to the subscriber21:44
fungiso it's probably something on the hyperkitty side of things, i'm guessing21:44
clarkbERROR 2022-08-12 21:40:40,204 439 Invalid HTTP_HOST header: ''. You may need to add '' to ALLOWED_HOSTS.21:45
clarkbit may be internal api requests between the mailman servers tripping over ^?21:45
fungiand yeah, i just double-checked that list is configured for public archives using hyperkitty, according to postorius, so inter-service communication problems would make sense21:46
clarkbI'm going to hack that into allowed hosts and restart the service21:46
clarkbin fact: "localhost",  # Archiving API from Mailman, keep it. exists in the file already so ya I bet adding will help21:47
clarkbfungi: that didn't cause it to reindex which is maybe expected. Do yo uthink you can try sending another email to see if that triggers it to happen?21:49
fungii can in a sec21:53
fungiclarkb: bingo, that was it21:58
fungithree messages in the archive now21:59
clarkbit logged that it hadnled all three too21:59
clarkbok let me make an update21:59
clarkbok I know why this happened I used instead of localhost in the docker compose env var urls because when you use localhost for mariadb it assumes a socket file not tcp. I switched it to then switched everything for consistency22:00
clarkbI think that consistency is still a good thing so I'll modify the config to allow localhost and both22:00
clarkbfungi: any concern with clearing out the two held nodes and holding a new one on the updated ps?22:01
funginone for me. delete them all as far as i'm concerned, there's nothing important on them22:01
clarkbI wasn't sure if you wanted to do more testing of things22:03
clarkbbah there is a merge conflict /me fixes22:05
opendevreviewClark Boylan proposed opendev/system-config master: WIP Add a mailman3 list server
fungii can test just about as easily on the next one22:07
fungithough i'm also about tested out for today22:08
clarkbholds have been rotated out ^ will grab another server we can poke at later22:08
clarkband ya its just about weekend time for you22:09
fungiwell, that but also i'm juggling a bunch of other stuff22:09
clarkbI think the job takes about 25 minutes to complete if you wanted to reapply your messages to that. But I thnk we tested that well enough for now and can pick this back up again later and probably look at migrated a list next22:10
clarkb*at migrating a list22:10
fungiyeah, that was really just making sure that default non-subscriber moderation works, subscriber non-moderation works, messages are being delivered to a subscriber, and archives are being created22:10
fungiand also basic list owner tasks (bulk subscribing, handling moderation approvals, looking at the list settings)22:11
clarkbI think if we didn't have preexisting lists to deal with this is about when we'd consider deploying to production :)22:12
clarkbit is getting close22:12
fungimainly we need to do import/migration testing and validation as a next step, i think22:14
fungiand decide on a rollout plan (new server vs trying to somehow do in-place, domain-by-domain or all-at-once, et cetera)22:15
fungii'm leaning toward a new server and migrating one domain at a time, starting with and letting it mellow for a bit before trying more22:15
clarkbdefinitely a new server due to the boot issues the old one has. Then probably an early domain ( make sure its happy then do the rest22:16
fungisounds like we agree22:16
clarkbwe also need to figure out how much disk space xapian needs22:16
clarkbtesting migration should give us a good idea though22:16
fungiyep, we'll want to remember to check that22:17
clarkbfungi: for when you get time, maybe you can automate your test email stuff and add it to the testing of the server?22:17
clarkbprobably in system-config/playbooks/zuul/test-lists3.yaml22:17
clarkbI don't think that needs to be rushed since you already check the behavior by hand and we're unlikely to do much that impacts it at this point22:17
fungioh, good idea. the tricky bits will probably be working out how to subscribe someone via the api, how to approve moderated messages through the api, and how to dump a raw smtp session into a socket22:18
clarkbI think the first thing is easy. Not sure about the second. And the third shouldn't be too bad with nc and file redirection22:18
clarkbfor the first thing I think you do the same thing we do to add list owners but you set the role value to subscriber instead of owner22:19
fungisounds reasonable, based on how i see the account relationships modeled in the webui at least22:23

Generated by 2.17.3 by Marius Gedminas - find it at!