Tuesday, 2020-08-25

prometheanfirethere's like 3-4 diferent ways to add apt keys in dib, heh00:05
fungiclarkb: i'm really not here, but the ghost of my connection feels compelled to suggest we check squid's behavior, its proxy caching at least used to be much easier to fine-tune than apache's comparable mods00:09
clarkbya squid seems like it would get these bits better00:09
fungii used to front some very popular web sites with squid caches at $oldjob and it did great00:10
fungiincluding working around some crazy site frameworks which incorporated all sorts of cache-busting tricks in service of making sure users weren't seeing old pages, except those same tricks usually also give caching proxies nightmares00:11
fungithings like setting content dates years in the past, negative cache ttls, et cetera00:12
*** DSpider has quit IRC00:26
prometheanfireianw: clarkb not fully done, but this seems to be working https://dpaste.com/FWV6QUFCN00:31
ianwcool, that seems reasonable now it can handle keys in the directory00:32
prometheanfireya, the dir only handles the binary format I think, that'd be the only drawback00:33
ianwi have great deja vu on that00:34
ianwsomething we did in zuul-jobs with ppas00:34
ianwit has to do with the .asc extensions or something ...00:34
prometheanfireheh, I converted gentoo to use a binary keyring as well (for image verification), easier to use00:34
prometheanfireno need to import stuff, just ship a gpg homedir00:35
ianwyeah, if it's .gpg it's binary, and then i think .asc works for ascii armored too00:36
prometheanfireah, well, guess that needs doc'd00:36
prometheanfireI'll submit this once I figure out why the image isn't building (failing on a python pre-install thing)00:37
prometheanfire2020-08-25 00:38:57.760 | + [[ 2 == 3 ]]00:39
prometheanfireneed to set DIB_PYTHON_VERSION=300:41
openstackgerritMerged opendev/system-config master: Add linaro Focal mirror  https://review.opendev.org/74761700:41
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: DNM: copy keys into dirs  https://review.opendev.org/74781000:55
prometheanfireianw: ^00:55
*** tkajinam_ is now known as tkajinam01:04
*** elod has quit IRC01:21
*** elod has joined #opendev01:23
ianwhrm, infra-prod-base failed :/01:31
ianwNo usable temporary directory found01:33
ianwodd ... full disk?01:33
clarkbperhaps the cache cleaner isntkeeping up01:34
ianw-bash: /etc/profile: Input/output error01:34
ianwwhen i try to log in ... i think this host is unhappy01:34
donnydianw: btw i fixed the block device thing with OE, so if you need to add more storages to that mirror feel free01:35
clarkbssh mirror df?01:35
clarkbthat should bypass a shell right?01:35
ianw[1650583.042388] Buffer I/O error on dev vda1, logical block 0, lost sync page write01:36
ianw[1650583.045683] EXT4-fs (vda1): I/O error while writing superblock01:36
ianwthat's on the console01:36
ianwdonnyd: ^ it looks like the mirror is pretty unhappy with it's storage backend01:36
clarkboh did it remount ro?01:36
ianwclarkb: yeah, i think that's part of it01:36
donnydMaybe reboots? Every other job on there is using the same backend01:36
donnydBut there was a small hiccup a few days ago. Apparently arista switches will flap all ports when you change the speed to 40g on just one01:37
ianwok, let me reboot it ... it's very unhappy01:38
ianwmirror01 login: [1364013.350470] blk_update_request: I/O error, dev vda, sector 130667408 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 001:38
donnydBut that was Friday last week01:38
ianwis when it started01:38
ianwever helpful relative timestamps01:38
prometheanfireianw: I think I'm gonna move that key stuff into debian minimal (as root.d still) in a earlier number (09 to go after 08-debootstrap)01:40
prometheanfireotherwise it kinda overloads debootstrap01:41
ianw#status log rebooted mirror01.us-east.openedge.opendev.org due to it corrupting it's disk.  seems ok after reboot01:41
openstackstatusianw: finished logging01:41
donnydianw: i also noticed a recent patch in dib for python3 with centos and rhel breaks rhel01:42
ianwdonnyd: which one is that?01:42
donnydI mentioned it in the dib channel, but i haven't gotten around to opening up a bug on it yet01:42
donnydThis one i do believe01:44
donnydIt runs in pre-install and for rhel it doesn't get a sub until after pre-install01:44
donnydAt least with satellite it doesn't01:44
ianwarrgghh, hrm, we need python3 to run the package map01:45
donnydWe should move the registration to earlier then IMO01:45
donnydIt makes sense to have the first thing rhel does is subscribe01:46
ianwindeed.  where does it happen now?01:47
ianwi guess if it was 00-0 it would come first ...01:48
ianwwhat a mess :/01:49
ianwwe should always start things at 10, to give us at least 9 iterations of being able to realise we need things earlier in the future01:49
donnydThat is interesting01:50
donnydI would share the logs, but from this particular build system i can't01:51
donnydBut it was throwing the i don't have a sub so i can't install this error01:51
prometheanfireianw: or to users01:52
ianwdonnyd: i'd be interested if you did mv diskimage_builder/elements/rhel-common/pre-install.d/00-rhel-registration diskimage_builder/elements/rhel-common/pre-install.d/00-0-rhel-registration01:52
prometheanfiresee https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/gentoo/pre-install.d :|01:52
ianwif it worked01:52
prometheanfirebecause something else was at 02 iirc01:52
donnydI can give that a whirl01:52
donnydI will open a real bug for it, so we can sort it out there. I don't want to derail anything else you already have in flight01:54
donnydWait so why does 00-1  come up before 00? I am no bashxerpt01:55
ianwumm, it comes up before 00-<alphabet>01:57
donnydFwiw dib is super reliable and works very well for some pretty complex stuff.. so everyones work on it is super appreciated01:57
donnydOh yea, that makes sense01:57
donnydSo we should just make rhel reg 00-0001:58
donnydOr start the sub nmbering at 10 and move the python3 install to 1101:59
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: copy apt gpg keys directly into trusted.gpg.d  https://review.opendev.org/74781002:11
prometheanfireenvironment.d happens after root.d iirc, maybe02:12
prometheanfirerunning a test build now02:14
prometheanfireoh, we are good, nice02:14
ianwenvironment.d should be sourced before running02:26
donnydOk, I got around to opening a bug. I can work on a patch, but probably not tonight. Gotta do that thing where i close my eyes and don't computer for a few hours.02:26
ianwdonnyd: thanks, our rhel testing is unfortunately non-existent in the gate so feedback is good02:26
ianwi think i might have to recommit something to effectively do the base->letsencrypt->mirror dance for the new linaro mirror02:28
donnydFor sure - I can tinker something up and submit it.02:29
prometheanfireianw: ya. built an image fine, now testing it booting02:30
prometheanfirebooted too, so that's neat02:33
prometheanfireyep, gonna call it good there02:38
prometheanfiremade the least invasive change to achieve goals :D02:39
openstackgerritIan Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org  https://review.opendev.org/74781202:42
openstackgerritIan Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org  https://review.opendev.org/74781202:46
openstackgerritIan Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org  https://review.opendev.org/74781302:46
openstackgerritIan Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org  https://review.opendev.org/74781203:14
openstackgerritIan Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org  https://review.opendev.org/74781303:14
openstackgerritIan Wienand proposed opendev/system-config master: letsencrypt test: copy account.conf  https://review.opendev.org/74781403:14
*** hashar has joined #opendev03:21
openstackgerritIan Wienand proposed opendev/system-config master: letsencrypt test: fix email match and copy account.conf  https://review.opendev.org/74781403:43
openstackgerritIan Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org  https://review.opendev.org/74781203:43
openstackgerritIan Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org  https://review.opendev.org/74781303:43
*** hashar has quit IRC03:47
*** hashar has joined #opendev03:48
prometheanfireianw: should I expect dib test to fail for now (should I not recheck)?03:59
ianwumm, not that i'm aware of, but that doesn't mean something hasn't broken :)04:03
ianw /opt/dib/tmp/dib_build.zsqbTWyt/hooks/root.d/09-apt-keyring: line 26: DIB_ADD_APT_KEYS: unbound variable04:05
ianwlooks like a valid error04:05
prometheanfireianw: hmm, it's exported in environment.d04:10
prometheanfireya, that's the if statement, not even inside it04:10
ianwprometheanfire: environment.d in debian-minimal though?  debootstrap is used by all the ubuntu roles too though04:11
prometheanfireianw: that's where the apt key stuff was originally04:12
prometheanfireoh, I see it04:12
prometheanfireya, I need to move that04:13
openstackgerritMatthew Thode proposed openstack/diskimage-builder master: copy apt gpg keys directly into trusted.gpg.d  https://review.opendev.org/74781004:15
prometheanfirethere are so many debian related elements it's hard to keep them in line04:15
prometheanfireat least gentoo keeps it to a single element :P04:15
openstackgerritIan Wienand proposed opendev/system-config master: letsencrypt test: fix email match and copy account.conf  https://review.opendev.org/74781404:15
openstackgerritIan Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org  https://review.opendev.org/74781204:15
openstackgerritIan Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org  https://review.opendev.org/74781304:15
*** ykarel has joined #opendev04:19
openstackgerritIan Wienand proposed opendev/system-config master: letsencrypt test: fix email match  https://review.opendev.org/74781404:43
openstackgerritIan Wienand proposed opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org  https://review.opendev.org/74781204:43
openstackgerritIan Wienand proposed opendev/system-config master: Remove mirror01.regionone.linaro-us.opendev.org  https://review.opendev.org/74781304:43
openstackgerritSampath Priyankara (samP) proposed opendev/irc-meetings master: Change Masakari Meeting time  https://review.opendev.org/74781904:43
*** raukadah is now known as chandankumar04:56
*** ysandeep|away is now known as ysandeep05:20
*** hashar has quit IRC05:31
openstackgerritMerged opendev/system-config master: letsencrypt test: fix email match  https://review.opendev.org/74781405:37
*** elod has quit IRC05:39
*** elod has joined #opendev05:53
openstackgerritMerged opendev/system-config master: Add LE bits for mirror02.regionone.linaro-us.opendev.org  https://review.opendev.org/74781206:08
openstackgerritIan Wienand proposed opendev/system-config master: mirror02.regionone.linaro.us : add missing LE file  https://review.opendev.org/74783906:39
*** andrewbonney has joined #opendev07:05
*** DSpider has joined #opendev07:14
*** iurygregory has joined #opendev07:21
*** dtantsur|afk is now known as dtantsur07:26
openstackgerritMerged opendev/system-config master: mirror02.regionone.linaro.us : add missing LE file  https://review.opendev.org/74783907:32
*** tosky has joined #opendev07:36
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: Update hashicorp jobs file with correct title  https://review.opendev.org/74785307:38
*** ysandeep is now known as ysandeep|lunch07:48
*** bolg has quit IRC07:51
*** hashar has joined #opendev07:59
*** bolg has joined #opendev07:59
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:03
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: WIP: Add nim roles  https://review.opendev.org/74786508:40
*** ysandeep|lunch is now known as ysandeep09:02
openstackgerritXinliang Liu proposed openstack/diskimage-builder master: Make iscsi-boot element support centos 8  https://review.opendev.org/74787809:29
*** johnsom has quit IRC10:16
*** johnsom has joined #opendev10:17
*** aannuusshhkkaa has quit IRC10:49
*** ildikov has quit IRC10:49
*** knikolla has quit IRC10:49
*** dviroel has quit IRC10:51
*** aannuusshhkkaa has joined #opendev10:59
*** dviroel has joined #opendev11:00
*** knikolla has joined #opendev11:01
*** ildikov has joined #opendev11:10
*** hashar is now known as hasharLunch11:14
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Add ansible collection roles  https://review.opendev.org/73036011:59
openstackgerritSorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible  https://review.opendev.org/73458411:59
*** ysandeep is now known as ysandeep|session12:06
*** hasharLunch is now known as hashar12:15
*** ykarel_ has joined #opendev12:16
*** ykarel has quit IRC12:18
*** ykarel_ is now known as ykarel12:19
openstackgerritLon Hohberger proposed openstack/diskimage-builder master: rhel-common: Provide method to select module streams  https://review.opendev.org/74773212:21
*** Gyuseok_Jung has joined #opendev12:41
*** Guest34946 has joined #opendev12:47
*** Guest34946 is now known as redrobot12:50
*** lpetrut has joined #opendev12:57
openstackgerritMerged openstack/project-config master: Do not update upper constraints for intermediate branches  https://review.opendev.org/74655312:58
*** ysandeep|session is now known as ysandeep13:16
*** ykarel is now known as ykarel|away13:28
*** ykarel|away has quit IRC13:45
*** roman_g has joined #opendev13:53
*** weshay is now known as weshay|interview13:54
dmsimardAnyone else see that dockerhub is implementing rate limiting for free/anonymous pulls ? https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/14:15
fungidmsimard: yep14:19
dmsimardI'd guess this could be a problem :(14:19
fungithey're supposedly going to be publishing guidelines/recommendations for ci systems soon14:26
fungiclarkb brought it up in here at 23:39 utc14:27
*** ysandeep is now known as ysandeep|away14:37
clarkbits worth noting we cached the blob layers that were previously limited. basically we tried to be good citizens there. We'll continue to try but details to be determined14:41
*** qchris has quit IRC14:57
*** mlavalle has joined #opendev15:06
*** qchris has joined #opendev15:10
*** chandankumar is now known as raukadah15:23
*** lpetrut has quit IRC15:32
corvusclarkb: i'm mostly unavailable today; may be able to join for 1st part of mtg.15:33
clarkbcorvus: ok15:33
clarkbI think fungi is out too, may just make it a less formal discussion if we're largely absent (and thats fine)15:33
fungiyeah, i can make no guarantees as to my availability at 1900z15:34
fungibut i'm clearly failing at this vacation thing so far15:34
clarkbyou should fail at it less15:34
clarkbgo kayak around the bay or something15:35
*** owalsh has quit IRC15:36
*** owalsh has joined #opendev15:45
*** owalsh has quit IRC15:45
fricklerdoes anybody know something about mordred? I know he was moving some time ago, but that was a month ago or longer?15:48
fricklerha, now that I write that he seems online again15:49
clarkbmwhahaha: related to https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/ I wonder if part of hte trouble that tripleo is seeing is the switch from rate limiting blobs to manifests15:55
clarkbmwhahaha: we should cache blobs reasonably well but don't cache manifests and so their switch (who knows when that goes into production) would be problematic15:55
mwhahahayou can't15:55
mwhahahabecause they require auth15:55
clarkbwell not with apache anyway15:55
clarkbbut the cache-control is marked public so a tool that honors that can cache it15:56
mwhahahawe're looking to just stop using docker.io15:56
clarkbah that would also work15:56
mwhahahaby switching to a build job parent to host content15:56
mwhahahaso the only thing we nmed is a base container15:56
mwhahahathen the dependencies will just be a single container + rdo mirror which we already have. it should reduce overall external requests as well15:57
mwhahahalooks like they are also doing an opensource plan per the blog post so it might be beneficial to look into that as well15:58
mwhahahaanyway it's high on our radar :/15:58
clarkbya a lot of the open source and ci details are still TBD15:58
clarkbbut will keep an eye on it to see if there are changes we can make to better accomodate their upstream updates15:58
*** sshnaidm is now known as sshnaidm|afk15:59
clarkband based on that we can decide if we need a different caching tool (fungi suggests squid as a reverse proxy)15:59
*** weshay|interview is now known as weshay16:04
*** dtantsur is now known as dtantsur|afk16:19
clarkbhrm is our gerrit replication config entirely in priviate host/group vars?16:19
clarkbah no we moved that into inventory/16:19
* clarkb learns to grep better16:19
*** owalsh has joined #opendev16:23
mwhahahaso i have a thought, could we use docker-distribution as a daily pull-through mirror where we just nuke the content on a daily basis (or some other time period) to replicate an expiration period16:26
clarkbwe would need to set up multiple instances of that and flip flop as you can only delete content with the server off16:27
clarkbthis is th emajor reason we've avoided the tool since other web caches don't have that problem16:28
mwhahahait's just containers tho16:29
mwhahaha+ an lb16:29
mwhahahajust a thought16:29
mwhahahawe can probably do something similar in our own job config however if they have an opensource credential set that would be best to be handled in infra16:30
clarkbits also all the disk and the orchestration16:30
clarkbwe hvae a system for that preexisting which is why we've used it16:30
clarkbyou would have to build this all from scratch. it is doable but someone has to do it16:30
clarkbI'd like to see what docker says is their preferred plan for CI systems which tehy claim will be out soon16:31
clarkbthen go from there16:31
openstackgerritLajos Katona proposed openstack/project-config master: Import netowrking-l2gw & networking-l2gw-tempest-plugin to x/  https://review.opendev.org/74411016:32
mwhahahasure but you will likely need to set the crednetials somewhere for the CI system16:32
mwhahahawhich i don't think apache provides today16:32
mwhahahaso you'd need something like docker-distribution where you can put those in place16:32
mwhahahawhich is what i'm sugesting16:32
mwhahahai'd be happy to investigate implementation but i'm not certain what the limitations would need to be in to make it opendev friendly16:33
clarkbyes, I think apache is likely to not be the best tool. I'm not convinced the docker registry is either16:33
clarkbwe need more information from docker and they haven't provided that yet16:33
mwhahahadocker registry does support swift backend16:34
mwhahahabut yea16:34
clarkbthe swift backend doesn't work16:34
mwhahahaof course not16:34
* mwhahaha head -> table16:34
clarkbit periodically writes empty blobs to swift16:34
clarkbthen you fetch them back and fail because the shas don't match the manifest16:34
clarkbwe ran it in production for a short period for zuul's intermediate registry16:35
clarkbthe replacement is zuul's container registry, but that isn't suitable for the pull through cache I don't think (though corvus can probably fill in more detail there)16:35
mwhahahaoh they have garbage collection now16:37
*** roman_g has quit IRC16:38
clarkb"Note: You should ensure that the registry is in read-only mode or not running at all. If you were to upload an image while garbage collection is running, there is the risk that the image’s layers are mistakenly deleted leading to a corrupted image."16:38
clarkbthats the problem16:38
clarkbwe can run mutliple registries and flip flop to work around that or use a different tool16:39
mwhahahayea i was thinkign haproxy + cron job to rotate16:40
clarkbthe problem then becomes disk16:40
clarkbwe'd need twice as much of it16:40
mwhahahawhat's the disk constraint issue?16:40
mwhahahalike we don't know how much will actually be used? or is it because it's just an unknown16:41
clarkbeach registry would need X disk. running two would require 2X disk. We currently have about 200GB of disk for all the caching on those mirror nodes. We'd effectively half our useable cache spcae for docker hub16:41
clarkboh we use the full amount currnetly16:41
clarkbthe apache cache cleaner is very busy16:41
clarkbI think we give 100GB to afs and 100GB to apache currently16:42
clarkband 2x docker registries probably looks like 20GB for apache, 35GB docker 1, 35GB docker 2 (with some headroom because apache does an after the fact cleanup)16:43
mwhahahayea the inability to cap storage is likely a larger issue16:43
fungii'm not here, but 35gb sounds like about enough docker images to make a cup of coffee16:47
fungii have a feeling a lot of the blob cache misses are due to us having to aggressively expire images which are actually being used regularly just because 100gb isn't enough16:48
mwhahahahttps://github.com/docker/docker.github.io/blob/master/registry/recipes/mirror.md#what-about-my-disk seems like it might purge but not certain if that's based on like the available size on the fs being provided or via garbage collection16:51
mwhahahawe'll see16:52
openstackgerritClark Boylan proposed opendev/system-config master: Stop replicating to the local Gerrit mirror  https://review.opendev.org/74798616:59
clarkbI've been digging into gerrit replication and notedb. We set replicatePermissions to false currently which means that any repo with HEAD set to refs/meta/config won't be replicated at all and the refs/meta/config branch isn't replicated for any repo17:00
clarkbcurrently All-Projects has HEAD set to refs/meta/config and All-Users has HEAD set to refs/heads/master. The upgrade process seems to change All-Users HEAD to refs/meta/config17:02
clarkbI think that means we're good for not replicating groups/accounts/etc on upgrade17:02
clarkbthat leaves the actual change data which is in refs/changes/XY/ABCDXY/meta17:03
clarkbI'm not concerned about replicating those from a disclosure perspective but from a "will this fill our gitea disks" perspective. I'm now reading git refspecs to understand if we can exclude meta files from a refs/changes replication directive17:04
clarkbI'm not sure we'll actually use that if it is possible, but understanding the options here seems worhtwhile17:04
*** hashar is now known as hasharDinner17:07
clarkbit seems we can exlucde revisions but not objects? that sort of makes sense the internal objects are supposed to be largely hidden away17:10
openstackgerritClark Boylan proposed opendev/base-jobs master: Exclude neutron q-svc logs from indexing  https://review.opendev.org/74798817:18
*** olaph has quit IRC17:24
clarkbya I wonder if that means we'd have to stop pushing refs/changes/ entirely17:24
clarkbor modify the replication plugin to filter out those bits?17:24
*** andrewbonney has quit IRC17:44
*** ildikov has quit IRC18:50
*** dviroel has quit IRC18:50
*** ildikov has joined #opendev18:52
*** dviroel has joined #opendev18:53
ianwsigh, the new arm64 focal mirror already has an openafs oops and it hasn't even been turned on yet19:03
openstackgerritMerged opendev/gerritlib master: Add branches arg to createProject  https://review.opendev.org/74127719:17
clarkbzbr: for https://review.opendev.org/#/c/729966/5 were you able to figure out what created the exceptional state in the poll?19:35
clarkbmaybe that is what the paramiko link is?19:35
zbrclarkb: that happenedto me only on macos and I never found time to go deep into it in order to discover what was the out of band message send19:43
zbrstill the spec is kinda clear that this can happen, so it would not be wrong to implement19:43
zbrif i remember well, i found this while trying to run e-r locally for development19:44
zbrbut as you can see paramiko is not really under active maintenance19:45
*** hasharDinner has quit IRC19:50
zbrclarkb: now i looked again at some of my tickets with paramiko and risk going into clinical depression. https://github.com/paramiko/paramiko/issues/138319:53
openstackgerritClark Boylan proposed opendev/base-jobs master: Exclude neutron q-svc logs from indexing  https://review.opendev.org/74798820:28
openstackgerritClark Boylan proposed opendev/base-jobs master: Skip ansible-lint E208  https://review.opendev.org/74802620:28
clarkbianw: ^ lets see if that is happier20:28
openstackgerritDouglas Mendizábal proposed openstack/project-config master: Update Ansible roles for ATOS and Thales HSMs  https://review.opendev.org/74802820:41
clarkbianw: yup jobs pass now20:42
ianwall lgtm, i guess go with it!20:52
clarkb++ I've approved them20:52
clarkbI'll let that land then go and resurrect the services that have been saddened20:53
openstackgerritMerged opendev/base-jobs master: Skip ansible-lint E208  https://review.opendev.org/74802620:57
openstackgerritMerged opendev/base-jobs master: Exclude neutron q-svc logs from indexing  https://review.opendev.org/74798821:01
ianwsigh openstack-zuul-jobs broken with same linter issues21:13
ianwseems 1.8.6 has made it into groovy, which is nice (no ~pre version) and it's backported to focal/bionic with just some minor tweaks to debhelper versions21:24
openstackgerritIan Wienand proposed openstack/project-config master: Add ansible-collections for system-config -devel job  https://review.opendev.org/74759622:16
clarkbI'm trying to fix the limestone cloud cert issue. I've discovered that the cert reported by keystone is different than what we have configured in our /etc/openstack dir22:18
clarkbhowever, switching to the new cert content doesn't seem to work either22:18
clarkband firefox says the cert is corrupted?22:18
clarkbI retrieved the new content with openssl s_client22:19
clarkband it seems that it is self signed so using the cert itself as part of the trust should be fine to verify it?22:19
clarkbusing s_client and setting -CAfile to the contents of the cert that says verification is ok22:21
clarkbhrm I think the issue may be my test setup isn't using the overridden clouds.yaml22:22
clarkbya that seems to be the problem22:23
clarkbaha its the docker exec issue22:24
openstackgerritClark Boylan proposed opendev/system-config master: Update the limestone cert in our clouds.yaml  https://review.opendev.org/74804022:26
clarkbinfra-root logan- ^ fyi that seems to work. Would probably be good to have logan- confirm that this change is expected and even better is confirming the new cert material is correct22:27
clarkbweird that firefox completely breaks on it22:29
clarkbchrome doesn't break on it22:29
ianwok https://mirror02.regionone.linaro-us.opendev.org/  working now22:41
clarkbI've filed an upstream firefox bug22:45
openstackgerritIan Wienand proposed opendev/zone-opendev.org master: Replace linaro 01 mirror with 02, update mirror CNAME  https://review.opendev.org/74804622:45
ianwclarkb: if i accept the self-signed cert i see the response from https://osa.continuous.pw:5000/22:48
clarkbya I think the bug is in the cert details viewer22:48
clarkbthe underlying tls implentations seem ok with it22:48
ianwclarkb: hrm, you mean clicking on the "view certificate" ?22:52
clarkbianw: yup22:52
clarkbhttps://bugzilla.mozilla.org/show_bug.cgi?id=1661163 is the bug fwiw22:52
openstackMozilla bug 1661163 in Untriaged "New firefox cert details viewer is unable to view a cert" [--,Unconfirmed] - Assigned to nobody22:52
ianwinteresting, if i go through the menu bar it sends me to a page "about:certificate?"22:56
clarkbya thats the page that breaks for me22:56
ianwfor another cert it sends me to "about:certificate?cert=...stuff"22:56
clarkboh it sends me to the ?cert=...stuff page22:57
clarkbif I go through the url bar or the navigation on the warning page22:57
*** mlavalle has quit IRC22:57
*** tosky has quit IRC23:04
*** DSpider has quit IRC23:07
ianwhuh, i'm on firefox 7923:19
openstackgerritMerged opendev/zone-opendev.org master: Replace linaro 01 mirror with 02, update mirror CNAME  https://review.opendev.org/74804623:24
openstackgerritIan Wienand proposed opendev/system-config master: [wip] Update to ARA 1.4.2  https://review.opendev.org/66447823:42
openstackgerritSagi Shnaidman proposed zuul/zuul-jobs master: Add jobs for testing ensure-ansible and galaxy roles  https://review.opendev.org/73458423:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!