| timburke | donnyd, i'll be anxious to see how well the expirers keep up this time ;-) | 00:00 |
|---|---|---|
| donnyd | LOL | 00:01 |
| timburke | sorry i couldn't help more when you ran into trouble last time | 00:01 |
| donnyd | well this time I am using ceph for object store - because I need it for a couple RBD volumes (like 50 ) | 00:01 |
| *** mattw4 has quit IRC | 00:02 | |
| clarkb | ceph has a different issue aiui | 00:02 |
| donnyd | it was not your issue - the expiring was working - but the two servers I had doing the work would have been done some time in 2035 | 00:02 |
| clarkb | with ceph it keep an index of objects that it updates on every write or something | 00:03 |
| clarkb | so writing lots of smaller files (like we do with logs) causes it problems? | 00:03 |
| clarkb | mnaser can probably explain it | 00:03 |
| donnyd | yea I planned to ask him if is willing to share | 00:03 |
| donnyd | ceph may turn into a whole bag of worms on its own | 00:04 |
| mnaser | feel free to ping me sometime next week and I can go in details, I’m in mobile now but yes the rgw index and omaps can blow up quickly | 00:04 |
| donnyd | Thanks mnaser - I will hit you up when you are settled down | 00:04 |
| donnyd | if ceph turns into a thing - I plan to move back to swift - because the object store will be heavily used for its new use case | 00:06 |
| donnyd | this time with expirations on | 00:07 |
| donnyd | LOL | 00:07 |
| donnyd | curious if anyone found the metrics useful from the last time around | 00:08 |
| donnyd | i found them useful - but not sure if anyone else used them for anything | 00:08 |
| *** lbragstad_ has quit IRC | 00:10 | |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Add Open Edge CI mirror https://review.opendev.org/711787 | 00:11 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Set up inventory and cert for Open Edge mirror https://review.opendev.org/711788 | 00:12 |
| fungi | ianw: clarkb: donnyd: ^ | 00:12 |
| donnyd | looking now | 00:12 |
| fungi | hopefully i got all that right | 00:13 |
| fungi | thankfully a lot of it is tested | 00:14 |
| * fungi needs to prep pizza dough for tomorrow | 00:14 | |
| donnyd | I double checked the v6 record - everything is right from what I can see | 00:15 |
| donnyd | I am making the assumption because you put in the records that the mirror is build | 00:16 |
| donnyd | built* | 00:16 |
| fungi | yup | 00:17 |
| fungi | and i can ssh into it | 00:17 |
| fungi | to be clear, the server for the mirror is built and has base configuration in place | 00:18 |
| donnyd | well that is good news | 00:18 |
| fungi | that change will add it to our ansible inventory and apply all the bits which make it into a mirror | 00:18 |
| donnyd | so the next puppet run the rest will be brought up right | 00:18 |
| donnyd | or next hourly ansible run | 00:18 |
| fungi | ~hourly (it's actually triggered every 15 minutes but has been taking 30-45 minutes to complete and skips if there's one already running) | 00:19 |
| clarkb | . | 00:19 |
| fungi | .. | 00:20 |
| donnyd | ... | 00:20 |
| clarkb | sorry trying to reset ssh connection :) | 00:20 |
| clarkb | there is a bug in the dns change writing it up now | 00:20 |
| fungi | +++ATH | 00:20 |
| fungi | NO CARRIER | 00:21 |
| fungi | thanks for catching my terribad friday night typing | 00:21 |
| donnyd | i suck | 00:22 |
| donnyd | lol - i read and reread that whole thing | 00:22 |
| clarkb | fungi: comment on https://review.opendev.org/#/c/711788/1 too but not sure if that need a new ps | 00:22 |
| openstackgerrit | Jeremy Stanley proposed opendev/zone-opendev.org master: Add Open Edge CI mirror https://review.opendev.org/711787 | 00:23 |
| fungi | thanks! | 00:23 |
| fungi | looking | 00:23 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Set up inventory and cert for Open Edge mirror https://review.opendev.org/711788 | 00:24 |
| * fungi starts weighing out ingredients while people review and tests run | 00:25 | |
| clarkb | both lgtm now | 00:25 |
| *** gyee has quit IRC | 00:28 | |
| *** yamamoto has joined #openstack-infra | 00:30 | |
| donnyd | ditto | 00:39 |
| fungi | and my doughball is now set up in the fridge to develop overnight | 00:48 |
| *** igordc has quit IRC | 00:49 | |
| fungi | fatal: [static01.opendev.org]: FAILED! => {"changed": false, "msg": "Unable to reload service apache2: Job for apache2.service failed because the control process exited with error code.\nSee \"systemctl status apache2.service\" and \"journalctl -xe\" for details.\n"} | 00:53 |
| fungi | that was from the system-config-run-static build on my system-config change above to add the mirror | 00:54 |
| donnyd | Is it because it depends on something else to already be therr | 00:54 |
| fungi | SSLCertificateFile: file '/etc/letsencrypt-certs/static.opendev.org/static.opendev.org.cer' does not exist or is empty | 00:54 |
| fungi | https://zuul.opendev.org/t/openstack/build/edb8a79ab1cf45938defd9aefbc59d55/log/static01.opendev.org/syslog.txt#1602 | 00:55 |
| fungi | no, should be entirely unrelated in theory | 00:55 |
| *** igordc has joined #openstack-infra | 00:55 | |
| fungi | unless i somehow broke letsencrypt stuff | 00:55 |
| fungi | that's a totally different server | 00:55 |
| fungi | looking through the ara report for what ran from the mock bridge.o.o, i don't see a separate task to create the static.opendev.org cert | 01:00 |
| fungi | ianw: if you have a free moment, i could use an expert eyeball or two on that build result | 01:02 |
| fungi | trying to figure out if that job is wedged by something we've merged recently which didn't get tested against it, or whether my change is somehow touching something which is breaking it, or if the job is inherently unstable | 01:03 |
| fungi | it ran successfully in the gate pipeline for another change at 2020-03-06T21:19:12 | 01:04 |
| *** KeithMnemonic1 has quit IRC | 01:05 | |
| fungi | aha, maybe this is it: https://zuul.opendev.org/t/openstack/build/edb8a79ab1cf45938defd9aefbc59d55/log/static01.opendev.org/acme.sh/acme.sh.log#12 | 01:08 |
| fungi | CURLE_SSL_CONNECT_ERROR (35) A problem occurred somewhere in the SSL/TLS handshake. You really want the error buffer and read the message there as it pinpoints the problem slightly more. Could be certificates (file formats, paths, permissions), passwords, and others. | 01:08 |
| donnyd | Maybe recheck? | 01:10 |
| fungi | yeah, comparing to a working build, that initial issue block normally does a bunch more | 01:10 |
| fungi | so i do think this must have just been a fluke | 01:10 |
| fungi | rechecking | 01:10 |
| fungi | $something went sideways when acme.sh called curl | 01:12 |
| donnyd | Where is that playbook executed? Test node? | 01:12 |
| fungi | yep | 01:13 |
| fungi | it's a multi-node job with one node representing a mock version of our ansible bastion and other nodes representing some servers | 01:14 |
| fungi | that specific job is for testing the setup of our static.opendev.org web content hosting platform, so a 2-node job with the bastion and a webserver | 01:15 |
| donnyd | I C | 01:15 |
| fungi | but the framework we have lets us set up any variety of combinations of representations of our infrastructure | 01:15 |
| *** lbragstad_ has joined #openstack-infra | 01:22 | |
| *** rh-jelabarre has joined #openstack-infra | 01:28 | |
| *** yamamoto has quit IRC | 01:31 | |
| *** rh-jelabarre has quit IRC | 01:35 | |
| donnyd | Yea just a fluke fungi | 01:41 |
| fungi | infra-root: if anyone has a few minutes to review 711788 and its dependency, that should get us to the point where we can start bringing the fortnebula replacement back online | 02:13 |
| *** slaweq has quit IRC | 02:26 | |
| *** igordc has quit IRC | 02:26 | |
| *** lbragstad_ has quit IRC | 02:50 | |
| *** jamesmcarthur has joined #openstack-infra | 03:06 | |
| *** jamesmcarthur has quit IRC | 03:06 | |
| *** jamesmcarthur has joined #openstack-infra | 03:06 | |
| *** jamesmcarthur has quit IRC | 03:10 | |
| *** jamesmcarthur has joined #openstack-infra | 03:12 | |
| *** dave-mccowan has quit IRC | 03:37 | |
| *** yamamoto has joined #openstack-infra | 04:22 | |
| *** yamamoto has quit IRC | 04:23 | |
| *** artom has quit IRC | 04:34 | |
| *** jamesmcarthur has quit IRC | 04:38 | |
| *** jamesmcarthur has joined #openstack-infra | 04:41 | |
| *** jamesmcarthur has quit IRC | 04:44 | |
| *** jamesmcarthur has joined #openstack-infra | 04:45 | |
| *** tetsuro has joined #openstack-infra | 05:25 | |
| *** tetsuro_ has joined #openstack-infra | 05:26 | |
| *** tetsuro has quit IRC | 05:30 | |
| *** jamesmcarthur has quit IRC | 05:35 | |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-infra | 05:35 | |
| *** jamesmcarthur has joined #openstack-infra | 05:58 | |
| *** dayou has quit IRC | 06:16 | |
| *** dchen has quit IRC | 06:16 | |
| *** tetsuro_ has quit IRC | 06:22 | |
| *** jamesmcarthur has quit IRC | 06:28 | |
| *** dayou has joined #openstack-infra | 06:43 | |
| *** rosmaita has quit IRC | 07:00 | |
| *** rosmaita has joined #openstack-infra | 07:13 | |
| *** rkukura has quit IRC | 07:14 | |
| *** rkukura has joined #openstack-infra | 07:15 | |
| *** tetsuro has joined #openstack-infra | 07:21 | |
| *** kopecmartin|afk is now known as kopecmartin | 07:43 | |
| *** Liang__ has quit IRC | 08:08 | |
| *** roman_g has quit IRC | 08:30 | |
| openstackgerrit | Merged zuul/zuul master: Match tag items against containing branches https://review.opendev.org/578557 | 08:32 |
| *** matt_kosut has joined #openstack-infra | 09:31 | |
| *** rcernin has quit IRC | 09:44 | |
| *** slaweq has joined #openstack-infra | 09:48 | |
| *** xek_ has joined #openstack-infra | 10:26 | |
| *** matt_kosut has quit IRC | 10:33 | |
| *** slaweq has quit IRC | 10:33 | |
| *** matt_kosut has joined #openstack-infra | 10:36 | |
| *** matt_kosut has quit IRC | 10:59 | |
| *** PrinzElvis has joined #openstack-infra | 11:01 | |
| *** Lucas_Gray has joined #openstack-infra | 11:40 | |
| *** matt_kosut has joined #openstack-infra | 11:43 | |
| *** Lucas_Gray has quit IRC | 12:02 | |
| *** Lucas_Gray has joined #openstack-infra | 12:27 | |
| *** xek_ has quit IRC | 13:07 | |
| *** PrinzElvis has quit IRC | 13:07 | |
| *** Lucas_Gray has quit IRC | 13:23 | |
| *** Lucas_Gray has joined #openstack-infra | 13:27 | |
| *** matt_kosut has quit IRC | 13:36 | |
| *** matt_kosut has joined #openstack-infra | 14:19 | |
| *** tetsuro has quit IRC | 14:29 | |
| *** yamamoto has joined #openstack-infra | 14:34 | |
| *** matt_kosut has quit IRC | 14:39 | |
| *** yamamoto has quit IRC | 14:43 | |
| *** ramishra has quit IRC | 14:45 | |
| mordred | fungi: those are reviewed and ready to go - wasn't sure if you wanted to be around to look | 14:47 |
| *** aedc_ has joined #openstack-infra | 14:58 | |
| *** aedc has quit IRC | 14:59 | |
| mordred | frickler: does https://review.opendev.org/#/c/711471/ look good to you? | 15:00 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Add Open Edge CI mirror https://review.opendev.org/711787 | 15:14 |
| fungi | mordred: sure, i'm around now and can test out the mirror once they're applied | 15:27 |
| fungi | dns is working already, so i'm approving the inventory/cert addition | 15:33 |
| fungi | will test it out once the mirror configuration has been applied to it | 15:34 |
| mordred | cool | 15:38 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update "Release and stable branches" for OpenDev https://review.opendev.org/711817 | 15:39 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Mark OpenStack specific steps in Project Retirement https://review.opendev.org/711818 | 15:47 |
| *** prometheanfire has quit IRC | 15:48 | |
| *** rosmaita has left #openstack-infra | 15:50 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Fix Peer Review section formatting https://review.opendev.org/711819 | 15:51 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Remove section "Updating devstack-vm-gate-wrap.sh" https://review.opendev.org/711820 | 15:54 |
| *** yamamoto has joined #openstack-infra | 15:55 | |
| *** jamesmcarthur has joined #openstack-infra | 16:01 | |
| openstackgerrit | Merged opendev/system-config master: Set up inventory and cert for Open Edge mirror https://review.opendev.org/711788 | 16:02 |
| *** yamamoto has quit IRC | 16:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Move Translation setup to i18n document https://review.opendev.org/711827 | 16:11 |
| AJaeger | clarkb: another round of infra-manual changes, I think that it is good enough. Still, we can make it nicer but I found no easy ways right now. Please review the etherpad and if you have specific changes, I'm happy to discuss (but probably not on the weekend ;) Just wanted to leave the message). | 16:16 |
| *** Lucas_Gray has quit IRC | 16:17 | |
| *** Lucas_Gray has joined #openstack-infra | 16:24 | |
| *** jamesmcarthur has quit IRC | 16:35 | |
| *** Lucas_Gray has quit IRC | 16:38 | |
| *** matt_kosut has joined #openstack-infra | 16:39 | |
| *** Lucas_Gray has joined #openstack-infra | 16:42 | |
| *** matt_kosut has quit IRC | 16:44 | |
| *** armax has quit IRC | 17:20 | |
| *** armax has joined #openstack-infra | 17:20 | |
| *** armax has quit IRC | 17:25 | |
| *** ociuhandu has joined #openstack-infra | 17:33 | |
| *** armax has joined #openstack-infra | 17:33 | |
| *** evrardjp has quit IRC | 17:35 | |
| *** evrardjp has joined #openstack-infra | 17:35 | |
| *** Lucas_Gray has quit IRC | 17:39 | |
| *** ociuhandu has quit IRC | 17:40 | |
| *** ociuhandu has joined #openstack-infra | 17:43 | |
| donnyd | fungi: any idea's when the mirror config will run? | 17:48 |
| donnyd | there is no rush to get it up - I am headed out the door anyways. Thanks everyone for all the work | 17:49 |
| *** ahosam has joined #openstack-infra | 17:55 | |
| *** ociuhandu has quit IRC | 17:57 | |
| fungi | donnyd: not sure, was just looking to see why it's not going yet | 17:58 |
| *** ociuhandu has joined #openstack-infra | 17:58 | |
| *** slaweq has joined #openstack-infra | 17:58 | |
| fungi | from /var/log/acme.sh/acme.sh.log: [Sat Mar 7 17:20:51 UTC 2020] '-' is not a valid domain for parameter '-d' | 17:59 |
| fungi | that's not super helpful | 17:59 |
| *** xek_ has joined #openstack-infra | 18:00 | |
| fungi | hah | 18:05 |
| fungi | Mar 7 17:20:51 mirror01 ansible-command: Invoked with chdir=/opt/acme.sh/ _raw_params=/opt/acme.sh/driver.sh issue "-d m -d i -d r -d r -d o -d r -d 0 -d 1 -d . -d u -d s -d - -d e -d a -d s -d t -d . -d o -d p -d e -d n -d e -d d -d g -d e -d . -d o -d p -d e -d n -d d -d e -d v -d . -d o -d r -d g -d -d m -d i -d r -d r -d o -d r -d . -d u -d s -d - -d e -d a -d s -d t -d . -d o -d p -d e -d n | 18:05 |
| fungi | -d e -d d -d g -d e -d . -d o -d p -d e -d n -d d -d e -d v -d . -d o -d r -d g "#012 _uses_shell=True warn=True stdin_add_newline=True strip_empty_ends=True argv=None executable=None creates=None removes=None stdin=None | 18:05 |
| fungi | i think i must have a string where it wants a list element | 18:05 |
| *** lbragstad_ has joined #openstack-infra | 18:06 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Fix formatting for Open Edge LE playbook https://review.opendev.org/711828 | 18:08 |
| * fungi sighs | 18:08 | |
| fungi | infra-root: i'm self-approving that one | 18:08 |
| *** ociuhandu_ has joined #openstack-infra | 18:10 | |
| *** ociuhandu has quit IRC | 18:14 | |
| *** ociuhandu_ has quit IRC | 18:21 | |
| *** ociuhandu has joined #openstack-infra | 18:23 | |
| *** ociuhandu has quit IRC | 18:31 | |
| *** slaweq has quit IRC | 18:45 | |
| openstackgerrit | Merged opendev/system-config master: Fix formatting for Open Edge LE playbook https://review.opendev.org/711828 | 18:56 |
| *** lbragstad_ has quit IRC | 18:56 | |
| *** jamesmcarthur has joined #openstack-infra | 18:57 | |
| *** jamesmcarthur has quit IRC | 19:02 | |
| mnaser | is there an example of a project inside opendev which is publishing images right now? | 19:33 |
| fungi | as in container images? virtual machine images? we have some of both | 19:37 |
| fungi | also when you say "inside opendev" you mean hosted by opendev? | 19:37 |
| mnaser | fungi: woops, sorry, i started writing the rest of my message and got carried away. container images to be published inside the opendev dockerhub tenant (built inside opendev) | 19:39 |
| mnaser | fungi: in this case, lodgeit :) | 19:39 |
| fungi | ahh, no i'm not aware of one yet | 19:41 |
| mnaser | i was sorta hoping to build the change to add that by somewhat leveraging existing jobs | 19:42 |
| fungi | being able to redo our lodgeit deployment with containers would be cool though | 19:42 |
| mnaser | my team convinced me that lodgeit is the pastebin they want to use so i think now py3 support is in it and we just need containers and helm charts | 19:42 |
| fungi | i assume we'd do something similar to what the zuul/nodepool images use in the zuul tenant | 19:42 |
| fungi | but that's really not my forte | 19:43 |
| fungi | so just sorta guessing | 19:43 |
| fungi | and yeah, lodgeit isn't terrible. most of our struggles with it have been the database backend, and to some extent making sure to prevent search engines from indexing it so that it's not a prime target for spammers | 19:44 |
| fungi | also occasionally the max paste size (which may just be limits to the size of the underlying field type in the db) | 19:45 |
| fungi | mnaser: i think we were planning to publish our gerrit images to the opendev namespace on dockerhub, mordred probably knows more than anyone else how close we are there | 19:50 |
| fungi | mnaser: oh! we're publishing gitea images there | 19:51 |
| fungi | and gerrit and storyboard and some other stuff | 19:52 |
| fungi | https://hub.docker.com/u/opendevorg | 19:52 |
| fungi | but those may all be built in the openstack zuul tenant | 19:52 |
| fungi | checking | 19:52 |
| fungi | yeah, i think all the publication jobs for those are in the openstack zuul tenant | 19:54 |
| clarkb | the base jobs are in the shared base-jobs repo | 20:11 |
| clarkb | you should be able to copy what system-config does in the opendev tenant | 20:11 |
| clarkb | (secrets will have to be separately encrypted though) | 20:12 |
| *** ociuhandu has joined #openstack-infra | 20:35 | |
| *** jamesmcarthur has joined #openstack-infra | 20:36 | |
| *** ociuhandu has quit IRC | 20:41 | |
| fungi | score! https://mirror.us-east.openedge.opendev.org/ | 20:41 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Start minimally using Open Edge environment https://review.opendev.org/711831 | 20:48 |
| fungi | config-core: donnyd: ^ | 20:48 |
| fungi | also i saw that same curl error number 35 hit the openedge mirror for real during an acme.sh run, so it seems it's not just iffy in tests | 20:49 |
| *** ijw has quit IRC | 20:49 | |
| *** jamesmcarthur has quit IRC | 21:05 | |
| *** factor has joined #openstack-infra | 21:12 | |
| donnyd | Booya | 21:32 |
| donnyd | that is a weird issue - of course how often do these plays get run? only when a new mirror is brought online? | 21:32 |
| fungi | the acme.sh script? it gets run regularly for every site we have let's encrypt certs on | 21:34 |
| donnyd | hrm | 21:34 |
| donnyd | I am good with turning OE on | 21:34 |
| donnyd | whenever anyone else is around to give us a mighty +2 - my paltry +1 won't get us very far | 21:35 |
| *** jamesmcarthur has joined #openstack-infra | 21:59 | |
| *** ijw has joined #openstack-infra | 21:59 | |
| *** imacdonn has quit IRC | 22:05 | |
| *** imacdonn has joined #openstack-infra | 22:07 | |
| *** xek_ has quit IRC | 22:09 | |
| *** ijw has quit IRC | 22:10 | |
| *** ijw has joined #openstack-infra | 22:30 | |
| *** jamesmcarthur has quit IRC | 22:33 | |
| *** jamesmcarthur has joined #openstack-infra | 22:34 | |
| *** ijw has quit IRC | 22:35 | |
| *** kopecmartin has quit IRC | 23:07 | |
| *** ykarel|away has quit IRC | 23:08 | |
| *** ykarel has joined #openstack-infra | 23:08 | |
| *** ijw has joined #openstack-infra | 23:09 | |
| *** kopecmartin has joined #openstack-infra | 23:09 | |
| *** ijw has quit IRC | 23:22 | |
| *** jamesmcarthur has quit IRC | 23:23 | |
| *** iokiwi5 has joined #openstack-infra | 23:25 | |
| *** iokiwi has quit IRC | 23:26 | |
| *** iokiwi5 is now known as iokiwi | 23:26 | |
| *** jrist has quit IRC | 23:35 | |
| *** jrist has joined #openstack-infra | 23:37 | |
| *** ijw has joined #openstack-infra | 23:43 | |
| *** ijw has quit IRC | 23:50 | |
| *** ijw has joined #openstack-infra | 23:51 | |
| *** jamesmcarthur has joined #openstack-infra | 23:59 | |
| *** factor has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!