| *** DSpider has quit IRC | 00:34 | |
| sgw | fungi: worked it out, might have not completed all the required steps. He is added. BTW, I still had the permission problem from a new machine, new email, same OneID though. I could do it via CLI | 00:45 |
|---|---|---|
| fungi | mmm... if you can add someone to the group with the ssh cli and not with the webui, that points to your ssh key being associated with a different account than your openid | 00:49 |
| fungi | ssh key and ssh username i mean | 00:50 |
| *** mlavalle has quit IRC | 00:50 | |
| *** mlavalle has joined #opendev | 00:51 | |
| fungi | sgw: bingo, you have two active accounts with different e-mail addresses, ids 28607 and 32381. the first has a username and the second does not | 00:57 |
| fungi | i bet the second is the one you're using via the webui, the first is the one which was in the group | 00:57 |
| fungi | likely the best solution is to move the new openid to the older 28607 account and set 32381 inactive, but you'll be the best judge of which you want to keep | 01:00 |
| *** qchris has quit IRC | 01:03 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update to gitea 1.12.4 https://review.opendev.org/753485 | 01:12 |
| ianw | after all that the templates look unchanged. however, there will be changes for the next version :) | 01:13 |
| *** qchris has joined #opendev | 01:17 | |
| *** mordred1 has joined #opendev | 01:33 | |
| *** mordred1 is now known as mordred | 01:34 | |
| *** diablo_rojo has quit IRC | 01:36 | |
| *** Topner has quit IRC | 01:54 | |
| *** dmellado has quit IRC | 02:38 | |
| openstackgerrit | Merged opendev/system-config master: install-ansible: fix collections install for devel job https://review.opendev.org/752846 | 03:07 |
| *** dmellado has joined #opendev | 03:09 | |
| *** Gyuseok_Jung has joined #opendev | 03:09 | |
| *** kden has quit IRC | 04:36 | |
| *** kden has joined #opendev | 04:40 | |
| *** Gyuseok_Jung has quit IRC | 04:46 | |
| *** marios has joined #opendev | 04:59 | |
| *** kden has quit IRC | 05:23 | |
| *** auristor has quit IRC | 05:25 | |
| openstackgerrit | Alexander Szakaly proposed opendev/git-review master: Fix bug in git_credentials() https://review.opendev.org/753946 | 05:56 |
| *** lpetrut has joined #opendev | 06:10 | |
| *** slaweq has joined #opendev | 06:11 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/753950 | 06:14 |
| *** ysandeep|away is now known as ysandeep | 06:15 | |
| *** slaweq has quit IRC | 06:29 | |
| *** slaweq has joined #opendev | 06:31 | |
| *** eolivare has joined #opendev | 06:32 | |
| *** hashar has joined #opendev | 06:49 | |
| *** andrewbonney has joined #opendev | 07:08 | |
| AJaeger | infra-root, https://review.opendev.org/753816 merged a new repo, but it does not exist, see https://review.opendev.org/753816 | 07:16 |
| AJaeger | infra-root, https://review.opendev.org/753816 merged a new repo, but it does not exist, see https://opendev.org/starlingx/meta-starlingx | 07:16 |
| AJaeger | (now with fixed pasto) | 07:17 |
| AJaeger | infra-root, and 753816 is *not* in git - check https://opendev.org/openstack/project-config/commits/branch/master | 07:18 |
| AJaeger | Somehow syncing to gitea looks broken ;( | 07:18 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Fix python-stow-versions https://review.opendev.org/751610 | 07:36 |
| openstackgerrit | Dmitriy Rabotyagov (noonedeadpunk) proposed openstack/diskimage-builder master: Fix python-stow-versions https://review.opendev.org/751610 | 07:37 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add support to use stow for ensure-python https://review.opendev.org/751611 | 07:39 |
| *** dtantsur|afk is now known as dtantsur | 07:47 | |
| *** ysandeep is now known as ysandeep|lunch | 07:59 | |
| *** moppy has quit IRC | 08:01 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add support to use stow for ensure-python https://review.opendev.org/751611 | 08:01 |
| *** moppy has joined #opendev | 08:02 | |
| *** tosky has joined #opendev | 08:04 | |
| *** fressi has joined #opendev | 08:21 | |
| *** hashar has quit IRC | 09:03 | |
| *** hashar has joined #opendev | 09:17 | |
| *** ysandeep|lunch is now known as ysandeep | 09:21 | |
| *** fressi has quit IRC | 09:25 | |
| *** AJaeger has quit IRC | 09:27 | |
| *** jaicaa has quit IRC | 09:27 | |
| *** dtantsur is now known as dtantsur|brb | 09:43 | |
| *** AJaeger has joined #opendev | 10:01 | |
| *** jaicaa has joined #opendev | 10:02 | |
| *** fressi has joined #opendev | 10:13 | |
| *** roman_g has joined #opendev | 10:13 | |
| ianw | AJaeger: pretty sure https://zuul.opendev.org/t/openstack/builds?job_name=system-config-run-gitea would get it | 10:47 |
| ianw | oh, yeah, it failed @ https://review.opendev.org/#/c/753816/ | 10:48 |
| ianw | fatal: [gitea02.opendev.org]: FAILED! => { | 10:52 |
| ianw | "changed": true | 10:52 |
| ianw | } | 10:52 |
| ianw | MSG: | 10:52 |
| ianw | 500 Server Error: Internal Server Error for url: https://localhost:3000/api/v1/orgs/openstack/repos?limit=50&page=2 | 10:52 |
| AJaeger | ianw: I don't see the change mirrored to the gitea farm either, so somehow project-config is not update in opendev.org | 10:53 |
| AJaeger | ianw: so, two problems: 1) repo creation failed; 2) No sync from gerrit to gitea for project-config | 10:54 |
| *** dtantsur|brb is now known as dtantsur | 10:57 | |
| ianw | AJaeger: ok, the replication queue doesn't look like it's stuck on review.opendev.org ; i.e. gerrit thinks it is sending out updates | 11:00 |
| sshnaidm | fungi, clarkb hi, I saw something suspicious in logs page: https://imgur.com/a/NTImq51 maybe it's that cause delays while opening logs page, fyi | 11:18 |
| *** auristor has joined #opendev | 11:30 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gitea: fix 3081 proxy https://review.opendev.org/754070 | 11:31 |
| ianw | AJaeger: https://gitea01.opendev.org:3000/openstack/project-config has it | 11:33 |
| ianw | it looks like gitea04/05 are out of sync | 11:34 |
| ianw | both have a lot of git defunct processes | 11:35 |
| ianw | AJaeger: sorry, i'm out of time to keep poking now | 11:35 |
| ianw | infra-root: if i had to suggest, maybe do the point-release upgrade to gitea with https://review.opendev.org/#/c/753485/ and then re-sync 04 & 05 | 11:36 |
| ianw | i guess i should take them out of rotation | 11:38 |
| ianw | #status log took gitea 04 and 05 out of the load-balancer because they are out of sync | 11:40 |
| openstackstatus | ianw: finished logging | 11:40 |
| ianw | AJaeger: ^ i think you and me are just lucky to get hashed to 04, but it should be good now | 11:40 |
| ttx | also (less urgent, just signalling in case there is a relationship) ethercalc seems offline (503) | 11:40 |
| *** ysandeep is now known as ysandeep|brb | 11:40 | |
| ianw | Sep 24 10:13:07 ethercalc02 bash[15741]: /opt/ethercalc/node_modules/redis/index.js:602 | 11:43 |
| ianw | Sep 24 10:13:07 ethercalc02 bash[15741]: throw err; | 11:43 |
| ianw | ttx: not related, i restarted it, that was the last error ^ | 11:43 |
| ianw | #status log restarted ethercalc service on ethercalc.openstack.org | 11:43 |
| ttx | beh | 11:43 |
| openstackstatus | ianw: finished logging | 11:43 |
| ttx | thanks ianw | 11:43 |
| ianw | ok, gitea02 is up-to-date with project-config, but not starlingx/meta-starlingx | 11:46 |
| ianw | infra-root: here's the status -> http://paste.openstack.org/show/798331/ | 11:48 |
| ianw | gite02 is up-to-date with project-config, but doesn't have starlingx/starlingx-meta (401 error?). gitea04 & 05 out of sync with project-config, and no repo too (500 Server Error: Internal Server Error for url: https://localhost:3000/api/v1/orgs/openstack/repos?limit=50&page=2 from both) | 11:49 |
| ianw | i will also turn off gitea02 | 11:50 |
| ianw | #status log disabled gitea02 in the load balancer due to missing projects (see previous note about 04 and 05) | 11:50 |
| openstackstatus | ianw: finished logging | 11:50 |
| ianw | so 02/04/05 are off in the LB pending further investigation | 11:50 |
| ianw | i think everything should be ok for now; i have to head off but will jump on early tomorrow | 11:51 |
| *** priteau has joined #opendev | 11:59 | |
| *** fressi has quit IRC | 12:00 | |
| *** fressi has joined #opendev | 12:02 | |
| *** sparkp1ug has joined #opendev | 12:04 | |
| AJaeger | ianw: thanks | 12:04 |
| AJaeger | ianw: good night! | 12:04 |
| sparkp1ug | hey everyone, I'm a new graduate with intermediate knowledge about linux systems and am quite interested in open source contribution | 12:06 |
| sparkp1ug | I don't know where should I begin with and any guidance would be helpful | 12:06 |
| *** ysandeep|brb is now known as ysandeep | 12:21 | |
| *** fressi has quit IRC | 12:36 | |
| *** fressi has joined #opendev | 12:45 | |
| *** Topner has joined #opendev | 12:52 | |
| *** Topner has quit IRC | 12:56 | |
| *** priteau has quit IRC | 13:02 | |
| *** sparkp1ug has quit IRC | 13:03 | |
| *** sparkp1ug has joined #opendev | 13:03 | |
| *** priteau has joined #opendev | 13:21 | |
| *** fressi has quit IRC | 13:24 | |
| roman_g | sparkp1ug Hi. Depending on your skills and interests, you could join pretty much any open source project. | 13:28 |
| roman_g | Find one project which is being developed in languages you know, and ask developers on IRC/mailing list/Slack/etc. for low-hanging-fruits for you to get you started. | 13:29 |
| *** priteau has quit IRC | 13:33 | |
| openstackgerrit | Gabriel Adrian Samfira proposed openstack/project-config master: Add Ironic charms to OpenStack charms https://review.opendev.org/754098 | 13:39 |
| *** ysandeep is now known as ysandeep|mtg | 14:02 | |
| *** hashar has quit IRC | 14:18 | |
| *** ysandeep|mtg is now known as ysandeep | 14:25 | |
| openstackgerrit | Merged openstack/diskimage-builder master: rhel-common: Provide method to select module streams https://review.opendev.org/747732 | 14:32 |
| *** sshnaidm is now known as sshnaidm|afk | 14:48 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Remove install unnecessary package https://review.opendev.org/753513 | 14:55 |
| openstackgerrit | Merged openstack/diskimage-builder master: Add support for CentOS 8 Stream cloud image https://review.opendev.org/737245 | 14:55 |
| clarkb | catching up on the gitea stuff. It sounds like 02, 04, 05 didn't create the new starlingx project? and they also are out of sync with git contents? | 14:56 |
| clarkb | there was OOMing on all three at that time | 14:57 |
| clarkb | what I'm confused about is how we ran the gerrit project creation if gitea failed since gitea happens first? | 14:58 |
| clarkb | AJaeger: do you have a link to the chagne that created the new project? | 14:58 |
| clarkb | I want to work backward from that and see how it broke | 14:58 |
| *** ysandeep is now known as ysandeep|away | 14:58 | |
| *** lpetrut has quit IRC | 15:00 | |
| fungi | maybe project creation didn't return a failure to ansible, but the servers hit oom conditions before the daemons could flush their writes? | 15:02 |
| clarkb | fungi: I just found the logs ansible definitely failed | 15:03 |
| clarkb | fungi: https://zuul.opendev.org/t/openstack/build/f602f8235cd4423a8236529316bfc2bf/log/manage-projects.yaml.log | 15:03 |
| clarkb | ianw has suggested we upgrade gitea first, but I think we should get gitea to a happy spot that way we don't have to debug upgrade problems and project creation problems at the same time if the upgrade has a sad | 15:05 |
| clarkb | fungi: since gerrit succeeded at manage projects I think what we want to do is run a modified manage-projects.yaml playbook that only does the gitea bits, then limit it to gitea02,04,05 | 15:06 |
| clarkb | fungi: run it then assuming that works now we reboot them to clear out any oom side effects and then tell gerrit to resync them | 15:06 |
| clarkb | fungi: I'll start a root screen on bridge | 15:07 |
| *** lpetrut has joined #opendev | 15:07 | |
| AJaeger | clarkb, 753816 was the change | 15:07 |
| clarkb | I'm worried the max_fail_percentage: 1 we have in manage-projects.yaml is not doing what we think it is doing and that is why gerrit ran after the failures | 15:11 |
| clarkb | but now that we have the log we can figure that out later | 15:11 |
| clarkb | fungi: does that command look correct to you in the bridge screen? | 15:13 |
| clarkb | I'll run disable-ansible prior to running the ansible command too | 15:14 |
| clarkb | `ansible-playbook --limit=gitea02.opendev.org /home/zuul/src/opendev.org/opendev/system-config/playbooks/manage-projects.yaml` <- is the command for anyone else looking | 15:15 |
| clarkb | then can run that for 04 and 05 | 15:16 |
| fungi | oh, sorry, split between multiple discussions this morning. jumping into your screen session now | 15:17 |
| fungi | clarkb: yep, lgtm, let's give it a shot | 15:18 |
| clarkb | ok disabling-ansible now then will do that | 15:19 |
| fungi | did a change to push description updates to gitea during those job runs merge yet? i forget where you got to in the meeting i skipped | 15:19 |
| fungi | just wondering if it could be related, or hasn't gone in yet | 15:19 |
| clarkb | fungi: not yet https://review.opendev.org/#/c/753464/ | 15:20 |
| clarkb | ok that says it changed 02 /me checks 02 directly | 15:21 |
| fungi | okay, that simplifies the investigation i guess, but also takes away one obvious suspect | 15:21 |
| clarkb | I think the OOMs line up reasonably well | 15:21 |
| clarkb | I'm guessing gitea was unhappy at the time | 15:21 |
| clarkb | https://gitea02.opendev.org:3000/starlingx/meta-starlingx exists | 15:22 |
| clarkb | running the command above for 04 and 05 now | 15:22 |
| clarkb | then I'll reboot them then I'll trigger gerrit project sync | 15:22 |
| fungi | does make me wonder what about that project creation would have pushed them over the edge | 15:24 |
| clarkb | https://gitea04.opendev.org:3000/starlingx/meta-starlingx and https://gitea05.opendev.org:3000/starlingx/meta-starlingx look good now too | 15:24 |
| clarkb | fungi: I'm guessing something else had increased memory pressure (we aren't under another ddos are we?) | 15:25 |
| clarkb | I'm rebooting 02 04 05 next | 15:25 |
| fungi | yeah, lgtm | 15:25 |
| clarkb | I've reenable ansible since the ansible runs are all done now | 15:27 |
| clarkb | there are now ~6.5k tasks replicating to gitea | 15:29 |
| clarkb | once I'm happy with 02 et al and have them back in the lb I'll quickly check the other 5 servers too | 15:29 |
| *** hashar has joined #opendev | 15:30 | |
| clarkb | sparkp1ug: as roman_g mentions finding something that interests you or overlaps with existing knowledge is a good place to start. I'm happy to help with suggestions if you have any interests you're willing to share | 15:32 |
| clarkb | I think instead of max_failure_percentage: 1 we may want any_errors_fatal? | 15:33 |
| clarkb | I wonder if the percentage is calculated against our entire inventory and not just the matching nodes? | 15:33 |
| clarkb | in which case 3 servers out of whatever the total number is may not trip the percentage? | 15:34 |
| roman_g | sparkp1ug if you don't possess development skills, there is still always lots of work to do to 1) improve CI tests, 2) improve CD procedures, 3) improve documentation, 4) work on translations, 5) help others answering questions on project's communication channels, etc. | 15:35 |
| * clarkb finds breakfast while gerrit replicates | 15:36 | |
| fungi | roman_g: sparkp1ug: well, and also if you don't possess development skills, that's no reason not to try your hand at software development anyway. there's no better way to learn, in my opinion | 15:41 |
| fungi | (assuming gaining those skills is an interest you have) | 15:42 |
| roman_g | Absolutely true. | 15:42 |
| * roman_g is looking for Go mentor | 15:42 | |
| fungi | it's how i learned, or continue to learn really | 15:42 |
| fungi | i still consider myself mostly a sysadmin. as a dev i'm a bit of a hack even after all these years | 15:43 |
| roman_g | Same for me. I was sending patches for C code to the mailing list of a project, but I was nowhere near C programming. | 15:45 |
| roman_g | I just can read and type in English... | 15:45 |
| *** _mlavalle_1 has joined #opendev | 15:54 | |
| *** mlavalle has quit IRC | 15:57 | |
| *** diablo_rojo has joined #opendev | 15:58 | |
| clarkb | AJaeger: does https://gitea05.opendev.org:3000/openstack/project-config/commits/branch/master look up to date to you now? (I think its happy on all 3 hosts but the sync isn't finished yet) | 15:59 |
| *** priteau has joined #opendev | 15:59 | |
| clarkb | if anyone is wondering look for MAINT vs UP to see the disabled servers in haproxy show stat output | 16:08 |
| clarkb | the other servers all have recetn OOMs too | 16:09 |
| clarkb | I'll rotate through those with restarts and replication once 02 04 and 05 are done | 16:09 |
| clarkb | that way we can get everythign in a happy place before we do the 1.12.4 upgrade | 16:10 |
| clarkb | 1k tasks left on 02 04 05 | 16:10 |
| clarkb | fungi: re the description updates, we should check the job logs and look at cost to do that timewise. We only run the playbook when projects.yaml changes so its impact should be low but if it is very slow then maybe we reconsider it | 16:13 |
| *** mlavalle has joined #opendev | 16:15 | |
| *** dtantsur is now known as dtantsur|afk | 16:17 | |
| clarkb | 2,4,5 are back in the rotation. I've pulled 1 and 3 out so I can reboot them post OOM and then will replicate and add them back | 16:19 |
| clarkb | gitea's indexer startup timeout is still 30s despite the config chagne to bump that to 300s | 16:26 |
| clarkb | I'll look into that too I guess | 16:26 |
| clarkb | I think it is a type issue. The value needs to be 300s not 300 | 16:32 |
| fungi | oh, neat. unit is not implied? | 16:33 |
| clarkb | I guess not | 16:34 |
| clarkb | I think we should land that before the upgrade too since the upgarde will do restarts | 16:34 |
| fungi | i concur | 16:35 |
| clarkb | I think that means our upgrade process should roughly be: reboot and replicate all servers (in progress now), bump startup timeout to 300s properly, upgrade to 1.12.4 | 16:35 |
| fungi | sounds right to me | 16:37 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Put a unit on gitea's indexer STARTUP_TIMEOUT https://review.opendev.org/754146 | 16:40 |
| clarkb | then after the upgrade we can look at landing the description update change | 16:40 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cleanup old suse 42.3 and 15.0 repos https://review.opendev.org/754148 | 16:47 |
| openstackgerrit | Merged openstack/diskimage-builder master: Revert "Avoid disabling rhel-7-server-rh-common-rpms" https://review.opendev.org/752851 | 16:57 |
| *** roman_g has quit IRC | 16:57 | |
| *** lpetrut has quit IRC | 17:06 | |
| clarkb | 6 and 7 have just started replicating now | 17:08 |
| *** lpetrut has joined #opendev | 17:09 | |
| *** andrewbonney has quit IRC | 17:11 | |
| *** marios is now known as marios|out | 17:25 | |
| *** eolivare has quit IRC | 17:26 | |
| *** iurygregory has quit IRC | 17:45 | |
| clarkb | 08 is replicating now and is the last one | 17:47 |
| clarkb | should be done in ~20 minutes? | 17:47 |
| fungi | cool, i'll be around but also in sb meeting | 17:55 |
| fungi | and then i need to start the grill for dinner and knock out some mowing while weather favors it | 17:55 |
| *** Eighth_Doctor is now known as Conan_Kudo | 17:58 | |
| *** Conan_Kudo is now known as Eighth_Doctor | 17:58 | |
| *** iurygregory has joined #opendev | 17:58 | |
| clarkb | 08 is back in the rotation now | 18:04 |
| *** hamalq has joined #opendev | 18:04 | |
| clarkb | I've +2'd the gitea upgrade change. I looked at the diffs quickly too and they are clean from what i can see (expected for minor updates like that) | 18:04 |
| clarkb | startup timeout change should land in about half an hour | 18:05 |
| clarkb | I'm going to look at how the playbook continued on to review.o.o during manage-projects more now as I think we're stable and just awiting for changes to alnd on other things | 18:06 |
| * clarkb does local ansible testing | 18:06 | |
| clarkb | the use of max_fail_percentage does seem to change how error handling happens locally but I can't get it to run a subsequent play (maybe beacuse I need to have many more hosts in my inventory | 18:13 |
| clarkb | interestingly that max_fail_percentage code comes from the old cgit ansible puppettry | 18:13 |
| clarkb | now I'm wondering if it is an ansible version specific thing | 18:17 |
| AJaeger | clarkb: gitea05 looks fine - thanks | 18:17 |
| AJaeger | config-core, please review https://review.opendev.org/753843 - our lint job was broken ;( | 18:18 |
| clarkb | oh its a bug in our python I think | 18:20 |
| clarkb | are we not setting hte appropriate failure flags? | 18:21 |
| clarkb | this is all very confusing /me reviews AJaeger's link | 18:21 |
| clarkb | AJaeger: I guess that means tox fixed their string interpolation issues | 18:22 |
| *** tosky has quit IRC | 18:22 | |
| AJaeger | clarkb: Ah! I wondered already ... | 18:24 |
| clarkb | AJaeger: ya at some point we had to add the extra % to esacpe because tox wasn't doing it for us | 18:25 |
| AJaeger | I see... Now it's broken and I only noticed since I looked at the logs ;( | 18:26 |
| openstackgerrit | Merged opendev/storyboard master: Allow Project Groups to be filtered by project https://review.opendev.org/753417 | 18:28 |
| AJaeger | donnyd: can we use openedge for logs again? AFAIU we're ready - config-core, please approve https://review.opendev.org/753499 | 18:29 |
| AJaeger | config-core, please review https://review.opendev.org/753545 and https://review.opendev.org/752735 as well | 18:29 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Simplify gitea project creation control flow https://review.opendev.org/754205 | 18:30 |
| clarkb | I think ^ may be the next step in better undstanding that weird ansible behavior. Basically simplify and go from there | 18:31 |
| clarkb | I'm unable to reproduce the issue locally | 18:31 |
| clarkb | AJaeger: I think donnyd wanted to get better monitoring installed first | 18:31 |
| openstackgerrit | Merged openstack/project-config master: Retire os-loganalyze https://review.opendev.org/753428 | 18:32 |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/753950 | 18:37 |
| openstackgerrit | Merged opendev/system-config master: Put a unit on gitea's indexer STARTUP_TIMEOUT https://review.opendev.org/754146 | 18:38 |
| openstackgerrit | Merged opendev/storyboard-webclient master: Allowing the user to choose what Columns are seen https://review.opendev.org/651270 | 18:52 |
| *** hashar has quit IRC | 18:56 | |
| *** priteau has quit IRC | 18:58 | |
| *** marios|out has quit IRC | 19:03 | |
| donnyd | clarkb: we can turn it back on, but if it breaks again I still won't know why | 19:09 |
| donnyd | it shouldn't break because I pinned the packages on the rgw nodes and took them out of the automation for updates | 19:09 |
| openstackgerrit | Merged opendev/system-config master: Cleanup old suse 42.3 and 15.0 repos https://review.opendev.org/754148 | 19:09 |
| *** openstackgerrit has quit IRC | 19:14 | |
| *** roman_g has joined #opendev | 19:32 | |
| clarkb | fungi: ianw I should be around for the next hour and a half or so if we want to land https://review.opendev.org/#/c/753485/ nowish, but then I plan to get out on a bike ride (happy for other people to approve it while I'm doing that too) | 19:41 |
| clarkb | ianw: fungi https://review.opendev.org/754205 may be another good one to get in after the issues we had previously. I think that may make it easier to understand ansible issues in gitea land | 19:41 |
| fungi | yeah, i'm here, just mowing and getting dinner going | 19:43 |
| fungi | approved just now | 19:43 |
| clarkb | thanks | 19:43 |
| clarkb | donnyd: cool, I'll probably approve that change once gitea is done upgrading and I'm back from my bike ride | 20:08 |
| *** roman_g has quit IRC | 20:16 | |
| donnyd | kk | 20:42 |
| *** openstackgerrit has joined #opendev | 20:46 | |
| openstackgerrit | Merged opendev/system-config master: Update to gitea 1.12.4 https://review.opendev.org/753485 | 20:46 |
| openstackgerrit | Merged opendev/system-config master: Mirror OpenSUSE Leap 15.2 https://review.opendev.org/745251 | 20:48 |
| clarkb | gitea upgrades should beging in about 4 minutes or so if I'm reading the zuul status page properly | 20:50 |
| * fungi is around, just flipping burgers (literally) | 20:56 | |
| clarkb | gitea01 is about to be upgraded, pulling the new image now | 20:56 |
| clarkb | (I'm tailing the log) | 20:56 |
| clarkb | and its been running for longer than the old 30 second startup timeout | 20:57 |
| clarkb | (looks good for that fix) | 20:57 |
| clarkb | https://gitea01.opendev.org:3000/zuul/zuul/src/branch/master/zuul/change_matcher.py looks good as does my navigation to that page | 20:58 |
| clarkb | through 04 are done now and they all look good so far | 21:02 |
| *** slaweq has quit IRC | 21:06 | |
| clarkb | "The value 1000 (type int) in a string field was converted to '1000' (type string). If this does not look like what you expect, quote the entire value to ensure it does not change." <- I wish it were clearer what types ansible expects where. Anyway we may want to update our ansible to use strings for uid and gids to avoid that warning | 21:07 |
| clarkb | justsomething I noticed, everything seems fine | 21:07 |
| clarkb | gitea has been upgraded to 1.12.4 across the whole cluster. I don't see an issues yet | 21:10 |
| clarkb | #status log Upgraded gitea to version 1.12.4 from 1.12.3 | 21:10 |
| openstackstatus | clarkb: finished logging | 21:10 |
| clarkb | if anyone notices oddities please let us know | 21:10 |
| clarkb | I'll keep an eye out as I get ready for bike ride in case something pop up in the next 10 minutes | 21:11 |
| fungi | awesome--thanks! | 21:18 |
| fungi | i'll try to continue keeping one eye on irc | 21:18 |
| ianw | hey, looks like i came in right as everything is done :) | 21:20 |
| *** lpetrut has quit IRC | 21:30 | |
| fungi | you, sir, have impeccable timing | 21:56 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: ensure-zookeeper: add role to setup zookeeper https://review.opendev.org/752757 | 22:00 |
| ianw | fungi: interested if you have any suggestions for the 3081 gitea proxy in https://review.opendev.org/#/c/754070/ ... ssl cert issues. in the gate, we proxy to localhost that works because we have a self-signed cert. in production that fails because the cert doesn't cover localhost | 22:16 |
| ianw | seems we could set SSLProxyCheckPeerName off to just ignore it always, or, swizzle the ProxyPass differently for testing v production | 22:17 |
| fungi | ianw: we could add an /etc/hosts entry and not refer to localhost? | 22:17 |
| ianw | umm, i think the cert we make still won't cover it | 22:18 |
| fungi | i guess you want a non-host-specific reference though | 22:18 |
| ianw | in testing | 22:18 |
| ianw | you get a 502 error | 22:18 |
| fungi | oh, the service listening there cares what http/1.1 hostname header is passed? | 22:19 |
| openstackgerrit | Adrian Turjak proposed openstack/project-config master: remove x/distil zuul configs https://review.opendev.org/754225 | 22:20 |
| ianw | so 3081 is the apache proxy we put in for potential user-agent filtering, etc that just proxies through to 3000. i tried to use it last night and noticed it was giving 502 errors | 22:21 |
| ianw | oh, i guess you're saying "the port 3000 service (gitea) cares about hostname header" | 22:22 |
| ianw | i'm pretty sure port 3000 in the container is a ngnix, that is connected up to our letsencrypt certs on disk, so yeah, it gets unhappy | 22:22 |
| ianw | maybe i'm thinking of something else. i guess it's all inside gitea actually | 22:23 |
| fungi | yeah, i don't think there's any nginx in there, just some go-based http listener | 22:26 |
| ianw | yeah i'm thinking of graphite :) | 22:26 |
| *** sparkp1ug has quit IRC | 22:38 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gitea: fix 3081 proxy https://review.opendev.org/754070 | 22:39 |
| ianw | that switches in the gate, see how that goes | 22:39 |
| *** mlavalle has quit IRC | 23:02 | |
| clarkb | ianw: fungi it might also be nice to land the descriptions upfate change if people areup to it | 23:04 |
| clarkb | I'll be back at a keyboard in a few | 23:04 |
| ianw | yeah that lgtm | 23:08 |
| clarkb | ianw: looks like you saw the simplification on the ansible side for gitea too | 23:17 |
| ianw | yeah, agree with that | 23:18 |
| clarkb | my best guess is some combo of strategy: free and max_fail_percentage: 1 is why gerrit ran in that failed manage projects run | 23:18 |
| clarkb | ok I'm going to approve the openedge logs storage change now | 23:20 |
| clarkb | promised to do that after my bike ride | 23:20 |
| openstackgerrit | Ian Wienand proposed opendev/base-jobs master: promote-deployment: fix synchronize https://review.opendev.org/754230 | 23:23 |
| clarkb | ianw: btw your apache change for gitea failed, but I didn't dig into logs yet (there were a number of other issues today :) I was just looking to see if that was necessary to fix gitea things) | 23:24 |
| ianw | clarkb: i've sent a v2 for that which is still running now; basically swizzle the hostname for testing v production | 23:26 |
| ianw | a bit annoying because it passed testing, but then we never switched it in live to notice the production issue | 23:27 |
| ianw | but i still think it's a good escape hatch | 23:27 |
| clarkb | ++ | 23:28 |
| openstackgerrit | Merged opendev/base-jobs master: Revert "base: temporarily disable openedge log upload" https://review.opendev.org/753499 | 23:28 |
| clarkb | donnyd: ^ fyi I'll keep an eye on it too | 23:28 |
| ianw | clarkb: and yep, it just reported and looks like it passed the gate at least :) https://review.opendev.org/#/c/754070/ | 23:30 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: ensure-zookeeper: add role to setup zookeeper https://review.opendev.org/752757 | 23:33 |
| ianw | what exactly does the javascript publishing publish anyway? clearly nobody has noticed it being old | 23:42 |
| clarkb | ianw: in the before docker container times opendev deployed zuul dashboard with that js | 23:43 |
| clarkb | at some point we switched to the bubdled js in the zuul packaging | 23:43 |
| ianw | zuul-master-py3-none-any.whl2020-02-20 05:23 11M | 23:43 |
| clarkb | and the docker container images use that too | 23:43 |
| clarkb | basically ist there for people to deploy the js without installing zuul itself aiui | 23:43 |
| ianw | that doesn't seem to be updating either? | 23:44 |
| clarkb | but ya clearly it not being updated hasn't been missed, maybe we should consider cleaning it up? | 23:44 |
| ianw | do we not publish zuul release tarballs to tarballs? | 23:48 |
| clarkb | ianw: not sdists I don't think | 23:49 |
| fungi | not since the switch to a dedicated zuul tenant i think | 23:49 |
| ianw | hrm, ok, so they only go to pypi? | 23:49 |
| fungi | afaik, yes | 23:49 |
| clarkb | and dockerhub via the docker images | 23:49 |
| ianw | we should probably rm https://tarballs.opendev.org/zuul/zuul/ to avoid confusion | 23:49 |
| clarkb | before we do that we should ensure this isn't an oversight corvus would prefer we fix | 23:50 |
| clarkb | (but it seems that no one is using that content so maybe cleaning it up is the way to go) | 23:50 |
| ianw | right; it uses https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/release-python/post.yaml which only updates to pypi | 23:52 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!