Monday, 2021-07-26

*** marios is now known as marios|ruck05:41
*** amoralej|off is now known as amoralej06:45
*** rpittau|afk is now known as rpittau07:55
*** ykarel_ is now known as ykarel08:52
*** ykarel_ is now known as ykarel09:03
opendevreviewAndreas Jaeger proposed openstack/project-config master: Retire ara projects
*** dviroel|out is now known as dviroel11:53
*** amoralej is now known as amoralej|lunch12:55
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container
*** amoralej|lunch is now known as amoralej13:50
*** gibi is now known as gibi_pto14:03
opendevreviewAnanya proposed opendev/elastic-recheck master: Run elastic-recheck container
*** ykarel is now known as ykarel|away14:56
clarkbinfra-root as indicated last week I intend on running the external id conflict cleanup script against the accounts that were retired recently today. Probably after I settle in a bit14:58
clarkbAs far as I know we haevn't seen any of those users complain since their accounts were retired so I don't need to reduce the list14:59
fungiyep, not a peep afaik15:00
fungisounds like a good plan15:00
clarkbI am proceeding with the external id cleanups now. I will rerun the consistency check after. I expect we'll have a couple of new errors in the consistency checks from the accounts we addressed during the downtime. Specifically the new accounts we didn't want will likely have preferred email addresses set without corresponding external ids15:53
clarkbif those two pop up as issues I'll clean them up too15:53
clarkbAnd then I think there are a few accounts like our old infra stackalytics account that I can also ninja cleanup15:54
fungiyeah, that can certainly go16:00
clarkbthen maybe next week it is time to start putting together a large all in one fixup change for our active users16:03
clarkbI think the list is approaching a reasonable size after this cleanup round16:03
clarkbcleanup script is done running. Log is in the usual location16:05
rpittauhi all! Quick question, is the "branches" option supposed to work on zuul templates?16:07
clarkbrpittau: I don't think so, but you can double check the zuul docs to confirm16:08
rpittauclarkb: I was afraid of that, I did a quick test and zuul complained :/16:09
rpittauI'll check the docs again16:09
dtantsurI couldn't find anything in the docs, maybe I haven't looked well enough16:10
fungiwhat specifically are you trying to make it do?16:10
rpittauI checked before too, maybe I missed it16:10
rpittaufungi: we'd like to run the lower-constraints job in master branch only16:10
fungiand when you say "zuul templates" you mean project-templates?16:10
rpittaufungi: yes, project-templates16:11
fungiwhat's the template, and which lower-constraints job is it including? any reason not to set the branches parameter in the job definition?16:11
clarkbconfig consistency checks run in about 90 seconds now. Much quicker than before \o/ confirmed the two accounts we addressed during the downtime have preferred email problems16:11
clarkbI'll cross check against my notes then properly retire those two unneeded accounts16:12
rpittaufungi: the template is openstack-lower-constraints-jobs16:12
rpittauwe can't exactly modify that as it's run by the entire openstack community16:12
fungiwell, don't be so hasty to assume... there seemed to be some consensus on the ml to switch to only running lower-constraints jobs on master branches16:13
rpittauah! :)16:13
fungimaybe everyone would be okay with that?16:13
rpittauwell I guess I cant try to submit a change in zuul and create a master-only template16:14
fungibut yeah, worst case, make a openstack-lower-constraints-master-branch-jobs temlpate which includes master-branch-only variants of the same jobs16:14
rpittauthanks fungi , if the community is interested maybe they will start adopting that template instead16:15
fungirpittau: that's also something for the qa team and tact sig to potentially push on16:17
gmannrpittau: fungi consensus on l-c testing is - it is up to project to test it or not or even test on master only 16:20
gmannnot sure what is benefit of making master-on template as openstack-lower-constraints-jobs can be removed from stable branch gate if any project do not want to run it on stable ?16:21
rpittaugmann: yes, speaking for the ironic project, we removed the l-c tests but we'd like to re-introduce it only for the master branch, that's why the questioning16:21
rpittaugmann: one less task to do during the stable branch cut16:21
gmannrpittau: i see. 16:22
gmannrpittau: +116:22
rpittauTL;DR I'm lazy :D16:22
rpittauthanks gmann :)16:22
rpittaubtw change is up
clarkbalright the three additional cleanups are done now. I've copied updated logs to the typical location. Rerunning consistency checks now16:28
clarkbconsistency checking looks a bit better after the followup cleanups. Those results and all the lgos are in the usual location now. Now to rerun the audit script to produce a current list of conflicts. But then I think this task is done for today16:31
*** amoralej is now known as amoralej|off16:34
*** rpittau is now known as rpittau|afk16:37
opendevreviewMerged opendev/puppet-infra-cookiecutter master: Fix gitreview host name
clarkbThere are 105 remaining conflicts.16:42
clarkbMaybe still a bit much for doing an all in one fixup change? I'll have to look through the audit results and see if there are additional accounts we can cleanup in a more brute force manner16:43
*** marios|ruck is now known as marios|out16:54
*** dviroel is now known as dviroel|afk16:56
sshnaidmclarkb, hi, do you know if zuul has a user on for publishing collections?17:04
opendevreviewMerged openstack/project-config master: Retire ara projects
sshnaidmI want to create a job which will publish collection after it's tagged, wonder if I can do it on behalf of zuul user17:05
fungii don't think we have any jobs publishing collections yet, but that would probably need to be done with per-project credentials anyway... the collections repository is namespaced right?17:06
fungiwould likely need to be done similar to how we push to project-specific namespaces on dockerhub or github17:07
clarkbfungi: ++17:08
fungidoing it with zuul secrets also puts the credentials management squarely in the hands of the project leadership or their delegates, so that the opendev sysadmins don't have to create or maintain those namespaces and accounts17:11
opendevreviewClark Boylan proposed opendev/system-config master: Test the rename_repos playbook
clarkbfungi: ^ thats not quite converged between testing and prod yet and I half expect it to fail due to the large scale chagnes I made to the test job side of things. But its getting closer17:14
clarkbfungi: two things come up from that. FIrst is do you know if the first user is always 'admin' in development mode? And the other is do you know if we need to update our host lists in rename_repos.yaml to be review:!disabled or similar to prevent it from trying to run against review01 as well?17:14
clarkbActually review01 may not be in inventory anymore /me checks17:14
clarkbya looks like review01 is out of the inventory so the second thing is probably a non issue17:15
fungii think in development mode it ignores the username entirely?17:15
clarkbcool in that case my change may work17:15
clarkbwhich was basically s/admin/openstack-project-creator/ to make testing look more like prod17:16
fungilooking to see what we did with git-review tests now17:16
fungiwe seem to use admin:secret as the username:password combo in the git-review tests with auth.type=DEVELOPMENT_BECOME_ANY_ACCOUNT and then call create-account via the ssh cli to create additional non-admin users17:19
clarkbif the account name change works then I think the last remaining piece to make that mergable is figuring out how to run rename_repos.yaml in a more general way17:20
clarkbfungi: you may want to look over the change to get a general idea for the types of change that are necessary17:20
clarkbI do think testing that if we can is a really good idea17:20
fungiagreed, if nothing else it will highlight where new gerrit releases may make our renaming process harder17:21
sshnaidmfungi, I will make credentials, but I think maybe to use a specific CI user for that? I definitely don't want to use my own. Which user do you use for publishing to dockerhub/github?17:51
fungisshnaidm: it depends on the namespace to which we're publishing. each namespace has its own publishing account and associated credentials17:57
sshnaidmok, so I need just to create a new one17:58
sshnaidmfungi, also can you please give a link to how to create job that is triggered by tagging? I think I saw such (post-release?)17:58
fungiproject leaders or someone to whom they've delegated the responsibility creates the account at the publication site and then encodes the related authentication credentials into a zuul secret to be used when that project calls the publication job17:58
sshnaidmfungi, ack17:59
fungiif you're thinking of the release-post pipeline, that's not a pipeline triggered by tagging, but rather a dedicated post-merge pipeline for the openstack/releases repository so it can be given priority for node requests18:00
fungii'd have to know more about how collections publication works to suggest a particular mechanism. for example publishing to dockerhub we recommend doing in the gate pipeline, and then using the promote and release pipelines to tag the created artifacts with certain labels or versions in the publication site18:01
fungior maybe it's that we build images in the gate pipeline and then have the promote jobs push those into dockerhub, i'd have to look back at the flowchart18:02
sshnaidmfungi, well, I plan just to push a tag with collection version, zull job will be triggered, will build a tarball with collection and will publish it to with API token (secret)18:07
sshnaidmfungi, it will build collection with a tag version inside18:07
sshnaidmso I need a job that is triggered by tag and the tag itself can be discovered inside the job18:09
fungisshnaidm: in that case any job run in the release pipeline will be triggered on a tag which looks like a 3-part semver version number, the pre-release pipeline will match on tags which look like semver pre-release strings, and the tag pipeline will match on any tag18:11
sshnaidmfungi, will be like that:
sshnaidmfungi, sounds like release pipeline is what I need18:12
sshnaidmfungi, what is tag variable in the job?18:13
fungieasiest way to find out is to look at an example by filtering the zuul builds ui for the release pipeline and then look at the inventory it includes:
fungiaccording to that i think it's zuul.tag in vars18:17
fungiyou could also parse it out of the zuul.ref var18:18
clarkbfungi: seems that the account being admin is implied18:23
clarkbthis means I need to do a bit of work to add another user18:23
opendevreviewClark Boylan proposed opendev/system-config master: Test the rename_repos playbook
clarkbmaybe that will work18:32
clarkbfungi: do you think project rename email announcement is worth going to -announce? I suppose so18:36
clarkbyuriys: hello18:39
yuriyshey clarkb, at least got this set up now lol18:40
yuriyswas the oft forceconvert? do i just nuke all of my previous freenode autojoins18:40
yuriysor is it per project18:40
yuriysas it usually is18:41
clarkbThis was pretty global. We can't easily run bots for both networks and projects like openstack decided to switch entirely18:41
clarkbin this case the writing on the wall awas pretty clear then a few weeks after we moved freenode dropped the original network too and switched to a new cluster with new services18:42
clarkbinfra-root I've grabbed the user cleanup notes I wanted from review-test. Would probably be good if others that used that server for testing etc double check it too before we clean it up? cc fungi mordred in particular I think18:54
clarkbbut then we should be able to clean up review01 and review-test together18:54
clarkbI still need to look at other info on review01 outside of the user cleanup stuff to see if I want to preserve any of that18:55
mordredclarkb: I don't think I have anything remaining on review-test19:05
clarkbmordred: thanks for confirming19:07
clarkbanything else to add to the infra meeting agenda? I'll probably wait for ianw to start the day before sending it out but get your updates now :)19:29
clarkbthat is really odd, the rename repo playbook test timed out after doing apparently nothing for 12 minutes?
clarkbthe sshd_log is empty making me think that we aren't actually successfully doing those ssh commands19:55
opendevreviewClark Boylan proposed opendev/system-config master: Test the rename_repos playbook
clarkbmaybe a known_hosts issue?19:59
*** dviroel|afk is now known as dviroel20:04
clarkbI took afs disk usage off the agenda beacuse we seem to have stablized below 90% utilization20:08
clarkband have plans to clean up stretch in the near future20:09
fungiyeah, i've seen no objections (nor responses of any kind) to my proposal to do that20:09
clarkbto answer my question about the github PR closer that is happening with the openstack-mirroring account I assume via the zuul jobs that mirror stuff20:11
fungiand so probably only happens for the repos being synced that way?20:13
clarkbyup, but I think those are the only ones we need to worry about20:15
fungigood point20:16
clarkbok I think it was a known_hosts issue because I haev successful jobs now looks like20:48
clarkbI guess that means I have to figure out a good way to run the rename playbook. Maybe I just do that from test-review.yaml20:48
clarkband test-gitea.yaml I guess20:49
opendevreviewClark Boylan proposed opendev/system-config master: Test the rename_repos playbook
clarkbif ^ works I think we may be in a mergable state. The only real delta on the production side is we use an ssh -p 29418 localhost instead of ssh -p 29418 when reindexing21:00
clarkbthat and the wait for gerrit to start up code is changed to use wait_for21:01
opendevreviewAlex Schultz proposed opendev/system-config master: Add cdn0{1,2}
fungiclarkb: huh, not sure if you saw my comments, but surprised openssh didn't mind /home/gerrit2/.ssh/known_hosts being root-owned21:31
clarkbfungi: ya I thought about that but I suspect it is because it is root. If it was another user it would probably complain?21:48
clarkb passes testing now and I think it is a mergable state. Going to double check the job logs to make sure it does what we expect22:03
clarkbyup seems to do what we want and actually runs the rename_repos.yaml playbook without redoing the base job framework22:06
clarkbfungi: I responded to your comments on the earlier ps22:11
opendevreviewMerged opendev/system-config master: Add cdn0{1,2}
clarkbianw: when your day starts can you let me know if you want ot add anything to the meeting agenda? I'll send that out once you've given the all clear (or in the next hour if not received otherwse)22:20
ianwclarkb: nothing from me22:37
clarkband agenda sent22:40
clarkbdonnyd: hey if you are around I'd like to fix a gerrit email address conflict between two accounts for you.22:48
clarkbdonnyd: if you can confirm the account id number you get when you login to that would help me confirm that my plan here is valid22:49
clarkbfungi: as a heads up I'm putting together anothe batch that I think we can do without much user involvement. I'm getting bolder now that we've tested the fixup for other accounts recently22:53
clarkbdefinitely still a number I'm skipping over as good for reaching out though22:54
fungisure, sounds great!22:54
fungiits in the home stretch now22:54
*** prometheanfire is now known as Guest235222:56
clarkbI also figure that once we get all of these done we'll be able to do online updates as well so getting there opens things up a bit more22:56
*** ChanServ changes topic to "OpenDev is a space for collaborative Open Source software development | | channel logs"22:57
clarkbwhat changed in that topic update?22:58
clarkbfungi: ^ do you know why we seemed to update our channel topics to their current channel topics?23:04
clarkbdpawlik: same question as for donnyd above. You have a couple of accounts that conflict on an email address. I'm fairly certain I see the correct way to clean it up but if you can pm me the account id you see on the gerrit settings page when you log in that would help me to confirm it23:10
fungiclarkb: my guess is a services restart did that23:14
clarkbok I've got new proposed cleanups list short dpawlik and donnyd's cleanups though I'm fairly certain we can do those two as well. But scp is failing because my ssh key manager is telling me its been a day already23:24
clarkbI'll push these up tomorrow when I reload keys23:24
fungiyep, it's time to knock off23:24
fungi(i'm not even sure why i'm here!)23:25
clarkbof course if I stop keyboarding I'll have to figure out dinner23:28
fungithat already happened to me, so i made (and consumed) phat si-io23:29
clarkbthat sounds really good23:30
clarkbianw: may interst you as it touches gerrit testing in particular.23:31
clarkbianw: trying to make a change that is mergable that will run our rename repos playbook against gerrit and gitea when we touch those services23:31
fungiit's nearly impossible to find chinese broccoli here, and we tried growing it with minimal success (thanks to a proliferation of cabbage moths/worms), so we either use store-bought western broccoli (and alter the cooking process with an additional step of steaming it by pouring sake in and slapping a lid on the wok) or we use greens from our garden (chard actually substitutes well)23:33
ianwclarkb: hrm, perhaps we should abstract things into a role so we don't have to call another nested ansible?23:45
Clark[m]ianw: but we'll run it during the downtime with that playbook... If we can keep it as much like what we'll actually run that may be better?23:54

Generated by 2.17.2 by Marius Gedminas - find it at!