Wednesday, 2021-04-14

*** ramishra has joined #openstack-infra00:01
*** ramishra has quit IRC00:06
*** ramishra has joined #openstack-infra00:12
*** dviroel has quit IRC00:32
*** gyee has quit IRC00:38
*** tkajinam has quit IRC00:55
*** tkajinam has joined #openstack-infra00:56
*** carloss has quit IRC01:02
*** zzzeek has quit IRC01:07
*** zzzeek has joined #openstack-infra01:08
*** ramishra has quit IRC01:08
*** ramishra has joined #openstack-infra01:14
*** zzzeek has quit IRC01:33
*** zzzeek has joined #openstack-infra01:35
*** zzzeek has quit IRC01:54
*** zzzeek has joined #openstack-infra01:57
*** beagles has quit IRC02:15
openstackgerritIan Wienand proposed openstack/project-config master: Add OSUOSL resources to nodepool  https://review.opendev.org/c/openstack/project-config/+/78615602:26
*** hamalq has quit IRC02:29
*** zzzeek has quit IRC02:55
*** zzzeek has joined #openstack-infra02:57
*** zzzeek has quit IRC03:30
*** zzzeek has joined #openstack-infra03:31
*** rcernin has quit IRC03:54
*** rcernin has joined #openstack-infra04:00
*** ykarel has joined #openstack-infra04:14
*** rlandy|rover has quit IRC04:33
*** vishalmanchanda has joined #openstack-infra04:44
*** whoami-rajat_ has joined #openstack-infra05:20
*** ralonsoh has joined #openstack-infra05:21
*** tinwood has quit IRC06:22
*** ysandeep|holiday is now known as ysandeep06:24
*** tinwood has joined #openstack-infra06:25
*** zbr has quit IRC06:30
*** zbr has joined #openstack-infra06:32
*** eolivare has joined #openstack-infra06:32
*** csatari has quit IRC06:34
*** ajitha has quit IRC06:34
*** PrinzElvis has quit IRC06:34
*** knikolla has quit IRC06:34
*** rpioso has quit IRC06:34
*** Nebby has quit IRC06:34
*** sboyron has joined #openstack-infra06:35
*** zaro has quit IRC06:35
*** srwilkers has quit IRC06:35
*** lxkong has quit IRC06:35
*** knikolla has joined #openstack-infra06:36
*** ajitha has joined #openstack-infra06:36
*** Nebby has joined #openstack-infra06:36
*** ajitha has quit IRC06:36
*** csatari has joined #openstack-infra06:36
*** jcapitao has joined #openstack-infra06:36
*** rpioso has joined #openstack-infra06:37
*** lxkong has joined #openstack-infra06:37
*** PrinzElvis has joined #openstack-infra06:37
*** zaro has joined #openstack-infra06:37
*** srwilkers has joined #openstack-infra06:37
*** lpetrut has joined #openstack-infra06:57
*** amoralej|off is now known as amoralej06:59
*** amoralej is now known as amoralej|off06:59
*** dklyle has quit IRC07:10
*** andrewbonney has joined #openstack-infra07:14
*** ociuhandu has joined #openstack-infra07:27
*** ociuhandu has quit IRC07:31
*** ociuhandu has joined #openstack-infra07:31
*** AJaeger has joined #openstack-infra07:35
*** ociuhandu has quit IRC07:41
*** ociuhandu has joined #openstack-infra07:42
*** ociuhandu has quit IRC07:44
*** ociuhandu has joined #openstack-infra07:44
*** tosky has joined #openstack-infra07:50
*** derekh has joined #openstack-infra07:54
*** rpittau|afk is now known as rpittau07:55
*** jpena|off is now known as jpena07:57
*** gfidente|afk is now known as gfidente08:01
*** lucasagomes has joined #openstack-infra08:01
*** AJaeger has quit IRC08:07
*** dchen has quit IRC08:09
*** slaweq_ has joined #openstack-infra08:19
*** slaweq has quit IRC08:19
*** rcernin has quit IRC08:24
*** ricolin has quit IRC08:39
hberaudHi infra team, how is the situation from your side?08:45
hberaudI take the temperature before starting the final release of Wallaby08:48
*** slaweq_ is now known as slaweq08:51
*** rcernin has joined #openstack-infra09:12
*** amoralej|off is now known as amoralej09:20
*** yamamoto has quit IRC09:28
*** hjensas|afk is now known as hjensas09:36
*** dtantsur|afk is now known as dtantsur09:45
*** yamamoto has joined #openstack-infra09:59
*** yamamoto has quit IRC09:59
*** yamamoto has joined #openstack-infra10:00
*** hberaud has quit IRC10:13
*** fresta has joined #openstack-infra10:16
*** whoami-rajat_ is now known as whoami-rajat10:17
*** hberaud has joined #openstack-infra10:17
*** carloss has joined #openstack-infra10:19
openstackgerritAndreas Jaeger proposed openstack/openstack-zuul-jobs master: Start translations for Wallaby  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/78619810:26
fungihberaud: i'm around now and things are looking okay so far10:35
hberaudfungi: excellent, thanks for feedback10:35
*** ociuhandu has quit IRC10:38
*** ociuhandu has joined #openstack-infra10:59
*** ociuhandu has quit IRC10:59
*** ociuhandu has joined #openstack-infra11:00
*** jcapitao is now known as jcapitao_lucnh11:01
*** jcapitao_lucnh is now known as jcapitao_lunch11:01
*** rcernin has quit IRC11:04
*** ociuhandu_ has joined #openstack-infra11:04
*** yamamoto has quit IRC11:06
*** ociuhandu has quit IRC11:08
*** eolivare_ has joined #openstack-infra11:12
*** tkajinam has quit IRC11:14
*** eolivare has quit IRC11:15
*** ociuhandu_ has quit IRC11:20
*** ociuhandu has joined #openstack-infra11:20
*** dviroel_ has joined #openstack-infra11:21
*** jpena is now known as jpena|lunch11:32
*** yamamoto has joined #openstack-infra11:32
*** dviroel_ is now known as dviroel__11:35
*** eolivare_ has quit IRC11:35
*** rlandy has joined #openstack-infra11:35
*** dviroel__ is now known as dviroel11:36
*** rlandy is now known as rlandy|rover11:36
*** ykarel_ has joined #openstack-infra11:37
*** hemna has quit IRC11:39
*** hemna has joined #openstack-infra11:39
*** yamamoto has quit IRC11:39
*** ykarel has quit IRC11:40
*** rcernin has joined #openstack-infra11:45
*** ykarel_ is now known as ykarel11:47
*** ociuhandu has quit IRC11:52
*** ociuhandu has joined #openstack-infra11:53
*** rcernin has quit IRC11:59
*** ociuhandu has quit IRC12:02
*** ociuhandu has joined #openstack-infra12:03
*** ociuhandu has quit IRC12:08
*** ociuhandu has joined #openstack-infra12:14
*** rlandy|rover is now known as rlandy|rover|mtg12:14
*** eolivare_ has joined #openstack-infra12:16
*** ociuhandu has quit IRC12:17
*** ociuhandu has joined #openstack-infra12:18
*** amoralej is now known as amoralej|lunch12:21
*** jcapitao_lunch is now known as jcapitao12:22
*** nweinber has joined #openstack-infra12:27
*** AJaeger has joined #openstack-infra12:28
AJaegerconfig-core, please review https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/786198 for Wallaby translations12:28
*** jpena|lunch is now known as jpena12:29
fungithanks AJaeger! approved it now12:39
openstackgerritMerged openstack/openstack-zuul-jobs master: Start translations for Wallaby  https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/78619812:44
*** ysandeep is now known as ysandeep|afk12:46
AJaegerthanks, fungi!12:49
fungimy pleasure, as always12:50
AJaeger;)12:50
janderssee you tomorrow Ironic o/13:01
openstackgerritMerged openstack/project-config master: Allow delete permissions in Gerrit ACLs  https://review.opendev.org/c/openstack/project-config/+/78608813:02
*** amoralej|lunch is now known as amoralej13:02
openstackgerritMerged openstack/project-config master: Add puppet-cinder-core and puppet-glance-core  https://review.opendev.org/c/openstack/project-config/+/78488813:03
*** AJaeger has quit IRC13:04
janderssorry wrong channel :)13:06
yoctozeptojanders: yup, in here we say "see you tomorrow, sarcastic"13:12
fungiwell, you might see them in here too ;)13:12
*** tkajinam has joined #openstack-infra13:14
tkajinamhi. may I ask somebody from infra team to add puppet-manager-core to puppet-cinder-core and puppet-glance-core ?13:16
tkajinam(I'm talking about groups in gerrit13:16
*** ysandeep|afk is now known as ysandeep13:25
fungitkajinam: sure, just a moment13:25
fungitkajinam: done13:28
*** ricolin has joined #openstack-infra13:34
*** rlandy|rover|mtg is now known as rlandy|rover13:35
tkajinamfungi, thanks !13:38
fungiany time13:38
*** arxcruz has quit IRC13:57
*** arxcruz has joined #openstack-infra13:59
*** yamamoto has joined #openstack-infra14:01
*** slaweq has quit IRC14:16
*** slaweq has joined #openstack-infra14:20
*** ykarel_ has joined #openstack-infra14:20
*** ykarel has quit IRC14:22
*** yamamoto has quit IRC14:24
*** ykarel_ is now known as ykarel14:29
*** lpetrut has quit IRC15:00
*** zxiiro has joined #openstack-infra15:07
*** dviroel is now known as dviroel|lunch15:07
*** dklyle has joined #openstack-infra15:09
*** ociuhandu has quit IRC15:09
*** ociuhandu has joined #openstack-infra15:12
*** d34dh0r53 has quit IRC15:20
*** d34dh0r53 has joined #openstack-infra15:22
*** ysandeep is now known as ysandeep|away15:25
*** ociuhandu_ has joined #openstack-infra15:36
*** rcernin has joined #openstack-infra15:39
*** ociuhandu has quit IRC15:40
*** ociuhandu_ has quit IRC15:40
*** rcernin has quit IRC15:43
*** dviroel|lunch is now known as dviroel15:45
*** rlandy|rover is now known as rlandy|rover|mtg16:01
*** lucasagomes has quit IRC16:04
*** gyee has joined #openstack-infra16:17
*** rpittau is now known as rpittau|afk16:21
*** gfidente is now known as gfidente|afk16:25
*** jcapitao has quit IRC16:25
*** hamalq has joined #openstack-infra16:28
*** geguileo has quit IRC16:38
*** geguileo has joined #openstack-infra16:42
*** ralonsoh has quit IRC16:45
*** eolivare_ has quit IRC16:48
*** geguileo has quit IRC16:55
*** geguileo has joined #openstack-infra16:55
*** rlandy|rover|mtg is now known as rlandy|rover17:05
*** derekh has quit IRC17:05
*** clarkb has quit IRC17:08
*** clarkb has joined #openstack-infra17:08
*** jpena is now known as jpena|off17:10
*** amoralej is now known as amoralej|off17:24
elodnow that wallaby is released... do i feel it right the this issue (which hits xenial based jobs) is the known pypi mirrors' "missing metadata" issue? >>> http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22distutils.errors.DistutilsError:%20Could%20not%20find%20suitable%20distribution%20for%20Requirement.parse('pbr%5C%22&from=864000s17:29
fungiwasn't that just fixed? maybe it hasn't been backported to stable branches yet17:30
fungihad to do with how stackviz was being installed17:31
fungimaybe ask in #openstack-qa if it's not clear from the ml thread17:31
clarkbrelated: centos7 will liekly have issues next month when pypi drops no sni support17:31
clarkbspecifically for any requests that happen via easy_install and don't hit our mirrors (pbr being a big example of this)17:32
elodfungi: the workaround for stackviz issue is merged and that works.17:33
elodthis "Could not find suitable distribution for Requirement.parse('pbr'>XXX)" thing comes sometimes more earlier, e.g. when tox is being installed17:34
clarkbelod: can you link to a specific instance (the logstash records have the log links in them)17:34
*** ykarel has quit IRC17:35
elodclarkb: let me search for one17:36
clarkbI half wonder if this is sni too17:38
elodhttps://zuul.opendev.org/t/openstack/build/7aab4575d0284ea2adfcbc2da9e0d4dc/log/job-output.txt#62017:38
clarkbbut I thought I double checked and xenial's python is new enough to sni17:38
elodclarkb: maybe this one as a good example? ^^^17:38
clarkbelod: ya, so that confirms it is failing in easy_install at least. And we don't configure our mirrors for easy_install17:39
*** rcernin has joined #openstack-infra17:39
clarkbthat means it should be trying to talk to https://pypi.org/simple/pbr to get that index listing17:39
elodso this is then - as i feared - an issue at pypi side? :S17:41
clarkbhttps://github.com/pypa/pypi-support/issues/978 says the brownouts would'ev happened around then17:41
clarkbelod: yes, either pypi actually failed to serve the content or xenial is susceptible to the lack of sni like centos717:42
fungiwarehouse is doing sni brownouts, so yeah could be17:43
clarkbpython2.7 supports sni when >= 2.7.9. Xenial reports 2.7.12 https://packages.ubuntu.com/xenial/python2.717:43
clarkbhttps://docs.python.org/2/whatsnew/2.7.html#pep-466-network-security-enhancements-for-python-2-717:43
clarkbthis is why I thought that python2.7 on xenial was fine but that is the sort of error that i think we had also seen on centos7 that we thought was likely the sni brownouts17:44
*** dtantsur is now known as dtantsur|afk17:44
clarkbthat said I think openstack should start looking at winding down xenial testing17:44
*** rcernin has quit IRC17:44
clarkb(doesn't have to happen overnight, but maybe we punt on this as that happens)17:44
fungiyeah, xenial is no longer receiving updates, it's hard to guarantee that we'll be able to continue building usable images for it17:45
elodxenial is up till Rocky17:45
elodI see17:46
fungidropping centos-7 testing is likely a harder pill to swallow17:46
fungibut yeah, xenial is eol from canonical's perspective17:46
fungitraditionally we haven't continued to maintain distro images past eol17:46
clarkbfungi: and centos-7 does report an older version of python2.7 that does not have sni support (though unsure if red hat backported it or not)17:46
clarkbthat likely means we'll want to address this somehow either way17:48
clarkb(assuming this is the sni brownouts causing the issue)17:48
clarkbone method would be to preinstall pbr17:49
clarkbwell more specifically preinstall it using pip17:49
clarkbthat way easy_install shoudl see it as already present. And pip won't fail because it will talk to our mirror which will do sni to pypi on the backend17:49
fungiright, that should be a tractable workaround if someone wants to work on it17:51
clarkbanother is to force easy_install to use our mirror (though that is lcunky and we removed it bceause it didn't work great iirc)17:51
elodthe pbr preinstall sounds interesting :) can it be done by some extra zuul task towards xenail based jobs?17:54
fungiyeah, you should be able to just add a role which does it17:54
elodsimilarly like ensure-* I guess?17:55
clarkbya, where it will get tricky is when it is a tox venv like in your example17:55
clarkbsince you'll want to find someway for tox to run pip install pbr before anything else after creating the venv17:55
fungiyou can anticipate the tox virtualenv name17:56
fungiand path17:56
clarkboh ya you could precreate the venv for tox17:56
fungiand tehn it shouldn't try to recreate17:56
elodok, I'll try to look into this as this pbr install issue hits quite many job runs :S17:59
*** andrewbonney has quit IRC17:59
clarkbelod: if you look at the brownout schedule they will happen more and more often until finally they turn it off. If you see it happening more often that would be another good indicator that it is related18:00
elodclarkb: according to the link it is quite often already, but yes, I'll try to doublecheck :S thanks!18:01
elodfungi: btw, about EOL branch deletion... o:) is there something that is missing for that? I want to run the delete script towards at least the eol'd repositories of ocata18:03
elod(ocata is unmaintained for ~ a year now, so most probably all ocata branch should be eol'd + deleted soon)18:06
*** ianychoi__ has quit IRC18:08
fungielod: the next step there is https://review.opendev.org/782830 and once that merges we can try having a project like openstack/release-test inherit from the openstack/meta-config acl18:08
fungiand then try to use release automation to delete a branch from openstack/release-test18:08
clarkbfungi:  Ithink you can approve that one if you are ready fwiw18:09
fungiahh, yeah will do now18:09
clarkbI didn't approve it only because we were having that converstaion about some of the potentialy weirdness at play18:09
elodfungi: sounds good!18:09
clarkbbut I think we reached teh conclusion that while weird it didn't present an actual problerm18:09
fungii've approved it, and will propose the change for openstack/release-test once that's created18:10
clarkbfungi: is that something we can manage through jeepyb? or do we have to do it through gerrit directly at the moment?18:10
clarkb(changing the parent I mean)18:10
elodfungi: btw, the delete script is interactive as it asks for deletion + gerrit username + password ( https://review.opendev.org/777446 )18:13
fungielod: in that case it can be tested initially by someone we add to the release managers group18:18
openstackgerritMerged openstack/project-config master: Add an empty project for an OpenStack base ACL  https://review.opendev.org/c/openstack/project-config/+/78283018:21
elodok, maybe I can test it some time? :) or do you want to test it by yourself?18:21
fungiif you're in the release managers group you'd be able to test it18:23
elodI am18:25
elodshould I try it today, or was the Wallaby release enough excitement for the day? :)18:27
clarkbI think there are a couple of bits to tie together (not sure what time zone you are in)18:28
elodCET18:28
elodso it's pretty late already here18:28
clarkbin that case maybe you should call it a day its getting late ;)18:28
elod:)18:29
elodwell, i could use some dinner...18:29
JayF118:31
JayFWhoops, sorry, misclick.18:31
JayFWill you all be emailing the list when stable/ocata branches are killed and ocata-eol tags made?18:31
JayFAnd I'm assuming it'd be too much to ask for there to be some overlap in the tag and branch existing :D18:31
elodso maybe some quiet period, like friday, when you are also around, I can do the test against an already eol'd ocata branch, just in case18:32
elodJayF: there are already some repositories EOL'd (but not yet deleted)18:32
elodfor ocata18:33
JayFI'm less concerned about the EOL itself, and more concerned about the branches disappearing, as it will have a negative impact on downstream CI that consumes them.18:33
JayF"No" is a fine answer, as I know it's not supported, but it never hurts to ask :D18:33
elodJayF: i see18:33
elodJayF: only such branches will be deleted (for now) that has already been announced that their stable/ocata is eol'd18:34
elodbut yes, maybe I can send some warning to the mailing list18:35
JayFIf you set a day for it, and document what repos, that'd be super helpful in helping us plan to adjust our CI in sync.18:35
*** gshippey has quit IRC18:37
elodJayF, ok, I will do that18:37
JayFI really appreciate it! Thank you.18:38
elodthen maybe it won't happen on friday, but we need some puffer time18:38
*** whoami-rajat has quit IRC18:38
*** geguileo has quit IRC18:40
elodand maybe it's worth to discuss it within the release team18:41
*** jlvillal has quit IRC18:41
*** jlvillal has joined #openstack-infra18:42
elodJayF: have you tried btw to run your downstream CI against the ocata-eol tag (for projects where it's already exists)?18:42
elodas it should remain there in the repository after the branch deletion18:43
elodlike at older releases18:43
elodanyway, I'll drop a mail to the list18:43
*** vishalmanchanda has quit IRC18:44
JayFelod: I expect it to work, but we haven't configured it. Honestly this wasn't even on the radar until I saw these messages. You do what you gotta do, we'll figure it out, the warning with a date is appreciated though so we can prep for it :D18:45
elod:)18:48
fungielod: the next step will be to work out how to configure the acl for openstack/release-test to depend on the openstack/meta-config acl, then push a change adding that and get it approved and deployed18:48
fungiafter that, we test deleting a branch from openstack/release-test (possibly after manually creating it)18:49
fungiif that works as expected, next we propose a similar change to make all repositories in the openstack/ git namespace inherit from the openstack/meta-config acl18:50
fungiafter that, using the script to manually delete branches from any of those projects should be possible as long as a release manager account is used to run it, and it could also be integrated into release automation18:50
fungiand then for cleanup, a gerrit admin needs to manually remove the openstack-specific allowances for the various other permissions from our global config, now that they're inheritable from openstack/meta-config by the specific projects which need them18:52
elodfungi: it sounds good to test it first with openstack/release-test! I can try to figure out tomorrow how to configure the acl for release-test (i'm not that familiar with gerrit configuration, though)18:57
elodso let's do that as a first step tomorrow :)18:58
fungiJayF: worth keeping in mind, any of the branches slated for eventual deletion are already tagged with a ${series}-eol tag, and will be receiving no new commits ever after that point18:58
fungideletion of those branches serves two purposes, first it makes it more obvious to consumers pulling from them that they should take a look at the repository since they may not be aware there are no more fixes appearing there ever again, but also it will signal to gerrit that it should reject any attempts to propose changes for those branches, so that they don't go unreviewed in perpetuity19:00
JayFfungi: yeah, that's what we're talking about downstream -- looking for an `ocata-eol` tag, if it exists, use that, if not, look for `stable/ocata` branch19:05
fungielod: https://review.opendev.org/Documentation/access-control.html#_project_access_control_lists refers to the documentation for a command-line solution https://review.opendev.org/Documentation/cmd-set-project-parent.html19:06
fungiprobably need to look into whether we can set that with the rest api instead19:07
fungimy guess is we'll need to extend the manage-projects tool in jeepyb to be able to set that from a field in the gerrit/projects.yaml file in openstack/project-config19:07
fungithe rest api solution appears to be https://review.opendev.org/Documentation/rest-api-projects.html#set-project-parent19:09
fungiclarkb: ianw: ^ you might have opinions19:10
*** sboyron has quit IRC19:30
*** nweinber has quit IRC20:00
*** ociuhandu has joined #openstack-infra20:00
*** ociuhandu has quit IRC20:04
clarkbfungi: I would probably use https://gerrit-review.googlesource.com/Documentation/cmd-set-project-parent.html and stick to ssh for simplicity20:38
clarkbyou may need to extend gerritlib for that, but all of the testing necessary to check it is owrking is there in gerritlib and jeepyb at least20:39
fungiyeah, that seems logical20:39
openstackgerritMerged openstack/project-config master: Add OSUOSL resources to nodepool  https://review.opendev.org/c/openstack/project-config/+/78615620:49
*** rcernin has joined #openstack-infra21:02
clarkbfungi: let me know if I can help with that at all. It was definitely a mistake to put the test job in gerritlib but I started adding the features there ..21:15
clarkbdefinitely would be easier to manage if the test was in jeepyb then you could add feature to gerritlib, depends on in jeepyb and confirm the stack before landing21:15
clarkbas is you have to do at least three changes instead, first to add the feature in gerritlib, next to update jeepyb, then third to udpate the test21:16
*** zxiiro has quit IRC21:45
*** rcernin has quit IRC21:45
*** rcernin has joined #openstack-infra21:46
*** jaicaa has quit IRC21:52
*** jaicaa has joined #openstack-infra21:53
openstackgerritIan Wienand proposed openstack/project-config master: OSU OSL : add diskimages to nb03  https://review.opendev.org/c/openstack/project-config/+/78633722:08
openstackgerritIan Wienand proposed openstack/project-config master: OSU OSL : add diskimages to nb03  https://review.opendev.org/c/openstack/project-config/+/78633722:10
*** rcernin has quit IRC22:20
*** yamamoto has joined #openstack-infra22:30
*** rlandy|rover is now known as rlandy|rover|bbl22:32
openstackgerritMerged openstack/project-config master: OSU OSL : add diskimages to nb03  https://review.opendev.org/c/openstack/project-config/+/78633722:34
*** tkajinam has quit IRC22:43
*** tkajinam has joined #openstack-infra22:52
*** hamalq has quit IRC22:57
*** hamalq has joined #openstack-infra22:57
*** yamamoto has quit IRC22:59
*** yamamoto has joined #openstack-infra22:59
*** rcernin has joined #openstack-infra23:06
*** tosky has quit IRC23:29
*** dchen has joined #openstack-infra23:32
*** irclogbot_3 has quit IRC23:50
*** irclogbot_3 has joined #openstack-infra23:56

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!