Tuesday, 2020-09-22

*** DSpider has quit IRC00:47
*** hamalq has quit IRC01:01
openstackgerritIan Wienand proposed opendev/bindep master: [WIP] Option to only match given profiles  https://review.opendev.org/75319801:03
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [dnm] merge-output test  https://review.opendev.org/75319902:05
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [dnm] merge-output test  https://review.opendev.org/75319902:06
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [dnm] merge-output test  https://review.opendev.org/75319902:10
*** hamalq has joined #opendev02:25
openstackgerritsebastian marcet proposed opendev/puppet-openstackid master: Fixed typo ( extra =)  https://review.opendev.org/75320503:27
*** ykarel|away has joined #opendev04:05
*** ykarel|away is now known as ykarel04:10
*** ykarel has quit IRC04:44
*** ykarel has joined #opendev04:46
AJaegerianw: could you review https://review.opendev.org/752745, please?05:19
openstackgerritMerged openstack/project-config master: Linaro US: Add a 16GB RAM label for CentOS 8  https://review.opendev.org/75303605:22
AJaegerclarkb, ianw , it's now the 22nd - want to switch fedora-latest to Fedora 32? https://review.opendev.org/#/c/752744/05:25
ianwAJaeger: not sure if devstack is working yet?  but anyway, i guess that's something we need to debug at any rate and waiting won't magically make it get better05:26
openstackgerritMerged openstack/project-config master: Switch grafana nodepool to Fedora 32  https://review.opendev.org/75274505:28
AJaegerianw: no idea about devstack...05:31
AJaegerianw: do you know why the  infra-prod-service-nodepool  deploy job failed for https://review.opendev.org/753036?05:31
*** ykarel has quit IRC05:33
cgoncalvesjob infra-prod-service-nodepool (Zuul deploy pipeline) in https://review.opendev.org/#/c/753036/ failed with: Timeout exception waiting for the logger. Please check connectivity to [bridge.openstack.org:19885]05:33
ianwcgoncalves: yeah, we don't log the output because it hasn't been fully audited to remove sensitive bits05:35
ianwi'm looking at the on-disk log file and it seems to be ok...05:35
ianwnb03.opendev.org           : ok=1    changed=0    unreachable=105:35
ianwi tell a lie05:35
ianwnb03.opendev.org                         | SHUTOFF05:36
ianwkevinz: ^ we need to figure out what's happening here :/05:36
AJaegerianw: https://review.opendev.org/#/c/752741/ remvoed nb03, didn't it?05:38
AJaegerWas that wrong?05:38
AJaegerclarkb: ^05:39
ianwAJaeger: that was nb03.openstack.org, not opendev.org (I hope :)05:39
AJaegerianw: Ah, good! Yeah, noticed the different domain now as well..05:39
ianwbut something shut the host down05:39
ianwhopefully i can catch kevinz at some point to see what logs might exist05:40
openstackgerritCarlos Goncalves proposed openstack/project-config master: CentOS 8 Stream initial deployment  https://review.opendev.org/73479105:50
*** ysandeep|away is now known as ysandeep05:50
openstackgerritIan Wienand proposed zuul/zuul-jobs master: [WIP] Revert "Disable broken fetch-sphinx-tarball test job"  https://review.opendev.org/75319905:52
openstackgerritCarlos Goncalves proposed openstack/diskimage-builder master: Remove doc files and autoremove packages  https://review.opendev.org/67554605:55
openstackgerritMerged zuul/zuul-jobs master: Disable broken fetch-sphinx-tarball test job  https://review.opendev.org/75295706:04
*** ykarel has joined #opendev06:08
*** hamalq has quit IRC06:11
openstackgerritMerged zuul/zuul-jobs master: Add managed jobs to periodic-weekly  https://review.opendev.org/74868206:20
*** qchris has quit IRC06:20
*** eolivare has joined #opendev06:27
*** diablo_rojo has quit IRC06:34
*** qchris has joined #opendev06:34
*** ykarel has quit IRC06:37
*** hashar has joined #opendev06:49
*** andrewbonney has joined #opendev06:53
*** slaweq has joined #opendev07:08
*** tosky has joined #opendev07:40
*** moppy has quit IRC08:01
*** moppy has joined #opendev08:02
*** kden has joined #opendev08:11
*** kden has quit IRC08:20
*** kden has joined #opendev08:20
*** priteau has joined #opendev08:23
*** priteau has quit IRC08:35
*** kden has quit IRC08:48
*** priteau has joined #opendev08:49
*** fressi has joined #opendev08:51
*** ysandeep is now known as ysandeep|lunch09:08
*** DSpider has joined #opendev09:17
openstackgerritAlfredo Moralejo proposed zuul/zuul-jobs master: Use Train repo to install openvswitch from multi-node-bridge in CentOS 8  https://review.opendev.org/75290809:20
*** fressi has quit IRC09:33
*** dtantsur has joined #opendev09:38
dtantsurhi folks! some log pages timeout currently, try e.g. https://zuul.opendev.org/t/openstack/build/737556b9764e424884d60758050dae8c/log/controller/logs/devstacklog.txt. May be temporary, but I decided to let you know.09:39
dtantsurthe underlying raw files download without issues09:39
*** ysandeep|lunch is now known as ysandeep09:56
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed zuul/zuul-jobs master: Add support to use stow for ensure-python  https://review.opendev.org/75161109:58
openstackgerritDmitriy Rabotyagov (noonedeadpunk) proposed zuul/zuul-jobs master: Add support to use stow for ensure-python  https://review.opendev.org/75161109:59
*** hashar is now known as hasharAway10:08
*** owalsh has quit IRC10:15
openstackgerritzbr proposed zuul/zuul-jobs master: Update ensure-docker for new releases  https://review.opendev.org/75263010:27
*** fressi has joined #opendev10:48
*** hasharAway is now known as hashar10:55
*** fressi has quit IRC11:02
*** priteau has quit IRC11:08
*** eolivare has quit IRC11:13
*** fressi has joined #opendev11:21
openstackgerritPierre-Louis Bonicoli proposed zuul/zuul-jobs master: explicit error when test_command is undef  https://review.opendev.org/75165911:39
*** owalsh has joined #opendev11:42
*** hashar has quit IRC11:43
*** owalsh has quit IRC11:49
fricklerdtantsur: that page works fine for me, might have been some temporary issue12:00
*** ysandeep is now known as ysandeep|brb12:05
*** owalsh has joined #opendev12:07
*** fressi has quit IRC12:11
*** hashar has joined #opendev12:16
fungiit would be interesting for the raw file to work but the cooked file to timeout... the latter is just javascript loading the raw file url and decorating it12:16
*** ysandeep|brb is now known as ysandeep12:16
fungiso the browser fetches the file content from the same place (cloud object storage) either way12:17
*** jhesketh has quit IRC12:32
openstackgerritMerged opendev/puppet-openstackid master: Fixed typo ( extra =)  https://review.opendev.org/75320512:32
*** fressi has joined #opendev12:46
*** fressi has joined #opendev12:48
*** jhesketh has joined #opendev13:19
*** fressi has quit IRC13:39
*** priteau has joined #opendev13:46
*** mlavalle has joined #opendev13:59
*** lpetrut has joined #opendev14:02
*** eolivare has joined #opendev14:24
*** ysandeep is now known as ysandeep|away14:27
*** hashar has quit IRC14:29
*** iurygregory has quit IRC14:46
*** iurygregory has joined #opendev14:51
*** lpetrut has quit IRC15:49
*** bhagyashri|rover has quit IRC16:08
*** bhagyashris has joined #opendev16:10
openstackgerritClark Boylan proposed openstack/project-config master: Add noop jobs to os-loganalyze  https://review.opendev.org/75339816:14
*** hamalq has joined #opendev16:16
*** redrobot has quit IRC16:25
*** johnsom has quit IRC16:26
*** johnsom has joined #opendev16:27
*** dtantsur is now known as dtantsur|afk16:34
*** mlavalle has quit IRC16:41
*** mlavalle has joined #opendev16:42
*** andrewbonney has quit IRC17:13
*** eolivare has quit IRC17:26
*** hamalq has quit IRC17:41
openstackgerritAdam Coldrick proposed opendev/storyboard master: Allow Project Groups to be filtered by project  https://review.opendev.org/75341717:52
*** diablo_rojo has joined #opendev18:03
openstackgerritClark Boylan proposed openstack/project-config master: Add noop jobs to os-loganalyze  https://review.opendev.org/75339818:35
fungilooks like i'm going to need to skip the meeting. we have a pet situation, headed to the vet's office now. i'll check back in later18:37
fungiclarkb: ^18:37
clarkbfungi: no worries, hope thinsg go well18:40
openstackgerritClark Boylan proposed openstack/project-config master: Retire os-loganalyze  https://review.opendev.org/75342818:43
*** hamalq has joined #opendev19:09
cgoncalveshey folks! it looks like there's a mirror issue with linaro us. at least two builds fail with curl timing out with "Operation too slow"19:13
*** priteau has quit IRC19:23
*** Tengu has quit IRC19:32
*** Tengu has joined #opendev19:33
openstackgerritMerged opendev/system-config master: launch-node: get sshfp entries from the host  https://review.opendev.org/74482119:39
openstackgerritMerged opendev/system-config master: Wait for ipv6 addrs when launching nodes  https://review.opendev.org/75004919:39
ianwcgoncalves: let me see if the mirror is unhappy20:02
ianw[Tue Sep 22 19:55:00 2020] afs: Lost contact with volume location server in cell openstack.org (code -1)20:03
ianw[Tue Sep 22 19:55:17 2020] afs: volume location server in cell openstack.org is back up (code 0)20:03
ianw[Tue Sep 22 18:15:27 2020] afs: file server in cell openstack.org is back up (code 105) (multi-homed address; other same-host interfaces may still be down)20:03
ianw[Tue Sep 22 18:15:55 2020] afs: Lost contact with file server in cell openstack.org (code -1) (all multi-homed ip addresses down for the server)20:03
ianwsorry, that's right about the same time20:04
ianwi think we sort of know things aren't great20:04
ianwthere's some hope that openafs 1.8 will give better performance20:04
cgoncalvesianw: thanks for checking! What can we do meanwhile? recheck? Any ETA for that openafs 1.8 update?20:07
ianwumm, no :)  it requires redoing the servers20:08
ianwi would say recheck, i guess the more that is cached the less networking affects things20:08
ianwwe also have some unresolved stability issues, but i haven't managed to catch kevinz yet20:09
cgoncalvesAlright, thanks20:10
*** tosky_ has joined #opendev21:09
*** tosky is now known as Guest3320021:10
*** tosky_ is now known as tosky21:10
*** slaweq has quit IRC21:18
mnaserinfra-root: is it possible that the docker mirror is currently broken?21:39
mnaserall loci jobs are failing https://review.opendev.org/#/c/753424/ and we use teh deb-docker mirror21:39
mnaserwith "apt cache update failed"21:39
mnasermore specifically http://mirror.bhs1.ovh.opendev.org/deb-docker/21:40
clarkbmnaser: it was updated recently to fix the per distro release packaging of all the various docker things21:41
clarkbpreviously we were just mirroring something old iirc21:41
clarkbI wonder if that broke things21:41
mnaseroh, that's quite possible, i don't think loci uses ensure-docker role yet21:41
clarkbmnaser: ah then that is likely the issue21:42
mnaseri think that setup-gate playbook could use some refactoring21:42
clarkbmnaser: basically everything got pushed down a level under distro specific dirs so that we can actually mirror it properly21:42
clarkbmnaser: looking at https://zuul.opendev.org/t/openstack/build/114d1775eca04df8a2829526beb2f69d/console#1/0/22/ubuntu-bionic I think you want it to be something like "deb http://mirror.iad.rax.opendev.org/deb-docker/$release $release stable" ?21:43
clarkbmnaser: see https://review.opendev.org/#/c/746272/1/zuul.d/docker-images/base.yaml21:44
mnaserclarkb: if i'm playing with that i'll just flip it to use configure-swap and ensure-docker and call it a day :) i think that's the path ill go towards21:44
clarkbmnaser: ok I think you may need to set the url as in the change link above but it should do the rest from there21:45
mnaseri wonder if it makes sense for us to add `docker_mirror_base_url` to the site.yml21:45
clarkbmnaser: I think we don't because we don't mirror for rpm distros?21:46
mnaserah, treu21:46
clarkband with podman == docker on rhel8 and friends I'm not sure if that is a generally safe thing to do21:47
mnaserwelp, that's unlucky21:47
ianwi'm getting Error Code:21:52
ianw    invalid_grant21:52
ianwtrying to log into storyboard21:52
fungiianw: i'm able to log out and in with no problem. when's the last time you were able to successfully log in?21:54
ianwumm, some time in the past :)  sorry i don't really know21:54
fungijust wondering if the issue is account-specific21:54
ianwhrm, well i closed the window and navigated directly to storyboard and now i can log in21:55
fungithat's an OAuthException subclass21:55
ianwpreviously i was following a link21:56
fungiwe don't explicitly raise the InvalidGrant exception in the sb api as far as i can see, we just define it (presumably so that we can pass a unique error string for it when the oauth lib raises it)21:57
ianwit is a link to a item i actually probably don't have permissions to see; let me see if i can log out and replicate it21:57
fungiwhat was it a link to?21:57
fungiheh, ignore my last question ;)21:57
fungiand yeah, i was about to guess it could have been for a private story which hasn't been made visible to you21:58
ianwit seems repeatable.  so if you are logged out, follow a link to a private story not made visible to you you get a 404 box, but then if you follow the login link you get to the InvalidGrant exception upon returning to storyboard from launchpad22:00
*** mwhahaha has quit IRC22:04
*** gmann has quit IRC22:04
*** mwhahaha has joined #opendev22:04
*** seongsoocho has quit IRC22:04
*** aannuusshhkkaa has quit IRC22:04
*** seongsoocho has joined #opendev22:05
*** gmann has joined #opendev22:05
*** gouthamr has quit IRC22:06
*** aannuusshhkkaa has joined #opendev22:06
clarkbI've approved the fedora-latest update change22:06
*** mnaser has quit IRC22:06
fungiianw: that does sound like enough of a bug worth filing22:07
fungithough probably not a common issue for folks since they would need to try to access a resource they're not authorized for while not logged in22:07
fungis/while/and while/22:08
clarkbI'll keep an eye out for any unexpected issues related to that22:08
*** mnaser has joined #opendev22:10
*** gouthamr has joined #opendev22:11
ianwfungi: done https://storyboard.openstack.org/#!/story/2008184 :)22:12
openstackgerritMerged opendev/base-jobs master: Switch fedora-latest to version 32  https://review.opendev.org/75274422:19
*** tosky has quit IRC22:48
*** tkajinam has joined #opendev22:57
*** mlavalle has quit IRC22:57
openstackgerritClark Boylan proposed opendev/system-config master: Update gitea project descriptions  https://review.opendev.org/75346423:16
clarkbcorvus: ianw ^ fyi followup from the meeting23:17
clarkboh it has a bug23:17
fungioh, i did catch up on the meeting log, but i didn't really have anything to add other than the docs timeouts (which are hopefully solved now)23:18
openstackgerritClark Boylan proposed opendev/system-config master: Update gitea project descriptions  https://review.opendev.org/75346423:18
clarkbnote that doesn't yet use the futures stuff23:19
clarkbI want to see it work serially (for simplicity) then add it into the thread pool23:19
openstackgerritMerged zuul/zuul-jobs master: Use Train repo to install openvswitch from multi-node-bridge in CentOS 8  https://review.opendev.org/75290823:44

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!