Tuesday, 2020-08-25

« Monday, 2020-08-24 Index Wednesday, 2020-08-26 »
*** armax has quit IRC00:24
*** artom has quit IRC00:26
*** Goneri has quit IRC00:27
*** armax has joined #openstack-infra00:50
*** tkajinam_ is now known as tkajinam01:04
*** rcernin has quit IRC01:14
*** rcernin has joined #openstack-infra01:15
*** elod has quit IRC01:21
*** elod has joined #openstack-infra01:23
*** armax has quit IRC01:37
*** gyee has quit IRC01:48
*** rfolco has quit IRC03:13
*** dave-mccowan has quit IRC03:16
*** hashar has joined #openstack-infra03:21
*** hashar has quit IRC03:47
*** hashar has joined #openstack-infra03:48
*** ijw has quit IRC04:07
*** ijw has joined #openstack-infra04:08
*** ijw has quit IRC04:10
*** ijw has joined #openstack-infra04:10
*** zxiiro has quit IRC04:16
*** ykarel has joined #openstack-infra04:19
*** rcernin has quit IRC04:32
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-infra04:35
*** hongbin has quit IRC04:39
*** raukadah is now known as chandankumar04:56
*** Lucas_Gray has joined #openstack-infra05:03
*** rcernin has joined #openstack-infra05:03
*** ysandeep|away is now known as ysandeep05:20
*** lmiccini has joined #openstack-infra05:27
*** hashar has quit IRC05:31
*** elod has quit IRC05:39
*** rcernin has quit IRC05:41
*** Lucas_Gray has quit IRC05:44
*** vishalmanchanda has joined #openstack-infra05:50
*** elod has joined #openstack-infra05:53
*** ricolin_ has joined #openstack-infra06:04
*** ricolin_ has quit IRC06:11
*** ralonsoh has joined #openstack-infra06:28
*** ijw has quit IRC06:31
*** slaweq has joined #openstack-infra06:36
*** xek has joined #openstack-infra06:44
*** ijw has joined #openstack-infra06:44
*** ijw has quit IRC06:49
*** apetrich has joined #openstack-infra06:56
*** rcernin has joined #openstack-infra06:56
*** xek has quit IRC07:02
*** rcernin has quit IRC07:03
*** rcernin has joined #openstack-infra07:05
*** andrewbonney has joined #openstack-infra07:05
*** rcernin has quit IRC07:14
*** ijw has joined #openstack-infra07:15
*** ijw has quit IRC07:20
*** iurygregory has joined #openstack-infra07:21
*** eolivare has joined #openstack-infra07:24
*** dtantsur|afk is now known as dtantsur07:26
noonedeadpunkHi everyone! Can I ask for a hold for https://review.opendev.org/#/c/747211/ of openstack-ansible-deploy-aio_distro_metal-centos-8 job? I was trying to reproduce in the sandbox but was not able to....07:32
noonedeadpunkoh, or wait tempest failed another way now....07:33
noonedeadpunkyeah, disregard for now07:34
*** tosky has joined #openstack-infra07:36
*** dklyle has quit IRC07:36
*** xek has joined #openstack-infra07:43
*** ijw has joined #openstack-infra07:47
*** ysandeep is now known as ysandeep|lunch07:48
*** ijw has quit IRC07:52
*** gfidente has joined #openstack-infra07:58
*** hashar has joined #openstack-infra07:59
*** piotrowskim has joined #openstack-infra07:59
*** xek has quit IRC08:04
*** lucasagomes has joined #openstack-infra08:13
*** Ajohn has joined #openstack-infra08:18
*** ijw has joined #openstack-infra08:18
*** ijw has quit IRC08:23
*** pkopec has joined #openstack-infra08:32
*** Ajohn has quit IRC08:40
*** derekh has joined #openstack-infra08:41
*** Ajohn has joined #openstack-infra08:45
*** ijw has joined #openstack-infra08:49
*** ijw has quit IRC08:54
*** dosaboy has quit IRC08:59
*** dosaboy has joined #openstack-infra08:59
*** ysandeep|lunch is now known as ysandeep09:02
*** yamamoto has quit IRC09:16
*** yamamoto has joined #openstack-infra09:18
*** ianychoi__ has joined #openstack-infra09:21
*** ijw has joined #openstack-infra09:21
*** ianychoi_ has quit IRC09:24
*** ijw has quit IRC09:26
*** verdurin has joined #openstack-infra09:29
*** yamamoto has quit IRC09:32
*** yamamoto has joined #openstack-infra09:45
*** yamamoto has quit IRC09:46
*** ijw has joined #openstack-infra09:53
*** yamamoto has joined #openstack-infra09:54
*** ijw has quit IRC09:57
*** hogepodge has quit IRC10:14
*** jamespage has quit IRC10:14
*** hogepodge has joined #openstack-infra10:15
*** johnsom has quit IRC10:16
*** jamespage has joined #openstack-infra10:16
*** johnsom has joined #openstack-infra10:17
sshnaidmDid anyone notice a slow disk write on OVH hosts? We have job timeouts as from what I see it takes a lot to install packages on the host (while download is pretty fast). For example: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_40e/747708/1/gate/tripleo-ci-centos-7-containers-multinode/40e0f65/logs/undercloud/home/zuul/install_packages.sh.log10:20
sshnaidmpackage install on disk took 10 mins, while on other host it's 2-3 minutes: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_da2/periodic/opendev.org/openstack/tripleo-quickstart-extras/master/tripleo-ci-centos-7-containers-multinode-rocky/da2eb1e/logs/undercloud/home/zuul/install_packages.sh.log10:20
sshnaidmalso OVH host ^10:20
sshnaidmclarkb, fungi any ideas..? ^^10:21
sshnaidmarxcruz|ruck, akahat|rover ^10:21
*** ijw has joined #openstack-infra10:24
*** ijw has quit IRC10:29
arxcruz|rucksshnaidm: there are so many things slowing down, than yeah, i haven't notice this in particular \10:32
*** howell has quit IRC10:49
*** knikolla has quit IRC10:49
*** csatari has quit IRC10:49
*** cjohnston has quit IRC10:50
*** rpioso has quit IRC10:50
*** dviroel has quit IRC10:51
*** lxkong has quit IRC10:51
*** yamamoto has quit IRC10:52
*** srwilkers has quit IRC10:52
*** yamamoto has joined #openstack-infra10:54
*** ijw has joined #openstack-infra10:56
*** rcernin has joined #openstack-infra10:57
*** srwilkers has joined #openstack-infra10:59
*** cjohnston has joined #openstack-infra10:59
*** rpioso has joined #openstack-infra10:59
*** howell has joined #openstack-infra10:59
*** dviroel has joined #openstack-infra11:00
*** ijw has quit IRC11:00
*** csatari has joined #openstack-infra11:01
*** knikolla has joined #openstack-infra11:01
*** lxkong has joined #openstack-infra11:02
*** hashar is now known as hasharLunch11:14
*** yamamoto has quit IRC11:24
*** yamamoto has joined #openstack-infra11:25
*** ijw has joined #openstack-infra11:27
*** trident has quit IRC11:31
*** ijw has quit IRC11:32
*** Lucas_Gray has joined #openstack-infra11:41
*** rfolco has joined #openstack-infra11:53
*** rlandy has joined #openstack-infra11:58
*** yamamoto has quit IRC11:58
*** ijw has joined #openstack-infra11:59
*** ijw has quit IRC12:03
*** ysandeep is now known as ysandeep|session12:06
*** xek has joined #openstack-infra12:10
*** Goneri has joined #openstack-infra12:11
*** hasharLunch is now known as hashar12:15
*** ykarel_ has joined #openstack-infra12:16
*** dave-mccowan has joined #openstack-infra12:18
*** ykarel has quit IRC12:18
*** ykarel_ is now known as ykarel12:19
*** ijw has joined #openstack-infra12:30
*** ijw has quit IRC12:35
*** yamamoto has joined #openstack-infra12:37
*** Gyuseok_Jung has joined #openstack-infra12:41
rpittauhello everyone! Can I please get a review on https://review.opendev.org/746553 ? missing just a +A :)12:42
*** artom has joined #openstack-infra12:43
*** Guest34946 has joined #openstack-infra12:47
*** yamamoto has quit IRC12:48
fricklerrpittau: done12:48
rpittaufrickler: thanks!12:48
*** Guest34946 is now known as redrobot12:50
*** lpetrut has joined #openstack-infra12:57
openstackgerritMerged openstack/project-config master: Do not update upper constraints for intermediate branches  https://review.opendev.org/74655312:58
*** bnemec has joined #openstack-infra13:00
*** vdrok has quit IRC13:02
*** ijw has joined #openstack-infra13:02
*** trident has joined #openstack-infra13:02
*** vdrok has joined #openstack-infra13:02
*** masayukig has quit IRC13:05
*** ijw has quit IRC13:07
*** ysandeep|session is now known as ysandeep13:16
*** masayukig has joined #openstack-infra13:16
*** KeithMnemonic has joined #openstack-infra13:22
*** ykarel is now known as ykarel|away13:28
*** ykarel|away has quit IRC13:45
*** d34dh0r53 has joined #openstack-infra13:49
*** sreejithp has joined #openstack-infra13:50
*** weshay is now known as weshay|interview13:54
*** gyee has joined #openstack-infra14:03
*** ijw has joined #openstack-infra14:06
*** ijw has joined #openstack-infra14:07
*** Lucas_Gray has quit IRC14:13
*** dklyle has joined #openstack-infra14:19
*** pkopec has quit IRC14:29
*** bauzas has quit IRC14:33
*** ijw_ has joined #openstack-infra14:34
*** ysandeep is now known as ysandeep|away14:37
*** zxiiro has joined #openstack-infra14:38
*** ijw has quit IRC14:38
*** rcernin has quit IRC14:45
*** bauzas has joined #openstack-infra14:47
*** bauzas has quit IRC14:52
*** bauzas has joined #openstack-infra14:54
*** pkopec has joined #openstack-infra15:10
*** Ajohn has quit IRC15:11
*** xek has quit IRC15:14
*** armax has joined #openstack-infra15:16
*** chandankumar is now known as raukadah15:23
*** eolivare has quit IRC15:28
*** lpetrut has quit IRC15:32
*** owalsh has quit IRC15:36
*** jtomasek has joined #openstack-infra15:41
*** owalsh has joined #openstack-infra15:45
*** owalsh has quit IRC15:45
*** sshnaidm is now known as sshnaidm|afk15:59
*** jtomasek has quit IRC16:03
*** weshay|interview is now known as weshay16:04
*** lucasagomes has quit IRC16:06
*** lmiccini has quit IRC16:08
*** dtantsur is now known as dtantsur|afk16:19
*** owalsh has joined #openstack-infra16:23
*** gfidente has quit IRC16:23
*** bnemec has quit IRC16:30
openstackgerritLajos Katona proposed openstack/project-config master: Import netowrking-l2gw & networking-l2gw-tempest-plugin to x/  https://review.opendev.org/74411016:32
*** bnemec has joined #openstack-infra16:35
*** hashar is now known as hasharDinner17:07
dansmithI wish my logstash fu was better, but.. I'm unable to even find results that we've seen in the last 24 hours17:09
dansmithis this not a valid query?17:09
dansmithmessage:"15 is not 0"17:09
dansmithshould match "2020-08-25 16:29:33.784264 | ubuntu-bionic |     RuntimeError: The exit code 15 is not 0" right?17:09
clarkbdansmith: yes that should match. Chances are we're still completely broken on the neutron logs being giant17:11
clarkbmelwitt was working with them on that most recently17:11
dansmithokay I see things in there from today from swift jobs17:11
dansmithso I thought this'd be in there17:11
clarkbya http://status.openstack.org/elastic-recheck/ says we're sad17:12
clarkbdansmith: ah so funny story. swift logs don't parse properly in all cases becaues they sometimes lack timestamps iirc17:12
dansmithit's unfortunate, because we think this *just* started happening, but it's really hard to tell without logstash :/17:12
dansmithah, behind by 200 hours, I see.. I should check that next time17:12
clarkbdansmith: when that happens we index them as if they happened now (but in reality they likely happened a week ago)17:12
dansmithclarkb: oh jeez :/17:12
clarkbwhen we lack timestamps we do our best but it can be misleading17:12
clarkband ya I agree, it would be good to get this stuff back into shape.17:13
clarkbI suppose we can just stop indexing neutron17:13
clarkbmelwitt ^ if you are around do you think that may be prudent given progress there?17:13
dansmith++17:13
clarkbI'll write that change now17:13
dansmithclarkb: so the thing I'm trying to chase,17:16
dansmithglance has functional tests that actually spawn api workers like normal,17:16
dansmithand it appears like those might be getting killed with SIGTERM17:16
dansmithI was just thinking maybe there's something else in the job that could be doing that, like maybe OOM (should be KILL though I think)17:17
dansmithlike all of a sudden a big rash of those and we can't repro locally17:17
dansmithanything magical you can think of? I notice there's no dstat logs for those, so i can't see if something is going off the rails17:18
clarkbdansmith: remote:   https://review.opendev.org/747988 Exclude neutron q-svc logs from indexing17:18
clarkbnothing comes to mind for sigterm17:18
*** derekh has quit IRC17:18
dansmithokay17:19
clarkbmy guess is it would be the control process if it is the children that are being terminated17:19
clarkbperhaps because there is an http request limit to rotate them out or similar17:19
clarkbapache does that with its own workers (maybe it will extend that to wsgi?)17:20
clarkbs/apache does that/apache can do that/17:20
dansmiththis is uuber standalone, so no apache17:21
dansmithI made a change a couple weeks ago that will try to kill workers that don't exit, and before they would just hang on waitpid forever, yielding the TIMED_OUT i was searching for before17:22
dansmithso it's possible that it's us doing that and we're not getting captured log output showing that it's happening17:22
dansmithbut I dunno how to explain the lack of logs17:22
dansmithanyway, just mumbling since I can't look at logstash info  for when it started :)17:23
*** ralonsoh has quit IRC17:27
EmilienMclarkb: we're trying to land 746993 asap, would it be possible to enqueue it into the gate please?17:28
clarkbEmilienM: all three of the depends on have merged. And it looks like the only change since the previous gate enqueue was a rebase?17:29
clarkbshould be safe enough give me a minute to load up my ssh keys and do that17:29
EmilienMthanks17:29
*** derekh has joined #openstack-infra17:33
clarkbEmilienM: done17:34
EmilienMclarkb: thanks a lot.17:34
weshaythanks all17:35
*** andrewbonney has quit IRC17:44
*** bnemec has quit IRC17:50
*** derekh has quit IRC17:52
*** bnemec has joined #openstack-infra17:53
*** dwalt has joined #openstack-infra18:26
*** calbers has quit IRC18:32
*** calbers has joined #openstack-infra18:32
dwalthey all! We're seeing a strange issue on some jobs in #airshipit where the "run" playbook does not execute. For example, this job definition [0] has a run playbook, but it did not execute in a recent run [1]. Is there anything that stands out in the job that may be causing this? We would really appreciate your advice if anyone has a second to take a look. [0]18:34
dwalthttps://opendev.org/airship/treasuremap/src/branch/v2/zuul.d/jobs.yaml#L61 [1] https://zuul.opendev.org/t/openstack/build/48f419e130bb45dcb35625c08db8ba61/console18:34
clarkbdwalt: if pre-run fails then run is skipped. It could be that? I'm looking at the logs now18:43
clarkbdwalt: it is actually because the run playbook ran and timed out. https://zuul.opendev.org/t/openstack/build/48f419e130bb45dcb35625c08db8ba61/log/job-output.txt#471718:44
clarkbdwalt: when the playbook times out ansible is forcefully killed and the log json is not written properly18:44
dwaltclarkb: oh, that makes sense. Next time, I will check that text output. Thanks for your help!18:45
clarkbupdating zuul to write a small "this playbook timed out" task in there may be useful18:47
*** howell has quit IRC18:49
*** srwilkers has quit IRC18:49
*** dviroel has quit IRC18:50
*** srwilkers has joined #openstack-infra18:52
*** dviroel has joined #openstack-infra18:53
*** howell has joined #openstack-infra18:53
*** xek has joined #openstack-infra18:59
spotzHas anyone seen an --include:: line break a doc build before on verification?19:06
clarkbspotz: have a link?19:10
spotzclarkb: https://review.opendev.org/#/c/747809/5..6 If I remove the include it passes19:10
clarkbspotz: https://zuul.opendev.org/t/openstack/build/798722ba70414699abf0d657f1494d20/log/job-output.txt#788-792 the issue is the included file has warnings19:13
spotzclarkb: Weird cause the included file itself passes19:15
clarkbif it isn't included anywhere else or added to a toctree it may be passed over?19:15
spotzit's in a toctree - https://review.opendev.org/#/c/747809/1..6/deploy-guide/source/environments/index.rst19:16
clarkbnot sure then, maybe its specific to the include behavior?19:17
clarkbinclude + the content == warning but just toctree + content doesn't?19:17
spotzI took the include line out of the file and it passsed and if I put it back in it failed, that's how I narrowed it down19:18
spotzSo yeah19:18
spotzNot that I ever have normal issues19:19
spotzWARNING: Unexpected section title?19:20
clarkbya I think it may be that you're including under a lower level section title then the higher level sections in the included data are balanced wrong?19:20
clarkbor maybe the section title underline characters need to be lined up between the files19:20
clarkbsince they aren't defined, its context sensitive I think it could be eithre thing?19:21
spotzI'm wondering, the page I based on isn't called anywhere else. Let me look at another page we include19:21
spotzThat one's an orphan19:23
*** ijw_ has quit IRC19:27
spotzOk trying having the included file with more sub subheading19:27
*** ijw has joined #openstack-infra19:28
*** ijw has quit IRC19:32
*** ijw has joined #openstack-infra19:34
*** pkopec has quit IRC19:39
*** ijw has quit IRC19:39
*** kberger_ has joined #openstack-infra19:39
*** kberger_ has quit IRC19:40
*** kberger_ has joined #openstack-infra19:40
*** KeithMnemonic has quit IRC19:41
*** piotrowskim has quit IRC19:45
*** rlandy is now known as rlandy|brb19:47
*** hasharDinner has quit IRC19:50
slaweqdansmith: clarkb: hi, regarding this huge neutron server log, we recently merged https://review.opendev.org/#/c/746714/ which makes them a bit smaller (about 50% IIRC)19:55
dansmithstill, 40M seems like a lot for each run19:55
slaweqdansmith: yes, I will check what else we can remove from it, that was first thing which we did19:56
slaweqbut I think that disabling indexing those logs temporary is good workaround19:56
dansmithagree, and if that log file can't be more reasonable in size, perhaps not indexing it permanently is the best thing all around19:59
slaweqdansmith: yes20:01
slaweqbut now I'm going afk as it's late for me, I will look into that tomorrow :)20:01
clarkbok maybe ianw can review 747988 and I can reetart logstash servicesto drop the big logs20:02
ianwclarkb: ^ is the linter failure there valid?20:03
clarkbianw: its the file permissions thing that ansible recently did20:06
*** ijw has joined #openstack-infra20:06
clarkbwe should probably fix those... I can take a look after lunch20:06
*** slaweq has quit IRC20:06
*** rlandy|brb is now known as rlandy20:06
ianwi've reported the openafs oops upstream with https://lists.openafs.org/pipermail/openafs-info/2020-August/042959.html20:08
ianwi'm going to try building 1.8.6 (we're on 1.8.5)20:08
*** vishalmanchanda has quit IRC20:10
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Bump openafs build to new 1.8.6 release  https://review.opendev.org/74802220:10
*** ijw has quit IRC20:11
*** slaweq has joined #openstack-infra20:18
*** Lucas_Gray has joined #openstack-infra20:21
clarkbianw: I'm half tempted to ignore that lint rule20:25
clarkbits so many things but I should see how painful it is to sort out permissions for each of these20:25
ianwi think we've ignored it in other places, so there's precedent20:26
clarkbah in that case let me write that change20:26
*** ijw has joined #openstack-infra20:37
openstackgerritDouglas Mendizábal proposed openstack/project-config master: Update Ansible roles for ATOS and Thales HSMs  https://review.opendev.org/74802820:41
*** ijw has quit IRC20:42
*** ijw has joined #openstack-infra20:43
*** dwalt has quit IRC21:15
clarkbdansmith: melwitt: I've restarted all the data pipeline processor processes now and the q-svc exclusion should take effect on jobs that started after the chnage for that merged21:17
clarkbI think we'll be in a happier state moving forward now21:17
*** yamamoto has joined #openstack-infra21:30
melwittgreat thanks clarkb++21:34
*** rfolco has quit IRC21:57
*** rfolco has joined #openstack-infra21:57
*** yamamoto has quit IRC22:00
*** rcernin has joined #openstack-infra22:00
*** rcernin has quit IRC22:01
*** rcernin has joined #openstack-infra22:02
*** rfolco has quit IRC22:02
*** rfolco has joined #openstack-infra22:03
*** rfolco has quit IRC22:04
dansmithwoot, thanks clarkb !22:06
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Bump openafs build to new 1.8.6 release  https://review.opendev.org/74802222:06
openstackgerritIan Wienand proposed openstack/openstack-zuul-jobs master: Skip ansible-lint 106/208  https://review.opendev.org/74803822:06
*** rfolco has joined #openstack-infra22:06
*** rfolco has quit IRC22:08
*** rfolco has joined #openstack-infra22:12
*** sreejithp has quit IRC22:14
*** bnemec has quit IRC22:14
openstackgerritIan Wienand proposed openstack/project-config master: Add ansible-collections for system-config -devel job  https://review.opendev.org/74759622:16
*** rfolco has quit IRC22:16
*** rlandy has quit IRC22:18
*** tosky has quit IRC23:04
*** rfolco has joined #openstack-infra23:19
*** yamamoto has joined #openstack-infra23:59
« Monday, 2020-08-24 Index Wednesday, 2020-08-26 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!