Friday, 2021-02-26

sreejithphello, i am seeing an issue with zuul jobs vms for airship. its not have 100gb disk space. seems like partition expansion is not work. how can i check and confirm this?00:07
clarkbsreejithp: can you link to an example? we have noticed that happening in kna1 and added some debugging info around growroot to try and debug it further00:07
clarkbif you link to an example we can check that debugging info00:08
sreejithpyes the slave node was on kna1. this is the job link https://zuul.opendev.org/t/openstack/build/8f677202146b49fb95c0e760e150a304/log/job-output.txt00:08
sreejithpi think this is the hostname ubuntu-bionic-expanded-airship-kna1-002320407200:09
clarkbsreejithp: https://zuul.opendev.org/t/openstack/build/8f677202146b49fb95c0e760e150a304/log/job-output.txt#314-385 is the growroot logs for that job00:09
clarkbwell specifically for the instance the job ran on00:09
clarkb'resize2fs: Bad message While checking for on-line resizing support' is why growroot failed and the disk remained small00:10
clarkbfungi: ^ fyi we finally caught one00:10
*** tosky has quit IRC00:12
clarkbunfortunately that doesn't give us a ton of info. We likely need to escalate that with the citynetwork operators and have them cross check disk/filesystem stability for instance 755aefe9-6ff6-4a75-af60-bbc176f77c1d running on hypervisor 2e57327cd82629e0855870473a1f85e93148eeadd4288f239a45884000:12
sreejithpsure, thanks clarkb00:14
clarkbsreejithp: is that something ya'll want to do from the airship side? I think historically you have done that as there is an ericsson intermediary?00:15
clarkbfungi: who was it that helped last time?00:15
fungisearching for that string turns this up from a decade ago: "https://sourceforge.net/p/e2fsprogs/bugs/293/"00:15
fungiclarkb: noonedeadpunk had contacts00:15
clarkbfungi: I don't see a bad message string in your link00:16
fungiinteresting00:16
fungiyeah, i guess ddg scraped the bottom of the barrel and found "Device or resource busy While checking for on-line resizing support"00:17
clarkbsreejithp: when we brought this up with some nova maintainers previously the one thing they thought of was maybe the disk is encrypted and we're hitting bugs in that00:17
clarkbbut it isn't clear to us if that is the case or not (we aren't requested encrypted disks)00:17
funginoonedeadpunk said they're only just investigating/entertaining encrypted backing devices there, so definitely not in place yet00:18
sreejithpi am not sure about the configuration/setup for the node. i will reach out to people from my end00:18
clarkbfungi: it wasn't clear to me if that was vexxhost or citynetwork though00:19
fungipretty sure he understood we were asking aboutcitynetwork00:21
fungihe's apparently involved in both though, yeah00:21
clarkba manjaro forum post, https://forum.manjaro.org/t/failed-to-open-directory-bad-message/6168/5, indicates that could be the result of a corupt fs00:24
fungii guess if the block device is misbehaving on boot that could happen00:24
clarkbI doubt it is our image's fs though since we're booting many nodes from the same image without trouble00:24
clarkbwhich makes me suspect something hypervisor side that is opaque to us00:25
fungiyeah, doubt it's image corruption or it would happen more, and would be likely to go away when we upload new images00:25
clarkbit would also affect other providers more00:25
clarkbwhereas we've largely seen this happen in kna100:25
fungiunless it's corruption of the image copies in the nova image cache00:26
clarkboh ya that is possible too I think00:26
fungiwe've only seen it happen in kna1 afaik00:26
clarkbfungi: we've seen bad message happen on some osa jobs elsewhere but the growroot disk size thing seems to be kna1 specific00:26
fungiright00:27
*** ianychoi__ is now known as ianychoi00:34
*** hamalq has quit IRC01:10
*** sorrison has quit IRC01:15
*** sorrison has joined #openstack-infra01:21
*** zzzeek has quit IRC01:26
*** zzzeek has joined #openstack-infra01:27
*** jhesketh_ is now known as jhesketh01:49
*** ramishra has joined #openstack-infra01:58
*** rcernin_ has joined #openstack-infra02:08
*** rcernin has quit IRC02:08
*** EmilienM has joined #openstack-infra02:46
*** sreejithp has quit IRC02:59
*** ricolin has joined #openstack-infra03:03
*** rcernin_ has quit IRC03:09
*** ricolin has quit IRC03:10
*** ricolin has joined #openstack-infra03:15
*** rcernin_ has joined #openstack-infra03:22
*** eharney has quit IRC03:28
*** rcernin_ has quit IRC03:32
*** rcernin has joined #openstack-infra03:32
*** zbr has joined #openstack-infra03:50
*** dmsimard has quit IRC04:08
*** dmsimard has joined #openstack-infra04:09
*** psachin has joined #openstack-infra04:28
*** ociuhandu has joined #openstack-infra04:30
*** ociuhandu has quit IRC04:35
*** rcernin has quit IRC06:58
*** rcernin_ has joined #openstack-infra06:58
*** lpetrut has joined #openstack-infra07:29
*** dklyle has quit IRC07:32
*** rcernin_ has quit IRC07:41
*** ysirndjuro has quit IRC07:42
*** icey_ is now known as icey07:43
*** elod has quit IRC07:45
*** elod has joined #openstack-infra07:49
*** slaweq_ has joined #openstack-infra07:50
*** dchen has quit IRC08:03
*** amoralej is now known as amoralej|off08:09
*** slaweq_ has quit IRC08:22
*** andrewbonney has joined #openstack-infra08:24
*** yoctozepto5 has joined #openstack-infra08:26
*** yoctozepto has quit IRC08:28
*** yoctozepto5 is now known as yoctozepto08:28
*** yoctozepto6 has joined #openstack-infra08:32
*** yoctozepto has quit IRC08:33
*** yoctozepto6 is now known as yoctozepto08:33
*** gfidente has joined #openstack-infra09:08
*** gagehugo has quit IRC09:37
*** coreycb has quit IRC09:37
*** gagehugo has joined #openstack-infra09:37
*** coreycb has joined #openstack-infra09:38
*** ricolin_ has joined #openstack-infra09:57
*** mugsie_ is now known as mugsie10:16
*** yoctozepto8 has joined #openstack-infra10:47
*** yoctozepto has quit IRC10:47
*** yoctozepto8 is now known as yoctozepto10:47
*** yoctozepto1 has joined #openstack-infra10:54
*** yoctozepto has quit IRC10:55
*** yoctozepto1 is now known as yoctozepto10:55
*** iurygregory has quit IRC11:13
*** gfidente has quit IRC11:15
*** tosky has joined #openstack-infra11:25
*** gfidente has joined #openstack-infra11:30
*** yoctozepto2 has joined #openstack-infra11:32
*** yoctozepto has quit IRC11:33
*** yoctozepto2 is now known as yoctozepto11:33
*** ricolin_ has quit IRC12:10
*** ociuhandu has joined #openstack-infra12:17
*** yoctozepto4 has joined #openstack-infra12:45
*** yoctozepto has quit IRC12:46
*** yoctozepto4 is now known as yoctozepto12:46
*** slaweq_ has joined #openstack-infra12:49
*** psachin has quit IRC12:58
*** zbr9 has joined #openstack-infra13:20
*** zigo has joined #openstack-infra13:21
*** zbr has quit IRC13:22
*** zbr9 is now known as zbr13:22
*** slaweq_ has quit IRC13:31
*** stephenfin has joined #openstack-infra13:38
*** zbr0 has joined #openstack-infra13:43
*** zbr has quit IRC13:46
*** zbr0 is now known as zbr13:46
*** iurygregory has joined #openstack-infra13:49
*** ociuhandu has quit IRC13:53
*** ociuhandu has joined #openstack-infra13:54
*** ociuhandu has quit IRC13:59
*** zbr3 has joined #openstack-infra14:03
*** ociuhandu has joined #openstack-infra14:04
*** zbr has quit IRC14:05
*** zbr3 is now known as zbr14:05
*** sreejithp has joined #openstack-infra14:06
*** zbr7 has joined #openstack-infra14:08
*** zbr has quit IRC14:10
*** zbr7 is now known as zbr14:10
*** zbr4 has joined #openstack-infra14:33
*** zbr has quit IRC14:35
*** zbr4 is now known as zbr14:35
*** ociuhandu has quit IRC14:49
*** zbr4 has joined #openstack-infra14:52
*** zbr has quit IRC14:53
*** zbr4 is now known as zbr14:53
*** zbr5 has joined #openstack-infra15:12
*** zbr has quit IRC15:14
*** zbr5 is now known as zbr15:14
*** lpetrut has quit IRC15:15
*** tosky has quit IRC15:19
*** tosky has joined #openstack-infra15:20
*** zbr8 has joined #openstack-infra15:30
*** zbr has quit IRC15:33
*** zbr8 is now known as zbr15:33
*** dklyle has joined #openstack-infra15:51
*** lpetrut has joined #openstack-infra16:05
*** ociuhandu has joined #openstack-infra16:07
*** zbr2 has joined #openstack-infra16:11
*** zbr has quit IRC16:13
*** zbr2 is now known as zbr16:13
*** zbr6 has joined #openstack-infra16:15
*** zbr has quit IRC16:17
*** zbr6 is now known as zbr16:17
*** calbers has quit IRC16:32
*** icey has quit IRC16:33
*** icey has joined #openstack-infra16:34
*** calbers has joined #openstack-infra16:35
*** zbr has quit IRC16:38
*** gyee has joined #openstack-infra16:40
*** harlowja has joined #openstack-infra16:41
*** hemna has quit IRC16:47
*** hemna has joined #openstack-infra16:49
*** harlowja has quit IRC16:59
*** slaweq_ has joined #openstack-infra17:08
*** ociuhandu_ has joined #openstack-infra17:09
*** ociuhandu has quit IRC17:13
*** ociuhandu_ has quit IRC17:14
*** hamalq has joined #openstack-infra17:16
*** ociuhandu has joined #openstack-infra17:23
*** gfidente has quit IRC17:23
*** ociuhandu has quit IRC17:28
*** __ministry1 has joined #openstack-infra17:34
*** arne_wiebalck has quit IRC17:35
*** arne_wiebalck has joined #openstack-infra17:36
openstackgerritJeremy Stanley proposed openstack/project-config master: Move git-review to opendev tenant  https://review.opendev.org/c/openstack/project-config/+/77777417:36
*** lpetrut has quit IRC17:40
*** slaweq_ has quit IRC17:54
*** andrewbonney has quit IRC18:15
*** __ministry1 has quit IRC18:29
*** slaweq_ has joined #openstack-infra18:33
*** slaweq_ has quit IRC18:44
openstackgerritMerged openstack/project-config master: Move git-review to opendev tenant  https://review.opendev.org/c/openstack/project-config/+/77777418:50
*** sreejithp has quit IRC19:16
*** slaweq_ has joined #openstack-infra19:44
*** tjgresha has joined #openstack-infra19:50
*** slaweq_ has quit IRC20:22
*** ociuhandu has joined #openstack-infra20:58
*** ociuhandu has quit IRC21:02
tjgreshaany special permissions needed for a 3rdparty ci account to be able to comment on project - say Nova or Neutron?21:33
tjgreshagerrit permissions that is21:37
clarkbtjgresha: yes for nova your third party ci account must be in the nova-ci group21:45
clarkbtjgresha: for neutron it is the neutron-ci group and so on21:45
clarkbthose groups are managed by the projects themselves21:45
tjgreshameaning i should hit up corresponding IRC channels for access?21:46
clarkbyes, it will be up to the respective projects to decide if they want you to vote +/-1 or not (all third party ci can leave comments iwthout vote values)21:47
tjgreshaok, Thanks clarkb21:48
*** tjgresha has quit IRC22:17
*** sreejithp has joined #openstack-infra22:21
*** hamalq has quit IRC23:40
*** zzzeek has quit IRC23:41
*** zzzeek has joined #openstack-infra23:43

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!