sreejithp | hello, i am seeing an issue with zuul jobs vms for airship. its not have 100gb disk space. seems like partition expansion is not work. how can i check and confirm this? | 00:07 |
---|---|---|
clarkb | sreejithp: can you link to an example? we have noticed that happening in kna1 and added some debugging info around growroot to try and debug it further | 00:07 |
clarkb | if you link to an example we can check that debugging info | 00:08 |
sreejithp | yes the slave node was on kna1. this is the job link https://zuul.opendev.org/t/openstack/build/8f677202146b49fb95c0e760e150a304/log/job-output.txt | 00:08 |
sreejithp | i think this is the hostname ubuntu-bionic-expanded-airship-kna1-0023204072 | 00:09 |
clarkb | sreejithp: https://zuul.opendev.org/t/openstack/build/8f677202146b49fb95c0e760e150a304/log/job-output.txt#314-385 is the growroot logs for that job | 00:09 |
clarkb | well specifically for the instance the job ran on | 00:09 |
clarkb | 'resize2fs: Bad message While checking for on-line resizing support' is why growroot failed and the disk remained small | 00:10 |
clarkb | fungi: ^ fyi we finally caught one | 00:10 |
*** tosky has quit IRC | 00:12 | |
clarkb | unfortunately that doesn't give us a ton of info. We likely need to escalate that with the citynetwork operators and have them cross check disk/filesystem stability for instance 755aefe9-6ff6-4a75-af60-bbc176f77c1d running on hypervisor 2e57327cd82629e0855870473a1f85e93148eeadd4288f239a458840 | 00:12 |
sreejithp | sure, thanks clarkb | 00:14 |
clarkb | sreejithp: is that something ya'll want to do from the airship side? I think historically you have done that as there is an ericsson intermediary? | 00:15 |
clarkb | fungi: who was it that helped last time? | 00:15 |
fungi | searching for that string turns this up from a decade ago: "https://sourceforge.net/p/e2fsprogs/bugs/293/" | 00:15 |
fungi | clarkb: noonedeadpunk had contacts | 00:15 |
clarkb | fungi: I don't see a bad message string in your link | 00:16 |
fungi | interesting | 00:16 |
fungi | yeah, i guess ddg scraped the bottom of the barrel and found "Device or resource busy While checking for on-line resizing support" | 00:17 |
clarkb | sreejithp: when we brought this up with some nova maintainers previously the one thing they thought of was maybe the disk is encrypted and we're hitting bugs in that | 00:17 |
clarkb | but it isn't clear to us if that is the case or not (we aren't requested encrypted disks) | 00:17 |
fungi | noonedeadpunk said they're only just investigating/entertaining encrypted backing devices there, so definitely not in place yet | 00:18 |
sreejithp | i am not sure about the configuration/setup for the node. i will reach out to people from my end | 00:18 |
clarkb | fungi: it wasn't clear to me if that was vexxhost or citynetwork though | 00:19 |
fungi | pretty sure he understood we were asking aboutcitynetwork | 00:21 |
fungi | he's apparently involved in both though, yeah | 00:21 |
clarkb | a manjaro forum post, https://forum.manjaro.org/t/failed-to-open-directory-bad-message/6168/5, indicates that could be the result of a corupt fs | 00:24 |
fungi | i guess if the block device is misbehaving on boot that could happen | 00:24 |
clarkb | I doubt it is our image's fs though since we're booting many nodes from the same image without trouble | 00:24 |
clarkb | which makes me suspect something hypervisor side that is opaque to us | 00:25 |
fungi | yeah, doubt it's image corruption or it would happen more, and would be likely to go away when we upload new images | 00:25 |
clarkb | it would also affect other providers more | 00:25 |
clarkb | whereas we've largely seen this happen in kna1 | 00:25 |
fungi | unless it's corruption of the image copies in the nova image cache | 00:26 |
clarkb | oh ya that is possible too I think | 00:26 |
fungi | we've only seen it happen in kna1 afaik | 00:26 |
clarkb | fungi: we've seen bad message happen on some osa jobs elsewhere but the growroot disk size thing seems to be kna1 specific | 00:26 |
fungi | right | 00:27 |
*** ianychoi__ is now known as ianychoi | 00:34 | |
*** hamalq has quit IRC | 01:10 | |
*** sorrison has quit IRC | 01:15 | |
*** sorrison has joined #openstack-infra | 01:21 | |
*** zzzeek has quit IRC | 01:26 | |
*** zzzeek has joined #openstack-infra | 01:27 | |
*** jhesketh_ is now known as jhesketh | 01:49 | |
*** ramishra has joined #openstack-infra | 01:58 | |
*** rcernin_ has joined #openstack-infra | 02:08 | |
*** rcernin has quit IRC | 02:08 | |
*** EmilienM has joined #openstack-infra | 02:46 | |
*** sreejithp has quit IRC | 02:59 | |
*** ricolin has joined #openstack-infra | 03:03 | |
*** rcernin_ has quit IRC | 03:09 | |
*** ricolin has quit IRC | 03:10 | |
*** ricolin has joined #openstack-infra | 03:15 | |
*** rcernin_ has joined #openstack-infra | 03:22 | |
*** eharney has quit IRC | 03:28 | |
*** rcernin_ has quit IRC | 03:32 | |
*** rcernin has joined #openstack-infra | 03:32 | |
*** zbr has joined #openstack-infra | 03:50 | |
*** dmsimard has quit IRC | 04:08 | |
*** dmsimard has joined #openstack-infra | 04:09 | |
*** psachin has joined #openstack-infra | 04:28 | |
*** ociuhandu has joined #openstack-infra | 04:30 | |
*** ociuhandu has quit IRC | 04:35 | |
*** rcernin has quit IRC | 06:58 | |
*** rcernin_ has joined #openstack-infra | 06:58 | |
*** lpetrut has joined #openstack-infra | 07:29 | |
*** dklyle has quit IRC | 07:32 | |
*** rcernin_ has quit IRC | 07:41 | |
*** ysirndjuro has quit IRC | 07:42 | |
*** icey_ is now known as icey | 07:43 | |
*** elod has quit IRC | 07:45 | |
*** elod has joined #openstack-infra | 07:49 | |
*** slaweq_ has joined #openstack-infra | 07:50 | |
*** dchen has quit IRC | 08:03 | |
*** amoralej is now known as amoralej|off | 08:09 | |
*** slaweq_ has quit IRC | 08:22 | |
*** andrewbonney has joined #openstack-infra | 08:24 | |
*** yoctozepto5 has joined #openstack-infra | 08:26 | |
*** yoctozepto has quit IRC | 08:28 | |
*** yoctozepto5 is now known as yoctozepto | 08:28 | |
*** yoctozepto6 has joined #openstack-infra | 08:32 | |
*** yoctozepto has quit IRC | 08:33 | |
*** yoctozepto6 is now known as yoctozepto | 08:33 | |
*** gfidente has joined #openstack-infra | 09:08 | |
*** gagehugo has quit IRC | 09:37 | |
*** coreycb has quit IRC | 09:37 | |
*** gagehugo has joined #openstack-infra | 09:37 | |
*** coreycb has joined #openstack-infra | 09:38 | |
*** ricolin_ has joined #openstack-infra | 09:57 | |
*** mugsie_ is now known as mugsie | 10:16 | |
*** yoctozepto8 has joined #openstack-infra | 10:47 | |
*** yoctozepto has quit IRC | 10:47 | |
*** yoctozepto8 is now known as yoctozepto | 10:47 | |
*** yoctozepto1 has joined #openstack-infra | 10:54 | |
*** yoctozepto has quit IRC | 10:55 | |
*** yoctozepto1 is now known as yoctozepto | 10:55 | |
*** iurygregory has quit IRC | 11:13 | |
*** gfidente has quit IRC | 11:15 | |
*** tosky has joined #openstack-infra | 11:25 | |
*** gfidente has joined #openstack-infra | 11:30 | |
*** yoctozepto2 has joined #openstack-infra | 11:32 | |
*** yoctozepto has quit IRC | 11:33 | |
*** yoctozepto2 is now known as yoctozepto | 11:33 | |
*** ricolin_ has quit IRC | 12:10 | |
*** ociuhandu has joined #openstack-infra | 12:17 | |
*** yoctozepto4 has joined #openstack-infra | 12:45 | |
*** yoctozepto has quit IRC | 12:46 | |
*** yoctozepto4 is now known as yoctozepto | 12:46 | |
*** slaweq_ has joined #openstack-infra | 12:49 | |
*** psachin has quit IRC | 12:58 | |
*** zbr9 has joined #openstack-infra | 13:20 | |
*** zigo has joined #openstack-infra | 13:21 | |
*** zbr has quit IRC | 13:22 | |
*** zbr9 is now known as zbr | 13:22 | |
*** slaweq_ has quit IRC | 13:31 | |
*** stephenfin has joined #openstack-infra | 13:38 | |
*** zbr0 has joined #openstack-infra | 13:43 | |
*** zbr has quit IRC | 13:46 | |
*** zbr0 is now known as zbr | 13:46 | |
*** iurygregory has joined #openstack-infra | 13:49 | |
*** ociuhandu has quit IRC | 13:53 | |
*** ociuhandu has joined #openstack-infra | 13:54 | |
*** ociuhandu has quit IRC | 13:59 | |
*** zbr3 has joined #openstack-infra | 14:03 | |
*** ociuhandu has joined #openstack-infra | 14:04 | |
*** zbr has quit IRC | 14:05 | |
*** zbr3 is now known as zbr | 14:05 | |
*** sreejithp has joined #openstack-infra | 14:06 | |
*** zbr7 has joined #openstack-infra | 14:08 | |
*** zbr has quit IRC | 14:10 | |
*** zbr7 is now known as zbr | 14:10 | |
*** zbr4 has joined #openstack-infra | 14:33 | |
*** zbr has quit IRC | 14:35 | |
*** zbr4 is now known as zbr | 14:35 | |
*** ociuhandu has quit IRC | 14:49 | |
*** zbr4 has joined #openstack-infra | 14:52 | |
*** zbr has quit IRC | 14:53 | |
*** zbr4 is now known as zbr | 14:53 | |
*** zbr5 has joined #openstack-infra | 15:12 | |
*** zbr has quit IRC | 15:14 | |
*** zbr5 is now known as zbr | 15:14 | |
*** lpetrut has quit IRC | 15:15 | |
*** tosky has quit IRC | 15:19 | |
*** tosky has joined #openstack-infra | 15:20 | |
*** zbr8 has joined #openstack-infra | 15:30 | |
*** zbr has quit IRC | 15:33 | |
*** zbr8 is now known as zbr | 15:33 | |
*** dklyle has joined #openstack-infra | 15:51 | |
*** lpetrut has joined #openstack-infra | 16:05 | |
*** ociuhandu has joined #openstack-infra | 16:07 | |
*** zbr2 has joined #openstack-infra | 16:11 | |
*** zbr has quit IRC | 16:13 | |
*** zbr2 is now known as zbr | 16:13 | |
*** zbr6 has joined #openstack-infra | 16:15 | |
*** zbr has quit IRC | 16:17 | |
*** zbr6 is now known as zbr | 16:17 | |
*** calbers has quit IRC | 16:32 | |
*** icey has quit IRC | 16:33 | |
*** icey has joined #openstack-infra | 16:34 | |
*** calbers has joined #openstack-infra | 16:35 | |
*** zbr has quit IRC | 16:38 | |
*** gyee has joined #openstack-infra | 16:40 | |
*** harlowja has joined #openstack-infra | 16:41 | |
*** hemna has quit IRC | 16:47 | |
*** hemna has joined #openstack-infra | 16:49 | |
*** harlowja has quit IRC | 16:59 | |
*** slaweq_ has joined #openstack-infra | 17:08 | |
*** ociuhandu_ has joined #openstack-infra | 17:09 | |
*** ociuhandu has quit IRC | 17:13 | |
*** ociuhandu_ has quit IRC | 17:14 | |
*** hamalq has joined #openstack-infra | 17:16 | |
*** ociuhandu has joined #openstack-infra | 17:23 | |
*** gfidente has quit IRC | 17:23 | |
*** ociuhandu has quit IRC | 17:28 | |
*** __ministry1 has joined #openstack-infra | 17:34 | |
*** arne_wiebalck has quit IRC | 17:35 | |
*** arne_wiebalck has joined #openstack-infra | 17:36 | |
openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Move git-review to opendev tenant https://review.opendev.org/c/openstack/project-config/+/777774 | 17:36 |
*** lpetrut has quit IRC | 17:40 | |
*** slaweq_ has quit IRC | 17:54 | |
*** andrewbonney has quit IRC | 18:15 | |
*** __ministry1 has quit IRC | 18:29 | |
*** slaweq_ has joined #openstack-infra | 18:33 | |
*** slaweq_ has quit IRC | 18:44 | |
openstackgerrit | Merged openstack/project-config master: Move git-review to opendev tenant https://review.opendev.org/c/openstack/project-config/+/777774 | 18:50 |
*** sreejithp has quit IRC | 19:16 | |
*** slaweq_ has joined #openstack-infra | 19:44 | |
*** tjgresha has joined #openstack-infra | 19:50 | |
*** slaweq_ has quit IRC | 20:22 | |
*** ociuhandu has joined #openstack-infra | 20:58 | |
*** ociuhandu has quit IRC | 21:02 | |
tjgresha | any special permissions needed for a 3rdparty ci account to be able to comment on project - say Nova or Neutron? | 21:33 |
tjgresha | gerrit permissions that is | 21:37 |
clarkb | tjgresha: yes for nova your third party ci account must be in the nova-ci group | 21:45 |
clarkb | tjgresha: for neutron it is the neutron-ci group and so on | 21:45 |
clarkb | those groups are managed by the projects themselves | 21:45 |
tjgresha | meaning i should hit up corresponding IRC channels for access? | 21:46 |
clarkb | yes, it will be up to the respective projects to decide if they want you to vote +/-1 or not (all third party ci can leave comments iwthout vote values) | 21:47 |
tjgresha | ok, Thanks clarkb | 21:48 |
*** tjgresha has quit IRC | 22:17 | |
*** sreejithp has joined #openstack-infra | 22:21 | |
*** hamalq has quit IRC | 23:40 | |
*** zzzeek has quit IRC | 23:41 | |
*** zzzeek has joined #openstack-infra | 23:43 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!