| sean-k-mooney | frickler: we have been seeing kernel panincs for a long time now mainly on some voluem tests | 07:09 |
|---|---|---|
| sean-k-mooney | like https://124392ceaf274fb4cef6-28d90efd9d81eee8549c62694c407521.ssl.cf1.rackcdn.com/920203/4/check/nova-ceph-multistore/d715129/testr_results.html | 07:09 |
| sean-k-mooney | /sbin/init: can't load library 'libtirpc.so.3' | 07:10 |
| sean-k-mooney | [ 11.595639] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00001000 | 07:10 |
| sean-k-mooney | im wondering if we should consider uing the unstable_test decoraotr to annotate the tests we see fail most often because of this | 07:10 |
| sean-k-mooney | altherniivtly do you know how hard it would be to do a new cirros release with a newer kernel? | 07:11 |
| sean-k-mooney | i have done some poc of trying to come up with an alternitive image to use in the past but we end up wasting a lot of ci time because of these test instablitys | 07:12 |
| sean-k-mooney | im kind of wondering if we could even crfeate a special decorator for kernel panic | 07:12 |
| sean-k-mooney | i.e. have the decorator check the reason the test failed and skip it if there is a panic in the log? | 07:13 |
| sean-k-mooney | https://github.com/openstack/tempest/blob/master/tempest/lib/decorators.py#L153C4-L192 | 07:13 |
| frickler | sean-k-mooney: I'm pretty convinced that this is not a cirros bug, but a volume corruption bug, but I cannot prove it | 07:40 |
| sean-k-mooney | well we see the same issue when using local storage with no cinder volumes too | 07:41 |
| sean-k-mooney | attachign/detaching cinder voluems even empty data volumes just seam to make it more likely to happen | 07:41 |
| sean-k-mooney | it may still be some kind of currption issue btu it happens in the lvm and ceph jobs and when using local stoage so that would make it more likely to be a glance currption issue or somethign like that | 07:42 |
| frickler | one could also look into making a cirros variant that has a proper pre-filled root-fs, not this copy-from-initrd setup, bu no idea how much effort that would be | 07:43 |
| sean-k-mooney | i had partly got an alpine image working https://fileshare.seanmooney.info/ with some dib patches | 07:44 |
| sean-k-mooney | but where that fialed is i was not doing grow root | 07:44 |
| sean-k-mooney | so we ran out of space | 07:44 |
| frickler | you could build it with a large enough root right from the start? | 07:44 |
| sean-k-mooney | sure or i could install one package :) | 07:45 |
| sean-k-mooney | but yes i could pad it to 1g by defualt | 07:45 |
| sean-k-mooney | i was trying to keep it small to not really increas the devstack disk usage | 07:45 |
| sean-k-mooney | https://review.opendev.org/q/topic:%22alpine%22 | 07:46 |
| sean-k-mooney | that was the patch serise i had proposed | 07:46 |
| sean-k-mooney | i think i just need to add it here | 07:47 |
| sean-k-mooney | but its been over a year since i last looks at this so i dont remember all the details | 07:47 |
| *** tosky_ is now known as tosky | 12:28 | |
| *** haleyb is now known as haleyb|out | 20:56 | |
| opendevreview | Merged openstack/whitebox-tempest-plugin master: Adds libvirt watchdog https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/921092 | 22:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!