| *** jamesmcarthur has joined #openstack-infra | 00:03 | |
| *** jamesmcarthur has quit IRC | 00:08 | |
| *** zzzeek has quit IRC | 00:08 | |
| *** zzzeek has joined #openstack-infra | 00:10 | |
| *** tosky has quit IRC | 00:36 | |
| melwitt | dansmith: are you doing build_status:FAILURE ? | 01:03 |
|---|---|---|
| *** xek has quit IRC | 01:04 | |
| *** jamesmcarthur has joined #openstack-infra | 01:05 | |
| *** jamesmcarthur has quit IRC | 01:37 | |
| *** jamesmcarthur has joined #openstack-infra | 01:39 | |
| *** rcernin has quit IRC | 02:19 | |
| *** rcernin has joined #openstack-infra | 02:33 | |
| *** ysandeep|out is now known as ysandeep|rover | 02:42 | |
| *** rcernin has quit IRC | 02:42 | |
| *** rcernin has joined #openstack-infra | 02:42 | |
| *** dviroel has quit IRC | 03:09 | |
| *** jamesmcarthur has quit IRC | 03:37 | |
| ysandeep|rover | #openstack-infra We're seeing some limestone mirror related RETRY_LIMIT failures again https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_99a/775067/1/check/openstack-tox-py37/99aa3cd/job-output.txt | 03:39 |
| *** jamesmcarthur has joined #openstack-infra | 03:45 | |
| *** jamesmcarthur has quit IRC | 03:49 | |
| *** jamesmcarthur has joined #openstack-infra | 03:50 | |
| ysandeep|rover | ^^ please ignore i got attention on opendev channel, thank you | 03:51 |
| *** psachin has joined #openstack-infra | 03:53 | |
| *** zzzeek has quit IRC | 03:57 | |
| *** zzzeek has joined #openstack-infra | 03:58 | |
| *** dciabrin_ has joined #openstack-infra | 04:03 | |
| *** dciabrin has quit IRC | 04:05 | |
| fungi | yep, i did ignore ;) | 04:05 |
| *** jamesmcarthur has quit IRC | 04:18 | |
| *** jamesmcarthur has joined #openstack-infra | 04:22 | |
| *** jamesmcarthur has quit IRC | 04:27 | |
| *** jamesmcarthur has joined #openstack-infra | 04:28 | |
| *** gyee has quit IRC | 04:30 | |
| *** jamesmcarthur has quit IRC | 04:33 | |
| *** zzzeek has quit IRC | 04:43 | |
| *** zzzeek has joined #openstack-infra | 04:45 | |
| *** ykarel_ has joined #openstack-infra | 04:51 | |
| *** jamesmcarthur has joined #openstack-infra | 04:55 | |
| *** jamesmcarthur has quit IRC | 04:59 | |
| *** jamesmcarthur has joined #openstack-infra | 05:07 | |
| *** zzzeek has quit IRC | 05:15 | |
| *** zzzeek has joined #openstack-infra | 05:16 | |
| *** jamesmcarthur has quit IRC | 05:34 | |
| *** jamesmcarthur has joined #openstack-infra | 05:36 | |
| *** jamesmcarthur has quit IRC | 05:38 | |
| *** jamesmcarthur has joined #openstack-infra | 05:42 | |
| *** jamesmcarthur has quit IRC | 05:45 | |
| *** jamesmcarthur has joined #openstack-infra | 05:52 | |
| *** ykarel_ is now known as ykarel | 05:59 | |
| *** jamesmcarthur has quit IRC | 05:59 | |
| *** jamesmcarthur has joined #openstack-infra | 06:01 | |
| *** zzzeek has quit IRC | 06:03 | |
| *** zzzeek has joined #openstack-infra | 06:04 | |
| *** zzzeek has quit IRC | 06:13 | |
| *** jamesmcarthur has quit IRC | 06:13 | |
| *** zzzeek has joined #openstack-infra | 06:14 | |
| *** zzzeek has quit IRC | 06:24 | |
| *** zzzeek has joined #openstack-infra | 06:30 | |
| *** zzzeek has quit IRC | 06:37 | |
| *** zzzeek has joined #openstack-infra | 06:37 | |
| *** eolivare has joined #openstack-infra | 06:55 | |
| *** zzzeek has quit IRC | 07:01 | |
| *** zzzeek has joined #openstack-infra | 07:02 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** rcernin has quit IRC | 07:37 | |
| *** zzzeek has quit IRC | 07:44 | |
| *** sboyron_ has joined #openstack-infra | 07:45 | |
| *** zzzeek has joined #openstack-infra | 07:46 | |
| *** ykarel_ has joined #openstack-infra | 07:47 | |
| *** ralonsoh has joined #openstack-infra | 07:50 | |
| *** ykarel has quit IRC | 07:50 | |
| *** hashar has joined #openstack-infra | 07:54 | |
| *** ysandeep|rover is now known as ysandeep|lunch | 07:56 | |
| *** rpittau|afk is now known as rpittau | 08:02 | |
| *** zzzeek has quit IRC | 08:05 | |
| *** zzzeek has joined #openstack-infra | 08:07 | |
| *** andrewbonney has joined #openstack-infra | 08:13 | |
| *** jamesmcarthur has joined #openstack-infra | 08:14 | |
| *** rcernin has joined #openstack-infra | 08:18 | |
| *** jamesmcarthur has quit IRC | 08:18 | |
| *** amoralej|off is now known as amoralej | 08:19 | |
| *** tosky has joined #openstack-infra | 08:24 | |
| *** zzzeek has quit IRC | 08:43 | |
| *** zzzeek has joined #openstack-infra | 08:46 | |
| *** zzzeek has quit IRC | 08:55 | |
| *** jpena|off is now known as jpena | 08:56 | |
| *** zzzeek has joined #openstack-infra | 08:56 | |
| *** ysandeep|lunch is now known as ysandeep|rover | 08:59 | |
| *** zzzeek has quit IRC | 09:05 | |
| *** zzzeek has joined #openstack-infra | 09:07 | |
| *** lucasagomes has joined #openstack-infra | 09:13 | |
| *** zzzeek has quit IRC | 09:19 | |
| *** zzzeek has joined #openstack-infra | 09:21 | |
| *** xek has joined #openstack-infra | 09:24 | |
| *** ociuhandu has joined #openstack-infra | 09:26 | |
| *** derekh has joined #openstack-infra | 09:32 | |
| *** yamamoto has quit IRC | 09:40 | |
| *** yamamoto has joined #openstack-infra | 09:41 | |
| *** yamamoto has quit IRC | 09:41 | |
| *** yamamoto has joined #openstack-infra | 09:42 | |
| *** ociuhandu has quit IRC | 09:45 | |
| *** yamamoto has quit IRC | 09:46 | |
| *** yamamoto has joined #openstack-infra | 09:46 | |
| *** yamamoto has quit IRC | 09:46 | |
| *** yamamoto has joined #openstack-infra | 09:47 | |
| tinwood | Hi, as part of our recent charms release, we accidentally created stable/21.01 branches for the openstack/charm-trilio-* repos. Is it possible/easy for these to be deleted or is a pain? | 09:48 |
| *** gfidente has joined #openstack-infra | 09:48 | |
| *** ociuhandu has joined #openstack-infra | 09:50 | |
| *** yamamoto has quit IRC | 09:51 | |
| *** yamamoto has joined #openstack-infra | 09:52 | |
| *** yamamoto has quit IRC | 09:53 | |
| *** yamamoto has joined #openstack-infra | 09:54 | |
| *** rcernin has quit IRC | 09:55 | |
| *** ociuhandu has quit IRC | 10:02 | |
| *** yamamoto has quit IRC | 10:03 | |
| *** rcernin has joined #openstack-infra | 10:09 | |
| *** ociuhandu has joined #openstack-infra | 10:11 | |
| *** ociuhandu has quit IRC | 10:11 | |
| *** ociuhandu has joined #openstack-infra | 10:18 | |
| *** rcernin has quit IRC | 10:23 | |
| *** redrobot9 has joined #openstack-infra | 10:27 | |
| *** redrobot has quit IRC | 10:27 | |
| *** redrobot9 is now known as redrobot | 10:27 | |
| *** strigazi has joined #openstack-infra | 10:30 | |
| *** psachin has quit IRC | 10:35 | |
| *** yamamoto has joined #openstack-infra | 10:35 | |
| *** ykarel_ is now known as ykarel | 10:37 | |
| *** psachin has joined #openstack-infra | 10:43 | |
| *** yamamoto has quit IRC | 10:46 | |
| *** dklyle has quit IRC | 10:58 | |
| *** xek has quit IRC | 10:58 | |
| *** ysandeep|rover is now known as ysandeep|afk | 11:18 | |
| *** yamamoto has joined #openstack-infra | 11:19 | |
| *** jamesmcarthur has joined #openstack-infra | 11:20 | |
| *** jamesmcarthur has quit IRC | 11:20 | |
| *** dviroel has joined #openstack-infra | 11:22 | |
| *** yamamoto has quit IRC | 11:23 | |
| *** ociuhandu has quit IRC | 11:39 | |
| *** sshnaidm|afk has quit IRC | 11:41 | |
| *** dtantsur|afk is now known as dtantsur | 11:41 | |
| *** ociuhandu has joined #openstack-infra | 11:42 | |
| *** ysandeep|afk is now known as ysandeep|rover | 11:42 | |
| *** zzzeek has quit IRC | 11:42 | |
| *** zzzeek has joined #openstack-infra | 11:45 | |
| *** yamamoto has joined #openstack-infra | 11:46 | |
| *** yamamoto has quit IRC | 11:47 | |
| *** yamamoto has joined #openstack-infra | 11:47 | |
| *** rcernin has joined #openstack-infra | 11:48 | |
| *** sshnaidm|afk has joined #openstack-infra | 11:49 | |
| *** sshnaidm|afk is now known as sshnaidm|off | 11:50 | |
| *** yamamoto has quit IRC | 11:52 | |
| *** ociuhandu has quit IRC | 11:54 | |
| *** yamamoto has joined #openstack-infra | 11:54 | |
| *** brtknr has quit IRC | 11:55 | |
| *** brtknr has joined #openstack-infra | 11:56 | |
| *** brtknr has quit IRC | 11:56 | |
| *** brtknr has joined #openstack-infra | 11:56 | |
| *** brtknr has quit IRC | 11:58 | |
| *** brtknr has joined #openstack-infra | 11:59 | |
| *** brtknr has quit IRC | 11:59 | |
| *** brtknr has joined #openstack-infra | 12:00 | |
| *** brtknr has quit IRC | 12:00 | |
| *** brtknr has joined #openstack-infra | 12:00 | |
| *** brtknr has quit IRC | 12:01 | |
| *** brtknr has joined #openstack-infra | 12:01 | |
| *** hashar has quit IRC | 12:03 | |
| *** yamamoto has quit IRC | 12:04 | |
| *** yamamoto has joined #openstack-infra | 12:05 | |
| *** yamamoto has quit IRC | 12:05 | |
| *** rcernin has quit IRC | 12:08 | |
| *** rcernin has joined #openstack-infra | 12:22 | |
| *** xek has joined #openstack-infra | 12:24 | |
| *** jpena is now known as jpena|lunch | 12:31 | |
| *** yamamoto has joined #openstack-infra | 12:33 | |
| *** rlandy has joined #openstack-infra | 12:38 | |
| *** yamamoto has quit IRC | 12:40 | |
| *** ociuhandu has joined #openstack-infra | 12:42 | |
| *** hashar has joined #openstack-infra | 12:42 | |
| *** rcernin has quit IRC | 12:53 | |
| *** ociuhandu has quit IRC | 12:55 | |
| *** ociuhandu has joined #openstack-infra | 13:00 | |
| *** ociuhandu has quit IRC | 13:07 | |
| *** ociuhandu has joined #openstack-infra | 13:11 | |
| *** ociuhandu has quit IRC | 13:15 | |
| *** jamesmcarthur has joined #openstack-infra | 13:21 | |
| *** jamesmcarthur has quit IRC | 13:25 | |
| *** nightmare_unreal has joined #openstack-infra | 13:25 | |
| *** rcernin has joined #openstack-infra | 13:27 | |
| *** amoralej is now known as amoralej|lunch | 13:29 | |
| *** jpena|lunch is now known as jpena | 13:30 | |
| *** ysandeep|rover is now known as ysandeep|mtg | 13:32 | |
| *** nweinber has joined #openstack-infra | 13:43 | |
| *** d34dh0r53 has quit IRC | 13:46 | |
| *** zzzeek has quit IRC | 13:48 | |
| *** zzzeek has joined #openstack-infra | 13:49 | |
| *** d34dh0r53 has joined #openstack-infra | 13:54 | |
| *** yamamoto has joined #openstack-infra | 13:55 | |
| *** zzzeek has quit IRC | 13:59 | |
| *** ociuhandu has joined #openstack-infra | 13:59 | |
| *** zzzeek has joined #openstack-infra | 14:04 | |
| *** ociuhandu has quit IRC | 14:04 | |
| *** ysandeep|mtg is now known as ysandeep | 14:05 | |
| *** ysandeep is now known as ysandeep|away | 14:07 | |
| *** rlandy has quit IRC | 14:07 | |
| *** rlandy has joined #openstack-infra | 14:07 | |
| *** yamamoto has quit IRC | 14:08 | |
| *** ociuhandu has joined #openstack-infra | 14:08 | |
| *** ociuhandu has quit IRC | 14:13 | |
| *** ociuhandu has joined #openstack-infra | 14:26 | |
| *** amoralej|lunch is now known as amoralej | 14:29 | |
| *** rcernin has quit IRC | 14:30 | |
| *** ociuhandu has quit IRC | 14:31 | |
| *** psachin has quit IRC | 14:32 | |
| dansmith | melwitt: yeah | 14:33 |
| dansmith | clarkb: okay yeah, I was wondering wtf was going on | 14:33 |
| *** hemna has quit IRC | 14:35 | |
| dansmith | fungi: did we ever get somewhere on the low disk nodes? | 14:41 |
| dansmith | manifested like this: https://zuul.opendev.org/t/openstack/build/8997228d3c5740688d78d8f9c94e3421/log/job-output.txt#2724 | 14:41 |
| dansmith | recently | 14:41 |
| dansmith | and you can see the ~15g df above | 14:42 |
| *** ociuhandu has joined #openstack-infra | 14:43 | |
| *** ociuhandu has quit IRC | 14:43 | |
| *** ociuhandu has joined #openstack-infra | 14:43 | |
| *** d34dh0r53 has quit IRC | 14:45 | |
| *** d34dh0r53 has joined #openstack-infra | 14:45 | |
| *** hemna has joined #openstack-infra | 14:47 | |
| gmann | fungi, mnaser; can you check this https://review.opendev.org/c/openstack/project-config/+/774378 | 14:58 |
| *** rpittau is now known as rpittau|afk | 15:03 | |
| frickler | dansmith: fungi: that looks like the rootfs isn't being grown properly, the partition has 100g, the fs only 15g, compare with https://ccc35cfa38f56032f297-95ab28bd06b01f2c7089eb38812248e0.ssl.cf2.rackcdn.com/759091/5/gate/nova-grenade-multinode/8997228/zuul-info/host-info.primary.yaml | 15:18 |
| dansmith | frickler: ah yeah, we had that suspicion last weekend when we first saw it | 15:19 |
| frickler | and with 11g already used at the start of the run, it's no wonder that we don't get far with that | 15:19 |
| fungi | dansmith: oh, sorry, i followed up in #openstack-nova before i saw your question in here, but yeah what frickler just said | 15:35 |
| fungi | it's not every node in that provider, it seems to only be a small fraction of them, i kept checking other nodes booted there from the same image and flavor and they were normal | 15:36 |
| fungi | my guess, with no solid evidence, is that somehow growroot is occasionally failing there when some nodes boot | 15:37 |
| fungi | as for why it seems to turn up mostly for grenade, i'm baffled, but maybe it's the only thing that runs devtsack on ubuntu-bionic and somehow ubuntu-focal isn't hitting the same bug? | 15:38 |
| fungi | i could add an lsblk to the validate-host role, i suppose | 15:39 |
| *** dklyle has joined #openstack-infra | 15:39 | |
| frickler | or maybe grenade with two installations triggers the small step from barely running within 3G free space to failing | 15:39 |
| fungi | and see if it reports a larger block device than the filesystem on it | 15:39 |
| *** ociuhandu has quit IRC | 15:40 | |
| frickler | or make a job which fails when there's like less than 20G free in /, keep rechecking until it fails and look at the held node you could from that for further analysis | 15:40 |
| frickler | maybe I can do it over the weekend when I get bored of all the snow | 15:41 |
| *** ykarel has quit IRC | 15:42 | |
| *** ykarel has joined #openstack-infra | 15:42 | |
| *** lbragstad_ has joined #openstack-infra | 15:46 | |
| *** lbragstad has quit IRC | 15:50 | |
| *** sboyron_ has quit IRC | 15:58 | |
| *** ociuhandu has joined #openstack-infra | 16:04 | |
| *** EmilienM has quit IRC | 16:07 | |
| *** yamamoto has joined #openstack-infra | 16:07 | |
| *** ociuhandu has quit IRC | 16:10 | |
| *** EmilienM has joined #openstack-infra | 16:16 | |
| *** rlandy is now known as rlandy|mtg | 16:22 | |
| *** ociuhandu has joined #openstack-infra | 16:24 | |
| *** lbragstad_ is now known as lbragstad | 16:25 | |
| *** yamamoto has quit IRC | 16:27 | |
| *** nightmare_unreal has quit IRC | 16:27 | |
| *** ociuhandu has quit IRC | 16:31 | |
| *** xek has quit IRC | 16:32 | |
| *** ociuhandu has joined #openstack-infra | 16:35 | |
| *** ociuhandu has quit IRC | 16:48 | |
| *** ociuhandu has joined #openstack-infra | 16:49 | |
| *** ociuhandu has quit IRC | 16:51 | |
| *** ociuhandu has joined #openstack-infra | 16:52 | |
| *** ociuhandu has quit IRC | 16:52 | |
| *** ociuhandu has joined #openstack-infra | 16:53 | |
| *** ykarel is now known as ykarel|away | 16:54 | |
| *** jamesmcarthur has joined #openstack-infra | 16:58 | |
| *** ociuhandu has quit IRC | 16:58 | |
| *** ociuhandu has joined #openstack-infra | 17:00 | |
| *** lucasagomes has quit IRC | 17:00 | |
| *** ociuhandu has quit IRC | 17:05 | |
| *** ociuhandu has joined #openstack-infra | 17:05 | |
| *** ykarel|away has quit IRC | 17:06 | |
| *** rlandy|mtg is now known as rlandy | 17:12 | |
| *** jpena is now known as jpena|brb | 17:13 | |
| *** hashar has quit IRC | 17:18 | |
| *** xek has joined #openstack-infra | 17:18 | |
| *** d34dh0r53 has quit IRC | 17:21 | |
| *** d34dh0r53 has joined #openstack-infra | 17:25 | |
| *** eolivare has quit IRC | 17:30 | |
| *** gfidente is now known as gfidente|afk | 17:32 | |
| *** gmann is now known as gmann_afk | 17:40 | |
| *** ociuhandu has quit IRC | 17:42 | |
| *** ociuhandu has joined #openstack-infra | 17:58 | |
| *** jamesmcarthur has quit IRC | 17:59 | |
| *** jamesmcarthur has joined #openstack-infra | 18:00 | |
| *** derekh has quit IRC | 18:00 | |
| *** andrewbonney has quit IRC | 18:02 | |
| *** jpena|brb is now known as jpena | 18:03 | |
| *** jamesmcarthur_ has joined #openstack-infra | 18:03 | |
| *** ociuhandu has quit IRC | 18:04 | |
| *** jamesmcarthur has quit IRC | 18:04 | |
| *** d34dh0r53 has quit IRC | 18:22 | |
| *** d34dh0r53 has joined #openstack-infra | 18:24 | |
| *** yamamoto has joined #openstack-infra | 18:24 | |
| *** hamalq has joined #openstack-infra | 18:36 | |
| *** yamamoto has quit IRC | 18:37 | |
| *** dtantsur is now known as dtantsur|afk | 18:39 | |
| *** gmann_afk is now known as gmann | 18:52 | |
| *** jpena is now known as jpena|off | 18:58 | |
| fungi | tinwood: i haven't forgotten your request, just now getting caught up from a busy yesterday... can you post something about it on the openstack-discuss ml first, or get fnordahl (as charms ptl) to confirm the request? | 19:23 |
| *** jamesmcarthur_ has quit IRC | 19:24 | |
| *** jamesmcarthur has joined #openstack-infra | 19:25 | |
| *** jamesmcarthur has quit IRC | 19:30 | |
| *** ociuhandu has joined #openstack-infra | 19:40 | |
| *** ociuhandu has quit IRC | 19:46 | |
| *** jamesmcarthur has joined #openstack-infra | 19:51 | |
| *** jamesmcarthur has quit IRC | 19:58 | |
| *** jamesmcarthur has joined #openstack-infra | 20:01 | |
| *** jamesmcarthur has quit IRC | 20:01 | |
| *** jamesmcarthur has joined #openstack-infra | 20:02 | |
| *** jamesmcarthur has quit IRC | 20:03 | |
| *** hashar has joined #openstack-infra | 20:19 | |
| *** auristor has quit IRC | 20:21 | |
| *** auristor has joined #openstack-infra | 20:22 | |
| *** jamesmcarthur has joined #openstack-infra | 20:22 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| *** jamesmcarthur has joined #openstack-infra | 20:24 | |
| *** yamamoto has joined #openstack-infra | 20:34 | |
| *** rcernin has joined #openstack-infra | 20:35 | |
| *** ralonsoh has quit IRC | 20:37 | |
| *** jamesmcarthur has quit IRC | 20:43 | |
| *** jamesmcarthur has joined #openstack-infra | 20:44 | |
| *** hashar has quit IRC | 20:46 | |
| *** jamesmcarthur has quit IRC | 20:49 | |
| *** gfidente|afk has quit IRC | 20:51 | |
| *** yamamoto has quit IRC | 20:55 | |
| *** slaweq has quit IRC | 21:00 | |
| *** dciabrin has joined #openstack-infra | 21:01 | |
| *** dciabrin_ has quit IRC | 21:03 | |
| *** jamesmcarthur has joined #openstack-infra | 21:14 | |
| *** jamesmcarthur has quit IRC | 21:18 | |
| *** jamesmcarthur has joined #openstack-infra | 21:18 | |
| *** rcernin has quit IRC | 21:22 | |
| *** jamesmcarthur has quit IRC | 21:36 | |
| *** jamesmcarthur has joined #openstack-infra | 21:38 | |
| *** jamesmcarthur has quit IRC | 21:43 | |
| *** jamesmcarthur has joined #openstack-infra | 21:44 | |
| *** yamamoto has joined #openstack-infra | 22:01 | |
| *** rlandy has quit IRC | 22:05 | |
| *** jamesmcarthur has quit IRC | 22:12 | |
| *** jamesmcarthur has joined #openstack-infra | 22:17 | |
| *** yamamoto has quit IRC | 22:22 | |
| *** nweinber has quit IRC | 22:22 | |
| *** jamesmcarthur has quit IRC | 22:33 | |
| *** yamamoto has joined #openstack-infra | 22:48 | |
| *** dchen has joined #openstack-infra | 23:07 | |
| *** hamalq has quit IRC | 23:51 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!