| *** Guest95463 has joined #openstack-infra | 00:00 | |
| *** Guest95463 is now known as logan- | 00:00 | |
| *** goldyfruit_ has joined #openstack-infra | 00:01 | |
| *** dave-mccowan has joined #openstack-infra | 00:15 | |
| *** bobh has quit IRC | 00:37 | |
| *** rfolco has joined #openstack-infra | 00:42 | |
| *** bobh has joined #openstack-infra | 01:15 | |
| *** markvoelker has joined #openstack-infra | 01:20 | |
| *** dave-mccowan has quit IRC | 01:23 | |
| *** markvoelker has quit IRC | 01:25 | |
| cmurphy | my first time encountering a real life http 402 https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e47/680797/1/check/openstack-tox-py36/e477716/ | 01:54 |
|---|---|---|
| *** bobh has quit IRC | 01:55 | |
| clarkb | ya we've reached out to them again. Happened earlier this weel and was fixed but has since returned | 02:04 |
| donnyd | Well the panel is finally installed and operational (mostly). I had the wrong size feeder breaker and had to order one that won't be here till monday. | 02:11 |
| donnyd | FN only took about an hour outage | 02:11 |
| donnyd | Its back up and has been since about 6pm EST | 02:12 |
| donnyd | I am not sure we want to bring it back online as there is a small risk that the breaker replacement will take more time than I have battery | 02:12 |
| donnyd | Its unlikely but possible | 02:12 |
| donnyd | The whole operation should take about 3 minutes and I have 9 minutes of power | 02:13 |
| clarkb | We can probably leave it off for the weekend if that puts lessstress on your end | 02:14 |
| clarkb | weekends are quiet | 02:14 |
| *** jcoufal has joined #openstack-infra | 02:22 | |
| donnyd | Well it won't really be 100% until late monday night | 02:33 |
| donnyd | Its no stress on my end, it should all "just work" | 02:34 |
| donnyd | But I also don't want to be 70 jobs deep and lose everyones build either | 02:34 |
| donnyd | During the breaker replacement I should say. The one that is in there now is only 10amps short and the whole thing is over provisioned by 50%... so it should be just fine | 02:35 |
| donnyd | I will leave it up to infra | 02:35 |
| donnyd | I am happy either way | 02:35 |
| donnyd | I will be much happier when the gas finally gets hooked up.. but for now having the electrical part done is good progress | 02:36 |
| * donnyd goes to bed because im sleepy | 02:39 | |
| *** benj_ has joined #openstack-infra | 02:50 | |
| *** spsurya has joined #openstack-infra | 03:09 | |
| *** auristor has quit IRC | 03:48 | |
| *** auristor has joined #openstack-infra | 04:01 | |
| *** jamesmcarthur has joined #openstack-infra | 04:27 | |
| *** jamesmcarthur has quit IRC | 04:38 | |
| *** yamamoto has joined #openstack-infra | 05:33 | |
| *** yamamoto has quit IRC | 05:46 | |
| openstackgerrit | Merged zuul/zuul master: web: fix connections list github base_url https://review.opendev.org/678950 | 05:55 |
| openstackgerrit | Merged zuul/zuul master: Annotate QueueItem logger https://review.opendev.org/678226 | 05:59 |
| *** kjackal has joined #openstack-infra | 06:02 | |
| *** ricolin has joined #openstack-infra | 06:05 | |
| *** yamamoto has joined #openstack-infra | 06:19 | |
| *** yamamoto has quit IRC | 06:21 | |
| *** yamamoto has joined #openstack-infra | 06:21 | |
| *** yolanda has quit IRC | 06:30 | |
| *** fdegir has quit IRC | 06:30 | |
| *** georgk has quit IRC | 06:30 | |
| *** fdegir has joined #openstack-infra | 06:31 | |
| *** georgk has joined #openstack-infra | 06:31 | |
| *** yamamoto has quit IRC | 06:47 | |
| *** yamamoto has joined #openstack-infra | 06:54 | |
| prometheanfire | is galooks like zuul is stalled or something? feels odd to see only 3 reviews in the queue http://zuul.openstack.org/status | 07:09 |
| prometheanfire | also, looks like the requirements propose-updates job stalled due to timeouts, any chance it can be rekicked? | 07:09 |
| prometheanfire | https://f705a0bd6b0052614dfb-5f206d1ac6837606572ca22039b53566.ssl.cf2.rackcdn.com/periodic/opendev.org/openstack/requirements/master/propose-updates/b417c5f/ specifically | 07:10 |
| *** yamamoto_ has joined #openstack-infra | 07:26 | |
| *** yamamoto has quit IRC | 07:27 | |
| *** yamamoto_ has quit IRC | 07:41 | |
| *** wpp has quit IRC | 07:46 | |
| *** snapiri has joined #openstack-infra | 08:00 | |
| *** yamamoto has joined #openstack-infra | 08:14 | |
| *** yamamoto has quit IRC | 08:27 | |
| *** yamamoto has joined #openstack-infra | 08:31 | |
| *** yamamoto has quit IRC | 08:38 | |
| *** yamamoto has joined #openstack-infra | 08:41 | |
| *** yamamoto has quit IRC | 08:58 | |
| *** ociuhandu has joined #openstack-infra | 09:01 | |
| *** ralonsoh has joined #openstack-infra | 09:02 | |
| ralonsoh | fungi, hi, one question if possible? maybe I'm doing something wrong, but now I can't decompress the logs files stored by the CI | 09:03 |
| ralonsoh | e.g.: https://f53895d70580881b785a-30f2d4bfe90ac8b488e5b54b3e170d95.ssl.cf5.rackcdn.com/680427/2/check/tempest-multinode-full-py3/0662559/controller/logs/screen-q-svc.txt.gz | 09:03 |
| ralonsoh | gzip: screen-q-svc.txt.gz: not in gzip format | 09:03 |
| *** ociuhandu has quit IRC | 09:05 | |
| *** snapiri has quit IRC | 09:07 | |
| *** jcoufal has quit IRC | 09:23 | |
| *** slaweq has joined #openstack-infra | 09:25 | |
| *** snapiri has joined #openstack-infra | 09:38 | |
| *** slaweq has quit IRC | 09:46 | |
| Roamer` | ralonsoh, it's a bit weird, but the webserver is configured so that it sends the file to the browser and announces that it is compressed, so the browser will automatically uncompress it. The saved file has a .gz extension, but it is actually a plain text file - open it with a file viewer (less, more, any text editor) | 09:50 |
| Roamer` | you may rename it first if you wish so that your system will automatically open it with a suitable program | 09:50 |
| ralonsoh | Roamer`, hmmm that's curious but it's true, this is a plain text file | 09:51 |
| ralonsoh | Roamer`, thanks! | 09:51 |
| *** spsurya has quit IRC | 09:54 | |
| fungi | infra-root: i'm going to be on the road all day heading home, so won't be around in here again until 21:00 at the earliest (but quite likely not until tomorrow). amorin has replied via e-mail that ovh should be working for us again if we want to revert the change which removed it from the log upload rotation | 10:22 |
| *** ralonsoh has quit IRC | 10:30 | |
| *** slaweq has joined #openstack-infra | 10:42 | |
| *** xenos76 has joined #openstack-infra | 11:05 | |
| *** slaweq has quit IRC | 11:06 | |
| *** e0ne has joined #openstack-infra | 11:30 | |
| *** Lucas_Gray has joined #openstack-infra | 11:33 | |
| *** e0ne has quit IRC | 11:33 | |
| *** Lucas_Gray has quit IRC | 11:34 | |
| *** Lucas_Gray has joined #openstack-infra | 11:36 | |
| *** yamamoto has joined #openstack-infra | 11:56 | |
| *** nicolasbock has joined #openstack-infra | 12:05 | |
| *** kjackal has quit IRC | 12:18 | |
| *** e0ne has joined #openstack-infra | 12:32 | |
| *** e0ne_ has joined #openstack-infra | 12:38 | |
| *** e0ne has quit IRC | 12:39 | |
| *** e0ne has joined #openstack-infra | 12:44 | |
| *** e0ne_ has quit IRC | 12:44 | |
| *** e0ne has quit IRC | 12:46 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove openSUSE 42.3 https://review.opendev.org/677158 | 12:48 |
| *** ricolin_ has joined #openstack-infra | 12:49 | |
| *** ricolin has quit IRC | 12:50 | |
| *** yamamoto has quit IRC | 12:50 | |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Revert "Stop storing logs on OVH" https://review.opendev.org/680855 | 12:55 |
| AJaeger_ | fungi, safe travels. Hope your home is fine! | 12:56 |
| *** jcoufal has joined #openstack-infra | 12:56 | |
| AJaeger_ | here's the revert for swift on OVH ^ Before we merge, could somebody test the account, please? ^ | 12:56 |
| *** jcoufal has quit IRC | 13:01 | |
| *** dklyle has quit IRC | 13:03 | |
| openstackgerrit | Merged opendev/base-jobs master: Test switch to prepare-workspace-git role https://review.opendev.org/680695 | 13:16 |
| *** yamamoto has joined #openstack-infra | 13:26 | |
| *** nicolasbock has quit IRC | 13:34 | |
| *** yamamoto has quit IRC | 13:35 | |
| *** yamamoto has joined #openstack-infra | 13:41 | |
| *** zzehring has quit IRC | 13:56 | |
| *** markvoelker has joined #openstack-infra | 13:57 | |
| *** markvoelker has quit IRC | 14:01 | |
| *** bobh has joined #openstack-infra | 14:08 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add prepare-workspace-openshift role https://review.opendev.org/631402 | 14:11 |
| *** bobh has quit IRC | 14:12 | |
| *** eharney has quit IRC | 14:15 | |
| *** kjackal has joined #openstack-infra | 14:20 | |
| *** Lucas_Gray has quit IRC | 14:38 | |
| *** e0ne has joined #openstack-infra | 14:42 | |
| clarkb | infra-root AJaeger_ I get 401's using osc to try a server list against ovh | 15:27 |
| clarkb | Not sure if our clouds.yaml is actually wrong or if still unhappy from earlier | 15:28 |
| *** eernst has joined #openstack-infra | 15:29 | |
| AJaeger_ | clarkb: let's ask amorin - and I'll WIP the change for now... thanks | 15:30 |
| clarkb | AJaeger_: cmurphy's link from yesterday still errors too: https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_e47/680797/1/check/openstack-tox-py36/e477716/ | 15:30 |
| AJaeger_ | ;( | 15:31 |
| amorin | AJaeger_: I checked the tenant | 15:35 |
| amorin | it's suspended again | 15:35 |
| amorin | ... | 15:35 |
| *** eernst has quit IRC | 15:36 | |
| *** dklyle has joined #openstack-infra | 15:36 | |
| AJaeger_ | amorin: thanks for double checking | 15:38 |
| amorin | AJaeger_: I launch a task to unsuspend | 15:39 |
| amorin | but there is something wrong with the suspending robot | 15:39 |
| AJaeger_ | should we wait until that one is fixed? | 15:40 |
| AJaeger_ | might be better to take the pressure away from you... | 15:41 |
| AJaeger_ | thanks, amorin | 15:41 |
| amorin | can you try again? | 15:41 |
| amorin | because it;s supposed to be open | 15:41 |
| amorin | now | 15:41 |
| AJaeger_ | clarkb: can you check tenant? | 15:42 |
| AJaeger_ | amorin: the link clarkb gave works now for me, thanks | 15:42 |
| AJaeger_ | clarkb, amorin , should we wait and give OVH a chance to fix the suspend bot before we use OVH again? | 15:43 |
| *** icarusfactor has joined #openstack-infra | 15:43 | |
| amorin | AJaeger_: I dont know | 15:44 |
| amorin | the suspend bot should not come back today AFAIK | 15:44 |
| amorin | but tomorrow... | 15:45 |
| AJaeger_ | that's what I feared ;) | 15:45 |
| *** factor has quit IRC | 15:46 | |
| *** icarusfactor has quit IRC | 15:46 | |
| *** icarusfactor has joined #openstack-infra | 15:47 | |
| *** ricolin_ has quit IRC | 15:49 | |
| *** icarusfactor has quit IRC | 15:49 | |
| *** factor has joined #openstack-infra | 15:49 | |
| *** jtomasek has joined #openstack-infra | 15:51 | |
| *** kjackal has quit IRC | 15:52 | |
| *** Goneri has joined #openstack-infra | 16:02 | |
| *** e0ne has quit IRC | 16:02 | |
| *** e0ne has joined #openstack-infra | 16:03 | |
| AJaeger_ | ovh dashboard for nodes looks good again, see http://grafana.openstack.org/d/BhcSH5Iiz/nodepool-ovh?orgId=1 | 16:03 |
| *** yamamoto has quit IRC | 16:06 | |
| *** icarusfactor has joined #openstack-infra | 16:14 | |
| *** factor has quit IRC | 16:14 | |
| openstackgerrit | Mohammed Naser proposed zuul/nodepool master: Fix Kubernetes driver documentation https://review.opendev.org/680879 | 16:19 |
| *** factor has joined #openstack-infra | 16:23 | |
| *** icarusfactor has quit IRC | 16:24 | |
| *** ccamacho has quit IRC | 16:27 | |
| *** jbadiapa has quit IRC | 16:28 | |
| openstackgerrit | Mohammed Naser proposed zuul/nodepool master: Add extra spacing to avoid monospace rendering https://review.opendev.org/680880 | 16:33 |
| *** yamamoto has joined #openstack-infra | 16:39 | |
| *** factor has quit IRC | 16:44 | |
| *** factor has joined #openstack-infra | 16:44 | |
| *** yamamoto has quit IRC | 16:45 | |
| *** factor has quit IRC | 16:51 | |
| *** factor has joined #openstack-infra | 16:51 | |
| openstackgerrit | Mohammed Naser proposed zuul/nodepool master: Fix chroot type https://review.opendev.org/680881 | 16:53 |
| *** slaweq has joined #openstack-infra | 17:01 | |
| clarkb | ya maybe we wait for more confirmation the bot is fixed before enabling swoft to avoid post failures tomorrow | 17:20 |
| *** slaweq has quit IRC | 17:24 | |
| *** slaweq_ has joined #openstack-infra | 17:29 | |
| *** slaweq_ has quit IRC | 17:39 | |
| *** ianychoi_ has joined #openstack-infra | 17:39 | |
| *** ianychoi has quit IRC | 17:42 | |
| *** e0ne has quit IRC | 17:53 | |
| *** markvoelker has joined #openstack-infra | 17:59 | |
| *** markvoelker has quit IRC | 18:06 | |
| *** e0ne has joined #openstack-infra | 18:10 | |
| *** e0ne has quit IRC | 18:12 | |
| *** e0ne has joined #openstack-infra | 18:22 | |
| *** e0ne has quit IRC | 19:07 | |
| *** e0ne has joined #openstack-infra | 19:07 | |
| *** yamamoto has joined #openstack-infra | 19:18 | |
| *** panda|rover has quit IRC | 19:18 | |
| *** panda has joined #openstack-infra | 19:18 | |
| *** yamamoto has quit IRC | 19:23 | |
| *** e0ne has quit IRC | 19:30 | |
| *** pcaruana has quit IRC | 19:36 | |
| *** yolanda has joined #openstack-infra | 19:54 | |
| *** logan- has quit IRC | 20:12 | |
| *** hwoarang has quit IRC | 20:12 | |
| *** hwoarang has joined #openstack-infra | 20:12 | |
| *** logan- has joined #openstack-infra | 20:12 | |
| *** Lucas_Gray has joined #openstack-infra | 20:14 | |
| *** bobh has joined #openstack-infra | 20:22 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed zuul/zuul master: Make default merge-failure-message more generic https://review.opendev.org/680886 | 20:29 |
| *** ccamacho has joined #openstack-infra | 20:30 | |
| *** jtomasek has quit IRC | 20:31 | |
| *** dciabrin_ has quit IRC | 20:43 | |
| *** dciabrin has joined #openstack-infra | 20:46 | |
| *** xenos76 has quit IRC | 21:06 | |
| *** dtantsur|afk has quit IRC | 21:09 | |
| *** dtantsur has joined #openstack-infra | 21:14 | |
| *** bobh has quit IRC | 21:35 | |
| *** bobh has joined #openstack-infra | 21:42 | |
| *** bobh has quit IRC | 21:46 | |
| *** factor has quit IRC | 22:05 | |
| *** factor has joined #openstack-infra | 22:05 | |
| *** factor has quit IRC | 22:12 | |
| *** bobh has joined #openstack-infra | 22:19 | |
| *** goldyfruit_ is now known as goldyufruit | 22:25 | |
| *** goldyufruit is now known as goldyfruit | 22:25 | |
| *** bobh has quit IRC | 22:30 | |
| *** bobh has joined #openstack-infra | 22:31 | |
| *** markvoelker has joined #openstack-infra | 22:32 | |
| *** bobh has quit IRC | 22:36 | |
| *** markvoelker has quit IRC | 22:37 | |
| *** tkajinam has joined #openstack-infra | 22:48 | |
| *** eernst_ has joined #openstack-infra | 22:57 | |
| *** rcernin has joined #openstack-infra | 23:00 | |
| *** eernst_ has quit IRC | 23:02 | |
| johnsom | Do the core counts vary by hosting provider? | 23:02 |
| johnsom | I.e. do jobs get varying numbers of cores. | 23:03 |
| ianw | johnsom: no, all the flavours we have should be 8x | 23:06 |
| johnsom | Hmm, ok. Thank you. | 23:07 |
| ianw | although, i think we turned up arm64 to 16, if you're using that | 23:07 |
| johnsom | Nope, probably could, but we are not yet. | 23:07 |
| johnsom | I am just trying to brainstorm ideas of why we are seeing these random RETRY_LIMIT failures. I was thinking maybe the tempest concurrency was different and we were running out of memory. | 23:08 |
| johnsom | Seems unlikely though, the times it passes peakmem seems to be ok | 23:09 |
| fungi | we have at times varies the vcpu count though, depending on how well or poorly some providers performed | 23:10 |
| ianw | i assume you've correlated them and they're not on the same provider? | 23:10 |
| *** hamzy_ has joined #openstack-infra | 23:11 | |
| fungi | "CPU count, speed, and supported processor flags differ, sometimes even within the same cloud region." https://docs.openstack.org/infra/manual/testing.html#known-differences-to-watch-out-for | 23:11 |
| johnsom | Most have been on RAX. | 23:12 |
| *** hamzy has quit IRC | 23:12 | |
| johnsom | It's just super hard to debug because we end up with no logs and the console output is... not very helpful | 23:12 |
| fungi | johnsom: also as i mentioned earlier, rax is our only xen-based provider (afaik) so is it possible we're tickling a hypervisor bug there? | 23:12 |
| fungi | and yeah, from the console stream you pasted earlier, zuul/ansible just up and loses access to the instance, so it may have crashed out utterly with a guest termination | 23:13 |
| johnsom | Any way to look into that further? Does RAX have an instance log we could look at? | 23:14 |
| *** Lucas_Gray has quit IRC | 23:22 | |
| *** Lucas_Gray has joined #openstack-infra | 23:23 | |
| ianw | johnsom: we can't really see what's "behind the curtain" as such ... you can query logstash by provider however | 23:23 |
| fungi | not really, just a novnc-like (but not actually) console, though also nodepool deletes the instances automatically | 23:23 |
| *** Lucas_Gray has quit IRC | 23:27 | |
| *** armax has quit IRC | 23:29 | |
| *** bobh has joined #openstack-infra | 23:38 | |
| *** dchen has joined #openstack-infra | 23:40 | |
| *** bobh has quit IRC | 23:43 | |
| *** bobh has joined #openstack-infra | 23:45 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Set zuul_work_dir for tox testing https://review.opendev.org/680895 | 23:46 |
| *** markvoelker has joined #openstack-infra | 23:47 | |
| *** bobh has quit IRC | 23:50 | |
| *** yamamoto has joined #openstack-infra | 23:53 | |
| *** markvoelker has quit IRC | 23:57 | |
| *** markvoelker has joined #openstack-infra | 23:57 | |
| *** yamamoto has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!