| *** ociuhandu has quit IRC | 00:00 | |
| *** ociuhandu_ has joined #openstack-infra | 00:00 | |
| *** ociuhandu_ has quit IRC | 00:08 | |
| *** rosmaita has left #openstack-infra | 00:20 | |
| *** rfolco has joined #openstack-infra | 01:01 | |
| *** ociuhandu has joined #openstack-infra | 01:04 | |
| *** ociuhandu has quit IRC | 01:09 | |
| *** diablo_rojo has joined #openstack-infra | 01:09 | |
| *** diablo_rojo has quit IRC | 01:15 | |
| *** ricolin has joined #openstack-infra | 01:20 | |
| *** ricolin_ has joined #openstack-infra | 01:27 | |
| *** jungleboyj has joined #openstack-infra | 01:27 | |
| jungleboyj | Hey, you guys know anything about etherpad going down. | 01:27 |
|---|---|---|
| jungleboyj | Putting a damper on Upstream Institute. | 01:27 |
| clarkb | the server seems to ping but has no http server? | 01:29 |
| clarkb | fungi ^ | 01:29 |
| jungleboyj | clarkb: fungi Thanks for looking! | 01:30 |
| *** ricolin has quit IRC | 01:30 | |
| clarkb | http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=114&rra_id=all | 01:31 |
| clarkb | iowait seems high | 01:31 |
| mnaser | hmm | 01:32 |
| mnaser | if any other infra-root is around? ^ | 01:32 |
| jungleboyj | Have we tried turning it off and back on again? | 01:32 |
| clarkb | is that something that happens during a live migrate? | 01:33 |
| mnaser | it could be an underlying hypervisor issue but the only way is to get into it, i guess. | 01:33 |
| mnaser | not likely, not for that long anyways | 01:33 |
| clarkb | mnaser ya I suspect something with the hosting given that cpu status | 01:33 |
| *** linshiyx has joined #openstack-infra | 01:33 | |
| mnaser | its not even pingable anymore | 01:33 |
| mnaser | 5 packets transmitted, 0 received, 100% packet loss, time 4076ms | 01:34 |
| clarkb | I suspect we will need to:check the console then reboot if it doesnt self:correct | 01:35 |
| fungi | "This message is to inform you that the host your cloud server '863c1545-38f6-4238-838d-cb108388bb32' resides on alerted our monitoring systems at 2019-11-03T01:31:55.844193. We are currently investigating the issue and will update you as soon as we have additional information regarding what is causing the alert. Please do not access or modify '863c1545-38f6-4238-838d-cb108388bb32' during this | 01:36 |
| fungi | process. Please reference this incident ID if you need to contact support: CSHD-4ef24d8c" | 01:36 |
| fungi | i love that they've stopped including instance names, volume names, et cetera in these notifications some months ago | 01:36 |
| mnaser | "resides on alerted our monitoring systems" | 01:36 |
| fungi | i totally recognize that uuid | 01:36 |
| clarkb | col so dont do anything | 01:36 |
| mnaser | did we accidentally a word | 01:36 |
| jungleboyj | Oy vey. Ok. Thank you for the quick investigation. We have a backup etherpad at the moment. | 01:37 |
| clarkb | the : is really close to the space on this device | 01:37 |
| mnaser | o | 01:38 |
| fungi | mnaser: no idea, the message i quoted made sense to me ("the host your cloud server [...] resides on alerted our monitoring systems [...]") | 01:38 |
| mnaser | its pretty neat integration | 01:38 |
| fungi | anyway, i'll do an openstack server list on dfw in a sec and double check that uuid is etherpad.o.o but i strongly suspect it is | 01:39 |
| fungi | odds are it'll get booted on another host shortly | 01:39 |
| fungi | rackspace support has also opened trouble ticket 191103-ord-0000049 for us on this | 01:42 |
| fungi | confirmed via openstack server list, 863c1545-38f6-4238-838d-cb108388bb32 is etherpad01.openstack.org | 01:43 |
| fungi | #status log the hypervisor host for etherpad.o.o has suffered an outage as of 01:30 utc, provider trouble ticket 191103-ord-0000049 | 01:45 |
| openstackstatus | fungi: finished logging | 01:45 |
| mnaser | yep server is fully dead now | 01:45 |
| fungi | okay, ticket update says it should be back online now... logging into it to double-check | 01:50 |
| fungi | 01:50:19 up 9 min | 01:50 |
| clarkb | seems to work via http too | 01:50 |
| fungi | so i guess it was up as of ~01:41z | 01:50 |
| fungi | yeah, seems to be working again | 01:51 |
| fungi | jungleboyj: ^ | 01:51 |
| clarkb | jungleboyj: ^ can you confirm it works for you too | 01:51 |
| jungleboyj | clarkb: You guys are the best. We are back. Thank you! | 01:52 |
| clarkb | all I did was look at cacti :) | 01:52 |
| * fungi read some e-mail and regurgitated it in here ;) | 01:52 | |
| fungi | #status log etherpad.o.o was back online and confirmed working as of 01:50 utc | 01:53 |
| openstackstatus | fungi: finished logging | 01:53 |
| clarkb | we still have snmp holes | 01:55 |
| clarkb | 01:55 | |
| clarkb | not sure if just waiting for next poll after services started | 01:55 |
| *** rfolco has quit IRC | 01:56 | |
| *** diablo_rojo has joined #openstack-infra | 02:00 | |
| clarkb | ya its back with new data in cacti now too | 02:04 |
| *** diablo_rojo has quit IRC | 02:06 | |
| *** ociuhandu has joined #openstack-infra | 02:13 | |
| fungi | ianw: you've left an edit-secrets session running, blocking additional gpg-agent invocations. looks like it was started several days ago | 02:18 |
| fungi | if you're around, would you mind closing it out? | 02:18 |
| rpioso | diablo_rojo_phon: Is the Shanghai registration desk open on Sunday for picking up badges? | 02:19 |
| clarkb | rpioso: yes from 1pm to 4:30pm. You have to enter via gate 3A | 02:21 |
| *** ociuhandu has quit IRC | 02:22 | |
| rpioso | clarkb: Thank you so much :-) | 02:24 |
| mordred | clarkb, corvus: https://review.opendev.org/#/c/690872/ - the fix to gitea got released in a point release -so we should be good to try 1.9.5 ... I get an API issue with 1.10 that seems real that we should figure out what changed | 02:27 |
| clarkb | I feel like this week is a bad one for me to help with gitea updates | 02:28 |
| fungi | what's fixed in 1.9.5? | 02:29 |
| clarkb | the api checking for root user in 1.9.4 | 02:33 |
| clarkb | we upgraded and had to revert due to that | 02:33 |
| clarkb | it seemed to work otherwise | 02:34 |
| clarkb | it is supposed to 404 but was 500ing | 02:34 |
| fungi | ahh | 02:34 |
| mordred | yeah. we submitted a patch upstream which they accepted - it's in 1.10.0-rc2 and 1.9.5 and master now | 02:43 |
| mordred | I haven't sorted out what's wrong with the 1.10.0-rc2 yet | 02:43 |
| *** ociuhandu has joined #openstack-infra | 02:50 | |
| *** ociuhandu has quit IRC | 02:55 | |
| *** diablo_rojo has joined #openstack-infra | 03:01 | |
| *** goldyfruit___ has quit IRC | 03:05 | |
| *** ricolin_ has quit IRC | 03:12 | |
| *** ricolin_ has joined #openstack-infra | 03:16 | |
| *** diablo_rojo__ has joined #openstack-infra | 03:20 | |
| *** diablo_rojo has quit IRC | 03:23 | |
| *** ricolin has joined #openstack-infra | 03:24 | |
| *** ricolin_ has quit IRC | 03:25 | |
| *** eernst has joined #openstack-infra | 03:30 | |
| *** eernst has quit IRC | 03:33 | |
| *** diablo_rojo__ has quit IRC | 03:58 | |
| *** ricolin has quit IRC | 04:16 | |
| fungi | i mentioned it a couple days ago, but it's still weekend and lots of us have been travelling/jetlagged... how are we handling ssl certs for the opendev.org lb/gitea? our certcheck has been warning us that it's less than a month before the current cert expires, so if there is automation in place i worry it's broken down | 04:31 |
| clarkb | it was a namecheap bought cert. But it should be converted to LE now | 04:32 |
| clarkb | we would have each backend get its own cert for opendev.org I think | 04:32 |
| clarkb | tricky bit is restartimg gitea on update | 04:33 |
| clarkb | but we can have an ansible handler run dockercommands for that | 04:33 |
| mordred | speaking of certs - we should convert storyboard-dev and review-dev to LE too - I've hit a bit of a wall with the ansible+docker and the self-signed cert because we need to inject the CA for storyboard-dev into the java environment - but the java environment is in the container - so things get super strange ... but the problem goes away if storyboard-dev is LE managed | 04:34 |
| fungi | ahh, yep, i see now it's not le, for some reason i thought it already was but couldn't find where we were updating it | 04:34 |
| *** eernst has joined #openstack-infra | 04:48 | |
| *** eernst has quit IRC | 04:55 | |
| *** ociuhandu has joined #openstack-infra | 05:15 | |
| *** ociuhandu has quit IRC | 05:19 | |
| *** ricolin has joined #openstack-infra | 06:07 | |
| *** Goneri has quit IRC | 06:18 | |
| *** ricolin has quit IRC | 06:18 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Update gitea to v1.10.0-rc2 https://review.opendev.org/690873 | 06:20 |
| *** Goneri has joined #openstack-infra | 06:31 | |
| tonyb | ... Fun question do we run the gerrit ssh on any other port than 29418? | 06:54 |
| clarkb | no | 06:54 |
| clarkb | 22 hosts the host ssh | 06:55 |
| clarkb | and that is all the ssh on the server | 06:55 |
| tonyb | clarkb: okay | 06:58 |
| tonyb | clarkb: clearly the failed attempts on 22 were me but the fact they failed indicates that it's a port problem | 06:59 |
| fungi | you're able to reach review.opendev.org:22 but not review.opendev.org:29418 from that network i guess? | 07:00 |
| tonyb | That's what I thought but nc review.opendev.org 29418 | 07:01 |
| tonyb | gives the ssh banner | 07:01 |
| fungi | well, there's plenty of ways to interfere with connections besides rejecting/dropping them at the beginning | 07:02 |
| tonyb | but ssh | 07:02 |
| tonyb | True | 07:03 |
| tonyb | ssh just sits at: debug3: mux_client_request_session: session request sent | 07:04 |
| tonyb | Ahh and the the local NOC has "fixed" it | 07:07 |
| fungi | nice | 07:07 |
| fungi | probably some data exfiltration detector trying to spot ssh tunnels on nonstandard ports | 07:08 |
| fungi | it's good to have confirmation that folks on company networks in china *can* reach 29418/tcp to push changes as long as the corporate network security folks allow it, so really no different than the rest of the world | 07:10 |
| clarkb | fungi: we should confirm from hotel or other local net too | 07:10 |
| clarkb | $corp may backhaul | 07:10 |
| clarkb | with a fi sim :) | 07:11 |
| fungi | sure. also it's been suggested that hotel networks may be exempted from gfw blocks too | 07:11 |
| clarkb | ah | 07:11 |
| fungi | at least western-style hotels | 07:12 |
| fungi | i have been extra paranoid about double-checking issuers on ssl certs before i use sites over https, but so far they've matched the certs i fetch on remote systems outside china for the same sites | 07:13 |
| clarkb | I have found that fi has been more reliable so stopped trying to:wifi | 07:14 |
| clarkb | I'll just pay $10/gb | 07:14 |
| mordred | clarkb: that's what I'm doing | 07:15 |
| fungi | i've just been switching to local networks here and there out of curiosity | 07:15 |
| fungi | i have noticed that https://duckduckgo.com/ is blocked from the marriott network | 07:17 |
| mordred | corvus: I think the template fix for gitea 1.10 worked from watching the stream | 07:18 |
| mordred | yup. job went green | 07:19 |
| mordred | so - I think https://review.opendev.org/#/c/690873/ means we'll be good to go when 1.10 final hits | 07:22 |
| *** Goneri has quit IRC | 07:30 | |
| *** markvoelker has joined #openstack-infra | 07:45 | |
| *** jamesmcarthur has joined #openstack-infra | 08:00 | |
| *** ociuhandu has joined #openstack-infra | 08:00 | |
| *** slaweq has joined #openstack-infra | 08:04 | |
| *** jtomasek has joined #openstack-infra | 08:07 | |
| *** ociuhandu has quit IRC | 08:08 | |
| *** psachin has joined #openstack-infra | 08:17 | |
| *** jamesmcarthur has quit IRC | 08:22 | |
| *** psachin has quit IRC | 08:23 | |
| *** linshiyx has quit IRC | 08:29 | |
| *** yolanda has quit IRC | 08:33 | |
| *** icarusfactor has quit IRC | 08:38 | |
| *** hamzy has quit IRC | 08:59 | |
| *** kmarc_ has joined #openstack-infra | 09:01 | |
| *** slaweq_ has joined #openstack-infra | 09:01 | |
| *** amorin has quit IRC | 09:01 | |
| *** kmarc has quit IRC | 09:05 | |
| *** slaweq has quit IRC | 09:05 | |
| *** amorin has joined #openstack-infra | 09:05 | |
| *** stephenfin has quit IRC | 09:05 | |
| *** stephenfin has joined #openstack-infra | 09:06 | |
| *** jamesdenton has quit IRC | 09:07 | |
| *** openstackstatus has quit IRC | 09:07 | |
| *** jamesdenton has joined #openstack-infra | 09:08 | |
| *** markvoelker has quit IRC | 09:10 | |
| *** slaweq__ has joined #openstack-infra | 09:13 | |
| *** slaweq_ has quit IRC | 09:14 | |
| *** jamesmcarthur has joined #openstack-infra | 09:16 | |
| *** ociuhandu has joined #openstack-infra | 09:17 | |
| *** jamesmcarthur has quit IRC | 09:20 | |
| *** ociuhandu has quit IRC | 09:21 | |
| *** pcaruana has joined #openstack-infra | 09:26 | |
| *** factor has joined #openstack-infra | 09:27 | |
| *** ociuhandu has joined #openstack-infra | 10:15 | |
| *** ociuhandu has quit IRC | 10:20 | |
| *** jamesmcarthur has joined #openstack-infra | 10:27 | |
| *** prometheanfire has quit IRC | 10:37 | |
| *** prometheanfire has joined #openstack-infra | 10:38 | |
| *** jamesmcarthur has quit IRC | 10:43 | |
| *** ociuhandu has joined #openstack-infra | 10:56 | |
| *** slaweq__ has quit IRC | 11:16 | |
| *** slaweq_ has joined #openstack-infra | 11:17 | |
| *** hamzy has joined #openstack-infra | 11:27 | |
| *** ociuhandu has quit IRC | 11:31 | |
| *** Lucas_Gray has joined #openstack-infra | 11:32 | |
| *** yamamoto has joined #openstack-infra | 12:13 | |
| *** slaweq_ has quit IRC | 12:26 | |
| *** slaweq has joined #openstack-infra | 12:29 | |
| *** Lucas_Gray has quit IRC | 12:38 | |
| *** yolanda has joined #openstack-infra | 12:44 | |
| *** yamamoto has quit IRC | 12:46 | |
| *** yamamoto has joined #openstack-infra | 12:51 | |
| *** Lucas_Gray has joined #openstack-infra | 12:53 | |
| *** yamamoto has quit IRC | 12:56 | |
| *** yamamoto has joined #openstack-infra | 13:09 | |
| *** bnemec has joined #openstack-infra | 13:09 | |
| *** yamamoto has quit IRC | 13:14 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles. https://review.opendev.org/691114 | 13:29 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles. https://review.opendev.org/691114 | 13:32 |
| *** ociuhandu has joined #openstack-infra | 13:34 | |
| *** slaweq has quit IRC | 13:42 | |
| *** ykarel has joined #openstack-infra | 13:44 | |
| *** slaweq__ has joined #openstack-infra | 13:44 | |
| *** yamamoto has joined #openstack-infra | 13:49 | |
| *** ociuhandu has quit IRC | 13:49 | |
| *** yamamoto has quit IRC | 13:55 | |
| *** mordred has quit IRC | 13:58 | |
| *** sgw has joined #openstack-infra | 14:00 | |
| *** mordred has joined #openstack-infra | 14:01 | |
| *** bnemec has quit IRC | 14:02 | |
| *** Lucas_Gray has quit IRC | 14:03 | |
| *** Lucas_Gray has joined #openstack-infra | 14:05 | |
| *** sgw has quit IRC | 14:17 | |
| *** Lucas_Gray has quit IRC | 14:18 | |
| *** bnemec has joined #openstack-infra | 14:25 | |
| *** bnemec has quit IRC | 14:27 | |
| *** bnemec has joined #openstack-infra | 14:37 | |
| *** slaweq__ has quit IRC | 14:43 | |
| *** yamamoto has joined #openstack-infra | 14:43 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles. https://review.opendev.org/691114 | 14:48 |
| *** yamamoto has quit IRC | 14:48 | |
| *** kjackal has joined #openstack-infra | 14:54 | |
| *** ykarel has quit IRC | 14:58 | |
| *** ykarel has joined #openstack-infra | 14:59 | |
| *** pcaruana has quit IRC | 15:06 | |
| *** hwoarang has quit IRC | 15:07 | |
| *** hwoarang has joined #openstack-infra | 15:13 | |
| *** yamamoto has joined #openstack-infra | 15:23 | |
| *** yamamoto has quit IRC | 15:30 | |
| *** ianychoi has quit IRC | 15:54 | |
| *** ianychoi has joined #openstack-infra | 15:54 | |
| *** ykarel has quit IRC | 16:03 | |
| *** yamamoto has joined #openstack-infra | 16:18 | |
| *** yamamoto has quit IRC | 16:22 | |
| *** Goneri has joined #openstack-infra | 16:22 | |
| *** jistr has quit IRC | 16:46 | |
| *** jistr has joined #openstack-infra | 16:46 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles. https://review.opendev.org/691114 | 16:48 |
| *** yamamoto has joined #openstack-infra | 16:53 | |
| *** jistr has quit IRC | 16:55 | |
| *** jistr has joined #openstack-infra | 16:55 | |
| *** yamamoto has quit IRC | 16:58 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Adds go jobs and roles. https://review.opendev.org/691114 | 17:05 |
| *** ociuhandu has joined #openstack-infra | 17:36 | |
| *** ociuhandu has quit IRC | 17:36 | |
| *** ociuhandu has joined #openstack-infra | 17:36 | |
| *** ociuhandu has quit IRC | 17:39 | |
| *** ociuhandu has joined #openstack-infra | 17:39 | |
| *** ociuhandu has quit IRC | 17:45 | |
| *** yamamoto has joined #openstack-infra | 17:48 | |
| *** yamamoto has quit IRC | 17:53 | |
| *** goldyfruit___ has joined #openstack-infra | 17:57 | |
| *** kjackal has quit IRC | 17:59 | |
| *** goldyfruit___ has quit IRC | 18:05 | |
| *** goldyfruit___ has joined #openstack-infra | 18:06 | |
| *** yamamoto has joined #openstack-infra | 18:21 | |
| *** yamamoto has quit IRC | 18:26 | |
| *** kjackal has joined #openstack-infra | 18:34 | |
| *** bdodd has quit IRC | 18:35 | |
| *** kjackal has quit IRC | 18:54 | |
| *** yamamoto has joined #openstack-infra | 19:00 | |
| *** ociuhandu has joined #openstack-infra | 19:13 | |
| *** ociuhandu has quit IRC | 19:18 | |
| *** bdodd has joined #openstack-infra | 19:40 | |
| *** ociuhandu has joined #openstack-infra | 19:42 | |
| *** ociuhandu has quit IRC | 19:48 | |
| *** jtomasek has quit IRC | 19:59 | |
| *** yamamoto has quit IRC | 20:04 | |
| *** ociuhandu has joined #openstack-infra | 20:47 | |
| *** ociuhandu has quit IRC | 20:52 | |
| *** yamamoto has joined #openstack-infra | 20:54 | |
| *** yamamoto has quit IRC | 20:58 | |
| *** ociuhandu has joined #openstack-infra | 21:10 | |
| *** markvoelker has joined #openstack-infra | 21:14 | |
| *** ociuhandu has quit IRC | 21:21 | |
| *** yamamoto has joined #openstack-infra | 21:32 | |
| *** dciabrin_ has quit IRC | 21:32 | |
| *** dciabrin has joined #openstack-infra | 21:32 | |
| *** markvoelker has quit IRC | 21:32 | |
| *** yamamoto has quit IRC | 21:37 | |
| *** ociuhandu has joined #openstack-infra | 21:45 | |
| *** persia has quit IRC | 21:48 | |
| *** rajinir has quit IRC | 21:48 | |
| *** gsantomaggio has quit IRC | 21:48 | |
| *** prometheanfire has quit IRC | 21:50 | |
| *** ociuhandu has quit IRC | 21:50 | |
| *** prometheanfire has joined #openstack-infra | 21:50 | |
| *** markvoelker has joined #openstack-infra | 21:52 | |
| *** markvoelker has quit IRC | 21:53 | |
| *** rajinir has joined #openstack-infra | 21:54 | |
| *** persia has joined #openstack-infra | 21:54 | |
| *** gsantomaggio has joined #openstack-infra | 21:54 | |
| *** markvoelker has joined #openstack-infra | 21:55 | |
| *** markvoelker has quit IRC | 21:59 | |
| *** ociuhandu has joined #openstack-infra | 22:00 | |
| *** ociuhandu has quit IRC | 22:07 | |
| *** yamamoto has joined #openstack-infra | 22:26 | |
| *** yamamoto has quit IRC | 22:30 | |
| *** ociuhandu has joined #openstack-infra | 22:55 | |
| *** yamamoto has joined #openstack-infra | 22:58 | |
| *** ociuhandu has quit IRC | 23:01 | |
| *** yamamoto has quit IRC | 23:02 | |
| *** dciabrin_ has joined #openstack-infra | 23:10 | |
| *** dciabrin has quit IRC | 23:12 | |
| *** ociuhandu has joined #openstack-infra | 23:25 | |
| *** ociuhandu has quit IRC | 23:26 | |
| *** dchen has joined #openstack-infra | 23:26 | |
| *** ociuhandu has joined #openstack-infra | 23:27 | |
| *** ociuhandu has quit IRC | 23:32 | |
| *** sgw has joined #openstack-infra | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!