| donnyd | clarkb: I think I can work something like that up | 00:01 |
|---|---|---|
| donnyd | I was also thinking maybe io load | 00:01 |
| *** jamesmcarthur has joined #openstack-infra | 00:02 | |
| *** jbadiapa has quit IRC | 00:04 | |
| *** jamesmcarthur has quit IRC | 00:06 | |
| *** jamesmcarthur has joined #openstack-infra | 00:10 | |
| *** goldyfruit___ has joined #openstack-infra | 00:11 | |
| donnyd | Is there a way we could publish the nodepool logs to the log server | 00:15 |
| donnyd | I would like to run down the source of some jobs not being able to ssh inbound | 00:15 |
| *** mtreinish has joined #openstack-infra | 00:18 | |
| *** jamesmcarthur has quit IRC | 00:27 | |
| *** armax has quit IRC | 00:32 | |
| *** jamesmcarthur has joined #openstack-infra | 00:39 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: Add remove-zuul-sshkey https://review.opendev.org/680712 | 00:41 |
| *** eharney has quit IRC | 00:42 | |
| *** gyee has quit IRC | 00:42 | |
| *** jamesmcarthur has quit IRC | 00:44 | |
| ianw | speaking of mtu's, i feel like it's likely afs01 -> afs02 could use jumbo packets? | 00:44 |
| ianw | we could enable that with "-jumbo" to the volserver process i think | 00:45 |
| ianw | anyway ... let's sort out one problem at a time :) some stats on zero delta release for fedora -> https://lists.openafs.org/pipermail/openafs-info/2019-September/042864.html | 00:46 |
| *** armax has joined #openstack-infra | 00:56 | |
| *** dychen has joined #openstack-infra | 01:01 | |
| *** dchen has quit IRC | 01:02 | |
| *** jamesmcarthur has joined #openstack-infra | 01:02 | |
| *** dingyichen has joined #openstack-infra | 01:03 | |
| *** dychen has quit IRC | 01:06 | |
| *** mriedem has joined #openstack-infra | 01:06 | |
| *** dingyichen has quit IRC | 01:11 | |
| *** dchen has joined #openstack-infra | 01:13 | |
| *** dychen has joined #openstack-infra | 01:20 | |
| *** dchen has quit IRC | 01:22 | |
| *** tkajinam has quit IRC | 01:25 | |
| *** tkajinam has joined #openstack-infra | 01:26 | |
| *** dingyichen has joined #openstack-infra | 01:39 | |
| *** HenryG has joined #openstack-infra | 01:40 | |
| *** dychen has quit IRC | 01:41 | |
| *** nicolasbock has quit IRC | 01:44 | |
| *** nicolasbock has joined #openstack-infra | 01:45 | |
| *** rlandy has quit IRC | 01:50 | |
| ianw | auristor / others: for our own info, fedora mirror seemed to go wrong @ | 01:53 |
| ianw | Could not end transaction on a ro volume: rxk: authentication expired | 01:53 |
| ianw | Could not update VLDB entry for volume 536871006 | 01:53 |
| ianw | in http://files.openstack.org/mirror/logs/rsync-mirrors/fedora.log.7.gz | 01:53 |
| ianw | unfortunately, that's the last log, it was ~ 2019-08-31 15:56:59 | 01:53 |
| openstackgerrit | Akihiro Motoki proposed openstack/project-config master: Update horizon grafana dashboard https://review.opendev.org/681361 | 01:54 |
| ianw | afaict, every update since then hit a locked volume | 01:54 |
| ianw | until our disk issues and restarts etc for the last few days | 01:55 |
| ianw | this probably explains how r/w delta has got up to 50gb ... it's many days of mirroring | 01:55 |
| auristor | ianw: more bad news - openafs can't do jumbograms with rxkad/crypt connections :( | 01:58 |
| *** nicolasbock has quit IRC | 01:58 | |
| *** jamesmcarthur has quit IRC | 01:59 | |
| *** dingyichen has quit IRC | 02:04 | |
| *** jamesmcarthur has joined #openstack-infra | 02:04 | |
| *** dchen has joined #openstack-infra | 02:04 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** FlorianFa has quit IRC | 02:15 | |
| *** slaweq has quit IRC | 02:15 | |
| *** auristor has quit IRC | 02:20 | |
| *** njohnston|lunch has quit IRC | 02:22 | |
| *** auristor has joined #openstack-infra | 02:27 | |
| *** roman_g has quit IRC | 02:34 | |
| *** FlorianFa has joined #openstack-infra | 02:35 | |
| *** mriedem has quit IRC | 02:48 | |
| *** yamamoto has joined #openstack-infra | 02:56 | |
| *** jamesmcarthur has quit IRC | 03:00 | |
| *** jamesmcarthur has joined #openstack-infra | 03:06 | |
| *** ykarel|away has joined #openstack-infra | 03:08 | |
| *** ccamacho has quit IRC | 03:11 | |
| *** jamesmcarthur has quit IRC | 03:13 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: fedora mirror update : add sleep https://review.opendev.org/681367 | 03:16 |
| *** jamesmcarthur has joined #openstack-infra | 03:25 | |
| *** rh-jelabarre has quit IRC | 03:31 | |
| *** rfolco has quit IRC | 03:32 | |
| *** PrinzElvis has quit IRC | 03:39 | |
| *** larainema has joined #openstack-infra | 03:39 | |
| *** abelur has quit IRC | 03:40 | |
| *** srwilkers has quit IRC | 03:41 | |
| *** rpioso has quit IRC | 03:42 | |
| *** knikolla has quit IRC | 03:42 | |
| *** davecore has quit IRC | 03:42 | |
| *** csatari has quit IRC | 03:42 | |
| *** setuid has quit IRC | 03:42 | |
| *** rpioso has joined #openstack-infra | 03:44 | |
| *** srwilkers has joined #openstack-infra | 03:44 | |
| *** knikolla has joined #openstack-infra | 03:45 | |
| *** setuid has joined #openstack-infra | 03:45 | |
| *** csatari has joined #openstack-infra | 03:45 | |
| *** davecore has joined #openstack-infra | 03:45 | |
| *** PrinzElvis has joined #openstack-infra | 03:45 | |
| *** abelur has joined #openstack-infra | 03:45 | |
| *** jamesmcarthur has quit IRC | 03:57 | |
| *** udesale has joined #openstack-infra | 04:00 | |
| *** ykarel|away has quit IRC | 04:08 | |
| *** ianychoi_ has joined #openstack-infra | 04:25 | |
| *** ykarel|away has joined #openstack-infra | 04:25 | |
| *** ykarel|away is now known as ykarel | 04:25 | |
| *** ianychoi has quit IRC | 04:27 | |
| *** soniya29 has joined #openstack-infra | 04:30 | |
| *** exsdev has quit IRC | 04:42 | |
| *** pcaruana has joined #openstack-infra | 04:42 | |
| *** exsdev has joined #openstack-infra | 04:46 | |
| *** kjackal has joined #openstack-infra | 04:57 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** pcaruana has quit IRC | 05:12 | |
| *** jtomasek has joined #openstack-infra | 05:13 | |
| *** slaweq has quit IRC | 05:16 | |
| *** rcernin has quit IRC | 05:22 | |
| *** redrobot has quit IRC | 05:25 | |
| *** rcernin has joined #openstack-infra | 05:38 | |
| *** kopecmartin|off is now known as kopecmartin | 05:46 | |
| *** rcernin has quit IRC | 05:51 | |
| cmurphy | AJaeger: can you propose the correction to https://review.opendev.org/681161 and i'll approve? | 05:57 |
| *** rpittau|afk is now known as rpittau | 06:01 | |
| AJaeger | cmurphy: good morning - will do ;) | 06:02 |
| AJaeger | cmurphy: https://review.opendev.org/681380 - found one more problem | 06:05 |
| *** yamamoto has quit IRC | 06:06 | |
| cmurphy | thanks AJaeger | 06:07 |
| * cmurphy -> bed | 06:07 | |
| *** rcernin has joined #openstack-infra | 06:09 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** slaweq has quit IRC | 06:15 | |
| *** pcaruana has joined #openstack-infra | 06:21 | |
| *** slaweq has joined #openstack-infra | 06:21 | |
| AJaeger | good night, cmurphy ! | 06:25 |
| *** pgaxatte has joined #openstack-infra | 06:26 | |
| *** AJaeger has quit IRC | 06:28 | |
| *** AJaeger has joined #openstack-infra | 06:35 | |
| *** ricolin has joined #openstack-infra | 06:38 | |
| *** ricolin has quit IRC | 06:39 | |
| *** ociuhandu has joined #openstack-infra | 06:42 | |
| *** ociuhandu has quit IRC | 06:46 | |
| *** lpetrut has joined #openstack-infra | 06:52 | |
| *** ykarel is now known as ykarel|lunch | 06:57 | |
| *** ociuhandu has joined #openstack-infra | 07:04 | |
| *** ociuhandu has quit IRC | 07:04 | |
| *** ociuhandu has joined #openstack-infra | 07:07 | |
| *** ociuhandu has quit IRC | 07:08 | |
| *** trident has quit IRC | 07:08 | |
| *** tesseract has joined #openstack-infra | 07:11 | |
| *** pkopec has joined #openstack-infra | 07:15 | |
| *** roman_g has joined #openstack-infra | 07:16 | |
| *** gfidente has joined #openstack-infra | 07:17 | |
| *** trident has joined #openstack-infra | 07:17 | |
| ianw | #status log all volumes released and mirror-update.opendev.org returned to operation. for info on the debugging done with fedora volume; see https://lists.openafs.org/pipermail/openafs-info/2019-September/042865.html | 07:18 |
| openstackstatus | ianw: finished logging | 07:18 |
| *** ralonsoh has joined #openstack-infra | 07:19 | |
| *** trident has quit IRC | 07:22 | |
| *** rcosnita has joined #openstack-infra | 07:24 | |
| *** ociuhandu has joined #openstack-infra | 07:24 | |
| ianw | i've also disabled fileaudit logging on servers; it doesn't appear related to rsync updates | 07:26 |
| rcosnita | hello. Has anyone got some time to review a very small pull request: https://review.opendev.org/#/c/678786? -> It is a minor improvement in the openstack ironic helm chart. | 07:27 |
| *** trident has joined #openstack-infra | 07:31 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** xenos76 has joined #openstack-infra | 07:38 | |
| *** ccamacho has joined #openstack-infra | 07:38 | |
| *** rcernin has quit IRC | 07:38 | |
| AJaeger | rcosnita: this is the channel for running the OpenStack infrastructure. For helm charts, check the proper channel. Full list of channels is at https://wiki.openstack.org/wiki/IRC | 07:41 |
| rcosnita | sorry, I've just realised that. I added my question on openstack-helm channel. Sorry for the noise. | 07:41 |
| *** ykarel|lunch is now known as ykarel | 07:45 | |
| *** e0ne has joined #openstack-infra | 07:56 | |
| *** jbadiapa has joined #openstack-infra | 08:01 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 08:09 | |
| *** ykarel is now known as ykarel|meeting | 08:12 | |
| *** e0ne has quit IRC | 08:12 | |
| *** dchen has quit IRC | 08:12 | |
| *** ociuhandu has quit IRC | 08:12 | |
| *** ykarel_ has joined #openstack-infra | 08:14 | |
| *** ociuhandu has joined #openstack-infra | 08:15 | |
| *** ykarel|meeting has quit IRC | 08:16 | |
| *** ykarel_ is now known as ykarel|meeting | 08:18 | |
| *** derekh has joined #openstack-infra | 08:20 | |
| *** tkajinam has quit IRC | 08:27 | |
| *** e0ne has joined #openstack-infra | 08:40 | |
| zbr | ianw: morning! | 08:48 |
| *** priteau has joined #openstack-infra | 08:53 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/openstack-zuul-jobs master: openstack-tox-molecule: replace success-url and failure-url https://review.opendev.org/681251 | 09:02 |
| *** lpetrut has quit IRC | 09:04 | |
| *** ianychoi_ has quit IRC | 09:09 | |
| *** rcernin has joined #openstack-infra | 09:10 | |
| *** gfidente has quit IRC | 09:10 | |
| *** rcosnita has quit IRC | 09:11 | |
| *** soniya29 has quit IRC | 09:18 | |
| *** dtantsur|afk is now known as dtantsur | 09:30 | |
| *** ykarel_ has joined #openstack-infra | 09:32 | |
| *** ykarel|meeting has quit IRC | 09:34 | |
| *** ricolin_phone has joined #openstack-infra | 09:35 | |
| *** ricolin_phone has quit IRC | 09:38 | |
| *** ykarel_ is now known as ykarel | 09:39 | |
| *** gfidente has joined #openstack-infra | 09:40 | |
| *** rcernin has quit IRC | 09:41 | |
| *** ociuhandu has quit IRC | 09:43 | |
| *** prometheanfire has quit IRC | 09:43 | |
| *** prometheanfire has joined #openstack-infra | 09:44 | |
| *** kjackal has quit IRC | 09:49 | |
| *** ociuhandu has joined #openstack-infra | 09:49 | |
| *** kjackal has joined #openstack-infra | 09:51 | |
| *** pgaxatte has quit IRC | 09:52 | |
| *** ociuhandu has quit IRC | 09:54 | |
| *** pkopec has quit IRC | 10:24 | |
| *** pkopec has joined #openstack-infra | 10:26 | |
| *** ianychoi has joined #openstack-infra | 10:30 | |
| *** ianychoi has quit IRC | 10:45 | |
| *** ianychoi has joined #openstack-infra | 10:45 | |
| sshnaidm|ruck | hi, how can I build new containers with zuul? The current containers on docker.io/zuul has zuul version 3.5.0 which is quite old | 11:00 |
| sshnaidm|ruck | mordred, clarkb ^^ | 11:00 |
| AJaeger | where are you looking? https://hub.docker.com/r/zuul/zuul/tags has the latest build, doesn't it? | 11:01 |
| *** mahajan-abhishek has joined #openstack-infra | 11:02 | |
| AJaeger | sshnaidm|ruck: ^ | 11:03 |
| *** shachar has quit IRC | 11:08 | |
| *** snapiri has joined #openstack-infra | 11:08 | |
| *** rh-jelabarre has joined #openstack-infra | 11:16 | |
| sshnaidm|ruck | AJaeger, weird, this build has 3.5.0 | 11:20 |
| *** sshnaidm|ruck is now known as sshnaidm|bbl | 11:20 | |
| *** ykarel is now known as ykarel|afk | 11:20 | |
| AJaeger | sshnaidm|bbl: but is it recent? So, is the version number wrong or the content also outdated? Best discuss on #zuul in either case... | 11:21 |
| *** udesale has quit IRC | 11:31 | |
| zbr | ouch those zuul/zuul does not even have tags on it, maybe there is another one that has versioned tags? clearly it would be very useful to have versioned tags and also a mobile one like "latest" that points to latest release. | 11:32 |
| *** nicolasbock has joined #openstack-infra | 11:39 | |
| *** mattymo has joined #openstack-infra | 11:40 | |
| *** jpena is now known as jpena|lunch | 11:40 | |
| mattymo | Anyone around familiar with zuul/nodepool? I know I can add secret env vars to zuul jobs, but for nodepool, you can't embed secrets in nodepool.yaml. I have to host the nodepool conf on a public git repo and I don't want to post secrets required for building RHEL disk images | 11:41 |
| mattymo | Any suggestions? | 11:41 |
| zbr | ianw: can you please abandon https://review.opendev.org/#/c/335520/ ? thanks | 11:57 |
| pabelanger | mattymo: one option is to use ansible to encrypt your secrets and tempate the config | 11:57 |
| pabelanger | mattymo: another is use secure.conf: https://zuul-ci.org/docs/nodepool/installation.html?#configuration | 11:58 |
| *** spsurya has joined #openstack-infra | 11:58 | |
| pabelanger | mattymo: you may also want to join #zuul for nodepool / zuul specific questions too | 11:58 |
| *** rfolco has joined #openstack-infra | 12:06 | |
| mattymo | pabelanger: I looked at the source code for secure.conf. It only supports zookeeper settings. I thought about hacking it, but I don't want to fork if I don't have to. I thought of another dirty hack of adding the secrets to the shell env inside the nodepool container | 12:07 |
| pabelanger | mattymo: what is it you are looking to do? | 12:10 |
| *** pgaxatte has joined #openstack-infra | 12:10 | |
| pabelanger | you likely best to encrypt them using vault from ansible | 12:10 |
| pabelanger | or some cfgmgmt tool in this case | 12:11 |
| mattymo | pabelanger: You're right. I'm going to try that route first | 12:11 |
| *** auristor has quit IRC | 12:11 | |
| pabelanger | mattymo: also look to join #zuul too, there are more humans doing container things, which might know a better way to use secrets from those tools | 12:12 |
| *** lpetrut has joined #openstack-infra | 12:13 | |
| *** ociuhandu has joined #openstack-infra | 12:13 | |
| *** ociuhandu has quit IRC | 12:14 | |
| *** goldyfruit___ has quit IRC | 12:15 | |
| *** derekh has quit IRC | 12:16 | |
| *** ociuhandu has joined #openstack-infra | 12:16 | |
| *** ociuhandu has quit IRC | 12:16 | |
| *** auristor has joined #openstack-infra | 12:17 | |
| *** rlandy has joined #openstack-infra | 12:23 | |
| *** jamesmcarthur has joined #openstack-infra | 12:25 | |
| *** auristor has quit IRC | 12:28 | |
| *** happyhemant has joined #openstack-infra | 12:30 | |
| *** jamesmcarthur has quit IRC | 12:30 | |
| *** auristor has joined #openstack-infra | 12:31 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** auristor has quit IRC | 12:37 | |
| *** ociuhandu has joined #openstack-infra | 12:38 | |
| *** ykarel|afk is now known as ykarel | 12:38 | |
| *** noama has joined #openstack-infra | 12:41 | |
| *** auristor has joined #openstack-infra | 12:44 | |
| *** e0ne has quit IRC | 12:46 | |
| *** mriedem has joined #openstack-infra | 12:49 | |
| *** eharney has joined #openstack-infra | 12:51 | |
| *** e0ne has joined #openstack-infra | 12:52 | |
| *** derekh has joined #openstack-infra | 13:01 | |
| *** Goneri has joined #openstack-infra | 13:08 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 13:11 |
| *** ociuhandu has quit IRC | 13:12 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 13:13 |
| *** ociuhandu has joined #openstack-infra | 13:13 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: add-build-sshkey: add centos/rhel-8 support https://review.opendev.org/674092 | 13:13 |
| *** mahajan-abhishek has left #openstack-infra | 13:14 | |
| *** kjackal has quit IRC | 13:15 | |
| *** kjackal has joined #openstack-infra | 13:18 | |
| *** ociuhandu has quit IRC | 13:18 | |
| *** jaosorior has quit IRC | 13:26 | |
| *** ociuhandu has joined #openstack-infra | 13:26 | |
| *** goldyfruit___ has joined #openstack-infra | 13:29 | |
| *** ociuhandu has quit IRC | 13:31 | |
| *** eharney has quit IRC | 13:32 | |
| *** eharney has joined #openstack-infra | 13:32 | |
| *** sthussey has joined #openstack-infra | 13:40 | |
| *** panda is now known as panda|ruck | 13:40 | |
| *** ociuhandu has joined #openstack-infra | 13:41 | |
| *** aaronsheffield has joined #openstack-infra | 13:42 | |
| *** sshnaidm|bbl is now known as sshnaidm|ruck | 13:44 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 13:44 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: add-build-sshkey: add centos/rhel-8 support https://review.opendev.org/674092 | 13:44 |
| *** panda|ruck is now known as panda|rover | 13:45 | |
| zbr | is the success-url and failure-url are going away or that is only a temporary issue? asking so I know if i adopt the tox-docs approach in order places or not. | 13:48 |
| *** redrobot has joined #openstack-infra | 13:50 | |
| AJaeger | zbr: my expectation: success-url will work while a buildset is running and work for non-swift logs. But discuss with corvus | 13:50 |
| *** rpittau is now known as rpittau|afk | 13:54 | |
| zbr | corvus: pabelanger: please help me merge https://review.opendev.org/#/c/681251/ to fix broken urls on tox-molecule jobs. | 13:55 |
| *** nicolasbock has quit IRC | 14:10 | |
| *** nicolasbock has joined #openstack-infra | 14:11 | |
| *** nicolasbock has quit IRC | 14:11 | |
| *** nicolasbock has joined #openstack-infra | 14:13 | |
| *** diablo_rojo has joined #openstack-infra | 14:15 | |
| *** nicolasbock has quit IRC | 14:15 | |
| *** ricolin has joined #openstack-infra | 14:18 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Rename x/ansible-role-cloud-launcher -> opendev/ https://review.opendev.org/662530 | 14:22 |
| *** electrofelix has joined #openstack-infra | 14:26 | |
| *** mattymo has quit IRC | 14:32 | |
| *** priteau has quit IRC | 14:34 | |
| *** ykarel is now known as ykarel|away | 14:37 | |
| *** nhicher has quit IRC | 14:39 | |
| openstackgerrit | Graham Hayes proposed openstack/project-config master: Allow all TC members +W access https://review.opendev.org/678214 | 14:39 |
| *** sshnaidm|ruck is now known as sshnaidm|rover | 14:40 | |
| mriedem | clarkb: fyi if you see ssh failures in tempest runs, i've been noticing this in the guest console log output: | 14:41 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22sh%3A%20write%20error%3A%20No%20space%20left%20on%20device%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | 14:41 |
| mriedem | i can't say for sure that's the reason for the ssh failure though because the log goes on to show cloud-init got the network info | 14:41 |
| *** ykarel|away has quit IRC | 14:46 | |
| *** panda|rover is now known as panda|ruck | 14:52 | |
| *** jbadiapa has quit IRC | 14:54 | |
| *** pgaxatte has quit IRC | 14:54 | |
| *** nhicher has joined #openstack-infra | 14:56 | |
| *** prometheanfire has quit IRC | 14:57 | |
| *** ykarel has joined #openstack-infra | 15:03 | |
| *** jamesmcarthur has joined #openstack-infra | 15:08 | |
| *** chandankumar has quit IRC | 15:11 | |
| *** chandankumar has joined #openstack-infra | 15:12 | |
| *** gyee has joined #openstack-infra | 15:23 | |
| *** bdodd has joined #openstack-infra | 15:23 | |
| *** igordc has joined #openstack-infra | 15:31 | |
| *** mattw4 has joined #openstack-infra | 15:34 | |
| clarkb | pabelanger: mattymo ya an important distinction here is that zuul is managing secrets for the jobs it runs not for zuul's own configs | 15:37 |
| *** eharney has quit IRC | 15:37 | |
| clarkb | in the nodepool case if you need to write secrets to the config I think configuration management tooling is what you want to rely on | 15:37 |
| pabelanger | ++ | 15:38 |
| clarkb | mriedem: I think that can be a few potential issues: 1) growroot or equivalent isn't working to expand the image size on boot 2) the image flavor disk size is too small for the workload being tested on the image 3) we are running out of disk space in /var/lib/libvirt/images (or whatever that path was) like ironic jobs are doing | 15:39 |
| clarkb | mriedem: considering that we are able to have ansible ssh in and collect logs I think 3) is unlikely as that is usually more catastrophic but could still be possible | 15:40 |
| clarkb | mriedem: checking 2) should be easy. checking 1) will depend on the image and whether or not we collect console logs | 15:40 |
| clarkb | corvus: fungi canI get a review on https://review.opendev.org/#/c/681354/ so that I can test only collecting last effort debug logs on job failures? | 15:41 |
| *** e0ne has quit IRC | 15:42 | |
| *** e0ne has joined #openstack-infra | 15:42 | |
| *** jaosorior has joined #openstack-infra | 15:44 | |
| clarkb | mriedem: 'GROWROOT: NOCHANGE: partition 1 is size 2078687. it cannot be grown' | 15:44 |
| clarkb | mriedem: I think that means the image is already as big as it can be based on flavor and image size? | 15:44 |
| clarkb | is that 2MB? | 15:45 |
| clarkb | so ya maybe we bump the flavor size up to 50MB or something | 15:45 |
| clarkb | oh its volume backed so that must be 2GB not 2MB? | 15:49 |
| *** eharney has joined #openstack-infra | 15:50 | |
| *** ykarel is now known as ykarel|away | 15:51 | |
| *** Garyx has quit IRC | 15:53 | |
| *** Garyx has joined #openstack-infra | 15:54 | |
| clarkb | ok we create a volume with the cirros image with a size of '1'. I think that means the volume itself is 1GB large | 15:54 |
| mriedem | the default volume size in tempest runs is 1gb | 15:56 |
| clarkb | mriedem: ya so now I wonder where the size in the growroot error comes from | 15:57 |
| clarkb | it should be in the range of 1073741824 | 15:58 |
| clarkb | or 1073741 if the units are kb | 15:58 |
| *** diablo_rojo has quit IRC | 15:59 | |
| *** diablo_rojo has joined #openstack-infra | 15:59 | |
| clarkb | mriedem: I'm going to guess that cirros issue is locally reproduceable unless it is a hypervisor running out of disk | 16:00 |
| clarkb | https://9fb8b67e1f094c1ca6a3-8e4483e68b70671d63c5bd54e8baf97f.ssl.cf5.rackcdn.com/628076/4/check/cinder-tempest-dsvm-lvm-lio-barbican/125dba2/logs/df.txt.gz at least at the end of the job we have plenty of disk | 16:00 |
| openstackgerrit | Merged opendev/base-jobs master: Update cleanup tasks to only happen on failure https://review.opendev.org/681354 | 16:01 |
| clarkb | fungi: thanks re ^ | 16:01 |
| mugsie | anyone able to give an ACL change a +W? - https://review.opendev.org/#/c/678214/ | 16:02 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22GROWROOT%3A%20NOCHANGE%3A%20partition%201%20is%20size%5C%22%20AND%20message%3A%5C%22it%20cannot%20be%20grown%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | 16:03 |
| *** mattw4 has quit IRC | 16:04 | |
| *** mattw4 has joined #openstack-infra | 16:04 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix evaluating nodepool_ip and switch_ip facts https://review.opendev.org/681182 | 16:05 |
| clarkb | mriedem: it is also entirely possible those errors happen on successful jobs too we just don't collect console logs in that case | 16:06 |
| clarkb | that said I think removing errors even if benign is worthwhile as it removes distractions when debugging | 16:06 |
| mriedem | clarkb: true | 16:06 |
| mriedem | i wonder if it would be useful to have tempest ssh and df and dump that out when we get an ssh failure? | 16:07 |
| mriedem | since the console output shows the disks but not their usage | 16:07 |
| AJaeger | mugsie: why did you rebase that change? That is completely unneeded unless they are merge conflicts - and only wastes test resources that are rare at the momemnt with all the feature freeze rush... | 16:08 |
| clarkb | mriedem: if we have an ssh failure we may not be able to ssh in :) | 16:08 |
| AJaeger | (a single one does not hurt - just wanted to point it out before it happens large scale ;) | 16:08 |
| clarkb | mriedem: but ya I think figuring that out would be good | 16:08 |
| mriedem | clarkb: heh right | 16:08 |
| mriedem | regardless i guess i should open a new bug for this and track it in e-r | 16:09 |
| mugsie | AJaeger: habit - I went to double check it would still rebase cleanly, and just g-r'd it | 16:09 |
| clarkb | mugsie: note that git review will bail out if a rebase is necessary | 16:09 |
| clarkb | though if you aren't making changes then it will also error because no changes | 16:10 |
| mugsie | clarkb: I know, it just have a habit of git fetch, git rebase, git review for older patches | 16:10 |
| *** dtantsur is now known as dtantsur|afk | 16:10 | |
| clarkb | we try really hard to tell you when a rebase is necessary so that you only do it then :) | 16:10 |
| *** mattw4 has quit IRC | 16:10 | |
| clarkb | gerrit, zuul and git review will all tell you | 16:10 |
| mugsie | yeah, unfortunately most of my work these days is outside of zuul, gerrit, and other good CI/CD toolchains | 16:11 |
| fungi | also git-restack is good for avoiding unnecessary rebasing onto different parents when updating a change series | 16:11 |
| fungi | (related) | 16:11 |
| mriedem | clarkb: could this be closer to the reason for the ssh fail? | 16:13 |
| mriedem | ip-route6:unreachable default dev lo metric -1 error -101 | 16:13 |
| clarkb | mriedem: is https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_c4c/671072/18/gate/nova-tox-functional/c4ca604/job-output.txt the functional test problem you were saying has a fix? it just reset the gate | 16:13 |
| mriedem | though i guess in this test it's trying to ssh into an ipv4 address | 16:13 |
| clarkb | mriedem: ya I think that is only a problem if using ipv6 which that test may not be | 16:13 |
| mriedem | clarkb: i noticed TestInstanceNotificationSampleWithMultipleCompute.test_multiple_compute_actions on another change this morning, it's not related to the one i pointed out yesterday (which is merged now), | 16:14 |
| mriedem | so i suspect something new has merged which is tickling a new failure, | 16:14 |
| mriedem | i'll e-r that one after i get this growroot one | 16:14 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for growroot fail bug 1843610 https://review.opendev.org/681527 | 16:16 |
| openstack | bug 1843610 in OpenStack-Gate "Tempest ssh to guest intermittently fails, "GROWROOT: NOCHANGE: partition 1 is size 2078687. it cannot be grown" seen in guest console log" [Undecided,New] https://launchpad.net/bugs/1843610 | 16:16 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Allow ensure-tox to upgrade tox version https://review.opendev.org/676464 | 16:16 |
| openstackgerrit | Merged openstack/project-config master: Allow all TC members +W access https://review.opendev.org/678214 | 16:17 |
| *** ricolin has quit IRC | 16:22 | |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for test_multiple_compute_actions race fail bug 1843615 https://review.opendev.org/681530 | 16:27 |
| mriedem | clarkb: ^ | 16:27 |
| openstack | bug 1843615 in OpenStack Compute (nova) "TestInstanceNotificationSampleWithMultipleCompute.test_multiple_compute_actions intermittently failing since Sept 10, 2019" [High,Confirmed] https://launchpad.net/bugs/1843615 | 16:27 |
| *** ociuhandu has quit IRC | 16:30 | |
| *** chandankumar is now known as raukadah | 16:31 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: For pre blocks to wrap text https://review.opendev.org/681532 | 16:32 |
| *** ralonsoh has quit IRC | 16:37 | |
| *** lpetrut has quit IRC | 16:40 | |
| *** hwoarang has quit IRC | 16:42 | |
| *** jpena is now known as jpena|off | 16:44 | |
| *** hwoarang has joined #openstack-infra | 16:45 | |
| *** jaosorior has quit IRC | 16:46 | |
| *** ociuhandu has joined #openstack-infra | 16:46 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: openstack-tox-molecule: replace success-url and failure-url https://review.opendev.org/681251 | 16:47 |
| clarkb | corvus: we don't seem to set zuul_success on the cleanup playbook http://paste.openstack.org/show/775150/ | 16:49 |
| *** ykarel|away has quit IRC | 16:52 | |
| *** e0ne has quit IRC | 16:54 | |
| corvus | clarkb: good thing we tested :/ | 16:54 |
| clarkb | Isee why now, working on a zuul change | 16:54 |
| *** tesseract has quit IRC | 16:57 | |
| *** mriedem is now known as gibi_zeppelin | 16:58 | |
| *** gibi_zeppelin is now known as gibi_submarine | 16:58 | |
| *** gibi_submarine is now known as mriedem | 16:58 | |
| *** derekh has quit IRC | 17:00 | |
| *** eernst has joined #openstack-infra | 17:01 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for growroot fail bug 1843610 https://review.opendev.org/681527 | 17:02 |
| openstack | bug 1843610 in OpenStack-Gate "Tempest ssh to guest intermittently fails, "GROWROOT: NOCHANGE: partition 1 is size 2078687. it cannot be grown" seen in guest console log" [Undecided,New] https://launchpad.net/bugs/1843610 | 17:02 |
| clarkb | running test now to confirm a fix | 17:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Add support for the Gerrit checks plugin https://review.opendev.org/680778 | 17:06 |
| openstackgerrit | Paul Belanger proposed zuul/nodepool master: Disable interface_ip check, when host-key-checking is disable https://review.opendev.org/681544 | 17:06 |
| *** ociuhandu has quit IRC | 17:08 | |
| *** ykarel|away has joined #openstack-infra | 17:08 | |
| *** gfidente has quit IRC | 17:13 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for test_multiple_compute_actions race fail bug 1843615 https://review.opendev.org/681530 | 17:15 |
| openstack | bug 1843615 in OpenStack Compute (nova) "TestInstanceNotificationSampleWithMultipleCompute.test_multiple_compute_actions intermittently failing since Sept 10, 2019" [High,In progress] https://launchpad.net/bugs/1843615 - Assigned to Matt Riedemann (mriedem) | 17:15 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: For pre blocks to wrap text https://review.opendev.org/681532 | 17:19 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Pass zuul_success to cleanup playbooks https://review.opendev.org/681552 | 17:21 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: For pre blocks to wrap text https://review.opendev.org/681532 | 17:21 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: synchronize: add support for kubectl connection https://review.opendev.org/681553 | 17:22 |
| openstackgerrit | Merged openstack/project-config master: New project request: airship/images https://review.opendev.org/677944 | 17:26 |
| *** ociuhandu has joined #openstack-infra | 17:27 | |
| *** sshnaidm|rover is now known as sshnaidm|off | 17:28 | |
| *** ociuhandu has quit IRC | 17:31 | |
| openstackgerrit | Paul Belanger proposed zuul/nodepool master: Disable interface_ip check, when host-key-checking is disable https://review.opendev.org/681544 | 17:32 |
| *** spsurya has quit IRC | 17:32 | |
| zbr | clarkb: if you can wf https://review.opendev.org/#/c/680962/ it would be great. | 17:37 |
| *** kopecmartin is now known as kopecmartin|off | 17:38 | |
| openstackgerrit | Merged zuul/zuul master: Fix timestamp race occurring on fast systems https://review.opendev.org/680937 | 17:44 |
| clarkb | zbr: why do we not need to mock platform.system to match 'Darwin' in the darwin case? | 17:46 |
| clarkb | that is what we check against in depends.py | 17:46 |
| zbr | clarkb: see https://opendev.org/opendev/bindep/src/branch/master/bindep/depends.py#L307-L314 | 17:48 |
| zbr | mainly we have a very different approach. | 17:48 |
| zbr | clarkb: but probably we could mock more and avoid that conditionals | 17:48 |
| clarkb | zbr: ya that code only runs if platform.system == Darwin but we never set that I don't think | 17:48 |
| zbr | i cannot say I liked it in particular. | 17:49 |
| clarkb | what sets platform.system to Darwin is my question I gess ( the change sets it to Linux if testing linux but I don't see that for darwin) | 17:49 |
| zbr | clarkb: if you want, make a comment and I will try to rework the code tomorrow. | 17:49 |
| clarkb | I don't know that we need to rework the code. I'm just trying to undersatnd it | 17:50 |
| clarkb | the test passes which means something must set platform.system to darwin right? | 17:50 |
| zbr | clarkb: because platform.system was not always mocked, when you would run unittest tests on Darwin, they would fail | 17:51 |
| clarkb | right I get that part. but for the test to work something must set platform.system to Darwin. What does that | 17:51 |
| zbr | sorry i need to go, family duty | 17:52 |
| clarkb | oh I see it now | 17:52 |
| clarkb | we haev two mocks happening and they are nested | 17:52 |
| clarkb | my brain read it as a single one | 17:52 |
| *** e0ne has joined #openstack-infra | 17:55 | |
| *** electrofelix has quit IRC | 17:56 | |
| *** e0ne has quit IRC | 17:59 | |
| *** ramishra has quit IRC | 18:00 | |
| *** ykarel|away has quit IRC | 18:00 | |
| *** ykarel|away has joined #openstack-infra | 18:00 | |
| *** jamesmcarthur has quit IRC | 18:06 | |
| *** jamesmcarthur has joined #openstack-infra | 18:07 | |
| *** jamesmcarthur has quit IRC | 18:11 | |
| *** Garyx has quit IRC | 18:12 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: FN needs to give a little more juice https://review.opendev.org/681569 | 18:16 |
| *** Garyx has joined #openstack-infra | 18:16 | |
| *** e0ne has joined #openstack-infra | 18:17 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Add support for the Gerrit checks plugin https://review.opendev.org/680778 | 18:18 |
| *** panda|ruck is now known as panda|ruck|off | 18:21 | |
| openstackgerrit | Merged opendev/bindep master: Fix test execution failure on Darwin https://review.opendev.org/680962 | 18:21 |
| donnyd | I am gonna watch the utilization, if all is going well I am gonna push FN to it's limits so hopefully people have to wait less on jobs | 18:24 |
| *** jbadiapa has joined #openstack-infra | 18:27 | |
| *** ykarel|away has quit IRC | 18:28 | |
| openstackgerrit | Merged openstack/project-config master: FN needs to give a little more juice https://review.opendev.org/681569 | 18:34 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Add several missing ssldomains to certcheck config https://review.opendev.org/681570 | 18:36 |
| fungi | clarkb: ^ | 18:36 |
| fungi | i spotted this because namecheap sent me a warning about the docs.starlingx.io cert coming up for renewal and on a whim i double-checked it | 18:37 |
| fungi | expires in a few weeks, which got me wondering why i hadn't seen any warnings from certcheck about it... and that was why | 18:39 |
| clarkb | fungi: thanks | 18:48 |
| *** pkopec has quit IRC | 18:51 | |
| *** eharney has quit IRC | 18:55 | |
| *** kjackal has quit IRC | 18:57 | |
| AJaeger | donnyd: thanks! | 19:01 |
| donnyd | :) | 19:01 |
| donnyd | hopefully it has a bit more to give.. | 19:02 |
| AJaeger | clarkb, fungi, want to +2A 681569 to give us some more nodes? | 19:02 |
| AJaeger | oh, fungi did already.. Thanks | 19:02 |
| AJaeger | and it's merged - mea culpa | 19:02 |
| donnyd | It hasn't actually started scheduling on it yet though | 19:02 |
| AJaeger | donnyd: it first needs deployment run to change our config, that can take an hour | 19:03 |
| donnyd | ah i see | 19:04 |
| AJaeger | but looking at http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 we have 20 nodes more already | 19:05 |
| AJaeger | also visible here: http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1 | 19:05 |
| AJaeger | quota problem? | 19:05 |
| AJaeger | donnyd: we don't reach more than 73 nodes | 19:06 |
| donnyd | nah | 19:07 |
| donnyd | quota is set at 100 | 19:07 |
| paladox | corvus another thing to look out for is https://bugs.chromium.org/p/gerrit/issues/detail?id=7645 (i'm not sure if it affects 2.13) but does 2.14+ | 19:07 |
| fungi | AJaeger: donnyd: i haven't looked at nodepool, but if there are quotas around processors, ram or disk those could also cause it not to boot additional nodes | 19:08 |
| donnyd | i think that grafana may lag a little behind reality | 19:08 |
| fungi | also that, for sure | 19:08 |
| paladox | we are being heavily affected by that issue at wikimedia :( (had gerrit go down three times because of it) | 19:08 |
| *** eharney has joined #openstack-infra | 19:08 | |
| donnyd | Oh i disabled those in the beginning | 19:08 |
| paladox | *today | 19:08 |
| donnyd | oh well i also guess i should put all my hypervisors back in service | 19:11 |
| donnyd | oh well i also guess i should put all my hypervisors back in service | 19:12 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix secret for promote-tox-docs-special-base https://review.opendev.org/681579 | 19:12 |
| AJaeger | clarkb, fungi, sorry, another wrong name, this breaks infra-manual ^ | 19:12 |
| donnyd | should be able to get about 16 more jobs out of it | 19:12 |
| donnyd | nope... grafana is accurate | 19:14 |
| donnyd | http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=1568227429047&to=1568229489128 | 19:18 |
| donnyd | been almost 30 minutes.. so i guess we must be waiting on something else maybe | 19:18 |
| clarkb | donnyd: its set to max-servers: 90 in the config now | 19:22 |
| clarkb | oh I see the max servers value went up but the actual used values did not. ya quota is likely to blame | 19:24 |
| openstackgerrit | Merged opendev/system-config master: Add several missing ssldomains to certcheck config https://review.opendev.org/681570 | 19:25 |
| donnyd | ugg... i suck... must have turned it down for something or other | 19:26 |
| donnyd | ok its set back to 120 where it should be | 19:26 |
| *** bdodd has quit IRC | 19:27 | |
| *** pgaxatte has joined #openstack-infra | 19:33 | |
| *** pgaxatte has quit IRC | 19:36 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add base promote job for moving static.o.o https://review.opendev.org/681582 | 19:44 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Switch governance sites to AFS publishing https://review.opendev.org/681583 | 19:44 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Move security site to AFS publishing https://review.opendev.org/681584 | 19:44 |
| *** rh-jelabarre has quit IRC | 19:44 | |
| AJaeger | ianw: that should get us started with static publishing. The first job can merge directly, the others for each site. Is that what you have in mind? | 19:45 |
| clarkb | donnyd: http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1 shows the expected jump in usage now | 19:45 |
| donnyd | yea now its rolling along | 19:45 |
| *** eharney has quit IRC | 19:45 | |
| clarkb | corvus: can you review https://review.opendev.org/#/c/681552/ if that looks sane I may hold off on implementing the cleanup playbook until after that gets in and we restart exectuors | 19:46 |
| donnyd | I am going to add the last hypervisor back in, the other testing can wait | 19:46 |
| clarkb | if not then maybe we'll just run cleanup on every change again? | 19:46 |
| donnyd | then then juice it up a little more | 19:46 |
| *** rh-jelabarre has joined #openstack-infra | 19:48 | |
| *** eernst has quit IRC | 19:49 | |
| clarkb | also worth noting that while adding resources definitely helps with the backlog, much of that aid is negated if we keep having a flaky gate | 19:50 |
| clarkb | if anyone is able to help with that I can give direction but I can't really devote time to debugging why cinder tests fail or nova etc | 19:51 |
| *** eernst has joined #openstack-infra | 19:52 | |
| *** panda|ruck|off has quit IRC | 19:55 | |
| *** panda has joined #openstack-infra | 19:57 | |
| AJaeger | config-core, please review https://review.opendev.org/681579 to fix publishing for infra-manual | 19:58 |
| donnyd | clarkb: yea that makes sense | 19:59 |
| donnyd | If you look at my grafana I don't push FN too hard because i don't want to see failing jobs from timeouts and what not | 20:00 |
| *** eernst has quit IRC | 20:05 | |
| *** jamesmcarthur has joined #openstack-infra | 20:06 | |
| *** e0ne has quit IRC | 20:06 | |
| clarkb | donnyd: ya not saying you should spin it up more. Mostly want to make the projects consuming the resources to understand the best thing they can do to make the situation better is improve reliability of the jobs | 20:13 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add enqueue reporter action https://review.opendev.org/681132 | 20:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add no-jobs reporter action https://review.opendev.org/681278 | 20:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add report time to item model https://review.opendev.org/681323 | 20:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add Item.formatStatusUrl https://review.opendev.org/681324 | 20:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add support for the Gerrit checks plugin https://review.opendev.org/680778 | 20:15 |
| *** e0ne has joined #openstack-infra | 20:15 | |
| *** e0ne has quit IRC | 20:16 | |
| *** e0ne has joined #openstack-infra | 20:16 | |
| *** eernst has joined #openstack-infra | 20:17 | |
| corvus | clarkb: looks legit | 20:17 |
| clarkb | corvus: http://paste.openstack.org/show/775158/ is that a known swift problem? | 20:18 |
| *** eernst has quit IRC | 20:18 | |
| clarkb | also thank you for the review | 20:18 |
| clarkb | corvus: thats the create a container step right? | 20:19 |
| corvus | clarkb: i'm not sure i can tell what request failed from that | 20:20 |
| corvus | is there a traceback or anything? | 20:20 |
| clarkb | corvus: no traceback, was just going off of the url | 20:20 |
| clarkb | fwiw that container exists and has 617 objects in it according to osc | 20:21 |
| corvus | so probably not the create step | 20:21 |
| corvus | clarkb: could be https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/upload-logs-swift/library/zuul_swift_upload.py#L511 ? | 20:23 |
| corvus | that lacks a retry | 20:24 |
| openstackgerrit | Merged openstack/project-config master: Fix secret for promote-tox-docs-special-base https://review.opendev.org/681579 | 20:26 |
| *** markvoelker has quit IRC | 20:27 | |
| clarkb | corvus: maybe? but it looks like if that fails it is expecting to create the container? | 20:28 |
| corvus | clarkb: well, it got a 401 not a 404, so that may be an unhandled exception | 20:28 |
| corvus | (i'm not sure, but seems plausible) | 20:29 |
| *** e0ne has quit IRC | 20:29 | |
| *** e0ne_ has joined #openstack-infra | 20:29 | |
| clarkb | ah | 20:29 |
| clarkb | and 401 is what swift uses to say this isn't consistent yet right | 20:29 |
| clarkb | timburke: ^ is my memory off there? | 20:29 |
| timburke | 401 means i don't know who you are. typically, invalid token | 20:30 |
| pabelanger | clarkb: make sure there isn't another swift container owned by somebody else? | 20:30 |
| timburke | not sure how you're uploading, but could also be an expired or otherwise invalid tempurl | 20:31 |
| corvus | pabelanger: this is rax swift (so more swiftlike than ceph) | 20:31 |
| clarkb | pabelanger: thats not a problem with swift aiui | 20:31 |
| pabelanger | kk | 20:31 |
| corvus | this is not using tempurl | 20:32 |
| clarkb | I'm going to create a test object in that container | 20:33 |
| donnyd | clarkb: is it just a coincidence that 16 jobs all finished at the same time??? | 20:35 |
| donnyd | i should have looked a little closer 32 | 20:35 |
| corvus | donnyd: they may have been aborted | 20:35 |
| clarkb | corvus: I am able to create an object in that container from bridge | 20:35 |
| donnyd | ok... just want to make sure FN working correctly | 20:35 |
| *** markvoelker has joined #openstack-infra | 20:36 | |
| *** kjackal has joined #openstack-infra | 20:37 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: Also include nodepool inventory variables https://review.opendev.org/681601 | 20:39 |
| *** factor has quit IRC | 20:40 | |
| *** markvoelker has quit IRC | 20:41 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-javascript-tarball: introduce zuul_do_synchronize https://review.opendev.org/681603 | 20:41 |
| *** factor has joined #openstack-infra | 20:42 | |
| *** rh-jelabarre has quit IRC | 20:43 | |
| *** xenos76 has quit IRC | 20:47 | |
| *** ociuhandu has joined #openstack-infra | 20:50 | |
| *** exsdev has quit IRC | 20:51 | |
| *** rh-jelabarre has joined #openstack-infra | 20:52 | |
| *** exsdev has joined #openstack-infra | 20:53 | |
| *** ociuhandu has quit IRC | 20:57 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: fetch-javascript-tarball: introduce zuul_do_synchronize https://review.opendev.org/681603 | 21:02 |
| *** slaweq has quit IRC | 21:10 | |
| *** slaweq has joined #openstack-infra | 21:11 | |
| *** rcernin has joined #openstack-infra | 21:15 | |
| *** slaweq has quit IRC | 21:18 | |
| *** jamesmcarthur has quit IRC | 21:20 | |
| mriedem | clarkb: fyi in case you need to promote anything again https://review.opendev.org/#/c/681540 fixes one of the known gate race reset bugs | 21:24 |
| mriedem | the one you pointed out to me this morning | 21:24 |
| mriedem | i just saw something else fail on it | 21:24 |
| *** prometheanfire has joined #openstack-infra | 21:26 | |
| *** noama has quit IRC | 21:26 | |
| *** slaweq has joined #openstack-infra | 21:27 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: FN can bump up another 20 https://review.opendev.org/681615 | 21:28 |
| donnyd | things don't look too bad and i just brought the other hypervisor online. It seems to be running the instances ok, so up 20 more | 21:29 |
| *** slaweq has quit IRC | 21:32 | |
| *** goldyfruit_ has joined #openstack-infra | 21:32 | |
| guilhermesp | infra-root: Is it possible to get a hold on openstack-ansible-deploy-aio_metal-debian-stable over project openstack/openstack-ansible (review #670051 ).... btw, keys https://github.com/guilhermesteinmuller.keys | 21:33 |
| guilhermesp | thanks in advance! | 21:33 |
| clarkb | mriedem: I have enqueued it | 21:33 |
| clarkb | guilhermesp: yes I can request the hold now | 21:34 |
| guilhermesp | thanks clarkb ! let me know when I can recheck it | 21:34 |
| *** goldyfruit___ has quit IRC | 21:34 | |
| clarkb | guilhermesp: what can I put in the notes as far as the purpose (helps us to figure out when we can delte the node) | 21:34 |
| guilhermesp | oh ok | 21:36 |
| clarkb | corvus: re the swift thing I'm guessing we don't print full traceback to avoid leaking data? | 21:36 |
| clarkb | corvus: I'm thinking that might be the next step in debugging is getting that info | 21:36 |
| clarkb | guilhermesp: just a short thing like "debugging osa bug foo" or whatever | 21:36 |
| guilhermesp | investigate galera_new_cluster command that is trying to connect to database with different user then localhost/127.0.0.1 | 21:37 |
| clarkb | thanks | 21:37 |
| guilhermesp | btw, I can tell you too when Im done with the node as usual | 21:37 |
| clarkb | it is in place and yes you should still tell us :) we add the notes so that if we miss you or whatever we have enough info to make a judgement call | 21:38 |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Add cleanup playbook to all base jobs https://review.opendev.org/681322 | 21:39 |
| *** e0ne_ has quit IRC | 21:41 | |
| *** kjackal has quit IRC | 21:45 | |
| openstackgerrit | Merged openstack/project-config master: FN can bump up another 20 https://review.opendev.org/681615 | 21:51 |
| *** mriedem is now known as mriedem_afk | 21:53 | |
| ianw | AJaeger: thanks will look today | 21:55 |
| *** markvoelker has joined #openstack-infra | 21:59 | |
| *** panda has quit IRC | 22:00 | |
| *** panda has joined #openstack-infra | 22:03 | |
| guilhermesp | cool clarkb | 22:06 |
| guilhermesp | so could I recheck it now? | 22:06 |
| clarkb | ya should be ready for you to trigger the job | 22:07 |
| guilhermesp | thanks clarkb ! rechecked | 22:07 |
| clarkb | Im out running errands for the next little bit but can check in after to see if we have caight one yet | 22:07 |
| *** slaweq has joined #openstack-infra | 22:11 | |
| *** whoami-rajat has quit IRC | 22:11 | |
| *** slaweq has quit IRC | 22:16 | |
| rm_work | fungi: do you know if Matthias is on IRC? I don't know his nick | 22:20 |
| rm_work | I didn't mean to start a whole debate about this thing, I just want my patch merged, lol | 22:21 |
| fungi | rm_work: yeah, i didn't mean to imply there was no value in patching it, but it's up to the horizon maintainers to decide, not the vmt | 22:23 |
| *** rlandy is now known as rlandy|bbl | 22:24 | |
| fungi | rm_work: and yeah, i don't know for sure whether he hangs out in irc much. i want to say i've seen him on occasionally as "mrunge" but i don't see him in #openstack-horizon right now nor in recent channel history there | 22:25 |
| corvus | clarkb: i think we should be able to print the full traceback; we probably just need to work out the best way to do that in an ansible module. | 22:27 |
| rm_work | fungi: yeah I just shouldn't have checked the security box when i submitted the bug | 22:27 |
| rm_work | i didn't even submit it until after he told me i needed a LP# on my patch | 22:27 |
| rm_work | >_< | 22:27 |
| rm_work | and I already had a +2 on the patch once... | 22:27 |
| donnyd | fungi: I have 10 more instances I can give over to the CI, but I want to watch it for a little bit | 22:32 |
| fungi | donnyd: sounds great! also, we should figure out how to get you listed at https://www.openstack.org/foundation/companies/#infra-donors now that you're providing in excess of 100 test nodes | 22:34 |
| *** threestrands has joined #openstack-infra | 22:37 | |
| *** jcoufal has joined #openstack-infra | 22:38 | |
| cmurphy | proposed another timeout increase for keystone https://review.opendev.org/681621 we have a better fix ready to go but it conflicts with everything trying to make it through the gate right now so would rather wait on it | 22:40 |
| donnyd | fungi: That would be pretty cool | 22:46 |
| fungi | donnyd: "figure out" in this sense entails figuring out how you'd want to be displayed (name/logo), the actual logistics are mainly that we ask the osf website admins to stick it on there and they do. not terribly complicated | 22:48 |
| *** mtreinish has quit IRC | 22:54 | |
| *** dchen has joined #openstack-infra | 22:55 | |
| *** goldyfruit_ has quit IRC | 22:57 | |
| *** tkajinam has joined #openstack-infra | 23:04 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** slaweq has quit IRC | 23:15 | |
| *** igordc has quit IRC | 23:18 | |
| *** jamesmcarthur has joined #openstack-infra | 23:32 | |
| *** igordc has joined #openstack-infra | 23:33 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add support for the Gerrit checks plugin https://review.opendev.org/680778 | 23:34 |
| *** exsdev has quit IRC | 23:40 | |
| *** igordc has quit IRC | 23:40 | |
| *** exsdev has joined #openstack-infra | 23:42 | |
| *** igordc has joined #openstack-infra | 23:42 | |
| *** sthussey has quit IRC | 23:42 | |
| *** jamesmcarthur has quit IRC | 23:46 | |
| *** jamesmcarthur has joined #openstack-infra | 23:46 | |
| *** mriedem_afk has quit IRC | 23:49 | |
| *** jamesmcarthur has quit IRC | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!