| *** jamesmcarthur has quit IRC | 00:06 | |
| *** dychen has joined #openstack-infra | 00:24 | |
| clarkb | fungi: I've noticed https://gitea01.opendev.org:3000/openstack/election/commits/branch/master is not up to date | 00:36 |
|---|---|---|
| clarkb | 05 06 and 07 seem to share this state | 00:36 |
| clarkb | the others are fine | 00:37 |
| clarkb | I've run repliations against 01 but that didn't help | 00:37 |
| clarkb | I'll have to dig out my root ssh key to do more | 00:37 |
| *** xenos76 has quit IRC | 00:39 | |
| clarkb | I'm going to guess stale lock files again | 00:41 |
| *** ccamacho has quit IRC | 00:42 | |
| clarkb | I think we should update our docker images to rm that file on startup | 00:43 |
| fungi | thanks for the heads up | 00:44 |
| clarkb | I've got my key now and am checking my theory | 00:44 |
| fungi | i'm unfortunately not in a situation where i can help, sorry :/ | 00:45 |
| ianw | i can take al ook | 00:45 |
| *** markvoelker has joined #openstack-infra | 00:46 | |
| clarkb | ya on 01 there is a lockfile since 17:29 sept 2 | 00:46 |
| clarkb | not sure yet if it is safe to rm it | 00:46 |
| clarkb | looking at logs | 00:46 |
| *** gregoryo has joined #openstack-infra | 00:47 | |
| clarkb | that timestamp on the file coincides with journald being restarted according to dmesg -T | 00:49 |
| ianw | ahh yeah Sep 2 17:29 /var/gitea/data/git/.gitconfig.lock | 00:49 |
| clarkb | package updates maybe? I think we probably can rm that file safely given the dmesg output | 00:50 |
| ianw | is that around an auto update? | 00:50 |
| ianw | Start-Date: 2019-09-02 06:55:47 | 00:51 |
| *** markvoelker has quit IRC | 00:51 | |
| ianw | Upgrade: libsystemd0:amd64 (237-3ubuntu10.25, 237-3ubuntu10.26), libpam-systemd:amd64 (237-3ubuntu10.25, 237-3ubuntu10.26), systemd:amd64 (237-3ubuntu10.25, 237-3ubuntu10.26), libnss-systemd:amd64 (237-3ubuntu10.25, 237-3ubuntu10.26) | 00:51 |
| clarkb | hrm thats almost half a day away | 00:51 |
| ianw | [Mon Sep 2 06:54:24 2019] systemd[1]: Starting Journal Service... | 00:51 |
| ianw | [Mon Sep 2 06:54:24 2019] systemd[1]: Started Journal Service. | 00:51 |
| clarkb | neither gitea nor sshd have been restarted in a number of days so they didn't crash | 00:52 |
| clarkb | there appear to only be defunct git processes running | 00:53 |
| clarkb | I think it is safe to remove that file | 00:53 |
| clarkb | we could also stop gitea, rm the file, then start gitea | 00:53 |
| clarkb | I guess lsof would tell us if anything has that file open | 00:53 |
| ianw | https://bugzilla.redhat.com/show_bug.cgi?id=1270616 ... journald self-destructs when system hibernates, or running 'git gc' | 00:53 |
| openstack | bugzilla.redhat.com bug 1270616 in systemd "journald self-destructs when system hibernates, or running 'git gc'" [Unspecified,Closed: eol] - Assigned to systemd-maint | 00:53 |
| clarkb | we don't have lsof installed | 00:53 |
| ianw | [Mon Sep 2 17:18:40 2019] systemd[1]: systemd-journald.service: State 'stop-sigabrt' timed out. Terminating. | 00:53 |
| ianw | "This is known issue. If your I/O is painfully slow and journald is blocked in some system call waiting for I/O request to complete then it can miss to send watchdog ping and will be SIGABRT'ed by systemd." | 00:54 |
| clarkb | or running git gc | 00:54 |
| ianw | this sounds very likley | 00:54 |
| *** onovy has quit IRC | 00:54 | |
| clarkb | huh | 00:54 |
| clarkb | so maybe that file was left around because something else was also unhappy about hte available IO? | 00:54 |
| ianw | or timed out similarly, yeah | 00:55 |
| clarkb | ianw: do you want to rm the file (and check 05, 06, 07) or should I? | 00:56 |
| ianw | i can, i'm just trying to see what the last thing in the logs is | 00:56 |
| clarkb | great thanks. Probably want to trigger replication for gitea01 05 06 07 too: replication start --url gitea01 for example afterwards | 00:57 |
| ianw | clarkb: you think just remove the file, no need to drop from lb and restart? | 00:59 |
| *** jamesmcarthur has joined #openstack-infra | 01:00 | |
| *** yamamoto has joined #openstack-infra | 01:00 | |
| *** onovy has joined #openstack-infra | 01:01 | |
| clarkb | ianw: ya | 01:02 |
| clarkb | if that odesn't work then we can do the full rotation | 01:02 |
| clarkb | but I think whatever had that lockfile is now gone | 01:02 |
| clarkb | and it is stale | 01:02 |
| *** yamamoto has quit IRC | 01:05 | |
| ianw | ok, for the record the files are -> http://paste.openstack.org/show/769796/ | 01:07 |
| *** xek has quit IRC | 01:13 | |
| *** onovy has quit IRC | 01:14 | |
| ianw | #status log /var/gitea/data/git/.gitconfig.lock removed on gitea01, 05, 06, 07 after suspected i/o issues ~ 2-sep-17:20. gerrit replication restarted for each | 01:17 |
| openstackstatus | ianw: finished logging | 01:17 |
| *** jamesmcarthur has quit IRC | 01:17 | |
| clarkb | we may also want to add that lockfile being a replication blocker to our gitea docs | 01:17 |
| clarkb | I can do that tomorrow | 01:17 |
| clarkb | note beacuse the docker contianer didn't restart the lockfile being stale wouldn't be fixed by a cleanup on docker image start | 01:18 |
| clarkb | ianw: thank you for addressing that | 01:19 |
| ianw | np, the replication queue is falling | 01:19 |
| clarkb | and https://gitea01.opendev.org:3000/openstack/election/commits/branch/master is up to date now | 01:19 |
| clarkb | thinking out loud here: is ionice something we should consider? we could nice the git gc | 01:21 |
| clarkb | we run the gc in cron so could update that command to run uner ionice? | 01:22 |
| clarkb | oh neat within the best effort class ionice is automatically derived from the cpu nice value | 01:23 |
| ianw | i think it's suspicious that across all the hosts it was ~17:20 when it seemed to go bad | 01:23 |
| clarkb | they are all backed by the same ceph system | 01:23 |
| clarkb | possible that it was external too (and nicing won't help) | 01:24 |
| ianw | the git gc seems to kick off @ 37 12 * * 0 | 01:24 |
| ianw | it seems more likely a common back-end issue, i'd say? | 01:25 |
| clarkb | ah I had associated the two due to that bugzilla bug but I guess they likely aren't related to the git gc then | 01:25 |
| clarkb | ianw: ya | 01:25 |
| ianw | i think it's just "heavy i/o thing" | 01:25 |
| *** onovy has joined #openstack-infra | 01:26 | |
| *** threestrands has joined #openstack-infra | 01:37 | |
| *** hongbin has joined #openstack-infra | 01:44 | |
| *** yamamoto has joined #openstack-infra | 02:01 | |
| *** apetrich has quit IRC | 02:10 | |
| *** larainema has joined #openstack-infra | 02:12 | |
| *** jamesmcarthur has joined #openstack-infra | 02:17 | |
| *** bobh has joined #openstack-infra | 02:25 | |
| *** jamesmcarthur has quit IRC | 02:27 | |
| *** ramishra has joined #openstack-infra | 02:27 | |
| *** bobh has quit IRC | 02:32 | |
| *** roman_g has quit IRC | 02:34 | |
| *** bobh has joined #openstack-infra | 02:40 | |
| *** bobh has quit IRC | 02:47 | |
| gmann | AJaeger_: +1 for zuul v3 migration as goal. thanks for adding that. | 03:20 |
| *** hongbin has quit IRC | 03:26 | |
| *** hongbin has joined #openstack-infra | 03:28 | |
| *** dchen has quit IRC | 03:40 | |
| *** e0ne has joined #openstack-infra | 03:47 | |
| *** jaosorior has quit IRC | 04:24 | |
| *** jaosorior has joined #openstack-infra | 04:25 | |
| *** hongbin has quit IRC | 04:29 | |
| *** jamesmcarthur has joined #openstack-infra | 04:31 | |
| *** jamesmcarthur has quit IRC | 04:35 | |
| *** dchen has joined #openstack-infra | 04:40 | |
| *** e0ne has quit IRC | 04:49 | |
| *** hongbin has joined #openstack-infra | 04:49 | |
| *** hongbin has quit IRC | 04:49 | |
| *** e0ne has joined #openstack-infra | 04:50 | |
| *** e0ne has quit IRC | 04:51 | |
| *** raukadah is now known as chandankumar | 04:59 | |
| *** udesale has joined #openstack-infra | 04:59 | |
| *** Garyx has quit IRC | 05:07 | |
| *** e0ne has joined #openstack-infra | 05:08 | |
| *** Garyx has joined #openstack-infra | 05:09 | |
| *** yamamoto has quit IRC | 05:12 | |
| *** e0ne has quit IRC | 05:16 | |
| *** bhavikdbavishi has joined #openstack-infra | 05:17 | |
| *** jaosorior has quit IRC | 05:22 | |
| *** yamamoto has joined #openstack-infra | 05:29 | |
| *** prometheanfire has quit IRC | 05:39 | |
| *** fdegir has quit IRC | 05:40 | |
| *** georgk has quit IRC | 05:40 | |
| *** georgk has joined #openstack-infra | 05:40 | |
| *** fdegir has joined #openstack-infra | 05:40 | |
| *** yamamoto has quit IRC | 05:50 | |
| *** prometheanfire has joined #openstack-infra | 05:57 | |
| *** e0ne has joined #openstack-infra | 06:00 | |
| *** e0ne has quit IRC | 06:03 | |
| *** xek has joined #openstack-infra | 06:11 | |
| *** e0ne has joined #openstack-infra | 06:12 | |
| *** e0ne has quit IRC | 06:20 | |
| *** yamamoto has joined #openstack-infra | 06:21 | |
| *** pgaxatte has joined #openstack-infra | 06:21 | |
| *** yamamoto has quit IRC | 06:25 | |
| *** yamamoto has joined #openstack-infra | 06:32 | |
| *** ccamacho has joined #openstack-infra | 06:33 | |
| *** jaosorior has joined #openstack-infra | 06:39 | |
| *** jpena|off is now known as jpena | 06:49 | |
| *** tridde has quit IRC | 07:00 | |
| *** pcaruana has joined #openstack-infra | 07:06 | |
| *** tesseract has joined #openstack-infra | 07:06 | |
| *** trident has joined #openstack-infra | 07:09 | |
| *** slaweq has joined #openstack-infra | 07:09 | |
| *** gfidente has joined #openstack-infra | 07:10 | |
| *** threestrands has quit IRC | 07:18 | |
| *** rcernin has quit IRC | 07:23 | |
| *** kaisers has joined #openstack-infra | 07:24 | |
| *** ofosos has joined #openstack-infra | 07:32 | |
| *** jbadiapa has quit IRC | 07:40 | |
| *** kopecmartin|off is now known as kopecmartin | 07:44 | |
| *** priteau has joined #openstack-infra | 07:46 | |
| *** zhurong has quit IRC | 07:56 | |
| *** jbadiapa has joined #openstack-infra | 07:56 | |
| *** lucasagomes has joined #openstack-infra | 07:58 | |
| *** tkajinam has quit IRC | 08:02 | |
| *** e0ne has joined #openstack-infra | 08:09 | |
| *** xek has quit IRC | 08:10 | |
| *** xek has joined #openstack-infra | 08:10 | |
| *** ociuhandu has joined #openstack-infra | 08:14 | |
| *** pkopec has joined #openstack-infra | 08:22 | |
| *** jaosorior has quit IRC | 08:29 | |
| *** dchen has quit IRC | 08:34 | |
| *** dychen has quit IRC | 08:35 | |
| *** ralonsoh has joined #openstack-infra | 08:37 | |
| *** derekh has joined #openstack-infra | 08:38 | |
| *** tesseract has quit IRC | 08:40 | |
| *** ociuhandu has quit IRC | 08:43 | |
| *** jaosorior has joined #openstack-infra | 08:59 | |
| *** rcernin has joined #openstack-infra | 08:59 | |
| *** gregoryo has quit IRC | 09:10 | |
| *** chandankumar has quit IRC | 09:18 | |
| *** chandankumar has joined #openstack-infra | 09:19 | |
| *** slaweq has quit IRC | 09:22 | |
| *** kjackal has quit IRC | 09:22 | |
| *** slaweq has joined #openstack-infra | 09:23 | |
| *** kjackal has joined #openstack-infra | 09:25 | |
| *** apetrich has joined #openstack-infra | 09:36 | |
| zbr | ianw: clarkb: any change to get https://review.opendev.org/#/c/674092/ ? | 09:39 |
| *** dtantsur|afk is now known as dtantsur | 09:42 | |
| *** Roamer` has joined #openstack-infra | 09:44 | |
| *** tkajinam has joined #openstack-infra | 09:48 | |
| *** yamamoto has quit IRC | 09:56 | |
| *** ianw has quit IRC | 10:00 | |
| *** jaosorior has quit IRC | 10:05 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Improve job and node information banner https://review.opendev.org/677971 | 10:06 |
| *** ianw has joined #openstack-infra | 10:08 | |
| *** pkopec has quit IRC | 10:16 | |
| *** pkopec has joined #openstack-infra | 10:17 | |
| *** tkajinam has quit IRC | 10:18 | |
| *** ianw has quit IRC | 10:27 | |
| *** ianw has joined #openstack-infra | 10:28 | |
| *** ianw has quit IRC | 10:28 | |
| *** ianw has joined #openstack-infra | 10:29 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/project-config master: Finish retiring networking-generic-switch-tempest-plugin https://review.opendev.org/679743 | 10:29 |
| dtantsur | AJaeger_: ^^ | 10:29 |
| *** yamamoto has joined #openstack-infra | 10:29 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Better error handling for the PagureAPIClient https://review.opendev.org/679746 | 10:33 |
| *** ianw has quit IRC | 10:34 | |
| *** yamamoto has quit IRC | 10:35 | |
| *** ianw has joined #openstack-infra | 10:36 | |
| *** bhavikdbavishi has quit IRC | 10:36 | |
| *** ociuhandu has joined #openstack-infra | 10:37 | |
| *** roman_g has joined #openstack-infra | 10:40 | |
| *** ianw_ has joined #openstack-infra | 10:41 | |
| *** ianw_ has quit IRC | 10:41 | |
| *** xenos76 has joined #openstack-infra | 10:43 | |
| *** ianw_ has joined #openstack-infra | 10:43 | |
| *** ianw_ has quit IRC | 10:44 | |
| *** ianw has quit IRC | 10:44 | |
| *** ianw has joined #openstack-infra | 10:44 | |
| *** xek has quit IRC | 10:45 | |
| *** tosky has joined #openstack-infra | 10:45 | |
| *** xek has joined #openstack-infra | 10:45 | |
| *** ianw has quit IRC | 10:46 | |
| *** ianw has joined #openstack-infra | 10:50 | |
| *** ianw has quit IRC | 10:52 | |
| *** ianw has joined #openstack-infra | 10:52 | |
| *** ginopc has quit IRC | 10:55 | |
| *** dave-mccowan has joined #openstack-infra | 11:00 | |
| *** dave-mccowan has quit IRC | 11:06 | |
| *** markvoelker has joined #openstack-infra | 11:09 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/project-config master: Finish retiring networking-generic-switch-tempest-plugin https://review.opendev.org/679743 | 11:10 |
| *** markvoelker has quit IRC | 11:14 | |
| *** nicolasbock has joined #openstack-infra | 11:19 | |
| *** xek has quit IRC | 11:20 | |
| *** ociuhandu has quit IRC | 11:23 | |
| *** jpena is now known as jpena|lunch | 11:31 | |
| *** udesale has quit IRC | 11:37 | |
| *** Garyx has quit IRC | 11:38 | |
| *** Garyx has joined #openstack-infra | 11:41 | |
| *** Garyx has quit IRC | 11:41 | |
| *** Garyx has joined #openstack-infra | 11:41 | |
| *** ociuhandu has joined #openstack-infra | 11:43 | |
| *** xek has joined #openstack-infra | 11:48 | |
| *** ociuhandu has quit IRC | 11:48 | |
| *** jaosorior has joined #openstack-infra | 11:51 | |
| *** udesale has joined #openstack-infra | 11:51 | |
| *** ricolin_ has joined #openstack-infra | 11:51 | |
| *** ociuhandu has joined #openstack-infra | 11:52 | |
| *** ricolin has quit IRC | 11:54 | |
| *** dtantsur is now known as dtantsur|brb | 11:57 | |
| *** Lucas_Gray has joined #openstack-infra | 12:01 | |
| *** rh-jelabarre has joined #openstack-infra | 12:02 | |
| *** bhavikdbavishi has joined #openstack-infra | 12:13 | |
| *** larainema has quit IRC | 12:15 | |
| *** markvoelker has joined #openstack-infra | 12:18 | |
| openstackgerrit | Benedikt Löffler proposed zuul/zuul master: Report retried builds in a build set via mqtt. https://review.opendev.org/632727 | 12:24 |
| openstackgerrit | Benedikt Löffler proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 12:24 |
| *** yamamoto has joined #openstack-infra | 12:25 | |
| *** jistr has quit IRC | 12:29 | |
| *** jistr has joined #openstack-infra | 12:29 | |
| *** jcoufal has joined #openstack-infra | 12:29 | |
| *** jistr has quit IRC | 12:30 | |
| *** ociuhandu has quit IRC | 12:30 | |
| *** ociuhandu has joined #openstack-infra | 12:30 | |
| *** ociuhandu has quit IRC | 12:31 | |
| *** ociuhandu_ has joined #openstack-infra | 12:31 | |
| *** ociuhandu has joined #openstack-infra | 12:32 | |
| *** Lucas_Gray has quit IRC | 12:32 | |
| *** rosmaita has joined #openstack-infra | 12:33 | |
| *** weshay has joined #openstack-infra | 12:34 | |
| *** Lucas_Gray has joined #openstack-infra | 12:35 | |
| *** ociuhandu_ has quit IRC | 12:36 | |
| *** jcoufal has quit IRC | 12:39 | |
| *** jpena|lunch is now known as jpena | 12:40 | |
| *** jistr has joined #openstack-infra | 12:40 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Switch PDF file name to doc-PROJECT.pdf https://review.opendev.org/679777 | 12:43 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Better error handling for the PagureAPIClient https://review.opendev.org/679746 | 12:44 |
| *** jamesmcarthur has joined #openstack-infra | 12:46 | |
| *** Goneri has joined #openstack-infra | 12:47 | |
| *** redrobot has quit IRC | 12:55 | |
| *** dmsimard has quit IRC | 12:56 | |
| *** dmsimard has joined #openstack-infra | 12:58 | |
| *** david-lyle has quit IRC | 13:02 | |
| *** dklyle has joined #openstack-infra | 13:02 | |
| *** pcaruana has quit IRC | 13:03 | |
| *** smarcet has joined #openstack-infra | 13:03 | |
| *** jcoufal has joined #openstack-infra | 13:06 | |
| *** witek has joined #openstack-infra | 13:09 | |
| *** sthussey has joined #openstack-infra | 13:12 | |
| *** signed8b_ has quit IRC | 13:13 | |
| *** signed8bit has joined #openstack-infra | 13:13 | |
| *** smarcet has quit IRC | 13:20 | |
| *** ociuhandu has quit IRC | 13:21 | |
| *** ociuhandu has joined #openstack-infra | 13:22 | |
| *** xek has quit IRC | 13:26 | |
| *** ociuhandu has quit IRC | 13:28 | |
| *** ociuhandu has joined #openstack-infra | 13:29 | |
| *** mriedem has joined #openstack-infra | 13:29 | |
| *** Garyx has quit IRC | 13:30 | |
| *** pcaruana has joined #openstack-infra | 13:34 | |
| *** redrobot_ has joined #openstack-infra | 13:36 | |
| *** redrobot_ is now known as redrobot | 13:37 | |
| *** dtantsur|brb is now known as dtantsur | 13:46 | |
| *** KeithMnemonic has joined #openstack-infra | 13:47 | |
| *** pcaruana has quit IRC | 13:47 | |
| *** eharney has joined #openstack-infra | 13:49 | |
| *** yamamoto has quit IRC | 13:55 | |
| *** kjackal has quit IRC | 13:57 | |
| *** yamamoto has joined #openstack-infra | 13:59 | |
| *** pcaruana has joined #openstack-infra | 13:59 | |
| *** kjackal has joined #openstack-infra | 14:00 | |
| *** Garyx has joined #openstack-infra | 14:03 | |
| *** yamamoto has quit IRC | 14:04 | |
| *** smarcet has joined #openstack-infra | 14:05 | |
| *** bhavikdbavishi has quit IRC | 14:05 | |
| *** pcaruana has quit IRC | 14:07 | |
| *** pcaruana has joined #openstack-infra | 14:10 | |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Adding in proper NUMA labels for FN https://review.opendev.org/679789 | 14:10 |
| donnyd | Adding proper labels for FN NUMA testing | 14:11 |
| *** xek has joined #openstack-infra | 14:12 | |
| *** ociuhandu has quit IRC | 14:13 | |
| *** xek_ has joined #openstack-infra | 14:14 | |
| *** xek has quit IRC | 14:17 | |
| *** ociuhandu has joined #openstack-infra | 14:17 | |
| *** udesale has quit IRC | 14:20 | |
| *** udesale has joined #openstack-infra | 14:21 | |
| *** Lucas_Gray has quit IRC | 14:27 | |
| *** jaosorior has quit IRC | 14:27 | |
| *** michael-beaver has joined #openstack-infra | 14:31 | |
| *** rcernin has quit IRC | 14:32 | |
| *** ociuhandu has quit IRC | 14:34 | |
| *** ociuhandu has joined #openstack-infra | 14:35 | |
| *** ramishra_ has joined #openstack-infra | 14:36 | |
| *** smarcet has quit IRC | 14:36 | |
| *** ramishra has quit IRC | 14:36 | |
| *** ociuhandu has quit IRC | 14:39 | |
| donnyd | So if I needed to schedule a long term outage (defined as more than 24H), should I just update project config to reflect 0's and let everyone know | 14:42 |
| donnyd | To get my generator online it looks like a significant amount of change will need to be done in my electrical and the power will be out for a day | 14:43 |
| *** jaosorior has joined #openstack-infra | 14:46 | |
| clarkb | ya I think that updating the config ti set max seevers to zero is the best way to communicate that | 14:46 |
| clarkb | we tend to review those changes quickly | 14:46 |
| *** Garyx has quit IRC | 14:47 | |
| *** piotrowskim has quit IRC | 14:52 | |
| *** Garyx has joined #openstack-infra | 14:52 | |
| *** smarcet has joined #openstack-infra | 14:59 | |
| *** pkopec has quit IRC | 15:02 | |
| *** pkopec has joined #openstack-infra | 15:04 | |
| *** ociuhandu has joined #openstack-infra | 15:07 | |
| *** jaosorior has quit IRC | 15:11 | |
| clarkb | donnyd: can you check my comment on https://review.opendev.org/679789 | 15:15 |
| donnyd | So you want me to pull the extra pool off and just use the expanded memory test pool? | 15:17 |
| *** pkopec has quit IRC | 15:17 | |
| clarkb | donnyd: pull off the expanded memory test pull too and put all of those labels in the main pool | 15:17 |
| donnyd | ok | 15:17 |
| clarkb | this assumes that you are ok with potential more than 10 of those flavors being booted at once | 15:17 |
| donnyd | Yea, that should work without too much issue | 15:17 |
| clarkb | in vexxhost we split it because they don't manage things with quotas and instead rely on nodepool max-servers values and so we have separate pools to separate out the different limits | 15:18 |
| donnyd | but I think we originally did it so there would be guaranteed space for those tests | 15:18 |
| clarkb | cool I assumed it wasn't a problem for your cloud and I think it will make the scheduling better to hav ea single pool | 15:18 |
| donnyd | ok, I am fine with that | 15:18 |
| openstackgerrit | Scott Hussey proposed openstack/project-config master: [WIP] Add airshipui project https://review.opendev.org/679806 | 15:19 |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Adding in proper NUMA labels for FN https://review.opendev.org/679789 | 15:20 |
| dmsimard | infra-root: I'm back from vacation and have time to help with the issues with ara static reports on swift. I'm still catching up.. has anything particular been done yet ? | 15:20 |
| clarkb | dmsimard: nothing specific to ara | 15:20 |
| fungi | dmsimard: i don't think so, no, other than taking vexxhost out of the rotation and purging all the logs we uploaded there | 15:20 |
| clarkb | I've got a change up that needs testing to improve overall container sharding which will help but not address ara specifically | 15:21 |
| donnyd | clarkb: Ok, I added more to the pool because I can support more normal jobs without the dedicated space for expanded jobs | 15:23 |
| *** kaiokmo has joined #openstack-infra | 15:23 | |
| dmsimard | even if the amount of files (and directories) in 1.x has slightly improved, static html reports are not efficient at all at scale :( | 15:24 |
| clarkb | donnyd: +2 thanks | 15:24 |
| donnyd | Thanks clarkb | 15:24 |
| *** smarcet has quit IRC | 15:26 | |
| *** pkopec has joined #openstack-infra | 15:27 | |
| *** pcaruana has quit IRC | 15:27 | |
| *** smarcet has joined #openstack-infra | 15:29 | |
| *** chandankumar is now known as raukadah | 15:32 | |
| *** kaiokmo has quit IRC | 15:33 | |
| dmsimard | clarkb, fungi: I don't have any ideas that wouldn't involve the sqlite middleware in some way or another. It's exactly the problem it was designed to solve. I'll try and see if I can figure how we might use it with Swift. | 15:34 |
| fungi | i guess javascript couldn't retrieve the sqlite file and query it locally on the client | 15:35 |
| * fungi knows little about javascript, thinks it's mostly made of magic | 15:36 | |
| timburke | swift supports (encourages, even) custom middleware, fwiw -- but it's more of an operator feature than a user-facing one. tricky when working with public clouds... | 15:36 |
| *** kaiokmo has joined #openstack-infra | 15:36 | |
| dmsimard | fungi: I am not a very javascript person :) | 15:37 |
| clarkb | fungi: I think it is possible, but would require writing ara renderer in js to run in the browser rather than server side python middleware | 15:37 |
| *** jaosorior has joined #openstack-infra | 15:37 | |
| dmsimard | clarkb: ara-web (from 1.x) is meant to be that javascript web client but it currently expects an API endpoint to retrieve data from | 15:38 |
| timburke | maybe useful? https://github.com/kripken/sql.js | 15:38 |
| *** tosky_ has joined #openstack-infra | 15:40 | |
| *** tosky has quit IRC | 15:41 | |
| *** tosky_ is now known as tosky | 15:41 | |
| *** xek_ has quit IRC | 15:42 | |
| *** ociuhandu has quit IRC | 15:43 | |
| dmsimard | timburke: I'd rather not go there if we can help it :p | 15:43 |
| *** ociuhandu has joined #openstack-infra | 15:44 | |
| dmsimard | well, at least for ara 0.x -- maybe it could be some sort of backend in ara-web for 1.x | 15:44 |
| timburke | fair enough ;-) | 15:44 |
| dmsimard | for now I'm trying to see if there is a quick win available to mitigate ongoing issues | 15:45 |
| *** ociuhandu has quit IRC | 15:47 | |
| *** udesale has quit IRC | 15:47 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Document globally unique container name requirement https://review.opendev.org/679814 | 15:47 |
| *** ociuhandu has joined #openstack-infra | 15:47 | |
| AJaeger_ | config-core, could you review https://review.opendev.org/679777, please? We need one more change for PDFs ;/ | 15:49 |
| dmsimard | Removing CSS/JS assets from the generated bundle and using a CDN instead /could/ save us ~73 files and ~15MB (unzipped) per report | 15:50 |
| AJaeger_ | dmsimard: we can host them on docs.openstack.org easily - I can import into openstack-manuals ;) | 15:50 |
| *** beekneemech is now known as bnemec | 15:51 | |
| AJaeger_ | or we host in a central place on opendev... | 15:51 |
| AJaeger_ | dmsimard: and then run ara with an option "use-cdn=https://someplace/" ? | 15:51 |
| dmsimard | AJaeger_: something like that | 15:51 |
| dmsimard | It would be an improvement but not a long term solution | 15:52 |
| AJaeger_ | depends on how long the long term solution needs ;) | 15:52 |
| dmsimard | there are some good opportunities with the new api but it needs more discussion | 15:54 |
| dmsimard | something that I'm excited about would be the opportunity for jobs to send their results back to the executor's instance of ara so that everything is in a single report instead of being nested in different directories like OSA, TripleO and Kolla does | 15:55 |
| dmsimard | but that also means that the callback might be sending data from across the world in a faraway openstack region back to the executor | 15:56 |
| *** e0ne has quit IRC | 16:00 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths https://review.opendev.org/679145 | 16:00 |
| *** smarcet has quit IRC | 16:00 | |
| clarkb | not sure why that couldn't be a long term improvement | 16:01 |
| clarkb | could make it a flag when generating the report to set where to find static assets | 16:01 |
| clarkb | however 73 files isn't that many compared to the size of the report so not sure it will help address the file count problem | 16:02 |
| *** lucasagomes has quit IRC | 16:03 | |
| clarkb | AJaeger_: why do the svgs get converted to .pdf files? | 16:03 |
| dmsimard | clarkb: right, this is what I meant -- not duplicating these 73 files and 15MB per report would be helpful but in the grand scheme of things, if there's 2000 task results, there's going to be 2000+ files | 16:07 |
| AJaeger_ | clarkb: that's the way latex builds them when included. Projects include README.rst which has the badges - and then the badges get converted from SVG to PDF and inlined by latex into the final PDF | 16:09 |
| clarkb | dmsimard: ya | 16:09 |
| clarkb | AJaeger_: ah so it is an intermediate result | 16:09 |
| AJaeger_ | yes | 16:09 |
| AJaeger_ | but also the name of the endresult - so that gives some kind of recursion ;( | 16:10 |
| openstackgerrit | Merged openstack/project-config master: Adding in proper NUMA labels for FN https://review.opendev.org/679789 | 16:10 |
| AJaeger_ | clarkb: so, there are two ways: Change the PDF file name for the endresult - or do not include badges. The docs team dsicussed and proposed to change the PDF file of the endresult so that the badges can be included if desired | 16:10 |
| AJaeger_ | config-core, the discussion above with clarkb is about https://review.opendev.org/679777 | 16:11 |
| clarkb | makes sense, I +2'd since the SVG to PDF conversion is done by sphinx and this role is meant to support sphinx | 16:11 |
| clarkb | (I think it shoul dhandle that behavior) | 16:11 |
| *** gyee has joined #openstack-infra | 16:12 | |
| *** pgaxatte has quit IRC | 16:13 | |
| AJaeger_ | thanks, clarkb | 16:13 |
| *** jpena is now known as jpena|off | 16:13 | |
| *** jcoufal has quit IRC | 16:15 | |
| *** jcoufal has joined #openstack-infra | 16:16 | |
| openstackgerrit | Sorin Sbarnea proposed opendev/system-config master: Recognize DISK_FULL failure messages https://review.opendev.org/631509 | 16:17 |
| *** dtantsur is now known as dtantsur|afk | 16:20 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths https://review.opendev.org/679145 | 16:24 |
| *** ricolin_ has quit IRC | 16:25 | |
| *** jaosorior has quit IRC | 16:25 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths https://review.opendev.org/679145 | 16:26 |
| *** kopecmartin is now known as kopecmartin|off | 16:27 | |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Test improvements to swift hosted logs https://review.opendev.org/679819 | 16:28 |
| *** ociuhandu_ has joined #openstack-infra | 16:30 | |
| AJaeger_ | clarkb: I left comments on 679145,4 , those are still valid | 16:31 |
| *** smarcet has joined #openstack-infra | 16:32 | |
| *** ociuhandu has quit IRC | 16:33 | |
| clarkb | AJaeger_: thanks responded, I'll fix those once I get test results back | 16:33 |
| clarkb | AJaeger_: in particular I want to know if the default() works after setting it in a previous play | 16:34 |
| *** ociuhandu_ has quit IRC | 16:35 | |
| clarkb | we may have to set it explicitly as you suggest but was going to check test results to see | 16:35 |
| AJaeger_ | I see | 16:36 |
| *** pkopec has quit IRC | 16:37 | |
| *** yamamoto has joined #openstack-infra | 16:39 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths https://review.opendev.org/679145 | 16:39 |
| clarkb | tha should fix a test problem as well as address your docs comments | 16:39 |
| *** smarcet has quit IRC | 16:40 | |
| AJaeger_ | great | 16:42 |
| *** virendra-sharma has joined #openstack-infra | 16:43 | |
| *** yamamoto has quit IRC | 16:43 | |
| virendra-sharma | Hi, My zuul is not connecting to gerrit (review.opendev.org) | 16:46 |
| virendra-sharma | can anyone please help? | 16:47 |
| clarkb | virendra-sharma: sure, can you provide more information on why/how it isn't connecting? | 16:47 |
| virendra-sharma | ssh -p 29418 -i /var/lib/zuul/ssh/id_rsa hp-storage-blr-ci@review.opendev.org | 16:48 |
| virendra-sharma | even if i run ssh command it's taking too long and failing | 16:48 |
| clarkb | dmsimard: is what I'm doing at https://review.opendev.org/#/c/679145/6/roles/emit-job-header/tasks/main.yaml not valid? specifically reusing the variable name to pass from one role to anohter? | 16:49 |
| clarkb | virendra-sharma: can you run your ssh command with the -vvv flag? | 16:49 |
| clarkb | that should tell us how far in the connection ssh is getting | 16:49 |
| virendra-sharma | zuul@cssos-openstack-ci:~$ ssh -p 29418 -i /var/lib/zuul/ssh/id_rsa hp-storage-blr-ci@review.opendev.org -vOpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014debug1: Reading configuration data /etc/ssh/ssh_configdebug1: /etc/ssh/ssh_config line 20: Applying options for *debug1: Connecting to review.opendev.org [104.130.246.32] port 29418.debug1: connect to ad | 16:50 |
| virendra-sharma | dress 104.130.246.32 port 29418: Connection timed outdebug1: Connecting to review.opendev.org [2001:4800:7819:103:be76:4eff:fe04:9229] port 29418.debug1: connect to address 2001:4800:7819:103:be76:4eff:fe04:9229 port 29418: Network is unreachablessh: connect to host review.opendev.org port 29418: Network is unreachable | 16:50 |
| clarkb | virendra-sharma: that sounds like you have a firewall blocking the connection | 16:51 |
| virendra-sharma | clarkb even after disabling firewall service I am getting same error | 16:51 |
| clarkb | those IP addresses look correct | 16:52 |
| clarkb | virendra-sharma: I am able to connect from here. This is why I suspect a firewall | 16:53 |
| clarkb | let me double check we don't have any rules on our server that may be blocking the traffic | 16:53 |
| AJaeger_ | virendra-sharma: might be a corporate firewall in betwween | 16:53 |
| virendra-sharma | Seems this is not issue with server firewall.. may be AJaeger_ suggests it can be corporate firewall issue. | 16:54 |
| *** jaosorior has joined #openstack-infra | 16:54 | |
| clarkb | virendra-sharma: we do have a 100 connection limit | 16:55 |
| clarkb | I'm checking netstat to see if any IPs are near that now | 16:55 |
| virendra-sharma | sure clarkb That will help | 16:56 |
| AJaeger_ | clarkb: in that case https://review.opendev.org/#/c/678413/ would help - but that needs a restart of gerrit... | 16:58 |
| *** derekh has quit IRC | 17:00 | |
| *** jtomasek has quit IRC | 17:01 | |
| clarkb | virendra-sharma: the most connections from a single IP address are 42 that I can see | 17:01 |
| clarkb | I don't think we are hitting any connection limits | 17:01 |
| clarkb | You should check your firewalls | 17:01 |
| *** smarcet has joined #openstack-infra | 17:02 | |
| virendra-sharma | Thank you clarkb for cross verification, will check with corporate firewall. | 17:02 |
| *** ociuhandu has joined #openstack-infra | 17:02 | |
| clarkb | virendra-sharma: traceoute -T -p 29418 may help | 17:03 |
| AJaeger_ | clarkb: traceroute6 ? | 17:05 |
| virendra-sharma | root@cssos-openstack-ci:/home/stack# traceroute -T -p 29418 104.130.246.32traceroute to 104.130.246.32 (104.130.246.32), 30 hops max, 60 byte packets 1 16.184.12.2 (16.184.12.2) 1.170 ms 1.124 ms 1.081 ms 2 10.64.218.193 (10.64.218.193) 1.259 ms 1.827 ms 2.477 ms 3 * * * 4 * * * 5 * * * | 17:05 |
| AJaeger_ | virendra-sharma: that looks like corporate firewall | 17:05 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Add option for object store friendly log paths https://review.opendev.org/679145 | 17:05 |
| AJaeger_ | but you used wrong address, use: | 17:06 |
| clarkb | AJaeger_: the earlier paste tried ipv4 and ipv6 addrs | 17:06 |
| AJaeger_ | clarkb: ah, only noticed ipv6 - ok | 17:06 |
| clarkb | checking both is probably worthwhile with traceroute but I agree looks like corporate firewall | 17:06 |
| virendra-sharma | AJaeger_ I thinks it's IP of gerrit so won't it be same ? | 17:06 |
| *** ociuhandu has quit IRC | 17:07 | |
| AJaeger_ | virendra-sharma: IP address looks indeed fine, I was confused | 17:07 |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Test improvements to swift hosted logs https://review.opendev.org/679819 | 17:07 |
| virendra-sharma | also I am looking to port my third party CI from python2.7 to python3.7 but currently using ubuntu14.04 (newton EOL) so will it be compatible | 17:08 |
| dmsimard | clarkb: I've done something like that before in a set_fact task, not positive whether it works in "vars" without testing it | 17:08 |
| clarkb | dmsimard: I'm testing using a different name with patest patchsets to see if that is indeed the problem. We may know shortly | 17:09 |
| virendra-sharma | I stick to ubuntu14.04(newton EOL) due to FC passthrough issue, on my devstack instance.. Any suggestion ? | 17:09 |
| *** smarcet has quit IRC | 17:09 | |
| clarkb | virendra-sharma: you will want to go to ubuntu 18.04 | 17:10 |
| clarkb | that has python2.7 3.6 and 3.7 available | 17:10 |
| virendra-sharma | clarkb I can not upgrade my ubuntu due to FC passthrough issue, newer version of ubuntu except 14.04 giving issues | 17:11 |
| *** smarcet has joined #openstack-infra | 17:12 | |
| virendra-sharma | Is there any compatibility for ubuntu kernel version with devstack ? | 17:12 |
| clarkb | virendra-sharma: devstack won't chnage your kernel if that is what you are asking | 17:12 |
| *** e0ne has joined #openstack-infra | 17:12 | |
| clarkb | but you could set up your jobs or your nodepool images to configure a working kernel | 17:12 |
| virendra-sharma | before running devstack on provider if i try to check fc passthrough then it gives error of hardware compatibility error | 17:14 |
| virendra-sharma | I tired with ubuntu16 & 18 both but same issue persists. | 17:14 |
| *** jaosorior has quit IRC | 17:14 | |
| clarkb | you may have better luck finding help for that with the cinder team | 17:15 |
| clarkb | since they are more likely to work with fiberchannel and possibly have run into problems like that in the past | 17:15 |
| *** roman_g has quit IRC | 17:15 | |
| AJaeger_ | virendra-sharma: there's #openstack-cinder channel if you want to discuss with them | 17:16 |
| *** jaosorior has joined #openstack-infra | 17:16 | |
| clarkb | dmsimard: changing the var names like that fixed it | 17:16 |
| *** iurygregory has joined #openstack-infra | 17:17 | |
| *** jaosorior has quit IRC | 17:18 | |
| *** mattw4 has joined #openstack-infra | 17:21 | |
| *** e0ne has quit IRC | 17:23 | |
| *** e0ne has joined #openstack-infra | 17:23 | |
| *** smarcet has quit IRC | 17:27 | |
| *** mattw4 has quit IRC | 17:29 | |
| *** pcaruana has joined #openstack-infra | 17:29 | |
| *** ramishra_ has quit IRC | 17:30 | |
| *** e0ne has quit IRC | 17:30 | |
| *** mattw4 has joined #openstack-infra | 17:34 | |
| *** priteau has quit IRC | 17:43 | |
| openstackgerrit | Gabriele Cerami proposed opendev/elastic-recheck master: Adding query for dlrn servers outage https://review.opendev.org/679835 | 17:46 |
| *** jtomasek has joined #openstack-infra | 17:46 | |
| *** virendra-sharma has quit IRC | 17:47 | |
| *** xeno_os76_xyz has joined #openstack-infra | 17:50 | |
| *** jtomasek has quit IRC | 17:50 | |
| clarkb | re ^ It seems that trunk-backup is now working but the http -> https:// redirect for trunk.rdoproject.org redirects to https://trunk-primary.rdoproject.org which has a cert for trunk.rdoproject.org so that aspect of the problem is still a problem | 17:52 |
| clarkb | though I suspect jobs won't notice that as long as they use our proxy since it will talk to https directly and not hit the http to https redirect | 17:52 |
| *** jtomasek has joined #openstack-infra | 17:52 | |
| *** xenos76 has quit IRC | 17:52 | |
| fungi | unless there is no working http | 17:55 |
| slittle1 | Hi all. StarlingX want's to do a history preserving git restructure that will create ~10 new repos, split off from existing repos. | 17:57 |
| slittle1 | We hope to do this in as quick and atomic a fashion as possible. | 17:58 |
| slittle1 | I have a tool to assist in the restructuring. | 17:58 |
| fungi | slittle1: the way we've recommended doing that in the past is with something like git filter-branch (fully scripted so you can refresh the filtered version shortly before you freeze the original) and then have the project creation import those repos | 17:59 |
| slittle1 | The prototype repos will be posted to github. | 17:59 |
| AJaeger_ | slittle1: any public accessible git server works ;) | 17:59 |
| fungi | and just coordinate with us to review/approve the new project additions quickly, ideally you'd just add them all with a single change to project-config | 17:59 |
| slittle1 | Yes, my script uses that under the hood | 17:59 |
| AJaeger_ | slittle1: do you want to retire old repos at same time? | 18:00 |
| slittle1 | Co-ordinating the quick review is exactly what I'm looking for. | 18:00 |
| slittle1 | No repos will be retired | 18:00 |
| AJaeger_ | ok | 18:00 |
| fungi | aside from that, make sure you abandon open reviews on the original repo and let all your reviewers know not to approve anything in it as soon as you generate your final split copies | 18:00 |
| AJaeger_ | fungi: since no repos are retired, abandon won't be needed, would it? | 18:01 |
| fungi | well, any changes altering files which are moving likely won't be mergeable anyway | 18:01 |
| AJaeger_ | fungi: yes, indeed - those will be merge conflicts | 18:02 |
| fungi | (if corresponding files are being deleted from the original) | 18:02 |
| slittle1 | Yes the originals will be deleted in the follow-up updates generated by my tool. | 18:03 |
| *** markvoelker has quit IRC | 18:03 | |
| slittle1 | I'm counting on those merge conflictsto highlight reviews that need to be reworked | 18:03 |
| fungi | yeah, that should do it | 18:03 |
| fungi | if the original project has its own dedicated reviewer group, you could always temporarily empty all reviewers besides yourself from it to ensure nobody else aside form you approves changes in it during the freeze | 18:04 |
| fungi | depending on whether you're worried there's much risk of coordinating that with them | 18:05 |
| slittle1 | Pending a green light from the final sanity test, I hope to process all of this Wednesday between 10am-5pm EST | 18:06 |
| *** iurygregory has quit IRC | 18:06 | |
| slittle1 | i think that is 2-9pm universal time | 18:06 |
| *** iurygregory_ has joined #openstack-infra | 18:06 | |
| fungi | yes, your tz conversion is correct | 18:07 |
| slittle1 | Unless you folks are unavailable in that time window | 18:07 |
| *** armax has joined #openstack-infra | 18:08 | |
| clarkb | I'll be around. Pushing up a change early would probably be good. You can WIP it until ready but that gives us a chance to review it ahead of time | 18:08 |
| slittle1 | K, I'll start on that next. | 18:09 |
| *** smarcet has joined #openstack-infra | 18:12 | |
| *** e0ne has joined #openstack-infra | 18:15 | |
| *** jamesmcarthur has quit IRC | 18:19 | |
| *** markvoelker has joined #openstack-infra | 18:20 | |
| *** xek has joined #openstack-infra | 18:23 | |
| *** tosky has quit IRC | 18:24 | |
| *** markvoelker has quit IRC | 18:24 | |
| *** smarcet has quit IRC | 18:31 | |
| slittle1 | Is it preferred to have a single review for all 10, or individual reviews? | 18:36 |
| clarkb | slittle1: fungi suggested a single review. I think that will help ensure we get things in as atomicly as possible | 18:37 |
| slittle1 | good | 18:37 |
| *** markvoelker has joined #openstack-infra | 18:43 | |
| *** gfidente is now known as gfidente|afk | 18:43 | |
| *** jamesmcarthur has joined #openstack-infra | 18:47 | |
| *** eernst has joined #openstack-infra | 18:50 | |
| *** gyee has quit IRC | 18:53 | |
| *** eernst has quit IRC | 18:55 | |
| *** Goneri has quit IRC | 18:55 | |
| clarkb | friendly reminder: we'll be having our team meeting in about 5 minutes over in #openstack-meeting | 18:55 |
| AJaeger_ | fungi, do you have time for a short review of https://review.opendev.org/679777 , please? | 18:56 |
| *** iurygregory_ is now known as iurygregory | 18:57 | |
| *** igordc has joined #openstack-infra | 19:02 | |
| *** AJaeger_ is now known as AJaeger | 19:03 | |
| *** jamesmcarthur has quit IRC | 19:04 | |
| *** gyee has joined #openstack-infra | 19:07 | |
| slittle1 | in projects.yaml ... is it correct to use 'upstream:' for the repo that will seed the opendev repo? I don't want to imply that the seed repo will always be the authoritative source. Or is there another directive for that? | 19:15 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: gitea: use random time for git gc run https://review.opendev.org/679846 | 19:16 |
| clarkb | slittle1: upstream is the correct directive | 19:18 |
| clarkb | its the upstream until we import into gerrit then gerrit becomes the canonical location (w eclean out the upstream directives once complete) | 19:18 |
| fungi | yeah, it could have been called "import" but its usage is overloaded with another feature for tracking remote branches | 19:19 |
| *** harlowja has joined #openstack-infra | 19:19 | |
| *** jamesmcarthur has joined #openstack-infra | 19:20 | |
| openstackgerrit | Scott Little proposed openstack/project-config master: StarlingX retructuring: Ten new repos https://review.opendev.org/679848 | 19:24 |
| *** e0ne_ has joined #openstack-infra | 19:24 | |
| *** ralonsoh has quit IRC | 19:25 | |
| *** e0ne_ has quit IRC | 19:26 | |
| *** e0ne has quit IRC | 19:27 | |
| AJaeger | slittle1: Just curious: is starlingx following airship? I see some names that sound familiar | 19:27 |
| AJaeger | oh, starlingx does not get IRC notifications anymore since the opendev rename and nobody seems to have missed them ;( Patch coming... | 19:28 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Update channels for starlingx https://review.opendev.org/679850 | 19:30 |
| AJaeger | slittle1: I suggest you WIP your change for now, use the workflow -1 | 19:30 |
| AJaeger | slittle1: also, a bit more wording in your commit message would also be appreciated | 19:31 |
| *** michael-beaver has quit IRC | 19:40 | |
| slittle1 | will do | 19:56 |
| slittle1 | zuul seems unhappy ... I think with the upstream urls. Is it my syntax? Or is it testing for the existence of the URL already ? | 19:57 |
| AJaeger | it tests for existence already - so, recheck, once you're done. | 19:58 |
| AJaeger | And I'll +2 only once the URLs are ready. Rest looks fine - but see my comment above asking for longer commit message | 19:58 |
| *** pcaruana has quit IRC | 20:00 | |
| slittle1 | coming right up .... | 20:10 |
| AJaeger | ianw, could you put https://review.opendev.org/679777 on your review queue, please? | 20:11 |
| * AJaeger waves good night | 20:11 | |
| openstackgerrit | Scott Little proposed openstack/project-config master: StarlingX retructuring: Ten new repos https://review.opendev.org/679848 | 20:14 |
| *** Lucas_Gray has joined #openstack-infra | 20:14 | |
| slittle1 | github projects have been created, but not yet populated | 20:16 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Add file matcher to releasenotes promote job https://review.opendev.org/679856 | 20:16 |
| fungi | all the zuul job cares is that the git url is valid and has a master branch, i think | 20:17 |
| *** slaweq has quit IRC | 20:17 | |
| *** witek has quit IRC | 20:19 | |
| slittle1 | master branch won't exist until I populate it tomorrow morning. | 20:19 |
| *** jamesmcarthur has quit IRC | 20:27 | |
| *** jamesmcarthur has joined #openstack-infra | 20:30 | |
| clarkb | fungi: if you have a chance can you review 679846? | 20:31 |
| *** iurygregory has quit IRC | 20:33 | |
| *** witek has joined #openstack-infra | 20:38 | |
| *** jtomasek has quit IRC | 20:42 | |
| *** jcoufal has quit IRC | 20:42 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Switch PDF file name to doc-PROJECT.pdf https://review.opendev.org/679777 | 20:43 |
| *** markvoelker has quit IRC | 20:45 | |
| *** Lucas_Gray has quit IRC | 20:46 | |
| *** Lucas_Gray has joined #openstack-infra | 20:47 | |
| *** ociuhandu has joined #openstack-infra | 20:48 | |
| *** Lucas_Gray has quit IRC | 20:50 | |
| fungi | clarkb: ianw: so that's a jinja2 macro to set a random time in the crontab? and does it get replaced every time ansible is run? | 20:51 |
| *** trident has quit IRC | 20:52 | |
| *** Lucas_Gray has joined #openstack-infra | 20:52 | |
| clarkb | fungi: the seed value fixes it and the 59 | sets a range | 20:52 |
| clarkb | it should not be set each run as long as the hostnames don't change | 20:52 |
| clarkb | 0-59 is the range | 20:53 |
| *** jamesmcarthur has quit IRC | 20:53 | |
| fungi | oh, got it, deterministic | 20:53 |
| fungi | shouldn't it be 60| and 24| or is that not a modulus? | 20:54 |
| clarkb | it is not a modulus it is a range | 20:55 |
| clarkb | with an implicit 0 start | 20:56 |
| *** Goneri has joined #openstack-infra | 20:58 | |
| *** trident has joined #openstack-infra | 21:03 | |
| fungi | ahh | 21:05 |
| fungi | so | is a range operator in jinja2? | 21:05 |
| clarkb | no I think its more like a pipe operator and the random filter takes that as input | 21:07 |
| *** witek has quit IRC | 21:08 | |
| clarkb | we could do this instead: {{ random(59, seed=inventory_hostname) }} too | 21:08 |
| clarkb | def rand(environment, end, start=None, step=None, seed=None) is the function signature (random is the jinja side name for rand) | 21:08 |
| fungi | makes sense, thanks | 21:08 |
| dmsimard | I was hoping to discuss ara but I'm being pulled for dinner, would appreciate opinions/comments on https://etherpad.openstack.org/p/Vz5IzxlWFz -- other proposals are most welcome :p | 21:09 |
| dmsimard | I have some thoughts that are more forward looking with 1.x, I'll write them down after dinner | 21:10 |
| clarkb | dmsimard: I added another option (the one we are currently using) | 21:12 |
| *** rmcall has joined #openstack-infra | 21:17 | |
| *** ociuhandu has quit IRC | 21:18 | |
| *** smarcet has joined #openstack-infra | 21:24 | |
| *** rmcall has quit IRC | 21:28 | |
| *** markvoelker has joined #openstack-infra | 21:31 | |
| *** markvoelker has quit IRC | 21:35 | |
| *** kjackal has quit IRC | 21:36 | |
| clarkb | announcements for the september 16 gerrit outage sent | 21:36 |
| openstackgerrit | Merged opendev/system-config master: gitea: use random time for git gc run https://review.opendev.org/679846 | 21:38 |
| paladox | https://bugs.chromium.org/p/gerrit/issues/detail?id=11418 is exciting! | 22:00 |
| pabelanger | interesting | 22:03 |
| dmsimard | clarkb: so was the issue specific to rgw then ? | 22:10 |
| *** mriedem has quit IRC | 22:10 | |
| clarkb | dmsimard: its specific to ceph I don't know what aspect of ceph | 22:10 |
| clarkb | dmsimard: mnaser made it sound like ceph will do rebalancing behind the scenes and that bogs thing sdown when there are too many objects | 22:11 |
| *** Lucas_Gray has quit IRC | 22:12 | |
| *** Wryhder has joined #openstack-infra | 22:12 | |
| *** Wryhder is now known as Lucas_Gray | 22:13 | |
| *** xeno_os76_xyz has quit IRC | 22:13 | |
| *** ociuhandu has joined #openstack-infra | 22:30 | |
| *** ociuhandu has quit IRC | 22:35 | |
| *** mgoddard has quit IRC | 22:47 | |
| *** iokiwi has joined #openstack-infra | 22:47 | |
| *** mgoddard has joined #openstack-infra | 22:47 | |
| *** armax has quit IRC | 22:48 | |
| *** tkajinam has joined #openstack-infra | 22:55 | |
| *** diablo_rojo has joined #openstack-infra | 22:56 | |
| *** Lucas_Gray has quit IRC | 22:56 | |
| dmsimard | clarkb: ack, I'm curious to know more so I'll reach out. | 23:00 |
| *** xek has quit IRC | 23:03 | |
| *** rcernin has joined #openstack-infra | 23:06 | |
| *** threestrands has joined #openstack-infra | 23:26 | |
| *** smarcet has quit IRC | 23:31 | |
| *** dchen has joined #openstack-infra | 23:39 | |
| *** dychen has joined #openstack-infra | 23:43 | |
| *** sthussey has quit IRC | 23:48 | |
| *** armax has joined #openstack-infra | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!