| *** gk__ has joined #openstack-infra | 00:02 | |
| *** jamesmcarthur has joined #openstack-infra | 00:05 | |
| *** andreww has quit IRC | 00:05 | |
| *** olaph1 has joined #openstack-infra | 00:09 | |
| *** salv-orlando has joined #openstack-infra | 00:09 | |
| *** jamesmcarthur has quit IRC | 00:10 | |
| *** olaph has quit IRC | 00:10 | |
| pabelanger | ianw: you online today? | 00:12 |
|---|---|---|
| ianw | yep | 00:12 |
| *** dingyichen has joined #openstack-infra | 00:13 | |
| pabelanger | ianw: k, mind if I share an update on zuul.o.o? | 00:13 |
| ianw | pabelanger: nope ... happy to help if i can | 00:13 |
| *** salv-orlando has quit IRC | 00:13 | |
| pabelanger | basically, http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=64792&rra_id=all is pretty flat now, but if you don't mind keeping an eye out for it rising. I believe most issues with memory might have been around too many topic:zuulv3-projects approved, but we haven't completely confirmed | 00:14 |
| pabelanger | but, when it was swapping, I was able to save queues using the crontab that dmsimard|afk setup. I had issues stopping zuul-scheduler, had to use kill -9, haven't looked into why that was | 00:15 |
| pabelanger | was going to do that in the morning | 00:15 |
| pabelanger | that's basically it for zuul today | 00:16 |
| ianw | ok | 00:17 |
| pabelanger | ianw: 537933 might need to be promoted to top of integrated change queue to help fix a timeout issue with nova, that would help cut down on amount of gate resets | 00:17 |
| pabelanger | ianw: incase mriedem is looking for it | 00:17 |
| ianw | i haven't restored queues from the json, i think i saw some instructions go by? | 00:17 |
| pabelanger | ianw: let me get the syntax I used | 00:17 |
| ianw | 537933 worth a force merge? | 00:18 |
| *** tosky has quit IRC | 00:18 | |
| ianw | i will watch it's progress, if it fails | 00:19 |
| pabelanger | ianw: I had to copy /var/lib/zuul/www/backup/status_<timestamp> to /tmp | 00:19 |
| pabelanger | then ran | 00:19 |
| pabelanger | python /opt/zuul/tools/zuul-changes.py file:///tmp openstack gate >gate.sh | 00:19 |
| pabelanger | and check too | 00:20 |
| pabelanger | then bash -x gate.sh as normal | 00:20 |
| pabelanger | and check | 00:20 |
| *** jamesmcarthur has joined #openstack-infra | 00:20 | |
| ianw | ok, cool | 00:21 |
| pabelanger | a patch would be to symlink status_<timestamp>.json to status.json, then we can just use the /var/lib/zuul/www/backup folder | 00:21 |
| pabelanger | something for another day | 00:21 |
| pabelanger | ianw: good luck :D | 00:21 |
| ianw | thanks :) catch you later | 00:22 |
| *** jamesmcarthur has quit IRC | 00:25 | |
| *** dhill_ has quit IRC | 00:26 | |
| ianw | playing with bringing up a mirror in linaro cloud | 00:31 |
| ianw | ironically, the thing it would benefit from mostly is probably a ... mirror :) | 00:31 |
| ianw | getting to ubuntu upstream repos seems a little unreliable | 00:32 |
| *** felipemonteiro_ has quit IRC | 00:32 | |
| *** dave-mccowan has quit IRC | 00:32 | |
| *** jamesmcarthur has joined #openstack-infra | 00:35 | |
| dmsimard|afk | ianw, pabelanger: I have a patch up to restore from file | 00:35 |
| dmsimard|afk | https://review.openstack.org/#/c/536622/ | 00:36 |
| dmsimard|afk | There is a system-config doc change as well: https://review.openstack.org/#/q/topic:zuul-changes | 00:36 |
| dmsimard|afk | Once that lands, we can consider moving the backups somewhere not web related | 00:37 |
| dmsimard|afk | FWIW I wasn't aware file:/// worked o_O | 00:38 |
| *** olaph1 has quit IRC | 00:38 | |
| *** olaph has joined #openstack-infra | 00:40 | |
| EmilienM | bnemec: ack, thx | 00:41 |
| *** jamesmcarthur has quit IRC | 00:41 | |
| *** shu-mutou has joined #openstack-infra | 00:41 | |
| *** gk__ has quit IRC | 00:42 | |
| *** cuongnv has joined #openstack-infra | 00:51 | |
| openstackgerrit | liusheng proposed openstack-infra/zuul master: Fix AttributeError when handle periodic job with github driver https://review.openstack.org/536645 | 00:52 |
| *** katkapilatova has left #openstack-infra | 00:54 | |
| *** jamesmcarthur has joined #openstack-infra | 00:56 | |
| *** gk__ has joined #openstack-infra | 00:57 | |
| *** andreas_s has joined #openstack-infra | 01:01 | |
| *** kiennt26 has joined #openstack-infra | 01:03 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: GPT partitioning support https://review.openstack.org/533490 | 01:04 |
| *** andreas_s has quit IRC | 01:05 | |
| *** melwitt has quit IRC | 01:10 | |
| *** jroll has quit IRC | 01:10 | |
| *** jamesmcarthur has quit IRC | 01:10 | |
| *** jroll has joined #openstack-infra | 01:11 | |
| *** gyee has quit IRC | 01:11 | |
| *** myoung|bbl is now known as myoung | 01:12 | |
| *** apetrich has quit IRC | 01:20 | |
| *** apetrich has joined #openstack-infra | 01:20 | |
| *** liujiong has joined #openstack-infra | 01:28 | |
| *** Swami has quit IRC | 01:29 | |
| EmilienM | ianw: hey | 01:32 |
| EmilienM | ianw: I'm testing yum puppetlabs mirror | 01:33 |
| EmilienM | ianw: it's failing on http://logs.openstack.org/72/537572/9/check/puppet-openstack-integration-4-scenario001-tempest-centos-7/2873abe/job-output.txt.gz#_2018-01-30_01_06_21_629963 | 01:33 |
| EmilienM | ianw: I wonder if we synced the mirror lately | 01:33 |
| *** jamesmcarthur has joined #openstack-infra | 01:33 | |
| *** salv-orlando has joined #openstack-infra | 01:33 | |
| *** liujiong has quit IRC | 01:36 | |
| *** liujiong has joined #openstack-infra | 01:36 | |
| *** derekjhyang has joined #openstack-infra | 01:37 | |
| *** jamesmcarthur has quit IRC | 01:38 | |
| *** hongbin has joined #openstack-infra | 01:39 | |
| *** salv-orlando has quit IRC | 01:39 | |
| *** jamesmcarthur has joined #openstack-infra | 01:40 | |
| *** felipemonteiro_ has joined #openstack-infra | 01:40 | |
| *** pramodrj07 has quit IRC | 01:43 | |
| *** namnh has joined #openstack-infra | 01:43 | |
| *** jamesmcarthur has quit IRC | 01:44 | |
| *** camunoz has quit IRC | 01:46 | |
| *** liujiong has quit IRC | 01:48 | |
| *** liujiong has joined #openstack-infra | 01:49 | |
| *** jamesmcarthur has joined #openstack-infra | 01:52 | |
| *** slaweq_ has joined #openstack-infra | 01:54 | |
| *** melwitt has joined #openstack-infra | 01:56 | |
| *** melwitt is now known as Guest94306 | 01:56 | |
| *** Guest94306 is now known as jgwentworth | 01:58 | |
| *** slaweq_ has quit IRC | 01:59 | |
| *** jgwentworth is now known as melwitt | 02:01 | |
| *** gcb has joined #openstack-infra | 02:04 | |
| *** pahuang has quit IRC | 02:05 | |
| *** pahuang has joined #openstack-infra | 02:05 | |
| *** yamamoto has joined #openstack-infra | 02:05 | |
| *** efried_hexchat has quit IRC | 02:06 | |
| ianw | EmilienM: i'll check the job in a little | 02:07 |
| ianw | and get back to you | 02:07 |
| EmilienM | ianw: ack | 02:07 |
| EmilienM | ianw: I'm here the next few hours | 02:07 |
| *** harlowja has quit IRC | 02:10 | |
| *** olaph1 has joined #openstack-infra | 02:11 | |
| prometheanfire | is there a bug I can subscribe to for why gate keeps on backing up? | 02:11 |
| *** gk__ has quit IRC | 02:12 | |
| *** olaph has quit IRC | 02:12 | |
| clarkb | prometheanfire: I dont think it wa sa single thing. log server lost a volume. eventlet broke some projects. zuul OOM'd. neutron unittests timeout. nova functional tests time out and so on | 02:13 |
| clarkb | thus is pretty normal for a feature freeze. We've just not had a busy feature freeze in a while :/ | 02:13 |
| *** ekcs has quit IRC | 02:14 | |
| prometheanfire | we haven't uncapped eventlet was that an os package thing? | 02:15 |
| *** jamesmcarthur has quit IRC | 02:15 | |
| prometheanfire | fair enough thoug :D | 02:15 |
| clarkb | ya it was some ubunut python update interaction that newer eventlet fixed | 02:16 |
| clarkb | I'm not sure what actions have been taken to address it | 02:16 |
| *** rossella_s has quit IRC | 02:16 | |
| prometheanfire | well, eventlet was on my hopes for uncapping this cycle, but didn't work out | 02:17 |
| prometheanfire | that's all | 02:17 |
| prometheanfire | it's on the shortlist for rocky | 02:17 |
| *** pramodrj07 has joined #openstack-infra | 02:17 | |
| *** rloo has quit IRC | 02:17 | |
| *** jamesmcarthur has joined #openstack-infra | 02:18 | |
| *** pramodrj07 has quit IRC | 02:18 | |
| *** felipemonteiro_ has quit IRC | 02:18 | |
| *** pramodrj07 has joined #openstack-infra | 02:18 | |
| *** efried_hexchat has joined #openstack-infra | 02:19 | |
| *** rossella_s has joined #openstack-infra | 02:20 | |
| *** jamesmcarthur has quit IRC | 02:22 | |
| *** jamesmcarthur has joined #openstack-infra | 02:23 | |
| *** ykarel|away has joined #openstack-infra | 02:23 | |
| *** slaweq_ has joined #openstack-infra | 02:29 | |
| *** Pramod has joined #openstack-infra | 02:31 | |
| *** pramodrj07 has quit IRC | 02:31 | |
| *** rlandy|bbl is now known as rlandy | 02:31 | |
| *** jamesmcarthur has quit IRC | 02:33 | |
| *** slaweq_ has quit IRC | 02:34 | |
| *** Pramod has quit IRC | 02:37 | |
| *** Pramod has joined #openstack-infra | 02:37 | |
| *** ykarel|away has quit IRC | 02:38 | |
| *** Pramod has quit IRC | 02:39 | |
| *** pramodrj07 has joined #openstack-infra | 02:39 | |
| *** olaph has joined #openstack-infra | 02:44 | |
| *** olaph1 has quit IRC | 02:45 | |
| *** hongbin has quit IRC | 02:46 | |
| *** yamamoto_ has joined #openstack-infra | 02:47 | |
| *** hongbin has joined #openstack-infra | 02:47 | |
| *** jamesmcarthur has joined #openstack-infra | 02:48 | |
| *** yamamoto has quit IRC | 02:50 | |
| *** d0ugal has quit IRC | 02:51 | |
| *** d0ugal has joined #openstack-infra | 02:52 | |
| *** jamesmcarthur has quit IRC | 02:53 | |
| *** edmondsw has joined #openstack-infra | 02:54 | |
| *** flwang has quit IRC | 02:55 | |
| *** ramishra has joined #openstack-infra | 02:55 | |
| *** edmondsw has quit IRC | 02:58 | |
| *** dave-mccowan has joined #openstack-infra | 02:58 | |
| *** cshastri has joined #openstack-infra | 02:59 | |
| *** zhurong has joined #openstack-infra | 03:03 | |
| *** yamahata has quit IRC | 03:05 | |
| *** armax has quit IRC | 03:08 | |
| *** apetrich has quit IRC | 03:09 | |
| *** apetrich has joined #openstack-infra | 03:13 | |
| *** cshastri has quit IRC | 03:15 | |
| *** jamesmcarthur has joined #openstack-infra | 03:20 | |
| *** psachin has joined #openstack-infra | 03:25 | |
| *** jamesmcarthur has quit IRC | 03:25 | |
| *** uberjay has quit IRC | 03:28 | |
| *** uberjay has joined #openstack-infra | 03:29 | |
| *** agopi|out has joined #openstack-infra | 03:31 | |
| *** dhajare_ has joined #openstack-infra | 03:34 | |
| ianw | EmilienM: ok, the mirror job seems to be going | 03:34 |
| EmilienM | ianw: syncing you mean? | 03:35 |
| ianw | well, running, yeah | 03:35 |
| ianw | that file isn't upstream? http://yum.puppetlabs.com/el/7/dependencies/ | 03:36 |
| *** salv-orlando has joined #openstack-infra | 03:36 | |
| *** yamamoto has joined #openstack-infra | 03:36 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config master: Remove reno jobs for some tripleo projects https://review.openstack.org/539081 | 03:36 |
| *** slaweq_ has joined #openstack-infra | 03:37 | |
| EmilienM | ianw: weird | 03:38 |
| EmilienM | ianw: yum trying to find it | 03:38 |
| EmilienM | let me check repo config | 03:38 |
| EmilienM | http://logs.openstack.org/72/537572/9/check/puppet-openstack-integration-4-scenario001-tempest-centos-7/2873abe/logs/repolist.txt.gz | 03:39 |
| EmilienM | sounds broken | 03:39 |
| *** yamamoto_ has quit IRC | 03:39 | |
| EmilienM | config is here: http://logs.openstack.org/72/537572/9/check/puppet-openstack-integration-4-scenario002-tempest-centos-7/8c9a7f2/job-output.txt.gz#_2018-01-30_01_16_00_923179 | 03:39 |
| EmilienM | let me find why | 03:40 |
| *** salv-orlando has quit IRC | 03:41 | |
| *** rlandy has quit IRC | 03:42 | |
| *** cshastri has joined #openstack-infra | 03:42 | |
| *** slaweq_ has quit IRC | 03:42 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/system-config master: Add aarch64 sources configuration https://review.openstack.org/539083 | 03:42 |
| *** uberjay has quit IRC | 03:43 | |
| EmilienM | ianw: I'll try without the second repo | 03:43 |
| EmilienM | I found out that our existing jobs don't use it | 03:43 |
| *** jamesmcarthur has joined #openstack-infra | 03:44 | |
| *** hongbin has quit IRC | 03:44 | |
| *** uberjay has joined #openstack-infra | 03:45 | |
| EmilienM | ianw: it's weird $basearch isn't use by the second repo | 03:46 |
| ianw | hmm, it's all noarch ruby stuff? | 03:46 |
| EmilienM | not sure | 03:47 |
| EmilienM | anyway, trying without this repo: https://review.openstack.org/#/c/537572/ | 03:47 |
| EmilienM | let's see how that works | 03:47 |
| *** uberjay has quit IRC | 03:47 | |
| *** uberjay has joined #openstack-infra | 03:47 | |
| *** yamahata has joined #openstack-infra | 03:47 | |
| *** jamesmcarthur has quit IRC | 03:48 | |
| *** daidv has quit IRC | 03:50 | |
| *** daidv has joined #openstack-infra | 03:50 | |
| *** dave-mccowan has quit IRC | 03:52 | |
| EmilienM | ianw: can you please review https://review.openstack.org/#/c/539081/ ? | 03:53 |
| *** jamesmcarthur has joined #openstack-infra | 03:57 | |
| *** slaweq_ has joined #openstack-infra | 03:58 | |
| *** coolsvap has joined #openstack-infra | 04:00 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/storyboard master: Add MQTT notification publisher https://review.openstack.org/538575 | 04:02 |
| *** slaweq_ has quit IRC | 04:03 | |
| *** jamesmcarthur has quit IRC | 04:03 | |
| *** sree has joined #openstack-infra | 04:04 | |
| *** olaph1 has joined #openstack-infra | 04:04 | |
| *** zhurong has quit IRC | 04:05 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/storyboard master: Add MQTT notification publisher https://review.openstack.org/538575 | 04:05 |
| *** jamesmcarthur has joined #openstack-infra | 04:05 | |
| *** olaph has quit IRC | 04:06 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a static driver for Nodepool https://review.openstack.org/535553 | 04:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: nodeutils: use socket.getaddrinfo instead of ipaddress https://review.openstack.org/539086 | 04:09 |
| *** mylu has quit IRC | 04:09 | |
| *** jamesmcarthur has quit IRC | 04:10 | |
| *** links has joined #openstack-infra | 04:10 | |
| EmilienM | ianw: same problem with http://logs.openstack.org/72/537572/11/check/puppet-openstack-integration-4-scenario001-tempest-centos-7/702106e/job-output.txt.gz#_2018-01-30_03_54_23_730924 | 04:13 |
| EmilienM | mhh | 04:13 |
| EmilienM | I feel like $basearch is interpreted | 04:15 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a static driver for Nodepool https://review.openstack.org/535553 | 04:16 |
| *** adriant has quit IRC | 04:18 | |
| *** jamesmcarthur has joined #openstack-infra | 04:18 | |
| EmilienM | ianw: nevermind, I think I used the wrong repo, let me fix | 04:20 |
| *** harlowja has joined #openstack-infra | 04:21 | |
| *** mylu has joined #openstack-infra | 04:21 | |
| *** jamesmcarthur has quit IRC | 04:22 | |
| *** armax has joined #openstack-infra | 04:24 | |
| *** harlowja has quit IRC | 04:26 | |
| *** harlowja has joined #openstack-infra | 04:26 | |
| *** harlowja has quit IRC | 04:27 | |
| *** armax has quit IRC | 04:28 | |
| *** jamesmcarthur has joined #openstack-infra | 04:30 | |
| *** adriant has joined #openstack-infra | 04:30 | |
| *** ykarel has joined #openstack-infra | 04:31 | |
| *** armax has joined #openstack-infra | 04:35 | |
| *** jamesmcarthur has quit IRC | 04:36 | |
| *** salv-orlando has joined #openstack-infra | 04:37 | |
| *** pgadiya has joined #openstack-infra | 04:38 | |
| *** rosmaita has quit IRC | 04:39 | |
| *** jamesmcarthur has joined #openstack-infra | 04:40 | |
| *** salv-orlando has quit IRC | 04:42 | |
| *** edmondsw has joined #openstack-infra | 04:42 | |
| *** jamesmcarthur has quit IRC | 04:44 | |
| *** andreas_s has joined #openstack-infra | 04:45 | |
| *** apetrich has quit IRC | 04:45 | |
| *** edmondsw has quit IRC | 04:46 | |
| *** apetrich has joined #openstack-infra | 04:46 | |
| EmilienM | ianw: i've got something working | 04:47 |
| * EmilienM off | 04:47 | |
| *** jamesmcarthur has joined #openstack-infra | 04:48 | |
| *** andreas_s has quit IRC | 04:49 | |
| ianw | ok, good :) happy to help if the mirror needs tweaking | 04:50 |
| *** jamesmcarthur has quit IRC | 04:53 | |
| *** jamesmcarthur has joined #openstack-infra | 04:58 | |
| *** jamesmcarthur has quit IRC | 05:02 | |
| *** armax has quit IRC | 05:05 | |
| *** mylu_ has joined #openstack-infra | 05:06 | |
| *** armax has joined #openstack-infra | 05:06 | |
| *** pcichy has quit IRC | 05:07 | |
| *** mylu has quit IRC | 05:08 | |
| *** slaweq_ has joined #openstack-infra | 05:12 | |
| *** jamesmcarthur has joined #openstack-infra | 05:13 | |
| *** sshnaidm|afk has quit IRC | 05:14 | |
| *** slaweq_ has quit IRC | 05:16 | |
| openstackgerrit | Merged openstack-dev/pbr master: Updated from global requirements https://review.openstack.org/537022 | 05:17 |
| *** jamesmcarthur has quit IRC | 05:19 | |
| *** cuongnv has quit IRC | 05:22 | |
| *** kjackal has joined #openstack-infra | 05:23 | |
| *** kjackal has quit IRC | 05:23 | |
| *** jamesmcarthur has joined #openstack-infra | 05:24 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a static driver for Nodepool https://review.openstack.org/535553 | 05:27 |
| *** cuongnv has joined #openstack-infra | 05:27 | |
| *** agopi|out has quit IRC | 05:30 | |
| *** jamesmcarthur has quit IRC | 05:30 | |
| *** sshnaidm has joined #openstack-infra | 05:31 | |
| *** pramodrj07 has quit IRC | 05:32 | |
| *** pramodrj07 has joined #openstack-infra | 05:33 | |
| *** jamesmcarthur has joined #openstack-infra | 05:34 | |
| *** salv-orlando has joined #openstack-infra | 05:38 | |
| *** jamesmcarthur has quit IRC | 05:38 | |
| prometheanfire | AJaeger: old restart or new one? | 05:38 |
| *** jamesmcarthur has joined #openstack-infra | 05:41 | |
| *** zhurong has joined #openstack-infra | 05:42 | |
| *** ekcs has joined #openstack-infra | 05:43 | |
| *** salv-orlando has quit IRC | 05:43 | |
| *** sshnaidm has quit IRC | 05:43 | |
| openstackgerrit | Merged openstack-infra/project-config master: Remove reno jobs for some tripleo projects https://review.openstack.org/539081 | 05:43 |
| *** agopi has joined #openstack-infra | 05:44 | |
| *** bhavik1 has joined #openstack-infra | 05:45 | |
| *** jamesmcarthur has quit IRC | 05:46 | |
| AJaeger | prometheanfire: what do you mean? | 05:47 |
| *** jamesmcarthur has joined #openstack-infra | 05:48 | |
| *** Pramod has joined #openstack-infra | 05:49 | |
| AJaeger | ianw: could you review https://review.openstack.org/#/c/538182/ and https://review.openstack.org/#/c/538185/ , please? Those are needed for queens release | 05:51 |
| *** esberglu_ has joined #openstack-infra | 05:51 | |
| *** pramodrj07 has quit IRC | 05:52 | |
| *** bhavik1 has quit IRC | 05:53 | |
| *** jamesmcarthur has quit IRC | 05:53 | |
| *** jamesmcarthur has joined #openstack-infra | 05:54 | |
| *** esberglu has quit IRC | 05:55 | |
| *** salv-orlando has joined #openstack-infra | 05:59 | |
| *** jamesmcarthur has quit IRC | 05:59 | |
| prometheanfire | you recheched something 30 min ago | 06:00 |
| *** slaweq has quit IRC | 06:00 | |
| *** jappleii__ has quit IRC | 06:01 | |
| AJaeger | prometheanfire: ah - that was for the restart from yesterday around 14:00 UTC | 06:02 |
| *** sshnaidm has joined #openstack-infra | 06:02 | |
| *** olaph has joined #openstack-infra | 06:05 | |
| *** olaph1 has quit IRC | 06:05 | |
| prometheanfire | ah, guess it was missed, thought the others went through | 06:06 |
| *** jamesmcarthur has joined #openstack-infra | 06:07 | |
| *** dsariel has joined #openstack-infra | 06:10 | |
| *** jamesmcarthur has quit IRC | 06:12 | |
| *** bhavik1 has joined #openstack-infra | 06:14 | |
| *** mylu_ has quit IRC | 06:16 | |
| *** slaweq has joined #openstack-infra | 06:16 | |
| *** slaweq_ has joined #openstack-infra | 06:17 | |
| *** jamesmcarthur has joined #openstack-infra | 06:20 | |
| *** dsariel has quit IRC | 06:21 | |
| *** jaosorior has joined #openstack-infra | 06:21 | |
| *** slaweq_ has quit IRC | 06:21 | |
| *** slaweq has quit IRC | 06:21 | |
| *** jtomasek has quit IRC | 06:24 | |
| *** jamesmcarthur has quit IRC | 06:24 | |
| *** jtomasek has joined #openstack-infra | 06:24 | |
| *** slaweq has joined #openstack-infra | 06:24 | |
| *** pgadiya has quit IRC | 06:27 | |
| *** edmondsw has joined #openstack-infra | 06:30 | |
| *** janki has joined #openstack-infra | 06:31 | |
| *** dbecker has quit IRC | 06:31 | |
| *** slaweq has quit IRC | 06:31 | |
| *** jamesmcarthur has joined #openstack-infra | 06:32 | |
| *** pgadiya has joined #openstack-infra | 06:32 | |
| *** edmondsw has quit IRC | 06:34 | |
| *** jamesmcarthur has quit IRC | 06:36 | |
| *** bhavik1 has quit IRC | 06:36 | |
| *** pgadiya has quit IRC | 06:41 | |
| *** pgadiya has joined #openstack-infra | 06:41 | |
| *** yamamoto_ has joined #openstack-infra | 06:43 | |
| ianw | zuul's memory usage looking pleasingly static | 06:43 |
| *** jamesmcarthur has joined #openstack-infra | 06:44 | |
| *** dbecker has joined #openstack-infra | 06:45 | |
| *** yamamoto has quit IRC | 06:46 | |
| *** Pramod has quit IRC | 06:48 | |
| *** jamesmcarthur has quit IRC | 06:49 | |
| openstackgerrit | Merged openstack-infra/project-config master: Use horizon publish job for blazar-dashboard https://review.openstack.org/538182 | 06:52 |
| *** jamesmcarthur has joined #openstack-infra | 06:52 | |
| *** esberglu has joined #openstack-infra | 06:55 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Remove legacy ec2-api jobs https://review.openstack.org/539109 | 06:56 |
| openstackgerrit | Merged openstack-infra/project-config master: Add publish-to-pypi in blazar-nova repo https://review.openstack.org/538185 | 06:56 |
| *** jamesmcarthur has quit IRC | 06:57 | |
| *** esberglu_ has quit IRC | 06:57 | |
| *** ekcs has quit IRC | 06:58 | |
| *** zhurong has quit IRC | 06:58 | |
| *** jamesmcarthur has joined #openstack-infra | 06:59 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/openstack-zuul-jobs master: Remove legacy ec2-api jobs https://review.openstack.org/539111 | 07:00 |
| *** zhurong has joined #openstack-infra | 07:02 | |
| openstackgerrit | Ghanshyam Mann proposed openstack-infra/project-config master: Remove legacy ec2-api jobs https://review.openstack.org/539109 | 07:03 |
| *** jamesmcarthur has quit IRC | 07:03 | |
| *** jtomasek has quit IRC | 07:05 | |
| *** ramishra has quit IRC | 07:07 | |
| *** jamesmcarthur has joined #openstack-infra | 07:09 | |
| *** ramishra has joined #openstack-infra | 07:09 | |
| *** pcaruana has joined #openstack-infra | 07:10 | |
| AJaeger | ianw, prometheanfire : Thanks to that static memory usage, I'll recheck one by one. | 07:10 |
| AJaeger | prometheanfire: we added all changes back after the restart with the exception of some that change zuul configs since we suspected them to be responsible. So, I'm rechecking them now.. | 07:11 |
| *** jamesmcarthur has quit IRC | 07:14 | |
| prometheanfire | AJaeger: do I need to recheck https://review.openstack.org/538842 ? | 07:15 |
| *** jamesmcarthur has joined #openstack-infra | 07:17 | |
| *** andreas_s has joined #openstack-infra | 07:17 | |
| AJaeger | prometheanfire: check http://zuul.openstack.org/ - and you see it's in the gate pipeline. But this is so long that the "Starting gate" message was not even send ;( So, nothing to do for this one | 07:20 |
| prometheanfire | ok | 07:20 |
| prometheanfire | that's what I though (been looking at the queue) | 07:20 |
| *** jamesmcarthur has quit IRC | 07:21 | |
| *** jamesmcarthur has joined #openstack-infra | 07:23 | |
| *** jtomasek has joined #openstack-infra | 07:27 | |
| *** annp has joined #openstack-infra | 07:27 | |
| *** jamesmcarthur has quit IRC | 07:29 | |
| *** janki is now known as janki-afk | 07:30 | |
| *** iyamahat has joined #openstack-infra | 07:30 | |
| *** jamesmcarthur has joined #openstack-infra | 07:32 | |
| *** d0ugal has quit IRC | 07:34 | |
| *** salv-orlando has quit IRC | 07:36 | |
| *** e0ne has joined #openstack-infra | 07:36 | |
| *** salv-orlando has joined #openstack-infra | 07:36 | |
| *** janki-afk is now known as janki | 07:36 | |
| *** jamesmcarthur has quit IRC | 07:37 | |
| *** slaweq has joined #openstack-infra | 07:39 | |
| *** dingyichen has quit IRC | 07:40 | |
| *** salv-orlando has quit IRC | 07:41 | |
| *** armax has quit IRC | 07:41 | |
| *** dingyichen has joined #openstack-infra | 07:43 | |
| *** agopi has quit IRC | 07:44 | |
| *** agopi has joined #openstack-infra | 07:44 | |
| *** slaweq has quit IRC | 07:44 | |
| *** jamesmcarthur has joined #openstack-infra | 07:46 | |
| *** alexchadin has joined #openstack-infra | 07:49 | |
| *** jamesmcarthur has quit IRC | 07:50 | |
| *** jamesmcarthur has joined #openstack-infra | 07:52 | |
| *** slaweq has joined #openstack-infra | 07:54 | |
| *** jtomasek has quit IRC | 07:54 | |
| *** jamesmcarthur has quit IRC | 07:56 | |
| *** slaweq has quit IRC | 07:58 | |
| *** jamesmcarthur has joined #openstack-infra | 07:59 | |
| *** salv-orlando has joined #openstack-infra | 07:59 | |
| *** tmorin has joined #openstack-infra | 08:01 | |
| *** d0ugal has joined #openstack-infra | 08:02 | |
| *** tmorin has quit IRC | 08:02 | |
| *** tmorin1 has joined #openstack-infra | 08:02 | |
| *** jamesmcarthur has quit IRC | 08:06 | |
| *** b_bezak has joined #openstack-infra | 08:07 | |
| *** jamesmcarthur has joined #openstack-infra | 08:11 | |
| *** slaweq has joined #openstack-infra | 08:11 | |
| *** oidgar has joined #openstack-infra | 08:13 | |
| *** sshnaidm is now known as sshnaidm|afk | 08:15 | |
| *** jamesmcarthur has quit IRC | 08:15 | |
| *** stakeda has quit IRC | 08:16 | |
| *** tesseract has joined #openstack-infra | 08:17 | |
| *** edmondsw has joined #openstack-infra | 08:18 | |
| *** armaan_ has quit IRC | 08:19 | |
| *** aviau has quit IRC | 08:19 | |
| *** aviau has joined #openstack-infra | 08:19 | |
| *** armaan has joined #openstack-infra | 08:19 | |
| *** edmondsw has quit IRC | 08:23 | |
| *** ralonsoh has joined #openstack-infra | 08:25 | |
| *** jamesmcarthur has joined #openstack-infra | 08:26 | |
| *** tmorin has joined #openstack-infra | 08:28 | |
| *** slaweq_ has joined #openstack-infra | 08:28 | |
| *** tmorin1 has quit IRC | 08:28 | |
| *** jamesmcarthur has quit IRC | 08:30 | |
| *** slaweq_ has quit IRC | 08:32 | |
| *** dingyichen has quit IRC | 08:36 | |
| *** jamesmcarthur has joined #openstack-infra | 08:37 | |
| *** kjackal has joined #openstack-infra | 08:40 | |
| cmurphy | i think the gate is broke https://zuul.openstack.org/builds.html | 08:41 |
| cmurphy | specifically the novnc package on ubuntu http://logs.openstack.org/84/537784/4/check/tempest-full-py3/05fec34/job-output.txt.gz#_2018-01-30_07_07_20_473050 | 08:41 |
| *** jamesmcarthur has quit IRC | 08:41 | |
| prometheanfire | I just got a notification from gate | 08:43 |
| prometheanfire | so not totallt broken | 08:43 |
| *** iyamahat has quit IRC | 08:43 | |
| *** dklyle has joined #openstack-infra | 08:43 | |
| *** david-lyle has quit IRC | 08:44 | |
| *** jamesmcarthur has joined #openstack-infra | 08:45 | |
| *** s-shiono has quit IRC | 08:45 | |
| *** jamesmcarthur has quit IRC | 08:50 | |
| *** andreas_s has quit IRC | 08:50 | |
| *** lennyb has quit IRC | 08:50 | |
| *** andreas_s has joined #openstack-infra | 08:50 | |
| *** jpena|off is now known as jpena | 08:50 | |
| cmurphy | novnc seems okay to me, maybe the problem already fixed itself | 08:53 |
| AJaeger | fixed by another mirror sync? | 08:54 |
| cmurphy | maybe? | 08:54 |
| *** eyalb has joined #openstack-infra | 08:55 | |
| AJaeger | hopefully ;) | 08:56 |
| * cmurphy crosses fingers | 08:56 | |
| *** jamesmcarthur has joined #openstack-infra | 08:57 | |
| *** jpich has joined #openstack-infra | 08:58 | |
| openstackgerrit | Balazs Gibizer proposed openstack-infra/project-config master: consolidate nova job definitions https://review.openstack.org/538908 | 08:58 |
| *** priteau has joined #openstack-infra | 08:59 | |
| *** slaweq_ has joined #openstack-infra | 09:00 | |
| *** vivsoni has quit IRC | 09:00 | |
| *** vivsoni_ has joined #openstack-infra | 09:00 | |
| *** jamesmcarthur has quit IRC | 09:01 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:04 | |
| *** amoralej|off is now known as amoralej | 09:04 | |
| *** slaweq_ has quit IRC | 09:04 | |
| *** jamesmcarthur has joined #openstack-infra | 09:07 | |
| *** yamahata has quit IRC | 09:08 | |
| *** jamesmcarthur has quit IRC | 09:12 | |
| *** arxcruz|off is now known as arxcruz | 09:12 | |
| *** florianf has joined #openstack-infra | 09:12 | |
| *** matbu has quit IRC | 09:14 | |
| *** jamesmcarthur has joined #openstack-infra | 09:16 | |
| *** florianf has quit IRC | 09:17 | |
| *** salv-orlando has quit IRC | 09:17 | |
| *** salv-orlando has joined #openstack-infra | 09:18 | |
| *** jamesmcarthur has quit IRC | 09:20 | |
| *** shardy has joined #openstack-infra | 09:20 | |
| *** ijw has joined #openstack-infra | 09:21 | |
| *** salv-orlando has quit IRC | 09:23 | |
| *** ijw has quit IRC | 09:26 | |
| *** shu-mutou is now known as shu-mutou-AWAY | 09:26 | |
| *** hrw has joined #openstack-infra | 09:29 | |
| *** janki has quit IRC | 09:30 | |
| *** janki has joined #openstack-infra | 09:30 | |
| *** jamesmcarthur has joined #openstack-infra | 09:30 | |
| *** kiennt26 has quit IRC | 09:30 | |
| *** andreas_s_ has joined #openstack-infra | 09:31 | |
| *** lennyb has joined #openstack-infra | 09:34 | |
| *** andreas_s has quit IRC | 09:35 | |
| *** jamesmcarthur has quit IRC | 09:36 | |
| *** oidgar has quit IRC | 09:40 | |
| *** jamesmcarthur has joined #openstack-infra | 09:41 | |
| *** derekh has joined #openstack-infra | 09:42 | |
| *** verdurin_ is now known as verdurin | 09:45 | |
| cmurphy | looks still broken :/ http://logs.openstack.org/46/525346/35/check/tempest-full/5ca494b/job-output.txt.gz#_2018-01-30_09_26_15_599484 | 09:46 |
| *** jamesmcarthur has quit IRC | 09:46 | |
| *** namnh has quit IRC | 09:51 | |
| *** cuongnv has quit IRC | 09:51 | |
| *** daidv has quit IRC | 09:51 | |
| *** daidv has joined #openstack-infra | 09:51 | |
| *** namnh has joined #openstack-infra | 09:51 | |
| *** cuongnv has joined #openstack-infra | 09:51 | |
| *** kopecmartin has joined #openstack-infra | 09:54 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 09:54 |
| *** jamesmcarthur has joined #openstack-infra | 09:55 | |
| eyalb | we also have the same problem http://logs.openstack.org/16/538816/1/gate/vitrage-dsvm-api-py27/b63f990/logs/devstacklog.txt.gz#_2018-01-30_07_38_41_362 | 09:58 |
| *** armaan has quit IRC | 09:59 | |
| *** matbu has joined #openstack-infra | 09:59 | |
| *** jamesmcarthur has quit IRC | 09:59 | |
| *** jtomasek has joined #openstack-infra | 10:00 | |
| *** sambetts|afk is now known as sambetts | 10:01 | |
| *** zhurong has quit IRC | 10:02 | |
| *** ijw has joined #openstack-infra | 10:02 | |
| *** pgadiya has quit IRC | 10:03 | |
| *** oidgar has joined #openstack-infra | 10:04 | |
| *** edmondsw has joined #openstack-infra | 10:06 | |
| cmurphy | not finding any recent changes in the upstream novnc or python-numpy or websockify packages | 10:06 |
| *** ijw has quit IRC | 10:07 | |
| *** yamamoto has joined #openstack-infra | 10:08 | |
| *** liujiong has quit IRC | 10:08 | |
| *** armaan has joined #openstack-infra | 10:08 | |
| *** jamesmcarthur has joined #openstack-infra | 10:09 | |
| *** edmondsw has quit IRC | 10:10 | |
| bauzas | is it me or the top changes in the gate pipeline are stuck ? | 10:10 |
| bauzas | speaking of the integrated queue | 10:11 |
| *** yamamoto_ has quit IRC | 10:11 | |
| bauzas | oh no | 10:11 |
| *** yamamoto has quit IRC | 10:12 | |
| bauzas | the top one is a requirements patch, which is potentially holding the nova change at the bottom | 10:12 |
| *** edmondsw has joined #openstack-infra | 10:12 | |
| *** edmondsw has quit IRC | 10:12 | |
| *** jamesmcarthur has quit IRC | 10:14 | |
| *** yamamoto has joined #openstack-infra | 10:14 | |
| *** yamamoto has quit IRC | 10:14 | |
| *** jamesmcarthur has joined #openstack-infra | 10:16 | |
| *** hjensas is now known as hjensas|lunch | 10:17 | |
| *** rubasov has joined #openstack-infra | 10:20 | |
| *** pgadiya has joined #openstack-infra | 10:21 | |
| *** jamesmcarthur has quit IRC | 10:21 | |
| *** olaph1 has joined #openstack-infra | 10:21 | |
| *** olaph has quit IRC | 10:23 | |
| *** cuongnv has quit IRC | 10:24 | |
| *** rubasov has left #openstack-infra | 10:24 | |
| *** jamesmcarthur has joined #openstack-infra | 10:28 | |
| *** sree has quit IRC | 10:29 | |
| *** matbu has quit IRC | 10:30 | |
| *** alexchadin has quit IRC | 10:30 | |
| *** salv-orlando has joined #openstack-infra | 10:31 | |
| *** alexchadin has joined #openstack-infra | 10:31 | |
| *** dave-mccowan has joined #openstack-infra | 10:32 | |
| *** jamesmcarthur has quit IRC | 10:32 | |
| *** andreas_s has joined #openstack-infra | 10:33 | |
| *** pgadiya has quit IRC | 10:33 | |
| *** ijw has joined #openstack-infra | 10:35 | |
| *** tosky has joined #openstack-infra | 10:35 | |
| *** andreas_s_ has quit IRC | 10:36 | |
| *** jamesmcarthur has joined #openstack-infra | 10:38 | |
| *** ijw has quit IRC | 10:39 | |
| *** lucas-afk is now known as lucasagomes | 10:41 | |
| *** ganso has joined #openstack-infra | 10:43 | |
| *** tellesnobrega has quit IRC | 10:43 | |
| *** jamesmcarthur has quit IRC | 10:43 | |
| *** tpsilva has joined #openstack-infra | 10:43 | |
| *** ldnunes has joined #openstack-infra | 10:43 | |
| *** pbourke has quit IRC | 10:44 | |
| *** olaph has joined #openstack-infra | 10:46 | |
| *** olaph1 has quit IRC | 10:46 | |
| *** pbourke has joined #openstack-infra | 10:46 | |
| lyarwood | also seeing the novnc deps issue here FWIW | 10:46 |
| *** alexchadin has quit IRC | 10:47 | |
| *** slaweq_ has joined #openstack-infra | 10:48 | |
| *** alexchadin has joined #openstack-infra | 10:49 | |
| *** armaan has quit IRC | 10:52 | |
| *** slaweq_ has quit IRC | 10:52 | |
| *** oidgar has quit IRC | 10:52 | |
| *** armaan has joined #openstack-infra | 10:53 | |
| *** jamesmcarthur has joined #openstack-infra | 10:54 | |
| *** slaweq_ has joined #openstack-infra | 10:54 | |
| *** andreas_s_ has joined #openstack-infra | 10:55 | |
| *** andreas_s has quit IRC | 10:58 | |
| *** jamesmcarthur has quit IRC | 10:58 | |
| *** yamamoto has joined #openstack-infra | 10:59 | |
| *** slaweq_ has quit IRC | 11:02 | |
| *** alexchadin has quit IRC | 11:02 | |
| *** alexchadin has joined #openstack-infra | 11:03 | |
| *** alexchadin has quit IRC | 11:03 | |
| *** alexchadin has joined #openstack-infra | 11:03 | |
| *** alexchadin has quit IRC | 11:04 | |
| *** yamamoto has quit IRC | 11:04 | |
| *** oidgar has joined #openstack-infra | 11:04 | |
| *** alexchadin has joined #openstack-infra | 11:04 | |
| *** yamamoto has joined #openstack-infra | 11:04 | |
| *** alexchadin has quit IRC | 11:04 | |
| *** jamesmcarthur has joined #openstack-infra | 11:05 | |
| *** namnh has quit IRC | 11:05 | |
| *** alexchadin has joined #openstack-infra | 11:05 | |
| *** alexchadin has quit IRC | 11:05 | |
| *** alexchadin has joined #openstack-infra | 11:06 | |
| *** alexchadin has quit IRC | 11:06 | |
| *** jamesmcarthur has quit IRC | 11:09 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 11:11 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul web: add admin endpoint, enqueue commands https://review.openstack.org/539004 | 11:15 |
| *** jamesmcarthur has joined #openstack-infra | 11:16 | |
| *** hjensas|lunch is now known as hjensas | 11:18 | |
| *** andreas_s_ has quit IRC | 11:19 | |
| *** jamesmcarthur has quit IRC | 11:20 | |
| *** andreas_s has joined #openstack-infra | 11:20 | |
| *** jamesmcarthur has joined #openstack-infra | 11:22 | |
| *** rfolco|off is now known as rfolco | 11:27 | |
| *** jamesmcarthur has quit IRC | 11:28 | |
| *** jamesmcarthur has joined #openstack-infra | 11:30 | |
| *** salv-orlando has quit IRC | 11:32 | |
| *** salv-orlando has joined #openstack-infra | 11:32 | |
| *** Nil_ has quit IRC | 11:33 | |
| *** d0ugal has quit IRC | 11:34 | |
| *** jamesmcarthur has quit IRC | 11:35 | |
| *** armaan has quit IRC | 11:35 | |
| *** armaan has joined #openstack-infra | 11:35 | |
| *** yamamoto has quit IRC | 11:36 | |
| *** salv-orlando has quit IRC | 11:37 | |
| *** ijw has joined #openstack-infra | 11:40 | |
| *** dsariel has joined #openstack-infra | 11:41 | |
| *** ijw has quit IRC | 11:45 | |
| *** jamesmcarthur has joined #openstack-infra | 11:45 | |
| *** yamamoto has joined #openstack-infra | 11:46 | |
| *** andreas_s has quit IRC | 11:47 | |
| *** andreas_s has joined #openstack-infra | 11:47 | |
| *** alexchadin has joined #openstack-infra | 11:49 | |
| *** jamesmcarthur has quit IRC | 11:50 | |
| *** yamamoto has quit IRC | 11:51 | |
| *** dhajare_ has quit IRC | 11:52 | |
| *** d0ugal has joined #openstack-infra | 11:54 | |
| *** d0ugal has quit IRC | 11:54 | |
| *** d0ugal has joined #openstack-infra | 11:54 | |
| *** jamesmcarthur has joined #openstack-infra | 11:54 | |
| *** jamesmcarthur has quit IRC | 11:59 | |
| *** tellesnobrega has joined #openstack-infra | 12:03 | |
| *** jamesmcarthur has joined #openstack-infra | 12:05 | |
| *** ijw has joined #openstack-infra | 12:08 | |
| *** edmondsw has joined #openstack-infra | 12:08 | |
| *** ijw has quit IRC | 12:09 | |
| *** ijw has joined #openstack-infra | 12:10 | |
| *** jamesmcarthur has quit IRC | 12:10 | |
| *** yamamoto has joined #openstack-infra | 12:10 | |
| *** zhurong has joined #openstack-infra | 12:13 | |
| *** yamamoto has quit IRC | 12:14 | |
| *** alexchadin has quit IRC | 12:17 | |
| *** alexchadin has joined #openstack-infra | 12:18 | |
| *** sree has joined #openstack-infra | 12:19 | |
| *** coolsvap has quit IRC | 12:19 | |
| *** jamesmcarthur has joined #openstack-infra | 12:20 | |
| *** alexchadin has quit IRC | 12:22 | |
| *** alexchadin has joined #openstack-infra | 12:22 | |
| *** jamesmcarthur has quit IRC | 12:25 | |
| *** agopi has quit IRC | 12:25 | |
| *** liusheng has joined #openstack-infra | 12:26 | |
| *** jamesmcarthur has joined #openstack-infra | 12:27 | |
| tosky | anyone else noticed issues with the xenial images and devstack? | 12:27 |
| tosky | I see "The following packages have unmet dependencies:" "novnc : Depends: python-numpy but it is not going to be installed" "Depends: websockify but it is not going to be installed" | 12:28 |
| tosky | namely here https://review.openstack.org/#/c/304019/ | 12:28 |
| * tosky bbl | 12:28 | |
| *** bfournie has quit IRC | 12:29 | |
| tosky | and I'm not the only one, see https://review.openstack.org/531206 (that's your job, chandankumar ) | 12:29 |
| *** bfournie has joined #openstack-infra | 12:29 | |
| *** wolverineav has joined #openstack-infra | 12:30 | |
| chandankumar | tosky: yup | 12:30 |
| chandankumar | tosky: cmurphy was talking about this in the morning, i donot know the status about the same | 12:31 |
| *** tmorin1 has joined #openstack-infra | 12:31 | |
| cmurphy | i posted https://review.openstack.org/#/c/539178/2 to try to help debug but it would be nice of an infra-root was around to look | 12:32 |
| *** jamesmcarthur has quit IRC | 12:32 | |
| *** tmorin has quit IRC | 12:32 | |
| *** bfournie has quit IRC | 12:33 | |
| *** jamesmcarthur has joined #openstack-infra | 12:34 | |
| cmurphy | aha, i can reproduce | 12:35 |
| *** yamamoto has joined #openstack-infra | 12:35 | |
| *** liusheng has quit IRC | 12:36 | |
| cmurphy | infra-root: I think it's something wrong with the latest image, I can reproduce the problem on a new image built with the build-image.sh script but on my regular ubuntu vms using the CI mirrors the packages are fine | 12:36 |
| *** liusheng has joined #openstack-infra | 12:36 | |
| odyssey4me | that sounds like the images were built using packages more recent than available in the mirrors | 12:37 |
| cmurphy | that sounds about right | 12:38 |
| cmurphy | libgfortran3 : Depends: gcc-5-base (= 5.4.0-6ubuntu1~16.04.5) but 5.4.0-6ubuntu1~16.04.6 is to be installed | 12:38 |
| *** sshnaidm is now known as sshnaidm|afk | 12:39 | |
| *** jamesmcarthur has quit IRC | 12:39 | |
| *** yamamoto_ has joined #openstack-infra | 12:40 | |
| *** dhajare_ has joined #openstack-infra | 12:40 | |
| *** yamamoto_ has quit IRC | 12:43 | |
| *** makowals_ has quit IRC | 12:43 | |
| *** yamamoto has quit IRC | 12:43 | |
| *** katkapilatova has joined #openstack-infra | 12:43 | |
| *** jpena is now known as jpena|lunch | 12:45 | |
| *** makowals has joined #openstack-infra | 12:46 | |
| *** janki has quit IRC | 12:46 | |
| *** links has quit IRC | 12:49 | |
| *** edmondsw_ has joined #openstack-infra | 12:49 | |
| *** jamesmcarthur has joined #openstack-infra | 12:50 | |
| *** edmondsw has quit IRC | 12:51 | |
| *** salv-orlando has joined #openstack-infra | 12:52 | |
| *** jamesmcarthur has quit IRC | 12:54 | |
| *** makowals_ has joined #openstack-infra | 12:55 | |
| *** gcb has quit IRC | 12:56 | |
| *** makowals has quit IRC | 12:57 | |
| *** slaweq_ has joined #openstack-infra | 12:58 | |
| *** edmondsw_ has quit IRC | 12:58 | |
| *** jamesmcarthur has joined #openstack-infra | 12:59 | |
| *** edmondsw has joined #openstack-infra | 13:00 | |
| *** zhurong_ has joined #openstack-infra | 13:03 | |
| *** jamesmcarthur has quit IRC | 13:04 | |
| *** slaweq_ has quit IRC | 13:05 | |
| *** bfournie has joined #openstack-infra | 13:05 | |
| *** sree has quit IRC | 13:08 | |
| *** jamesmcarthur has joined #openstack-infra | 13:12 | |
| *** olaph1 has joined #openstack-infra | 13:12 | |
| *** salv-orl_ has joined #openstack-infra | 13:13 | |
| *** olaph has quit IRC | 13:14 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix relaunch attempts when hitting quota errors https://review.openstack.org/536930 | 13:15 |
| *** dhajare_ has quit IRC | 13:16 | |
| *** jamesmcarthur has quit IRC | 13:16 | |
| *** salv-orlando has quit IRC | 13:16 | |
| *** zhurong has quit IRC | 13:17 | |
| *** hemna_ has joined #openstack-infra | 13:18 | |
| *** zhurong_ has quit IRC | 13:19 | |
| *** zhurong has joined #openstack-infra | 13:19 | |
| *** dprince has joined #openstack-infra | 13:19 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:19 | |
| *** zhurong has quit IRC | 13:20 | |
| *** tellesnobrega has quit IRC | 13:21 | |
| *** alexchadin has quit IRC | 13:21 | |
| *** chenying_ has quit IRC | 13:21 | |
| *** trown|outtypewww is now known as trown | 13:25 | |
| *** agopi has joined #openstack-infra | 13:25 | |
| *** trown is now known as trown|rover | 13:25 | |
| *** jamesmcarthur has joined #openstack-infra | 13:26 | |
| cmurphy | it actually just seems like the mirror is out of sync, on a new vm using archive.ubuntu.com things are fine | 13:27 |
| *** amoralej is now known as amoralej|lunch | 13:28 | |
| *** rlandy has joined #openstack-infra | 13:28 | |
| tosky | cmurphy: but of course we can't just blindy retry until we find a mirror | 13:30 |
| cmurphy | tosky: i think they're all out of sync | 13:30 |
| tosky | ouch | 13:30 |
| *** ykarel is now known as ykarel|away | 13:31 | |
| *** jamesmcarthur has quit IRC | 13:31 | |
| *** weshay|ruck|afk is now known as weshay|ruck | 13:31 | |
| *** ijw has quit IRC | 13:32 | |
| *** rosmaita has joined #openstack-infra | 13:32 | |
| AJaeger | cmurphy: the mirrors are on AFS - so in sync ;( | 13:34 |
| AJaeger | cmurphy, tosky, we need an infra-root to help again | 13:34 |
| cmurphy | AJaeger: in sync with each other but not in sync with upstream i think | 13:34 |
| AJaeger | cmurphy: yep | 13:34 |
| *** sree has joined #openstack-infra | 13:35 | |
| AJaeger | time for another #status notice? What should I send? | 13:35 |
| *** ykarel|away has quit IRC | 13:36 | |
| *** dave-mccowan has quit IRC | 13:37 | |
| *** jamesmcarthur has joined #openstack-infra | 13:37 | |
| *** slaweq has quit IRC | 13:37 | |
| *** slaweq has joined #openstack-infra | 13:38 | |
| cmurphy | "devstack jobs are currently failing due to mirror issues, more info shortly" ? | 13:39 |
| odyssey4me | not just devstack - anything using ubuntu-xenial | 13:39 |
| odyssey4me | well, at least every functional test | 13:40 |
| cmurphy | anything trying to install certain packages on ubuntu-xenial | 13:40 |
| cmurphy | "some ubuntu-xenial jobs are currently failing due to mirror issues, more info shortly" ? | 13:40 |
| AJaeger | What about #status notice Our ubuntu-xenial images (used for e.g. unit tests and devstack) are currently failing to install any packages, please restrain from *recheck* or *approve* until the issue has been investigated and fixed | 13:41 |
| cmurphy | +1 | 13:41 |
| AJaeger | cmurphy, odyssey4me thanks | 13:41 |
| odyssey4me | sgtm | 13:41 |
| AJaeger | #status notice Our ubuntu-xenial images (used for e.g. unit tests and devstack) are currently failing to install any packages, restrain from *recheck* or *approve* until the issue has been investigated and fixed. | 13:41 |
| openstackstatus | AJaeger: sending notice | 13:41 |
| *** sree has quit IRC | 13:41 | |
| cmurphy | not seeing anything crazy in cacti so far | 13:42 |
| *** jamesmcarthur has quit IRC | 13:42 | |
| AJaeger | #thanks cmurphy for your investigation | 13:42 |
| -openstackstatus- NOTICE: Our ubuntu-xenial images (used for e.g. unit tests and devstack) are currently failing to install any packages, restrain from *recheck* or *approve* until the issue has been investigated and fixed. | 13:43 | |
| cmurphy | :) | 13:43 |
| fungi | catching up, do we need to pause image builds and roll back the most recent additions? | 13:43 |
| *** yamamoto has joined #openstack-infra | 13:44 | |
| *** links has joined #openstack-infra | 13:44 | |
| fungi | ahh, image packages newer than mirror package lists | 13:44 |
| *** mihalis68 has joined #openstack-infra | 13:44 | |
| openstackstatus | AJaeger: finished sending notice | 13:44 |
| openstackstatus | AJaeger: Added your thanks to Thanks page (https://wiki.openstack.org/wiki/Thanks) | 13:44 |
| AJaeger | fungi: according to what cmurphy found out: Yes, please roll back | 13:45 |
| AJaeger | good morning, fungi! | 13:45 |
| *** jpena|lunch is now known as jpena | 13:46 | |
| *** jamesmcarthur has joined #openstack-infra | 13:46 | |
| *** yamamoto has quit IRC | 13:49 | |
| *** janki has joined #openstack-infra | 13:49 | |
| *** jamesmcarthur has quit IRC | 13:50 | |
| *** jamesmcarthur has joined #openstack-infra | 13:50 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Pause ubuntu-xenial image uploads https://review.openstack.org/539213 | 13:51 |
| AJaeger | fungi, took the liberty to +2A ^ | 13:52 |
| fungi | config-core: i think that ^ is how we pause uploads and can manually apply it (on the builders?) with puppet disabled if so | 13:52 |
| *** alexchadin has joined #openstack-infra | 13:52 | |
| *** jpich has quit IRC | 13:54 | |
| fungi | #status log nb01, nb02 and nb03 have beeen placed in the emergency maintenance list in preparation for a manual application of https://review.openstack.org/539213 | 13:54 |
| openstackstatus | fungi: finished logging | 13:54 |
| *** kgiusti has joined #openstack-infra | 13:54 | |
| *** alexchadin has quit IRC | 13:55 | |
| mordred | fungi, AJaeger: wow. what a week we've been having | 13:56 |
| *** alexchadin has joined #openstack-infra | 13:56 | |
| *** annp_ has joined #openstack-infra | 13:56 | |
| cmurphy | it must be an exciting life to wake up to fires every day | 13:56 |
| AJaeger | mordred: when it rains, it pours - or what's the saying? | 13:57 |
| *** psachin has quit IRC | 13:57 | |
| mordred | AJaeger: yup. that's right | 13:57 |
| mordred | cmurphy: 'exciting' ... yes | 13:57 |
| mordred | fungi: can I do anything to help - I'm assuming you're on the rollback already | 13:57 |
| *** sree has joined #openstack-infra | 13:58 | |
| *** tellesnobrega has joined #openstack-infra | 13:58 | |
| *** tmorin has joined #openstack-infra | 13:59 | |
| *** tmorin1 has quit IRC | 14:00 | |
| *** alexchadin has quit IRC | 14:00 | |
| *** jcoufal has joined #openstack-infra | 14:00 | |
| fungi | yeah, i'm fumbling my way through it | 14:01 |
| AJaeger | EmilienM: just saw your email on openstack-dev - are you tracking the same problem? ^ | 14:01 |
| EmilienM | AJaeger: in the tripleo meeting right now | 14:02 |
| fungi | i've updated the configs on the three builders and am trying to remember which order to delete in (image-delete the individual uploads, then dib-image-delete the source image? or the other way around?) | 14:02 |
| EmilienM | I'll catchup in 1h or less | 14:02 |
| tosky | what is the trigger that promote a newly built image? Shouldn't that be gated too with some tests? (any basic devstack test would have caught the issue) | 14:02 |
| *** sree_ has joined #openstack-infra | 14:02 | |
| *** sree_ is now known as Guest41574 | 14:03 | |
| *** makowals_ has quit IRC | 14:04 | |
| *** olaph has joined #openstack-infra | 14:04 | |
| *** olaph1 has quit IRC | 14:04 | |
| fungi | tosky: devstack fails too often due to other random reasons. we used to do something like that and had to stop because it was less reliable than just assuming the images are viable | 14:05 |
| tosky | uhm | 14:05 |
| rosmaita | maybe an "image freeze" during milestone week? | 14:05 |
| *** jamesmcarthur has quit IRC | 14:05 | |
| *** sree has quit IRC | 14:05 | |
| rosmaita | which i guess was actually last week anyway, so nevermind | 14:06 |
| *** jamesmcarthur has joined #openstack-infra | 14:06 | |
| *** dhajare_ has joined #openstack-infra | 14:06 | |
| *** kiennt26 has joined #openstack-infra | 14:06 | |
| tosky | maybe it's possible to isolate package installation failures from other failures - the logs from apt (or yum) can be identified | 14:06 |
| tosky | and on the other side, a failed devstack job for issues not related to the packages may slow down a new image by one or two days, but a real issues blocks the development for half a day in the best case | 14:08 |
| *** makowals has joined #openstack-infra | 14:08 | |
| fungi | mordred: do we have a convenience tool for this? i'm resorting to munging the tabular output from image-list into the several required parameters to image-delete | 14:08 |
| *** jamesmcarthur has quit IRC | 14:09 | |
| *** jamesmcarthur has joined #openstack-infra | 14:09 | |
| mordred | fungi: I don't think so - but I think we should have one "roll back to the previous ubuntu-xenial image" is a thing | 14:09 |
| mordred | Shrews: ^^ ? | 14:10 |
| *** Spazmotic has joined #openstack-infra | 14:10 | |
| *** yamamoto has joined #openstack-infra | 14:11 | |
| *** ldnunes has quit IRC | 14:11 | |
| fungi | okay, i seem to have gotten the four required parameters to the delete command correct | 14:13 |
| mordred | \o/ | 14:13 |
| *** Goneri has joined #openstack-infra | 14:13 | |
| *** yamamoto has quit IRC | 14:14 | |
| *** katkapilatova has quit IRC | 14:14 | |
| *** olaph1 has joined #openstack-infra | 14:14 | |
| *** yamamoto has joined #openstack-infra | 14:15 | |
| *** yamamoto has quit IRC | 14:15 | |
| fungi | working on dib-image-delete for that one next | 14:16 |
| * Shrews was heads down in code... catching up | 14:16 | |
| *** olaph has quit IRC | 14:16 | |
| *** ykarel|away has joined #openstack-infra | 14:17 | |
| mordred | Shrews: tl;dr - convenience command for nodepool to say "hey man, please to roll back image $label on all providers" | 14:17 |
| *** jamesmcarthur has quit IRC | 14:17 | |
| Shrews | mordred: fungi: dib-image-delete should take care of all uploads before deleting the on-disk image | 14:18 |
| *** jamesmcarthur has joined #openstack-infra | 14:18 | |
| fungi | Shrews: oh, it'll delete ready images based on that image id? | 14:18 |
| Shrews | fungi: it should | 14:18 |
| fungi | neat, i'll give that a shot next tim | 14:18 |
| Shrews | fungi: https://docs.openstack.org/infra/system-config/nodepool.html#bad-images | 14:18 |
| fungi | e | 14:18 |
| fungi | thanks! | 14:18 |
| fungi | certainly easier than | 14:19 |
| fungi | sudo -H -u nodepool nodepool image-list|grep ' ubuntu-xenial .* 00:0'|cut -d'|' -f2-4 | sed -e 's/^/sudo -H -u nodepool nodepool image-delete --image ubuntu-xenial --build-id/' -e 's/|/--upload-id/' -e 's/|/--provider/'>foo.sh | 14:19 |
| Shrews | fungi: if you do the uploads yourself first, the builder may aggressively try to re-upload them before you delete the dib | 14:19 |
| fungi | well, i had preconfigured them to pause | 14:19 |
| Shrews | fungi: ah | 14:20 |
| Shrews | just an extra step for yourself :) | 14:20 |
| fungi | though yes, it did that anyway because puppet was apparently already running when i added them to the emergency list so it went behind me and undid the pause while i was working on deletion syntax | 14:20 |
| fungi | i've currently got a bunch that i can't delete until their uploads finish | 14:21 |
| fungi | though sounds like nodepool should delete them for me as soon as they finish uploading, since i also told it to delete the source image | 14:22 |
| *** jamesmcarthur has quit IRC | 14:22 | |
| fungi | #status log most recent ubuntu-xenial images have been deleted from nodepool, so future job builds should revert to booting from the previous (working) image while we debug | 14:24 |
| Shrews | fungi: actually, i'm not 100% certain what happens when you try to delete a dib when there is a currently uploading image | 14:24 |
| openstackstatus | fungi: finished logging | 14:24 |
| Shrews | fungi: will have to remind myself with a code dive | 14:24 |
| cmurphy | yay thank you fungi and AJaeger | 14:24 |
| lennyb | Is it possible to configure zuul v2 so that one pipeline (gate) will abort the other one (test). I mean if commit A is already in the gate there is no reason to run another tests on it | 14:24 |
| *** ldnunes has joined #openstack-infra | 14:24 | |
| mordred | lennyb: I'm not sure what you mean by 'the other one'? | 14:25 |
| mordred | lennyb: youmean if check is still running and it goes in to gate to abort the jobs running in check (for instance) | 14:25 |
| *** derekjhyang has quit IRC | 14:26 | |
| *** watanabe_isao has joined #openstack-infra | 14:27 | |
| fungi | about the only time that situation crops up with our zuul config is when a change is already enqueued in the gate pipeline and someone leaves a "recheck" comment on it | 14:27 |
| AJaeger | config-core, could you review https://review.openstack.org/#/c/538306/ to remove a job that is moved in-tree, please? | 14:28 |
| *** eyalb has left #openstack-infra | 14:28 | |
| lennyb | mordred, yes exactly | 14:28 |
| AJaeger | fungi: we should sent a #status notice that it's safe to recheck/approve again - see my previous notice. Will you do that or shall I? | 14:28 |
| *** dklyle has quit IRC | 14:29 | |
| fungi | AJaeger: are jobs succeeding again already? | 14:29 |
| AJaeger | fungi: so, you suggest to better verify that everything works? Yeah, probably better... | 14:30 |
| *** alexchadin has joined #openstack-infra | 14:30 | |
| fungi | i'm not where i can check that easily yet, nor am i certain i read scrollback closely enough to know which jobs were impacted | 14:30 |
| AJaeger | cmurphy, tosky , any ideas? ^ | 14:30 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add buildset-artifacts-location and fetch roles https://review.openstack.org/530679 | 14:31 |
| Shrews | fungi: so, i *think* the builder will dtrt when you delete a dib that has an in-progress upload | 14:31 |
| fungi | need to finish up my morning routine and then revisit the details so i can start digging into the corresponding mirror problem | 14:31 |
| fungi | Shrews: thanks again. just going to tell myself i'm fuzz-testing nodepool commands ;) | 14:31 |
| *** dhajare_ has quit IRC | 14:32 | |
| Shrews | fungi: build is marked as DELETED, existing uploads removed, cleanup thread will periodically go back and keep trying to remove uploads until all are gone, then the dib is removed from disk | 14:32 |
| fungi | sounds perfect | 14:33 |
| cmurphy | AJaeger: i just sent a bunch of rechecks to the patches I care about so I can report back if they worked :) | 14:33 |
| fungi | thanks cmurphy! that's a huge help | 14:33 |
| cmurphy | i mostly just want my patches merged >.> | 14:33 |
| fungi | don't we all? ;) | 14:33 |
| * fungi will be back in a few moments to resume troubleshooting | 14:35 | |
| tosky | AJaeger: I can recheck my job that was failing | 14:36 |
| *** slaweq_ has joined #openstack-infra | 14:36 | |
| *** lucasagomes is now known as lucas-hungry | 14:36 | |
| *** amoralej|lunch is now known as amoralej | 14:39 | |
| *** yamamoto has joined #openstack-infra | 14:40 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul-jobs master: Propose to move submit-log-processor-jobs and submit-logstash-jobs in zuul-jobs https://review.openstack.org/537847 | 14:40 |
| *** watanabe_isao has quit IRC | 14:41 | |
| *** slaweq_ has quit IRC | 14:41 | |
| pabelanger | morning | 14:42 |
| openstackgerrit | Merged openstack-infra/project-config master: Pause ubuntu-xenial image uploads https://review.openstack.org/539213 | 14:42 |
| *** yamamoto has quit IRC | 14:43 | |
| *** eharney has joined #openstack-infra | 14:44 | |
| *** d0ugal has quit IRC | 14:44 | |
| pabelanger | cmurphy: AJaeger: Shrews: fungi: odd, we should have been protected from that on ubuntu, but I think we dropped some configuration when we migrated to nodepoolv3 | 14:45 |
| pabelanger | https://review.openstack.org/511492/ | 14:45 |
| pabelanger | I can repropose that | 14:45 |
| *** slaweq_ has joined #openstack-infra | 14:46 | |
| AJaeger | please do | 14:46 |
| *** d0ugal has joined #openstack-infra | 14:46 | |
| *** yamamoto has joined #openstack-infra | 14:48 | |
| *** jamesmcarthur has joined #openstack-infra | 14:50 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Switch (back) ubuntu DIB to use AFS mirror in rackspace https://review.openstack.org/539228 | 14:50 |
| pabelanger | config-core: cmurphy: ^ will help fix our DIB issues from this morning | 14:50 |
| *** adarazs is now known as adarazs_brb | 14:50 | |
| pabelanger | sorry about that, I proposed the original change to remove it (by mistake) | 14:50 |
| *** slaweq_ has quit IRC | 14:50 | |
| cmurphy | thanks pabelanger | 14:51 |
| *** katkapilatova has joined #openstack-infra | 14:52 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/project-config master: Revert "Pause ubuntu-xenial image uploads" https://review.openstack.org/539229 | 14:52 |
| pabelanger | and revert | 14:52 |
| *** ijw has joined #openstack-infra | 14:53 | |
| *** katkapilatova has left #openstack-infra | 14:53 | |
| pabelanger | I'll see if AFS mirrors are working properly | 14:53 |
| *** dave-mccowan has joined #openstack-infra | 14:56 | |
| AJaeger | thanks, pabelanger | 14:56 |
| pabelanger | yah, I seen an issue | 14:56 |
| openstackgerrit | Steve McLellan proposed openstack-infra/project-config master: Change searchlight-ui to use publish-to-pypi-horizon https://review.openstack.org/539231 | 14:56 |
| pabelanger | let me get some coffee and start to clean it up | 14:56 |
| *** r-daneel has joined #openstack-infra | 14:57 | |
| *** d0ugal has quit IRC | 14:58 | |
| *** ldnunes has quit IRC | 14:59 | |
| mordred | lennyb: if so, no - I do not believe that is possible. | 15:02 |
| *** dhajare_ has joined #openstack-infra | 15:03 | |
| *** tellesnobrega has quit IRC | 15:03 | |
| cmurphy | novnc successfully installs now https://zuul.openstack.org/stream.html?uuid=4728ed69d92048698a99fe3f597820e3&logfile=console.log | 15:04 |
| pabelanger | yah, our AFS mirrors haven't updated since 01-25-2018. Working to remove lock and verify repo, then will run reprepro again | 15:05 |
| pabelanger | for ubuntu | 15:05 |
| *** links has quit IRC | 15:05 | |
| *** alexchadin has quit IRC | 15:05 | |
| *** yamahata has joined #openstack-infra | 15:06 | |
| *** dsariel has quit IRC | 15:06 | |
| fultonj | hi pabelanger you mentioned on openstack-dev you have zuul/github doing initial testing w/ ansible/ansible ? is there a URL i could see for that? | 15:09 |
| *** annp_ has quit IRC | 15:09 | |
| *** andreas_s has quit IRC | 15:10 | |
| *** andreas_s has joined #openstack-infra | 15:11 | |
| pabelanger | fultonj: You'll see the PRs on http://zuul.openstack.org/ but in check pipeline, they'll only run against shade jobs. Maybe mordred has a PR which shows the results, I'm not 100% everything is working correctly | 15:12 |
| *** ldnunes has joined #openstack-infra | 15:12 | |
| *** yamamoto has quit IRC | 15:12 | |
| pabelanger | fultonj: they will just show as ansible/ansible for the project on zuul.o.o | 15:12 |
| AJaeger | fultonj, pabelanger - not anymore, see https://review.openstack.org/538447 | 15:13 |
| pabelanger | Oh, right. I forgot about that | 15:14 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Set remote url on every getRepo in merger https://review.openstack.org/535716 | 15:15 |
| pabelanger | clarkb: chown for jenkins user on logs.o.o is done | 15:16 |
| *** hongbin has joined #openstack-infra | 15:16 | |
| EmilienM | AJaeger: back here - not exactly I think, we had some timeouts in our centos multinode jobs | 15:16 |
| pabelanger | clarkb: we might want to 0775 directories too? Or update upload-logs to confirm that directory permission | 15:17 |
| EmilienM | trown|rover, weshay|ruck : do we know the reasons of our timeouts yet? or at least which jobs are affected? | 15:17 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config master: Add ansible-role-k8s-cinder to zuul.d https://review.openstack.org/534608 | 15:17 |
| *** zhongjun has quit IRC | 15:18 | |
| AJaeger | EmilienM: we had xenial problems as well - just wanted to understand whether you're hit by the same or have some others... | 15:18 |
| fultonj | AJaeger: i see you yanked out 14 lines of yaml, i'm now trying to understand what they did between zuul and github and clicking around in https://github.com/openstack-infra/project-config | 15:19 |
| *** efried has joined #openstack-infra | 15:19 | |
| EmilienM | AJaeger: I don't think our issues were xenial problems although I'll need to check | 15:19 |
| mordred | fultonj, pabelanger: yah - we had to turn them off for now | 15:19 |
| EmilienM | AJaeger: our main issue now is multinode timing out a lot | 15:19 |
| EmilienM | AJaeger: I'm checking details now | 15:20 |
| mordred | fultonj: there is a bug currently where we were reporting merge issues on changes that weren't what we had configured to report on (it's an edge case) | 15:20 |
| pabelanger | fetch-subunit-out just had a MODULE_FAILURE: http://logs.openstack.org/08/536508/2/gate/openstack-tox-validate/dd01df1/job-output.txt.gz#_2018-01-30_14_29_18_562411 | 15:20 |
| EmilienM | AJaeger: I'll continue on #tripleo | 15:20 |
| pabelanger | does somebody have a minute to dig into why? | 15:20 |
| fultonj | mordred: OK, so you plan to introduce it back after the bug is dealt with? | 15:21 |
| *** ijw has quit IRC | 15:21 | |
| mordred | fultonj: absolutely | 15:21 |
| fultonj | mordred: tripleo is using ceph-ansible (github hosted) and i am interested in getting cross-ci working | 15:22 |
| mordred | fultonj: yah - I thnk there's several good opportunities for using the support once we've got things ironed out | 15:22 |
| fultonj | and using ansible/ansible as an example to ci how that could be done | 15:22 |
| *** oidgar has quit IRC | 15:22 | |
| *** dsariel has joined #openstack-infra | 15:22 | |
| mordred | yes! | 15:22 |
| fultonj | mordred: would you mind pointing me in the direction of the right submissions to read on how the ansible/ansible submission works? | 15:23 |
| fultonj | by the time i figure it out you might have it fixed :) | 15:23 |
| * fultonj currently clicking through history of https://github.com/openstack-infra/project-config | 15:24 | |
| *** d0ugal has joined #openstack-infra | 15:24 | |
| *** camunoz has joined #openstack-infra | 15:25 | |
| pabelanger | fultonj: https://docs.openstack.org/infra/zuul/admin/drivers/github.html is specific to the driver integration for zuul. Then, users are able to use PRs from github as normal, and zuul will get the events. Zuul docs then expand on ideas of pipelines / jobs, which same as gerrit integration | 15:26 |
| fultonj | pabelanger: excellent, thank you! | 15:27 |
| weshay|ruck | EmilienM, I wanted to get w/ you re: the timeouts and start to record each step of the deployment | 15:27 |
| EmilienM | weshay|ruck: see #tripleo | 15:27 |
| EmilienM | weshay|ruck: it's on our side probably, the undercloud takes more time than before to deploy | 15:27 |
| weshay|ruck | k | 15:27 |
| pabelanger | fungi: AJaeger: https://review.openstack.org/539228/ and https://review.openstack.org/539229/ fix and unpause ubuntu DIBs. Are we ready to land it? | 15:28 |
| *** agopi has quit IRC | 15:28 | |
| *** agopi has joined #openstack-infra | 15:28 | |
| pabelanger | actually, I have to run an errand for a few hours in 20mins. So, also happy to wait until I am back to support | 15:28 |
| sc` | from the peanut gallery: one of my xenial chef jobs completed its first of three chef runs successfully. the three times is a legacy thing, because once is good enough | 15:28 |
| *** andreas_s has quit IRC | 15:29 | |
| AJaeger | sc`: and it failed before? | 15:30 |
| *** yamamoto has joined #openstack-infra | 15:30 | |
| *** andreas_s has joined #openstack-infra | 15:30 | |
| *** b_bezak has quit IRC | 15:31 | |
| *** gcb has joined #openstack-infra | 15:31 | |
| sc` | i had refrained from kicking off jobs earlier this morning. it's likely that it would have failed a couple hours ago | 15:31 |
| *** b_bezak has joined #openstack-infra | 15:31 | |
| *** eharney has quit IRC | 15:32 | |
| *** armax has joined #openstack-infra | 15:33 | |
| *** kiennt26 has quit IRC | 15:33 | |
| pabelanger | okay, no longer running errand, due to weather. | 15:33 |
| *** lucas-hungry is now known as lucasagomes | 15:34 | |
| AJaeger | pabelanger: 539229 has now two +2s - didn't +A - wanted you or fungi to approve and babysit | 15:34 |
| *** Goneri has quit IRC | 15:35 | |
| *** b_bezak has quit IRC | 15:35 | |
| fungi | pabelanger: that MODULE_FAILURE looks like it happened in the "Check for stestr directory" task. i wonder if it should have been created during the run phase? | 15:36 |
| *** adarazs_brb is now known as adarazs | 15:36 | |
| *** dsariel has quit IRC | 15:37 | |
| bcafarel | hi infra-core, can you take a look at https://review.openstack.org/#/c/535390/ ? small review mostly to get meetbot in our channel (#networking-sfc) | 15:37 |
| AJaeger | bcafarel: note we can only merge this while no meetings happen | 15:39 |
| *** andreas_s has quit IRC | 15:40 | |
| corvus | fungi: i believe we decided we should build images from our mirrors, in order to solve the problem of mismatches. | 15:41 |
| fungi | pabelanger: should i remove nb* and mirror-update from the emergency maintenance list? | 15:41 |
| mordred | fultonj: also, http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/main.yaml#n1361 is relevant config | 15:41 |
| fungi | corvus: correct, and we just approved the fix for the regression in that | 15:42 |
| fultonj | mordred: thank you | 15:42 |
| mordred | fungi: which MODULE_FAILURE? | 15:42 |
| mordred | ah - I see it | 15:42 |
| fungi | mordred: the one pabelanger linked | 15:42 |
| fungi | yeah | 15:42 |
| corvus | fungi: cool | 15:42 |
| fultonj | mordred: is there a bug for the revert patch? | 15:42 |
| bcafarel | AJaeger: ok thanks, as long as a merge window opens sometimes it will be fine :) | 15:42 |
| fultonj | https://review.openstack.org/#/c/538447 has no topic | 15:42 |
| * fultonj just wants to follow the bug if there is one (clicking through launchpad) | 15:43 | |
| EmilienM | we are seeing a bunch of multinode performances issues, at least with rax-dfw now | 15:44 |
| mordred | fultonj: I do not believe we've created one yet - I'll let you know when we do | 15:45 |
| *** andreas_s has joined #openstack-infra | 15:45 | |
| fultonj | thanks | 15:45 |
| *** slaweq_ has joined #openstack-infra | 15:45 | |
| mordred | fungi, pabelanger: that's weird- that's a stat task - I'm not sure why that would MODULE_FAILURE | 15:46 |
| EmilienM | AJaeger, pabelanger, mordred : not sure if you're aware about RAX DFW issue but I opened 5 random gate timeouts in tripleo gate, and all from RAX DFW | 15:46 |
| fungi | mordred: fultonj: https://storyboard.openstack.org/#!/story/2001333 looks close, from the one-sentence description, but hard to know without more detail | 15:46 |
| EmilienM | other clodus are fine | 15:46 |
| EmilienM | clouds even | 15:46 |
| fultonj | thanks fungi | 15:46 |
| corvus | fungi, mordred, fultonj: we also need to work with the tc to come up with a plan for approving the use of openstack zuul for third-party reporting on non-openstack git repos | 15:46 |
| clarkb | tosky: fungi we also dont want to end up with special images that are required to get tests to pass. Its hard to determine if failure is "global" or "local" in that case | 15:46 |
| mordred | corvus: yah | 15:46 |
| *** felipemonteiro__ has joined #openstack-infra | 15:47 | |
| corvus | fungi, mordred, fultonj: i'm sure it won't be a big deal, but we need at least something slightly more official than "it sounded good to the infra folks" :) | 15:47 |
| tosky | clarkb: it's not possible to think about at test for mirrors? Something that just install packages? | 15:47 |
| fungi | corvus: what are the potentially controversial bits? do you think it warrants a tc resolution, or what? | 15:47 |
| *** eharney has joined #openstack-infra | 15:47 | |
| EmilienM | rax-ord and rax-dfw cause TripleO timeouts in gate | 15:47 |
| tosky | clarkb: that would not be a test for a special image, just a normal acceptance test for a guest image | 15:48 |
| corvus | fungi: i'm thinking we should have a resolution establishing a process or criteria or something. because we're using openstack project resources for not-openstack-project things. the things we want to use them on are good things to use them on because they help advance the project, but it seems that the openstack project should decide to do that, and not just us. | 15:49 |
| *** Goneri has joined #openstack-infra | 15:49 | |
| *** dsariel has joined #openstack-infra | 15:50 | |
| fungi | EmilienM: what are the nature of the timeouts you're seeing? job/playbook timed out results? network timeouts (git fetch, package installs...) logged in the job run? disk access timeouts? some other sort of timeout? | 15:50 |
| corvus | (i think we've informally established that desire enough on all sides for testing on ansible/ansible to proceed, but when we move past that, we should have a plan) | 15:50 |
| *** slaweq_ has quit IRC | 15:50 | |
| fungi | corvus: makes sense. so basically we need to draft a process for determining how/when/whether we support that and seek edits/blessing from the tc | 15:51 |
| openstackgerrit | Merged openstack-infra/project-config master: Switch (back) ubuntu DIB to use AFS mirror in rackspace https://review.openstack.org/539228 | 15:51 |
| EmilienM | fungi: performances and network | 15:51 |
| EmilienM | let me get numbers | 15:51 |
| fungi | EmilienM: so job runs exceeding their configured playbook timeouts? | 15:51 |
| clarkb | tosky: ya we can test specific things but if the acceptance test is our tests run and pass (which is typivally what is suggested) we run the risk of requiring our "special" images | 15:52 |
| fultonj | corvus: perhaps a ptg topic? | 15:52 |
| EmilienM | packages take 837.54 to be downloaded + installed in RAX while 300s elsewhere | 15:52 |
| corvus | fultonj: i'm hoping it's not controversial enough to warrant that. maybe we can write something up and get it approved before then. | 15:52 |
| EmilienM | it's 8+ min more | 15:52 |
| fultonj | corvus: ok | 15:53 |
| EmilienM | keystone resources (endpoints, services, etc) take 10 min to be created! | 15:53 |
| EmilienM | instead of 3 | 15:53 |
| fungi | EmilienM: ooh, so also possible slowness on the package mirrors in those regions | 15:53 |
| EmilienM | yeah probably that | 15:53 |
| tosky | clarkb: right, and it's not what I'd request; even when I spoke about running our test, the goal was to either to focus on the part which allows us to call an image as "working" | 15:53 |
| fungi | EmilienM: but non-package-related slowness may indicate actual performance issues on the job nodes themselves, i guess | 15:53 |
| EmilienM | fungi: sure, but the mirrors slowness on RAX is killing us | 15:54 |
| fultonj | corvus: if you draft a spec i'd be happy to review | 15:54 |
| fungi | EmilienM: thanks, i'm taking a look at our resource graphs for those now | 15:54 |
| corvus | fultonj: cool, i'll let you know | 15:55 |
| clarkb | EmilienM: keystone respurces if created by osc are directly tied to disk performance due to how osc's plugin mechanism uses entry points | 15:55 |
| fultonj | thanks | 15:55 |
| fungi | looks like we might be capping out our network interface on mirror.dfw.rax.o.o | 15:55 |
| corvus | fungi, EmilienM: we seem to peak at 100Mbit outbound on mirror.dfw.rax http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=3104&rra_id=all | 15:55 |
| fungi | yep, exactly what i was just looking at | 15:56 |
| corvus | what's our max there? | 15:56 |
| fungi | the historic graphs appear to indicate that it used to be higher | 15:56 |
| corvus | ord goes well past 200 http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=3063&rra_id=all | 15:56 |
| corvus | they should be the same flavors and have the same cap | 15:56 |
| fungi | yeah, they look like the same flavor | 15:57 |
| *** pcichy has joined #openstack-infra | 15:57 | |
| *** xarses_ has joined #openstack-infra | 15:57 | |
| *** david-lyle has joined #openstack-infra | 15:57 | |
| fungi | but it really does appear that around utc midnight on the morning of january 25 it started getting capped to 100mbps outbound for the dfw mirror | 15:58 |
| *** andreww has joined #openstack-infra | 15:58 | |
| *** felipemonteiro has joined #openstack-infra | 15:58 | |
| *** slaweq_ has joined #openstack-infra | 15:58 | |
| fungi | the server's been up and running since days before that, so doesn't correspond to a reboot/replacement | 15:59 |
| *** rosmaita has quit IRC | 15:59 | |
| EmilienM | I have 11 jobs that ran over the last hours which have the same problem | 15:59 |
| EmilienM | they all run on dfw rax | 15:59 |
| *** andreas_s has quit IRC | 15:59 | |
| *** rosmaita has joined #openstack-infra | 15:59 | |
| fungi | EmilienM: yeah, that could easily be caused by packet loss we'd experience from the rate limiting on the mirror's network interface | 15:59 |
| pabelanger | okay, reprepro finish and packages look to all be there | 16:00 |
| *** olaph1 has quit IRC | 16:00 | |
| pabelanger | going to update packages now | 16:00 |
| fungi | i'm going spelunking in rackspace tickets for any hints | 16:00 |
| clarkb | 100mbps is supsicious could just be a link negotiating slower unexpectedly | 16:01 |
| *** andreas_s has joined #openstack-infra | 16:01 | |
| fungi | we can scale down our max-servers in dfw in the interim while we get the provider to look into it | 16:01 |
| *** rosmaita has quit IRC | 16:01 | |
| *** felipemonteiro__ has quit IRC | 16:01 | |
| *** xarses_ has quit IRC | 16:02 | |
| EmilienM | fungi: that would be great until they reply to your ongoing ticket | 16:02 |
| pabelanger | fungi: not mirror-update, I am still working on that. but nb* can be | 16:02 |
| *** olaph has joined #openstack-infra | 16:02 | |
| fungi | well, i don't know whether there is (yet) an ongoing ticket | 16:02 |
| weshay|ruck | EmilienM, I see one on ovh-gra1 too | 16:02 |
| EmilienM | weshay|ruck: link? | 16:02 |
| weshay|ruck | http://logs.openstack.org/06/534706/3/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/4615b2f/job-output.txt.gz#_2018-01-30_09_04_51_561838 | 16:03 |
| *** alexchadin has joined #openstack-infra | 16:03 | |
| EmilienM | weshay|ruck: no, it's not the same issue | 16:03 |
| EmilienM | weshay|ruck: see http://logs.openstack.org/06/534706/3/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/4615b2f/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-01-30_09_43_34 | 16:03 |
| EmilienM | weshay|ruck: package download was "fast", 300s | 16:03 |
| weshay|ruck | k | 16:04 |
| EmilienM | versus RAX which takes 800+ | 16:04 |
| honza | is there a way to use different templates in project-config/zuul.d/projects.yaml based on the branch? kind of like the branch: under jobs: etc | 16:04 |
| trown|rover | weshay|ruck: ya that was different | 16:04 |
| *** slaweq_ has quit IRC | 16:04 | |
| fungi | #status log removed nb01, nb02 and nb03 from the emergency maintenance list now that it's safe to start building new ubuntu-xenial images again | 16:04 |
| openstackstatus | fungi: finished logging | 16:04 |
| trown|rover | weshay|ruck: http://logs.openstack.org/06/534706/3/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/4615b2f/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-01-30_09_43_34 | 16:04 |
| trown|rover | too slow :( | 16:04 |
| *** armaan has quit IRC | 16:04 | |
| *** andreas_s has quit IRC | 16:05 | |
| prometheanfire | fungi: oh, cool | 16:05 |
| prometheanfire | so, rechecks? | 16:05 |
| corvus | honza: if the template itself is in a repo with branches, it will automatically apply implied branch matchers to all of its jobs. so a template defined on stable/pike should end up only adding jobs which run on stable/pike. | 16:05 |
| *** andreas_s has joined #openstack-infra | 16:06 | |
| fungi | trown|rover: weshay|ruck: EmilienM: what we're specifically looking into now is (seemingly low) network rate limiting on the mirror server in the rax dfw region causing packet loss (and so slowness/errors) for package retrieval in jobs | 16:06 |
| corvus | honza: if you want to do something more explicit (ie, define a template on master which runs jobs on stable/pike) you can add a 'branches:' attribute to the jobs themselves in the template. | 16:06 |
| EmilienM | fungi: exactly | 16:06 |
| fungi | prometheanfire: sure, though we're getting some poor performance in rax.dfw at the moment | 16:06 |
| honza | corvus: this template is general (doesn't specify branches), and i need to apply an older version of it to stable/pike | 16:06 |
| pabelanger | is it possible our cache in mirror.ord.rax is getting invalidated to fast and we have to download packages more often from web? | 16:07 |
| EmilienM | fungi: I said to tripleo folks to not approve patches or recheck for now until we sort things out | 16:07 |
| prometheanfire | fungi: just generally bad perf or something specific (so I can ping people) | 16:07 |
| pabelanger | I know we make it smaller because of tripleo-test-cloud-rh1, maybe we can raise the storage size, because we are now taking that cloud offline | 16:07 |
| *** rosmaita has joined #openstack-infra | 16:08 | |
| pabelanger | /dev/mapper/main-proxycache 99G 48G 46G 52% /var/cache/apache2 | 16:08 |
| pabelanger | currently | 16:08 |
| honza | corvus: my project uses nodejs8-x for master, and nodejs6-x for stable/pike | 16:09 |
| fungi | prometheanfire: http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=3104&rra_id=all shows what looks like our rate limit for the mirror in dfw suddenly dropped to 100mbps at midnight utc on the morning of the 25th | 16:09 |
| honza | corvus: nodejs{x}-jobs is a template which takes a param (nodejs version) | 16:09 |
| prometheanfire | ok | 16:10 |
| fungi | prometheanfire: as clarkb said, could indicate an interface negotiation problem on a physical link somewhere, so might be impacting more than just us | 16:10 |
| *** sree has joined #openstack-infra | 16:10 | |
| honza | corvus: maybe: templates: - name: nodejs6-jobs branch: stable/pike instead of just template: - nodejs6-jobs | 16:11 |
| * honza shrugs | 16:11 | |
| fungi | though if it is, we're monopolizing most of that link ;) | 16:11 |
| corvus | honza: are you asking about openstack's zuul, or something else? | 16:11 |
| honza | corvus: yes | 16:11 |
| corvus | honza: i don't think our templates take parameters | 16:12 |
| *** e0ne has quit IRC | 16:12 | |
| efried | Are we cleared to start rechecking at this point? | 16:12 |
| *** ldnunes has quit IRC | 16:12 | |
| corvus | honza: can you point me at your project's configuration? | 16:12 |
| honza | corvus: https://github.com/openstack-infra/project-config/blob/master/zuul.d/projects.yaml#L16792-L16801 | 16:13 |
| AJaeger | wow, what'S this? " : ERROR Project openstack/python-ironicclient does not have the default branch master" | 16:13 |
| *** Guest41574 has quit IRC | 16:13 | |
| AJaeger | on https://review.openstack.org/538826 | 16:13 |
| honza | corvus: the question is: can i set a specific template for a branch in that place? | 16:13 |
| *** ldnunes has joined #openstack-infra | 16:13 | |
| prometheanfire | fungi: pinged the team internally | 16:14 |
| fungi | prometheanfire: server show on mirror01.dfw.rax.openstack.org indicates it's built from their performance1-8 flavor, and flavor show on that claims a rxtx_factor of 1600.0 so i don't think we're getting anywhere near that | 16:14 |
| AJaeger | honza: you cannot. You can take the single jobs from the template and add the branches to those | 16:14 |
| corvus | honza: the best thing to do i think is remove the nodejs templates from that file, then create a .zuul.yaml in openstack/tripleo-ui on each branch, and add the appropriate template to the project in those files | 16:15 |
| mordred | honza, corvus: i was just about to type the same thing :) | 16:15 |
| prometheanfire | fungi: oh, do you not mean the rackspace mirror? | 16:15 |
| mordred | prometheanfire: no, we don't use the rackspace mirror | 16:15 |
| pabelanger | fungi: replacement mirrors are usually easy (what could go wrong?) to bring online. We could consider that if we thing something is wrong with our current instance, hoping we get onto better hardware. | 16:16 |
| fungi | prometheanfire: correct, a cloud compute instance we're running named mirror01.dfw.rax.openstack.org... i'm in the process of filling in a trouble ticket describing the issue now | 16:16 |
| prometheanfire | pabelanger: that's what I'd suggest | 16:16 |
| pabelanger | prometheanfire: ah, missed that. | 16:16 |
| pabelanger | oh, suggest :) | 16:17 |
| pabelanger | need more coffee | 16:17 |
| prometheanfire | pabelanger: I didn't get around to it, did't know that it was an os-infra run mirror :P | 16:17 |
| *** Goneri has quit IRC | 16:17 | |
| prometheanfire | :D | 16:17 |
| *** Nil_ has joined #openstack-infra | 16:17 | |
| mordred | honza, corvus: that said - if the node6 vs. node8 is global across openstack javascript projects ... like, all openstack projets doing javascript are expected to be doing node8 in master but node6 in stable/pike, then I bet we could make an 'openstack-javascript-jobs' for that - but we'd likely want to get that defined in the PTI first | 16:17 |
| *** yamamoto has quit IRC | 16:17 | |
| mordred | my understanding is that so far we're not quite that coordinated yet, right? | 16:18 |
| *** olaph1 has joined #openstack-infra | 16:19 | |
| *** olaph has quit IRC | 16:19 | |
| *** ykarel|away has quit IRC | 16:20 | |
| *** yamamoto has joined #openstack-infra | 16:20 | |
| fungi | #status log ticket 180130-ord-0000697 filed to investigate an apparent 100Mbps rate limit on the mirror01.dfw.rax instance | 16:24 |
| openstackstatus | fungi: finished logging | 16:24 |
| *** yamamoto has quit IRC | 16:25 | |
| AJaeger | infra-root, once again "ERROR Project openstack/python-ironicclient does not have the default branch master" - now on https://review.openstack.org/538845 | 16:26 |
| AJaeger | and "ERROR Project openstack/tripleo-heat-templates does not have the default branch master" on https://review.openstack.org/537787 ;( | 16:26 |
| corvus | AJaeger: could be a problem on a merger or executor | 16:26 |
| honza | corvus: AJaeger: mordred: cool, thanks | 16:27 |
| corvus | AJaeger: i'll see if i can correlate | 16:27 |
| AJaeger | thanks for checking, corvus - hope those three changes (first 12 mins ago) help. | 16:27 |
| * AJaeger will be back later... | 16:28 | |
| corvus | oh that's an executor only error | 16:29 |
| *** dhajare_ has quit IRC | 16:29 | |
| *** chandankumar is now known as chkumar|off | 16:30 | |
| *** Goneri has joined #openstack-infra | 16:33 | |
| *** inc0 has quit IRC | 16:33 | |
| *** inc0 has joined #openstack-infra | 16:33 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/project-config master: Use max-ready-age on nodepool nodes https://review.openstack.org/539248 | 16:33 |
| Shrews | oops. didn't mean to throw that out yet (stupid fast fingers) | 16:34 |
| *** david-lyle has quit IRC | 16:34 | |
| honza | AJaeger: corvus: would you mind having a quick look? https://review.openstack.org/#/c/539250/ | 16:35 |
| *** alexchadin has quit IRC | 16:35 | |
| *** jtomasek has quit IRC | 16:36 | |
| *** dhill_ has joined #openstack-infra | 16:36 | |
| pabelanger | okay, we have 2 corrupt packages in ubuntu reprepro: http://paste.openstack.org/show/658030/ | 16:38 |
| pabelanger | deleting them now to force download again | 16:38 |
| *** andreas_s has quit IRC | 16:40 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Temporarily reduce max-servers in rax-dfw https://review.openstack.org/539255 | 16:41 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config master: Revert "Temporarily reduce max-servers in rax-dfw" https://review.openstack.org/539256 | 16:41 |
| fungi | i'll wip 539256 for now | 16:41 |
| bnemec | EmilienM: weshay|ruck trown|rover: https://bugs.launchpad.net/tripleo/+bug/1746287 | 16:42 |
| openstack | Launchpad bug 1746287 in tripleo "RDO cloud ci jobs using Rackspace mirror" [High,Triaged] | 16:42 |
| fungi | i went through all the other mirrors and didn't see any similar bandwidth capping behavior | 16:42 |
| bnemec | ^may be related to the rax-fdw issues. | 16:42 |
| bnemec | *dfw | 16:42 |
| bnemec | I just noticed that last night. | 16:42 |
| fungi | bnemec: eep, thanks for finding | 16:42 |
| pabelanger | wait, rackspace dfw mirror is only having issues? | 16:43 |
| weshay|ruck | bnemec, that's what we were asked to do | 16:43 |
| weshay|ruck | pabelanger, and I discussed this a while back | 16:43 |
| weshay|ruck | can't prove that | 16:43 |
| weshay|ruck | but I think that's right | 16:43 |
| weshay|ruck | maybe david | 16:43 |
| weshay|ruck | :) | 16:43 |
| bnemec | Seriously? Why would we use another cloud's mirror when the repos are hosted _in_ RDO cloud? | 16:43 |
| fungi | i'd be surprised if we asked rdo to use our mirror with their ci system | 16:44 |
| EmilienM | bnemec: someone has to do the work and setup the mirror I guess | 16:44 |
| pabelanger | right, RDO is using mirror.dfw.rax.openstack.org too | 16:44 |
| bnemec | I'm arguing that we don't need a mirror. | 16:44 |
| bnemec | The repos are local to the cloud. | 16:44 |
| pabelanger | I would beat that is sucking up the bandwidth now | 16:44 |
| *** andreas_s has joined #openstack-infra | 16:44 | |
| fungi | we've previously held the position that our mirrors are not a stable api, and that we take them offline, rename or reorganize them at our whim without coordinating with outside organizations or broadly announcing | 16:45 |
| pabelanger | Yes, I've been pushing for RDO cloud to run an AFS mirror for months, sadly this hasn't happened yet | 16:45 |
| weshay|ruck | pabelanger, what ever you suggest my man | 16:45 |
| *** ramishra has quit IRC | 16:46 | |
| fungi | it would be unfortunate if us taking an extended maintenance in rax-dfw broke rdo's ci system | 16:46 |
| bnemec | I guess I could see an argument for an in-cloud mirror to take load off the repo vms, but it seems wrong to be round-tripping through another cloud. | 16:46 |
| *** david-lyle has joined #openstack-infra | 16:47 | |
| *** david-lyle has quit IRC | 16:48 | |
| pabelanger | fungi: yes, RDO has been made away of the issue, but haven't launched there on AFS mirror yet. There are open patches to support centos upstream now, but we also haven't reviewed them | 16:48 |
| *** david-lyle has joined #openstack-infra | 16:49 | |
| pabelanger | given that OVB jobs are not running live in RDO, an no longer using tripleo-test-cloud-rh1, it make sense now if that mirror is over run. The amount of jobs it has to support now has likely doubled | 16:49 |
| pabelanger | s/not/now* | 16:49 |
| *** olaph1 has quit IRC | 16:51 | |
| *** slaweq_ has joined #openstack-infra | 16:52 | |
| *** olaph has joined #openstack-infra | 16:53 | |
| corvus | the 3 default branch issues i've checked all localize to ze04 | 16:53 |
| corvus | but i don't see anything wrong with those repos | 16:53 |
| *** andreas_s has quit IRC | 16:53 | |
| *** andreas_s has joined #openstack-infra | 16:54 | |
| *** tmorin has quit IRC | 16:55 | |
| *** jtomasek has joined #openstack-infra | 16:56 | |
| *** jtomasek has quit IRC | 16:56 | |
| *** slaweq_ has quit IRC | 16:57 | |
| *** jtomasek has joined #openstack-infra | 16:57 | |
| *** jtomasek has quit IRC | 16:58 | |
| *** dprince has quit IRC | 16:59 | |
| *** gcb has quit IRC | 17:00 | |
| *** andreas_s has quit IRC | 17:00 | |
| *** aviau has quit IRC | 17:01 | |
| *** aviau has joined #openstack-infra | 17:01 | |
| *** slaweq has quit IRC | 17:03 | |
| *** erlon_ has quit IRC | 17:03 | |
| *** slaweq has joined #openstack-infra | 17:03 | |
| *** slaweq_ has joined #openstack-infra | 17:03 | |
| clarkb | anticw: ^ possible that your issues are related to the rax dfw mirror slowness? | 17:04 |
| corvus | ah ok, i think the executor must have run out of space | 17:04 |
| corvus | fatal: unable to write new index file | 17:04 |
| corvus | warning: Clone succeeded, but checkout failed. | 17:04 |
| corvus | that ^ is in the logs | 17:04 |
| corvus | so it looks like we're not trapping that error early enough | 17:05 |
| corvus | AJaeger: i think the 'default branch' errors are transient; it looks like we had some significant load spikes earlier, should be better now | 17:05 |
| corvus | we probably do need to add another executor or two :/ | 17:06 |
| *** tosky has quit IRC | 17:06 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Revert "Reduce apache reverse proxy cache to 40GB" https://review.openstack.org/539263 | 17:07 |
| *** slaweq_ has quit IRC | 17:07 | |
| *** Goneri has quit IRC | 17:07 | |
| *** sree has quit IRC | 17:08 | |
| *** slaweq has quit IRC | 17:08 | |
| *** janki has quit IRC | 17:11 | |
| *** sree has joined #openstack-infra | 17:12 | |
| *** kopecmartin has quit IRC | 17:12 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint add summit speaker assistance https://review.openstack.org/539264 | 17:13 |
| *** felipemonteiro has quit IRC | 17:14 | |
| *** felipemonteiro__ has joined #openstack-infra | 17:14 | |
| pabelanger | clarkb: this also might explain timeouts I've been seeing in integrated gate this week, if mirror is oversubcribed for jobs | 17:16 |
| pabelanger | and no bandwidth | 17:16 |
| pabelanger | okay, AFS mirror for ubuntu released | 17:16 |
| pabelanger | reprepro happy again | 17:16 |
| pabelanger | removing mirror-update.o.o from emergency file | 17:16 |
| *** sree has quit IRC | 17:17 | |
| clarkb | do we know why it got unhappy? | 17:17 |
| pabelanger | clarkb: yah, we had networking issue to AFS volume and then reprepro timed out, eventually killed, and left lock file | 17:18 |
| pabelanger | we had 2 packages corrupt, so had to manually delete and redownload | 17:18 |
| clarkb | so even if lock file was cleaned it wouldve needed intervention? | 17:18 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint add summit speaker assistance https://review.openstack.org/539264 | 17:18 |
| *** janki has joined #openstack-infra | 17:19 | |
| pabelanger | clarkb: yah, we wouldn't have released the volume, since CRC checks failed. So we were protected in that case | 17:19 |
| *** janki has quit IRC | 17:19 | |
| *** Goneri has joined #openstack-infra | 17:20 | |
| *** dprince has joined #openstack-infra | 17:20 | |
| *** yamamoto has joined #openstack-infra | 17:21 | |
| *** yamamoto has quit IRC | 17:27 | |
| pabelanger | clarkb: we just had another gate reset for nova timeout, 536936 is the fix but due to dib failure this morning didn't merged. Any objection is we enqueue and promote to land it? | 17:27 |
| pabelanger | we've been dealing with issue for a few days now | 17:27 |
| corvus | pabelanger: yeah, we should promote important integrated gate fixes | 17:28 |
| *** ralonsoh_ has joined #openstack-infra | 17:28 | |
| pabelanger | corvus: okay, will do so now | 17:29 |
| *** dprince has quit IRC | 17:29 | |
| clarkb | ++ | 17:29 |
| *** dprince has joined #openstack-infra | 17:29 | |
| pabelanger | sorry, 537933 is what I'll be promoting | 17:31 |
| *** Pramod has joined #openstack-infra | 17:31 | |
| *** ralonsoh has quit IRC | 17:31 | |
| inc0 | good morning, is zuul down? | 17:33 |
| inc0 | http://zuulv3.openstack.org no worky for me :( | 17:33 |
| smcginnis | inc0: drop the "v3" part of that. | 17:34 |
| fungi | inc0: that was only a temporary (302) redirect from zuul.openstack.org. the official status interface url is and has been http://zuul.openstack.org/ | 17:34 |
| *** panda is now known as panda|bbl | 17:34 | |
| inc0 | ok, thanks, that works:) | 17:34 |
| pabelanger | #status log 537933 promoted to help address with integrated gate timeout issue with nova | 17:34 |
| openstackstatus | pabelanger: finished logging | 17:34 |
| fungi | inc0: also findable via the status tab on http://status.openstack.org/ | 17:35 |
| fungi | when in doubt | 17:36 |
| *** markvoelker has joined #openstack-infra | 17:36 | |
| fungi | er, via the zuul tab on that page i mean | 17:36 |
| mriedem | pabelanger: thanks for promoting https://review.openstack.org/#/c/539266/1 | 17:37 |
| mriedem | oops | 17:37 |
| mriedem | 537933 | 17:37 |
| *** dsariel has quit IRC | 17:41 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: do not set OS_TEST_TIMEOUT in tox.ini https://review.openstack.org/539273 | 17:44 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: only try to reset the db user password if it needs to be https://review.openstack.org/539274 | 17:44 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: add a script to clean up test databases https://review.openstack.org/539275 | 17:44 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: log database management actions in tests https://review.openstack.org/539276 | 17:44 |
| *** myoung is now known as myoung|food | 17:44 | |
| *** tesseract has quit IRC | 17:46 | |
| *** felipemonteiro has joined #openstack-infra | 17:49 | |
| *** electrofelix has quit IRC | 17:49 | |
| *** electrofelix has joined #openstack-infra | 17:50 | |
| *** felipemonteiro__ has quit IRC | 17:51 | |
| *** ralonsoh__ has joined #openstack-infra | 17:54 | |
| *** jamesmcarthur has quit IRC | 17:56 | |
| *** trown|rover is now known as trown|lunch | 17:57 | |
| *** ralonsoh_ has quit IRC | 17:58 | |
| *** alexchadin has joined #openstack-infra | 17:58 | |
| *** iyamahat has joined #openstack-infra | 17:58 | |
| *** derekh has quit IRC | 17:59 | |
| fungi | config-core: any takers for temporarily reducing our max-servers in rax-dfw to lower the load on the mirror there? https://review.openstack.org/539255 | 18:00 |
| fungi | i haven't seen any response on our ticket yet, fwiw | 18:01 |
| *** kjackal has quit IRC | 18:01 | |
| corvus | fungi: done | 18:01 |
| fungi | i also have a revert for that prepped and uploaded, but set to wip for now | 18:01 |
| fungi | thanks corvus! | 18:01 |
| *** bhavik1 has joined #openstack-infra | 18:02 | |
| *** kjackal has joined #openstack-infra | 18:03 | |
| *** Pramod has quit IRC | 18:05 | |
| *** Pramod has joined #openstack-infra | 18:06 | |
| *** Swami has joined #openstack-infra | 18:07 | |
| *** slaweq has joined #openstack-infra | 18:08 | |
| corvus | EmilienM: i believe the change to use override-checkout in job branch matching is in production, so you might try that when you get a sec | 18:09 |
| *** bhavik1 has quit IRC | 18:10 | |
| EmilienM | corvus: that? https://review.openstack.org/#/c/537612/6/.zuul.yaml | 18:10 |
| EmilienM | I just did recheck to see how that works now | 18:10 |
| EmilienM | corvus: looks like it works | 18:10 |
| EmilienM | I see the jobs now \o/ | 18:10 |
| EmilienM | corvus: I'll confirm once the jobs have run and looked at logs to confirm the right branch was checked out | 18:11 |
| EmilienM | corvus: thanks | 18:11 |
| *** slaweq has quit IRC | 18:13 | |
| *** david-lyle has quit IRC | 18:13 | |
| openstackgerrit | Merged openstack-infra/project-config master: Temporarily reduce max-servers in rax-dfw https://review.openstack.org/539255 | 18:15 |
| *** andreww has quit IRC | 18:18 | |
| *** xarses_ has joined #openstack-infra | 18:19 | |
| *** cshastri has quit IRC | 18:19 | |
| corvus | EmilienM: great, thanks! | 18:20 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added endpoint update summit speaker assistance https://review.openstack.org/539285 | 18:23 |
| *** yamamoto has joined #openstack-infra | 18:23 | |
| *** priteau has quit IRC | 18:24 | |
| *** priteau has joined #openstack-infra | 18:25 | |
| *** alexchadin has quit IRC | 18:27 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 18:27 |
| pabelanger | so, lost internet access at home, down at coffee shop now | 18:28 |
| pabelanger | catching up | 18:28 |
| *** priteau has quit IRC | 18:29 | |
| *** yamamoto has quit IRC | 18:29 | |
| AJaeger | config-core, infra-root, I would like some feedback on https://review.openstack.org/#/c/538908 - that one removes integrated-gate from nova and adds the single jobs to it so that the irrelevant files lists are respected. I don't like the removal of the template, we will forget to update it ;( Does anybody have better ideas on how to solve these? | 18:31 |
| *** ralonsoh__ has quit IRC | 18:33 | |
| *** slaweq has joined #openstack-infra | 18:33 | |
| *** myoung|food is now known as myoung | 18:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Add third-party-check pipeline https://review.openstack.org/539286 | 18:34 |
| openstackgerrit | Monty Taylor proposed openstack-infra/project-config master: Revert "Temporarily remove ansible/ansible from pipelines" https://review.openstack.org/539287 | 18:34 |
| mordred | corvus, pabelanger: ^^ there's a stab at re-enabling ansible/ansible | 18:35 |
| mordred | AJaeger: looking | 18:35 |
| mordred | AJaeger: ugh. no - I don't have a beter idea yet | 18:36 |
| AJaeger | we could move all the lists to the template so that all projects share it... | 18:37 |
| *** alexchadin has joined #openstack-infra | 18:38 | |
| *** slaweq has quit IRC | 18:38 | |
| *** salv-orl_ has quit IRC | 18:39 | |
| *** salv-orlando has joined #openstack-infra | 18:39 | |
| *** alexchadin has quit IRC | 18:41 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added endpoint update summit speaker assistance https://review.openstack.org/539285 | 18:42 |
| *** armaan has joined #openstack-infra | 18:43 | |
| *** salv-orlando has quit IRC | 18:44 | |
| AJaeger | config-core, could you review https://review.openstack.org/538632 https://review.openstack.org/538631 and https://review.openstack.org/538306 , please? Those remove some jobs from project-config | 18:46 |
| mordred | Shrews: I put an autohold into zuul and it didn't seem to hold the node | 18:46 |
| mordred | Shrews: do I need to put in the autohold before the job starts? | 18:47 |
| *** olaph1 has joined #openstack-infra | 18:47 | |
| *** olaph has quit IRC | 18:49 | |
| mordred | clarkb, pabelanger : I'm still seeing devstack jobs using rax dfw for ubuntu-cloud-archive, but we landed the hack to always use rax for UCA https://review.openstack.org/#/c/535522/ | 18:50 |
| mordred | example: http://logs.openstack.org/32/539032/1/check/osc-functional-devstack/c1bde97/job-output.txt.gz#_2018-01-30_17_53_10_071483 | 18:50 |
| mordred | I don't believe that's why that job failed - but I just notcied it and was confused | 18:51 |
| clarkb | mordred: the jobs running on rax dfw should use rax dfw but the jobs running in other clouds should use the other clouds right? | 18:52 |
| pabelanger | yah, thought we did revert on that | 18:55 |
| clarkb | oh did we revert the fix? | 18:55 |
| clarkb | infra meeting in ~5 minutes. I am currently working to sort out the agenda much later than anticipated as babysitter didn't arrive until a few minutes ago | 18:56 |
| mordred | pabelanger: oh - well, that would explain it at leat | 18:56 |
| fungi | so far, reducing rax-dfw max-servers doesn't seem to have done much to lower the mirror network utilization | 18:56 |
| fungi | pabelanger: do you have any rough idea what job volume rdo has? is it possible _most_ of the utilization on that mirror is rdo and not us? | 18:57 |
| mordred | pabelanger: why did we revert it? and I don't see a revert for that in the tree | 18:57 |
| pabelanger | mordred: no, I thought we did. But I might be confused | 18:57 |
| pabelanger | mordred: Oh, 535522 is the revert. So yes, ignore me | 18:58 |
| *** armaan has quit IRC | 18:58 | |
| pabelanger | fungi: yah, I think that amount of jobs has grown now that they moved from tripleo-test-cloud-rh1 into RDO cloud. It looks like mirror.regionone.rdo-cloud.rdoproject.org is online now, working to switch RDo jobs to now use that | 18:59 |
| Shrews | mordred: just before it finishes, iirc | 18:59 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 18:59 |
| fungi | pabelanger: should we stop using nodes in dfw entirely until we can get rdo admins to stop using our mirror in their ci system? | 18:59 |
| *** armaan has joined #openstack-infra | 19:00 | |
| clarkb | meeting time | 19:00 |
| mordred | Shrews: ok. well, the node doesn't seem to be held :( | 19:00 |
| *** jpena is now known as jpena|off | 19:00 | |
| AJaeger | fungi, pabelanger, we have such a high load here that we need every node. | 19:00 |
| pabelanger | fungi: i think we are still a few hours out from stopping RDO, so we can either firewall their IP range (breaks them) or turn of cloud? | 19:01 |
| fungi | i guess we can discuss during open discussion in the meeting, or after | 19:01 |
| Shrews | mordred: try autohold-list | 19:01 |
| AJaeger | I'm happy to give them a chnace to update it - for a few hours | 19:01 |
| mordred | Shrews: yah- I did that: | openstack | git.openstack.org/openstack/python-openstackclient | osc-functional-devstack | 1 | mordred OSC release bug | 19:02 |
| Shrews | mordred: then it didn't match b/c it would have been removed from the list | 19:03 |
| anticw | does setting WF -1 kill any bulders in progress? | 19:03 |
| Shrews | mordred: i can help more after the meeting | 19:03 |
| Shrews | mordred: possible you might not need the git.o.o part? i forget | 19:04 |
| corvus | anticw: no (on purpose) | 19:04 |
| *** trown|lunch is now known as trown | 19:05 | |
| *** trown is now known as trown|rover | 19:05 | |
| anticw | corvus: is there a machanism to kill jobs in progress? | 19:06 |
| *** amoralej is now known as amoralej|off | 19:06 | |
| *** SumitNaiksatam has joined #openstack-infra | 19:07 | |
| AJaeger | anticw: abandon | 19:07 |
| anticw | thanks | 19:08 |
| *** alexchadin has joined #openstack-infra | 19:09 | |
| AJaeger | anticw: and when you restore, it goes to check again and needs again at +A | 19:09 |
| mordred | AJaeger: abandon doesn't kill the in-progress jobs, does it? | 19:10 |
| AJaeger | mordred: abandon kills running jobs AFAIK | 19:11 |
| *** alexchadin has quit IRC | 19:11 | |
| AJaeger | anticw: rebase also dequeues | 19:11 |
| *** Pramod has quit IRC | 19:12 | |
| *** slaweq has joined #openstack-infra | 19:12 | |
| EmilienM | corvus: I confirm, I checked logs and everything is fine, nice work! | 19:12 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: added endpoint get summit attendance by id https://review.openstack.org/539301 | 19:13 |
| *** camunoz has quit IRC | 19:13 | |
| corvus | EmilienM: yay! mordred ^ EmilienM tested the feature we plan to use to resolve ansible/devel | 19:14 |
| mordred | EmilienM: woot! | 19:14 |
| fungi | nice! | 19:15 |
| *** slaweq has quit IRC | 19:16 | |
| *** harlowja has joined #openstack-infra | 19:17 | |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: added endpoint get summit attendance by id https://review.openstack.org/539301 | 19:18 |
| *** jamesmcarthur has joined #openstack-infra | 19:21 | |
| *** armaan has quit IRC | 19:21 | |
| mordred | Shrews: fwiw, I didn't create the autohold with eh git.o.o prefix | 19:22 |
| *** armaan has joined #openstack-infra | 19:23 | |
| Shrews | mordred: that job did fail, yes? | 19:24 |
| *** yamamoto has joined #openstack-infra | 19:25 | |
| mordred | Shrews: oh yah | 19:25 |
| mordred | Shrews: I hit a recheck on it | 19:25 |
| mordred | so we can see if it catches this time | 19:26 |
| mordred | Shrews: oh - it did fail with RETRY_LIMIT rather than with a normal FAILURE | 19:26 |
| Shrews | mordred: ah, has to be FAILURE iirc | 19:26 |
| mordred | GOTCHA | 19:26 |
| mordred | I'll resubmit and change the job so that it'll reprt ath as failure so we can see what's broken | 19:26 |
| *** lucasagomes is now known as lucas-afk | 19:26 | |
| EmilienM | corvus: which btw I need this feature too for testing github/ceph-ansible with tripleo jobs running in openstack infra | 19:27 |
| mordred | Shrews: thanks | 19:28 |
| corvus | EmilienM: yes -- we're not quite ready for that yet (both technically and policy-wise) but we're getting close and eager to do it | 19:28 |
| *** Pramod has joined #openstack-infra | 19:28 | |
| EmilienM | awesome | 19:29 |
| *** yamamoto has quit IRC | 19:31 | |
| *** camunoz has joined #openstack-infra | 19:34 | |
| *** shardy has quit IRC | 19:42 | |
| *** alexchadin has joined #openstack-infra | 19:43 | |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Create a StoryBoard gui manual https://review.openstack.org/325474 | 19:47 |
| openstackgerrit | Kendall Nelson proposed openstack-infra/storyboard master: Create a StoryBoard gui manual https://review.openstack.org/325474 | 19:48 |
| *** david-lyle has joined #openstack-infra | 19:51 | |
| *** pcaruana has quit IRC | 19:52 | |
| clarkb | Shrews: pabelanger AJaeger I think Shrews intends on fixing the nodepool pool deadlock issue properly so this would be just a temporary workaround | 20:01 |
| clarkb | so I'm fine with it | 20:01 |
| AJaeger | clarkb: as mentioned, let'S try - and monitor ;) | 20:01 |
| *** sambetts is now known as sambetts|afk | 20:01 | |
| clarkb | I've +2'd it but not approved as I am not in a great place to do the monitoring | 20:01 |
| AJaeger | clarkb: same with me - I let others approve | 20:02 |
| tobiash | ups, missed the meeting, looks like I'm already mentally in summer time mode | 20:02 |
| AJaeger | tobiash: it's already dark outside ;) | 20:02 |
| pabelanger | clarkb: Shrews: AJaeger: yah, anything to help keep nodes in-use helps, I've +2'd | 20:03 |
| *** alexchadin has quit IRC | 20:03 | |
| AJaeger | config-core, could you review https://review.openstack.org/538632 https://review.openstack.org/538631 and https://review.openstack.org/538306 , please? Those remove some jobs from project-config | 20:03 |
| AJaeger | nothing urgent - but if you review, consider mine as well, please ;) | 20:04 |
| Shrews | clarkb: AJaeger: pabelanger: how about I WIP it until tomorrow morning my time. We can approve it then and I'll have all day to watch it? | 20:04 |
| AJaeger | Shrews: wfm | 20:04 |
| Shrews | only a couple of hours more until I EOD | 20:04 |
| pabelanger | ++ | 20:04 |
| Shrews | done | 20:05 |
| mordred | clarkb: got a sec for a devstack change? https://review.openstack.org/#/c/510939/1 ? | 20:06 |
| tobiash | AJaeger: almost forgot about that while doing reviews ;) | 20:07 |
| AJaeger | ;) | 20:08 |
| *** jamesmcarthur has quit IRC | 20:11 | |
| *** dsariel has joined #openstack-infra | 20:11 | |
| *** rfolco is now known as rfolco|off | 20:14 | |
| *** iyamahat has quit IRC | 20:14 | |
| *** felipemonteiro has quit IRC | 20:15 | |
| *** felipemonteiro__ has joined #openstack-infra | 20:15 | |
| *** ldnunes has quit IRC | 20:15 | |
| *** slaweq has joined #openstack-infra | 20:17 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: set up tests to run with sqlite https://review.openstack.org/539308 | 20:17 |
| fungi | #status log reenqueued release pipeline jobs for openstack/tripleo-ipsec 8.0.0 | 20:18 |
| openstackstatus | fungi: finished logging | 20:18 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: set up tests to run with sqlite https://review.openstack.org/539308 | 20:18 |
| clarkb | mordred: done | 20:24 |
| mriedem | can we be rechecking things now? | 20:26 |
| mriedem | https://review.openstack.org/#/c/537933/ is merged | 20:26 |
| *** d0ugal has quit IRC | 20:26 | |
| *** yamamoto has joined #openstack-infra | 20:27 | |
| pabelanger | mriedem: yah, believe so. We still having an issue with rax-dfw mirror, but working to address that | 20:29 |
| *** dprince has quit IRC | 20:29 | |
| mriedem | ack | 20:29 |
| *** sai is now known as sai-pto | 20:29 | |
| *** pcichy has quit IRC | 20:30 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Added endpoint get promo code csv format https://review.openstack.org/539312 | 20:32 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Added endpoint get promo code csv format https://review.openstack.org/539312 | 20:33 |
| *** yamamoto has quit IRC | 20:33 | |
| *** sshnaidm is now known as sshnaidm|afk | 20:38 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Provider wedge test https://review.openstack.org/539316 | 20:40 |
| *** d0ugal has joined #openstack-infra | 20:41 | |
| *** e0ne has joined #openstack-infra | 20:42 | |
| *** jamesmcarthur has joined #openstack-infra | 20:46 | |
| mordred | Shrews: poo. the autohold seems like it maybe worked this time (no autohold in the list) but the node is not actually held and is something else now | 20:47 |
| Shrews | mordred: gremlins? | 20:47 |
| mordred | Shrews: I suppose so | 20:48 |
| Shrews | mordred: 0002276213 node is held for you | 20:49 |
| * Shrews hands mordred more coffee | 20:49 | |
| Shrews | dmsimard|afk: you have 2 held nodes that are several days old | 20:50 |
| dmsimard|afk | Shrews: yes it's for a failing fedora job that I have no idea why it is failing, iirc there's the review number it's there for | 20:51 |
| dmsimard|afk | Shrews: between logs.o.o exploding and other things this week I won't have time to look | 20:51 |
| dmsimard|afk | but it's preventing a stack of patches from landing we've been waiting for a while | 20:51 |
| dmsimard|afk | feel free to release them and I can re-hold them once I have time to spend on it | 20:52 |
| Shrews | mordred: are you saying that held node is not what you were expecting? | 20:52 |
| Shrews | dmsimard|afk: i'm just poking you about them b/c we usually forget about our held nodes. i'll let you delete them when you're ready | 20:52 |
| dmsimard|afk | Shrews: I won't have time to look at it this week so they're probably wasted unless someone else wants to check :( | 20:53 |
| *** wolverineav has quit IRC | 20:53 | |
| dmsimard|afk | it's this patch: https://review.openstack.org/#/c/514489/ | 20:53 |
| pabelanger | going to start work on mirror02 | 20:54 |
| *** e0ne has quit IRC | 20:55 | |
| mordred | Shrews: oh - well - it's possible I looked for the node info wrongly | 21:00 |
| *** panda|bbl is now known as panda | 21:02 | |
| mordred | Shrews: yup. pebkac. thank you | 21:03 |
| pabelanger | clarkb: looks like we have 2 volumes for for afscache / proxycache in rax. Any objections to see about just using 1 now? | 21:03 |
| pabelanger | wonder if that was because we added proxycache after afscache volume was created | 21:04 |
| fungi | pabelanger: yeah, we could i suppose switch to different 100gb logical volumes on a single 200gb cinder volume | 21:04 |
| fungi | and you are correct about the timeline there | 21:04 |
| pabelanger | kk | 21:05 |
| fungi | we already had a 100gb cinder volume for afs cache, and so i added another for th apache proxy cache rather than take the outage necessary to migrate between filesystems | 21:05 |
| pabelanger | I might need some help with logical volumes, I usually just use mount_volume.sh script | 21:07 |
| *** iyamahat has joined #openstack-infra | 21:08 | |
| *** slaweq_ has joined #openstack-infra | 21:08 | |
| *** iyamahat has quit IRC | 21:09 | |
| *** iyamahat has joined #openstack-infra | 21:09 | |
| *** felipemonteiro has joined #openstack-infra | 21:10 | |
| *** slaweq_ has quit IRC | 21:13 | |
| *** felipemonteiro__ has quit IRC | 21:13 | |
| pabelanger | okay, believe I done it | 21:15 |
| pabelanger | rebooting to confirm | 21:16 |
| *** olaph has joined #openstack-infra | 21:18 | |
| pabelanger | http://mirror02.dfw.rax.openstack.org/ | 21:19 |
| pabelanger | still waiting for DNS | 21:19 |
| pabelanger | oh, ha. seen an issue | 21:19 |
| *** olaph1 has quit IRC | 21:19 | |
| pabelanger | I might need to expand LV for proxycache, i should have used 100% for lvcreate, but did 50%, so it is only using 1/4 of volume | 21:20 |
| pabelanger | fungi: any suggestions to do that?^ | 21:21 |
| *** eharney_ has joined #openstack-infra | 21:21 | |
| pabelanger | I think lvextend | 21:21 |
| pabelanger | yah | 21:22 |
| pabelanger | https://docs.openstack.org/infra/system-config/sysadmin.html#expanding-an-existing-logical-volume | 21:22 |
| fungi | yep | 21:22 |
| fungi | you got it | 21:22 |
| mordred | Shrews: ok. SO - that autoheld held a change that is not oneof the failng changes | 21:23 |
| *** eharney has quit IRC | 21:23 | |
| *** eharney_ is now known as eharney | 21:23 | |
| pabelanger | there we go | 21:24 |
| pabelanger | yah | 21:24 |
| openstackgerrit | Paul Belanger proposed openstack-infra/system-config master: Add mirror02.dfw.rax.o.o to cacti https://review.openstack.org/539328 | 21:26 |
| mordred | Shrews: but I'm juggling too many brain balls, so it's still probably my fault - I'm taking a stab at debugging via adding lots of prints | 21:26 |
| pabelanger | infra-root: adds mirror02.dfw.rax.o.o to cacti^ I haven't changed the DNS yet on mirror.dfw.rax.o.o. | 21:27 |
| smcginnis | We have a project-config change that needs to merge before we can do a release. | 21:28 |
| smcginnis | Can someone other than AJaeger take a look at https://review.openstack.org/#/c/539231/ | 21:28 |
| * mnaser looks | 21:28 | |
| mnaser | smcginnis: done | 21:29 |
| smcginnis | mnaser: Thanks! | 21:29 |
| *** kgiusti has left #openstack-infra | 21:29 | |
| mnaser | np | 21:29 |
| *** yamamoto has joined #openstack-infra | 21:29 | |
| *** eharney has quit IRC | 21:30 | |
| pabelanger | fungi: clarkb: okay, RDO is building replacement DIBs, but it doesn't look to be a fast filesystem. Something going on with hardware on their side, but rough math is another 8 hours for image to finish building (unless something changes behind the scenes). | 21:32 |
| *** ianychoi has quit IRC | 21:34 | |
| *** ianychoi has joined #openstack-infra | 21:35 | |
| *** yamamoto has quit IRC | 21:35 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: set up tests to run with sqlite https://review.openstack.org/539308 | 21:36 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: convert list of auth_codes to single value https://review.openstack.org/539331 | 21:36 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: do not allow story id to change via payload https://review.openstack.org/539332 | 21:36 |
| fungi | pabelanger: thanks, i've also sort of been following along in #rdo | 21:36 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove zuul._projects https://review.openstack.org/535518 | 21:37 |
| Shrews | mordred: that seems unpossible that it's not a failing change | 21:38 |
| openstackgerrit | Merged openstack-infra/project-config master: Change searchlight-ui to use publish-to-pypi-horizon https://review.openstack.org/539231 | 21:38 |
| *** eharney has joined #openstack-infra | 21:39 | |
| pabelanger | fungi: kk | 21:39 |
| *** threestrands has joined #openstack-infra | 21:40 | |
| *** trown|rover is now known as trown|outtypewww | 21:42 | |
| *** armaan has quit IRC | 21:46 | |
| pabelanger | fungi: clarkb: what time do we want to consider DNS update for mirror02? | 21:46 |
| *** armaan has joined #openstack-infra | 21:47 | |
| *** andreww has joined #openstack-infra | 21:47 | |
| clarkb | corvus mentioned a few hours from meeting so probably anytime after now if still no update on the ticket | 21:48 |
| clarkb | pabelanger: ^ | 21:48 |
| pabelanger | clarkb: okay, lets first land https://review.openstack.org/539328/ and make sure cacti is happy, then we can update DNS | 21:49 |
| *** xarses_ has quit IRC | 21:50 | |
| clarkb | still no update on ticket? (I'm not in a good spot to check | 21:51 |
| clarkb | I +2'd that change too | 21:51 |
| *** xarses_ has joined #openstack-infra | 21:51 | |
| pabelanger | trying to pull that up myself | 21:51 |
| *** andreww has quit IRC | 21:51 | |
| pabelanger | clarkb: yah, nothing back yet | 21:52 |
| mordred | Shrews: yah - I'm pretty sure I'm just looking at things wrong | 21:53 |
| *** Goneri has quit IRC | 21:54 | |
| *** slaweq has quit IRC | 21:55 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/storyboard master: set up tests to run with sqlite https://review.openstack.org/539308 | 21:55 |
| *** panda is now known as panda|off | 21:57 | |
| *** felipemonteiro has quit IRC | 21:57 | |
| *** eharney has quit IRC | 21:57 | |
| smcginnis | I see high-ish queue lengths in zuul.o.o. Something to worry about? | 21:57 |
| *** felipemonteiro has joined #openstack-infra | 21:57 | |
| openstackgerrit | Merged openstack-infra/nodepool master: zk: check for client in properties https://review.openstack.org/535559 | 21:58 |
| clarkb | smcginnis: if they dont fall in 5-10 minutes then maybe | 21:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Add third-party-check pipeline https://review.openstack.org/539286 | 21:58 |
| openstackgerrit | Merged openstack-infra/project-config master: Revert "Temporarily remove ansible/ansible from pipelines" https://review.openstack.org/539287 | 21:58 |
| smcginnis | clarkb: Oh, already have dropped. | 21:58 |
| smcginnis | Hmm, actually, fluctuating all over. Guess that means things are moving. | 21:59 |
| *** mylu has joined #openstack-infra | 21:59 | |
| clarkb | ya moving is good as long as it goes both directions | 21:59 |
| pabelanger | topic:zuulv3-projects has started again, I think we've entered increase of dynamic reloads on zuul, but memory looks flatish | 22:00 |
| *** olaph1 has joined #openstack-infra | 22:00 | |
| *** olaph has quit IRC | 22:03 | |
| corvus | pabelanger: it has? | 22:08 |
| corvus | (i haven't done anything) | 22:10 |
| pabelanger | corvus: I'm not sure how to confirm, aside from looking at topic and +W, but it does seem zuul is increased dynamic reloads. Again, I am not sure how best to track that | 22:11 |
| *** edmondsw has quit IRC | 22:11 | |
| corvus | pabelanger: is it something we need to track? | 22:12 |
| *** edmondsw has joined #openstack-infra | 22:12 | |
| *** mihalis68 has quit IRC | 22:13 | |
| *** priteau has joined #openstack-infra | 22:13 | |
| pabelanger | corvus: I think so, we spent a lot of time yesterday recovering from outage, but results queue grew, then emptied, and repeated for a few hours | 22:14 |
| *** salv-orlando has joined #openstack-infra | 22:14 | |
| corvus | pabelanger: that's pretty normal. | 22:14 |
| pabelanger | okay, maybe I'm over thinking it | 22:14 |
| pabelanger | but seem to have been growing ready nodes (upwards of 300 ready) then once results queue empties they get allocated to zuul. | 22:15 |
| *** xarses_ has quit IRC | 22:15 | |
| *** edmondsw has quit IRC | 22:17 | |
| *** rcernin has quit IRC | 22:18 | |
| *** xarses_ has joined #openstack-infra | 22:19 | |
| *** andreww has joined #openstack-infra | 22:20 | |
| *** andreww has quit IRC | 22:20 | |
| *** andreww has joined #openstack-infra | 22:21 | |
| *** tpsilva has quit IRC | 22:22 | |
| *** xarses_ has quit IRC | 22:23 | |
| pabelanger | ianw: I'll be EOD shortly, do you mind taking over on https://review.openstack.org/539328/ ? Thing we want to confirm cacti.o.o is good for mirror02.dfw.rax.o.o then, if no reply to open ticket in rax, update DNS for mirror.dfw.rax.o.o to point to mirror02 | 22:24 |
| pabelanger | clarkb: ^ | 22:24 |
| ianw | pabelanger: ok, so we're replacing or round-robining? | 22:25 |
| pabelanger | ianw: I think replace to start, I'm not sure what is involved with round-robining right now. | 22:26 |
| corvus | you can't rr a cname, you'd have to replace it with a records anyway. | 22:26 |
| corvus | (which is fine; just pointing out if we wanted to rr, we'd need to drop the cname and add 2 A records and 2 AAAA records) | 22:27 |
| *** yamamoto has joined #openstack-infra | 22:27 | |
| *** priteau has quit IRC | 22:27 | |
| *** felipemonteiro__ has joined #openstack-infra | 22:27 | |
| corvus | i don't see a need to rr though -- one server should supply plenty of bandwidth. | 22:28 |
| ianw | yep, anyway let's KISS with a replacement. so i'll wait until cacti is seeing it so we've actually got stats if it's any better then update | 22:28 |
| corvus | ++ | 22:28 |
| pabelanger | wfm | 22:28 |
| *** felipemonteiro has quit IRC | 22:31 | |
| ianw | should we run a md5sum or something on the afs to prime it? | 22:31 |
| corvus | we use a really small subset, that would probably take unreasonably long | 22:32 |
| *** esberglu has quit IRC | 22:32 | |
| corvus | however, a pip install of openstack/requirements might be worthwhile | 22:32 |
| *** jcoufal has quit IRC | 22:35 | |
| *** bfournie has quit IRC | 22:44 | |
| *** hemna_ has quit IRC | 22:45 | |
| *** andreww has quit IRC | 22:51 | |
| *** rcernin has joined #openstack-infra | 22:52 | |
| *** xarses_ has joined #openstack-infra | 22:54 | |
| *** xarses_ has quit IRC | 22:54 | |
| *** xarses_ has joined #openstack-infra | 22:55 | |
| *** armax has quit IRC | 22:56 | |
| *** andreas_s has joined #openstack-infra | 22:58 | |
| *** xarses_ has quit IRC | 22:59 | |
| *** mriedem is now known as mriedem_afk | 23:02 | |
| *** andreas_s has quit IRC | 23:02 | |
| *** dave-mccowan has quit IRC | 23:03 | |
| *** ianychoi has quit IRC | 23:05 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Merger: retry network operations https://review.openstack.org/539356 | 23:11 |
| *** yamamoto_ has joined #openstack-infra | 23:12 | |
| ianw | interesting that 8.8.8.8 still seems to work in cn1 | 23:12 |
| ianw | but whatever resolver is given by default has omissions | 23:14 |
| ianw | $ host twitter.com | 23:14 |
| ianw | Host twitter.com not found: 2(SERVFAIL) | 23:14 |
| *** yamamoto has quit IRC | 23:15 | |
| clarkb | puppet should set up unbound then use opendns and google dns | 23:16 |
| clarkb | so that may not be a major issue for us | 23:16 |
| ianw | yep, it seems resolving via them is fine ... just as long as people aren't doing "ping -c1 facebook.com # is the network up?" :) | 23:19 |
| fungi | some firewalls are just greater than others, that's all | 23:20 |
| *** salv-orlando has quit IRC | 23:21 | |
| *** salv-orlando has joined #openstack-infra | 23:22 | |
| mordred | corvus (and other infra-root) ... we are running an integration test on an ansible PR! | 23:22 |
| * fungi cheers from the sidelines | 23:24 | |
| *** jamesmcarthur has quit IRC | 23:24 | |
| mordred | corvus: fwiw, the link we provided to the status link on the GH PR is "https://zuul01.openstack.org/" | 23:24 |
| mordred | https://github.com/ansible/ansible/pull/20974 is the PR I just triggered, in case anyone wants to see | 23:25 |
| *** xarses_ has joined #openstack-infra | 23:25 | |
| *** salv-orlando has quit IRC | 23:26 | |
| *** andreww has joined #openstack-infra | 23:27 | |
| *** andreww has quit IRC | 23:27 | |
| *** andreww has joined #openstack-infra | 23:28 | |
| *** jamesmcarthur has joined #openstack-infra | 23:28 | |
| *** andreww has quit IRC | 23:28 | |
| *** andreww has joined #openstack-infra | 23:28 | |
| corvus | mordred: i don't see any zuul-related links on that; maybe 'contributors' only? | 23:29 |
| *** jamesmcarthur has quit IRC | 23:29 | |
| corvus | mordred: oh, the little yellow dot opens up a popup with zuul in it | 23:29 |
| *** xarses_ has quit IRC | 23:30 | |
| dmsimard|afk | Huh, Red Hat acquires CoreOS. | 23:30 |
| corvus | mordred: looks like we need to set webapp.status_url in zuul.conf? | 23:31 |
| corvus | mordred: (which probably needs to change to web.status_url ?) | 23:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Set status_url for zuul https://review.openstack.org/539357 | 23:33 |
| mordred | corvus: yah ^^ | 23:33 |
| mordred | corvus: but also, yeah - we can hold off on that until we update that to web.status_url | 23:34 |
| corvus | mordred: let's move in parallel | 23:35 |
| *** olaph has joined #openstack-infra | 23:36 | |
| *** olaph1 has quit IRC | 23:37 | |
| *** jamesmcarthur has joined #openstack-infra | 23:38 | |
| *** jamesmcarthur has quit IRC | 23:38 | |
| *** slaweq has joined #openstack-infra | 23:40 | |
| fungi | #status log scheduled maintenance will make the citycloud-sto2 api endpoint unavailable intermittently between 2018-01-31 07:00 and 2018-02-01 07:00 utc | 23:42 |
| openstackstatus | fungi: finished logging | 23:42 |
| mordred | fungi: have a sec for an easy +A? https://review.openstack.org/#/c/539357/ | 23:43 |
| fungi | #status log scheduled maintenance will make the citycloud-kna1 api endpoint unavailable intermittently between 2018-02-07 07:00 and 2018-02-08 07:00 utc | 23:43 |
| openstackstatus | fungi: finished logging | 23:43 |
| *** yamahata has quit IRC | 23:44 | |
| *** slaweq has quit IRC | 23:45 | |
| *** armaan has quit IRC | 23:45 | |
| *** iyamahat has quit IRC | 23:47 | |
| *** dhill_ has quit IRC | 23:47 | |
| *** dsariel has quit IRC | 23:48 | |
| *** stakeda has joined #openstack-infra | 23:49 | |
| *** abelur_ has joined #openstack-infra | 23:50 | |
| *** ganso has quit IRC | 23:53 | |
| *** dingyichen has joined #openstack-infra | 23:54 | |
| *** rcernin has quit IRC | 23:57 | |
| tonyb | I saw something that installing packagines in ubuntu nodes is failing and we shoudl avoid rechecks .... is that still the case? | 23:58 |
| ianw | tonyb: pabelanger fixed the mirror, so should be ok now | 23:59 |
| tonyb | ianw: Thanks. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!