| ianw | for the last 4+ hours, gra1 hasn't leaked any ports | 00:03 |
|---|---|---|
| *** hamzy has joined #openstack-infra | 00:05 | |
| openstackgerrit | MarcH proposed openstack-infra/git-review master: add option to pick an entire HEAD..FETCH_HEAD series https://review.openstack.org/612834 | 00:05 |
| clarkb | nice | 00:06 |
| *** jbadiapa has quit IRC | 00:08 | |
| *** jbadiapa has joined #openstack-infra | 00:11 | |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** smarcet has quit IRC | 00:15 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Don't let openstacksdk run a taskmanager during testing https://review.openstack.org/612835 | 00:16 |
| *** smarcet has joined #openstack-infra | 00:21 | |
| *** smarcet has quit IRC | 00:23 | |
| *** smarcet has joined #openstack-infra | 00:23 | |
| *** felipemonteiro has joined #openstack-infra | 00:25 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul master: Add the process environment to zuul.conf parser https://review.openstack.org/612824 | 00:44 |
| *** slaweq has quit IRC | 00:45 | |
| *** longkb has joined #openstack-infra | 00:53 | |
| *** smarcet has quit IRC | 00:59 | |
| *** slaweq has joined #openstack-infra | 01:12 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Turn on quiet mode when logfile specified https://review.openstack.org/612865 | 01:17 |
| *** mrsoul has joined #openstack-infra | 01:17 | |
| *** hongbin has joined #openstack-infra | 01:19 | |
| *** smarcet has joined #openstack-infra | 01:22 | |
| *** imacdonn has quit IRC | 01:23 | |
| *** imacdonn has joined #openstack-infra | 01:24 | |
| *** xinliang has quit IRC | 01:25 | |
| *** roman_g has quit IRC | 01:29 | |
| *** roman_g has joined #openstack-infra | 01:29 | |
| *** xinliang has joined #openstack-infra | 01:30 | |
| *** bobh has joined #openstack-infra | 01:38 | |
| *** smarcet has quit IRC | 01:40 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix epel repo rewrite, add to testing https://review.openstack.org/612636 | 01:43 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Turn on quiet mode when logfile specified https://review.openstack.org/612865 | 01:43 |
| *** slaweq has quit IRC | 01:44 | |
| *** fuentess has quit IRC | 01:53 | |
| *** smarcet has joined #openstack-infra | 02:06 | |
| *** cfriesen has quit IRC | 02:08 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** felipemonteiro has quit IRC | 02:20 | |
| *** slaweq has quit IRC | 02:44 | |
| *** jcoufal has quit IRC | 02:50 | |
| *** psachin has joined #openstack-infra | 02:55 | |
| *** sambetts|afk has quit IRC | 02:56 | |
| *** bhavikdbavishi has joined #openstack-infra | 02:56 | |
| *** truongnh__ has joined #openstack-infra | 03:03 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** yamamoto has quit IRC | 03:21 | |
| *** yamamoto has joined #openstack-infra | 03:21 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Don't let openstacksdk run a taskmanager during testing https://review.openstack.org/612835 | 03:21 |
| *** zhangfei has joined #openstack-infra | 03:38 | |
| *** bobh has quit IRC | 03:41 | |
| *** slaweq has quit IRC | 03:45 | |
| *** ramishra has joined #openstack-infra | 03:49 | |
| *** udesale has joined #openstack-infra | 03:50 | |
| *** udesale has quit IRC | 03:50 | |
| *** udesale has joined #openstack-infra | 03:51 | |
| *** slaweq has joined #openstack-infra | 04:12 | |
| *** smarcet has quit IRC | 04:12 | |
| *** zhangfei has quit IRC | 04:18 | |
| *** janki has joined #openstack-infra | 04:23 | |
| *** spsurya has joined #openstack-infra | 04:24 | |
| *** hongbin has quit IRC | 04:25 | |
| *** bhavikdbavishi has quit IRC | 04:35 | |
| *** bhavikdbavishi has joined #openstack-infra | 04:36 | |
| *** slaweq has quit IRC | 04:45 | |
| *** agopi has quit IRC | 04:45 | |
| *** ykarel has joined #openstack-infra | 04:46 | |
| *** agopi has joined #openstack-infra | 04:47 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Normalise more of the API stats calls https://review.openstack.org/605898 | 04:57 |
| *** dave-mccowan has quit IRC | 04:57 | |
| *** chkumar|off is now known as chandankumar | 05:03 | |
| *** auristor has quit IRC | 05:05 | |
| *** felipemonteiro has joined #openstack-infra | 05:07 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** auristor has joined #openstack-infra | 05:11 | |
| *** truongnh__ has quit IRC | 05:18 | |
| *** niedbalski has joined #openstack-infra | 05:20 | |
| *** rlandy|bbl has quit IRC | 05:21 | |
| *** ramishra has quit IRC | 05:34 | |
| *** ramishra has joined #openstack-infra | 05:39 | |
| *** armax has quit IRC | 05:40 | |
| *** slaweq has quit IRC | 05:44 | |
| *** kjackal has joined #openstack-infra | 05:44 | |
| *** sshnaidm|pto is now known as sshnaidm|ruck | 05:45 | |
| *** rkukura_ has joined #openstack-infra | 05:51 | |
| *** rkukura has quit IRC | 05:54 | |
| *** rkukura_ is now known as rkukura | 05:54 | |
| AJaeger | config-core, couple of reviews for your consideration, please https://review.openstack.org/602783 https://review.openstack.org/612419 https://review.openstack.org/612705 | 06:04 |
| *** truongnh__ has joined #openstack-infra | 06:05 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** ddurst has quit IRC | 06:15 | |
| *** armax has joined #openstack-infra | 06:15 | |
| *** diablo_rojo has quit IRC | 06:18 | |
| *** aojea has joined #openstack-infra | 06:18 | |
| *** diablo_rojo has joined #openstack-infra | 06:19 | |
| *** quiquell|off is now known as quiquell | 06:21 | |
| *** ddurst has joined #openstack-infra | 06:29 | |
| *** raginbajin has quit IRC | 06:32 | |
| *** raginbajin has joined #openstack-infra | 06:34 | |
| openstackgerrit | Merged openstack-infra/project-config master: Add starlingx-docs-core group https://review.openstack.org/612419 | 06:42 |
| openstackgerrit | Merged openstack-infra/project-config master: Added template 'publish-to-pypi-python3' to Iotronic projects. https://review.openstack.org/612705 | 06:42 |
| *** truongnh__ has quit IRC | 06:45 | |
| *** ddurst has quit IRC | 06:46 | |
| *** ccamacho has joined #openstack-infra | 06:47 | |
| *** ddurst has joined #openstack-infra | 06:55 | |
| quiquell | AJaeger: Do you know where can I find a centos8 docker image or virtual machine ? | 07:02 |
| *** bhavikdbavishi has quit IRC | 07:02 | |
| *** rcernin has quit IRC | 07:03 | |
| *** bhavikdbavishi has joined #openstack-infra | 07:03 | |
| *** eernst has joined #openstack-infra | 07:05 | |
| *** pcaruana has joined #openstack-infra | 07:05 | |
| *** felipemonteiro has quit IRC | 07:06 | |
| cgoncalves | ianw, oh, good catch! | 07:12 |
| *** threestrands has quit IRC | 07:13 | |
| scas | quiquell: 8 is highly unlikely to exist, as rhel does not have an alpha published at this time | 07:13 |
| quiquell | scas: ack | 07:19 |
| *** ykarel is now known as ykarel|lunch | 07:20 | |
| *** cfriesen has joined #openstack-infra | 07:21 | |
| *** bhavikdbavishi has quit IRC | 07:25 | |
| *** xek has joined #openstack-infra | 07:31 | |
| *** ginopc has joined #openstack-infra | 07:32 | |
| ianw | cgoncalves: i've got a couple of reviews out there trying to move the dib logs into a file ... hopefully they're close to working | 07:34 |
| *** carl_cai has joined #openstack-infra | 07:35 | |
| *** aojea has quit IRC | 07:36 | |
| *** aojea has joined #openstack-infra | 07:37 | |
| *** cfriesen has quit IRC | 07:39 | |
| *** ifat_afek has joined #openstack-infra | 07:40 | |
| *** lpetrut has joined #openstack-infra | 07:41 | |
| *** aojea has quit IRC | 07:48 | |
| *** jpena|off is now known as jpena | 07:48 | |
| *** roman_g has quit IRC | 07:51 | |
| *** tosky has joined #openstack-infra | 07:52 | |
| *** e0ne has joined #openstack-infra | 07:55 | |
| *** jpich has joined #openstack-infra | 08:01 | |
| *** ifat_afek has quit IRC | 08:01 | |
| *** ianychoi_ is now known as ianychoi | 08:03 | |
| *** diablo_rojo has quit IRC | 08:04 | |
| *** eernst has quit IRC | 08:08 | |
| *** bhavikdbavishi has joined #openstack-infra | 08:08 | |
| *** ramishra has quit IRC | 08:16 | |
| *** ramishra has joined #openstack-infra | 08:17 | |
| *** hashar has joined #openstack-infra | 08:18 | |
| *** gfidente has joined #openstack-infra | 08:23 | |
| cgoncalves | ianw, yep, I saw them. thanks! | 08:28 |
| *** ykarel|lunch is now known as ykarel | 08:28 | |
| *** roman_g has joined #openstack-infra | 08:32 | |
| *** derekh has joined #openstack-infra | 08:39 | |
| *** priteau has joined #openstack-infra | 08:48 | |
| *** vabada has quit IRC | 08:51 | |
| *** vabada has joined #openstack-infra | 08:52 | |
| *** maciejjozefczyk has quit IRC | 09:02 | |
| *** maciejjozefczyk has joined #openstack-infra | 09:02 | |
| *** kopecmartin|off is now known as kopecmartin | 09:09 | |
| *** Emine has joined #openstack-infra | 09:13 | |
| *** xinliang has quit IRC | 09:13 | |
| *** aojea has joined #openstack-infra | 09:16 | |
| *** haitruonghp has joined #openstack-infra | 09:25 | |
| *** ifat_afek has joined #openstack-infra | 09:27 | |
| *** haitruonghp is now known as truongnh___ | 09:30 | |
| *** truongnh___ has quit IRC | 09:30 | |
| openstackgerrit | Tung Doan proposed openstack-infra/project-config master: Add release tag for Apmec project https://review.openstack.org/612962 | 09:31 |
| *** dtantsur|afk is now known as dtantsur | 09:34 | |
| *** jesusaur has quit IRC | 09:45 | |
| *** jesusaur has joined #openstack-infra | 09:49 | |
| *** rossella_s has joined #openstack-infra | 09:53 | |
| *** panda has quit IRC | 09:54 | |
| *** panda has joined #openstack-infra | 09:55 | |
| *** yamamoto has quit IRC | 09:58 | |
| *** yamamoto has joined #openstack-infra | 09:59 | |
| *** yamamoto has quit IRC | 10:03 | |
| *** kjackal has quit IRC | 10:03 | |
| *** kjackal has joined #openstack-infra | 10:03 | |
| *** longkb has quit IRC | 10:04 | |
| *** kjackal has quit IRC | 10:08 | |
| *** kjackal_v2 has joined #openstack-infra | 10:08 | |
| *** eernst has joined #openstack-infra | 10:13 | |
| *** eernst has quit IRC | 10:19 | |
| *** quiquell has quit IRC | 10:29 | |
| *** quiquell has joined #openstack-infra | 10:30 | |
| *** ifat_afek has quit IRC | 10:36 | |
| *** rossella_s has quit IRC | 10:41 | |
| *** yamamoto has joined #openstack-infra | 10:42 | |
| *** ifat_afek has joined #openstack-infra | 10:44 | |
| *** pbourke has quit IRC | 10:48 | |
| *** pbourke has joined #openstack-infra | 10:48 | |
| *** udesale has quit IRC | 11:01 | |
| *** rook has joined #openstack-infra | 11:11 | |
| *** dave-mccowan has joined #openstack-infra | 11:14 | |
| rook | hey all - trying to get the openstack/openstackstatus bot to join openstack-browbeat, however, not having much luck | 11:15 |
| dtroyer | Good morning infra! could someone add me to the newly-minted starlingx-docs-core group in Gerrit? https://review.openstack.org/612419 | 11:16 |
| *** bhavikdbavishi has quit IRC | 11:18 | |
| *** xek has quit IRC | 11:21 | |
| *** xek has joined #openstack-infra | 11:22 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix issue in Github connection with large diffs https://review.openstack.org/612989 | 11:27 |
| *** ldnunes has joined #openstack-infra | 11:34 | |
| *** jpena is now known as jpena|lunch | 11:36 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix issue in Github connection with large diffs https://review.openstack.org/612989 | 11:45 |
| *** ansmith has quit IRC | 11:48 | |
| amorin | ianw: fungi AJaeger ping | 11:49 |
| amorin | on grafana node pool | 11:49 |
| amorin | I see that it's better on gra1, do you know if you still have some errors in your logs? | 11:49 |
| *** carl_cai has quit IRC | 11:51 | |
| *** rh-jelabarre has joined #openstack-infra | 11:57 | |
| *** trown|outtypewww is now known as trown | 11:59 | |
| *** haleyb has quit IRC | 12:07 | |
| *** haleyb has joined #openstack-infra | 12:10 | |
| *** yamamoto has quit IRC | 12:11 | |
| *** yamamoto has joined #openstack-infra | 12:11 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Reduce socket connect timeout in nodescan https://review.openstack.org/608897 | 12:14 |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Fix issue in Github connection with large diffs https://review.openstack.org/612989 | 12:14 |
| *** udesale has joined #openstack-infra | 12:17 | |
| *** quiquell is now known as quiquell|lunch | 12:19 | |
| *** yamamoto has quit IRC | 12:20 | |
| *** yamamoto has joined #openstack-infra | 12:20 | |
| *** yamamoto has quit IRC | 12:24 | |
| *** udesale has quit IRC | 12:25 | |
| *** ifat_afek has quit IRC | 12:28 | |
| *** ifat_afek has joined #openstack-infra | 12:30 | |
| *** rlandy has joined #openstack-infra | 12:31 | |
| *** fuentess has joined #openstack-infra | 12:32 | |
| *** jpena|lunch is now known as jpena | 12:37 | |
| *** jrist has quit IRC | 12:40 | |
| *** smarcet has joined #openstack-infra | 12:44 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: DNM: testing zookeeper oddities https://review.openstack.org/612750 | 12:50 |
| *** kgiusti has joined #openstack-infra | 12:51 | |
| *** yamamoto has joined #openstack-infra | 12:55 | |
| *** ifat_afek has quit IRC | 12:57 | |
| *** ansmith has joined #openstack-infra | 12:57 | |
| *** eharney has joined #openstack-infra | 12:58 | |
| *** sthussey has joined #openstack-infra | 12:59 | |
| *** Douhet has quit IRC | 13:02 | |
| fungi | rook: you need to propose a change to add the channel name to both the statusbot_channels and meetbot_channels lists in https://git.openstack.org/cgit/openstack-infra/system-config/tree/hiera/common.yaml | 13:02 |
| *** dave-mccowan has quit IRC | 13:02 | |
| fungi | dtroyer: done! | 13:02 |
| fungi | amorin: i'll check for errors now | 13:02 |
| *** Douhet has joined #openstack-infra | 13:03 | |
| amorin | ok thanks | 13:03 |
| amorin | I see that error node launch attempts is around 7 | 13:04 |
| *** dave-mccowan has joined #openstack-infra | 13:04 | |
| *** bnemec has joined #openstack-infra | 13:04 | |
| amorin | I dont know which region it is | 13:04 |
| *** jd_ has quit IRC | 13:04 | |
| fungi | looks like nl04 is the nodepool launcher for that region, so digging into its logs now | 13:04 |
| amorin | ok | 13:04 |
| *** dtantsur has quit IRC | 13:04 | |
| *** jd_ has joined #openstack-infra | 13:06 | |
| *** hashar is now known as hasharAway | 13:07 | |
| fungi | amorin: most recent error was in the last minute, but i don't think it's an error from the api. i wonder if we screwed up configuration on our end or whether we're hitting a bug in nodepool. creating 8d0cd3d9-fef7-4555-b612-c0df9fabf354 raised openstack.cloud.exc.OpenStackCloudCreateException from openstack/cloud/openstackcloud.py line 6831, in create_server with ":param block_device_mapping_v2: | 13:08 |
| fungi | (optional) A dict of block" | 13:08 |
| fungi | or maybe that's saying it didn't get a response it could parse? | 13:08 |
| *** dtantsur has joined #openstack-infra | 13:09 | |
| fungi | oh, though we're also getting openstack.exceptions.SDKException: Error in creating the server: No valid host was found. There are not enough hosts available. | 13:09 |
| fungi | amorin: so maybe that's the actual issue? | 13:10 |
| fungi | we've gotten that error a couple hundred times over the past hour | 13:11 |
| *** beagles is now known as beagles_mtg | 13:12 | |
| fungi | oh, though i didn't correlate these by region, just a sec | 13:13 |
| amorin | our database is under heavy load since one hour | 13:13 |
| fungi | unfortunately these errors don't say what cloud/region they map to so it will take a bit for me to provide a solid correlation | 13:13 |
| amorin | ok so nevermind, | 13:14 |
| amorin | on the hosts directly | 13:14 |
| amorin | I saw no errors | 13:14 |
| amorin | so maybe the errors were from BHS1 | 13:14 |
| amorin | we still need to apply some tuning on BHS1 | 13:14 |
| *** yamamoto has quit IRC | 13:14 | |
| amorin | we were trying GRA1 first | 13:14 |
| *** smarcet has quit IRC | 13:15 | |
| fungi | yeah, the first one i checked for "no valid hosts" was bhs1 | 13:15 |
| openstackgerrit | Tung Doan proposed openstack-infra/project-config master: Add release tag and remove python jobs for Apmec https://review.openstack.org/612962 | 13:15 |
| amorin | ok | 13:16 |
| fungi | the first "block_device_mapping_v2" one i checked was also in bhs1 | 13:16 |
| amorin | great | 13:16 |
| fungi | looks like we're getting some of both in gra1 as well | 13:21 |
| *** udesale has joined #openstack-infra | 13:22 | |
| fungi | at 13:18:37z for example we got a "no valid host was found" trying to launch node ubuntu-xenial-ovh-gra1-0003377451 | 13:22 |
| *** jcoufal has joined #openstack-infra | 13:23 | |
| amorin | ok | 13:23 |
| fungi | same for centos-7-ovh-gra1-0003377399 at 13:17:39 | 13:24 |
| fungi | we're getting around 1-2 launch failures a minute in gra1 | 13:25 |
| fungi | and keep in mind that we try to launch a node up to three times before we consider the launch a failure | 13:25 |
| fungi | so once or twice a minute we're failing three consecutive tries to launch an instance | 13:26 |
| *** markmcd has quit IRC | 13:27 | |
| fungi | not sure how many boot calls on average we're failing since some may be succeeding on the second or third try | 13:27 |
| amorin | 1-2 launch failure per minutes on gra1? | 13:27 |
| amorin | can you give me instance ID for example? | 13:27 |
| fungi | yes, that's what it looks like from the launcher log | 13:27 |
| fungi | 01ab3207-6ac6-438e-bdad-a58f6e294b80 was one from a minute or so ago | 13:28 |
| fungi | would have been named centos-7-ovh-gra1-0003377829 | 13:29 |
| fungi | raised openstack.exceptions.SDKException: Error in creating the server: No valid host was found. There are not enough hosts available. | 13:29 |
| *** jamesmcarthur has joined #openstack-infra | 13:31 | |
| *** boden has joined #openstack-infra | 13:32 | |
| openstackgerrit | Joe Talerico proposed openstack-infra/system-config master: Adding openstack-browbeat https://review.openstack.org/613022 | 13:33 |
| amorin | fungi: I will add a host in the aggregate, we are hitting max number of instances most of the time I think | 13:34 |
| rook | fungi: ^^ Thanks! | 13:34 |
| *** rossella_s has joined #openstack-infra | 13:36 | |
| *** agopi is now known as agopi|brb | 13:38 | |
| *** trown is now known as trown|brb | 13:38 | |
| fungi | thanks amorin! | 13:39 |
| *** agopi|brb has quit IRC | 13:43 | |
| *** smarcet has joined #openstack-infra | 13:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Collect docker logs after quick-start run https://review.openstack.org/613027 | 13:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add the process environment to zuul.conf parser https://review.openstack.org/612824 | 13:46 |
| *** yamamoto has joined #openstack-infra | 13:46 | |
| dtroyer | fungi: thanks! | 13:48 |
| *** felipemonteiro has joined #openstack-infra | 13:49 | |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove diskimage-builder meeting https://review.openstack.org/612696 | 13:54 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove Puppet-OpenStack team meeting https://review.openstack.org/612701 | 13:54 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove OSops team meeting https://review.openstack.org/612703 | 13:54 |
| openstackgerrit | Merged openstack-infra/irc-meetings master: Remove OpenStackClient team meeting https://review.openstack.org/612702 | 13:54 |
| *** trown|brb is now known as trown | 13:59 | |
| *** janki has quit IRC | 14:00 | |
| *** ifat_afek has joined #openstack-infra | 14:01 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: Add prepare-workspace-git role https://review.openstack.org/613036 | 14:04 |
| *** jrist has joined #openstack-infra | 14:06 | |
| *** udesale has quit IRC | 14:07 | |
| *** agopi|brb has joined #openstack-infra | 14:08 | |
| *** agopi|brb is now known as agopi | 14:08 | |
| *** jrist has quit IRC | 14:09 | |
| *** jrist has joined #openstack-infra | 14:09 | |
| *** markmcd has joined #openstack-infra | 14:21 | |
| *** jrist has quit IRC | 14:24 | |
| *** ifat_afek has quit IRC | 14:26 | |
| openstackgerrit | Stephen Finucane proposed openstack-infra/git-review master: docs: Call out use of an agent to store SSH passwords https://review.openstack.org/610616 | 14:27 |
| *** quiquell|lunch is now known as quiquell | 14:32 | |
| *** beagles_mtg is now known as beagles_food | 14:33 | |
| *** mmedvede has joined #openstack-infra | 14:36 | |
| *** eernst has joined #openstack-infra | 14:37 | |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources master: Updated speaker other_presentation_links collection for PUT/POST actions https://review.openstack.org/613056 | 14:40 |
| *** ykarel is now known as ykarel|away | 14:41 | |
| *** jrist has joined #openstack-infra | 14:41 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: DNM: testing zookeeper oddities https://review.openstack.org/612750 | 14:42 |
| openstackgerrit | Merged openstack-infra/openstackid-resources master: Updated speaker other_presentation_links collection for PUT/POST actions https://review.openstack.org/613056 | 14:43 |
| *** felipemonteiro has quit IRC | 14:44 | |
| *** eernst has quit IRC | 14:49 | |
| clarkb | Shrews: corvus mordred the nodepool builder move has run into what I expect is the first major hurdle. Image uploads to rackspace are not working | 14:57 |
| Shrews | clarkb: because of zk problems, or rax issues? | 14:58 |
| clarkb | haven't dug in yet (sorry still early for me and I have a meeting in ~2 minutes) | 14:58 |
| clarkb | just noticing its many retries on a bunch of image uploads to rax | 14:58 |
| Shrews | clarkb: i can't dig either. i'm already late for a chiro appt so have to leave now. biab... | 14:59 |
| *** cfriesen has joined #openstack-infra | 14:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Merger: automatically add new hosts to the known_hosts file https://review.openstack.org/608453 | 14:59 |
| corvus | clarkb: which node? | 15:03 |
| *** beagles_food is now known as beagles | 15:03 | |
| clarkb | corvus: nb01 and nb02 | 15:03 |
| clarkb | if you run image-list there you can see the in flight rax uploads | 15:04 |
| corvus | openstack.exceptions.HttpException: HttpException: 403: Client Error for url: https://ord.images.api.rackspacecloud.com/v2/images/3df2035c-e283-4cd7-a1d3-bd0e4f90ee5c, Forbidden | 15:05 |
| *** rossella_s has quit IRC | 15:06 | |
| clarkb | cloudnull: has offered to help with rax related things if we need to loop someone from rax in | 15:07 |
| *** ykarel|away has quit IRC | 15:07 | |
| corvus | clarkb: so in addition to the zk stuff, there are nodepool and openstacksdk changes? | 15:07 |
| clarkb | corvus: ya the openstacksdk changes caught me unawares debugging thatn odepool test issue yesterday. But it is possible this is a new sdk/shade bug? | 15:08 |
| corvus | openstacksdk==0.18.1 | 15:08 |
| clarkb | corvus: fwiw rax uses the glance task api and I am not sure that v2/images/foo stuff is how you interact with rax | 15:08 |
| clarkb | (I can help dig in at about 1600UTC I expect) | 15:09 |
| corvus | personally, i am in a position to help merge a pin to 0.17.2 | 15:11 |
| corvus | i'll ask in the openstack-sdks channel if anyone is able to help debug this further; otherwise, maybe we merge the pin for now | 15:12 |
| clarkb | corvus: that may be a good short term solution if we want to try switch say nb01 over and if it fixes nodepool merge the pin? | 15:12 |
| *** felipemonteiro has joined #openstack-infra | 15:12 | |
| clarkb | wfm | 15:13 |
| *** auristor has quit IRC | 15:15 | |
| *** gyee has joined #openstack-infra | 15:15 | |
| corvus | clarkb: yeah, i'll switch nb01 manually now | 15:15 |
| mordred[m] | clarkb I can help diagnose underlying issue in a couple hours | 15:15 |
| corvus | mordred[m]: would you like me to just leave things as they are? | 15:16 |
| mordred[m] | Pinning to 0.17.2 is great in the meantime | 15:16 |
| corvus | i don't think we're in a rush if you're able to help today. i think we just want to get it resolved by eod. | 15:16 |
| mordred[m] | Cool, yeah, I've got plenty of time to help in a little bit | 15:17 |
| mordred[m] | And seeing the break might make it easier | 15:17 |
| corvus | mordred[m], clarkb: okay, i will leave nb01 and nb02 untouched for mordred[m] to live-debug later. | 15:17 |
| clarkb | sounds like a plan, thanks! | 15:17 |
| *** lpetrut has quit IRC | 15:18 | |
| cloudnull | clarkb ++ is something going on ? | 15:19 |
| clarkb | cloudnull: I think we suspect openstacksdk at this point and probably nto a rax problem | 15:19 |
| clarkb | cloudnull: mordred[m] will dig in in a bit and if he finds evidence that rax is doing something I expect he can loop you in | 15:20 |
| *** Emine has quit IRC | 15:22 | |
| *** agopi is now known as agopi|food | 15:22 | |
| *** auristor has joined #openstack-infra | 15:23 | |
| *** kjackal has joined #openstack-infra | 15:25 | |
| *** ykarel|away has joined #openstack-infra | 15:28 | |
| *** ykarel|away is now known as ykarel | 15:28 | |
| cfriesen | if I've got commits in the "check" queue that I know are going to fail (new flake8 version) is there any way to just cancel them? | 15:28 |
| clarkb | cfriesen: you can abandon them or push a new patchset | 15:29 |
| *** felipemonteiro has quit IRC | 15:29 | |
| *** kjackal_v2 has quit IRC | 15:29 | |
| fungi | though the new patchset will also start new jobs in check, as will restoring them out of abandonment | 15:30 |
| *** rossella_s has joined #openstack-infra | 15:32 | |
| *** felipemonteiro has joined #openstack-infra | 15:36 | |
| *** agopi|food is now known as agopi | 15:42 | |
| *** smarcet has quit IRC | 15:44 | |
| *** rossella_s has quit IRC | 15:47 | |
| *** rossella_s has joined #openstack-infra | 15:48 | |
| *** pcaruana has quit IRC | 15:49 | |
| openstackgerrit | Andrey Volkov proposed openstack-infra/project-config master: New Airship project - Utils https://review.openstack.org/612820 | 15:50 |
| *** kjackal_v2 has joined #openstack-infra | 15:53 | |
| *** rossella_s has quit IRC | 15:53 | |
| *** kjackal has quit IRC | 15:53 | |
| *** ccamacho has quit IRC | 15:54 | |
| *** munimeha1 has joined #openstack-infra | 15:58 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Move openstack-browbeat zuul jobs to project repository https://review.openstack.org/613092 | 15:59 |
| *** jamesmcarthur has quit IRC | 16:00 | |
| *** jamesmcarthur has joined #openstack-infra | 16:01 | |
| *** rossella_s has joined #openstack-infra | 16:02 | |
| *** jcoufal_ has joined #openstack-infra | 16:02 | |
| *** hasharAway is now known as hashar | 16:02 | |
| *** felipemonteiro has quit IRC | 16:02 | |
| *** dave-mccowan has quit IRC | 16:02 | |
| *** jcoufal has quit IRC | 16:03 | |
| *** ykarel is now known as ykarel|away | 16:07 | |
| *** smarcet has joined #openstack-infra | 16:12 | |
| *** ramishra has quit IRC | 16:14 | |
| *** psachin has quit IRC | 16:16 | |
| *** rossella_s has quit IRC | 16:17 | |
| *** bhavikdbavishi has joined #openstack-infra | 16:19 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/nodepool master: Run test zookeeper on top of tmpfs https://review.openstack.org/612816 | 16:20 |
| *** pcrews has joined #openstack-infra | 16:21 | |
| clarkb | mordred[m]: http://paste.openstack.org/show/732974/ is the full upload traceback. It appears to be uploading successfully then failing to update the image properties after | 16:32 |
| *** ginopc has quit IRC | 16:32 | |
| clarkb | mordred[m]: one concern is that I'm not sure we'll properly clean things up in this error case | 16:32 |
| clarkb | we don't seem to try except around that property update | 16:33 |
| mordred[m] | <freenode_cla "mordred: http://paste.openstack."> clarkb awesome, super helpful | 16:33 |
| mordred[m] | clarkb I'm 99.9% certain it's going to be a bug in https://review.openstack.org/#/c/609683/ | 16:35 |
| *** lpetrut has joined #openstack-infra | 16:35 | |
| *** smarcet has quit IRC | 16:35 | |
| mordred[m] | That was a reorg intended to be functionally equiv to set the stage for using the shade code for the SDK image methods ... I'm guessing a logic error snuck in | 16:36 |
| mordred[m] | I'm about to take off on a 1.5 hour flight, will look at the code in the air but I don't think I'll have wifi. Will be online once I hit my hotel on the other side | 16:37 |
| clarkb | ya thats the code I've found diffing 0.18.1 and 0.17.2 | 16:37 |
| clarkb | ok | 16:37 |
| clarkb | I'll take a look until I've got to go visit the dentist | 16:38 |
| clarkb | will do my best to apss along anything I've found here so you can pick it up when you land | 16:38 |
| *** jpich has quit IRC | 16:38 | |
| mordred[m] | But good ... I was worried it was discovery related | 16:38 |
| mordred[m] | Sweet | 16:38 |
| mordred[m] | This is likely an else in the wrong place or something | 16:39 |
| clarkb | mordred[m]: really quickly at the end of that diff I think we change parameters to properties | 16:39 |
| *** ansmith has quit IRC | 16:39 | |
| *** imacdonn has quit IRC | 16:40 | |
| clarkb | though that is just an sdk side var looks like so maybe unrelated | 16:40 |
| mordred[m] | The variable naming is ... Confusing at best :) | 16:40 |
| mordred[m] | https://review.openstack.org/#/c/609684 is a patch that sketches out where this is going ... In case it's helpful intent-wise | 16:41 |
| clarkb | oh but ya we don't pop out parameters | 16:41 |
| clarkb | so maybe if we've kept parameters in there the patch could fail later? | 16:41 |
| *** imacdonn has joined #openstack-infra | 16:43 | |
| *** quiquell is now known as quiquell|off | 16:43 | |
| openstackgerrit | Sorin Sbarnea proposed openstack-infra/project-config master: Move openstack-browbeat zuul jobs to project repository https://review.openstack.org/613092 | 16:45 |
| mordred[m] | Maybe? But why are we getting 403 ... I feel like maybe we're using the wrong update mechanism? | 16:45 |
| *** dtantsur is now known as dtantsur|afk | 16:46 | |
| *** trown is now known as trown|lunch | 16:46 | |
| *** pcaruana has joined #openstack-infra | 16:46 | |
| clarkb | I was thinking 403 forbidden to update the field you want to update type of thing | 16:48 |
| clarkb | but ya maybe it is the wrong mechanism? | 16:48 |
| *** diablo_rojo has joined #openstack-infra | 16:51 | |
| *** icey has quit IRC | 16:56 | |
| *** panda is now known as panda|off | 16:59 | |
| *** e0ne has quit IRC | 17:00 | |
| *** jamesmcarthur has quit IRC | 17:01 | |
| *** icey has joined #openstack-infra | 17:04 | |
| *** derekh has quit IRC | 17:07 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul-jobs master: WIP: Add sar logging roles https://review.openstack.org/613112 | 17:07 |
| *** aojea has quit IRC | 17:08 | |
| *** irclogbot_2 has quit IRC | 17:08 | |
| *** hashar is now known as hasharDinner | 17:08 | |
| clarkb | mordred[m]: https://developer.openstack.org/api-ref/image/v2/index.html?expanded=update-image-detail has a section about when 403s will happen if we try to update the wrong details. I expect that is what is happening here | 17:09 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: DNM: Enable sar logging for unit tests https://review.openstack.org/613117 | 17:13 |
| *** smarcet has joined #openstack-infra | 17:14 | |
| *** jpena is now known as jpena|off | 17:15 | |
| *** kopecmartin is now known as kopecmartin|off | 17:16 | |
| *** Swami has joined #openstack-infra | 17:18 | |
| clarkb | mordred[m]: one theory is that it is related to disk_format and conatiner_format being set in 0.18. In 0.17 we never passed that along to the image upload task | 17:19 |
| clarkb | mordred[m]: if rax has set those properties to be read only we would potentially try to update them if they are nil type values | 17:19 |
| clarkb | I need to go eat breakfast now but ^ is my hunch. I expect we want to continue to not set those properties with rackspace | 17:20 |
| clarkb | and let the import task stuff figure it out on its own? | 17:20 |
| *** jamesmcarthur has joined #openstack-infra | 17:26 | |
| *** ansmith has joined #openstack-infra | 17:27 | |
| *** jamesmcarthur_ has joined #openstack-infra | 17:27 | |
| *** jamesmcarthur has quit IRC | 17:31 | |
| *** bhavikdbavishi has quit IRC | 17:41 | |
| *** bhavikdbavishi1 has joined #openstack-infra | 17:41 | |
| *** bhavikdbavishi1 has quit IRC | 17:41 | |
| corvus | o/ | 17:46 |
| *** kjackal_v2 has quit IRC | 17:50 | |
| *** kjackal has joined #openstack-infra | 17:51 | |
| *** gfidente has quit IRC | 17:51 | |
| *** lpetrut has quit IRC | 17:55 | |
| corvus | clarkb: yeah, i'd put my money on properties | 17:58 |
| AJaeger | config-core, astara repos are retired - could you review https://review.openstack.org/#/c/610361/2 , please? | 17:58 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Collect docker logs after quick-start run https://review.openstack.org/613027 | 18:03 |
| *** electrofelix has quit IRC | 18:05 | |
| *** gfidente has joined #openstack-infra | 18:06 | |
| *** tpsilva has joined #openstack-infra | 18:07 | |
| *** jamesmcarthur_ has quit IRC | 18:09 | |
| *** jamesmcarthur has joined #openstack-infra | 18:12 | |
| *** trown|lunch is now known as trown | 18:13 | |
| kmalloc | what is the typical rule on accepting patches posted to LP instead of through gerrit? i am unsure what risk we take at this time. | 18:13 |
| kmalloc | on that front | 18:13 |
| kmalloc | have a case in keystonemiddleware. | 18:13 |
| * kmalloc will comment and request submitting through gerrit from the OP. | 18:13 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Collect docker logs after quick-start run https://review.openstack.org/613027 | 18:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add the process environment to zuul.conf parser https://review.openstack.org/612824 | 18:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Pin flake8 https://review.openstack.org/613134 | 18:19 |
| fungi | kmalloc: officially we want to be sure the person who posted that patch has agreed to the openstack icla. unofficially, we tend not to really pay attention unless it's clear the patch is from someone who isn't a regular contributor, and even then we've taken the board's confirmation that we can switch to using the dco as an indication that having the patch author include a signed-off-by is sufficient | 18:25 |
| *** kjackal has quit IRC | 18:27 | |
| kmalloc | fungi: i think this is a normal contributor | 18:27 |
| kmalloc | anyway it is a good fix... | 18:27 |
| kmalloc | i'll get a signed-off-by if it's an issue | 18:27 |
| fungi | we frequently have one person attach patches to embargoed vulnerability reports and then another person push them to gerrit, which is why i say we tend not to pay attention to that rule in a lot of cases | 18:29 |
| kmalloc | right. this is a simple WSGI compat thing and i think the person already contributes to another project | 18:31 |
| kmalloc | sooooooo | 18:31 |
| AJaeger | fungi, clarkb, do you have time for a quick cleanup review, please? https://review.openstack.org/#/c/610361 - finish retiring astara | 18:36 |
| *** ssbarnea|bkp2 has joined #openstack-infra | 18:39 | |
| *** slaweq_ has joined #openstack-infra | 18:40 | |
| *** niska has joined #openstack-infra | 18:41 | |
| *** dkehn_ has joined #openstack-infra | 18:42 | |
| *** jistr_ has joined #openstack-infra | 18:42 | |
| *** tobiash_ has joined #openstack-infra | 18:43 | |
| *** roman_g_ has joined #openstack-infra | 18:43 | |
| *** kazsh_ has joined #openstack-infra | 18:44 | |
| *** jhesketh_ has joined #openstack-infra | 18:44 | |
| *** smarcet has quit IRC | 18:45 | |
| *** icey has quit IRC | 18:48 | |
| *** diablo_rojo has quit IRC | 18:48 | |
| *** markmcd has quit IRC | 18:48 | |
| *** haleyb has quit IRC | 18:48 | |
| *** jesusaur has quit IRC | 18:48 | |
| *** priteau has quit IRC | 18:48 | |
| *** roman_g has quit IRC | 18:48 | |
| *** raginbajin has quit IRC | 18:48 | |
| *** slaweq has quit IRC | 18:48 | |
| *** jbadiapa has quit IRC | 18:48 | |
| *** gary_perkins has quit IRC | 18:48 | |
| *** dhill_ has quit IRC | 18:48 | |
| *** tobiash has quit IRC | 18:48 | |
| *** ssbarnea has quit IRC | 18:48 | |
| *** kazsh has quit IRC | 18:48 | |
| *** zzzeek has quit IRC | 18:48 | |
| *** zul has quit IRC | 18:48 | |
| *** scas has quit IRC | 18:48 | |
| *** dkehn has quit IRC | 18:48 | |
| *** rook has quit IRC | 18:48 | |
| *** jistr has quit IRC | 18:48 | |
| *** SpamapS has quit IRC | 18:48 | |
| *** spotz has quit IRC | 18:48 | |
| *** radez has quit IRC | 18:48 | |
| *** SotK has quit IRC | 18:48 | |
| *** niska` has quit IRC | 18:48 | |
| *** dabukalam has quit IRC | 18:48 | |
| *** Diabelko has quit IRC | 18:48 | |
| *** jhesketh has quit IRC | 18:48 | |
| *** dkehn_ is now known as dkehn | 18:48 | |
| *** Diabelko has joined #openstack-infra | 18:49 | |
| *** jbadiapa has joined #openstack-infra | 18:49 | |
| *** jesusaur has joined #openstack-infra | 18:49 | |
| *** icey has joined #openstack-infra | 18:49 | |
| *** SpamapS has joined #openstack-infra | 18:49 | |
| *** dhill_ has joined #openstack-infra | 18:50 | |
| *** diablo_rojo has joined #openstack-infra | 18:50 | |
| *** SotK has joined #openstack-infra | 18:50 | |
| *** e0ne has joined #openstack-infra | 18:51 | |
| *** spotz has joined #openstack-infra | 18:52 | |
| *** rook has joined #openstack-infra | 18:54 | |
| *** rook has quit IRC | 18:54 | |
| *** rook has joined #openstack-infra | 18:54 | |
| openstackgerrit | Merged openstack-infra/project-config master: Create blazar-specs repo https://review.openstack.org/602783 | 18:54 |
| *** rook is now known as Guest79095 | 18:55 | |
| mordred | clarkb: I think I'm betting on disk_format/container_format | 18:55 |
| *** kjackal has joined #openstack-infra | 18:58 | |
| mordred | clarkb: remote: https://review.openstack.org/613141 Don't pass disk_format or container_format to image task upload | 19:00 |
| mordred | clarkb: I applied that by hand on nb02 | 19:01 |
| clarkb | mordred: cool Im about to get teeth cleaned so cant help right now | 19:01 |
| anteaya | clarkb: happy teeth cleaning | 19:01 |
| mordred | clarkb: enjoy the teeth! | 19:02 |
| clarkb | lets hope :) | 19:02 |
| anteaya | may they send you away with a new toothbrush | 19:02 |
| anteaya | I always look forward to my new toothbrush | 19:02 |
| mordred | clarkb: I'm probably going to grab some food or something but am tailing the logs in a terminal | 19:02 |
| *** openstackgerrit has quit IRC | 19:06 | |
| *** openstackgerrit has joined #openstack-infra | 19:10 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: WIP: support foreign required-projects https://review.openstack.org/613143 | 19:10 |
| *** gfidente has quit IRC | 19:11 | |
| *** e0ne has quit IRC | 19:13 | |
| *** Guest79095 is now known as rook | 19:13 | |
| *** harlowja has quit IRC | 19:14 | |
| *** harlowja has joined #openstack-infra | 19:14 | |
| *** e0ne has joined #openstack-infra | 19:18 | |
| *** felipemonteiro has joined #openstack-infra | 19:22 | |
| *** ykarel|away has quit IRC | 19:23 | |
| *** irclogbot_2 has joined #openstack-infra | 19:26 | |
| *** irclogbot_2 has quit IRC | 19:27 | |
| *** irclogbot_2 has joined #openstack-infra | 19:27 | |
| *** irclogbot_2 has quit IRC | 19:27 | |
| *** irclogbot_2 has joined #openstack-infra | 19:34 | |
| *** e0ne has quit IRC | 19:39 | |
| *** slaweq_ is now known as slaweq | 19:40 | |
| *** jamesmcarthur has quit IRC | 19:48 | |
| *** jamesmcarthur has joined #openstack-infra | 19:51 | |
| *** scas has joined #openstack-infra | 19:55 | |
| *** felipemonteiro has quit IRC | 19:57 | |
| *** apetrich has quit IRC | 20:00 | |
| *** felipemonteiro has joined #openstack-infra | 20:05 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Run test zookeeper on top of tmpfs https://review.openstack.org/612816 | 20:06 |
| *** smarcet has joined #openstack-infra | 20:06 | |
| *** felipemonteiro has quit IRC | 20:10 | |
| AJaeger | config-core, please review https://review.openstack.org/#/c/610361 and https://review.openstack.org/609531 | 20:11 |
| *** apetrich has joined #openstack-infra | 20:14 | |
| ianw | fungi: yes, we need to look at correlating the logs in nodepool with regions better, it's very inscrutable on a busy system like nl04 to figure out what error goes with what | 20:19 |
| *** irclogbot_2 has quit IRC | 20:21 | |
| clarkb | mordred: we have ready images in rax-* now | 20:31 |
| clarkb | mordred: so that must've been it? | 20:31 |
| clarkb | mordred: can we apply it to nb01 too? | 20:31 |
| clarkb | (I won't do that myself unless given a go ahead since I'm unsure if other debugging is going on) | 20:32 |
| *** ansmith has quit IRC | 20:34 | |
| cfriesen | is there something odd going on in the check queue, or is 5+ hrs delay normal for this time of day? | 20:36 |
| clarkb | cfriesen: currently looks to be a just under 3 hour delay? but the delays are a direct function of demand vs available resources | 20:37 |
| clarkb | http://grafana.openstack.org/d/rZtIH5Imz/nodepool?orgId=1 shows resources and http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 shows demand (with some simplified resource graphs too) | 20:38 |
| clarkb | cfriesen: looking at that I don't think there is much abnormality going on. Just a large amount of demand | 20:38 |
| cfriesen | zuul.openstack.org/status shows 5hr36min for the top commit, unless I'm reading it wrong. In any case, guess I'll check things later tonight. :) | 20:39 |
| cfriesen | thanks for the links | 20:40 |
| clarkb | cfriesen: thats how long that particular change has been there and is in part a function of how long tripleo test jobs take | 20:40 |
| clarkb | 3.5 hours iirc | 20:40 |
| cfriesen | yikes | 20:40 |
| clarkb | (I prefer to look at the most recently started jobs in the queuei nstead to understand how far behind we are to avoid self inflicted time costs like that) | 20:40 |
| cfriesen | got it | 20:41 |
| *** jamesmcarthur has quit IRC | 20:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Filter file comments for existing files https://review.openstack.org/613161 | 20:49 |
| clarkb | tripleo gate just reset :( | 20:49 |
| clarkb | we may be back to thrashing a bit in the gate which makes the queues worse | 20:50 |
| *** pcaruana has quit IRC | 20:50 | |
| corvus | that's a scheduler change ^ but if we land that and restart the scheduler, we should be able to safely try the pep8 thing again | 20:52 |
| *** e0ne has joined #openstack-infra | 20:52 | |
| *** e0ne has quit IRC | 20:52 | |
| clarkb | ok I'm going to eat lunch and if I don't hear from mordred I'm going to go ahead and apply the openstacksdk fix to nb01 and restart the builder there | 20:53 |
| *** kjackal has quit IRC | 20:55 | |
| mordred[m] | clarkb yes, I concur ... Apply to nb01 | 20:58 |
| *** jcoufal_ has quit IRC | 21:03 | |
| *** openstack has quit IRC | 21:03 | |
| *** openstack has joined #openstack-infra | 21:05 | |
| *** ChanServ sets mode: +o openstack | 21:05 | |
| *** xek has quit IRC | 21:07 | |
| *** xek_ has quit IRC | 21:08 | |
| clarkb | mordred[m]: how did you apply it to nb02? pbr freeze shows the old sha1. Was it not installed but patched straight in? I'm going to install it fwiw | 21:15 |
| *** ldnunes has quit IRC | 21:15 | |
| clarkb | and of course I used pip not pip3 | 21:15 |
| *** jamesmcarthur has joined #openstack-infra | 21:17 | |
| clarkb | ok after using pip3 I believe I have properly applied that openstacksdk patch to nb01 now as well | 21:17 |
| mordred[m] | clarkb I just patched it straight in | 21:20 |
| clarkb | rgr | 21:20 |
| clarkb | I'm watching nb01 now, but I think we are set | 21:20 |
| *** spsurya has quit IRC | 21:21 | |
| *** smarcet has quit IRC | 21:21 | |
| *** ansmith has joined #openstack-infra | 21:30 | |
| mordred[m] | Woot! | 21:31 |
| *** kgiusti has left #openstack-infra | 21:34 | |
| *** boden has quit IRC | 21:46 | |
| *** hasharDinner has quit IRC | 21:48 | |
| *** tosky has quit IRC | 21:53 | |
| *** smarcet has joined #openstack-infra | 21:53 | |
| *** bnemec has quit IRC | 21:55 | |
| clarkb | infra-root in prep for the zk switch tomorrow, care to review https://review.openstack.org/#/c/612442/ and https://review.openstack.org/#/c/612443/ these are the changes that actually update our configs. | 21:57 |
| clarkb | Probably don't want to approve them yet. We'll merge them in a controlled manner early tomorrow | 21:57 |
| *** jamesmcarthur has quit IRC | 22:05 | |
| *** slaweq has quit IRC | 22:06 | |
| *** fuentess has quit IRC | 22:09 | |
| *** rh-jelabarre has quit IRC | 22:10 | |
| clarkb | waiting for image uploads is not the most entertaining thing, but don't want to get into something else if this ends up not working | 22:13 |
| clarkb | one neat thing is that when I restarted nb01 it connected to a different zk server | 22:13 |
| clarkb | so it isn't always the first server listed that is used | 22:13 |
| *** efried has quit IRC | 22:15 | |
| *** efried has joined #openstack-infra | 22:15 | |
| *** felipemonteiro has joined #openstack-infra | 22:22 | |
| *** felipemonteiro has quit IRC | 22:25 | |
| *** jamesmcarthur has joined #openstack-infra | 22:27 | |
| ianw | clarkb / amorin: although the error rate for launching servers is up, the cleanup runs don't seem to be finding any ports to clear, so that looks good on the leaking front | 22:27 |
| clarkb | ianw: that is good news. | 22:29 |
| clarkb | On the rax image upload front we are successfully uploading images on nb01 now (it just takes an hour to get to that point) | 22:30 |
| clarkb | I think we will be ready to siwtch to the zk cluster tomorrow. I'm glad we gave ourselves more than a day too :) | 22:30 |
| *** jamesmcarthur has quit IRC | 22:31 | |
| *** rlandy is now known as rlandy|bbl | 22:31 | |
| *** vabada has quit IRC | 22:35 | |
| *** vabada has joined #openstack-infra | 22:37 | |
| *** eharney has quit IRC | 22:38 | |
| clarkb | #status log nb01 and nb02 patched to have https://review.openstack.org/#/c/613141/ installed so that image uploads to rax work. Both nodes are in ansible emergency file so this won't be undone automatically. Will need openstacksdk release before removing them from the emergency file | 22:46 |
| openstackstatus | clarkb: finished logging | 22:46 |
| *** jamesmcarthur has joined #openstack-infra | 23:00 | |
| *** rcernin has joined #openstack-infra | 23:05 | |
| openstackgerrit | Merged openstack-infra/project-config master: Retire astra projects https://review.openstack.org/610361 | 23:05 |
| openstackgerrit | Merged openstack-infra/project-config master: Import puppet-crane for TripleO https://review.openstack.org/609531 | 23:05 |
| openstackgerrit | Merged openstack-infra/zuul master: Pin flake8 https://review.openstack.org/613134 | 23:15 |
| clarkb | ok I think all images are uploaded to call regions at this point | 23:17 |
| clarkb | s/call/all/ | 23:18 |
| *** jamesmcarthur has quit IRC | 23:18 | |
| *** openstackgerrit has quit IRC | 23:20 | |
| clarkb | For tomorrow morning I think the rough plan should be put nl*.openstack.org, ze*.openstack.org, zm*.openstack.org, and zuul*.openstack.org in the emergency file. Merge https://review.openstack.org/#/c/612442/ and https://review.openstack.org/#/c/612443/, STop services on those hosts, kick.sh those hosts, start services as necessary | 23:22 |
| clarkb | This is basically what we did with nb* beacuse shrews wasn't sure how switching the db out from under a running process would go | 23:23 |
| clarkb | Then when that is done I'll work on cleanup | 23:23 |
| clarkb | and we can monitor cacti for the zk01/2/3 servers | 23:23 |
| clarkb | And I'll try to be awake and around bright and early to do that :) | 23:24 |
| *** agopi has quit IRC | 23:32 | |
| *** jesusaur has quit IRC | 23:34 | |
| *** smarcet has quit IRC | 23:35 | |
| *** ansmith has quit IRC | 23:38 | |
| *** sthussey has quit IRC | 23:40 | |
| *** Swami has quit IRC | 23:45 | |
| *** smarcet has joined #openstack-infra | 23:50 | |
| *** plestang has quit IRC | 23:50 | |
| *** smarcet has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!