| *** pfalleno1 has joined #openstack-infra | 00:04 | |
| *** pfallenop has quit IRC | 00:04 | |
| *** amitgandhinz has quit IRC | 00:09 | |
| *** pfalleno1 has quit IRC | 00:09 | |
| *** Sukhdev has joined #openstack-infra | 00:13 | |
| *** pfallenop has joined #openstack-infra | 00:14 | |
| *** thorst has joined #openstack-infra | 00:17 | |
| *** kaisers1 has joined #openstack-infra | 00:19 | |
| *** kaisers4 has joined #openstack-infra | 00:19 | |
| *** kaisers3 has quit IRC | 00:20 | |
| *** kaisers2 has quit IRC | 00:20 | |
| *** inc0 has joined #openstack-infra | 00:20 | |
| *** sdake has quit IRC | 00:22 | |
| *** spzala has joined #openstack-infra | 00:24 | |
| *** thorst has quit IRC | 00:25 | |
| *** inc0 has quit IRC | 00:26 | |
| *** Sukhdev has quit IRC | 00:26 | |
| *** sflanigan_afk has joined #openstack-infra | 00:27 | |
| *** Sukhdev has joined #openstack-infra | 00:28 | |
| *** roxanaghe has joined #openstack-infra | 00:28 | |
| *** sdake has joined #openstack-infra | 00:32 | |
| *** dimtruck is now known as zz_dimtruck | 00:33 | |
| *** roxanaghe has quit IRC | 00:33 | |
| *** masayukig_ is now known as masayukig | 00:33 | |
| *** ijw has joined #openstack-infra | 00:35 | |
| *** moravec1 has quit IRC | 00:43 | |
| *** moravec has joined #openstack-infra | 00:46 | |
| *** ijw has quit IRC | 00:46 | |
| *** claudiub has quit IRC | 00:47 | |
| tristanC | #success 55 PTL elected for Ocata cycle (plus 2 appointed by the TC) | 00:48 |
|---|---|---|
| *** moravec has quit IRC | 00:57 | |
| *** moravec has joined #openstack-infra | 00:57 | |
| *** tqtran has joined #openstack-infra | 00:58 | |
| *** ijw has joined #openstack-infra | 01:01 | |
| *** tqtran has quit IRC | 01:02 | |
| *** yamamoto_ has joined #openstack-infra | 01:03 | |
| *** amitgandhinz has joined #openstack-infra | 01:05 | |
| openstackgerrit | zhangyanxian proposed openstack-infra/project-config: Enable py35 voting on project watcher https://review.openstack.org/376062 | 01:08 |
| openstackgerrit | zhangyanxian proposed openstack-infra/project-config: Enable py35 voting on project watcher https://review.openstack.org/376062 | 01:10 |
| *** gongysh has joined #openstack-infra | 01:11 | |
| *** kaisers_ has joined #openstack-infra | 01:11 | |
| *** ianychoi has joined #openstack-infra | 01:13 | |
| *** zhurong has joined #openstack-infra | 01:14 | |
| *** kaisers has quit IRC | 01:15 | |
| *** jascott1_ has joined #openstack-infra | 01:18 | |
| *** zhurong has quit IRC | 01:19 | |
| *** thorst has joined #openstack-infra | 01:19 | |
| *** zhurong has joined #openstack-infra | 01:20 | |
| *** ijw has quit IRC | 01:21 | |
| *** baoli_ has quit IRC | 01:22 | |
| mriedem | logstash is jacked | 01:22 |
| *** thorst has quit IRC | 01:22 | |
| mriedem | cluster health is green but indexing is behind by 11 hours | 01:22 |
| *** baoli has joined #openstack-infra | 01:23 | |
| *** thorst has joined #openstack-infra | 01:24 | |
| *** hongbin has joined #openstack-infra | 01:25 | |
| *** yamamoto_ has quit IRC | 01:26 | |
| *** yanyanhu has joined #openstack-infra | 01:27 | |
| *** baoli has quit IRC | 01:28 | |
| *** roxanaghe has joined #openstack-infra | 01:30 | |
| *** amotoki has joined #openstack-infra | 01:30 | |
| *** thorst has quit IRC | 01:31 | |
| *** Sukhdev has quit IRC | 01:33 | |
| *** roxanaghe has quit IRC | 01:34 | |
| *** ijw has joined #openstack-infra | 01:34 | |
| *** yamamoto_ has joined #openstack-infra | 01:38 | |
| *** amitgandhinz has quit IRC | 01:39 | |
| *** ijw has quit IRC | 01:40 | |
| *** EricGonc_ has joined #openstack-infra | 01:40 | |
| *** EricGonczer_ has quit IRC | 01:42 | |
| *** shu-mutou has joined #openstack-infra | 01:43 | |
| *** thorst has joined #openstack-infra | 01:47 | |
| *** EricGonc_ has quit IRC | 01:47 | |
| *** thorst has quit IRC | 01:48 | |
| *** ijw has joined #openstack-infra | 01:48 | |
| *** EricGonczer_ has joined #openstack-infra | 01:49 | |
| *** mtanino has joined #openstack-infra | 01:59 | |
| *** ijw has quit IRC | 02:05 | |
| *** AlexxNica has joined #openstack-infra | 02:24 | |
| *** baoli has joined #openstack-infra | 02:29 | |
| *** njohnston has quit IRC | 02:29 | |
| *** mminesh has quit IRC | 02:30 | |
| *** njohnston has joined #openstack-infra | 02:30 | |
| *** stevemar has quit IRC | 02:30 | |
| *** jd__ has quit IRC | 02:31 | |
| *** loquacities has quit IRC | 02:31 | |
| *** loquacities has joined #openstack-infra | 02:31 | |
| *** stevemar has joined #openstack-infra | 02:31 | |
| *** spzala has quit IRC | 02:32 | |
| *** jd__ has joined #openstack-infra | 02:32 | |
| *** mminesh has joined #openstack-infra | 02:32 | |
| *** baoli has quit IRC | 02:33 | |
| *** sdake has quit IRC | 02:35 | |
| *** amitgandhinz has joined #openstack-infra | 02:35 | |
| *** tuanla has joined #openstack-infra | 02:36 | |
| *** amotoki has quit IRC | 02:41 | |
| *** thorst has joined #openstack-infra | 02:48 | |
| *** yuanying has quit IRC | 02:52 | |
| *** EricGonczer_ has quit IRC | 02:56 | |
| *** takashi has joined #openstack-infra | 02:57 | |
| *** thorst has quit IRC | 02:57 | |
| *** mriedem has quit IRC | 02:57 | |
| *** yamamoto_ has quit IRC | 03:03 | |
| *** david-lyle has quit IRC | 03:04 | |
| *** ijw has joined #openstack-infra | 03:06 | |
| *** amitgandhinz has quit IRC | 03:09 | |
| *** ijw has quit IRC | 03:11 | |
| *** roxanaghe has joined #openstack-infra | 03:15 | |
| openstackgerrit | linzi proposed openstack-infra/project-config: Networking-peregrine is a Neutron ml2 mechanism driver. https://review.openstack.org/376096 | 03:17 |
| *** roxanaghe has quit IRC | 03:20 | |
| *** rwsu has joined #openstack-infra | 03:24 | |
| *** amotoki has joined #openstack-infra | 03:30 | |
| *** Sukhdev has joined #openstack-infra | 03:38 | |
| *** vikrant has joined #openstack-infra | 03:41 | |
| *** mtanino has quit IRC | 03:41 | |
| *** yamamoto_ has joined #openstack-infra | 03:44 | |
| *** yuanying has joined #openstack-infra | 03:47 | |
| *** amotoki has quit IRC | 03:47 | |
| *** markvoelker has joined #openstack-infra | 03:48 | |
| *** yuanying has quit IRC | 03:49 | |
| *** markvoelker has quit IRC | 03:53 | |
| openstackgerrit | linzi proposed openstack-infra/project-config: Networking-peregrine is a Neutron ml2 mechanism driver. https://review.openstack.org/376096 | 03:54 |
| *** thorst has joined #openstack-infra | 03:55 | |
| *** yuanying has joined #openstack-infra | 03:55 | |
| *** gongysh has quit IRC | 03:57 | |
| *** thorst has quit IRC | 04:02 | |
| *** claudiub has joined #openstack-infra | 04:02 | |
| *** ianychoi has quit IRC | 04:03 | |
| *** amitgandhinz has joined #openstack-infra | 04:06 | |
| *** ijw has joined #openstack-infra | 04:07 | |
| *** hichihara has joined #openstack-infra | 04:08 | |
| *** gouthamr has quit IRC | 04:08 | |
| *** ijw has quit IRC | 04:13 | |
| *** AlexxNic_ has joined #openstack-infra | 04:15 | |
| *** AlexxNica has quit IRC | 04:16 | |
| *** baoli has joined #openstack-infra | 04:17 | |
| *** tonytan_brb has joined #openstack-infra | 04:17 | |
| *** hongbin has quit IRC | 04:18 | |
| *** tonytan4ever has quit IRC | 04:19 | |
| *** baoli has quit IRC | 04:21 | |
| *** roxanaghe has joined #openstack-infra | 04:31 | |
| *** spzala has joined #openstack-infra | 04:32 | |
| openstackgerrit | linzi proposed openstack-infra/project-config: ITRI networking-peregrine is a Neutron ml2 mechanism driver. https://review.openstack.org/376155 | 04:35 |
| *** roxanaghe has quit IRC | 04:35 | |
| *** kota_ has joined #openstack-infra | 04:35 | |
| *** spzala has quit IRC | 04:37 | |
| kota_ | someone? | 04:37 |
| *** amitgandhinz has quit IRC | 04:41 | |
| *** sdake has joined #openstack-infra | 04:41 | |
| *** sdake has quit IRC | 04:45 | |
| *** markvoelker has joined #openstack-infra | 04:49 | |
| *** markvoelker has quit IRC | 04:53 | |
| *** coolsvap has joined #openstack-infra | 04:57 | |
| *** StevenK_ is now known as StevenK | 04:58 | |
| *** thorst has joined #openstack-infra | 04:59 | |
| *** tqtran has joined #openstack-infra | 04:59 | |
| *** tqtran has quit IRC | 05:04 | |
| *** thorst has quit IRC | 05:07 | |
| *** gildub has joined #openstack-infra | 05:08 | |
| tobiash | Hi, just noticed that the timer trigger has a slightly different semantics than cron | 05:08 |
| tobiash | regarding day of week cron starts with 0 == sunday where zuul starts with 0 == monday | 05:09 |
| *** ijw has joined #openstack-infra | 05:09 | |
| openstackgerrit | linzi proposed openstack-infra/project-config: Add networking-peregrine project. https://review.openstack.org/376167 | 05:09 |
| tobiash | looks like the underlying apscheduler handles this different than cron | 05:10 |
| *** yee3793 has joined #openstack-infra | 05:11 | |
| *** florianf has joined #openstack-infra | 05:12 | |
| *** lezbar__ has joined #openstack-infra | 05:13 | |
| *** crinkle_ has joined #openstack-infra | 05:13 | |
| *** fmccrthy_ has joined #openstack-infra | 05:13 | |
| *** NobodyCam_ has joined #openstack-infra | 05:13 | |
| *** kozhukalov_ has joined #openstack-infra | 05:13 | |
| *** igorbelikov_ has joined #openstack-infra | 05:13 | |
| *** betherly_ has joined #openstack-infra | 05:13 | |
| *** zxiiro_ has joined #openstack-infra | 05:13 | |
| *** GheRivero_ has joined #openstack-infra | 05:13 | |
| *** tomhambleton_ has joined #openstack-infra | 05:13 | |
| *** auggy_ has joined #openstack-infra | 05:14 | |
| *** MarkAtwood_ has joined #openstack-infra | 05:14 | |
| *** adriant has quit IRC | 05:14 | |
| *** monester__ has joined #openstack-infra | 05:14 | |
| *** furlongm_ has joined #openstack-infra | 05:14 | |
| *** unrahul_ has joined #openstack-infra | 05:14 | |
| *** teran_ has joined #openstack-infra | 05:15 | |
| *** zaro_ has joined #openstack-infra | 05:15 | |
| *** DuncanT_ has joined #openstack-infra | 05:15 | |
| *** serverascode_ has joined #openstack-infra | 05:15 | |
| *** tobiash_ has joined #openstack-infra | 05:15 | |
| *** hwoarang_ has joined #openstack-infra | 05:15 | |
| *** ijw has quit IRC | 05:15 | |
| *** sslypushenko_ has joined #openstack-infra | 05:16 | |
| *** simonmcc_ has joined #openstack-infra | 05:16 | |
| *** firenka_ has joined #openstack-infra | 05:16 | |
| *** csmart_ has joined #openstack-infra | 05:16 | |
| *** Guest66676 has joined #openstack-infra | 05:16 | |
| *** darvon_ has joined #openstack-infra | 05:16 | |
| *** promethe1nfire has joined #openstack-infra | 05:16 | |
| *** fabo_ has joined #openstack-infra | 05:16 | |
| *** doffm_ has joined #openstack-infra | 05:16 | |
| *** aviau_ has joined #openstack-infra | 05:16 | |
| *** pleia2_ has joined #openstack-infra | 05:16 | |
| *** gema_ has joined #openstack-infra | 05:16 | |
| *** dosaboy_ has joined #openstack-infra | 05:16 | |
| *** BobBall_ has joined #openstack-infra | 05:16 | |
| *** matbu_ has joined #openstack-infra | 05:16 | |
| *** senk_ has joined #openstack-infra | 05:17 | |
| *** ggillies_ has joined #openstack-infra | 05:17 | |
| *** taron_ has joined #openstack-infra | 05:17 | |
| *** szaher_ has joined #openstack-infra | 05:17 | |
| *** stevebak` has joined #openstack-infra | 05:19 | |
| *** dgonzalez_ has joined #openstack-infra | 05:19 | |
| *** bexelbie- has joined #openstack-infra | 05:19 | |
| *** jlk` has joined #openstack-infra | 05:19 | |
| *** mjblack- has joined #openstack-infra | 05:19 | |
| *** filler_ has joined #openstack-infra | 05:19 | |
| *** chmouel_ has joined #openstack-infra | 05:19 | |
| *** kambiz_ has joined #openstack-infra | 05:19 | |
| *** tbarron_ has joined #openstack-infra | 05:19 | |
| *** jlvillal_ has joined #openstack-infra | 05:19 | |
| *** mrodden_ has joined #openstack-infra | 05:19 | |
| *** jasondotstar_ has joined #openstack-infra | 05:19 | |
| *** alaski_ has joined #openstack-infra | 05:19 | |
| *** weshay_ has joined #openstack-infra | 05:20 | |
| *** jklare_ has joined #openstack-infra | 05:20 | |
| *** beisner- has joined #openstack-infra | 05:20 | |
| *** sirushti_ has joined #openstack-infra | 05:20 | |
| *** mrmartin_ has joined #openstack-infra | 05:20 | |
| *** igormarnat__ has joined #openstack-infra | 05:20 | |
| *** amoralej_ has joined #openstack-infra | 05:20 | |
| *** rods has joined #openstack-infra | 05:20 | |
| *** clayton_ has joined #openstack-infra | 05:20 | |
| *** timburke_ has joined #openstack-infra | 05:20 | |
| *** dteselkin_ has joined #openstack-infra | 05:20 | |
| *** sileht has quit IRC | 05:20 | |
| *** pzhurba_ has quit IRC | 05:20 | |
| *** doffm has quit IRC | 05:20 | |
| *** chmouel has quit IRC | 05:20 | |
| *** hwoarang has quit IRC | 05:21 | |
| *** prometheanfire has quit IRC | 05:21 | |
| *** kun_huang has quit IRC | 05:21 | |
| *** simonmcc has quit IRC | 05:21 | |
| *** betherly has quit IRC | 05:21 | |
| *** monester_ has quit IRC | 05:21 | |
| *** serverascode has quit IRC | 05:21 | |
| *** unrahul has quit IRC | 05:21 | |
| *** kozhukalov has quit IRC | 05:21 | |
| *** antonym has quit IRC | 05:21 | |
| *** GheRivero has quit IRC | 05:21 | |
| *** zxiiro has quit IRC | 05:21 | |
| *** tsufiev has quit IRC | 05:21 | |
| *** kambiz has quit IRC | 05:21 | |
| *** teran has quit IRC | 05:21 | |
| *** igorbelikov has quit IRC | 05:21 | |
| *** mrhillsman has quit IRC | 05:21 | |
| *** dmellado has quit IRC | 05:21 | |
| *** bexelbie has quit IRC | 05:21 | |
| *** NobodyCam has quit IRC | 05:21 | |
| *** taron has quit IRC | 05:21 | |
| *** Douhet has quit IRC | 05:21 | |
| *** zaro has quit IRC | 05:21 | |
| *** jlvillal has quit IRC | 05:21 | |
| *** beisner has quit IRC | 05:21 | |
| *** timburke has quit IRC | 05:21 | |
| *** filler has quit IRC | 05:21 | |
| *** jlk has quit IRC | 05:21 | |
| *** AJaeger has quit IRC | 05:21 | |
| *** jasondotstar has quit IRC | 05:21 | |
| *** jak-jak_ has quit IRC | 05:21 | |
| *** trash_ has quit IRC | 05:21 | |
| *** vkmc has quit IRC | 05:21 | |
| *** csmart has quit IRC | 05:21 | |
| *** clayton has quit IRC | 05:21 | |
| *** mrodden has quit IRC | 05:21 | |
| *** tbarron has quit IRC | 05:21 | |
| *** tangchen has quit IRC | 05:21 | |
| *** sirushti has quit IRC | 05:21 | |
| *** sslypushenko has quit IRC | 05:21 | |
| *** dosaboy has quit IRC | 05:21 | |
| *** gema has quit IRC | 05:21 | |
| *** yee379 has quit IRC | 05:21 | |
| *** tomhambleton has quit IRC | 05:21 | |
| *** ggillies has quit IRC | 05:21 | |
| *** dteselkin has quit IRC | 05:21 | |
| *** darvon has quit IRC | 05:21 | |
| *** weshay has quit IRC | 05:21 | |
| *** furlongm has quit IRC | 05:21 | |
| *** igormarnat has quit IRC | 05:21 | |
| *** lezbar has quit IRC | 05:21 | |
| *** MarkAtwood has quit IRC | 05:21 | |
| *** auggy has quit IRC | 05:21 | |
| *** jesusaur has quit IRC | 05:21 | |
| *** paul___ has quit IRC | 05:21 | |
| *** mnaser has quit IRC | 05:21 | |
| *** aunnam has quit IRC | 05:21 | |
| *** fmccrthy has quit IRC | 05:21 | |
| *** katyafervent_awa has quit IRC | 05:21 | |
| *** DuncanT has quit IRC | 05:21 | |
| *** amoralej has quit IRC | 05:21 | |
| *** rattboi has quit IRC | 05:21 | |
| *** sshnaidm has quit IRC | 05:21 | |
| *** winggundamth has quit IRC | 05:21 | |
| *** vsaienk0 has quit IRC | 05:21 | |
| *** Shrews_ has quit IRC | 05:21 | |
| *** kragniz has quit IRC | 05:21 | |
| *** aviau has quit IRC | 05:21 | |
| *** jklare has quit IRC | 05:21 | |
| *** tobiash has quit IRC | 05:21 | |
| *** mjblack has quit IRC | 05:21 | |
| *** hogepodge has quit IRC | 05:21 | |
| *** stevebaker has quit IRC | 05:21 | |
| *** devananda|bbl has quit IRC | 05:21 | |
| *** crinkle has quit IRC | 05:21 | |
| *** Guest27215 has quit IRC | 05:21 | |
| *** szaher has quit IRC | 05:21 | |
| *** dgonzalez has quit IRC | 05:21 | |
| *** pleia2 has quit IRC | 05:21 | |
| *** alaski has quit IRC | 05:21 | |
| *** mrmartin has quit IRC | 05:21 | |
| *** Guest66666 has quit IRC | 05:21 | |
| *** firenka has quit IRC | 05:21 | |
| *** SpamapS has quit IRC | 05:21 | |
| *** BobBall has quit IRC | 05:21 | |
| *** fabo has quit IRC | 05:21 | |
| *** matbu has quit IRC | 05:21 | |
| *** logan- has quit IRC | 05:21 | |
| *** mrmartin_ is now known as mrmartin | 05:21 | |
| *** bexelbie- is now known as bexelbie | 05:21 | |
| *** kambiz_ is now known as kambiz | 05:21 | |
| *** tbarron_ is now known as tbarron | 05:21 | |
| *** mrodden_ is now known as mrodden | 05:21 | |
| *** hichihara has quit IRC | 05:21 | |
| *** filler_ is now known as filler | 05:21 | |
| *** beisner- is now known as beisner | 05:21 | |
| *** dgonzalez_ is now known as dgonzalez | 05:21 | |
| *** mjblack- is now known as mjblack | 05:21 | |
| *** igormarnat__ is now known as igormarnat | 05:21 | |
| *** tonytan_brb has quit IRC | 05:21 | |
| *** clayton_ is now known as clayton | 05:21 | |
| *** Douhet has joined #openstack-infra | 05:21 | |
| *** sirushti_ is now known as sirushti | 05:21 | |
| *** monester__ is now known as monester_ | 05:21 | |
| *** logan- has joined #openstack-infra | 05:21 | |
| *** katyafervent_awa has joined #openstack-infra | 05:21 | |
| *** trash has joined #openstack-infra | 05:21 | |
| *** paul___ has joined #openstack-infra | 05:22 | |
| *** tomhambleton_ is now known as tomhambleton | 05:22 | |
| *** dmellado has joined #openstack-infra | 05:22 | |
| *** igorbelikov_ is now known as igorbelikov | 05:22 | |
| *** sslypushenko_ is now known as sslypushenko | 05:22 | |
| *** zaro_ is now known as zaro | 05:22 | |
| *** zxiiro_ is now known as zxiiro | 05:22 | |
| *** GheRivero_ is now known as GheRivero | 05:22 | |
| *** kozhukalov_ is now known as kozhukalov | 05:22 | |
| *** tsufiev has joined #openstack-infra | 05:22 | |
| *** NobodyCam_ is now known as NobodyCam | 05:22 | |
| *** teran_ is now known as teran | 05:22 | |
| *** betherly_ is now known as betherly | 05:23 | |
| *** jlvillal_ is now known as jlvillal | 05:23 | |
| *** fmccrthy_ is now known as fmccrthy | 05:23 | |
| *** dham1 has quit IRC | 05:23 | |
| *** jak-jak has joined #openstack-infra | 05:23 | |
| *** vkmc has joined #openstack-infra | 05:23 | |
| *** jlvillal is now known as Guest97128 | 05:23 | |
| *** jak-jak is now known as Guest22950 | 05:24 | |
| *** simonmcc_ is now known as simonmcc | 05:25 | |
| *** auggy_ is now known as auggy | 05:25 | |
| *** sflanigan_afk is now known as sflanigan | 05:25 | |
| *** unrahul_ is now known as unrahul | 05:25 | |
| *** MarkAtwood_ is now known as MarkAtwood | 05:26 | |
| *** yuval has joined #openstack-infra | 05:26 | |
| *** psachin has joined #openstack-infra | 05:26 | |
| *** winggundamth has joined #openstack-infra | 05:27 | |
| *** rattboi has joined #openstack-infra | 05:27 | |
| *** SpamapS has joined #openstack-infra | 05:27 | |
| *** antonym has joined #openstack-infra | 05:27 | |
| *** mrhillsman has joined #openstack-infra | 05:27 | |
| *** aunnam has joined #openstack-infra | 05:28 | |
| *** devananda|bbl has joined #openstack-infra | 05:28 | |
| *** tangchen has joined #openstack-infra | 05:28 | |
| *** kun_huang has joined #openstack-infra | 05:28 | |
| *** Shrews_ has joined #openstack-infra | 05:28 | |
| *** AJaeger has joined #openstack-infra | 05:28 | |
| *** pzhurba_ has joined #openstack-infra | 05:28 | |
| *** jesusaur has joined #openstack-infra | 05:28 | |
| *** vsaienk0 has joined #openstack-infra | 05:28 | |
| *** sshnaidm has joined #openstack-infra | 05:28 | |
| *** sileht has joined #openstack-infra | 05:28 | |
| *** kragniz has joined #openstack-infra | 05:29 | |
| *** dham1 has joined #openstack-infra | 05:30 | |
| *** rcernin has joined #openstack-infra | 05:30 | |
| *** darvon_ is now known as darvon | 05:31 | |
| *** serverascode_ is now known as serverascode | 05:31 | |
| *** mnaser has joined #openstack-infra | 05:31 | |
| *** pgadiya has joined #openstack-infra | 05:32 | |
| *** DuncanT_ is now known as DuncanT | 05:36 | |
| *** amitgandhinz has joined #openstack-infra | 05:36 | |
| *** gongysh has joined #openstack-infra | 05:38 | |
| *** yonglihe has joined #openstack-infra | 05:39 | |
| *** caowei has joined #openstack-infra | 05:41 | |
| *** ramishra has joined #openstack-infra | 05:48 | |
| *** ihrachys has joined #openstack-infra | 05:49 | |
| *** sandanar has joined #openstack-infra | 05:49 | |
| *** amotoki has joined #openstack-infra | 05:50 | |
| *** promethe1nfire is now known as prometheanfire | 05:50 | |
| *** matbu_ is now known as matbu | 05:55 | |
| *** senk_ has quit IRC | 05:55 | |
| *** martinkopec has joined #openstack-infra | 05:57 | |
| *** ihrachys has quit IRC | 05:58 | |
| *** thorst has joined #openstack-infra | 06:05 | |
| *** nherciu has joined #openstack-infra | 06:06 | |
| *** vikrant has quit IRC | 06:06 | |
| *** ccamacho has joined #openstack-infra | 06:08 | |
| *** Ravikiran_K has joined #openstack-infra | 06:08 | |
| *** tangchen has quit IRC | 06:09 | |
| *** amitgandhinz has quit IRC | 06:10 | |
| *** ijw has joined #openstack-infra | 06:12 | |
| *** thorst has quit IRC | 06:12 | |
| *** drifterza has quit IRC | 06:13 | |
| *** makowals has joined #openstack-infra | 06:14 | |
| *** ijw has quit IRC | 06:18 | |
| *** yolanda has quit IRC | 06:18 | |
| *** vikrant has joined #openstack-infra | 06:22 | |
| AJaeger | kota_: if you have any questions or comments, just speak up and later somebody will read backscroll and might answer. | 06:23 |
| kota_ | AJaeger: ok, thanks | 06:24 |
| kota_ | AJaeger: my question is related to gerrit job, probably something on project-config | 06:25 |
| *** slaweq has joined #openstack-infra | 06:25 | |
| kota_ | I'd like to know how we can make the release tarball in tarball site, https://tarballs.openstack.org | 06:25 |
| kota_ | i'm working on swift3 project (S3API on top of OpenStack Swift) and it uses openstack namespace | 06:26 |
| *** tangchen has joined #openstack-infra | 06:26 | |
| kota_ | swift3 already has some jobs in jenkins and the master tarball exists on https://tarballs.openstack.org/swift3/ | 06:27 |
| kota_ | but currently we do not keep tarballs for each release. I'd like to make it as well as swift, https://tarballs.openstack.org/swift/ (something like "swift3-1.10.tar.gz") | 06:29 |
| *** andreas_s has joined #openstack-infra | 06:29 | |
| kota_ | I wonder if someone helps me to set the job to make it. | 06:29 |
| kota_ | currently, I'm following http://docs.openstack.org/infra/manual/drivers.html to make the release tag. | 06:30 |
| *** ihrachys has joined #openstack-infra | 06:30 | |
| AJaeger | kota_: such tarballs are done during release | 06:30 |
| AJaeger | So, let me check in project-config your setup... | 06:31 |
| kota_ | AJaeger: sure, wait a sec, will paste the link for github | 06:31 |
| AJaeger | kota_: there's a release pipeline setup. | 06:31 |
| AJaeger | kota_: no need, have it locally | 06:32 |
| AJaeger | kota_: so, with the release pipeline setup, a tag should get uploaded | 06:32 |
| *** senk_ has joined #openstack-infra | 06:32 | |
| AJaeger | kota_: What tag did you made for swift3? | 06:34 |
| kota_ | AJaeger: https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L16134-L16158 for layout in project-config | 06:34 |
| kota_ | wait, | 06:34 |
| kota_ | I tagged "v1.11", recently | 06:34 |
| *** rlu has joined #openstack-infra | 06:35 | |
| *** Sukhdev has quit IRC | 06:35 | |
| kota_ | AJaeger: https://github.com/openstack/swift3/releases/tag/v1.11 this one | 06:35 |
| AJaeger | kota_: that's not a valid tag format, see http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/layout.yaml#n133 | 06:35 |
| AJaeger | let me find you a link... | 06:35 |
| kota_ | ah, we should likely 1.11 (not set the "v" prefix)? | 06:36 |
| AJaeger | kota_: we follow pep440 for versions - so it should indeed be 1.11 | 06:36 |
| AJaeger | Let me update the drivers.html file for this... | 06:37 |
| kota_ | AJaeger: alright, try to tag with 1.11, thanks | 06:38 |
| *** rlu has quit IRC | 06:38 | |
| *** verdurin has joined #openstack-infra | 06:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: drivers: Mention PEP 440 for tagging https://review.openstack.org/376211 | 06:40 |
| AJaeger | kota_: documentation update ^ | 06:41 |
| kota_ | AJaeger: thanks, looking at PEP 440 restriction.... | 06:43 |
| kota_ | AJaeger: quick question. the PEP 440 allows just a "Major.miner" format? | 06:43 |
| kota_ | AJaeger: in your doc at patch 376211, it includes rev like "major.miner.rev" | 06:44 |
| *** eeiden has quit IRC | 06:44 | |
| kota_ | ah, it looks ok just 'major.miner', I found the example, https://www.python.org/dev/peps/pep-0440/#final-releases | 06:45 |
| *** flip214_ is now known as flip214 | 06:45 | |
| *** pcaruana has joined #openstack-infra | 06:45 | |
| *** flip214 has quit IRC | 06:45 | |
| *** flip214 has joined #openstack-infra | 06:45 | |
| AJaeger | kota_: See also the regex I cited in layout.yaml - Major.minor is fine. But most projects use all three components, so wanted to show that in the message. Let me say "1.2.3" or "1.2" | 06:47 |
| kota_ | AJaeger: sounds great. | 06:47 |
| *** ttx has quit IRC | 06:48 | |
| *** ttx has joined #openstack-infra | 06:48 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: drivers: Mention PEP 440 for tagging https://review.openstack.org/376211 | 06:48 |
| *** Na3iL has joined #openstack-infra | 06:50 | |
| *** psachin has quit IRC | 06:51 | |
| *** eeiden has joined #openstack-infra | 06:55 | |
| *** rlu has joined #openstack-infra | 06:55 | |
| zigo | AJaeger: Hi there! Do you know what triggers the merge of upstream branches? | 06:56 |
| zigo | AJaeger: https://github.com/openstack/networking-mlnx/releases <--- Has a 9.0.0.0b1 but deb-networking-mlnx doesn't yet. | 06:56 |
| kota_ | AJeager: that update looks fine. I'll add +1 atter, I get success with the tarballing. | 06:56 |
| *** tangchen has quit IRC | 06:56 | |
| AJaeger | zigo: no idea. | 06:59 |
| AJaeger | kota_: tell us whether it works, please | 06:59 |
| kota_ | AJaeger: sure | 07:00 |
| *** psachin has joined #openstack-infra | 07:00 | |
| *** dizquierdo has joined #openstack-infra | 07:02 | |
| AJaeger | kota_: looking at http://status.openstack.org/zuul/ I see swift3 in the release pipeline ;) | 07:02 |
| AJaeger | So, it should work ;) | 07:02 |
| kota_ | AJaeger: fantastic! thanks a lot! | 07:03 |
| *** flepied1 has quit IRC | 07:03 | |
| AJaeger | you're welcome | 07:04 |
| kota_ | yey, i got the release tarball https://tarballs.openstack.org/swift3/ !!!! | 07:04 |
| *** flepied has joined #openstack-infra | 07:04 | |
| AJaeger | party time ;) | 07:05 |
| *** amitgandhinz has joined #openstack-infra | 07:06 | |
| *** nmagnezi has joined #openstack-infra | 07:08 | |
| *** thorst has joined #openstack-infra | 07:10 | |
| *** flepied has quit IRC | 07:11 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Multinode periodic temepest with no skips https://review.openstack.org/376237 | 07:12 |
| *** acabot has joined #openstack-infra | 07:13 | |
| *** ijw has joined #openstack-infra | 07:14 | |
| *** makowals has quit IRC | 07:14 | |
| *** yaume has joined #openstack-infra | 07:14 | |
| *** migi has joined #openstack-infra | 07:16 | |
| *** jascott1_ has quit IRC | 07:17 | |
| *** thorst has quit IRC | 07:17 | |
| *** jascott1_ has joined #openstack-infra | 07:18 | |
| *** ijw has quit IRC | 07:19 | |
| *** nt has quit IRC | 07:19 | |
| *** roxanaghe has joined #openstack-infra | 07:19 | |
| *** jpena|off is now known as jpena | 07:21 | |
| *** nt has joined #openstack-infra | 07:22 | |
| *** jtomasek has quit IRC | 07:22 | |
| *** roxanaghe has quit IRC | 07:24 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 07:25 |
| *** matrohon has joined #openstack-infra | 07:25 | |
| *** vikrant has quit IRC | 07:27 | |
| *** drifterza has joined #openstack-infra | 07:28 | |
| *** ggnel_t has joined #openstack-infra | 07:29 | |
| *** jaosorior has joined #openstack-infra | 07:29 | |
| *** HeOS has joined #openstack-infra | 07:30 | |
| *** rlu has quit IRC | 07:31 | |
| *** vikrant has joined #openstack-infra | 07:31 | |
| *** jlanoux has joined #openstack-infra | 07:31 | |
| *** vikrant is now known as vikrant|lunch | 07:31 | |
| *** jklare_ has quit IRC | 07:32 | |
| *** jklare has joined #openstack-infra | 07:33 | |
| *** Hal1 has joined #openstack-infra | 07:33 | |
| *** rlu has joined #openstack-infra | 07:37 | |
| *** claudiub has quit IRC | 07:39 | |
| *** Hal1 has quit IRC | 07:40 | |
| *** oanson has joined #openstack-infra | 07:40 | |
| *** amitgandhinz has quit IRC | 07:41 | |
| *** jpich has joined #openstack-infra | 07:42 | |
| *** amoralej_ is now known as amoralej | 07:43 | |
| zigo | AJaeger: Would you know how to remove a package from jessie-newton-backports? | 07:46 |
| zigo | AJaeger: onovy broke everything by uploading debhelper there, we need to get it removed. | 07:46 |
| *** Hal1 has joined #openstack-infra | 07:47 | |
| AJaeger | zigo, I'm project-config core, not an infra root. I cannot help with those operational questions - as it'S nothing that can be done AFAIK automatically. | 07:47 |
| *** pilgrimstack has joined #openstack-infra | 07:47 | |
| zigo | AJaeger: Ok, thanks. I'll wait until an infra root gets up then. | 07:48 |
| *** hashar has joined #openstack-infra | 07:49 | |
| *** bhavik1 has joined #openstack-infra | 07:52 | |
| *** liusheng has joined #openstack-infra | 07:53 | |
| *** vsaienk0 has quit IRC | 07:55 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** aspiers_ is now known as aspiers | 08:01 | |
| *** zzzeek has joined #openstack-infra | 08:01 | |
| *** shu-mutou is now known as shu-mutou-AFK | 08:10 | |
| *** ijw has joined #openstack-infra | 08:10 | |
| *** bethwhite has joined #openstack-infra | 08:14 | |
| *** gnuoy has quit IRC | 08:14 | |
| strigazi | hi AJaeger, I'm trying to create a ci for magnum on our cloud. Is there any documentation for this? | 08:14 |
| *** thorst has joined #openstack-infra | 08:15 | |
| *** gnuoy` is now known as gnuoy | 08:15 | |
| *** bethwhite has quit IRC | 08:16 | |
| *** bethwhite has joined #openstack-infra | 08:16 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:18 |
| *** ijw has quit IRC | 08:18 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:19 |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 08:19 |
| *** tonytan4ever has joined #openstack-infra | 08:22 | |
| *** thorst has quit IRC | 08:22 | |
| *** ijw has joined #openstack-infra | 08:23 | |
| *** tonytan4ever has quit IRC | 08:27 | |
| strigazi | AJaeger, I have found it, I'll give it a go http://docs.openstack.org/infra/openstackci/third_party_ci.html#overview | 08:28 |
| *** pedroalvarez_ has left #openstack-infra | 08:28 | |
| *** asettle has joined #openstack-infra | 08:30 | |
| AJaeger | strigazi: yeah, that's the link I just wanted to give ;) | 08:30 |
| AJaeger | strigazi: talk with anteaya - and join the 3rd party IRC channel and meetings. | 08:31 |
| AJaeger | strigazi: #openstack-third-party-ci | 08:31 |
| *** vikrant|lunch has quit IRC | 08:31 | |
| strigazi | AJaeger, ok thanks. the goal of this is to have xlarge or 2xlarge nodes | 08:32 |
| *** vikrant has joined #openstack-infra | 08:32 | |
| *** vaggarwa_ has joined #openstack-infra | 08:33 | |
| *** yolanda has joined #openstack-infra | 08:35 | |
| yolanda | good morning | 08:35 |
| *** ijw has quit IRC | 08:37 | |
| strigazi | Good Morning yolanda | 08:37 |
| *** amitgandhinz has joined #openstack-infra | 08:37 | |
| strigazi | yolanda, Where can i check the logs of magnum's periodic jobs? | 08:38 |
| *** snarwade has joined #openstack-infra | 08:40 | |
| *** e0ne has joined #openstack-infra | 08:41 | |
| *** rossella_s has joined #openstack-infra | 08:41 | |
| jhesketh | Morning | 08:42 |
| *** jascott1_ has quit IRC | 08:42 | |
| AJaeger | morning, yolanda, jhesketh ! | 08:42 |
| AJaeger | yolanda, jhesketh: can you check https://review.openstack.org/375939 , please? | 08:43 |
| AJaeger | strigazi: If it's in openstack-health, you can use something like: | 08:43 |
| AJaeger | http://status.openstack.org/openstack-health/#/g/build_queue/periodic?groupKey=build_queue&searchJob=translation | 08:43 |
| *** jascott1_ has joined #openstack-infra | 08:43 | |
| *** derekh has joined #openstack-infra | 08:43 | |
| yolanda | hi, going to start with the review queue | 08:43 |
| AJaeger | thanks, yolanda | 08:44 |
| *** vaggarwa_ has quit IRC | 08:44 | |
| jhesketh | AJaeger: lgtm | 08:45 |
| *** florianf has quit IRC | 08:45 | |
| AJaeger | strigazi: which periodic magnum job are you talking about? | 08:45 |
| *** esikachev has joined #openstack-infra | 08:45 | |
| AJaeger | strigazi: check also http://logs.openstack.org/periodic/ - and then dig into job and sort by last changed | 08:45 |
| yolanda | strigazi, http://logs.openstack.org/periodic/periodic-dib-buildimage-ubuntu-mesos/ | 08:45 |
| yolanda | i think you could rename these jobs to mention magnum somewhere. I created the initial one, so my fault... | 08:45 |
| strigazi | AJaeger yolanda, thanks | 08:46 |
| AJaeger | yolanda, strigazi yes, please rename! | 08:46 |
| AJaeger | thanks, jhesketh | 08:46 |
| *** florianf has joined #openstack-infra | 08:46 | |
| *** jascott1_ has quit IRC | 08:48 | |
| *** e0ne has quit IRC | 08:51 | |
| *** Hal1 has quit IRC | 08:52 | |
| *** baoli has joined #openstack-infra | 08:54 | |
| *** Hal1 has joined #openstack-infra | 08:56 | |
| *** flepied has joined #openstack-infra | 08:57 | |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config: [magnum] Add magnum in dib jobs names https://review.openstack.org/376310 | 08:57 |
| *** dtardivel has joined #openstack-infra | 08:57 | |
| *** baoli has quit IRC | 08:59 | |
| *** acoles_ is now known as acoles | 09:01 | |
| *** e0ne has joined #openstack-infra | 09:01 | |
| *** acoles has left #openstack-infra | 09:01 | |
| *** chmouel_ is now known as chmouel | 09:03 | |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config: [magnum] Fix unbound variable in dib job template https://review.openstack.org/376313 | 09:05 |
| *** nijaba has quit IRC | 09:06 | |
| openstackgerrit | Dmitry Tantsur proposed openstack-infra/project-config: Switch ironic-inspector discovery job to Xenial https://review.openstack.org/376314 | 09:06 |
| dtantsur | AJaeger, ^^^ | 09:06 |
| AJaeger | thanks, dtantsur ! | 09:08 |
| AJaeger | clarkb: ^ | 09:08 |
| *** sambetts_ is now known as sambetts | 09:11 | |
| *** nijaba has joined #openstack-infra | 09:11 | |
| *** aglarendil__ has joined #openstack-infra | 09:11 | |
| *** amitgandhinz has quit IRC | 09:12 | |
| *** mhickey has joined #openstack-infra | 09:14 | |
| *** chem has joined #openstack-infra | 09:15 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove translations for releasenotes from branches https://review.openstack.org/375939 | 09:19 |
| openstackgerrit | Merged openstack-infra/project-config: Enable py35 voting on project watcher https://review.openstack.org/376062 | 09:19 |
| *** rossella_s has quit IRC | 09:19 | |
| *** rossella_s has joined #openstack-infra | 09:19 | |
| *** thorst has joined #openstack-infra | 09:20 | |
| *** jistr has quit IRC | 09:20 | |
| *** florianf has quit IRC | 09:20 | |
| *** chem has quit IRC | 09:21 | |
| *** chem has joined #openstack-infra | 09:22 | |
| *** florianf has joined #openstack-infra | 09:24 | |
| *** jlanoux has quit IRC | 09:25 | |
| *** thorst has quit IRC | 09:27 | |
| *** tosky has joined #openstack-infra | 09:28 | |
| openstackgerrit | Merged openstack-infra/project-config: Remove ovh-bhs1 from nodepool.yaml https://review.openstack.org/376037 | 09:31 |
| openstackgerrit | Merged openstack-infra/project-config: horizon: Update job skip condition for doc-only changes https://review.openstack.org/375935 | 09:31 |
| openstackgerrit | Merged openstack-infra/project-config: Enable py35 voting on Cinder https://review.openstack.org/374030 | 09:31 |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Disable test_volume_from_snapshot tests for Cinder BDD job" https://review.openstack.org/375700 | 09:31 |
| openstackgerrit | Merged openstack-infra/project-config: Add Craton's channels to accessbot https://review.openstack.org/375677 | 09:31 |
| *** jlanoux has joined #openstack-infra | 09:32 | |
| *** esikachev has quit IRC | 09:34 | |
| *** pgadiya has quit IRC | 09:35 | |
| *** andreykurilin_ has left #openstack-infra | 09:36 | |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config: [magnum] Add magnum in dib jobs names https://review.openstack.org/376310 | 09:36 |
| *** nickola has joined #openstack-infra | 09:38 | |
| *** flaper87 has joined #openstack-infra | 09:39 | |
| *** flaper87 has quit IRC | 09:39 | |
| *** flaper87 has joined #openstack-infra | 09:39 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci: Add IPv6 network configuration for ipv6 job types https://review.openstack.org/363674 | 09:39 |
| *** gongysh has quit IRC | 09:42 | |
| *** ijw has joined #openstack-infra | 09:43 | |
| AJaeger | strigazi: for changes like the EXTRA_PROJECTS, looking at the compare-xml output is usefull: http://logs.openstack.org/13/376313/1/check/gate-project-config-compare-xml/a467ac3/console.html#_2016-09-26_09_22_47_565387 shows that your change is not really fixing the problem. | 09:44 |
| *** panda is now known as panda|afk | 09:44 | |
| AJaeger | strigazi: I commented with what I think is broken. | 09:44 |
| strigazi | AJaeger: thanks, looking | 09:44 |
| *** pgadiya has joined #openstack-infra | 09:49 | |
| *** andreykurilin_ has joined #openstack-infra | 09:49 | |
| *** nickola has quit IRC | 09:49 | |
| *** paulobanon has joined #openstack-infra | 09:49 | |
| *** ijw has quit IRC | 09:49 | |
| *** yanyanhu has quit IRC | 09:51 | |
| openstackgerrit | Spyros Trigazis proposed openstack-infra/project-config: [magnum] Fix unbound variable in dib job template https://review.openstack.org/376313 | 09:51 |
| *** daemontool has joined #openstack-infra | 09:54 | |
| *** esikachev has joined #openstack-infra | 09:58 | |
| *** zhurong has quit IRC | 10:01 | |
| openstackgerrit | Merged openstack-infra/python-storyboardclient: Update 'usage' https://review.openstack.org/372454 | 10:01 |
| openstackgerrit | Merged openstack-infra/release-tools: decode reno notes to unicode https://review.openstack.org/374953 | 10:01 |
| openstackgerrit | Merged openstack-infra/release-tools: update propose-final-releases to handle cycle-trailing projects https://review.openstack.org/375075 | 10:01 |
| openstackgerrit | Stephen Finucane proposed openstack-dev/hacking: Add optional checks for many more asserts https://review.openstack.org/376352 | 10:05 |
| *** sdake has joined #openstack-infra | 10:06 | |
| *** yamamoto_ has quit IRC | 10:07 | |
| *** amitgandhinz has joined #openstack-infra | 10:08 | |
| openstackgerrit | Merged openstack-infra/project-config: [magnum] Add magnum in dib jobs names https://review.openstack.org/376310 | 10:08 |
| *** OferBarber has joined #openstack-infra | 10:09 | |
| *** claudiub has joined #openstack-infra | 10:11 | |
| *** OferBarber_ has quit IRC | 10:13 | |
| *** sdague has joined #openstack-infra | 10:14 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/os-testr: Update home-page url https://review.openstack.org/356179 | 10:14 |
| *** panda|afk is now known as panda | 10:15 | |
| *** aarefiev_ is now known as aarefiev | 10:17 | |
| *** bhavik1 has left #openstack-infra | 10:19 | |
| *** takashi has quit IRC | 10:24 | |
| *** thorst has joined #openstack-infra | 10:26 | |
| *** jtomasek has joined #openstack-infra | 10:27 | |
| *** flepied has quit IRC | 10:31 | |
| *** jlanoux has quit IRC | 10:32 | |
| *** thorst has quit IRC | 10:33 | |
| *** nickola has joined #openstack-infra | 10:33 | |
| *** nherciu_ has joined #openstack-infra | 10:35 | |
| *** PalTale has quit IRC | 10:38 | |
| *** nherciu has quit IRC | 10:38 | |
| *** PalTale has joined #openstack-infra | 10:39 | |
| AJaeger | zigo, please don't just -1 changes without a comment like you did on https://review.openstack.org/#/c/375851/ | 10:39 |
| AJaeger | zigo, it's important for *all* reviewers to see why things change. | 10:40 |
| *** sshnaidm is now known as sshnaidm|afk | 10:41 | |
| *** berendt has joined #openstack-infra | 10:41 | |
| *** AlexxNic_ has quit IRC | 10:42 | |
| *** baoli has joined #openstack-infra | 10:43 | |
| *** rodrigods has quit IRC | 10:43 | |
| *** amitgandhinz has quit IRC | 10:43 | |
| *** rodrigods has joined #openstack-infra | 10:43 | |
| *** ijw has joined #openstack-infra | 10:45 | |
| *** Hunner_ is now known as Hunner | 10:46 | |
| *** AlexxNica has joined #openstack-infra | 10:47 | |
| *** baoli has quit IRC | 10:48 | |
| *** roxanaghe has joined #openstack-infra | 10:48 | |
| *** tuanla has quit IRC | 10:50 | |
| *** ijw has quit IRC | 10:51 | |
| *** roxanaghe has quit IRC | 10:52 | |
| *** rhallisey has joined #openstack-infra | 10:55 | |
| *** jkilpatr has quit IRC | 10:59 | |
| *** jlanoux has joined #openstack-infra | 11:00 | |
| *** tqtran has joined #openstack-infra | 11:01 | |
| *** Ravikiran_K has quit IRC | 11:02 | |
| *** rtheis has joined #openstack-infra | 11:03 | |
| *** sdake_ has joined #openstack-infra | 11:04 | |
| *** sdake has quit IRC | 11:04 | |
| *** yamamoto has joined #openstack-infra | 11:05 | |
| *** tqtran has quit IRC | 11:05 | |
| *** vikrant has quit IRC | 11:07 | |
| *** sulo_ is now known as sulo | 11:09 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** mkoderer has joined #openstack-infra | 11:13 | |
| *** dizquierdo has quit IRC | 11:17 | |
| openstackgerrit | Robin Naundorf proposed openstack-infra/system-config: Fix formatting in firehose docs https://review.openstack.org/376408 | 11:21 |
| *** vikrant has joined #openstack-infra | 11:22 | |
| *** dizquierdo has joined #openstack-infra | 11:29 | |
| *** ldnunes has joined #openstack-infra | 11:30 | |
| openstackgerrit | Merged openstack-infra/storyboard-webclient: Order projects in task list alphabetically https://review.openstack.org/373485 | 11:32 |
| *** jpena is now known as jpena|lunch | 11:34 | |
| *** yuval has quit IRC | 11:36 | |
| *** AlexxNica has quit IRC | 11:37 | |
| *** onovy has quit IRC | 11:38 | |
| *** onovy has joined #openstack-infra | 11:38 | |
| *** jkilpatr has joined #openstack-infra | 11:38 | |
| *** amitgandhinz has joined #openstack-infra | 11:38 | |
| *** schang has quit IRC | 11:40 | |
| *** dougshelley66 has quit IRC | 11:42 | |
| *** amoralej is now known as amoralej|lunch | 11:43 | |
| *** sdake_ has quit IRC | 11:43 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4 https://review.openstack.org/371209 | 11:44 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4 https://review.openstack.org/371209 | 11:44 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: WIP - Deploy TripleO with Puppet 4 https://review.openstack.org/371209 | 11:44 |
| *** tpsilva has joined #openstack-infra | 11:45 | |
| *** esikachev has quit IRC | 11:46 | |
| *** OferBarber_ has joined #openstack-infra | 11:46 | |
| *** cdent has joined #openstack-infra | 11:46 | |
| *** schang has joined #openstack-infra | 11:46 | |
| *** gildub has quit IRC | 11:47 | |
| *** gildub has joined #openstack-infra | 11:47 | |
| *** ijw has joined #openstack-infra | 11:47 | |
| *** gongysh has joined #openstack-infra | 11:47 | |
| *** dougshelley66 has joined #openstack-infra | 11:47 | |
| *** jtomasek has quit IRC | 11:48 | |
| *** OferBarber has quit IRC | 11:49 | |
| *** thorst has joined #openstack-infra | 11:49 | |
| *** dprince has joined #openstack-infra | 11:51 | |
| *** EricGonczer_ has joined #openstack-infra | 11:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 11:52 | |
| pabelanger | guten Morgen | 11:52 |
| *** artmr has joined #openstack-infra | 11:53 | |
| *** tonytan4ever has joined #openstack-infra | 11:54 | |
| *** ijw has quit IRC | 11:54 | |
| *** alaski_ is now known as alaski | 11:54 | |
| *** vikrant has quit IRC | 11:54 | |
| openstackgerrit | mathieu bultel proposed openstack-infra/tripleo-ci: Implement overcloud upgrade job - Mitaka -> Newton https://review.openstack.org/323750 | 11:55 |
| *** OferBarber has joined #openstack-infra | 11:57 | |
| *** raildo has joined #openstack-infra | 11:58 | |
| AJaeger | good morning, pabelanger | 11:58 |
| *** tonytan4ever has quit IRC | 11:58 | |
| AJaeger | pabelanger: will you great us in NOvember with "Buenos dias"? ;) | 11:58 |
| *** OferBarber_ has quit IRC | 11:58 | |
| pabelanger | AJaeger: :) | 11:59 |
| *** zhurong has joined #openstack-infra | 11:59 | |
| *** abramley has quit IRC | 12:00 | |
| *** asettle has quit IRC | 12:03 | |
| AJaeger | pabelanger: I cannot log via webui into gerrit right now, seems launchpad openid provider is broken ;( | 12:03 |
| AJaeger | pabelanger: could you review 375696 later, please? That doubles the osic cloud quota - to 1024 nodes. Change looks fine but I didn't want to merge without an infra-root around... | 12:04 |
| pabelanger | AJaeger: sure, I think we wanted to hold off until later this morning | 12:05 |
| AJaeger | fine with me;) | 12:05 |
| *** vikrant has joined #openstack-infra | 12:05 | |
| *** abramley has joined #openstack-infra | 12:06 | |
| *** dosaboy has joined #openstack-infra | 12:07 | |
| *** dosaboy has quit IRC | 12:09 | |
| *** zhurong has quit IRC | 12:09 | |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Multinode periodic tempest with no skips https://review.openstack.org/376237 | 12:10 |
| *** boden has joined #openstack-infra | 12:10 | |
| *** kota_ has left #openstack-infra | 12:10 | |
| *** zhurong has joined #openstack-infra | 12:10 | |
| boden | Is there a current issue with review.openstack.org + OpenID? When I go to the login page I get “Provider is not supported, or was incorrectly entered.” | 12:11 |
| *** icey has joined #openstack-infra | 12:11 | |
| *** amitgandhinz has quit IRC | 12:11 | |
| *** yamamoto has joined #openstack-infra | 12:12 | |
| AJaeger | boden: that would be a problem with launchpad itself... | 12:13 |
| boden | ah ok | 12:14 |
| *** asettle has joined #openstack-infra | 12:14 | |
| *** dosaboy_ has quit IRC | 12:14 | |
| AJaeger | boden, check https://bugs.launchpad.net/ | 12:14 |
| pabelanger | https://twitter.com/launchpadstatus/status/780374871821942785 | 12:14 |
| AJaeger | nice "Launchpad is currently unavailable due to a firewall issue. We're looking into it." | 12:14 |
| pabelanger | looks like launchpad.net is down right now | 12:14 |
| *** caowei has quit IRC | 12:14 | |
| boden | Thanks guys; appreciate it | 12:15 |
| *** gordc has joined #openstack-infra | 12:16 | |
| *** pcarver has joined #openstack-infra | 12:18 | |
| *** gildub has quit IRC | 12:18 | |
| *** yamamoto has quit IRC | 12:19 | |
| *** markvoelker has joined #openstack-infra | 12:20 | |
| *** psilvad has joined #openstack-infra | 12:20 | |
| *** kgiusti has joined #openstack-infra | 12:21 | |
| *** dosaboy has joined #openstack-infra | 12:21 | |
| AJaeger | boden: try again | 12:23 |
| boden | AJaeger: I’m in.. Thanks | 12:24 |
| AJaeger | pabelanger: could you +2A a xenial change for ironic, please? https://review.openstack.org/#/c/376314/ | 12:24 |
| rodrigods | so quick question, if i want to create a job that sets/changes a localrc setting from devstack, the correct way is by using "export THE_VAR_I_WANT"? | 12:25 |
| AJaeger | rodrigods: quick answer: no. | 12:26 |
| AJaeger | rodrigods: you can set devstack vars directly. | 12:26 |
| AJaeger | Let me get you an example for other stuff... | 12:27 |
| rodrigods | AJaeger, cool, thanks | 12:27 |
| AJaeger | http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/devstack-gate.yaml#n240 | 12:27 |
| AJaeger | rodrigods: best to check that directory for other examples as well... | 12:28 |
| mtreinish | fungi: so it looks like mosquitto is down again, is there enough evidence to blame logrotate now? (I can't remember what we were supposed to look for) | 12:28 |
| rodrigods | AJaeger, hmm great | 12:28 |
| rodrigods | thanks AJaeger | 12:29 |
| *** moravec has quit IRC | 12:29 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci: Add TripleO scenarios jobs in status dashboard https://review.openstack.org/367324 | 12:29 |
| openstackgerrit | Maciej Relewicz proposed openstack-infra/puppet-apps_site: Glare support for app-catalog https://review.openstack.org/359029 | 12:29 |
| *** gouthamr has joined #openstack-infra | 12:30 | |
| *** mdrabe has joined #openstack-infra | 12:30 | |
| *** asettle has quit IRC | 12:31 | |
| *** yamamoto has joined #openstack-infra | 12:31 | |
| *** asettle has joined #openstack-infra | 12:32 | |
| *** amotoki has quit IRC | 12:32 | |
| *** rfolco has joined #openstack-infra | 12:33 | |
| *** ccamacho is now known as ccamacho|lunch | 12:35 | |
| *** roxanaghe has joined #openstack-infra | 12:36 | |
| openstackgerrit | Merged openstack-infra/project-config: Switch ironic-inspector discovery job to Xenial https://review.openstack.org/376314 | 12:36 |
| *** yamamoto has quit IRC | 12:37 | |
| *** daemontool has quit IRC | 12:38 | |
| *** yamamoto has joined #openstack-infra | 12:39 | |
| *** roxanaghe has quit IRC | 12:41 | |
| *** markvoelker has quit IRC | 12:41 | |
| *** edmondsw has joined #openstack-infra | 12:43 | |
| *** jpena|lunch is now known as jpena | 12:43 | |
| *** asettle has quit IRC | 12:43 | |
| *** asettle has joined #openstack-infra | 12:43 | |
| *** daemontool has joined #openstack-infra | 12:44 | |
| *** esikachev has joined #openstack-infra | 12:44 | |
| pabelanger | mordred: clarkb: do we want to tag a glean release today? Would be nice to try and bring rax-iad back online | 12:46 |
| *** florianf has quit IRC | 12:46 | |
| *** trown is now known as trown|brb | 12:47 | |
| *** ijw has joined #openstack-infra | 12:51 | |
| *** dmsimard2 is now known as dmsimard | 12:51 | |
| *** esikachev has quit IRC | 12:52 | |
| *** Goneri has joined #openstack-infra | 12:53 | |
| *** ijw has quit IRC | 12:56 | |
| *** david-lyle has joined #openstack-infra | 12:56 | |
| *** markvoelker has joined #openstack-infra | 12:57 | |
| *** trown|brb is now known as trown | 12:57 | |
| *** sdake has joined #openstack-infra | 12:57 | |
| *** amoralej|lunch is now known as amoralej | 12:58 | |
| *** florianf has joined #openstack-infra | 12:58 | |
| *** xyang1 has joined #openstack-infra | 12:59 | |
| *** yamamoto has quit IRC | 12:59 | |
| *** EricGonczer_ has quit IRC | 12:59 | |
| *** yamamoto has joined #openstack-infra | 13:00 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Use configdrive for ironic tempest jobs https://review.openstack.org/376454 | 13:00 |
| *** yamamoto has quit IRC | 13:00 | |
| *** andrewbogott has quit IRC | 13:01 | |
| *** andrewbogott has joined #openstack-infra | 13:01 | |
| *** daemontool has quit IRC | 13:01 | |
| *** Ravikiran_K has joined #openstack-infra | 13:01 | |
| *** xyang1 has quit IRC | 13:02 | |
| *** sdake_ has joined #openstack-infra | 13:03 | |
| *** daemontool has joined #openstack-infra | 13:03 | |
| pabelanger | jeblair: clarkb: any objections to cleaning up the held nodes in nodepool? | 13:03 |
| *** thorst is now known as thorst_afk | 13:05 | |
| *** sdake has quit IRC | 13:06 | |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Force configdrive for ironic jobs https://review.openstack.org/376454 | 13:07 |
| *** amitgandhinz has joined #openstack-infra | 13:08 | |
| *** baoli has joined #openstack-infra | 13:09 | |
| *** amitgandhinz has quit IRC | 13:12 | |
| *** gongysh has quit IRC | 13:12 | |
| *** xyang1 has joined #openstack-infra | 13:13 | |
| *** caowei has joined #openstack-infra | 13:13 | |
| *** vikrant has quit IRC | 13:13 | |
| *** gongysh has joined #openstack-infra | 13:14 | |
| *** amitgandhinz has joined #openstack-infra | 13:14 | |
| pabelanger | AJaeger: zigo: project-config-core: we are ready to start having debian-jessie nodes use our AFS mirrors for apt: https://review.openstack.org/#/c/347140 | 13:14 |
| openstackgerrit | Merged openstack-infra/project-config: [magnum] Fix unbound variable in dib job template https://review.openstack.org/376313 | 13:16 |
| clarkb | pabelanger: +1 on glean release and yes you can vlear out my held nodes. Think I got that ssl stuff worling yeaterday | 13:16 |
| openstackgerrit | Merged openstack-infra/project-config: Add experimental networking-ovn multinode job https://review.openstack.org/373528 | 13:17 |
| pabelanger | clarkb: great. I have to run an errand this morning but ready on this side too | 13:18 |
| *** rlandy has joined #openstack-infra | 13:18 | |
| *** vrovachev has quit IRC | 13:18 | |
| openstackgerrit | Merged openstack-infra/project-config: Merge zuul-git-prep-upper-constraints/zuul-release-git-prep-upper-constraints https://review.openstack.org/352575 | 13:18 |
| *** jaosorior has quit IRC | 13:19 | |
| *** rossella_s has quit IRC | 13:19 | |
| *** esikachev has joined #openstack-infra | 13:19 | |
| *** rossella_s has joined #openstack-infra | 13:19 | |
| *** jaosorior has joined #openstack-infra | 13:19 | |
| *** amitgandhinz has quit IRC | 13:20 | |
| *** thorst_afk is now known as thorst | 13:20 | |
| *** amitgandhinz has joined #openstack-infra | 13:20 | |
| *** claudiub has quit IRC | 13:21 | |
| *** cardeois has joined #openstack-infra | 13:21 | |
| *** matt-borland has joined #openstack-infra | 13:21 | |
| *** amotoki has joined #openstack-infra | 13:26 | |
| *** nadya has joined #openstack-infra | 13:27 | |
| *** eharney has joined #openstack-infra | 13:27 | |
| *** mriedem has joined #openstack-infra | 13:28 | |
| *** srobert has joined #openstack-infra | 13:29 | |
| *** mdbooth_ is now known as mdbooth | 13:34 | |
| *** moravec has joined #openstack-infra | 13:34 | |
| *** esberglu has joined #openstack-infra | 13:34 | |
| *** Guest97128 is now known as jlvillal | 13:36 | |
| *** moravec has quit IRC | 13:36 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-subunit2sql: Switch to storing run wall time in runs database https://review.openstack.org/376512 | 13:37 |
| *** nmagnezi has quit IRC | 13:38 | |
| *** nadya has quit IRC | 13:39 | |
| *** annegentle has joined #openstack-infra | 13:40 | |
| *** sandanar has quit IRC | 13:40 | |
| openstackgerrit | Brad P. Crochet proposed openstack-infra/tripleo-ci: Only ask for the overcloud-full image https://review.openstack.org/363592 | 13:41 |
| *** rcarrillocruz has joined #openstack-infra | 13:44 | |
| *** jlk` is now known as jlk | 13:44 | |
| *** jlk has quit IRC | 13:44 | |
| *** jlk has joined #openstack-infra | 13:44 | |
| *** akshai has joined #openstack-infra | 13:44 | |
| *** tuannguyen has joined #openstack-infra | 13:45 | |
| *** fguillot has joined #openstack-infra | 13:46 | |
| *** annegentle has quit IRC | 13:46 | |
| *** sdake has joined #openstack-infra | 13:47 | |
| *** woodster_ has joined #openstack-infra | 13:48 | |
| *** yamamoto has joined #openstack-infra | 13:48 | |
| *** ccamacho|lunch is now known as ccamacho | 13:50 | |
| *** sdake_ has quit IRC | 13:50 | |
| *** sdake_ has joined #openstack-infra | 13:51 | |
| *** ijw has joined #openstack-infra | 13:52 | |
| *** cinerama` is now known as cinerama | 13:52 | |
| *** sdake has quit IRC | 13:54 | |
| *** tonytan4ever has joined #openstack-infra | 13:55 | |
| *** moravec has joined #openstack-infra | 13:56 | |
| *** mriedem is now known as mriedem_meeting | 13:56 | |
| *** ijw has quit IRC | 13:58 | |
| *** amotoki_ has joined #openstack-infra | 13:58 | |
| *** tonytan4ever has quit IRC | 13:59 | |
| *** moravec has quit IRC | 13:59 | |
| *** tonytan4ever has joined #openstack-infra | 13:59 | |
| *** amotoki has quit IRC | 14:01 | |
| *** zhurong has quit IRC | 14:01 | |
| *** moravec has joined #openstack-infra | 14:02 | |
| dmellado | rcarrillocruz: ping | 14:02 |
| *** hongbin has joined #openstack-infra | 14:03 | |
| *** moravec has quit IRC | 14:04 | |
| *** Jokke__ is now known as Jokke_ | 14:04 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Enable tlsproxy on ocata and above https://review.openstack.org/375776 | 14:04 |
| clarkb | sdague: ^ now using rm-services on the older branches | 14:04 |
| *** amotoki_ has quit IRC | 14:05 | |
| sdague | clarkb: I did a quick respin on the devstack patch earlier as well to be more consistent on cleanup | 14:05 |
| clarkb | sdague: would be great if you could review https://review.openstack.org/#/c/375778/1 as well which is the next step in that chain to actually use tls-proxy | 14:05 |
| clarkb | sdague: yup saw that, thanks | 14:05 |
| *** rbrndt has joined #openstack-infra | 14:05 | |
| *** yamamoto has quit IRC | 14:05 | |
| *** florianf has quit IRC | 14:06 | |
| *** inc0 has joined #openstack-infra | 14:06 | |
| *** yamamoto has joined #openstack-infra | 14:06 | |
| rcarrillocruz | dmellado: helo, what's up ? | 14:06 |
| sdague | clarkb: yeh, I think that's right, but we should probably get these other things landed first, right? | 14:06 |
| clarkb | sdague: yup the depends-on are all correct there. So d-g needs to go in first then project-config | 14:07 |
| sdague | ok | 14:07 |
| clarkb | sdague: then in parallel we can work on the multinode chain | 14:07 |
| anteaya | fungi: rcarrillocruz cacti is now not showing anything at all for localhost: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=7&select_first=true | 14:07 |
| zigo | pabelanger: Are you doing a mirror of security.debian.org as well? | 14:08 |
| clarkb | sdague: https://review.openstack.org/373219 is annoying because I had to update the version of ansible to fix a bug with the sync module which changed the behavior of running against an empty inventory group | 14:08 |
| zigo | pabelanger: Also, is this using ftp-sync? | 14:08 |
| *** florianf has joined #openstack-infra | 14:08 | |
| zigo | Or just the reprepro sync? | 14:08 |
| anteaya | fungi: rcarrillocruz nor is it showing any data for any other node that I have clicked on so far | 14:08 |
| *** cardeois_ has joined #openstack-infra | 14:08 | |
| fungi | AJaeger: i missed your reply to kota_ earlier, but beware that even though our jobs potentially support versions without 3 components, python projects that use pbr need to have 3 component versions or the jobs will break (because pbr/setuptools will normalize the version to 3 components and then that will fail to match the tag) | 14:09 |
| *** cardeois has quit IRC | 14:10 | |
| clarkb | sdague: I think I am going to flip around the dependency ordering for the multinode changes so that the fixes can merge (right now they can't merge as they are behind the hack to turn on tls-proxy in devstack change) | 14:10 |
| wznoinsk | hi infra | 14:10 |
| *** moravec has joined #openstack-infra | 14:10 | |
| sdague | clarkb: ok, cool | 14:10 |
| mwhahaha | hey folks, we're seeing some failures with zuul-cloner for our releasenotes jobs, did anything change around that? | 14:11 |
| mwhahaha | it appears that the branch is not properly being calculated | 14:11 |
| mwhahaha | http://logs.openstack.org/99/376499/1/check/gate-puppet-ovn-releasenotes/b6a1129/console.html#_2016-09-26_13_49_28_936124 | 14:11 |
| *** yamamoto has quit IRC | 14:11 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/devstack-gate: Copy ssl certs for multinode https://review.openstack.org/373219 | 14:11 |
| *** kashyap has joined #openstack-infra | 14:13 | |
| anteaya | mwhahaha: any idea when zuul-cloner: error: too few arguments started showing up for you? | 14:13 |
| *** pilgrimstack has quit IRC | 14:13 | |
| kashyap | Whoops, LaunchPad is down, it said "will be back in a couple of minutes" 10 minutes ago | 14:13 |
| clarkb | sdague: ok thats done. Should mean that the entire stack si properly reviewable now and can merge without needing to somehow get that enable tls proxy hack change in | 14:14 |
| *** pilgrimstack has joined #openstack-infra | 14:14 | |
| mwhahaha | anteaya: today | 14:14 |
| anteaya | mwhahaha: have you a sucessful run of gate-puppet-ovn-releasenotes recently? | 14:14 |
| fungi | kashyap: yep. i noticed the same unfortunately. we don't operate lp so if you need an eta you might check in #launchpad | 14:14 |
| mwhahaha | we're getting it on all of our modules | 14:14 |
| kashyap | fungi: I realize, it's Canonical that does | 14:14 |
| anteaya | mwhahaha: :( | 14:14 |
| wznoinsk | out of curiosity (cna't find much in the source code), do you plan to have nodepool support baremetal nodes? (I'm on a stage that ironic needs kernel/initrd uploaded apart from the main filesystem image) | 14:14 |
| mwhahaha | anteaya: http://logs.openstack.org/03/376503/1/check/gate-puppet-trove-releasenotes/5e091bf/console.html#_2016-09-26_13_49_41_221167 | 14:15 |
| clarkb | wznoinsk: nodepool should already support them just fine. Since nova boot should handle it all for us | 14:15 |
| anteaya | mwhahaha: that last time a patch merged to zuul was 11 days ago: http://git.openstack.org/cgit/openstack-infra/zuul/log/ | 14:15 |
| *** florianf has quit IRC | 14:15 | |
| kashyap | fungi: Are they on OFTC? | 14:16 |
| anteaya | mwhahaha: I believe you are seeing it in multiple jobs, I had asked if you knew when the last passing job was | 14:16 |
| clarkb | anteaya: mwhahaha https://review.openstack.org/352575 just merged this morning according to my irc scrollback | 14:16 |
| clarkb | maybe that is related? | 14:16 |
| kashyap | fungi: Err, they're on Freenode | 14:16 |
| kashyap | Disregard me | 14:16 |
| wznoinsk | clarkb: yes, boot works ok as long as the kernel/initrd images are uploaded to glance and specified in the filesystem's image (ubuntu, fedora...) properties, I'm thinking of this upload images stage from nodepool | 14:16 |
| *** florianf has joined #openstack-infra | 14:16 | |
| anteaya | clarkb: thank you | 14:17 |
| anteaya | mwhahaha: were jobs older than 2 hours passing for you? | 14:17 |
| clarkb | wznoinsk: I thought jroll said you *should* be able to boot qcow2s | 14:17 |
| mwhahaha | yea | 14:17 |
| rcarrillocruz | anteaya , fungi : | 14:17 |
| rcarrillocruz | root@cacti:/var/log/cacti# ping compute023.vanilla.ic.openstack.org | 14:17 |
| rcarrillocruz | bash: fork: Cannot allocate memory | 14:17 |
| rcarrillocruz | so yeah, we need a new cacti it seems :/ | 14:17 |
| anteaya | mwhahaha: hmmmm | 14:17 |
| clarkb | wznoinsk: adding that functionality to nodepool is probably not a bad idea but I am pretty sure ironic has said its not necessary | 14:17 |
| jroll | clarkb: wznoinsk: you can boot whole-disk qcow2 | 14:18 |
| wznoinsk | clarkb: ok, I'll dig the ironic side of things a bit further, thanks | 14:18 |
| mwhahaha | https://review.openstack.org/#/c/376055/ merged ~45 mins ago | 14:18 |
| anteaya | AJaeger: can you help here? it appears this patch may have broken puppets release notes jobs: https://review.openstack.org/#/c/352575/ | 14:18 |
| *** rlu has quit IRC | 14:18 | |
| mwhahaha | err didn't have a release notes on it | 14:18 |
| wznoinsk | jroll: ok, will test that on ironic then | 14:18 |
| mwhahaha | let me see if i can find something else | 14:18 |
| jroll | wznoinsk: probably want to use "local boot" because pxe booting user images is insanity, but otherwise it should just work :) | 14:19 |
| anteaya | AJaeger: they are now getting zuul cloner too few arguments errors, and the timing is matching up I'm told | 14:19 |
| mwhahaha | but looking at https://review.openstack.org/#/c/352575/3 i would suspect it's that | 14:19 |
| *** burgerk has joined #openstack-infra | 14:20 | |
| wznoinsk | jroll: when pxe booting (local boot off) it's only tftping kernel/ramdisk over lan, no fs image booting over network (it should already be on the local machine's disk if you use one) | 14:20 |
| anteaya | mwhahaha: is it just release notes jobs that are breaking for you? | 14:20 |
| mwhahaha | yes | 14:20 |
| jroll | wznoinsk: I'm aware, I still think it's insane :) | 14:20 |
| mwhahaha | all of them for all of our modules that have that job | 14:21 |
| kashyap | fungi: FYI - I just checked on #launchpad, they said: "not yet unfortunately", when I asked if they had an ETA about when it'd be back up again. | 14:21 |
| anteaya | mwhahaha: okay thanks | 14:21 |
| fungi | anteaya: clarkb: AJaeger: sdague: mwhahaha: let's revert 352575 for now. i'd rather not try to untangle release automation failures this close to release week | 14:21 |
| wznoinsk | jroll: depending on the scale, 50MB over 10GB network a few times a day doesn't do much harm, for bigger scale it's "virtual insanity" | 14:21 |
| anteaya | fungi: agreed | 14:21 |
| clarkb | fungi: wfm | 14:22 |
| sdague | fungi: wfm | 14:22 |
| sdague | fungi: is the revert up? | 14:22 |
| jroll | wznoinsk: well, it requires control plane to be up to be able to reboot and such. more of an opinion though, I won't argue about it :) | 14:22 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/project-config: Revert "Merge zuul-git-prep-upper-constraints/zuul-release-git-prep-upper-constraints" https://review.openstack.org/376535 | 14:22 |
| fungi | anteaya: clarkb: AJaeger: sdague: mwhahaha: ^ | 14:23 |
| njohnston | Hi! Quick question. This change got +A 80 minutes ago, but it looks like it never made it into Zuul for the Jenkins +2, I was just wondering if that might be related to any other current issues. | 14:23 |
| mwhahaha | thanks | 14:23 |
| *** rajinir has joined #openstack-infra | 14:23 | |
| openstackgerrit | Anita Kuno proposed openstack-infra/project-config: Revert "Merge zuul-git-prep-upper-constraints/zuul-release-git-prep-upper-constraints" https://review.openstack.org/376536 | 14:23 |
| njohnston | https://review.openstack.org/#/c/359320/ | 14:23 |
| wznoinsk | jroll: agree on those two points, let's leave it there ;-) | 14:23 |
| mwhahaha | njohnston: the depends on | 14:23 |
| fungi | anteaya: clarkb: AJaeger: sdague: mwhahaha: can one of you update the commit message there with details as to what broke? | 14:23 |
| anteaya | fungi: oh, okay, I'll abandon mine | 14:23 |
| openstackgerrit | Andrew Laski proposed openstack-infra/project-config: Run Nova post_test_hook on placement job https://review.openstack.org/376537 | 14:23 |
| fungi | anteaya: oh, i missed you created one | 14:23 |
| *** yamamoto has joined #openstack-infra | 14:23 | |
| njohnston | mwhahaha: Oh crap. Thanks! | 14:23 |
| fungi | anteaya: i'll abandon mine | 14:23 |
| fungi | anteaya: ah, added after mine. let's just stick with 376535 | 14:24 |
| fungi | i'm trying to follow too many conversations at once at the moment | 14:24 |
| anteaya | fungi: I understand, I abandoned mine and approveed yours | 14:24 |
| anteaya | thank you | 14:24 |
| *** roxanaghe has joined #openstack-infra | 14:24 | |
| fungi | or we can go with 376536 since it actually has details in the commit message | 14:24 |
| fungi | oh, okay | 14:24 |
| anteaya | I added some post approval | 14:25 |
| *** beekneemech is now known as bnemec | 14:25 | |
| *** amotoki has joined #openstack-infra | 14:25 | |
| anteaya | njohnston: can you not swear in here please? | 14:25 |
| njohnston | anteaya: My apologies | 14:25 |
| anteaya | thank you | 14:25 |
| *** Guest92615 is now known as zeus | 14:26 | |
| *** nadya has joined #openstack-infra | 14:26 | |
| *** zeus has quit IRC | 14:26 | |
| *** zeus has joined #openstack-infra | 14:26 | |
| rcarrillocruz | anteaya: if you want to put an item on the infra agenda for the cacti, go ahead, otherwise i can put it myself | 14:28 |
| cloudnull | mornings | 14:28 |
| anteaya | rcarrillocruz: an infra agenda item? I would hope we have a working cacti server before the end of today | 14:29 |
| anteaya | rcarrillocruz: do you not feel that is something you are able to help with today? | 14:29 |
| *** roxanaghe has quit IRC | 14:29 | |
| anteaya | cloudnull: good morning | 14:29 |
| cloudnull | o/ anteaya | 14:29 |
| cloudnull | how are you doing on this fine monday :) | 14:29 |
| anteaya | cloudnull: no fires like monday fires | 14:29 |
| rcarrillocruz | well, i'm not familiar with cacti to do a DB migration... so I rather wait for jeblair and other folks that have more experience with it first | 14:30 |
| anteaya | and yourself? | 14:30 |
| cloudnull | I love a good monday fire | 14:30 |
| cloudnull | :) | 14:30 |
| anteaya | rcarrillocruz: that is fair, can you perhaps find out from fungi what size server cacti should have now and spin one up? | 14:30 |
| cloudnull | I'm well, raining here in TX so I'm enjoying that. | 14:30 |
| anteaya | cloudnull: rain in texas? wow, how unusual | 14:31 |
| *** annegentle has joined #openstack-infra | 14:31 | |
| cloudnull | yup, and there's a flash flood warning in effect right now; though the rain has tappered off from this morning. | 14:32 |
| anteaya | given that cacti has stopped serving any data of any kind I have no idea what the size the current or prior cacti servers were | 14:32 |
| anteaya | the storm drains will take care of the flood will they not? | 14:32 |
| anteaya | or flood warning the storm drains can't handle | 14:32 |
| cloudnull | I'm not sure. | 14:32 |
| anteaya | :( | 14:33 |
| anteaya | be safe | 14:33 |
| *** mtanino has joined #openstack-infra | 14:33 | |
| *** akshai has quit IRC | 14:33 | |
| cloudnull | I live on a hill. so if it floods where I am there are bigger problems | 14:33 |
| cloudnull | :) | 14:33 |
| fungi | anteaya: rcarrillocruz: i'm adding a 4gb swapfile to the current cacti.o.o. hopefully that (1) gets it sort of working again and (2) will get us some idea how much ram it wants | 14:33 |
| anteaya | cloudnull: oh are you at home? | 14:33 |
| cloudnull | yes. | 14:33 |
| anteaya | fungi: thank you | 14:33 |
| anteaya | cloudnull: oh good, stay safe at home | 14:33 |
| cloudnull | indeed. bad weather == me working from home. | 14:33 |
| *** akshai has joined #openstack-infra | 14:34 | |
| *** nickola has quit IRC | 14:34 | |
| anteaya | cloudnull: glad to hear it | 14:34 |
| *** akshai has quit IRC | 14:35 | |
| zul | hmmm...is there some weirdness going on? http://logs.openstack.org/54/375654/4/check/gate-nova-lxd-pep8-ubuntu-xenial/116bd1e/console.html | 14:35 |
| *** vgusev has joined #openstack-infra | 14:35 | |
| anteaya | hey chuck | 14:35 |
| smcginnis | Anyone know what's happening wiht launchpad? | 14:36 |
| *** artmr has quit IRC | 14:36 | |
| anteaya | zul: this should fix it: https://review.openstack.org/#/c/376535/1 | 14:36 |
| cloudnull | zul: that doesn't look good. | 14:36 |
| anteaya | smcginnis: I was able to login, some other folks are reporting that launchpad is saying it is down | 14:37 |
| fungi | smcginnis: kashyap said that in #launchpad they're saying they don't know when they'll have it back up yet | 14:37 |
| anteaya | zul: after the fix merges then it will take some time to propigate, then recheck | 14:37 |
| fungi | anteaya: the "launchpad sso" is fine, launchpad the service is broken (they're separate things) | 14:37 |
| cloudnull | anteaya: does that mean all of the images will need to be updated? | 14:37 |
| smcginnis | anteaya, fungi: OK, thanks. Got a 500 on a script I tried to run, then web page just hung. | 14:37 |
| zul | ok thanks | 14:37 |
| anteaya | fungi: ah, thank you | 14:38 |
| anteaya | cloudnull: I don't think so, since this issue showed up about 45 minutes after the first project config change merged | 14:38 |
| cloudnull | ok. | 14:38 |
| anteaya | smcginnis: ah thank you | 14:38 |
| anteaya | cloudnull: good question | 14:38 |
| *** senk_ has quit IRC | 14:39 | |
| * cloudnull wants to try and capture bits when images are uploaded and the cache stampede happens | 14:39 | |
| anteaya | that issue took out everything in the gate, AJaeger sdague if you want to recheck patches once this propigates | 14:39 |
| anteaya | cloudnull: understandable | 14:40 |
| anteaya | cloudnull: I think pabelanger and clarkb are the folks to pay attention to there | 14:40 |
| anteaya | I know next to nothing about images and their caching | 14:40 |
| anteaya | rcarrillocruz: can you join #openstack-infra-incident please | 14:41 |
| rcarrillocruz | i cannot | 14:41 |
| rcarrillocruz | i have a call shortly | 14:41 |
| anteaya | okay | 14:41 |
| anteaya | so far nova, neutron and tempest patches are affected in the gate | 14:42 |
| *** andreww has quit IRC | 14:43 | |
| *** vgridnev has joined #openstack-infra | 14:43 | |
| vgridnev | hello infra team, is anyone faced with issue like that: http://logs.openstack.org/46/376446/1/gate/gate-sahara-docs-ubuntu-xenial/8bf8820/console.html ? | 14:44 |
| fungi | anteaya: rcarrillocruz: i was able to find a free window between memory exhaustion events to add that swapfile and reboot for good measure (since the oomkiller may have sacrificed running services too and it's better to make sure everything is properly restarted now) | 14:44 |
| *** esberglu has quit IRC | 14:44 | |
| anteaya | fungi: do you want to wait for this to finish testing or do you want to put it in now? https://review.openstack.org/#/c/376535/ it is taking out everything in gate and check | 14:44 |
| *** dosaboy has quit IRC | 14:44 | |
| anteaya | fungi: thank you | 14:45 |
| *** esberglu has joined #openstack-infra | 14:45 | |
| anteaya | vgridnev: is that zuul cloner too few arguments? if yes, we are working on the fix | 14:45 |
| fungi | anteaya: oh, i thought it was only breaking release notes jobs? how is it breaking everything in check and gate? | 14:45 |
| *** vgusev has left #openstack-infra | 14:45 | |
| vgridnev | anteaya, yep, ok, thanks | 14:46 |
| *** dosaboy has joined #openstack-infra | 14:46 | |
| fungi | i guess i should look at the errors shortly, but i'll take your word for it right now and manually merge it | 14:46 |
| *** esberglu has quit IRC | 14:46 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add 3 missing deb-python-xstatic for Horizon https://review.openstack.org/376551 | 14:46 |
| *** dosaboy has quit IRC | 14:46 | |
| vgridnev | fungi, for us docs are also broken | 14:46 |
| anteaya | fungi: the gate reset | 14:46 |
| anteaya | yeah docs jobs | 14:46 |
| *** isaacb has joined #openstack-infra | 14:46 | |
| anteaya | fungi: thank you | 14:46 |
| *** esberglu has joined #openstack-infra | 14:47 | |
| *** spzala has joined #openstack-infra | 14:48 | |
| openstackgerrit | Merged openstack-infra/project-config: Revert "Merge zuul-git-prep-upper-constraints/zuul-release-git-prep-upper-constraints" https://review.openstack.org/376535 | 14:48 |
| *** esberglu has quit IRC | 14:48 | |
| anteaya | thank you | 14:48 |
| vgridnev | anteaya, could you put announcement when fix is done (to avoid unneeded rechecks) | 14:48 |
| *** esberglu has joined #openstack-infra | 14:49 | |
| *** akshai has joined #openstack-infra | 14:49 | |
| anteaya | well the fix is in, and I'm going with fungi is ensuring zuul is updated | 14:49 |
| *** moravec has quit IRC | 14:49 | |
| *** bnemec has quit IRC | 14:49 | |
| anteaya | and then yeah I think a notice to the channels when recheck will work is a good idea | 14:50 |
| anteaya | thanks | 14:50 |
| *** florianf has quit IRC | 14:51 | |
| *** edtubill has joined #openstack-infra | 14:51 | |
| fungi | as for the actual bug, now that i've looked at the log it appears the config was missing a git_base_url parameter to zuul-cloner | 14:51 |
| *** mriedem_meeting is now known as mriedem | 14:51 | |
| anteaya | ah thank you | 14:52 |
| *** marst has joined #openstack-infra | 14:52 | |
| fungi | though actually what was happening was it was passing --branch and the variable for it (if any) expanded to an empty string so it thought the base url was actuallt the branch | 14:52 |
| *** dmsimard has quit IRC | 14:52 | |
| fungi | yeah, it was doing --branch TARGET_BRANCH | 14:52 |
| fungi | er, --branch $TARGET_BRANCH | 14:53 |
| fungi | and TARGET_BRANCH was "" | 14:53 |
| smcginnis | In case anyone else comes asking - lauchpad down due to network hardware issue. No ETA at the moment, but they are working on it. | 14:53 |
| fungi | or undefined maybe. anyway, resolved to an empty string | 14:53 |
| fungi | thanks smcginnis | 14:53 |
| anteaya | fungi: I've posted that as a comment to the patch | 14:54 |
| *** russellb has quit IRC | 14:54 | |
| *** russellb has joined #openstack-infra | 14:54 | |
| anteaya | smcginnis: thank you | 14:54 |
| *** armax has joined #openstack-infra | 14:54 | |
| smcginnis | :) | 14:54 |
| *** OferBarber_ has joined #openstack-infra | 14:54 | |
| *** jordanP has joined #openstack-infra | 14:54 | |
| fungi | anteaya: rcarrillocruz: looks like we have cacti data again | 14:54 |
| AJaeger | I'm sorry for the breakage. anteaya, fungi, sdague, thanks for fixing! | 14:54 |
| anteaya | fungi: that is helpful | 14:54 |
| Jeffrey4l | are u talking this "zuul-cloner -m /tmp/tmp.9pQgdUQD5D --cache-dir /opt/git --branch git://git.openstack.org openstack/requirements " | 14:55 |
| *** ijw has joined #openstack-infra | 14:55 | |
| *** tphummel has joined #openstack-infra | 14:55 | |
| anteaya | AJaeger: maybe in the future, adjusting release templates after release week? | 14:55 |
| fungi | AJaeger: well, we didn't fix it, just reverted it | 14:55 |
| Jeffrey4l | i saw the gate is busted. | 14:55 |
| Jeffrey4l | do we have any fix for this? | 14:55 |
| AJaeger | anteaya: the change itself was weeks old... | 14:55 |
| fungi | anteaya: to his credit, that patch had been out there for more than a month, sdague just happened to get approved this morning | 14:55 |
| *** slaweq_ has joined #openstack-infra | 14:56 | |
| fungi | er, happened to approve it | 14:56 |
| anteaya | Jeffrey4l: yes the fix has merged, waiting for it to propigate | 14:56 |
| Jeffrey4l | Got. thanks. | 14:56 |
| anteaya | AJaeger: ah, my mistake | 14:56 |
| * AJaeger hasn't read backscroll yet, just his notices... | 14:56 | |
| *** tonytan_brb has joined #openstack-infra | 14:56 | |
| *** moravec has joined #openstack-infra | 14:56 | |
| inc0 | hello, is review.openstack.org auth broken? | 14:56 |
| *** ArxCruz has joined #openstack-infra | 14:56 | |
| *** OferBarber has quit IRC | 14:57 | |
| *** yamamoto has quit IRC | 14:57 | |
| fungi | AJaeger: summary is it ended up doing --branch $TARGET_BRANCH with an empty TARGET_BRANCH | 14:57 |
| anteaya | inc0: what are you seeing? | 14:57 |
| *** daemontool_ has joined #openstack-infra | 14:57 | |
| inc0 | Provider is not supported, or was incorrectly entered. <- error and really ugly css-less form | 14:57 |
| fungi | inc0: anteaya: maybe the network hardware failure for launchpad has now started to impact launchpad sso? | 14:57 |
| *** mfisch has joined #openstack-infra | 14:57 | |
| anteaya | fungi: yes, that is what I'm wondering | 14:57 |
| fungi | that certainly sounds like how gerrit acts when it can't reach lp sso | 14:57 |
| inc0 | fungi, sounds plausable | 14:57 |
| mfisch | sounds like you guys know ^ | 14:58 |
| *** hogepodge has joined #openstack-infra | 14:58 | |
| jroll | I see similar on the wiki fwiw | 14:58 |
| *** bin_ has joined #openstack-infra | 14:58 | |
| anteaya | inc0: we are tracking a launchpad issue, but prior to now sso had been working | 14:58 |
| *** mjblack has left #openstack-infra | 14:58 | |
| *** mjblack has joined #openstack-infra | 14:58 | |
| fungi | if this persists for more than a few minutes i'll do a status alert until logins work again | 14:58 |
| *** bnemec has joined #openstack-infra | 14:58 | |
| *** esberglu has quit IRC | 14:58 | |
| anteaya | it is a hardware issue and smcginnis tells us they have no eta on a fix | 14:58 |
| inc0 | I'll keep checking | 14:58 |
| anteaya | fungi: sounds reasonable | 14:58 |
| jroll | and yeah, login.launchpad.net throwing 503 | 14:58 |
| anteaya | jroll: okay thank you for confirming | 14:58 |
| jroll | np :) | 14:59 |
| anteaya | sounds like we need to just skip today | 14:59 |
| anteaya | and try today again tomorrow | 14:59 |
| *** esberglu has joined #openstack-infra | 14:59 | |
| *** tonytan4ever has quit IRC | 14:59 | |
| jroll | ++ | 14:59 |
| fungi | since it was working until a few minutes ago, i'm willing to bet this is a brief outage resulting from their attempts to fix launchpad itself | 14:59 |
| inc0 | heh, good thing that it happened in one of quietest times in OpenStack | 14:59 |
| fungi | anteaya: it's a very monday monday, yeah | 14:59 |
| * anteaya hopes fungi's reality works | 14:59 | |
| smcginnis | Still saying no ETA on launchpad. Looks like it may take some time. | 14:59 |
| anteaya | fungi: 'tis | 14:59 |
| *** daemontool has quit IRC | 14:59 | |
| *** sdake_ has quit IRC | 14:59 | |
| anteaya | smcginnis: thank you :) | 14:59 |
| AJaeger | fungi, thanks for the info about pbr. | 14:59 |
| anteaya | smcginnis: appreciate you taking the monitoring on this one, thank you | 15:00 |
| *** akshai has quit IRC | 15:00 | |
| AJaeger | fungi, should I update the infra-manual change and show only 3 components? | 15:00 |
| smcginnis | anteaya: I feel bad for them watching the IRC. It's every few minutes someone coming on to the channel and asking. :) | 15:00 |
| anteaya | inc0: well it is release week | 15:00 |
| fungi | AJaeger: having examples that work with pbr sounds preferable, agreed. please do | 15:00 |
| anteaya | inc0: so timing isn't ideal even though patch load has decreased | 15:00 |
| *** ijw has quit IRC | 15:00 | |
| anteaya | smcginnis: yeah, thank goodness most of our hundreds read backscroll first :) | 15:01 |
| anteaya | yay our development community | 15:01 |
| fungi | smcginnis: i can sympathize with them, for sure ;) | 15:01 |
| *** gouthamr has quit IRC | 15:01 | |
| *** pots has joined #openstack-infra | 15:01 | |
| slaweq_ | mdbooth: hello | 15:01 |
| slaweq_ | I just saw Your comment in https://review.openstack.org/#/c/365756/4/nova/virt/libvirt/guest.py | 15:01 |
| smcginnis | fungi: Hah, no doubt. :) | 15:01 |
| *** akshai has joined #openstack-infra | 15:02 | |
| *** gouthamr has joined #openstack-infra | 15:02 | |
| slaweq_ | and I also thought that it's not proper to check if status != -1 there :) | 15:02 |
| slaweq_ | I will try to change it tomorrow I push new patch there | 15:02 |
| *** asettle has quit IRC | 15:02 | |
| slaweq_ | mdbooth: sorry, wrong channel :) | 15:02 |
| *** annegentle has quit IRC | 15:02 | |
| *** moravec has quit IRC | 15:03 | |
| *** annegentle has joined #openstack-infra | 15:03 | |
| *** martinkopec has quit IRC | 15:04 | |
| *** florianf has joined #openstack-infra | 15:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: drivers: Mention PEP 440 for tagging https://review.openstack.org/376211 | 15:06 |
| AJaeger | fungi ^ | 15:06 |
| fungi | AJaeger: thanks! | 15:06 |
| anteaya | this also took out bashate: http://logs.openstack.org/41/375441/1/gate/gate-devstack-bashate/885e92d/console.html | 15:07 |
| *** jlvillal has quit IRC | 15:07 | |
| *** rohanion has joined #openstack-infra | 15:07 | |
| anteaya | is a different infra root avaiable to help zuul reconfigure itself? | 15:07 |
| rohanion | hey guys | 15:07 |
| rohanion | we're facing a weird problem | 15:07 |
| anteaya | the fix is in but it hasn't updated for several hours | 15:07 |
| rohanion | http://logs.openstack.org/34/376034/2/check/gate-performance-docs-docs-ubuntu-xenial/657b619/console.html | 15:07 |
| anteaya | if it is zuul cloner too few argument we are aware | 15:07 |
| anteaya | also I'm not a guy | 15:07 |
| *** jlvillal has joined #openstack-infra | 15:08 | |
| rohanion | could you look at it please and tell why TARGET_BRANCH is empty? | 15:08 |
| rohanion | anteaya: sorry ^_^ | 15:08 |
| anteaya | the fix is in and I'm trying to get some assistance to get zuul cloner updated | 15:08 |
| *** moravec has joined #openstack-infra | 15:08 | |
| *** piet_ has joined #openstack-infra | 15:08 | |
| AJaeger | rohanion: please wait an hour and then recheck. I pushed a change that was broken, it was reverted and needs time to get deployed | 15:08 |
| *** zz_dimtruck is now known as dimtruck | 15:08 | |
| rohanion | thanks a lot! | 15:08 |
| *** psachin has quit IRC | 15:08 | |
| *** electrical_ has quit IRC | 15:08 | |
| *** xarses has joined #openstack-infra | 15:09 | |
| *** vgridnev has quit IRC | 15:09 | |
| *** esberglu has quit IRC | 15:09 | |
| *** dimtruck is now known as zz_dimtruck | 15:09 | |
| *** electrical_ has joined #openstack-infra | 15:09 | |
| *** xarses has quit IRC | 15:09 | |
| *** xarses has joined #openstack-infra | 15:09 | |
| fungi | anteaya: AJaeger: rohanion: the fix is in the process of being applied in the puppet run which started about 10 minutes ago, so unless we also just merged new project creation changes or anything that should take effect in moments | 15:10 |
| anteaya | fungi: ah thank you | 15:10 |
| rohanion | thaaanks~ | 15:10 |
| *** electrical_ has quit IRC | 15:10 | |
| *** electrical_ has joined #openstack-infra | 15:10 | |
| *** electrical_ has quit IRC | 15:10 | |
| *** electrical_ has joined #openstack-infra | 15:10 | |
| *** esberglu has joined #openstack-infra | 15:11 | |
| anteaya | <-- needs to find something to eat, back in a few | 15:11 |
| *** Vek has joined #openstack-infra | 15:12 | |
| *** esberglu has quit IRC | 15:12 | |
| *** yamamoto has joined #openstack-infra | 15:12 | |
| Vek | Is there a known issue with gerrit login? I'm getting redirected to a page saying that "Provider is not supported, or was incorrectly entered." | 15:13 |
| openstackgerrit | Ondřej Nový proposed openstack-infra/project-config: Added Swift related Debian packaging projects https://review.openstack.org/375851 | 15:13 |
| fungi | Vek: yes, the launchpad network problems seem to have spread to launchpad sso now as well | 15:13 |
| AJaeger | fungi, looking at http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/macros.yaml#n221 , I wonder how TARGET_BRANCH can be empty. What is wrong in that function? | 15:14 |
| *** electrical_ has quit IRC | 15:14 | |
| Vek | OK, so it's a network problem at LaunchPad. Good to know... | 15:14 |
| *** esberglu has joined #openstack-infra | 15:14 | |
| *** rbrndt has quit IRC | 15:14 | |
| anteaya | Vek: well it is a hardware problem at launchpad | 15:14 |
| *** claudiub has joined #openstack-infra | 15:14 | |
| *** electrical_ has joined #openstack-infra | 15:14 | |
| *** esikachev has quit IRC | 15:14 | |
| fungi | as soon as this job fix is in place, i'll do a #status notice about being ready for rechecks, and then i'll do a #status alert about launchpad sso in case it's still broken | 15:14 |
| Vek | cool. | 15:15 |
| anteaya | Vek: we are hoping that their efforts to restart the launchpad service are what have taken sso down for a few minutes | 15:15 |
| anteaya | fungi: thank you | 15:15 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid: Upgrade to LV 5.2 https://review.openstack.org/305521 | 15:15 |
| fungi | just trying not to conflate two unrelated issues with persistent irc status alerts | 15:15 |
| *** roxanaghe has joined #openstack-infra | 15:15 | |
| *** esikachev has joined #openstack-infra | 15:15 | |
| anteaya | AJaeger: can we focus on the fires today? so far we have launchpad, release jobs and cacti server | 15:15 |
| *** sdake has joined #openstack-infra | 15:15 | |
| anteaya | fungi: makes sense | 15:15 |
| fungi | anteaya: i suspect cacti is in good enough shape for now, though would definitely help to keep an eye on it | 15:16 |
| anteaya | fungi: okay so what is our next step there? new server? | 15:16 |
| *** Vek has left #openstack-infra | 15:17 | |
| AJaeger | anteaya: this is diagnosing what went wrong with my change. Or is there another issue with release jobs? | 15:17 |
| *** rbrndt has joined #openstack-infra | 15:17 | |
| AJaeger | anteaya: but we can do the post mortem later... | 15:17 |
| fungi | anteaya: yeah, we need to figure out how much ram it's actually wanting (by attempting to profile its swap utilization) and then make a bigger server that provides at least that much plus some room for growth | 15:17 |
| anteaya | AJaeger: yeah thanks | 15:17 |
| *** rohanion has quit IRC | 15:18 | |
| fungi | AJaeger: no other job-related failures i'm aware of no, just several unrelated issues have been going on | 15:18 |
| *** msmol has joined #openstack-infra | 15:18 | |
| *** dmsimard has joined #openstack-infra | 15:18 | |
| fungi | #status log A 4gb swapfile has been added on cacti.openstack.org at /swap while we try to work out what flavor its replacement should run | 15:18 |
| fungi | openstackstatus is missing in action too | 15:19 |
| *** msmol has quit IRC | 15:19 | |
| fungi | add that to the list of things that need debugging | 15:19 |
| fungi | i'll take a look | 15:19 |
| anteaya | thanks | 15:19 |
| *** dosaboy has joined #openstack-infra | 15:19 | |
| *** esberglu has quit IRC | 15:19 | |
| fungi | though the zuul jobs just updated | 15:19 |
| anteaya | yay | 15:19 |
| fungi | so we should be in the clear there | 15:19 |
| *** sandanar has joined #openstack-infra | 15:19 | |
| *** esberglu has joined #openstack-infra | 15:20 | |
| anteaya | my last reconfigured timestamp hasn't changed Last reconfigured: Mon Sep 26 2016 09:43:41 GMT-0400 (EDT) | 15:20 |
| *** slaweq_ has quit IRC | 15:21 | |
| anteaya | refreshed many times in chrome and even opened it in firefox | 15:21 |
| *** kvcobb has joined #openstack-infra | 15:21 | |
| clarkb | the last reconfigured timestamp is only for the pipeline config not the jjb configs | 15:21 |
| clarkb | If the only thing that changed was jjb I don't think we will see that timestamp update | 15:21 |
| anteaya | clarkb: thank you | 15:22 |
| anteaya | clarkb: and welcome | 15:22 |
| anteaya | clarkb: feeling less jetlagged? | 15:22 |
| clarkb | a little. Weekends help | 15:22 |
| fungi | correct, i was waiting to see the puppet updates applied on launchers in the ansible log because there was no accompanying layout change for the scheduler | 15:22 |
| anteaya | clarkb: glad to hear it | 15:23 |
| anteaya | fungi: ah thanks | 15:23 |
| *** electrical_ is now known as electrical | 15:23 | |
| anteaya | clarkb: feel like helping us figure out what size the new cacti server should be? | 15:23 |
| anteaya | I'm looking at http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=1&rra_id=all | 15:24 |
| openstackgerrit | Jordan Pittier proposed openstack/os-client-config: Fix AttributeError in `get_config` https://review.openstack.org/376570 | 15:24 |
| anteaya | and to my eyes it looks like cacti used to use 1M of swap quite regularly, when it had swap configured | 15:24 |
| anteaya | I think when we checked friday night no swap was configured when we switched servers from precise to trusty | 15:25 |
| fungi | no indication in statusbot's main or debug logs as to why it's not in any channels. i see logs from the last time it rejoined channels a couple of days ago. i'm just going to restart it | 15:25 |
| openstackgerrit | Isaac Beckman proposed openstack/diskimage-builder: Enable ssh password authentication https://review.openstack.org/376572 | 15:25 |
| anteaya | fungi: ack | 15:25 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/project-config: Fix zuul-release-git-prep-upper-constraints https://review.openstack.org/376573 | 15:25 |
| clarkb | fungi: there were a ton of issues with freenode over the weekend, I am going to guess that it just ended up in a corner and not part of the main network | 15:25 |
| fungi | anteaya: yeah, i think the cacti server got rebuilt without any swap, and was doing okay enough without any until we added infracloud | 15:26 |
| AJaeger | fungi, anteaya, I hope I found the problem - please review later 376573. | 15:26 |
| anteaya | I feel the same | 15:26 |
| anteaya | AJaeger: thank you | 15:26 |
| fungi | clarkb: yep, which is why if this doesn't work i'll have to start fiddling with the configured server | 15:26 |
| anteaya | AJaeger: I'll look at it later in the week | 15:26 |
| AJaeger | anteaya: ok | 15:26 |
| *** openstackstatus has joined #openstack-infra | 15:26 | |
| *** ChanServ sets mode: +v openstackstatus | 15:26 | |
| anteaya | AJaeger: after the release team has all the things they need | 15:27 |
| fungi | looks like that did it | 15:27 |
| anteaya | fungi: yay | 15:27 |
| amrith | fungi, it is stil broken | 15:27 |
| fungi | i'll wait for it to finish joining other channels before i start doing alerts | 15:27 |
| anteaya | amrith: is what? | 15:27 |
| *** moravec1 has joined #openstack-infra | 15:27 | |
| amrith | launchpad sso | 15:27 |
| anteaya | amrith: we have many broken things today | 15:27 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci: WIP: Multinode periodic tempest with no skips https://review.openstack.org/376237 | 15:27 |
| fungi | amrith: thanks for the update | 15:27 |
| fungi | #status log A 4gb swapfile has been added on cacti.openstack.org at /swap while we try to work out what flavor its replacement should run | 15:28 |
| clarkb | the docs jobs et al look happy again | 15:28 |
| amrith | fungi, more precise update. it is broken in las vegas as well. maybe we should get people to provide local updates :) kidding | 15:28 |
| anteaya | clarkb: thank you | 15:28 |
| anteaya | and yay working jobs | 15:28 |
| openstackstatus | fungi: finished logging | 15:28 |
| fungi | amrith: maybe you should be doing more fun things in las vegas ;) | 15:28 |
| clarkb | looks like cacti is a 1GB server instance now | 15:28 |
| clarkb | maybe just go up to 4GB which should give plenty of headroom | 15:28 |
| anteaya | clarkb: thank you | 15:29 |
| jeblair | pabelanger: no objection to cleaning up held nodes, thanks | 15:29 |
| anteaya | +1 4GB for new cacti | 15:29 |
| *** caowei has quit IRC | 15:29 | |
| jeblair | we didn't cacti cacti did we? | 15:29 |
| anteaya | we did | 15:29 |
| anteaya | and it stopped reporting anything over the weekend | 15:29 |
| *** moravec has quit IRC | 15:30 | |
| anteaya | well we only have 3 or 4 graphs from the cacti server | 15:30 |
| jeblair | i mean, we don't graph stats for cacti, right? | 15:30 |
| fungi | jeblair: cacti does some very basic localhost checks, but we could stand to have it treat itself as a "remote" server too for better data | 15:30 |
| jeblair | fungi: yeah, the localhost stuff is nearly useless, we should do the snmp checks | 15:31 |
| fungi | agreed. i was thinking about that earlier | 15:31 |
| fungi | just juggling too many things today | 15:31 |
| jeblair | it's currently a 1g host | 15:31 |
| jeblair | and, i guess it has run out of ram | 15:31 |
| fungi | right, that's our conclusion as well | 15:32 |
| jeblair | so it probably needs at least 2g? | 15:32 |
| anteaya | clarkb: had suggested 4 | 15:32 |
| dansmith | apologies if this has been asked a lot, but .. new openid login for gerrit this morning? | 15:32 |
| * dansmith didn't see anything on the dev list | 15:32 | |
| anteaya | dansmith: well not new that we know of but not working as reported by others | 15:32 |
| fungi | jeblair: probably, but since it was the addition of all the infracloud hosts at once which pushed it over, might be good to confirm 2gb will actually be sufficient | 15:33 |
| dansmith | anteaya: ah okay | 15:33 |
| jeblair | fungi, anteaya, clarkb: i could get on board with 4g | 15:33 |
| anteaya | dansmith: so far what we know is that launchpad is having hardware issues | 15:33 |
| dansmith | anteaya: so, I should not type my credentials into this weird login. yes? | 15:33 |
| anteaya | dansmith: correct, do not do that | 15:33 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/infra-manual: drivers: Mention PEP 440 for tagging https://review.openstack.org/376211 | 15:33 |
| fungi | anybody have a previously failing change they can recheck to make sure the revert has put things back on the right track, or has someone already? | 15:33 |
| anteaya | dansmith: we are hoping sso will come back soon, that is just went away as a consquence of them toggling their hardware | 15:33 |
| dansmith | anteaya: okay, and workarounds for being able to login? | 15:33 |
| dansmith | okay | 15:34 |
| AJaeger | fungi, now with even more details in 376211 - something for later to review, please | 15:34 |
| *** esikachev has quit IRC | 15:34 | |
| anteaya | dansmith: this is all conjecture on our part | 15:34 |
| dansmith | anteaya: ack | 15:34 |
| anteaya | dansmith: we have no workarounds for login | 15:34 |
| fungi | dansmith: no workarounds unless you have an api password set | 15:34 |
| jeblair | anteaya, fungi: it's entirely possible that people will be able to create accounts with random openid providers with gerrit in this state. | 15:34 |
| dansmith | fungi: I do, does that help me somehow? | 15:34 |
| anteaya | dansmith: we have been having a fire filled monday with only fungi at the helm until about 10 minutes ago | 15:34 |
| dansmith | anteaya: ack, sorry, will return to the sidelines | 15:34 |
| jeblair | (which, once it is fixed, they will not be able to log into) | 15:34 |
| fungi | dansmith: gertty for example is configured to authenticate rest api calls | 15:34 |
| fungi | dansmith: doesn't help if you want to use the webui though, no | 15:35 |
| openstackgerrit | Dan Smith proposed openstack-infra/project-config: Run nova's post_test_hook in the placement job for testing https://review.openstack.org/376585 | 15:35 |
| *** mhickey has quit IRC | 15:35 | |
| *** esberglu has quit IRC | 15:35 | |
| fungi | jeblair: good point, i'll give that a test | 15:35 |
| anteaya | jeblair: oh my | 15:35 |
| dansmith | fungi: okay | 15:35 |
| jeblair | fungi, anteaya, clarkb: perhaps we should consider using mod_rewrite to 500 /login/ | 15:35 |
| anteaya | dansmith: no no, not at all, just giving you a view of things | 15:36 |
| fungi | though i probably need to fire up a separate browser to test that | 15:36 |
| *** esberglu has joined #openstack-infra | 15:36 | |
| fungi | oh! i can test it on review-dev | 15:36 |
| clarkb | jeblair: that seems reasonable if indeed this is "exploitable" | 15:36 |
| fungi | without logging out of review.o.o | 15:36 |
| *** harlowja_at_home has joined #openstack-infra | 15:36 | |
| jeblair | clarkb: well, i just mean it will cause confusion, not like serious badness. | 15:36 |
| *** ccamacho is now known as ccamacho|away | 15:36 | |
| fungi | or so can anyone i guess. i'm going to get back to confirming jobs are working again first | 15:36 |
| anteaya | jeblair: yeah, I think we should control the 500 rather than have some other page pop up | 15:37 |
| jeblair | i may be lacking vocabulary. this morning. | 15:37 |
| jeblair | #status log added review.openstack.org to emergency disabled file | 15:37 |
| openstackstatus | jeblair: finished logging | 15:37 |
| rcarrillocruz | fungi: awesome, great news | 15:38 |
| anteaya | okay I'm staring at https://review-dev.openstack.org/#/q/status:open in firefox | 15:39 |
| clarkb | fungi: I have been keeping an eye on them and they seem to be happy again | 15:39 |
| *** mkoderer has quit IRC | 15:39 | |
| fungi | anteaya: you said docs jobs were broken previously? http://logs.openstack.org/36/375536/3/check/gate-neutron-docs-ubuntu-xenial/05c1a71/console.html looks like it succeeded | 15:39 |
| clarkb | fungi: change 375616 for example | 15:39 |
| openstackgerrit | Jordan Pittier proposed openstack-infra/project-config: Enable py35 voting on project os-client-config https://review.openstack.org/376597 | 15:39 |
| anteaya | I'm going with once jeblair's mod_rewrite change is in place I click sign in to check it | 15:39 |
| inc0 | hey, qq, who can add topic in project channels? | 15:39 |
| fungi | anteaya: so if that should have failed before, we're probably in the clear | 15:39 |
| fungi | inc0: which channel? | 15:39 |
| anteaya | fungi: oh good, also bashate for devstack | 15:40 |
| inc0 | #openstack-kolla | 15:40 |
| fungi | inc0: you should be able to query chanserv for that information | 15:40 |
| AJaeger | inc0: http://docs.openstack.org/infra/system-config/irc.html shows how to query chanserv | 15:41 |
| fungi | #status notice Earlier job failures for "zuul-cloner: error: too few arguments" should now be solved, and can safely be rechecked | 15:41 |
| openstackstatus | fungi: sending notice | 15:41 |
| anteaya | yay fungi, thank you | 15:41 |
| *** breton has joined #openstack-infra | 15:41 | |
| *** daniel3 is now known as Odd_Bloke | 15:41 | |
| AJaeger | thanks, fungi! | 15:41 |
| breton | hi! Is everything fine with gerrit and lp? I am getting "Provider is not supported, or was incorrectly entered." | 15:42 |
| anteaya | breton: no things are not fine | 15:42 |
| openstackgerrit | Jordan Pittier proposed openstack/os-client-config: List py35 in the default tox env list https://review.openstack.org/376600 | 15:42 |
| inc0 | AJaeger, fungi thanks, I'll also ask how to add +o to my nick;) | 15:42 |
| anteaya | they have a hardware issue with launchpad and that appears to be affecting sso | 15:42 |
| -openstackstatus- NOTICE: Earlier job failures for "zuul-cloner: error: too few arguments" should now be solved, and can safely be rechecked | 15:42 | |
| breton | ok, glad you know about it, thanks. | 15:42 |
| fungi | as soon as the status notice finishes, i'll do a status alert about the lp sso issues | 15:42 |
| anteaya | breton: we don't have an eta on a fix | 15:42 |
| anteaya | breton: thank you | 15:42 |
| anteaya | fungi: thanks | 15:42 |
| *** yamamoto has quit IRC | 15:44 | |
| openstackstatus | fungi: finished sending notice | 15:44 |
| fungi | there we go | 15:44 |
| fungi | #status alert Launchpad SSO is offline, preventing login to https://review.openstack.org/, https://wiki.openstack.org/ and many other sites; no ETA has been provided by the LP admin team | 15:44 |
| openstackstatus | fungi: sending alert | 15:44 |
| *** akshai has quit IRC | 15:44 | |
| inc0 | nvm figured it out, thanks guys! | 15:44 |
| anteaya | lennyb: thank you for stepping in and chairing the meeting on short notice | 15:45 |
| anteaya | lennyb: I appreciate it | 15:45 |
| harlowja_at_home | hmmm, come back launchpad SSO | 15:45 |
| *** jascott1_ has joined #openstack-infra | 15:45 | |
| *** yamamoto has joined #openstack-infra | 15:45 | |
| AJaeger | harlowja_at_home: a single-sign-on shouldn't be a single-point-of-failure ;) | 15:46 |
| *** jascott1_ has quit IRC | 15:46 | |
| anteaya | clicking sigin on review-dev gives me a 403: https://review-dev.openstack.org/login/%23%2Fq%2Fstatus%3Aopen | 15:46 |
| harlowja_at_home | AJaeger :-P | 15:46 |
| anteaya | jeblair: is that with your mod_rewrite change in place? | 15:46 |
| jeblair | anteaya: yeah, i'm working on review-dev | 15:46 |
| anteaya | awesome | 15:46 |
| -openstackstatus- NOTICE: Launchpad SSO is offline, preventing login to https://review.openstack.org/, https://wiki.openstack.org/ and many other sites; no ETA has been provided by the LP admin team | 15:47 | |
| jeblair | if someone can throw some text in an etherpad, we can put up a fail-whale webpage | 15:47 |
| *** ChanServ changes topic to "Launchpad SSO is offline, preventing login to https://review.openstack.org/, https://wiki.openstack.org/ and many other sites; no ETA has been provided by the LP admin team" | 15:47 | |
| *** adrian_otto has joined #openstack-infra | 15:47 | |
| *** thinrichs has joined #openstack-infra | 15:47 | |
| *** dprince has quit IRC | 15:47 | |
| lennyb | anteaya: np | 15:47 |
| openstackgerrit | Vladyslav Drok proposed openstack-infra/project-config: Skip slow ironic tests on install-guide changes https://review.openstack.org/376603 | 15:48 |
| clarkb | jeblair: "Ubuntu One/Launchpad OpenID servers are currently offline. We will notify the development community via IRC and email when this status changes. Sorry for the interruption." ? | 15:49 |
| jeblair | #status log gerrit login manually set to error page in apache config to avoid accidental account creation while lp sso is offline | 15:49 |
| jeblair | clarkb: https://review.openstack.org/login/ | 15:50 |
| openstackstatus | fungi: finished sending alert | 15:50 |
| openstackstatus | jeblair: finished logging | 15:50 |
| anteaya | my house is cold and I have to put on an actual fire, be back in about 30 minutes | 15:50 |
| clarkb | jeblair: lgtm | 15:50 |
| jeblair | clarkb, fungi: /home/gerrit2/review_site/static/broken.html | 15:50 |
| *** yamamoto has quit IRC | 15:50 | |
| *** moravec1 has quit IRC | 15:50 | |
| jeblair | also, our robots.txt was broken. | 15:50 |
| *** shardy has joined #openstack-infra | 15:51 | |
| openstackgerrit | Sean Dague proposed openstack-infra/devstack-gate: Remove tracing since ansible seems to be working https://review.openstack.org/178398 | 15:52 |
| *** rcernin has quit IRC | 15:52 | |
| openstackgerrit | Tony Xu proposed openstack/os-client-config: Update homepage with developer documentation page https://review.openstack.org/376605 | 15:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/puppet-gerrit: Add apache2.4 acls and permission to gerrit static dir https://review.openstack.org/376606 | 15:52 |
| jeblair | clarkb: ^ | 15:52 |
| *** tuannguyen has quit IRC | 15:53 | |
| *** nherciu has joined #openstack-infra | 15:53 | |
| jeblair | i have undone my changes on review-dev, so we can use it to test lp when back online | 15:53 |
| *** panda is now known as panda|bbl | 15:54 | |
| *** pgadiya has quit IRC | 15:54 | |
| *** annegentle has quit IRC | 15:55 | |
| *** tuannguyen has joined #openstack-infra | 15:55 | |
| *** nherciu_ has quit IRC | 15:55 | |
| *** moravec has joined #openstack-infra | 15:55 | |
| *** dosaboy has quit IRC | 15:56 | |
| *** dosaboy has joined #openstack-infra | 15:56 | |
| fungi | jeblair: in what way was our robots.txt broken? | 15:56 |
| *** ijw has joined #openstack-infra | 15:57 | |
| rcarrillocruz | jeblair , fungi : not sure if i missed something else on scrollback, happy to start working on cacti replacement if there's consensus on using 4GB flavor | 15:57 |
| *** isaacb has quit IRC | 15:57 | |
| *** akshai has joined #openstack-infra | 15:57 | |
| fungi | want to make sure i didn't accidentally crib brokenness from it onto https://wiki-dev.openstack.org/robots.txt | 15:57 |
| *** tuannguyen has quit IRC | 15:57 | |
| *** annegentle has joined #openstack-infra | 15:57 | |
| *** Sukhdev has joined #openstack-infra | 15:57 | |
| *** moravec has quit IRC | 15:58 | |
| clarkb | fungi: no perms to read the file by apache vhost | 15:58 |
| *** sandanar_ has joined #openstack-infra | 15:58 | |
| clarkb | fungi: see 376606 | 15:58 |
| fungi | aha | 15:58 |
| fungi | i guess it's fixed locally now since i can get to https://review.openstack.org/robots.txt | 15:58 |
| jeblair | fungi: yeah, i manually fixed it so i could serve broken.html | 15:59 |
| jeblair | fungi: but it should be re-broken on review-dev now | 15:59 |
| fungi | got it | 15:59 |
| fungi | and confirmed i get a 403 there | 15:59 |
| *** vinaypotluri has joined #openstack-infra | 15:59 | |
| jeblair | rcarrillocruz: ++4g | 15:59 |
| fungi | yeah, 4gb is probably plenty | 15:59 |
| jeblair | er, well, to be clear -- i like 4g. i did not mean to say "pre-increment 4g by one". | 16:00 |
| fungi | at worst infra-cloud maybe doubled the number of oids we're polling (very rough guess, don't actually have any research to back that up though) | 16:00 |
| *** sandanar has quit IRC | 16:00 | |
| *** moravec has joined #openstack-infra | 16:01 | |
| jeblair | fungi: ic has much more hardware (network/disk) and therefore more oids per-host, so it may actually be 'worse' :) | 16:01 |
| jhesketh | Anybody looking at the zuul-cloner failers? | 16:01 |
| clarkb | jhesketh: yup, fix should have gone in and jobs are happy again | 16:01 |
| *** berendt has quit IRC | 16:01 | |
| fungi | jhesketh: fixed about 45 minutes ago | 16:01 |
| clarkb | jhesketh: was a misconfigured jjb updated | 16:01 |
| jhesketh | ah cool, thanks | 16:01 |
| jeblair | 15:57 < axino> Launchpad should be back | 16:01 |
| jeblair | 15:57 < cjwatson> axino: Likely to stay up now? | 16:01 |
| jeblair | 16:00 < axino> cjwatson: likely, yes. | 16:01 |
| fungi | ooh! | 16:01 |
| * jhesketh doesn't keep up with scrollback sorry | 16:02 | |
| fungi | jhesketh: i status logged it too | 16:02 |
| fungi | well, status notice | 16:02 |
| jeblair | i will backout my local changes from review.o.o | 16:02 |
| jhesketh | fungi: right, I just saw the launchpad sso stuff, I should have checked the wiki | 16:02 |
| *** ijw has quit IRC | 16:02 | |
| fungi | jeblair: i just successfully logged in on review-dev | 16:03 |
| fungi | we, well almost anyway | 16:03 |
| jeblair | fungi: and i on review.o.o | 16:03 |
| fungi | i got through the lp sso steps | 16:03 |
| rcarrillocruz | jeblair, fungi : ok, i'll start working on spinning the node and configuring it today | 16:03 |
| *** lucasagomes is now known as lucas-hungry | 16:03 | |
| jeblair | fungi: from my pov, i think we're okay for 'status ok'; i'll let you further test and make the call | 16:04 |
| *** mriedem is now known as mriedem_away | 16:04 | |
| clarkb | I can write up mail for the dev list too | 16:04 |
| clarkb | will wait for status ok though | 16:04 |
| fungi | rcarrillocruz: though now that it has swap, it may not be urgent if we have a steady stream of stats again. could do like jeblair suggested and add cacti.o.o as a "remote" node in cacti configuration and start tracking its swap usage to see if a 4gb flavor will actually be sufficient | 16:04 |
| fungi | thanks clarkb! | 16:04 |
| rcarrillocruz | k | 16:05 |
| fungi | okay, confirmed now i'm _properly_ logged into review-dev | 16:06 |
| fungi | thanks jeblair! | 16:06 |
| openstackgerrit | Andrew Laski proposed openstack-infra/project-config: Run Nova post_test_hook on placement job https://review.openstack.org/376537 | 16:06 |
| fungi | #status ok Launchpad SSO logins are confirmed working correctly again | 16:06 |
| openstackstatus | fungi: sending ok | 16:06 |
| *** dprince has joined #openstack-infra | 16:07 | |
| *** andreas_s has quit IRC | 16:08 | |
| *** tphummel has quit IRC | 16:08 | |
| openstackgerrit | Ricardo Carrillo Cruz proposed openstack-infra/system-config: Monitor cacti in cacti itself https://review.openstack.org/376613 | 16:08 |
| *** mriedem_away has quit IRC | 16:08 | |
| breton | yey, works for me | 16:08 |
| *** mriedem has joined #openstack-infra | 16:08 | |
| *** mriedem is now known as mriedem_away | 16:08 | |
| rcarrillocruz | fungi: so, is localhost reporting different to 'cacti.openstack.org' ? | 16:09 |
| *** ChanServ changes topic to "Discussion of OpenStack Developer and Community Infrastructure | docs http://docs.openstack.org/infra/ | bugs https://storyboard.openstack.org/ | source https://git.openstack.org/cgit/openstack-infra/ | channel logs http://eavesdrop.openstack.org/irclogs/%23openstack-infra/" | 16:09 | |
| -openstackstatus- NOTICE: Launchpad SSO logins are confirmed working correctly again | 16:09 | |
| *** Swami has joined #openstack-infra | 16:09 | |
| rcarrillocruz | i mean, i see localhost on the UI, I understand it gets less stuff monitored as having it as an external node? | 16:09 |
| *** Sukhdev has quit IRC | 16:09 | |
| *** pilgrimstack has quit IRC | 16:09 | |
| fungi | rcarrillocruz: the "localhost" checks don't use snmp stats at all, and are very limited and rudimentary | 16:09 |
| rcarrillocruz | gotcha | 16:09 |
| *** yaume_ has joined #openstack-infra | 16:11 | |
| *** moravec has quit IRC | 16:11 | |
| openstackstatus | fungi: finished sending ok | 16:12 |
| fungi | launchpad itself appears to be working again too | 16:12 |
| clarkb | fungi: mail sent | 16:13 |
| fungi | thanks clarkb | 16:13 |
| fungi | the crazy today means i haven't caught up on the ml yet | 16:13 |
| *** david-lyle has quit IRC | 16:13 | |
| *** david-lyle has joined #openstack-infra | 16:14 | |
| *** yaume has quit IRC | 16:14 | |
| *** yaume has joined #openstack-infra | 16:14 | |
| pabelanger | zigo: yes, jessie-security, not still reprepro. Plan to test drive ftp-sync in the coming weeks | 16:15 |
| *** nickola has joined #openstack-infra | 16:16 | |
| *** daemontool_ has quit IRC | 16:17 | |
| *** yaume_ has quit IRC | 16:17 | |
| *** nickola has quit IRC | 16:17 | |
| *** makowals has joined #openstack-infra | 16:18 | |
| pabelanger | wow, fun backscroll | 16:19 |
| *** watersoul has quit IRC | 16:20 | |
| mdbooth | slaweq: Hey, you pinged (in the wrong channel ;)) | 16:20 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Remove cleanup directive https://review.openstack.org/329410 | 16:20 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Rename expand-groups to ansible-expand-groups https://review.openstack.org/326465 | 16:20 |
| openstackgerrit | Joshua Hesketh proposed openstack-infra/system-config: Tidy up ansible directories https://review.openstack.org/326464 | 16:20 |
| *** jordanP has quit IRC | 16:21 | |
| *** senk_ has joined #openstack-infra | 16:21 | |
| *** denisra has joined #openstack-infra | 16:22 | |
| *** moravec has joined #openstack-infra | 16:23 | |
| *** amotoki has quit IRC | 16:24 | |
| zaro | morning | 16:24 |
| *** e0ne has quit IRC | 16:25 | |
| anteaya | morning zaro | 16:25 |
| anteaya | fungi: have we fixed all the crazy? | 16:25 |
| anteaya | or at least got it tamped down? | 16:25 |
| anteaya | appears to be so from backscroll | 16:25 |
| anteaya | I also can log into review-dev now | 16:27 |
| *** drifterza has quit IRC | 16:27 | |
| zaro | good news, gerrit maintainers want to drop drafts feature. | 16:27 |
| anteaya | thanks for the email to the list clarkb | 16:27 |
| anteaya | zaro: yay! | 16:27 |
| clarkb | zaro: that is excellent news. No moreconfusion over that feature will be nice | 16:27 |
| fungi | anteaya: it's only smoldering now, i think. once lp was back up i needed to switch gears to check a bunch of pending updates to vulnerability reports | 16:27 |
| anteaya | fungi: sorry to bother you | 16:28 |
| *** armax has quit IRC | 16:28 | |
| fungi | anteaya: not a bother at all | 16:28 |
| *** matrohon has quit IRC | 16:28 | |
| anteaya | zaro: what version can we expect that change to be released in? | 16:28 |
| zaro | clarkb: yeah, it was announced at the end very end of the hackathon. | 16:28 |
| zaro | anteaya: it can already be turned off now and that's what we have done. so effectively it's not there for us | 16:29 |
| fungi | zaro: did they indicate there was any plan for another mechanism for indicating a change is not yet ready for review? | 16:29 |
| zaro | yeah, WIP | 16:29 |
| zaro | haha! | 16:29 |
| fungi | oh?!? | 16:29 |
| fungi | wow | 16:29 |
| *** pcaruana has quit IRC | 16:29 | |
| zaro | they finally realized openstack was correct about this whole thing | 16:30 |
| fungi | zaro: well, "turned off" makes it sound like there was a proper switch to disable drafts. we instead used acls to deny pushing to drafts refs, which results in obscure errors | 16:30 |
| anteaya | did they say that? | 16:30 |
| zaro | no, they didn't admit that. but actions do say that | 16:30 |
| anteaya | was wondering how far their language went | 16:30 |
| anteaya | but in any case, yay! | 16:31 |
| *** breton has left #openstack-infra | 16:31 | |
| zaro | so the plan is to add WIP and just keep allowing users to disable drafts until at a time when drafts can be completely removed. | 16:31 |
| *** jheroux has joined #openstack-infra | 16:31 | |
| zaro | WIP will need 2 things before drafts go aways. WIP on/off state and a WIP visibility acl | 16:32 |
| fungi | i don't care if they acknowledge that we figured this out years ago, i'll just be glad to see wip get some upstream/community momentum behind it so it's not just us looking like an odd duck | 16:32 |
| fungi | though Shrews_ might feel a little vindicated now ;) | 16:32 |
| zaro | the important point is that they are willing to accept adding WIP to core now | 16:32 |
| openstackgerrit | Paul Belanger proposed openstack/diskimage-builder: Create runtime-ssh-host-keys element https://review.openstack.org/374366 | 16:32 |
| *** jlanoux has quit IRC | 16:33 | |
| *** senk_ has quit IRC | 16:33 | |
| clarkb | right seems like an acceptance of the workflow that we have used whcih si great and at the same time removal of a feature that tends to only cause confusion | 16:33 |
| sslypushenko | Hi folks! Whom can I ask to re-trigger generate-constraints job? I put wrong version in to upper-constraints.txt in this patchset https://review.openstack.org/#/c/370293/ I guess next run of generate-constraints will fix it but looks like it will happen only in two days... | 16:33 |
| *** amitgandhinz has quit IRC | 16:33 | |
| fungi | clarkb: are you saying that the workflow we've used is great? or it's great that they've accepted our workflow? because i agree with both interpretations | 16:34 |
| *** amitgandhinz has joined #openstack-infra | 16:34 | |
| clarkb | fungi: both then :P | 16:34 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add 3 missing deb-python-xstatic for Horizon https://review.openstack.org/376551 | 16:34 |
| sdake | say are things on fire or is now a ogod time to make a feature request ;) | 16:34 |
| *** krtaylor has joined #openstack-infra | 16:34 | |
| zaro | not really important for us but they want wip changes to be visible to specific users like how drafts works now. | 16:34 |
| anteaya | sdake: we are post fire lull, so it depends on how much attention you want | 16:35 |
| sdake | anteaya not alot | 16:35 |
| sdake | just something to float in your minds | 16:35 |
| anteaya | folks are not trying to catch up on the things they couldn't do before | 16:35 |
| clarkb | zaro: meaning not visible to everyone or just more visible to a subset of people? | 16:35 |
| anteaya | you are welcome to post | 16:35 |
| anteaya | s/are not/are now | 16:35 |
| *** tphummel has joined #openstack-infra | 16:35 | |
| sdake | I notice a pattern acrosss openstack that the mor red in a gerrit review (on the left side indicating the length of the patch) the less likely it is to be reviewed | 16:36 |
| sdake | unfortunately many of these patches are critical | 16:36 |
| *** dizquierdo has quit IRC | 16:36 | |
| anteaya | sslypushenko: you might want to try the #openstack-requirements channel to see if anyone has any advice for you there | 16:36 |
| sdake | it sure would be nice if gerrit showed bug severity in the review list if there was one | 16:36 |
| anteaya | sdake: offer a patch to gerrit's gui | 16:37 |
| sdake | left side/right side ;) | 16:37 |
| sdake | anteaya zug- don't know anthing about ui programming | 16:37 |
| zaro | clarkb: both, in general visibility of any change should be configurable. you can't configure visibility of changes now. | 16:37 |
| sdake | just a suggestion for others to follow up on ;) | 16:37 |
| anteaya | well I don't think anyone in here has time to be honest | 16:37 |
| sdake | roger anteaya | 16:38 |
| anteaya | nor have I heard anyone else voice your concern, but you never know | 16:38 |
| sdake | anteaya i feel the same wy | 16:38 |
| fungi | sdake: gerrit doesn't have any understanding of bug details beyond being able to display a hyperlink. we have some hooks in place right now to get bugs updated from gerrit events. also it's worth noting that at least with storyboard we're moving away from a notion of global bug priority to individual bug priority | 16:38 |
| sslypushenko | anteaya: according to README https://github.com/openstack/requirements#for-upper-constraintstxt-changes re-triggering update of upper-constraints should be asked in infra channel) | 16:38 |
| fungi | sdake: likely reviewday is what you're thinking of though | 16:38 |
| anteaya | sslypushenko: okay, I don't know who can do that for you | 16:38 |
| sslypushenko | But I'll ask #openstack-requirements too) Thank you | 16:39 |
| anteaya | sslypushenko: your welcome | 16:39 |
| zaro | clarkb: that is much more involved feature than just making wip visible to all or only change owner. | 16:39 |
| sdake | fungi could you expand on storyboard? | 16:39 |
| *** harlowja_at_home has quit IRC | 16:39 | |
| sdake | the global priority vs individual priority part | 16:39 |
| fungi | sdake: http://status.openstack.org/reviews/ actually munges together bug severity and change linkages to give you some sense of priority | 16:39 |
| zaro | too bad it was only announced at the end of the hackathon otherwise i would have started on it already. | 16:39 |
| anteaya | zaro: thanks for sharing the news now | 16:40 |
| fungi | sdake: launchpad bug priority has so far mostly been a vague feel-good knob but isn't an accurate reflection of task prioritization. storyboard has kanban-like worklists and you can link tasks from stories into them | 16:40 |
| sdake | fungi is that integrated into storyboard in some way? | 16:40 |
| *** yaume has quit IRC | 16:40 | |
| sdake | (th eabove reviewday lilnk) | 16:41 |
| fungi | sdake: so you could create for example a kolla team tasklist and prioritize story tasks in it, or you could have a personal tasklist with your own priorities and people could subscribe to it too | 16:41 |
| fungi | sdake: no, reviewday and storyboard were separate comments i was making | 16:41 |
| *** Apoorva has joined #openstack-infra | 16:41 | |
| sdake | cool sounds like the problem is being solved to me | 16:42 |
| sdake | thanks fungi | 16:42 |
| fungi | sdake: reviewday is what we have now to get an overview of review priority taking bug criticality into account. storyboard is solving things in a different way | 16:42 |
| sdake | i'll have to dig more into storyboard and reviewday | 16:42 |
| sdake | i was awaare of storyboard | 16:42 |
| sdake | not aware of reviewday | 16:42 |
| *** ijw has joined #openstack-infra | 16:43 | |
| sdake | fungi re reviewday - is there a spot documented how to add a project to it? | 16:43 |
| fungi | sdake: some teams have fairly recently added links for prioritized gerrit review dashboards into reviewday as well | 16:43 |
| *** amitgandhinz has quit IRC | 16:43 | |
| *** amitgandhinz has joined #openstack-infra | 16:44 | |
| *** mriedem_away has quit IRC | 16:44 | |
| fungi | sdake: i'm not super familiar with reviewday, but hopefully others here can give you pointers on it | 16:44 |
| sdake | fungi is it accurate to say that reeviewday takes launchpad priority into account as well as action on the patchset to determine itss priority? | 16:44 |
| pabelanger | pleia2_: the twitter bot is excellent | 16:45 |
| *** shardy has quit IRC | 16:45 | |
| fungi | sdake: yes, and also linked lp blueprints i think | 16:45 |
| *** irtermit- is now known as irtermite | 16:45 | |
| sdake | fungi hotness | 16:45 |
| *** panda|bbl is now known as panda | 16:45 | |
| sdake | fungi that will serve well :) | 16:45 |
| sdake | fungi any points on who i should request info from to get kolla on the list? | 16:46 |
| sdake | or which repo its in? | 16:46 |
| *** xarses has quit IRC | 16:46 | |
| fungi | sdake: i'm digging to find where the config lives now | 16:46 |
| sdake | if you have a repo i cna figure out th rest | 16:46 |
| sdake | thanks fungi | 16:46 |
| *** xarses has joined #openstack-infra | 16:46 | |
| *** ijw has quit IRC | 16:48 | |
| *** dprince has quit IRC | 16:50 | |
| *** jpena is now known as jpena|off | 16:51 | |
| *** akshai has quit IRC | 16:52 | |
| *** Rockyg has joined #openstack-infra | 16:52 | |
| *** akshai has joined #openstack-infra | 16:53 | |
| *** derekh has quit IRC | 16:53 | |
| *** pleia2_ is now known as pleia2 | 16:53 | |
| openstackgerrit | Merged openstack-infra/devstack-gate: Use local AFS mirror for network_sanity_check https://review.openstack.org/318714 | 16:54 |
| *** moravec1 has joined #openstack-infra | 16:54 | |
| *** erlon has joined #openstack-infra | 16:56 | |
| *** amitgandhinz has quit IRC | 16:56 | |
| *** moravec has quit IRC | 16:57 | |
| *** lucas-hungry is now known as lucasagomes | 16:57 | |
| pleia2 | pabelanger: oh good, nice to see it broke the long message into 2 tweets | 16:57 |
| fungi | pleia2: i'm trying to figure out how to add projects to reviewday and so far it's pretty impenetrable... do you perchance any pointers? | 16:58 |
| pabelanger | pleia2: Ya, I seen that | 16:58 |
| *** akshai has quit IRC | 16:58 | |
| fungi | i've been digging in the reviewday repo, project-config, system-config, puppet-reviewday... | 16:58 |
| *** ijw has joined #openstack-infra | 16:59 | |
| *** akshai has joined #openstack-infra | 17:00 | |
| clarkb | fungi: its hardcoded in reviewday/bin/reviewday | 17:00 |
| fungi | clarkb: oh wow, so it is! | 17:01 |
| fungi | i thought you were pulling my leg for a moment | 17:01 |
| fungi | sdake: https://git.openstack.org/cgit/openstack-infra/reviewday/tree/bin/reviewday#n31 | 17:01 |
| *** senk_ has joined #openstack-infra | 17:02 | |
| fungi | i skimmed recent commits to the repo and didn't see any project-specific additions other than the neutron dashboard stuff, which i figured was entirely separate | 17:02 |
| sdake | fungi thanks dude apprecite it ;) | 17:02 |
| fungi | i should have skimmed back further | 17:02 |
| *** yamamoto has joined #openstack-infra | 17:02 | |
| fungi | i guess the last "addition" was two years ago when marconi was renamed | 17:02 |
| *** trown is now known as trown|lunch | 17:03 | |
| *** electrichead is now known as redrobot | 17:03 | |
| *** gyee has joined #openstack-infra | 17:05 | |
| *** ijw has quit IRC | 17:05 | |
| openstackgerrit | Steven Dake proposed openstack-infra/reviewday: Add kolla deliverables to reviewday https://review.openstack.org/376685 | 17:05 |
| *** dprince has joined #openstack-infra | 17:07 | |
| *** amitgandhinz has joined #openstack-infra | 17:07 | |
| *** jaosorior has quit IRC | 17:07 | |
| *** esikachev has joined #openstack-infra | 17:09 | |
| *** jpich has quit IRC | 17:10 | |
| *** tqtran has joined #openstack-infra | 17:11 | |
| *** yamamoto has quit IRC | 17:11 | |
| *** devananda|bbl is now known as devananda | 17:11 | |
| *** xarses has quit IRC | 17:12 | |
| rcarrillocruz | fungi: i'm a bit torn with https://review.openstack.org/#/c/376613/ and https://review.openstack.org/#/c/369357/ | 17:12 |
| *** xarses has joined #openstack-infra | 17:12 | |
| rcarrillocruz | should I hold on on adding chocolate till we get more data on cacti swap usage? | 17:12 |
| rcarrillocruz | otherwise it will be hard to compare if i throw more stress to the host by adding more computes | 17:12 |
| *** armax has joined #openstack-infra | 17:13 | |
| *** Guest68174 is now known as med_ | 17:14 | |
| *** med_ has quit IRC | 17:14 | |
| *** med_ has joined #openstack-infra | 17:14 | |
| clarkb | fungi: any idea why http://logs.openstack.org/74/372374/17/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/6bbbcae/logs/devstacklog.txt.gz#_2016-09-26_17_00_36_961 has happened? I believe the list on the right is the valdi set of names from the cert and one of them does match the left hand side so that seems odd | 17:15 |
| *** thinrichs has left #openstack-infra | 17:15 | |
| clarkb | rcarrillocruz: it shouldn't take long for that right? the swap is in place now and has been for a few hours so once we get cacti cactiing itself we should be happy a few hours after that as far as data collection goes? | 17:15 |
| rcarrillocruz | ok, if folks are good with that amount of data, i can get the chocolate rolling in a couple hours | 17:16 |
| anteaya | in terms of priority my vote is on cacti ahead of additional infra cloud nodes | 17:17 |
| fungi | yeah, i agree let's get some (quick) trending on cacti for a few hours and then decide what size replacement it needs | 17:17 |
| *** kgiusti has left #openstack-infra | 17:18 | |
| *** ArxCruz has quit IRC | 17:18 | |
| rcarrillocruz | and roll chocolate after few hours or after new cacti replacement ? | 17:18 |
| *** ArxCruz has joined #openstack-infra | 17:19 | |
| *** rossella_s has quit IRC | 17:19 | |
| *** ArxCruz has quit IRC | 17:19 | |
| *** rossella_s has joined #openstack-infra | 17:19 | |
| *** ArxCruz has joined #openstack-infra | 17:19 | |
| fungi | rcarrillocruz: if we're only barely using swap then we can probably add more nodes to the config and build the replacement in parallel | 17:20 |
| *** ArxCruz has quit IRC | 17:20 | |
| rcarrillocruz | ++ | 17:20 |
| clarkb | ugh looks like a new urllib3 release today is what may be breaking us | 17:20 |
| *** ArxCruz has joined #openstack-infra | 17:20 | |
| fungi | clarkb: oh, did this just spontaneously start? | 17:20 |
| clarkb | fungi: ya this morning and urllib3 just released 1.18 :( | 17:21 |
| *** ArxCruz has quit IRC | 17:21 | |
| fungi | i was looking into potential known behaviors instead (and not very successfully) | 17:21 |
| clarkb | fungi: it was working yesterday | 17:21 |
| pabelanger | clarkb: fungi: are we okay using c0f71f7 for glean 1.7.0? Was going to try and tag a release for rax-iad fixes. | 17:21 |
| fungi | okay, so yes this sounds exceedingly likely | 17:21 |
| *** ArxCruz has joined #openstack-infra | 17:21 | |
| clarkb | pabelanger: let me check git log | 17:21 |
| *** ArxCruz has quit IRC | 17:21 | |
| *** ArxCruz has joined #openstack-infra | 17:22 | |
| *** watersoul has joined #openstack-infra | 17:22 | |
| *** ArxCruz has quit IRC | 17:22 | |
| clarkb | pabelanger: ya I tested that one and it worked locally | 17:22 |
| fungi | clarkb: pabelanger: skimming i don't see any obvious backward incompatibilities introduced since 1.6.1, so should probably be fine as 1.7.0 | 17:22 |
| clarkb | pabelanger: also I think you said you did testing on your side as well | 17:22 |
| *** ArxCruz has joined #openstack-infra | 17:23 | |
| *** gaurangt has joined #openstack-infra | 17:23 | |
| *** ArxCruz has quit IRC | 17:23 | |
| *** Na3iL has quit IRC | 17:23 | |
| *** gaurangt has left #openstack-infra | 17:23 | |
| *** ArxCruz has joined #openstack-infra | 17:23 | |
| clarkb | sigmavirus: if you are around any chance you can help track down this urllib3 issue? | 17:23 |
| pabelanger | clarkb: fungi: Yes, build new images for rax-iad and uploaded. Worked well | 17:23 |
| *** ArxCruz has quit IRC | 17:24 | |
| *** ArxCruz has joined #openstack-infra | 17:24 | |
| clarkb | pabelanger: no objection from me then | 17:25 |
| *** ArxCruz has quit IRC | 17:25 | |
| clarkb | sigmavirus: specifically http://logs.openstack.org/74/372374/17/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/6bbbcae/logs/devstacklog.txt.gz#_2016-09-26_17_00_36_961 just started today and there is a new urllib3 release. was working yesterday | 17:25 |
| *** ArxCruz has joined #openstack-infra | 17:25 | |
| anteaya | clarkb: sorry what is the error that is symptomatic of urlib3 1.18? | 17:25 |
| clarkb | anteaya: the log link I just linked | 17:25 |
| anteaya | oh you just said | 17:25 |
| anteaya | sorry | 17:25 |
| *** ArxCruz has quit IRC | 17:26 | |
| *** ArxCruz has joined #openstack-infra | 17:26 | |
| *** ArxCruz has quit IRC | 17:26 | |
| *** ArxCruz has joined #openstack-infra | 17:27 | |
| clarkb | I can work on a bisect in a bit but goign to eat breakfast first | 17:27 |
| *** ArxCruz has quit IRC | 17:27 | |
| pabelanger | clarkb: fungi: 1.7.0 pushed | 17:27 |
| *** ArxCruz has joined #openstack-infra | 17:27 | |
| *** ArxCruz has quit IRC | 17:28 | |
| *** drawsmcgraw has joined #openstack-infra | 17:28 | |
| fungi | clarkb: i'm not seeing any obvious issues filed for that yet, so bisecting is probably in order | 17:28 |
| fungi | i don't have enough context to put together a minimal reproducer anyway | 17:28 |
| *** ArxCruz has joined #openstack-infra | 17:28 | |
| *** ArxCruz has quit IRC | 17:29 | |
| *** ArxCruz has joined #openstack-infra | 17:29 | |
| *** ArxCruz has quit IRC | 17:30 | |
| *** harlowja has joined #openstack-infra | 17:30 | |
| *** ArxCruz has joined #openstack-infra | 17:30 | |
| *** pvaneck has joined #openstack-infra | 17:30 | |
| *** ArxCruz has quit IRC | 17:30 | |
| *** ArxCruz has joined #openstack-infra | 17:31 | |
| *** ArxCruz has quit IRC | 17:31 | |
| *** ArxCruz has joined #openstack-infra | 17:31 | |
| sigmavirus | clarkb: looking | 17:32 |
| *** ArxCruz has quit IRC | 17:32 | |
| sigmavirus | clarkb: ah I think I know what work caused this to break | 17:32 |
| *** ArxCruz has joined #openstack-infra | 17:32 | |
| *** e0ne has joined #openstack-infra | 17:33 | |
| *** ArxCruz has quit IRC | 17:33 | |
| *** ArxCruz has joined #openstack-infra | 17:33 | |
| *** ArxCruz has quit IRC | 17:34 | |
| *** pvaneck_ has joined #openstack-infra | 17:34 | |
| *** HeOS has quit IRC | 17:34 | |
| *** ArxCruz has joined #openstack-infra | 17:34 | |
| *** ArxCruz has quit IRC | 17:34 | |
| *** ArxCruz has joined #openstack-infra | 17:35 | |
| *** tosky has quit IRC | 17:35 | |
| *** ArxCruz has quit IRC | 17:35 | |
| sigmavirus | clarkb: do you happen to know if you inject pyopenssl for urllib3 or not? | 17:35 |
| clarkb | sigmavirus: I dont. not sure what that means | 17:36 |
| clarkb | sigmavirus: if you scrollup a bit you cansee the installation logs | 17:36 |
| *** pvaneck has quit IRC | 17:37 | |
| *** flepied has joined #openstack-infra | 17:37 | |
| sigmavirus | Right I see you're installing pyOpenSSL and idna and was wondering if you're using pyOpenSSL in urllib3 or not. There should be code that literally calls `urllib3.contrib.pyopenssl.inject_into_urllib3()` | 17:38 |
| *** nadya has quit IRC | 17:39 | |
| sigmavirus | ah that's a different installation section | 17:39 |
| clarkb | mtreinish: ^ do you know? tempest is the thing using urllib3 here | 17:40 |
| clarkb | sigmavirus: git grep says no | 17:42 |
| *** salv-orlando has joined #openstack-infra | 17:43 | |
| sigmavirus | clarkb: is it possible to take a look at the cert generated for keystone here? | 17:43 |
| sigmavirus | also this is 2.7 on 14.04, yes? | 17:43 |
| clarkb | sigmavirus: I can restart the tests and hold the instance then get the cert | 17:44 |
| clarkb | sigmavirus: no its 2.7 on 16.04 | 17:44 |
| dmsimard | Does openstack-infra VMs in-gate use Google's DNS servers (i.e, 8.8.8.8, 8.8.4.4) for DNS resolution ? | 17:44 |
| *** dtantsur is now known as dtantsur|afk | 17:44 | |
| clarkb | dmsimard: they run a local unbound server that is caching and forwarding to google (except for rax iirc) | 17:44 |
| clarkb | but rax should be made to be like the others, this is a bug | 17:44 |
| dmsimard | It looks like there may have been transient/intermittent failures with them today, at least it looks like I'm not crazy and there's some anecdotal evidence on twitter about it. | 17:45 |
| sigmavirus | clarkb: can you point me at the code that generates the cert? (Assuming that's something we know where it lives) | 17:45 |
| *** tonytan_brb is now known as tonytan4ever | 17:45 | |
| *** Shrews_ is now known as Shrews | 17:45 | |
| clarkb | sigmavirus: devstack/lib/tls | 17:45 |
| anteaya | dmsimard: that would fit with the rest of our day so far | 17:45 |
| clarkb | sigmavirus: let me get you a link | 17:45 |
| anteaya | dmsimard: thanks for sharing | 17:45 |
| dmsimard | anteaya: ah, you've received reports of DNS issues ? | 17:45 |
| sigmavirus | clarkb: let me scan that first then if necessary we can hold an instance | 17:45 |
| clarkb | sigmavirus: ok | 17:45 |
| *** dosaboy_ has joined #openstack-infra | 17:45 | |
| *** degorenko is now known as _degorenko|afk | 17:45 | |
| *** akshai has quit IRC | 17:45 | |
| anteaya | dmsimard: no | 17:46 |
| clarkb | sigmavirus: https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/tls | 17:46 |
| anteaya | dmsimard: but we lost launchpad and sso earlier today | 17:46 |
| dmsimard | anteaya: ah, perhaps indirectly related. Okay. | 17:46 |
| anteaya | dmsimard: and had a patch take out our tests until we reverted | 17:46 |
| Shrews | fungi: zaro: sean finally accepting of WIP?? hah. i do not care anymore | 17:46 |
| clarkb | sigmavirus: make_cert is the specific function in that file that makes the cert | 17:46 |
| *** rbrndt has quit IRC | 17:46 | |
| anteaya | dmsimard: not related at all | 17:46 |
| *** dosaboy_ has quit IRC | 17:46 | |
| anteaya | dmsimard: perfect storm of unrelated things | 17:46 |
| *** dosaboy has quit IRC | 17:46 | |
| dmsimard | anteaya: by indirectly related I meant that perhaps that's why launchpad/SSO went K.O. :) | 17:47 |
| anteaya | ah, now I see | 17:47 |
| anteaya | and perhaps you are correct | 17:47 |
| *** pilgrimstack has joined #openstack-infra | 17:47 | |
| anteaya | however we heard reports of hardware failures | 17:47 |
| *** dosaboy has joined #openstack-infra | 17:47 | |
| anteaya | as the cause of launchpad issues | 17:47 |
| dmsimard | I tried to find confirmed reports of Google DNS issues but I can't find any official source .. | 17:47 |
| anteaya | but I didn't follow anything up | 17:47 |
| anteaya | dmsimard: thank you | 17:47 |
| zaro | Shrews: David B. is now the main guy on gerrit now and he's the one that thinks drafts should go away. | 17:48 |
| *** esikachev has quit IRC | 17:49 | |
| sigmavirus | clarkb: Okay, I should have a fix for devstack/lib/tls but I want to know if you want this to work on older versions of urllib3 as well or not | 17:49 |
| sigmavirus | Actually, I guess it might as well work on everything, right? | 17:49 |
| clarkb | sigmavirus: I think ideally it would work with both yes. Though if thats difficult we can roll forward on it | 17:49 |
| sigmavirus | clarkb: not difficult at all | 17:49 |
| sigmavirus | Just wanted to get it as close to "right" on the first try as possible :) | 17:49 |
| zaro | so actually not Shawn. but the other maintainers agree with David so there's consensus | 17:50 |
| clarkb | sigmavirus: let me give you a change id to depends on when you push your patch to lib/tls to make sure it will get tested | 17:50 |
| Rockyg | dmsimard, from downdetector.com: Google is having issues since 11:32 AM EDT. Are you also affected? Leave a message in the comments. | 17:50 |
| sigmavirus | clarkb: sounds good | 17:50 |
| *** mriedem has joined #openstack-infra | 17:50 | |
| *** esikachev has joined #openstack-infra | 17:50 | |
| clarkb | sigmavirus: https://review.openstack.org/372374 you should actually edit that change to depends on your change | 17:50 |
| clarkb | then if the jobs to 372374 work we know its fixed. Also what is the problem? | 17:51 |
| clarkb | (if you want me to figure out dependencies I can do that once the code is pushed) | 17:51 |
| Shrews | zaro: i'm just glad that i'm not the one that has to code that again | 17:52 |
| sigmavirus | clarkb: so here's the problem, urllib3's hostname matching became more correct (and hence more strict) and so we need to specify the TLS IP as both DNS: and IP: | 17:52 |
| sigmavirus | I can update 372374 for that | 17:52 |
| sigmavirus | Is there a bug for this clarkb? | 17:52 |
| clarkb | sigmavirus: oh is it only listed as a name rather than an IP and since its an ip addr not in dns its mad? | 17:52 |
| sigmavirus | Yeah | 17:53 |
| clarkb | sigmavirus: no bug yet I just started noticing it | 17:53 |
| jeblair | Shrews: ++ you submitted some lovely code; they can dust it off if they want :) | 17:53 |
| *** EricGonczer_ has joined #openstack-infra | 17:53 | |
| clarkb | sigmavirus: no don't update 372374 as thats a throwaway change that cannot merge that just turns on tls in all devstack runs for testing | 17:53 |
| clarkb | sigmavirus: I am very close to having it enabled in a subset of devstack runs but need this fixed first :) once that happens testing this will be less crazy with the dep trees | 17:53 |
| zaro | Shrews: yeah, just push as-is to let them know that openstack tried to introduce long time ago :) | 17:54 |
| *** flepied has quit IRC | 17:54 | |
| Shrews | https://github.com/davido/gerrit-wip-plugin | 17:54 |
| *** e0ne has quit IRC | 17:54 | |
| zaro | Shrews: that's a broken attempt that came after yours :) | 17:54 |
| sigmavirus | clarkb: so https://review.openstack.org/376983 should fix it | 17:55 |
| *** EricGonczer_ has quit IRC | 17:55 | |
| clarkb | sigmavirus: thanks I will adjust the dep tree and we should find out in a little bit | 17:55 |
| *** panda is now known as panda|dinner | 17:55 | |
| Shrews | zaro: that was work based on my code to make it a plugin (to hopefully get it accepted) | 17:55 |
| Shrews | but it's there if someone wants it | 17:56 |
| *** piet_ has quit IRC | 17:56 | |
| *** cdent has quit IRC | 17:56 | |
| *** esikachev has quit IRC | 17:56 | |
| clarkb | sigmavirus: 372374 has been updated with the correct depends on (I just realized I could git tree that and I didn't but the end resutl should be the same as far as testing goes) | 17:56 |
| zaro | Shrews: that attempted to reuse draft status which doesn't work queit right. there's are many problems with that one. | 17:57 |
| Shrews | meh | 17:57 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Ironic: Replace *_ssh jobs with *_ipmitool https://review.openstack.org/376990 | 17:57 |
| clarkb | sigmavirus: re pyopenssl injection would we have seen these failures earlier if we were injecting pyopenssl? | 17:58 |
| *** amoralej is now known as amoralej|off | 17:58 | |
| sigmavirus | clarkb: re pyopenssl, no. but there were separate bugs in 1.17 with pyopenssl | 17:59 |
| * sigmavirus wanted to rule that out before digging down rabbit holes | 17:59 | |
| clarkb | gotcha | 17:59 |
| *** piet_ has joined #openstack-infra | 17:59 | |
| clarkb | sigmavirus: thank you for jumping on that we are so very close to being able to test this on every change just need this least set of changes to get in then turn on tls in the jobs | 18:00 |
| *** e0ne has joined #openstack-infra | 18:00 | |
| sigmavirus | clarkb: no worries. Thank you for enabling TLS in devstack and at the gate | 18:00 |
| *** mat128 is now known as mat128|mtg | 18:01 | |
| *** srobert has quit IRC | 18:01 | |
| *** spzala has quit IRC | 18:01 | |
| *** EricGonczer_ has joined #openstack-infra | 18:01 | |
| *** spzala has joined #openstack-infra | 18:02 | |
| *** spzala_ has joined #openstack-infra | 18:03 | |
| *** csomerville has joined #openstack-infra | 18:05 | |
| *** spzala has quit IRC | 18:07 | |
| *** spzala_ has quit IRC | 18:08 | |
| *** cody-somerville has quit IRC | 18:08 | |
| *** sambetts is now known as sambetts|afk | 18:09 | |
| *** trown|lunch is now known as trown | 18:10 | |
| *** AnarchyAo has joined #openstack-infra | 18:10 | |
| clarkb | sigmavirus: maybe also the error message in urllib3 should be updated too? the message as is is confusing because 127.0.0.1 is in the list that it says it was looking at | 18:10 |
| *** spzala has joined #openstack-infra | 18:11 | |
| *** ijw has joined #openstack-infra | 18:11 | |
| sigmavirus | Right | 18:11 |
| sigmavirus | So this is code that's backported from Python 3 I think that we're relying on | 18:11 |
| sigmavirus | And that message is unhelpfully generated by that code | 18:11 |
| *** ihrachys has quit IRC | 18:12 | |
| clarkb | ah | 18:12 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Convert tripleo-container jobs into multinode https://review.openstack.org/377000 | 18:13 |
| *** ijw has quit IRC | 18:14 | |
| *** ijw has joined #openstack-infra | 18:14 | |
| *** esikachev has joined #openstack-infra | 18:14 | |
| *** ijw has quit IRC | 18:15 | |
| anteaya | yes having the error say 127.0.0.1 doesn't match 127.0.0.1 made me wonder what was going on as well | 18:15 |
| *** ijw has joined #openstack-infra | 18:17 | |
| *** markvoelker_ has joined #openstack-infra | 18:18 | |
| *** tuannguyen has joined #openstack-infra | 18:19 | |
| *** markvoelker has quit IRC | 18:19 | |
| *** rook_ is now known as rook | 18:19 | |
| *** hongbin has quit IRC | 18:20 | |
| *** markvoelker has joined #openstack-infra | 18:21 | |
| *** ijw has quit IRC | 18:21 | |
| *** EricGonczer_ has quit IRC | 18:23 | |
| *** dane-fichter has joined #openstack-infra | 18:23 | |
| *** markvoelker_ has quit IRC | 18:24 | |
| dhellmann | infra-core: I'm seeing errors in the release tagging job because launchpad lib is missing from the node. We fixed this once before, I think, but I don't know what was done to make the fix. A puppet change? http://logs.openstack.org/d4/d43245a42b8f8442e76a39e260da06dc6d5bd2b7/release-post/tag-releases/1f717c2/console.html#_2016-09-26_18_12_37_589866 | 18:24 |
| *** ijw has joined #openstack-infra | 18:24 | |
| *** vhosakot has joined #openstack-infra | 18:24 | |
| clarkb | dhellmann: I think we may have fixed it on proposal.slave but this ran on signing01.ci | 18:25 |
| dhellmann | clarkb : ah | 18:25 |
| clarkb | we may just need to apply the fix from the one to the other /me looks | 18:25 |
| dhellmann | clarkb : thanks | 18:25 |
| openstackgerrit | Dan Smith proposed openstack-infra/project-config: Run Nova post_test_hook on placement job https://review.openstack.org/376537 | 18:25 |
| fungi | clarkb: dhellmann: i fixed it on the signing node | 18:25 |
| fungi | guessing the release node also runs some similar tasks? | 18:26 |
| clarkb | hrm ya I see python-launchpadlib is in the manifest | 18:26 |
| clarkb | fungi: it failed on signing01 according to the log dhellmann posted | 18:26 |
| fungi | huh, i'll look too | 18:26 |
| *** EricGonczer_ has joined #openstack-infra | 18:26 | |
| fungi | d'oh! | 18:26 |
| fungi | wait | 18:26 |
| rcarrillocruz | clarkb , fungi : can you please review https://review.openstack.org/#/c/376613/ so we start collecting things for cacti hosts ? | 18:26 |
| clarkb | and aptitude says its installed | 18:27 |
| fungi | nope, new enough lp should be working again too | 18:27 |
| clarkb | rcarrillocruz: +2 | 18:27 |
| fungi | and yeah, it's an importerror | 18:28 |
| *** peterlisak has quit IRC | 18:28 | |
| *** onovy has quit IRC | 18:28 | |
| rcarrillocruz | thx | 18:28 |
| clarkb | fungi: dhellmann I just ran import launchpadlib.launchpad and it worked on that host | 18:28 |
| *** panda|dinner is now known as panda | 18:29 | |
| clarkb | I am fairly stumped | 18:29 |
| dhellmann | clarkb, fungi: weird | 18:31 |
| *** sandanar_ has quit IRC | 18:31 | |
| clarkb | ok best guess is something is shadowing that name | 18:32 |
| *** sandanar has joined #openstack-infra | 18:32 | |
| clarkb | though it looks like that job cds into all the repo dirs before running that command and I doubt all of those repos area all shadowing it | 18:33 |
| openstackgerrit | Rodrigo Duarte proposed openstack-infra/project-config: Enable security compliance feature https://review.openstack.org/377008 | 18:33 |
| dhellmann | ooh, we had 2 zuul nodes a while back, didn't we? | 18:33 |
| dhellmann | oh, you meant the library, not the host | 18:33 |
| clarkb | wya | 18:34 |
| clarkb | *ya though it may be the case the host is also shadowed/ | 18:34 |
| *** akshai has joined #openstack-infra | 18:34 | |
| clarkb | fungi: ^ | 18:34 |
| *** e0ne has quit IRC | 18:36 | |
| *** EricGonczer_ has quit IRC | 18:36 | |
| mat128|mtg | Anyone free for a quick zuul layout change for skipping (most) tests on install-guide changes? https://review.openstack.org/#/c/376603/ | 18:36 |
| *** bin_ has quit IRC | 18:37 | |
| *** akshai has quit IRC | 18:37 | |
| *** rbrndt has joined #openstack-infra | 18:38 | |
| *** akshai has joined #openstack-infra | 18:38 | |
| fungi | clarkb: i don't think we have an additional signing01.ci.o.o, but double-checking now | 18:38 |
| *** moravec1 has quit IRC | 18:39 | |
| *** e0ne has joined #openstack-infra | 18:39 | |
| fungi | clarkb: yeah, confirmed only one signing01.ci.o.o according to nova | 18:40 |
| fungi | and our inventory | 18:40 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: run launchpad update script from temp directory https://review.openstack.org/377015 | 18:40 |
| dhellmann | clarkb, fungi : just in case ^^ | 18:40 |
| *** moravec has joined #openstack-infra | 18:40 | |
| clarkb | I am pretty stumped then. Doesn't look like we use tox/virtualenv and if I run /usr/bin/env python and then import launchpadlib.launchpad on that host it works | 18:41 |
| *** nemish has left #openstack-infra | 18:41 | |
| *** Marx314 has joined #openstack-infra | 18:41 | |
| *** Marx314 has quit IRC | 18:41 | |
| openstackgerrit | Timothy R. Chavez proposed openstack-infra/jenkins-job-builder: Add support for the random string parameter https://review.openstack.org/351384 | 18:42 |
| *** nadya has joined #openstack-infra | 18:43 | |
| *** Marx314 has joined #openstack-infra | 18:43 | |
| *** akshai has quit IRC | 18:43 | |
| *** nadya has quit IRC | 18:44 | |
| mtreinish | clarkb: we don't explicitly do anything with pyopenssl that I know of in tempest | 18:44 |
| *** pilgrimstack has quit IRC | 18:44 | |
| *** dane-fichter has quit IRC | 18:44 | |
| clarkb | pip also sees the installation so I don't think it somehow got broken | 18:45 |
| *** liusheng has quit IRC | 18:45 | |
| clarkb | mtreinish: thanks, I think sigmavirus was able to sort it out. Waiting on tests now | 18:45 |
| *** liusheng has joined #openstack-infra | 18:45 | |
| sigmavirus | yeah I also realized afterwards that I read the wrong list of dependencies that included pyopenssl so I'm welcome | 18:46 |
| fungi | clarkb: dhellmann: also dpkg-log doesn't indicate the python-launchpadlib package was being updated at that time or anything | 18:47 |
| openstackgerrit | Lucas Alvares Gomes proposed openstack-infra/project-config: Ironic: Replace *_ssh jobs with *_ipmitool https://review.openstack.org/376990 | 18:47 |
| fungi | er, /var/log/dpkg.log doesn't i mean | 18:47 |
| AJaeger | team, do we want to update osic cloud1 to 1024 nodes? then let's merge https://review.openstack.org/375696 | 18:47 |
| *** dtardivel has quit IRC | 18:47 | |
| irtermite | cloudnull: ^^ | 18:47 |
| anteaya | AJaeger: fungi clarkb and I chatted about it on friday and we are okay with it | 18:47 |
| fungi | dhellmann: is this a consistent failure, or did it only happen once? | 18:47 |
| anteaya | AJaeger: I did raise the concern about what that would mean regarding testing on the rest of our hosts | 18:48 |
| dhellmann | fungi : I only noticed it this once. The job actually ignores failures to update launchpad, though, because it's not considered "critical" so I don't know how long it has been going on. I can check some of the other jobs from today. | 18:48 |
| anteaya | AJaeger: and was reassured the algorithm will continue to allocate on all host clouds | 18:48 |
| AJaeger | anteaya: so, merge it now? | 18:48 |
| anteaya | AJaeger: I have no objeciton | 18:48 |
| anteaya | the agorithm allocates based on proportion of quota | 18:49 |
| clarkb | fungi: I wonder if this was somehow related to the LP issues a cuople hours ago | 18:49 |
| fungi | dhellmann: at least another correlation that it's happened more than once would be good... though also with launchpad being broken earlier today you may end up seeing other failures | 18:49 |
| clarkb | fungi: timestamps say no but maybe we hit something there | 18:49 |
| fungi | clarkb: well, it ran well after | 18:49 |
| AJaeger | fungi, clarkb, then I'll +2A the change to raise the cloud1 quota now | 18:50 |
| dhellmann | fungi : the job before that one was fine, but the job after that one also had the import error: http://logs.openstack.org/f3/f37cf3a761d343e9a697738950ab050a93867a2a/release-post/tag-releases/131de0a/console.html | 18:50 |
| fungi | clarkb: ohhh... like maybe the launchpad package has some code it runs at init before the module is successfully indexed? | 18:50 |
| *** adrian_otto has quit IRC | 18:50 | |
| fungi | AJaeger: sounds good | 18:50 |
| clarkb | fungi: ya lots of libs do that in python :/ | 18:50 |
| clarkb | AJaeger: ok | 18:50 |
| AJaeger | done | 18:50 |
| dhellmann | fungi , clarkb : is it possible this is a path, or PYTHONPATH, issue? | 18:50 |
| clarkb | dhellmann: I was looking at the ansible losg and I don't think so, but maybe | 18:50 |
| clarkb | http://logs.openstack.org/d4/d43245a42b8f8442e76a39e260da06dc6d5bd2b7/release-post/tag-releases/1f717c2/_zuul_ansible/scripts/0eeecf2bd2b7426398a9e66dc8a43a83.sh doesn't set python path for example | 18:51 |
| fungi | if we override pythonpath (seems unlikely) then it could be | 18:51 |
| *** vhosakot has quit IRC | 18:51 | |
| anteaya | AJaeger: thanks for asking | 18:51 |
| dhellmann | I don't think I'm doing anything like that in any of the scripts I've written for this stuff | 18:51 |
| *** ccamacho|away has left #openstack-infra | 18:51 | |
| anteaya | AJaeger: i too thought it was worthy of discussion | 18:51 |
| anteaya | :) | 18:51 |
| clarkb | I also checked that it was using python2 and not 3 | 18:51 |
| *** ianychoi has joined #openstack-infra | 18:52 | |
| *** dprince has quit IRC | 18:52 | |
| AJaeger | pabelanger, could you delete http://grafana.openstack.org/dashboard/db/networking-ovn-failure-rate , please? It got renamed to http://grafana.openstack.org/dashboard/db/networking-ovn-failure-rates | 18:52 |
| AJaeger | anteaya: yeah - thanks | 18:52 |
| * dhellmann makes a note to talk to the infra team about porting the release tools to python 3 | 18:52 | |
| anteaya | :) | 18:52 |
| fungi | dhellmann: also make a note that python-launchpadlib doesn't seem to support py3k | 18:53 |
| *** cdent has joined #openstack-infra | 18:53 | |
| *** dprince has joined #openstack-infra | 18:53 | |
| fungi | dhellmann: at least on trusty | 18:53 |
| fungi | the python-launchpadlib package doesn't install into the python 3 search path, and there's no separate python3-launchpadlib package either | 18:53 |
| *** tuannguyen has quit IRC | 18:53 | |
| dhellmann | fungi : yeah, I'll need to look at our dependencies first | 18:54 |
| pabelanger | AJaeger: sure, let me see why it wasn't deleted | 18:54 |
| dhellmann | the stuff in release-tools might be a better early candidate | 18:54 |
| clarkb | sigmavirus: http://logs.openstack.org/74/372374/18/check/gate-tempest-dsvm-platform-fedora24-nv/d8c3801/logs/devstacklog.txt.gz#_2016-09-26_18_53_04_606 it is still failing but I think that is because we need to add 127.0.0.1 to the list as an IP as well | 18:54 |
| fungi | dhellmann: it's possible, as clarkb said, that the actual exception (maybe an endpoint test?) is being caught and hidden behind the importerror | 18:55 |
| dhellmann | let me look at that lib and see if it does anything like that | 18:56 |
| fungi | in which case perhaps the problem has already been resolved and that's why our manual testing now doesn't recreate it | 18:56 |
| *** moravec has quit IRC | 18:56 | |
| *** senk_ has quit IRC | 18:56 | |
| fungi | might be able to recreate it by setting up a bad entry for the api server in /etc/hosts | 18:56 |
| dhellmann | can you give me the version for the lib on that node? | 18:56 |
| *** vhosakot has joined #openstack-infra | 18:57 | |
| clarkb | sigmavirus: I can push a new patchset if you don't mind | 18:57 |
| fungi | dhellmann: 1.10.2 | 18:57 |
| sigmavirus | clarkb: oh if you already have it ready, go for it | 18:57 |
| fungi | dhellmann: from the python-launchpadlib 1.10.2+ds-2 package in trusty | 18:57 |
| dhellmann | fungi : thanks | 18:58 |
| fungi | dhellmann: be prepared for a bzr repo or retrieving a source deb | 18:58 |
| clarkb | sigmavirus: hrm well now its tricky because I think its SERVICE_HOST that is set to 127.0.0.1 which may or may not be an IP address | 18:58 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Move ENV var setting to a common function https://review.openstack.org/377035 | 18:59 |
| clarkb | sigmavirus: and we seem to deal with SERVICE_HOST twice which is weird so probably some general cleanup we can do | 18:59 |
| fungi | dhellmann: oh! they actually published it to pypi. how un-ubuntu | 18:59 |
| dhellmann | I think we complained to them a while back? Or maybe someone else complained. | 18:59 |
| openstackgerrit | Rodrigo Duarte proposed openstack-infra/project-config: Enable security compliance feature https://review.openstack.org/377008 | 19:00 |
| *** rbrndt has quit IRC | 19:00 | |
| dhellmann | I don't see anything obviously indicating that importing the library is hitting the API | 19:00 |
| mtreinish | fungi: it's been on pypi for a few years at least. I remember being surprised by that way back when jogo and I started elastic-recheck | 19:00 |
| sigmavirus | clarkb: I wonder if there's a good way to check if SERVICE_HOST is an IP or not | 19:01 |
| fungi | dhellmann: yeah, i also tested setting api.launchpad.net to an unreachable ip address in /etc/hosts | 19:01 |
| *** vhosakot has quit IRC | 19:01 | |
| fungi | still no importerror | 19:01 |
| clarkb | sigmavirus: I think we can hack something togetehr with a regex maybe | 19:01 |
| clarkb | looking to see if devstack already has this sort of utility function though | 19:01 |
| sigmavirus | clarkb: I have some gnarly regexp's in rfc3986 | 19:02 |
| AJaeger | jeblair: is https://review.openstack.org/371341 fine? That adds missing root markers for afs publishing | 19:02 |
| dhellmann | fungi : is there some way to retrigger one of these jobs so we can try it again? they should be idempotent w.r.t. the actual tags | 19:02 |
| fungi | dhellmann: yes, i can do that. just a sec | 19:02 |
| dhellmann | before you do that let me make sure it will run this step if the branch exists | 19:02 |
| fungi | okay | 19:02 |
| dhellmann | yes, it does. a while back it used to skip the remaining steps, but we took that out | 19:03 |
| mtreinish | pleia2, nibalizer, fungi: btw, mosquitto is down again, so someone probably need to go in and restart it. Also there was something we were supposed to check about logrotate to confirm it's the cause, but I can't remember what that was | 19:03 |
| fungi | mtreinish: whether it created a new logfile, or if the last log entry was the reload message in the rotated log | 19:03 |
| openstackgerrit | Merged openstack-infra/project-config: Update the max-servers for cloud1 to 1024 https://review.openstack.org/375696 | 19:04 |
| mtreinish | fungi: ah, right | 19:04 |
| clarkb | sigmavirus: is it an error to have something like IP:foo.bar.com in your cert? | 19:04 |
| clarkb | sigmavirus: as thats the other way we could work around this | 19:04 |
| *** moravec has joined #openstack-infra | 19:04 | |
| sigmavirus | clarkb: let me ask a different question: are we going to use IPv6 anytime soon? | 19:05 |
| anteaya | mtreinish: you still have the mostquito killer magic | 19:05 |
| nibalizer | nn | 19:05 |
| * nibalizer dances | 19:05 | |
| * anteaya applaudes | 19:05 | |
| clarkb | sigmavirus: ipv6 is another thing I would like to see us push more for though its not currently possible in our multinode case because we can't run tunnels over it | 19:06 |
| jeblair | AJaeger: +3 | 19:06 |
| AJaeger | thanks, jeblair | 19:06 |
| sigmavirus | clarkb: so we could be practical and just use a regexp for ipv4, eh? | 19:06 |
| clarkb | sigmavirus: ya I think so | 19:06 |
| anteaya | AJaeger: thank you | 19:07 |
| sigmavirus | I just don't like the idea of having devstack generate a cert with a non-IP address value in the ipAddress SAN | 19:07 |
| sigmavirus | because while tempest/python clients are the primary testing targets, other tools might use it too | 19:07 |
| *** sandanar has quit IRC | 19:07 | |
| *** ijw_ has joined #openstack-infra | 19:07 | |
| fungi | dhellmann: i'm all set to retrigger the job with the details from http://logs.openstack.org/d4/d43245a42b8f8442e76a39e260da06dc6d5bd2b7/release-post/tag-releases/1f717c2/_zuul_ansible/playbook whenever you give me the go ahead | 19:07 |
| *** adrian_otto has joined #openstack-infra | 19:07 | |
| clarkb | sigmavirus: tahts true | 19:07 |
| clarkb | let me cobble together a cheap " | 19:08 |
| clarkb | is this ipv4 address" check | 19:08 |
| dhellmann | fungi : fire when ready | 19:08 |
| fungi | dhellmann: logs should appear at http://logs.openstack.org/d4/d43245a42b8f8442e76a39e260da06dc6d5bd2b7/release-post/tag-releases/1f717c2,byhand0/ when complete | 19:09 |
| dhellmann | fungi : ack, thanks | 19:09 |
| *** AlexxNica has joined #openstack-infra | 19:09 | |
| *** ijw has quit IRC | 19:09 | |
| mtreinish | anteaya: heh, well I'm not really doing anything special. I just try to use it :) | 19:09 |
| openstackgerrit | Merged openstack-infra/system-config: Monitor cacti in cacti itself https://review.openstack.org/376613 | 19:10 |
| *** gongysh has quit IRC | 19:11 | |
| *** gongysh has joined #openstack-infra | 19:11 | |
| *** gongysh has quit IRC | 19:11 | |
| sigmavirus | clarkb: I have one of those | 19:12 |
| fungi | mtreinish: no such luck i'm afraid... last logged entry was "New connection from 127.0.0.1 on port 1883." at 11:51:58 utc today | 19:12 |
| fungi | mtreinish: so this one doesn't seem to have been related to log rotation at all | 19:12 |
| fungi | mtreinish: [Mon Sep 26 11:51:57 2016] traps: mosquitto[26916] general protection ip:7f1cfb5ba072 sp:7ffdcd87d1b0 error:0 in libwebsockets.so.7[7f1cfb5a5000+1f000] | 19:13 |
| sigmavirus | clarkb: is a lazy one https://github.com/sigmavirus24/rfc3986/blob/master/rfc3986/misc.py#L87 | 19:13 |
| anteaya | mtreinish: seems to be enough | 19:13 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: run launchpad update script from temp directory https://review.openstack.org/377015 | 19:13 |
| clarkb | sigmavirus: super lazy :P | 19:14 |
| fungi | mtreinish: i also see the same in dmesg from Fri Sep 23 06:25:00 2016, Wed Sep 21 06:25:01 2016 and Wed Sep 21 02:54:47 2016 | 19:14 |
| mtreinish | fungi: sigh, so we're tripping a bug in libwebsockets | 19:14 |
| fungi | mtreinish: i started it back up for now | 19:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/shade: Add simple field for disabled flavors https://review.openstack.org/377049 | 19:15 |
| mtreinish | fungi: that's harder to fix | 19:15 |
| fungi | mtreinish: amusing that the one before this was triggered during log rotation, so may have been related to something the signal handler was doing to open sockets? | 19:15 |
| fungi | mtreinish: other possibility is that the gp there was death throes after the daemon began to collapse from something else unrelated | 19:16 |
| fungi | mtreinish: so maybe the bug in websockets is a secondary one that only occurs during teardown, and the primary bug crashing the service is just leaving us no other trace of its presence | 19:17 |
| *** e0ne has quit IRC | 19:17 | |
| fungi | mtreinish: we could probably get this set up to core dump with an appropriate wrapper? | 19:18 |
| dhellmann | fungi : same error: http://logs.openstack.org/d4/d43245a42b8f8442e76a39e260da06dc6d5bd2b7/release-post/tag-releases/1f717c2,byhand0/console.html#_2016-09-26_19_09_41_272278 | 19:18 |
| fungi | dhellmann: indeed, this is baffling | 19:18 |
| mtreinish | fungi: hmm, yeah any of those could be a possiblity. it's hard to know | 19:19 |
| fungi | dhellmann: i could temporarily stop puppet for this node, add some debug statements into the script and then trigger again | 19:19 |
| dhellmann | fungi : yeah, that seems like a good plan | 19:19 |
| dhellmann | I was going to suggest letting me write a little probe script, too | 19:19 |
| mtreinish | fungi: yeah it should be too hard to wrap the call to start mosquitto with something | 19:19 |
| openstackgerrit | Merged openstack-infra/project-config: Add missing root-marker for afs-publishing https://review.openstack.org/371341 | 19:20 |
| fungi | dhellmann: i'll try to get the output of env and the sys.path from python | 19:20 |
| mtreinish | fungi: it might be easier to just do it in the systemd unit file for mosquitto | 19:20 |
| AJaeger | wow, 325 jobs in the post queue - looks mainly like deb repos | 19:20 |
| dhellmann | fungi : python -v -c 'import launchpadlib.launchpad' | 19:20 |
| fungi | dhellmann: oh, that's a fine place to start, sure | 19:20 |
| dhellmann | we'll need the path, too, I think that only shows imports that work | 19:21 |
| anteaya | AJaeger: new record? | 19:21 |
| *** matrohon has joined #openstack-infra | 19:21 | |
| AJaeger | anteaya: might be. Do we record these records anywhere? | 19:21 |
| clarkb | odyssey4me: ok new patch pushed | 19:21 |
| *** e0ne has joined #openstack-infra | 19:22 | |
| anteaya | AJaeger: just irc that I know of | 19:22 |
| anteaya | AJaeger: 326 in my browser | 19:22 |
| fungi | dhellmann: though i'll need to work out how to do that from within python. the only script that's going to be easy for me to edit here is a python script. the shell scripts are dynamically created by the launcher | 19:22 |
| *** moravec has quit IRC | 19:23 | |
| AJaeger | anteaya: let's wait 10mins and we're at 330 ;) | 19:23 |
| anteaya | okey dokey | 19:24 |
| mtreinish | fungi: oh, did you restart the lpmqtt and germqtt daemons too? I'm not seeing any messages on the restarted mosquitto | 19:24 |
| anteaya | the test node graph in status.openstack.org/zuul looks spikier than usual | 19:24 |
| anteaya | I have to admit that I can't read the grafana graphs as well as i would like | 19:24 |
| *** esikachev has quit IRC | 19:24 | |
| anteaya | so I can't tell from the nodepool graph in grafana if there is an issue or not | 19:25 |
| AJaeger | weird, http://grafana.openstack.org/dashboard/db/zuul-status shows 102 post jobs while http://status.openstack.org/zuul/ shows 329 | 19:25 |
| dhellmann | fungi : I don't see a way to set the verbose flag from within a python program :-( | 19:25 |
| fungi | mtreinish: i restarted the other two just now at your urging | 19:25 |
| clarkb | anteaya: looks like it thinks we have a ton of instances that are ready that may not be getting used? | 19:25 |
| anteaya | AJaeger: I confirm 329 in my browser from status.o.o/zuul | 19:25 |
| clarkb | anteaya: I will take a quick look at that | 19:25 |
| fungi | dhellmann: then i'll go back to my plan of spewing os.environ and sys.path to stdout | 19:26 |
| anteaya | clarkb: thank you | 19:26 |
| mtreinish | fungi: thanks. I should look into the code for the daemons and see if there is a way I can prevent them from crashing when mosquitto goes down | 19:26 |
| mtreinish | fungi: based on the stacktrace I remember from before paho retries the connection for a while but since the outage is more than a few min it eventually bails out | 19:26 |
| dhellmann | fungi : ack. we could replace /usr/local/jenkins/slave_scripts/release-tools/launchpad_add_comment.py with a shell script that calls a separate copy of that script using the -v (or even -vv) option | 19:26 |
| *** jamielennox|away has quit IRC | 19:26 | |
| clarkb | anteaya: looks like they are ready across all zuul launchers and clouds and none are more than an hour old | 19:27 |
| anteaya | clarkb: wow | 19:27 |
| clarkb | I think that nodepool may be able to get them ready before we can shcedule to them | 19:27 |
| dhellmann | fungi: the -vv option will print each file that is checked when an import is tried | 19:27 |
| clarkb | so things are "ok" | 19:27 |
| anteaya | clarkb: thanks for looking, I appreciate it | 19:27 |
| *** trown is now known as trown|brb | 19:28 | |
| clarkb | anteaya: we just went from 250 ready to 160 ish | 19:29 |
| *** jamielennox|away has joined #openstack-infra | 19:29 | |
| clarkb | so I think that that is just delay between nodepool booting them and zuul grabbing them for jobs | 19:29 |
| anteaya | awesome | 19:29 |
| *** jamielennox|away is now known as jamielennox | 19:30 | |
| anteaya | thank you | 19:30 |
| anteaya | AJaeger: 330 in post | 19:30 |
| AJaeger | clarkb, do you know why there's difference between http://grafana.openstack.org/dashboard/db/zuul-status and http://status.openstack.org/zuul/ on post jobs? 105 vs 330? | 19:31 |
| *** trown|brb is now known as trown | 19:31 | |
| clarkb | AJaeger: I do not | 19:31 |
| fungi | dhellmann: agreed, i'll reserve that as an option if this doesn't get us anything helpful | 19:31 |
| dhellmann | k | 19:31 |
| AJaeger | clarkb: ok | 19:31 |
| mtreinish | fungi: so one thing I did just check is on my local mosquitto at home I'm not running with a logfile (or logrotate on it) | 19:32 |
| mtreinish | I just have it log to stderr and let systemd handle it | 19:32 |
| *** esikachev has joined #openstack-infra | 19:32 | |
| *** samyewell has quit IRC | 19:32 | |
| mtreinish | and I don't think I've had mosquitto crash on me since I started it up | 19:32 |
| *** sdake has quit IRC | 19:33 | |
| mtreinish | although I also don't run it with a websocket listener so it could be that too | 19:33 |
| fungi | mtreinish: we could always try to run it without logging, but i wouldn't want to so that as an indefinite workaround | 19:35 |
| *** moravec has joined #openstack-infra | 19:36 | |
| clarkb | mtreinish: fungi did the copytruncate thing not help/ | 19:38 |
| fungi | clarkb: we hadn't tried it yet, looking for correlation first. this time it didn't crash during log rotation, just at random | 19:38 |
| clarkb | huh | 19:39 |
| *** tuannguyen has joined #openstack-infra | 19:39 | |
| *** moravec has quit IRC | 19:40 | |
| *** onovy has joined #openstack-infra | 19:40 | |
| anteaya | AJaeger: 332 in post | 19:40 |
| fungi | and tripped a gpf in the websockets library, though no idea if that points to the cause or is just a secondary affect. but the timestamps in dmesg/syslog do at least correlate closely to the crashes | 19:40 |
| anteaya | new nodes came online | 19:40 |
| anteaya | cloudnull: ^^ | 19:40 |
| cloudnull | wooo! | 19:41 |
| * cloudnull goes home | 19:41 | |
| mtreinish | fungi: hmm, I'm not sure if we switched it to stderr how well it would work. I just looked at the package source and I only see an upstart file, no systemd unit files | 19:41 |
| *** peterlisak has joined #openstack-infra | 19:41 | |
| mtreinish | not sure how ubuntu is hanlding the migration or how any of that works | 19:41 |
| anteaya | building 744 nodes: http://grafana.openstack.org/dashboard/db/nodepool | 19:41 |
| anteaya | cloudnull: I thought you were home | 19:41 |
| cloudnull | you caught me :) | 19:42 |
| clarkb | ya we seem to be building them alright then putting them into ready state where they hang out until the zuul launchers can catch up | 19:42 |
| anteaya | :) | 19:42 |
| AJaeger | thanks, cloudnull | 19:42 |
| clarkb | ~300 ready now | 19:42 |
| fungi | dhellmann: http://logs.openstack.org/d4/d43245a42b8f8442e76a39e260da06dc6d5bd2b7/release-post/tag-releases/1f717c2,byhand1/console.html#_2016-09-26_19_34_41_170875 | 19:42 |
| anteaya | clarkb: thanks for keeping an eye on this | 19:42 |
| cloudnull | thank you all. I appreciate you beating on our env :) | 19:42 |
| *** salv-orl_ has joined #openstack-infra | 19:42 | |
| anteaya | AJaeger: 336 in post | 19:43 |
| fungi | dhellmann: oh! | 19:43 |
| * anteaya hears a light go off | 19:43 | |
| dhellmann | zuul-env | 19:43 |
| fungi | why, why, oh why is /usr/zuul-env in sys.path? | 19:43 |
| dhellmann | I did that to allow zuul-cloner to be found in $PATH instead of having it hard-coded, because hard-coding it breaks the manual operation of the script | 19:44 |
| dhellmann | I can fix this, gimme a sec | 19:44 |
| fungi | aha! | 19:44 |
| * AJaeger remember reviewing it ;( | 19:44 | |
| AJaeger | thanks, fungi | 19:44 |
| *** crinkle_ is now known as crinkle | 19:45 | |
| AJaeger | anteaya: could you review https://review.openstack.org/#/c/363356/ , please? | 19:45 |
| AJaeger | dougwig: please reconsider your -1 on https://review.openstack.org/#/c/368948/ | 19:45 |
| *** salv-orlando has quit IRC | 19:45 | |
| fungi | dhellmann: okay, if you get a prospective patch linked in here i can take it for a spin before i turn puppet back on for that node | 19:45 |
| *** zz_dimtruck is now known as dimtruck | 19:46 | |
| *** sdake has joined #openstack-infra | 19:46 | |
| dhellmann | fungi : ack | 19:46 |
| fungi | that way we'll be 100% sure it works | 19:46 |
| dhellmann | I wonder if it would be enough to move zuul-env to the end of $PATH instead of the front | 19:47 |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Remove Cinder GlusterFS volume driver jobs https://review.openstack.org/377059 | 19:47 |
| *** thiagolib has joined #openstack-infra | 19:48 | |
| anteaya | thingee: thank you for continuing to do such a wonderful job composing and posting the dev digest | 19:50 |
| fungi | dhellmann: probably? i mean, the real issue is that it's finding /usr/zuul-env/bin/python before /usr/bin/python | 19:50 |
| *** ihrachys has joined #openstack-infra | 19:50 | |
| thingee | anteaya: thank you. it has been kind of slow, but maybe that's good | 19:51 |
| *** florianf has quit IRC | 19:51 | |
| fungi | dhellmann: a slightly less risky solution is what i do on my workstation... add a directory to PATH and then populate that directory with symlinks to the executables i want to use | 19:51 |
| dhellmann | fungi : yeah, if we had several of these I'd do something like that, but in this case I'll just put in the logic to look for the executable | 19:52 |
| *** vhosakot has joined #openstack-infra | 19:52 | |
| fungi | okay | 19:52 |
| anteaya | thingee: indeed | 19:53 |
| anteaya | thingee: :) | 19:53 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config: do not add zuul-env to the PATH https://review.openstack.org/377064 | 19:53 |
| dhellmann | fungi : ^^ | 19:53 |
| AJaeger | dhellmann: LGTM | 19:54 |
| *** e0ne has quit IRC | 19:54 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool: Add UploadWorker skeleton to the builder https://review.openstack.org/377065 | 19:55 |
| dhellmann | AJaeger : thanks | 19:55 |
| fungi | giving it a try now, just a sec | 19:55 |
| dhellmann | now to decide whether to re-enable error reporting for the launchpad script and have the job fail if that fails | 19:56 |
| *** vhosakot_ has joined #openstack-infra | 19:56 | |
| *** vhosakot has quit IRC | 19:56 | |
| * AJaeger waves good night | 19:56 | |
| dhellmann | AJaeger: good night! | 19:57 |
| openstackgerrit | Merged openstack-infra/project-config: Add networking-ovn grenade graph to grafana dashboard https://review.openstack.org/373534 | 19:57 |
| anteaya | AJaeger: good night | 19:57 |
| openstackgerrit | Merged openstack-infra/project-config: networking-odl: add job for OpenDaylight carbon https://review.openstack.org/363356 | 19:57 |
| fungi | dhellmann: i've left the path debugging in for now in this run with your 377064 manually applied | 19:57 |
| dhellmann | fungi : good plan | 19:58 |
| *** vhosakot_ has quit IRC | 19:58 | |
| *** e0ne has joined #openstack-infra | 19:59 | |
| pabelanger | clarkb: ya, scheduling issues for sure. We have 640+ ready nodes ATM, but 440 in use | 19:59 |
| pabelanger | zuul slowing grabbing them 1 at a time | 19:59 |
| clarkb | ya I wonder if that is related to general gear slowness | 20:00 |
| Shrews | clarkb: did you ever get an answer to your ansible question in #ansible-devel? | 20:00 |
| clarkb | Shrews: no so I gave up and just added the logic to devstack-gate | 20:01 |
| *** vhosakot has joined #openstack-infra | 20:01 | |
| pabelanger | clarkb: I'm thinking so | 20:01 |
| clarkb | I think I have realized that ansible doesn't care much about backward compatibleness | 20:01 |
| clarkb | since every single release in the 2.x series has broken something | 20:01 |
| pabelanger | the good news is, we are launching nodes faster then we can consume them :) | 20:01 |
| pabelanger | a good problem to have | 20:01 |
| clarkb | yes though I wonder if there is some balance there that might make throughput better. In any case a good problem to have and fix if it is indeed a problem | 20:02 |
| Shrews | clarkb: it's certainly not that they don't care | 20:02 |
| anteaya | I see 739 ready, 459 in use, 83 deleting | 20:03 |
| clarkb | Shrews: ya maybe care is the wrong word but basiocally many things have broken and no effort has been made to correct those issues instead we most roll forward with them | 20:03 |
| anteaya | AJaeger: and 378 in post | 20:03 |
| openstackgerrit | John Trowbridge proposed openstack-infra/tripleo-ci: Move ENV var setting to a common function https://review.openstack.org/377035 | 20:04 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-mosquitto: Use the mosquitto ppa for ubuntu packages https://review.openstack.org/377071 | 20:05 |
| *** e0ne has quit IRC | 20:05 | |
| mtreinish | fungi: ^^^ not sure if that's a good idea or not. But I feel like I need to do something :) | 20:06 |
| *** nherciu has quit IRC | 20:06 | |
| jeblair | clarkb, Shrews: in the future, we will be in a better position to bring attention to things that break us earlier :) | 20:07 |
| jeblair | clarkb: did you get a response on the empty inventory thing? is there a tldr i can read? | 20:07 |
| Shrews | jeblair: we just brought attention to it again in #ansible-devel | 20:08 |
| jeblair | cool, i'll catch up there | 20:08 |
| *** e0ne has joined #openstack-infra | 20:10 | |
| anteaya | we seem to have nodes reach the delete state at the same speed as zuul can assign jobs to nodes | 20:10 |
| anteaya | so far we haven't broken 500 nodes in use | 20:11 |
| jeblair | anteaya: it looks like we have too many ready nodes; is someone looking into it? | 20:12 |
| *** inc0_ has joined #openstack-infra | 20:12 | |
| anteaya | well clarkb looked at the graph and pabelanger agreed, that was as far as I have been informed about action | 20:13 |
| *** tuannguyen has quit IRC | 20:13 | |
| anteaya | jeblair: the patch to increase osic cloud1 from 512 to 1024 merged not long ago | 20:13 |
| *** inc0 has quit IRC | 20:13 | |
| anteaya | so that is the cause of the bulk of ready nodes | 20:13 |
| jeblair | anteaya: that alone is not sufficient cause for ready nodes | 20:13 |
| *** tuannguyen has joined #openstack-infra | 20:13 | |
| anteaya | the fact that they are all in the ready state however is new and I belive unexpected | 20:13 |
| anteaya | jeblair: okay | 20:14 |
| *** nherciu has joined #openstack-infra | 20:14 | |
| anteaya | that is the cause of our increase in quota, perhaps that is accurate | 20:14 |
| cloudnull | if i can help out in any way with regard to zuul and such let me know. | 20:14 |
| jeblair | anteaya: ya, and the increase means we may be able to see a new behavior in the system | 20:15 |
| anteaya | jeblair: agreed | 20:15 |
| mriedem | clarkb: any idea what's going on with logstash? | 20:15 |
| clarkb | mriedem: no fungi and jhesketh were maybe goign to look at it? | 20:16 |
| *** EricGonczer_ has joined #openstack-infra | 20:16 | |
| * mtreinish hides | 20:16 | |
| *** tuannguyen has quit IRC | 20:18 | |
| anteaya | we now have 842 ready, 449 in use, we aren't gaining any ground here | 20:19 |
| jeblair | anteaya: i examined one ready node -- it ran a job but nodepool does not seem to have observed it | 20:19 |
| *** mat128|mtg is now known as mat128 | 20:19 | |
| anteaya | jeblair: really? | 20:19 |
| anteaya | that is new behaviour is it not? | 20:19 |
| openstackgerrit | sebastian marcet proposed openstack-infra/openstackid-resources: New Locations endpoints https://review.openstack.org/377078 | 20:19 |
| anteaya | at least I haven' heard of that before | 20:19 |
| jeblair | anteaya: sadly no, it's similar to a known behavior of zeromq | 20:19 |
| anteaya | ah | 20:20 |
| anteaya | we might be triggering a zeromq bug? | 20:20 |
| jeblair | i see zeromq traffic now, so i wonder if rax had a network issue or something | 20:20 |
| *** lucasagomes is now known as lucas-afk | 20:21 | |
| *** cdent has quit IRC | 20:21 | |
| *** akshai has joined #openstack-infra | 20:21 | |
| jeblair | hrm | 20:21 |
| anteaya | when I search for rax-dfw status I get https://launchpad.net/~fungi | 20:21 |
| anteaya | ha ha ha | 20:21 |
| *** flepied has joined #openstack-infra | 20:21 | |
| jeblair | i picked one from 30 mins ago and 5 mins ago | 20:21 |
| jeblair | and they both are missing zmq events in nodepool | 20:21 |
| jeblair | so either it's an ongoing network issue, or (perhaps we should consider this more likely) zmq on nodepool is dropping events? | 20:22 |
| anteaya | I don't see any alerts at https://status.rackspace.com/ | 20:22 |
| *** akshai has quit IRC | 20:22 | |
| *** morgabra_ is now known as morgabra | 20:22 | |
| *** hashar has quit IRC | 20:22 | |
| *** slaweq_ has joined #openstack-infra | 20:22 | |
| jeblair | or, nodepool is just really, really behind | 20:23 |
| jeblair | we know it was already hitting its scalability limit before now | 20:23 |
| anteaya | we knew that? | 20:23 |
| jeblair | yep | 20:23 |
| clarkb | it looked like ready would decrease by about what used would increase | 20:23 |
| anteaya | if we knew that I missed that | 20:23 |
| anteaya | sorry | 20:23 |
| jeblair | anteaya: it's part of why we are making zuulv3 | 20:23 |
| clarkb | so I think its managing to transition them. But maybe not all of them | 20:23 |
| anteaya | jeblair: okay thank you | 20:24 |
| anteaya | jeblair: I heard you gave a great zuulv3 presentation in germany, I'm sorry I missed it | 20:24 |
| fungi | dhellmann: sorry, juggling too many things at once, but it seems to have worked http://logs.openstack.org/d4/d43245a42b8f8442e76a39e260da06dc6d5bd2b7/release-post/tag-releases/1f717c2,byhand2/console.html | 20:24 |
| fungi | dhellmann: i'll approve your fix | 20:24 |
| *** ijw_ has quit IRC | 20:24 | |
| *** ijw has joined #openstack-infra | 20:25 | |
| jeblair | anteaya: well, less of a presentation and more of mordred and i setting up some introductory material for a great discussion :) | 20:25 |
| *** Ravikiran_K has quit IRC | 20:25 | |
| anteaya | I'm sorry I missed that too | 20:25 |
| mordred | ++ | 20:25 |
| anteaya | in any case, what would you like to do at this moment? | 20:25 |
| dhellmann | fungi : thanks, I appreciate your debugging help today, as always | 20:26 |
| anteaya | well regarding nodepool and zuul in any case | 20:26 |
| *** gouthamr has quit IRC | 20:26 | |
| * anteaya scopes better | 20:26 | |
| *** ijw has quit IRC | 20:26 | |
| jeblair | i expect to have some written materials come out of that though -- mordred is writing up some examples; i think i have a spec update in the future, and i'll have an update for the meeting tomorrow i think | 20:26 |
| anteaya | okay thank you | 20:26 |
| jeblair | s/think/hope/ :) | 20:26 |
| cloudnull | jeblair: clarkb: anteaya: are the zuul/nodepool scalability issues something that we could help with by hosting a some of the infra? | 20:27 |
| anteaya | I always get more out of inperson things though as I tend to associate what was said with where someone was in the room when they said it | 20:27 |
| cloudnull | IE: could adding providers help at this point? | 20:27 |
| anteaya | cloudnull: well personally I am thinking we might need to revert the osic quota increase | 20:27 |
| cloudnull | that's fair | 20:27 |
| jeblair | cloudnull: unfortunately, i don't think so -- there are some fundamental issues in nodepool that we are addressing in v3, but don't have easy fixes today -- but we should talk more about hosting... :) | 20:28 |
| anteaya | as to my mind that may have been a trigger for todays event | 20:28 |
| jeblair | cloudnull, anteaya: i don't think we need to revert | 20:28 |
| anteaya | okay great | 20:28 |
| anteaya | I'm wrong then | 20:28 |
| jeblair | we are not using resources optimally, but we are using them at the highest rate that we can at the moment | 20:28 |
| jeblair | clarkb: yeah, i'm checking back in on the nodes i picked out earlier to see if they eventually see zmq events | 20:29 |
| jeblair | that may help illustrate what's going on | 20:29 |
| openstackgerrit | Merged openstack-infra/openstackid-resources: New Locations endpoints https://review.openstack.org/377078 | 20:29 |
| *** cjloader has joined #openstack-infra | 20:30 | |
| jeblair | clarkb, anteaya: aha -- the zmq event for the 30min old ready node arrived 45 minutes late | 20:31 |
| jeblair | (er, i guess it's now a 45 min old ready node ;) | 20:31 |
| anteaya | :) | 20:31 |
| anteaya | but it arrived | 20:31 |
| anteaya | yay? | 20:31 |
| anteaya | AJaeger: 417 in post | 20:31 |
| anteaya | AJaeger: definitely a record | 20:32 |
| cjloader | I have a question: why couldn't we create multiple zuuls and run parallelism to run tasks to be more efficient with resources? | 20:32 |
| *** amitgandhinz has quit IRC | 20:32 | |
| jeblair | so yeah, i think nodepool is so thread starved from all of the nodes launching that the thread which receives these events is heavily backlogged | 20:32 |
| jeblair | cjloader: is there a particular resource inefficiency you are trying to solve? | 20:33 |
| *** amitgandhinz has joined #openstack-infra | 20:33 | |
| cjloader | no, just a general question. | 20:33 |
| *** dimtruck is now known as zz_dimtruck | 20:34 | |
| jeblair | cjloader: oh, i asked because zuul is *highly* parallel, so i was trying to figure out what aspect you wanted to improve | 20:34 |
| *** adriant has joined #openstack-infra | 20:35 | |
| *** ihrachys has quit IRC | 20:35 | |
| *** Jeffrey4l has quit IRC | 20:35 | |
| cjloader | well, I wanted to find out if possibly increasing zuuls to allow for better resource allocation | 20:35 |
| *** spzala has quit IRC | 20:35 | |
| openstackgerrit | Eric Harney proposed openstack-infra/project-config: Remove Cinder GlusterFS volume driver jobs https://review.openstack.org/377059 | 20:35 |
| cjloader | just a idea | 20:35 |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Disable mqtt output plugin on logstash workers https://review.openstack.org/377087 | 20:36 |
| *** spzala has joined #openstack-infra | 20:36 | |
| cjloader | ignore me if not a good idea | 20:36 |
| mtreinish | mriedem, clarkb, pleia2: ^^^ | 20:36 |
| *** Jeffrey4l has joined #openstack-infra | 20:36 | |
| mtreinish | that probably won't hurt | 20:36 |
| *** rbrndt has joined #openstack-infra | 20:37 | |
| cloudnull | jeblair: if noodpool is thread starved do you think updating soft limits in limits.conf or setting kernel.max-threads higher would help ? | 20:37 |
| cjloader | what won't hurt? | 20:37 |
| cjloader | lol | 20:37 |
| cloudnull | are they system or app limits do you think? | 20:37 |
| *** trown is now known as trown|outtypewww | 20:38 | |
| cloudnull | s/kernel.max-threads/kernel.threads-max/ | 20:38 |
| *** krtaylor has quit IRC | 20:39 | |
| jeblair | cloudnull: i think the issue is that we have too many. i believe we're hitting python's limit of coping with threads -- it has more than 1k of them, and despite spending most of their time waiting on network io, the main thread is still starved for cpu time. | 20:39 |
| cloudnull | ah. bummer. | 20:40 |
| jeblair | (the machine itself has a load average of 10 and is 50% idle) | 20:40 |
| *** spzala has quit IRC | 20:40 | |
| cloudnull | moar clock speed! | 20:40 |
| jeblair | cloudnull: so the design for nodepool in v3 will take care of this, by allowing us to split out launching into multiple processes, or even machines | 20:40 |
| cloudnull | ^ that makes sense. -cc cjloader | 20:40 |
| jeblair | but it's not easy to fix in current nodepool -- at least without doing something hacky. it may be time to do something hacky though. :) | 20:41 |
| cloudnull | how far off is v3 ? | 20:41 |
| jeblair | cloudnull: months at least | 20:41 |
| clarkb | we could attempt multiprocessing but seems like that time we did that we ran itno all the multiprocessing issues | 20:41 |
| jeblair | (we could start using the new nodepool before zuul itself is ready, but even that is still months) | 20:42 |
| cloudnull | anything folks can help with to make then be now? :) | 20:42 |
| jeblair | clarkb: yeah, if we did, we would probably want to avoid using the magic shared datastructure stuff | 20:42 |
| clarkb | since in theory the abstraction is replace threads with processes but reality not so much | 20:42 |
| anteaya | cloudnull: here is the spec: http://specs.openstack.org/openstack-infra/infra-specs/specs/zuulv3.html | 20:42 |
| jeblair | (that's where most of the issues came in) | 20:42 |
| *** HeOS has joined #openstack-infra | 20:42 | |
| * cloudnull reading | 20:42 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/system-config: Disable mqtt output plugin on logstash workers https://review.openstack.org/377087 | 20:43 |
| *** zz_dimtruck is now known as dimtruck | 20:44 | |
| jeblair | cloudnull: if there are some good python programmers looking for a full-time or near full-time task, i can put them to work -- it's still not quite formed well enough for less dedicated work. | 20:44 |
| fungi | mriedem: clarkb: oh, yep, i never got any further digging into logstash issues. was going to pull up eshead and bigdesk but then other things broke | 20:45 |
| cloudnull | kencjohnston: ^ -- maybe something OSIC can help with | 20:45 |
| cloudnull | irtermite: | 20:45 |
| mtreinish | fungi: I think https://review.openstack.org/#/c/377087/ will fix it | 20:45 |
| cloudnull | jeblair: i'll start bothering people | 20:45 |
| cloudnull | :) | 20:45 |
| jeblair | cloudnull: cool! :) | 20:45 |
| Shrews | jeblair: there's no spec yet for that portion is there? | 20:46 |
| mtreinish | fungi: the last elasticsearch index is about the same time pleia2 and I flipped the switch on the mqtt plugin | 20:46 |
| cloudnull | idk if we can, but i'll see if we can get a couple of folks to pay attention to it. | 20:46 |
| cloudnull | seems like a good OSIC task imo | 20:46 |
| mtreinish | so probably better to just turn that off | 20:46 |
| jeblair | Shrews: it's idling in review https://review.openstack.org/305506 and will need an update to address comments left on it and think we have learned for stage 1 of the zk work | 20:46 |
| jeblair | s/think/things/ | 20:47 |
| Shrews | jeblair: ah ha. no wonder i couldn't find it | 20:47 |
| clarkb | fungi: mtreinish if you turn that off in logstash and indexing doesn't come back you might try restarting the ES cluster | 20:47 |
| clarkb | we have seen ES get into a deadlocked situation where the cluster isn't happy doing anything. I can help walk through that process if it becomes necessary | 20:48 |
| anteaya | cloudnull: a few people who can actually read all the history and test and review patches is better than a crowd | 20:48 |
| cloudnull | indeed. | 20:48 |
| anteaya | cloudnull: thanks | 20:48 |
| mtreinish | clarkb: heh, I don't think I'll be able to do that :) | 20:48 |
| anteaya | so if you can find any of those, I believe there are patches in gerrit they can start testing reviewing | 20:48 |
| clarkb | mtreinish: oh right yhou like to give out passwords. I will see if fungi can do it :P | 20:49 |
| *** jkilpatr has quit IRC | 20:49 | |
| fungi | mtreinish: clarkb: thanks. approved the temporarily disablement to see if it returns to sanity or not | 20:49 |
| anteaya | cloudnull: have them search for branch:feature/zuulv3 in gerrit | 20:50 |
| fungi | s/temporarily.temporary/ | 20:50 |
| clarkb | fungi: also make sure that it restarts lgostash service after that goes in | 20:50 |
| mtreinish | clarkb: haha, yep I'm a security nightmare | 20:50 |
| clarkb | fungi: since I think a full service restart is required there | 20:50 |
| fungi | clarkb: i'll double-check the process start time, definitely | 20:50 |
| cloudnull | anteaya: will do | 20:51 |
| anteaya | cloudnull: thank you | 20:51 |
| jeblair | clarkb: the 'easy' way to multiprocess nodepool would be to s/thread/process/, however, that messes up our taskmanager (the shared object that we run all of the openstack api calls through for rate-limiting purposes). so the unit of disaggregation needs to be the provider, which is more complex. :( | 20:51 |
| *** baoli has quit IRC | 20:52 | |
| *** esikachev has quit IRC | 20:52 | |
| *** Goneri has quit IRC | 20:53 | |
| *** dimtruck is now known as zz_dimtruck | 20:54 | |
| *** inc0_ has quit IRC | 20:54 | |
| *** inc0 has joined #openstack-infra | 20:55 | |
| *** gyee has quit IRC | 21:00 | |
| *** e0ne has quit IRC | 21:06 | |
| *** stewie925 is now known as Guest63324 | 21:08 | |
| *** raildo has quit IRC | 21:09 | |
| *** stew925 has joined #openstack-infra | 21:09 | |
| *** vhosakot has quit IRC | 21:10 | |
| *** chem has quit IRC | 21:12 | |
| *** eharney has quit IRC | 21:12 | |
| *** apetrich has quit IRC | 21:13 | |
| *** apetrich has joined #openstack-infra | 21:14 | |
| *** chem has joined #openstack-infra | 21:14 | |
| *** zz_dimtruck is now known as dimtruck | 21:14 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit https://review.openstack.org/375113 | 21:15 |
| *** ldnunes has quit IRC | 21:16 | |
| *** moravec has joined #openstack-infra | 21:17 | |
| *** mriedem has quit IRC | 21:17 | |
| *** dprince has quit IRC | 21:18 | |
| *** rossella_s has quit IRC | 21:19 | |
| *** piet_ has quit IRC | 21:19 | |
| zigo | What did I do wrong here? http://logs.openstack.org/51/376551/2/check/gate-project-config-layout/41fb730/console.html | 21:19 |
| zigo | Oh, sorry, got it ! :) | 21:19 |
| *** rossella_s has joined #openstack-infra | 21:20 | |
| *** nherciu_ has joined #openstack-infra | 21:20 | |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Add 3 missing deb-python-xstatic for Horizon https://review.openstack.org/376551 | 21:20 |
| jeblair | clarkb, Shrews, cloudnull: we *might* be able to make the taskmanager a separate process which would then enable us to make the launch threads separate processes. the threads would communicate with a queue over pipes, which doesn't require the magic multiprocessing server. this might not be too complex to implement and might make a performance difference. i'm poking at it a little bit to see if it's worthwhile. | 21:22 |
| *** nherciu has quit IRC | 21:22 | |
| jeblair | (yay taskmanager api) | 21:22 |
| *** piet_ has joined #openstack-infra | 21:23 | |
| anteaya | yay | 21:23 |
| *** piet_ has quit IRC | 21:23 | |
| anteaya | I have to go run errands | 21:23 |
| anteaya | offline for the rest of the night | 21:23 |
| cloudnull | jeblair: that'd be awesome! | 21:23 |
| *** psilvad has quit IRC | 21:23 | |
| cloudnull | have a good one anteaya | 21:23 |
| *** psilvad has joined #openstack-infra | 21:23 | |
| cloudnull | jeblair: i have to go make dinner, then respond to email but if there's something i can lend a hand with please let me know. | 21:24 |
| *** dimtruck is now known as zz_dimtruck | 21:24 | |
| *** akshai has joined #openstack-infra | 21:24 | |
| *** krtaylor has joined #openstack-infra | 21:24 | |
| zigo | fungi: It's been more than a day that we're waiting for deb-networking-mlnx to sync the latest tag from upstream, is there anyway to trigger it? | 21:25 |
| *** jheroux has quit IRC | 21:25 | |
| *** dave-mccowan has joined #openstack-infra | 21:25 | |
| *** matrohon has quit IRC | 21:26 | |
| *** jkilpatr has joined #openstack-infra | 21:26 | |
| *** ijw has joined #openstack-infra | 21:27 | |
| Shrews | jeblair: seems like a feasible idea | 21:27 |
| *** sdague has quit IRC | 21:27 | |
| fungi | zigo: as mentioned back when you set this up, i think we need to switch it to a cron job, right now the only time it gets updated is when changes merge to project-config altering project configurations | 21:27 |
| *** burgerk_ has joined #openstack-infra | 21:27 | |
| *** matrohon has joined #openstack-infra | 21:27 | |
| Shrews | i love how python forces one to find the right magical combination between threads and processes to make things work at scale | 21:27 |
| fungi | zigo: which, while pretty frequent, is not a refular frequency | 21:28 |
| * Shrews says with MUCH sarcasm | 21:28 | |
| fungi | er, regular | 21:28 |
| zigo | fungi: But as infra-root, don't you have a magic button? :P | 21:28 |
| Shrews | fungi IS our magic button | 21:29 |
| * Shrews presses fungi | 21:29 | |
| *** akshai has quit IRC | 21:29 | |
| zigo | BUZZZZZZZ | 21:29 |
| zigo | :) | 21:29 |
| *** Apoorva_ has joined #openstack-infra | 21:29 | |
| fungi | zigo: sort of. i'll run the script to confirm my hypothesis | 21:29 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Log possible KeyError https://review.openstack.org/377105 | 21:30 |
| *** thorst has quit IRC | 21:30 | |
| *** salv-orl_ has quit IRC | 21:30 | |
| *** nherciu_ has quit IRC | 21:30 | |
| *** akshai has joined #openstack-infra | 21:30 | |
| *** ijw has quit IRC | 21:31 | |
| *** akshai has quit IRC | 21:31 | |
| EmilienM | hi infra, any chance to have a review on https://review.openstack.org/#/c/377000/ ? thanks | 21:31 |
| *** ijw has joined #openstack-infra | 21:31 | |
| *** burgerk has quit IRC | 21:31 | |
| *** rtheis has quit IRC | 21:32 | |
| *** salv-orlando has joined #openstack-infra | 21:32 | |
| *** akshai has joined #openstack-infra | 21:32 | |
| *** akshai has quit IRC | 21:32 | |
| *** claudiub has quit IRC | 21:33 | |
| *** Apoorva has quit IRC | 21:33 | |
| *** csomerville has quit IRC | 21:33 | |
| *** sdake has quit IRC | 21:33 | |
| openstackgerrit | Merged openstack-infra/project-config: do not add zuul-env to the PATH https://review.openstack.org/377064 | 21:35 |
| *** thorst has joined #openstack-infra | 21:36 | |
| *** gordc has quit IRC | 21:37 | |
| *** annegentle has quit IRC | 21:37 | |
| clarkb | EmilienM: reviewed | 21:37 |
| *** psilvad has quit IRC | 21:38 | |
| *** adrian_otto has quit IRC | 21:38 | |
| *** esberglu has quit IRC | 21:40 | |
| *** r-mibu has quit IRC | 21:40 | |
| *** r-mibu has joined #openstack-infra | 21:40 | |
| *** thorst has quit IRC | 21:41 | |
| *** burgerk_ has quit IRC | 21:41 | |
| *** mriedem has joined #openstack-infra | 21:43 | |
| EmilienM | clarkb: nice catch, thx | 21:44 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/project-config: Convert tripleo-container jobs into multinode https://review.openstack.org/377000 | 21:45 |
| *** esikachev has joined #openstack-infra | 21:48 | |
| clarkb | I am going to pop out for a really late lunch nowish | 21:49 |
| *** thorst has joined #openstack-infra | 21:49 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config: Enable tlsproxy on "core" tempest jobs https://review.openstack.org/375778 | 21:50 |
| clarkb | would be great if people can review the changes with topic:devstack-tls we are ready to move ahead with turning that on in our jobs if we can get ~3-4 changes merged | 21:51 |
| clarkb | infra-core and qa core ^ | 21:51 |
| *** gouthamr has joined #openstack-infra | 21:51 | |
| *** sdague has joined #openstack-infra | 21:51 | |
| mtreinish | clarkb: heh, I don't think qa core will highlight for anyone | 21:51 |
| *** esikachev has quit IRC | 21:52 | |
| *** matt-borland has quit IRC | 21:53 | |
| *** thorst has quit IRC | 21:54 | |
| *** dfflanders has joined #openstack-infra | 21:56 | |
| *** Swami has quit IRC | 21:57 | |
| *** marst has quit IRC | 21:59 | |
| *** edtubill has quit IRC | 21:59 | |
| *** marst has joined #openstack-infra | 21:59 | |
| *** moravec has quit IRC | 22:00 | |
| *** marst has quit IRC | 22:01 | |
| *** annegentle has joined #openstack-infra | 22:01 | |
| zigo | fungi: https://review.openstack.org/376551 <--- Add 3 missing deb-python-xstatic for Horizon | 22:02 |
| rajinir | Is the devstack-gate broken? Bad subnets request: allocation_pools allowed only for specific subnet requests.. | 22:02 |
| *** annegent_ has joined #openstack-infra | 22:05 | |
| *** annegentle has quit IRC | 22:06 | |
| EmilienM | clarkb: have you seen some ssh timeouts on multinode/rax jobs? | 22:08 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit https://review.openstack.org/375113 | 22:08 |
| EmilienM | https://goo.gl/42w5XL - tripleo timeouts | 22:08 |
| *** salv-orlando has quit IRC | 22:08 | |
| *** dstanek_ is now known as dstanek | 22:08 | |
| *** boden has left #openstack-infra | 22:09 | |
| *** inc0 has quit IRC | 22:10 | |
| EmilienM | clarkb: wrong url | 22:11 |
| *** xyang1 has quit IRC | 22:11 | |
| EmilienM | but query is build_name: *tripleo-ci* AND build_status: FAILURE AND message: "port 22: Connection timed out" | 22:11 |
| fungi | looks like we can't do full manage-projects runs... it blocks prompting for a github username for some reason | 22:11 |
| fungi | anyone have time to look into that? | 22:11 |
| fungi | i'm rerunning right now with -v to see if i can work out which project it's hitting that on | 22:12 |
| *** camunoz_ has joined #openstack-infra | 22:12 | |
| openstackgerrit | Matt Riedemann proposed openstack-infra/devstack-gate: DNM: test nova-manage db archive_deleted_rows post_test_hook https://review.openstack.org/377113 | 22:12 |
| *** camunoz_ has quit IRC | 22:13 | |
| *** amitgandhinz has quit IRC | 22:13 | |
| fungi | i had hoped that </dev/null would cause it not to think there was anybody around to answer interactive password prompts, but no, it still sits there waiting at a prompt | 22:14 |
| *** pfallenop has quit IRC | 22:15 | |
| *** tuannguyen has joined #openstack-infra | 22:15 | |
| *** gordc has joined #openstack-infra | 22:16 | |
| *** ijw has quit IRC | 22:16 | |
| *** yolanda has quit IRC | 22:17 | |
| *** yolanda has joined #openstack-infra | 22:18 | |
| jeblair | clarkb, fungi, mordred, Shrews: as i look into this, it gets slightly more complicated, and i'm brought around to considering what i think cjloader may have been suggesting earlier, which is running multiple nodepool daemons. | 22:18 |
| jeblair | the answer to "why not?" for that is that they would be partially unaware of what the others are doing and would sometimes respond inaccurately to demand, likely producing more nodes than needed. | 22:19 |
| jeblair | the rejoinder to that is "do we care?" and possibly at the moment, the answer is "no -- we need to throw more nodes at the system more than we need to throw the right number at it" | 22:20 |
| *** tuannguyen has quit IRC | 22:21 | |
| fungi | jeblair: how do you think we'd partition the quota? or is that something we'd coordinate between them through zk? | 22:21 |
| jeblair | if we had multiple nodepoold's sharing a database, then they could *approximate* the correct supply of nodes and use the same set of images. they would just occasionally cycle at close enough to the same time that they would both see a demand for 500 nodes and both try to launch 500 nodes. | 22:21 |
| fungi | ahh, got it | 22:21 |
| fungi | so shared db | 22:21 |
| jeblair | fungi: oh, i'm working on a solution we can implement in *hours*. the long term plan is still the same. :) | 22:22 |
| *** ijw has joined #openstack-infra | 22:22 | |
| fungi | makes sense | 22:22 |
| *** marst has joined #openstack-infra | 22:22 | |
| jeblair | fungi: and yeah, i think we would halve the quota for each provider if we ran two instances | 22:22 |
| fungi | do we configure all but one of them to build images once a year or something? | 22:22 |
| jeblair | that way if they did both decide to max out a provider, they wouldn't see an adverse impact. | 22:22 |
| jeblair | fungi: yeah, i'm thinking two config files which are otherwise identical, but only one of them builds images. | 22:23 |
| zigo | fungi: Can you please approve this patch ? We need it for the Horizon package: https://review.openstack.org/376551 | 22:24 |
| fungi | zigo: i'm trying to troubleshoot why manage-projects isn't working | 22:24 |
| *** AlexxNica has quit IRC | 22:24 | |
| zigo | Oh ! | 22:24 |
| fungi | i think it's choking trying to remote update jsonrpclib for you | 22:24 |
| jeblair | it's a sloppy solution, but i'm weighing it against code changes (which seem to be getting complex quickly and are distracting from v3), and the status quo, which is beginning to appear more sloppy to me. | 22:25 |
| *** rkukura has joined #openstack-infra | 22:25 | |
| fungi | zigo: the last thing i see in the debug log is this: | 22:25 |
| jeblair | (i mean, the system is unbalanced either way -- i'm starting to look at it as a choice between 'unbalanced with fewer or more nodes') | 22:25 |
| fungi | manage_projects - INFO - Executing command: git --git-dir=/opt/lib/jeepyb/openstack/deb-python-jsonrpclib/.git --work-tree=/opt/lib/jeepyb/openstack/deb-python-jsonrpclib remote update --prune | 22:25 |
| fungi | and then it goes to... Username for 'https://github.com': | 22:25 |
| fungi | i'm betting something is wrong with that repo | 22:25 |
| zigo | fungi: There's a leading dot in project-config for that repo. | 22:26 |
| zigo | fungi: ie: https://github.com/joshmarshall/jsonrpclib. | 22:27 |
| fungi | zigo: trailing maybe? says it's supposed to use: https://github.com/joshmarshall/jsonrpclib. | 22:27 |
| zigo | not leading, at the end ... | 22:27 |
| zigo | Yup. | 22:27 |
| fungi | yeah, wondering if that's supposed to be there | 22:27 |
| jeblair | the gearman aspect of this should be fine -- either nodepool daemon can run the 'node-return' gearman function... | 22:27 |
| fungi | i can clone it without the . but not with | 22:27 |
| jeblair | zmq is worth considering more though... they would both see zmq events and attempt to act on them | 22:27 |
| jeblair | so they would both try to delete used nodes at the same time | 22:28 |
| openstackgerrit | Thomas Goirand proposed openstack-infra/project-config: Fix deb-python-jsonrpclib https://review.openstack.org/377117 | 22:28 |
| zigo | Right, that's a mistake, IMO. | 22:28 |
| fungi | jeblair: and i guess we'd (roughly) divide the min-ready by the number of daemons? | 22:28 |
| jeblair | fungi: i think so | 22:28 |
| jeblair | oh! | 22:28 |
| jeblair | what if, instead, we had one of them launch nodes, and the other delete nodes? | 22:29 |
| fungi | zigo: you want to submit a change to fix that? | 22:29 |
| zigo | Yeah, done ! :) | 22:29 |
| zigo | Just did ... | 22:29 |
| fungi | oh, perfect | 22:29 |
| zigo | https://review.openstack.org/377117 | 22:29 |
| fungi | yeah, i see it above now | 22:29 |
| fungi | just doing too many things at once | 22:29 |
| *** ijw has quit IRC | 22:29 | |
| jeblair | that could be quickly and would allow us to reduce the load on nodepoold by ~half | 22:29 |
| *** Swami has joined #openstack-infra | 22:29 | |
| *** matrohon has quit IRC | 22:29 | |
| zigo | I wonder how come the patch to add track-upstream still went through with that pb. | 22:30 |
| fungi | zigo: i suspect, while having not confirmed, that our upstream url tester only fires if track-upstream isn't set | 22:31 |
| fungi | jeblair: i guess it's a code change either way, but i agree that's pretty minor (boot worker and delete worker) | 22:31 |
| *** pfallenop has joined #openstack-infra | 22:31 | |
| zigo | Oh... And to be honest, I wasn't so careful, because I was relying on it to tell me my mistakes... :/ | 22:32 |
| *** fguillot has quit IRC | 22:32 | |
| fungi | jeblair: or were you thinking just configure one of the daemons with no quota and leave the code as-is? | 22:33 |
| fungi | it wouldn't be busy tracking booting nodes and would likely end up finding and handling most of the deletions | 22:34 |
| *** spzala has joined #openstack-infra | 22:34 | |
| fungi | at least under heavy activity | 22:34 |
| fungi | though at the expense of racing and occasionally issuing extra deletes | 22:34 |
| jeblair | fungi: well, the other would still try to delete things... though... i guess if it didn't see any zmq events it wouldn't... so... i guess you're right, no code changes needed? | 22:35 |
| jeblair | (so, one has quota but no zmq listeners; the other has zmq listeners but no quota) | 22:36 |
| fungi | it just might work | 22:36 |
| fungi | also it might asplode, of course | 22:36 |
| *** spzala has quit IRC | 22:36 | |
| jeblair | yes, both are possibilities | 22:36 |
| *** spzala has joined #openstack-infra | 22:37 | |
| openstackgerrit | Abhishek Raut proposed openstack-infra/project-config: Use python-db-jobs for vmware-nsx https://review.openstack.org/375766 | 22:38 |
| *** pfallenop has quit IRC | 22:38 | |
| *** pfallenop has joined #openstack-infra | 22:39 | |
| *** rkukura has quit IRC | 22:39 | |
| *** spzala has quit IRC | 22:41 | |
| jeblair | fungi: a code change may be better than trying to keep two 1k-line files half-synced. | 22:42 |
| fungi | jeblair: also true. and could be boiled down to a simple cli option or something | 22:44 |
| *** tpsilva has quit IRC | 22:44 | |
| fungi | or envvar or whatever | 22:44 |
| jeblair | fungi: yeah, i'm poking at a cli now | 22:45 |
| *** zz_dimtruck is now known as dimtruck | 22:48 | |
| *** esikachev has joined #openstack-infra | 22:48 | |
| *** annegent_ has quit IRC | 22:49 | |
| *** esikachev has quit IRC | 22:53 | |
| *** dimtruck is now known as zz_dimtruck | 22:57 | |
| *** Swami_ has joined #openstack-infra | 22:59 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add options to skip deleting or launching nodes https://review.openstack.org/377123 | 23:00 |
| jeblair | fungi: ^ | 23:00 |
| fungi | exciting | 23:01 |
| *** edmondsw has quit IRC | 23:01 | |
| *** rbrndt has quit IRC | 23:01 | |
| fungi | jeblair: i guess that also opens up the option to have one only scheduling image updates? though with the separate build worker that's maybe not too useful for us anyway | 23:02 |
| *** lucas-afk has quit IRC | 23:02 | |
| mordred | jeblair: I like the split-along-create-delete lines idea | 23:02 |
| jeblair | fungi: yeah, that's kind of low-impact. i threw it in in case we wanted to have multiple launchers after all, then we could easily just have one do image updates. | 23:02 |
| mordred | jeblair: seems clean and also least amount of churn in a time/place we're not really wanting to do larger thoughts atm | 23:03 |
| *** cardeois_ has quit IRC | 23:03 | |
| *** Swami has quit IRC | 23:03 | |
| jeblair | yeah, hoping it's a good enough band-aid that we don't have to implement v3 before we have a chance to implement v3 | 23:03 |
| mordred | ++ | 23:03 |
| mordred | we didn't hit the problem until we gained an additional 500 nodes | 23:04 |
| mordred | so I'd imagine that unless we get another large influx we should be fine ) | 23:04 |
| openstackgerrit | Tim Burke proposed openstack-infra/jenkins-job-builder: Remove dead tool jenkins-jobs.py https://review.openstack.org/377128 | 23:05 |
| *** lucasagomes has joined #openstack-infra | 23:05 | |
| fungi | yeah, lgtm. there's no testing nor documentation with it, but i guess this isn't expected to stick around and could be considered "experimental" for now | 23:05 |
| jeblair | mordred: first-world ci problems | 23:05 |
| jeblair | fungi: i'm still under the misapprehension that if i don't document the terrible production hacks i write no one will depend on them. | 23:06 |
| *** adrian_otto has joined #openstack-infra | 23:06 | |
| jeblair | (but truly, we could document it if we want; i just wanted to get the idea out there) | 23:07 |
| *** aeng_ has joined #openstack-infra | 23:08 | |
| *** aeng has quit IRC | 23:09 | |
| openstackgerrit | Matthew Treinish proposed openstack-infra/puppet-mosquitto: Use the mosquitto ppa for ubuntu packages https://review.openstack.org/377071 | 23:12 |
| *** amitgandhinz has joined #openstack-infra | 23:14 | |
| *** zz_dimtruck is now known as dimtruck | 23:15 | |
| *** EricGonc_ has joined #openstack-infra | 23:15 | |
| *** EricGonczer_ has quit IRC | 23:17 | |
| *** Swami__ has joined #openstack-infra | 23:18 | |
| *** EricGonc_ has quit IRC | 23:20 | |
| *** AnarchyAo has quit IRC | 23:21 | |
| *** verdurin has quit IRC | 23:21 | |
| *** Swami_ has quit IRC | 23:22 | |
| *** EricGonczer_ has joined #openstack-infra | 23:22 | |
| openstackgerrit | Merged openstack-infra/system-config: Disable mqtt output plugin on logstash workers https://review.openstack.org/377087 | 23:24 |
| *** thorst has joined #openstack-infra | 23:24 | |
| *** verdurin has joined #openstack-infra | 23:24 | |
| *** dimtruck is now known as zz_dimtruck | 23:25 | |
| mtreinish | fungi, clarkb, mriedem: ^^^ I'm not sure if we need to restart the workers to pick that config change up or not | 23:27 |
| *** aeng_ has quit IRC | 23:28 | |
| fungi | yeah, i'm going to give it a few minutes and then check the process start time once the config looks updated | 23:28 |
| *** ijw has joined #openstack-infra | 23:28 | |
| * clarkb catches up after lunch | 23:29 | |
| *** hongbin has joined #openstack-infra | 23:30 | |
| clarkb | any other infra core willing to review https://review.openstack.org/#/c/375776/ ? once we get that in and one last devstack change we can start enabling tls in our tempest jobs | 23:31 |
| *** cody-somerville has joined #openstack-infra | 23:31 | |
| *** cody-somerville has joined #openstack-infra | 23:31 | |
| *** ijw has quit IRC | 23:32 | |
| *** Apoorva_ has quit IRC | 23:33 | |
| fungi | zaro: i added the gerrit jgit upgrade discussion to the agenda for tomorrow's meeting, i hope you don't mind? if you won't be around, i can do a pretty good job of covering the topic | 23:33 |
| clarkb | neat idea as far as splitting the nodepool daemon in half | 23:34 |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/zuul: Log possible KeyError https://review.openstack.org/377105 | 23:36 |
| *** tuannguyen has joined #openstack-infra | 23:37 | |
| *** adrian_otto has quit IRC | 23:38 | |
| *** salv-orlando has joined #openstack-infra | 23:39 | |
| clarkb | jeblair: commented on the nodepool change | 23:39 |
| pleia2 | fungi: did you figure out how to add a project to reviewday? | 23:39 |
| clarkb | unittests are failing and I have pointed out the reason | 23:39 |
| *** adrian_otto has joined #openstack-infra | 23:40 | |
| fungi | pleia2: clarkb had to show me | 23:40 |
| fungi | using very small words | 23:40 |
| *** Apoorva has joined #openstack-infra | 23:40 | |
| fungi | but thanks for following up! | 23:40 |
| *** Swami has joined #openstack-infra | 23:40 | |
| pleia2 | fungi: hah, ok, great :) | 23:40 |
| fungi | we could stand to get some minimal service documentation in place for that | 23:40 |
| clarkb | he thought I was pulling his leg | 23:40 |
| * pleia2 nods | 23:40 | |
| *** aeng_ has joined #openstack-infra | 23:40 | |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Only query indexes that exist https://review.openstack.org/377129 | 23:40 |
| jeblair | clarkb: doh | 23:40 |
| *** adrian_otto has quit IRC | 23:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool: Add options to skip deleting or launching nodes https://review.openstack.org/377123 | 23:41 |
| fungi | indeed, "the list of projects is hard-coded in the executable" sounded like he was having one over on me | 23:41 |
| clarkb | fungi: any luck getting ES to work? | 23:41 |
| clarkb | I guess thats in progress as we speak? | 23:41 |
| fungi | clarkb: i'm about to see if mtreinish's patch has taken effect | 23:41 |
| clarkb | kk let me know if I can help | 23:41 |
| *** zz_dimtruck is now known as dimtruck | 23:42 | |
| jeblair | clarkb: i fixed the typo and +3d the change (since i'm assuming there weren't other objections) | 23:42 |
| clarkb | jeblair: kk | 23:43 |
| *** Swami__ has quit IRC | 23:43 | |
| fungi | i've also about reached the point in today's program where beer is served, so if mtreinish's intuition was surprisingly wrong i'm not sure how much else todaying i've got left in me | 23:43 |
| *** sdake has joined #openstack-infra | 23:44 | |
| *** salv-orlando has quit IRC | 23:45 | |
| openstackgerrit | Merged openstack-infra/project-config: Fix deb-python-jsonrpclib https://review.openstack.org/377117 | 23:45 |
| mtreinish | fungi: heh, I'm very often wrong about things. Although in this case it would be really surprising if adding mqtt wasn't the cause of everything falling apart there | 23:45 |
| clarkb | fungi: tyty for tls review | 23:46 |
| mtreinish | since everything stopped working right after pleia2 and I merged that stuff last week | 23:46 |
| fungi | i have a bottle of clipper city brewing's "cross bones" session ipa that says you're right about it, mtreinish | 23:46 |
| *** timburke_ is now known as timburke | 23:46 | |
| clarkb | just need one more +2 and then things to merge then we can test with tls | 23:46 |
| clarkb | its like the future. We have ipv6 AND TLS | 23:46 |
| *** xarses has quit IRC | 23:46 | |
| fungi | the 1990s called, they want their future back | 23:47 |
| *** tphummel has quit IRC | 23:47 | |
| *** zhurong has joined #openstack-infra | 23:48 | |
| fungi | mtreinish: clarkb: start time on the jvm process on worker01 is 23:38 utc, so looks right? | 23:48 |
| *** ijw has joined #openstack-infra | 23:48 | |
| clarkb | yup that looks right | 23:49 |
| *** thorst has quit IRC | 23:49 | |
| clarkb | so now we watch if bigdesk/elasticsearch-head say we are indexing things | 23:49 |
| *** amitgandhinz has quit IRC | 23:49 | |
| *** AlexxNica has joined #openstack-infra | 23:49 | |
| *** esikachev has joined #openstack-infra | 23:49 | |
| fungi | and if not, rolling restart of es nodes, or big bang? | 23:49 |
| clarkb | hrm small sample of at least worker01 ways we no longer have our gearman workers running | 23:50 |
| clarkb | fungi: I think it has to be big bang if it is the deadlock issue | 23:50 |
| fungi | oh, i didn't check the gearman worker processes | 23:50 |
| *** puiterwijk has quit IRC | 23:50 | |
| clarkb | I have started them on 01 | 23:50 |
| fungi | clarkb: i see them, start time of now | 23:50 |
| clarkb | ya I just started them | 23:50 |
| fungi | oh, that would be why :/ | 23:50 |
| fungi | i see one running on worker05 | 23:51 |
| fungi | should be four? | 23:51 |
| clarkb | ya A-D | 23:51 |
| mtreinish | fungi: yep | 23:51 |
| clarkb | but even with the ones started on 01 I don't see indexing happening | 23:51 |
| fungi | none running on 08 | 23:51 |
| fungi | something has tanked them? | 23:52 |
| mtreinish | heh, maybe it wasn't all my fault then :) | 23:52 |
| *** dimtruck is now known as zz_dimtruck | 23:52 | |
| clarkb | fungi: I think they tank on their own when ES gets unhappy | 23:52 |
| fungi | none running on worker16 | 23:52 |
| openstackgerrit | Matt Riedemann proposed openstack-infra/devstack-gate: DNM: test nova-manage db archive_deleted_rows post_test_hook https://review.openstack.org/377113 | 23:52 |
| fungi | i think we can safely say almost all of them have died | 23:52 |
| clarkb | basically the pipes get clogged and the longer that situation exists the further upstream you notice | 23:52 |
| clarkb | fungi: I think we should start them and see if that changes anything but likely need to big bang ES restart | 23:53 |
| fungi | so restart es across the cluster at once, and then restart all gearman workers? | 23:53 |
| fungi | oh, got it | 23:53 |
| *** notmyname has quit IRC | 23:53 | |
| fungi | so i can loop across all the workers and restart the gearman client | 23:53 |
| *** ijw has quit IRC | 23:53 | |
| clarkb | the gearman worker | 23:53 |
| *** sdague has quit IRC | 23:53 | |
| clarkb | jenkins-log-gearman-worker-A - D | 23:53 |
| fungi | yeah, that thing | 23:53 |
| clarkb | er | 23:53 |
| *** notmyname has joined #openstack-infra | 23:53 | |
| clarkb | jenkins-log-worker-A | 23:54 |
| *** esikachev has quit IRC | 23:54 | |
| clarkb | oh! I see things indexing now | 23:54 |
| fungi | we have 24 of these? | 23:54 |
| clarkb | 20 | 23:54 |
| *** Sukhdev has joined #openstack-infra | 23:54 | |
| fungi | okay, so then hopefully restarting the rest will solve this now | 23:54 |
| clarkb | so maybe thats all we need to do is get them started | 23:54 |
| clarkb | ya | 23:54 |
| *** puiterwijk has joined #openstack-infra | 23:55 | |
| *** gordc has quit IRC | 23:56 | |
| clarkb | fungi: and worker15 hasn't restared its logstash, the others have | 23:56 |
| clarkb | maybe just ignore it until tomorrow and focus on getting the rest of them happy | 23:57 |
| *** dfflanders has quit IRC | 23:57 | |
| mtreinish | clarkb, fungi: when we're ready to try mqttifying this again maybe we should start with 1 worker node to test it manually before we push it to all of them at once | 23:58 |
| openstackgerrit | Ramy Asselin proposed openstack-infra/elastic-recheck: Make bug name, type, and url explicit https://review.openstack.org/375113 | 23:58 |
| fungi | mtreinish: sounds safer | 23:58 |
| *** ijw has joined #openstack-infra | 23:59 | |
| mtreinish | I'm not sure the best mechanism for doing that though, well besides adding the config block back in on one node by hand | 23:59 |
| clarkb | fungi: looks like you did A and B ? gonna do C and D next? | 23:59 |
| fungi | clarkb: yeah, still running | 23:59 |
| fungi | okay, finished, now double-checking them all | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!