1

Trying to deploy a multi-node openstack Juno, on 14.04. Here is my charm config file:

nova-cloud-controller:
  openstack-origin: cloud:trusty-juno
  network-manager: "Neutron"
openstack-dashboard:
  openstack-origin: cloud:trusty-juno
keystone:
  openstack-origin: cloud:trusty-juno
glance:
  openstack-origin: cloud:trusty-juno
cinder:
  openstack-origin: cloud:trusty-juno
ext1-quantum-gateway:
  openstack-origin: cloud:trusty-juno
ext2-quantum-gateway:
  openstack-origin: cloud:trusty-juno
nova-compute:
  openstack-origin: cloud:trusty-juno

On Juju gui, all nodes and relation lines are green; however, I cannot access Project->Access&Security page, under horizon. Looking at rabbitmq log files, it seems that all access, from various services, are denied:

# tail -f /var/log/rabbitmq/rabbit@rabbitmq1.log | grep access_refused -B2 -A3
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'neutron' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17775.0> (10.0.0.77:34733 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'neutron' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17778.0> (10.0.0.77:34734 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'neutron' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17781.0> (10.0.0.77:34735 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'neutron' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17784.0> (10.0.0.77:34736 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'neutron' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17787.0> (10.0.0.75:54438 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'neutron' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17790.0> (10.0.0.62:39103 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'nova' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17793.0> (10.0.0.62:39104 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'nova' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17796.0> (10.0.0.62:39105 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'nova' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17799.0> (10.0.0.75:54439 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'neutron' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17803.0> (10.0.0.62:39106 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'nova' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17806.0> (10.0.0.62:39107 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'nova' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17809.0> (10.0.0.62:39108 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'nova' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17812.0> (10.0.0.62:39109 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'nova' - invalid credentials",
                             'connection.start_ok'}}

--
closing AMQP connection <0.17815.0> (10.0.0.77:34737 -> 10.0.0.60:5672):
{handshake_error,starting,0,
                 {amqp_error,access_refused,
                             "AMQPLAIN login refused: user 'neutron' - invalid credentials",
                             'connection.start_ok'}}

Looking under /etc/rabbitmq, I don't see any files! Removing and adding relations, per Juju didn't configure rabbitmq for openstack?, did not resolve the problem. Here are some typical diagnostics:

rabbitmqctl list_users
Listing users ...
cinder  []
glance  []
guest   [administrator]
nagios-rabbitmq-server-0    []
neutron []
nova    []
...done.
# rabbitmqctl list_permissions -p openstack
Listing permissions in vhost "openstack" ...
cinder  .*  .*  .*
glance  .*  .*  .*
neutron .*  .*  .*
nova    .*  .*  .*
...done.

rabbitmqctl list_vhosts
Listing vhosts ...
/
nagios-rabbitmq-server-0
openstack
...done.

Any thoughts?
Cheers,

Edit 1: Seems that cloud:trusty-juno charms are not compatible with rabbitmq charm. For cinder and nova-cloud-controller had to set notification_driver = rabbit. Furthermore, had to manually set password, in rabbitmq server, for nova, neutron, and cinder users. In all cases, had to restart relevant services, manually. (Note that in case of cinder and cloud-controller, juju overwrites notification_driver.) Unfortunately, still can't access Access&security page. Keystone is reporting the following:

(keystone.common.wsgi): 2014-11-24 23:51:42,238 WARNING Authorization failed. The request you have made requires authentication. from 10.0.0.62
(keystone.common.wsgi): 2014-11-24 23:51:42,251 WARNING Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

10.0.0.62 being nova-cloud-controller.

Edit 2: (vis-a-vis 1st response.)

# sudo rabbitmqctl cluster_status
Cluster status of node rabbit@rabbitmq1 ...
[{nodes,[{disc,[rabbit@rabbitmq1]}]},
 {running_nodes,[rabbit@rabbitmq1]},
 {partitions,[]}]
...done.

Looking forward to the next release.

Community
  • 1
Nastooh
  • 625

1 Answers1

1

The charm updates to support the Juno OpenStack release are fully compatible with the RabbitMQ charm - we test this on every commit we make to any of the OpenStack charms as part of the day-to-day charm development cycle.

The RabbitMQ (rabbitmq-server) charm should be automatically creating users under the openstack vhost for all related services, and providing a password back via relation data to cinder, nova, neutron etc... This is all done directly in RabbitMQ's underlying state database - so don't expect anything to appear in /etc/rabbitmq!

For some reason, it looks like this password does not correspond with the password in RabbitMQ - you're running in a clustered configuration so I wonder whether there is some issue with the clustering - try running:

sudo rabbitmqctl cluster_status

on one of the RabbitMQ nodes to see if all units are listed; I suspect there may have been some issue with the RabbitMQ clustering set-up, and you might be running in some sort of split brain configuration, so the passwords remote services have only match in a subset of the RabbitMQ units.

I also checked on the notification driver configuration for the cinder and nova-cc charms; although they are using the older deprecated name for the rpc driver, this gets remapped to the correct oslo.messaging (the shared OpenStack messaging library) driver. We'll update this during our next dev cycle to use the oslo.messaging one directly.

jamespage
  • 1,144