CLI Commands for Troubleshooting Infoblox

With Infoblox you’re almost doing everything through the WebUI on the Infoblox Grid Master. At least the daily business such as adding/changing/deleting/moving/whatever DNS, DHCP, and IPAM stuff. Even troubleshooting is almost done through this HTTPS-based GUI. However, some circumstances require the use of the CLI on an Infoblox appliance/VM, called “Remote Console Access” aka SSH. Here are the most common troubleshooting CLI commands for Infoblox DDI. Samples on how to use the IPMI/LOM features round things up:

Note that this blogpost is a living document. Whenever I use some “new” commands for troubleshooting issues, I will update it. In case there are any useful commands missing, please write a comment!

Prerequisites

Unless you’re using the serial console (or the remote console through VMware or the like), you need to enable the SSH access in the GUI through: Grid Manager -> Grid Properties -> Security -> Advanced -> Enable Remote Concole Access.

After that you have remote SSH access with your admin account to any Grid member, either through the MGMT or the LAN1 port, depending on your config. (Note: If you’re using remote authentication such as RADIUS, those outgoing authentication connections will be sourced from the MGMT or LAN1 port as well, again depending on your config.)

Furthermore you should set the prompt to something other than the (annoying) default of “Infoblox >”. This must be done on the Grid master and applies to all members afterwards. Choose one of those:

In HA scenarios you get an additional (A) or (P) depending on the device state. Example:

 

General Stuff

The very basics:

The following “show config …” commands are the same as within the GUI, e.g., Data Management -> DNS -> Members -> View -> View Configuration. However, sometimes it’s a bit easier to have a quick look through the CLI. Only working on the members when the appropriate services are enabled:

Show, tail, or follow log files of different types. Optionally with /regex/:

 

Network ‘n Interfaces

(Note that I was not able to use ping via IPv6 on NIOS version 8.3.4. Maybe a bug?)

 

dig & expertmode dig

There are two slightly different digs on the CLI. The normal one when using it directly on the CLI (which is customized by Infoblox in some way), and the one under “expertmode” which has a couple of more options.

Normal dig:

Expertmode dig:

Traffic Capture & tcpdump

You can easily use the traffic capture within the GUI (Grid -> Grid Manager -> select member -> Traffic Capture):

However, in HA scenarios you can only start/stop and download the traffic capture on the current active node and NOT on the passive one. :( Hence you need to use the CLI. At first you can use the set traffic_capture ... command along with show traffic_capture_status. Finally you need to copy the two captures (one from each cluster member) via set traffic_capture transfer scp <server-ip> <user-name> <user-password> . You cannot specify a server-name but only the mere IP. You can use a dash for the password to have a prompt for it instead of typing it in plain text. Also note that the filenames, unless you specify their names, do NOT tell you on which cluster member they were taken. Example:

Please note that if you’re capturing on “All” interfaces instead of a single one such as LAN1 or HA, you will loose the original Ethernet frame header. Wireshark will only display a “Linux cooked capture” then which includes only the source MAC address but not the destination MAC and so on.

tcpdump

Another way of troubleshooting network/DNS/DHCP issues is to use tcpdump in the hidden expertmode CLI section. With this you can use tcpdump as always except that you can’t write (-w filename) the output somewhere. Hence I am using it with specific capture filters, -v or even -vv, and PuTTY logging. ;)

You need to set the interface via “-i”. The NICs are:

  1. NIC1 = eth0 = MGMT
  2. NIC2 = eth1 = LAN1
  3. NIC3 = eth2 = HA
  4. NIC4 = eth3 = LAN2
Sample run:
 

DNS Related

Aka standard BIND output:

Note that regex version of “show dns cache_ex” which is quite useful. Sample:

 

Anycast Routing (OSPF, BGP)

Easy one:

 

Maintenance Mode

There’s a hidden CLI mode for maintenance purposes. You can enter it via:

Here you can show/delete backups and core dumps:

Sample for showing/deleting a backup:

Sample of coresummary on the Grid master (without any files ;)):

Furthermore you can watch a process list in this maintenance mode via:

Sample run:

 

IPMI aka LOM

Using the Intelligent Platform Management Interface port which is called Lights Out Management on Infoblox you can power on/off the device, get the sensor values, read out the system event log, and finally open a serial console session (which is great!). I am using ipmitool on Linux: sudo apt-get install ipmitool. Here are some samples:

Power off/on/status

Sensors

System Event Log

Serial Console aka Serial over LAN (SOL)

This is very cool. Terminate the session with ~. :

IPMI on Windows

I wasn’t able to use all IPMI commands that easy on Windows. However, using ipmiutil I was able to power off and on the devices:

 

Support Bundle

Always good to know: Download of the support bundle for every single member via Grid -> Grid Manager -> Members -> select member -> Download -> Support Bundle:

Exporting it via SCP on the CLI is this. Use a dash “-” sign for the password to avoid having it in the command history. You have to type it in a new line after you hit enter. And don’t forget to add the keywords to include the logs:

 

Factory Reset

In order to reset the configuration to its defaults you can use one of these commands:

Pound It. Noggin’. Cu!

Featured image “Screwdrivers on white background. Top view” by Marco Verch is licensed under CC BY 2.0.

2 thoughts on “CLI Commands for Troubleshooting Infoblox

  1. This is super helpful, especially for those times I want to quickly check something without going through the whole setting up a webex with support hassle.

    One we had to use recently to see database transactions:
    set debug ibap on

    Then to display the relevant messages:
    show log debug follow /regex/

    And when done:
    set debug ibap off

Leave a Reply

Your email address will not be published. Required fields are marked *