It was MTU! Zscaler over GRE behind Palo, blocking ICMP Frag Needed

ICMP/ICMPv6, Palo Alto Networks, , , , , , , ,

The other day, I was troubleshooting an issue where users reported that “some websites are working while some are not“. Uh. This is almost the worst scenario to face from a networker’s perspective. It’s way easier if things do or don’t work at all, but not this “some don’t” situation.

The scenario: Using Zscaler for outbound Internet connections, connected via a GRE tunnel from a Palo Alto Networks firewall. TL;DR: If it’s not DNS, it’s MTU. 😂 The “Suppress ICMP Frag Needed” option within the ICMP Drop section of the Zone Protection Profile did what it is meant to do: block “ICMP fragmentation needed” messages. Unfortunately, this killed *some* sessions which had the “Don’t fragment” bit set but exceeded the (lower) MTU of the GRE tunnel.

Continue reading It was MTU! Zscaler over GRE behind Palo, blocking ICMP Frag Needed →

Palo vs. PlayStation: How a Security Feature Blocked Our PlayStation Updates

Palo Alto Networks, , , , , ,

For a few weeks, our PlayStation stopped downloading game updates. I figured it was just a temporary issue with the PS4. Since it didn’t affect me directly but only the kids, I didn’t pay much attention at first. I planned to wait for a firmware update from Sony. When such an update eventually came but didn’t solve the issue, I started getting suspicious – especially when I found almost no relevant results online for the official error code, which reads “(HTTP Status Code : 416) (CE-40862-0)”.

After conducting further detailed searches, I finally came across a post in the Palo Alto Networks LIVEcommunity. That definitely caught my attention. If there’s one thing that sets my home network apart from most “normal” households, it’s the fact that I have a Palo Alto firewall running – not your average consumer-grade router. 😂

Continue reading Palo vs. PlayStation: How a Security Feature Blocked Our PlayStation Updates →

Redundant VPN with Failover on a Palo NGFW

Internet Access, IPsec/VPN, Palo Alto Networks, Tutorial/Howto, , , , , ,

This goes out to anyone who uses more than one Site-to-Site VPN tunnel between two locations that are secured by firewalls from Palo Alto Networks. Using two (or even more) VPN tunnels, you need an automatic way to failover the traffic flow from one VPN to the other in case of failures. Here’s how to accomplish that requirement:

Continue reading Redundant VPN with Failover on a Palo NGFW →

Wireshark Feature Added: Connecting ICMP Errors

ICMP/ICMPv6, Wireshark, , , ,

It’s really just a small thing, but very practical for me: In Wireshark, a feature request I submitted has been implemented. Now, when you click on an ICMP error, the corresponding (original) packet is highlighted.

Previously, clicking on a packet belonging to a flow would show all related packets, including any ICMP errors. However, if you selected an ICMP error packet itself, nothing happened. If you had many ICMP errors from different sessions, you had to go through the cumbersome process of figuring out which sessions they actually belonged to.

Now, you can simply scroll through the packet list as usual and immediately see whether related packets are present — and if so, which ones. Very handy.

Continue reading Wireshark Feature Added: Connecting ICMP Errors →

Quizauflösung: Fehlersuche mittels ICMP-RĂŒckmeldungen

ICMP/ICMPv6, , , , , , , ,

Die Fehlersuche in IP-Netzwerken fĂ€llt nicht leicht, denn einem Netzwerkschluckauf können viele Ursachen zugrunde liegen. Profi-Admins kennen aber Wege, um das klassische und meist aufwendige Troubleshooting abzukĂŒrzen. Beispielsweise kann man Fehlerquellen anhand von ICMP-RĂŒckmeldungen der NetzwerkgerĂ€te eingrenzen, die an einem fehlgeschlagenen IP-Dialog beteiligt sind. Welche Meldungen das sind und wie man sie interpretiert, haben wir hier ausfĂŒhrlich beschrieben.

Am Ende dieses Beitrags haben wir vier Netzwerkanalyse-Aufgaben gestellt. Die Grundlage dafĂŒr bildet ein Verkehrsmitschnitt, den man mit dem Analysetool Wireshark öffnet und mit einem Display-Filter siebt. Hier folgen die Antworten zu den Aufgaben.

Continue reading Quizauflösung: Fehlersuche mittels ICMP-RĂŒckmeldungen →

ICMP-Meldungen zur Fehlersuche im Netz einspannen

ICMP/ICMPv6, Internet Access, Routing, , , , , , , , , , , ,

Sie sind Admin und Ihr Netz krĂ€nkelt. Wo fangen Sie an mit der Fehlersuche? Unser Tipp: Tasten Sie Ihre Netzwerkpatienten mal nach ICMP-Symptomen ab. Viele fĂŒhren direkt zur Ursache.

Wenn man Netzwerkschluckauf behandeln muss, gilt Wireshark als eines der Lieblingswerkzeuge von Netzwerkadmins. Denn falsch angestöpselten oder fehlkonfigurierten Servern kommt man oft schon anhand eines Netzwerkmitschnitts auf die Spur und erspart sich so den Adminzugriff auf Abteilungsrouter oder -switches. Als behandelnder Admin mĂŒssen Sie das aufgefangene Paketkonfetti nur noch mit einem geeigneten Display-Filter sieben, um jene Paketsorte im Kescher zu behalten, die Fehlerhinweise gratis unter Ihre wissenden Augen bringt: die ICMP-PĂ€ckchen.

Continue reading ICMP-Meldungen zur Fehlersuche im Netz einspannen →

ICMP ‘Destination Unreachable’ Messages @ SharkFest’24 EU

Conference Talks, ICMP/ICMPv6, Internet Access, Wireshark, , , , , , ,

I did a presentation at SharkFest’24 EU in Vienna, the “Wireshark Developer and User Conference“, about the topic: “Unveiling Network Errors – A Deep Dive into ICMP ‘Destination Unreachable’ Messages“. It covers the following:

“Effective troubleshooting of network issues is a critical concern for network technicians. While many are familiar with basic ICMP tools like ping and traceroute, the breadth of ICMP capabilities often goes underutilised. This session delves into ICMP messages, specifically the ‘Destination Unreachable’ type, and the insights they provide into network errors.

We will explore methods for capturing and analysing network traffic, highlighting practical tips and tricks for using Wireshark to diagnose and resolve issues efficiently. Attendees will gain a deeper understanding of ICMP message functions and how to leverage them for improved network troubleshooting.”

You can watch the whole session and download the slides. And you can do the six challenges at the end of the session as well. (The answers are not in the PDF, but shown in the video.)

Continue reading ICMP ‘Destination Unreachable’ Messages @ SharkFest’24 EU →

Security-as-a-Podcast

Conference Talks,

Es ist so weit – mein Kollege Florian und ich haben einen Podcast gestartet! 🎉

Der Security-as-a-Podcast beschĂ€ftigt sich mit Netzwerksicherheit – aber nicht im klassischen „Wir erklĂ€ren euch die RFCs“-Stil, sondern eher aus der Perspektive von zwei Menschen, die selbst regelmĂ€ĂŸig mit Security-Buzzwords bombardiert werden und manchmal nur halb wissen, worum es wirklich geht.

Begriffe wie SASE, ZTA oder OT begegnen uns im Alltag stĂ€ndig – in Projekten, in Meetings, auf Konferenzen. Doch wie oft bleibt davon wirklich was hĂ€ngen? Und was davon ist mehr Marketing als Substanz?

In unserem Podcast sprechen wir genau darĂŒber – miteinander und mit Experten. Ziel ist es, Wissen zu entwirren, einzuordnen und so aufzubereiten, dass es greifbar und verstĂ€ndlich wird – und im besten Fall auch im Kopf bleibt. Vom Business-Case hin zum Tool.

Continue reading Security-as-a-Podcast →

Editing Palo Configs by Scripts: pan-os-php

Palo Alto Networks, Tutorial/Howto, , , , , , ,

There are recurring cases where tasks cannot be edited quickly and easily using the classic Palo Alto Networks GUI or Panorama. For example, editing multiple policies at once, such as during a zone migration. Or checking which policies haven’t log forwarding enabled, hence enabling it directly. Or finding unused objects, including deleting them.

For these situations (and many more!), there’s a tool with a wealth of predefined scripts: pan-os-php. This first blog post covers installation and some initial use cases.

Continue reading Editing Palo Configs by Scripts: pan-os-php →

Azure PTP Accuracy

Microsoft Azure, NTP, , ,

The Network Time Protocol (NTP) is widely used to synchronize computer clocks. The Precision Time Protocol (PTP) can be used as a time source as well, which is expected to be accurate within microseconds. However, at Microsoft Azure VMs, PTP-derived time-of-day errors could exceed 50,000 microseconds, which may be inadequate. Let’s go into some details:

Continue reading Azure PTP Accuracy →

Bad IPv6 Approaches

IPv6, , , , , , , , , , ,

I just got a few emails from an administrator of a medium-sized company, asking some IPv6 questions. They want to use IPv6 to reach the Internet, using two ISPs, while remaining IPv4-only on their internal networks. For whatever reason, they came across three different ideas that were almost completely wrong, speaking of a sound IPv6 design. But why? Maybe because IPv4 thinking is a bigger problem than we ever thought? Or because admins rely on firewall vendors (like Fortinet) that suggest completely wrong network approaches?

Let’s dig into some misconceptions concerning IPv6:

Continue reading Bad IPv6 Approaches →

Palo Alto Networks Announces Strategic Shift to Apparel Manufacturing

Palo Alto Networks, , , ,

Palo Alto Networks, a global leader in cybersecurity solutions, has announced a significant strategic shift. The company will transition from its core cybersecurity business to exclusively focus on apparel manufacturing.

Continue reading Palo Alto Networks Announces Strategic Shift to Apparel Manufacturing →

Path MTU Discovery

ICMP/ICMPv6, Internet Access, Routing, , , , , , ,

One of the mysteries for me in IP networks was the Path MTU Discovery (PMTUD) process. I’ve seldom seen any problems with the MTU at all. Fortunately, while troubleshooting some router issues, I captured several ICMP “packet too big” errors along with the original packets. đŸ‘đŸ»

Let’s have a look at those PMTUD processes for IPv6 and legacy IP with Wireshark. Of course, these captured connections are part of the Ultimate PCAP as well, hence, you can download the most current version of it and analyze it by yourself.

Continue reading Path MTU Discovery →

Which KPIs to monitor on a Palo Alto Firewall?

Monitoring, Palo Alto Networks, , , , , ,

We wanted to monitor some of our Palo firewalls from our monitoring system via the API. But: Which enhanced metrics/KPIs shall we monitor? While there are some obvious ones such as interface counters, uptime, software versions, license expiry dates, or HA-states, we dug a little deeper to get more out of it, such as mgmt-/data-plane stats, packet rates, drop counters (all global counters?), and routing entries.

Here are some ideas on which values a monitoring system could observe. I’m listing the required API calls along with some demo values that can be used to develop monitoring tools/scripts.

Continue reading Which KPIs to monitor on a Palo Alto Firewall? →