Packet Marking using /etc/shorewall/tcrules

Tom Eastep

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover, and with no Back-Cover Texts. A copy of the license is included in the section entitled “GNU Free Documentation License”.

2009/06/17


Table of Contents

Packet and Connection Marks
Packet Marking "Programs"
Mark and Mask Values
Shorewall-defined Chains in the Mangle Table
An Example
Examining the Marking Programs on a Running System

Caution

This article includes information that applies to Shorewall version 3.2.5 and later. Not all features described here will be available in earlier releases.

Packet and Connection Marks

Perhaps no aspect of Shorewall causes more confusion than packet marking. This article will attempt to clear up some of that confusion.

Each packet has a mark whose value is initially 0. Mark values are stored in the skb (socket buffer) structure used by the Linux kernel to track packets; the mark value is not part of the packet itself and cannot be seen with tcpdump, ethereal or any other packet sniffing program. They can be seen in an iptables/ip6tables trace -- see the iptrace command in shorewal(8) and shorewall6(8).

Each active connection (even those that are not yet in ESTABLISHED state) has a mark value that is distinct from the packet marks. Connection mark values can be seen using the shorewall show connections command. The default connection mark value is 0.

Example (output has been folded for display ):

shorewall show connections
Shorewall 3.3.2 Connections at gateway - Mon Oct  2 09:08:18 PDT 2006

tcp      6 19 TIME_WAIT src=206.124.146.176 dst=192.136.34.98 sport=58597 dport=80
         packets=23 bytes=4623 src=192.136.34.98 dst=206.124.146.176 sport=80 dport=58597
         packets=23 bytes=22532 [ASSURED] mark=256 use=1
…

Packet marks are valid only while the packet is being processed by the firewall. Once the packet has been given to a local process or sent on to another system, the packet's mark value is no longer available. Connection mark values, on the other hand, persist for the life of the connection.

Important

Other parts of the system such as Traffic Shaping and Policy Routing cannot use connection marks — they can only use packet marks.

Packet Marking "Programs"

Packet marking occurs in Netfilter's mangle table. See the Netfilter Overview article.

You can think of entries in the tcrules file like instructions in a program coded in a crude assembly language. The program gets executed for each packet.

That is another way of saying that if you don't program, you may have difficulty making full use of Netfilter/Shorewall's Packet Marking.

Actually, the tcrules define several programs. Each program corresponds to one of the built-in chains in the mangle table.

  • PREROUTING program — If MARK_IN_FORWARD_CHAIN=No in shorewall.conf, then by default entries in /etc/shorewall/tcrules are part of the PREROUTING program. Entries specifying the ":P" suffix in the MARK column are also part of the PREROUTING program. The PREROUTING program gets executed for each packet entering the firewall.

  • FORWARD program — If MARK_IN_FORWARD_CHAIN=Yes in shorewall.conf, then by default entries in /etc/shorewall/tcrules are part of the FORWARD program. Entries specifying the ":F" suffix in the MARK column are also part of the FORWARD program. The FORWARD program gets executed for each packet forwarded by the firewall.

  • OUTPUT program — Entries with $FW in the SOURCE column are part of the OUTPUT program. The OUTPUT program is executed for each packet originating on the firewall itself.

  • POSTROUTING program — Entries with a class-id in the MARK column (and that don't specify $FW in the SOURCE column) are part of the POSTROUTING program. These rules are executed for each packet leaving the firewall. Entries specifying the ":T" suffix in the MARK column are also part of the POSTROUTING program (Shorewall version 3.4.0 and later).

  • INPUT program — No entries in tcrules will add entries to this program. It is executed for each packet that is targeted to the firewall itself.

Note that a packet being forwarded by your firewall actually gets processed by three different programs: PREROUTING, FORWARD and POSTROUTING. Similarly, packets addressed to the firewall itself are processed by two programs (PREROUTING and INPUT) while packets originating on the firewall are likewise processed by two programs (OUTPUT and POSTROUTING).

Rules in each program are executed as follows:

  • Rules are conditionally executed based on whether the current packet matches the contents of the SOURCE, DEST, PROTO, PORT(S), CLIENT PORT(S_, USER, TEST, LENGTH and TOS columns.

  • When a rule is executed, either:

    1. the current packet receives a new mark value; or

    2. the connection to which the current packet belongs receives a new mark value (":C", ":CF" or ":CP" suffix in the MARK column); or

    3. the packet is classified for traffic shaping (class-id in the MARK column); or

    4. the packet mark in the current packet is moved to the connection mark for the connection that the current packet is part of ("SAVE" in the MARK column); or

    5. the connection mark value for the connection that the current packet is part of is moved to the current packet's mark ("RESTORE" in the MARK column); or

    6. jump to a subroutine (another chain in the mangle table). These jumps are generated by Shorewall; or

    7. exit the current subroutine ("CONTINUE" in the MARK column).

  • Unless the subroutine is exited using CONTINUE, the current packet is always passed to the next tcrule in the subroutine.

Mark and Mask Values

The mark value is held in a 32-bit field. Because packet marking is the Netfilter kludge of last resort for solving many hard technical problems, Shorewall reserves half of this field (16 bits) for future use. The remainder is split into two 8-bit values:

  • The low-order eight bits are used for traffic shaping marks. These eight bits are also used for selecting among multiple providers when HIGH_ROUTE_MARKS=No in shorewall.conf. Some rules that deal with only these bits use a mask value of 0xff.

  • The next 8 bits are used for selecting among multiple providers when HIGH_ROUTE_MARKS=Yes in shorewall.conf. These bits are manipulated using a mask value of 0xff00.

As hinted above, marking rules can specify both a mark value and a mask. The mask determines the subset of the 32 bits in the mark to be used in the operation — only those bits that are on in the mask are manipulated when the rule is executed. For entries in tcrules, Shorewall-generated rules use a mask value that depends on which program the rule is part of, what the rule does, and the setting of HIGH_ROUTE_MARKS.

For entries in tcrules, the default mask value is 0xffff except in these cases:

  • RESTORE rules use a default mask value of 0xff.

  • SAVE rules use a default mask value of 0xff.

  • Connection marking rules use a mask value of 0xff.

Shorewall-defined Chains in the Mangle Table

Shorewall creates a set of chains in the mangle table to hold rules defined in your /etc/shorewall/tcrules file. As mentioned above, chains are like subroutines in the packet marking programming language. By placing all of your rules in subroutines, CONTINUE (which generates a Netfilter RETURN rule) can be used to stop processing your rules while still allowing following Shorewall-generated rules to be executed.

tcpre

PREROUTING rules.

tcfor

FORWARD rules.

tcout

OUTPUT rules.

tcpost

POSTROUTING rules.

Shorewall generates jumps to these chains from the built-in chains (PREROUTING, FORWARD, etc.).

An Example

Here's the example (slightly expanded) from the comments at the top of the /etc/shorewall/tcrules file.

#MARK    SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST    LENGTH  TOS
#                                                        PORT(S)
1        0.0.0.0/0       0.0.0.0/0       icmp    echo-request                 #Rule 1
1        0.0.0.0/0       0.0.0.0/0       icmp    echo-reply                   #Rule 2
1        $FW             0.0.0.0/0       icmp    echo-request                 #Rule 3
1        $FW             0.0.0.0/0       icmp    echo-reply                   #Rule 4

RESTORE  0.0.0.0/0       0.0.0.0/0       all     -       -       -       0    #Rule 5
CONTINUE 0.0.0.0/0       0.0.0.0/0       all     -       -       -       !0   #Rule 6
4        0.0.0.0/0       0.0.0.0/0       ipp2p:all                            #Rule 7
SAVE     0.0.0.0/0       0.0.0.0/0       all     -       -       -       !0   #Rule 8
##LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

Let's take a look at each rule:

  1. This straight-forward rule simply marks all 'ping' requests passing through the firewall with mark value 1. Note that it does not mark pings that originate on the firewall itself.

  2. Similarly, this rule marks 'ping' replies.

  3. This rule marks 'ping' requests that originate on the firewall. This rule and the next ones are part of the OUTPUT program.

  4. Similarly, this rule marks 'ping' replies from the firewall itself.

  5. Remember that even though 'ping' packets were marked in one of the first two rules, they are still passed on to rule 5 (note that packets marked by rules 3 and 4 are not processed by this rule since it is in a different program). That rule moves the connection mark to the packet mark, if the packet mark is still zero (note the '0' in the TEST column). Without the '0' in the TEST column, this rule would overwrite the marks assigned in the first two rules.

  6. If the packet mark is non-zero (note the '!0' in the TEST column), then exit — The remaining rules will not be executed in this case. The packet mark will be non-zero if this is a 'ping' packet, or if the connection mark restored in rule 5 was non-zero.

  7. The packet mark is still zero. This rule checks to see if this is a P2P packet and if it is, the packet mark is set to 4.

  8. If the packet mark is non-zero (meaning that it was set to 4 in rule 7), then save the value (4) in the connection. The next time that a packet from this same connection comes through this program, rule 6 will be executed and the P2P check will be avoided.

Examining the Marking Programs on a Running System

You can see the tcrules in action using the shorewall show mangle command.

The sample output from that command shown below has the following in /etc/shorewall/providers:

#NAME   NUMBER  MARK    DUPLICATE       INTERFACE       GATEWAY         OPTIONS         COPY
Blarg   1       0x100   main            eth3            206.124.146.254 track,balance   br0,eth1
#LAST LINE -- ADD YOUR ENTRIES ABOVE THIS LINE -- DO NOT REMOVE

Here is /etc/shorewall/tcrules:

#MARK   SOURCE          DEST            PROTO   PORT(S) CLIENT  USER    TEST
#                                                       PORT(S)
1:110   192.168.0.0/22  eth3                                            #Our internal nets get priority
                                                                        #over the server
1:130   206.124.146.177 eth3            tcp     -       873
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

And here is /etc/shorewall/tcdevices and /etc/shorewall/tcclasses:

#INTERFACE      IN-BANDWITH     OUT-BANDWIDTH
eth3            1.3mbit         384kbit
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

#INTERFACE      MARK    RATE            CEIL            PRIORITY        OPTIONS
eth3            10      full            full            1               tcp-ack,tos-minimize-delay
eth3            20      9*full/10       9*full/10       2               default
eth3            30      6*full/10       6*full/10       3
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE

I've annotated the following output with comments beginning with "<<<<" and ending with ">>>>". This example uses HIGH_ROUTE_MARKS=Yes and TC_EXPERT=No in shorewall.conf.

gateway:~ # shorewall show mangle
Shorewall 3.3.2 Mangle Table at gateway - Mon Oct  2 15:07:32 PDT 2006

Counters reset Mon Oct  2 07:49:52 PDT 2006

<<<< The PREROUTING Program >>>>

Chain PREROUTING (policy ACCEPT 409K packets, 122M bytes)
 pkts bytes target     prot opt in     out     source               destination

<<<< Restore the provider mark from the connection, if any >>>>

 185K   77M CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK match !0x0/0xff00 CONNMARK restore mask 0xff00

<<<< If there is no mark in the connection and the packet came in on eth3, then jump to the routemark chain 
     This rule is generated as a result of 'track' being specified in the providers file entry for eth3 >>>>

 8804 1396K routemark  all  --  eth3   *       0.0.0.0/0            0.0.0.0/0           MARK match 0x0/0xff00

<<<< If the packet came in on eth3, jump the the tcpre chain -- packets entering on a 'track'ed interface can have their mark set to zero there >>>>

 102K   52M tcpre      all  --  eth3   *       0.0.0.0/0            0.0.0.0/0

<<<< Otherwise, jump to the tcpre chain if there is no current provider mark -- 
     if we would have had TC_EXPERT=Yes, this jump would have been unconditional>>>>

 215K   44M tcpre      all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK match 0x0/0xff00

<<<< End of PREROUTING program >>>>

<<<< INPUT Program -- Shorewall generates the single rule here which turns off the provider mark in the packet after routing
                      The rule does that by logically ANDing the mark value with 0xff which will turn off all but the low-order 8 bits >>>>

Chain INPUT (policy ACCEPT 98238 packets, 16M bytes)
 pkts bytes target     prot opt in     out     source               destination
98234   16M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK and 0xff

<<<< End of INPUT program >>>>

<<<< FORWARD Program -- Shorewall generates the first rule here which turns off the provider mark in the packet after routing >>>>

Chain FORWARD (policy ACCEPT 312K packets, 106M bytes)
 pkts bytes target     prot opt in     out     source               destination
 312K  106M MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK and 0xff

<<<< Jump unconditionally to the tcfor chain >>>>

 312K  106M tcfor      all  --  *      *       0.0.0.0/0            0.0.0.0/0

<<<< End of FORWARD program >>>>

<<<< OUTPUT Program >>>>

Chain OUTPUT (policy ACCEPT 1462K packets, 396M bytes)
 pkts bytes target     prot opt in     out     source               destination

<<<< Restore the provider mark from the connection -- this rule was generated by Shorewall because of the 'track' option >>>>

 3339  615K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           CONNMARK match !0x0/0xff00 CONNMARK restore mask 0xff00

<<<< If there is no provider mark, then jump to the tcout chain -- 
     if we would have had TC_EXPERT=Yes, this jump would have been unconditional >>>>

92747   28M tcout      all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK match 0x0/0xff00

<<<< End of FORWARD program >>>>

<<<< POSTROUTING Program -- Unconditionally jump to the tcpost chain >>>>

Chain POSTROUTING (policy ACCEPT 407K packets, 135M bytes)
 pkts bytes target     prot opt in     out     source               destination
 407K  135M tcpost     all  --  *      *       0.0.0.0/0            0.0.0.0/0

<<<< End of FORWARD program >>>>

Chain routemark (1 references)
 pkts bytes target     prot opt in     out     source               destination

<<<< Set connection 'track' mark for packets coming in on eth3 >>>>

 8804 1396K MARK       all  --  eth3   *       0.0.0.0/0            0.0.0.0/0           MARK or 0x100

<<<< Save any mark added above in the connection mark >>>>

 8804 1396K CONNMARK   all  --  *      *       0.0.0.0/0            0.0.0.0/0           MARK match !0x0/0xff00 CONNMARK save mask 0xff00

Chain tcfor (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain tcout (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain tcpost (1 references)
 pkts bytes target     prot opt in     out     source               destination

<<<< The next two rules are the entries in the /etc/shorewall/tcrules file >>>>

65061   11M CLASSIFY   all  --  *      eth3    192.168.0.0/22       0.0.0.0/0           CLASSIFY set 1:110
 2224 2272K CLASSIFY   tcp  --  *      eth3    206.124.146.177      0.0.0.0/0           tcp spt:873 CLASSIFY set 1:130

<<<< The following rules are generated by Shorewall and classify the traffic according to the marks in /etc/shorewall/classes >>>>

    0     0 CLASSIFY   all  --  *      eth3    0.0.0.0/0            0.0.0.0/0           MARK match 0xa/0xff CLASSIFY set 1:110
    0     0 CLASSIFY   all  --  *      eth3    0.0.0.0/0            0.0.0.0/0           MARK match 0x14/0xff CLASSIFY set 1:120
    0     0 CLASSIFY   all  --  *      eth3    0.0.0.0/0            0.0.0.0/0           MARK match 0x1e/0xff CLASSIFY set 1:130

Chain tcpre (2 references)
 pkts bytes target     prot opt in     out     source               destination
gateway:~ #