Boundary Client Agent

This feature requires HCP Boundary or Boundary Enterprise

Beta feature

Beta functionality is stable, but possibly incomplete and subject to change. We strongly discourage using beta features in production deployments of Boundary.

The Boundary Client Agent is Boundary's DNS daemon. When the Boundary Client Agent runs alongside an authenticated Boundary client, the agent establishes itself as the primary resolver on the system and intercepts DNS requests.

If you enter a hostname that matches a Boundary alias that the client is authorized to establish a session to, Boundary automatically generates the session and transparently proxies the connection on your behalf. If the Boundary Client Agent cannot find an alias, or if there are any issues with authentication, network connectivity, or latency, the Client Agent defers DNS resolution to the previously configured DNS resolvers.

Security

When you successfully authorize a session on a Boundary controller, the response includes a list of any brokered credentials, which include the decoded secrets. When the Boundary Client Agent receives a DNS request, Boundary creates a new session. An OS user can only connect to an authorized session managed by the Boundary Client Agent daemon if they are the same OS user that added the Boundary auth token used for authorizing the session.

Note

Currently, you cannot authenticate to multiple Boundary controllers at once. If you authenticate to a different Boundary controller, any existing sessions are terminated and any new transparent sessions would be established with the new controller.

The Boundary Client Agent stores the credentials and some other information related to the session in memory. The in-memory store removes the session information:

when the session ends.
if the auth token stored in the Boundary expires.
if the current OS user authenticates with a different Boundary user.
if the current OS user authenticates to a new Boundary controller.
if the Boundary Client Agent is paused.
if the Boundary Client Agent is terminated.

API requests are authenticated in the same way as session proxy access.

Credential brokering is supported for transparent sessions. A notification appears when you establish a session against a target that is configured with credential brokering. You can retrieve the credentials later using the following command:

$ boundary client-agent sessions

Grants

The default grants that Boundary creates for anonymous and authenticated users are sufficient to get started with the Client Agent for the public beta. However, in a production scenario, you may want to provide the least amount of privileges necessary for users. For a Boundary user to be able to use the Client Agent to establish a transparent session, they must:

be able to authenticate using an auth method.
have read permissions for their auth token.
have permission to establish a session to one or more targets.

You can use the following grant strings to grant those permissions:

type=auth-method;ids=*;actions=authorize
type=target;ids=*;actions=authorize-session
type=auth-token;ids=*;actions=read:self

HashiCorp highly recommends that you also grant users the permission to list resolvable aliases, as the Client Agent periodically fetches a list of aliases to match incoming DNS requests against. Without that permission, every DNS request on the system is sent to the Boundary controller, which can easily overwhelm it. You can use the following grant string to grant the permission to list resolvable aliases:

type=user;ids=*;actions=list-resolvable-aliases

Configuration

The default configuration included with the Boundary Client Agent upon installation will be suitable for most users. If you want to make changes to the configuration, the configuration file is located in the following directory:

/Library/Application Support/HashiCorp/Boundary/boundary-client-agent.hcl

The configuration file contains the following fields:

alias_refresh_interval - Specifies how often to refresh the alias cache. The default value is 1 minute.
Example:
```
alias_refresh_interval=60s
```
dns_request_timeout - Specifies for how long the Client Agent DNS request handling, including any recursion, is allowed to run before it is canceled.
Example:
```
dns_request_timeout=300s
```
interface_to_use - Specifies the interface to use instead of the default.
Example:
```
interface_to_use=en1
```

log_file - Specifies where to write the Boundary Client Agent log file to.

Example:

log_file="/Library/Application\ /Support/HashiCorp/Boundary/boundary-client-agent.log"

log_level - Specifies the verbosity of the Client Agent logs.
Example:
```
log_level="DEBUG"
```
log_to_stdout - Logs to STDOUT in addition to the boundary-client-agent.log file.
Example:
```
log_to_stdout=false
```
override_upstream_dns_servers - Lists the DNS servers that should be used for recursing non-Boundary requests, overriding those configured on the system.
Example:
```
override_upstream_dns_servers = ["8.8.8.8", "8.8.4.4"]
```
state_file - Specifies where to write the Boundary Client Agent state file to. This is an ephemeral file which is removed on successful shutdown.
Example:
```
state_file="/Library/Application\ /Support/HashiCorp/Boundary/boundary-client-agent-state.json"
```
v4_prefix - Specifies an alternate prefix to use for generating IPs. Currently must be between /8 and /16
Example:
```
v4_prefix=1.1.1.1/8
```

Change the configuration

Complete the following steps to change the configuration of the Client Agent:

As a privileged user, open the Boundary Client Agent configuration file in the editor of your choice. By default, it is located in the following directory:
/Library/Application Support/HashiCorp/Boundary/boundary-client-agent.hcl
Change the configuration settings, and save the file.
Note
You must restart the Client Agent to update some configuration settings. However, when you restart the Client Agent, it closes any existing sessions. Other configuration settings can be updated by only reloading the configuration file, which does not affect any existing sessions.
Either reload the configuration file or restart the Client Agent.
You can change the following configuration values by reloading the configuration file, which will not disrupt any existing sessions:
Run the following command to reload the configuration file:
```
$ sudo pkill -1 boundary-client-agent
```
If you want to update another configuration value, you can restart the Client Agent using the following commands, however it will close any existing sessions:
```
$ sudo launchctl stop com.hashicorp.boundary.boundary-client-agent
$ sudo launchctl start com.hashicorp.boundary.boundary-client-agent
```

As a privileged user, open the Boundary Client Agent configuration file in the editor of your choice. By default, it is located in the following directory:
C:\Program Files\Hashicorp Boundary\boundary-client-agent.hcl
Change the configuration settings, and save the file.
Run the following commands to restart the Client Agent.
```
net stop BoundaryClientAgent
net start BoundaryClientAgent
```
Note that when you restart the Client Agent, it closes any existing sessions.

Manage the Client Agent

Refer to the following sections for more information about managing the Client Agent. You can monitor the Client Agent's status and retrieve information about any transparent sessions. If you want to temporarily defer DNS resolution to any previously configured DNS resolvers, you can pause the Client Agent. You can also disable the Client Agent, if you no longer want to use it for DNS resolution.

Monitor status and sessions

You can check the status of the Client Agent to ensure it is running. Use the following command to check the Client Agent's status:

$ boundary client-agent status

You can retrieve information about the sessions that the Client Agent is managing. Use the following command to list any sessions currently being managed by the Client Agent, as well as any brokered credentials for those sessions:

$ boundary client-agent sessions

Note that this command does not list sessions that are not managed by the Client Agent. Use boundary sessions list -recursive to see all sessions.

Pause the Client Agent

You can temporarily disable the Boundary Client Agent by pausing it with the following command:

$ boundary client-agent pause

When the Client Agent is paused, it does not intercept any DNS requests, and you are unable to use transparent sessions.

To resume the Client Agent, use the following command:

$ boundary client-agent resume

Disable the Client Agent

If you want to disable the Boundary Client Agent, you can stop it with the following commands:

$ sudo launchctl stop com.hashicorp.boundary.boundary-client-agent

Troubleshooting

The following sections can help you to troubleshoot the Client Agent's behavior. You should proceed through these steps from top to bottom.

Check the status of the Client Agent

If you experience unexpected behavior, you should first check on the status of the Client Agent. You can check the status using the Boundary CLI or the Desktop Client. To check the Client Agent status through the Boundary CLI, use the following command:

$ boundary client-agent status

Status:
  Address:
  https://boundary.corp.com
  Auth Token Expiration:   167h58m9s
  Auth Token Id:           at_GBqZUK2ihv
  Status:                  running
  Version:                 0.0.1-e561e69839cce148ee5045684bce5b7168c65026

In the Desktop Client, you can find the status of the Client Agent by navigating to Settings, and then scrolling to the Boundary Client Agent section.

The status command includes various information about the Client Agent, including the runtime status. In this example, the runtime status is "running". If the status is "paused", the Client Agent is not currently intercepting DNS requests and must be resumed. Users can pause the Client Agent, and it will also pause itself if it detects a large number of network failures in a short period of time.

The status also allows you to see whether the current user is authenticated. If the response looks like the example above, including showing an auth token ID and expiration, your current user is authenticated. If not, you may need to first authenticate to the Client Agent using the CLI or Desktop Client.

The status also sometimes contains a list of errors that have been encountered by the Client Agent. The list is ordered by most recent first. These errors can help you understand why the Client Agent may not be behaving as expected. Please see the section below on commonly seen errors to help diagnose specific errors. Note that this list of errors will not be cleared until the next reboot, so it may not necessarily be a sign of something being wrong.

If the status command returns an error, the Client Agent may not be running. You can attempt to start the Client Agent using the following commands:

$ sudo launchctl start com.hashicorp.boundary.boundary-client-agent

Resume the Client Agent

You can resume the Client Agent using either the Boundary CLI or the Desktop Client. In the CLI, run the following command to resume the Client Agent:

$ boundary client-agent resume
The Client Agent has been successfully resumed.

In the Desktop Client, you can resume the Client Agent by selecting the Resume button in the Boundary Client Agent section of the settings. Once the Client Agent has resumed, test if it has started working as expected again.

Inspect the log file

If you are not able to diagnose the problem by looking at the status or resuming the Client Agent, another step can be to inspect the log file produced by the Client Agent.

The log file should be located in /Library/Application Support/HashiCorp/Boundary/boundary-client-agent.log.

Once you have found the log file, you can look through it to see if you can understand why the Client Agent is not working as expected. The list below provides some common errors and explanations.

It may be necessary to increase the logging verbosity of the Client Agent. You can increase the verbosity by setting the log_level option in the configuration file to "DEBUG". See the section on changing the configuration for more information.

Establish the behavior of the local DNS configuration

The Client Agent works by intercepting DNS requests before they are sent to your regular DNS server. If the DNS requests on your system are not sent to the right place, or they are not being answered appropriately, transparent sessions will not work.

You can use the nslookup command to understand where the DNS requests are being sent. Start by sending a DNS request for hashicorp.com:

$ nslookup hashicorp.com
;; Truncated, retrying in TCP mode.
Server:     100.88.241.86
Address:    100.88.241.86#53

Non-authoritative answer:
Name:   hashicorp.com
Address: 76.76.21.21

The important part here is the Server field, which contains an IP in the CGNAT range (from 100.64.0.0 to 100.127.255.255). This is a good indication that the Client Agent DNS server is being used as expected.

Next, you can try to make a DNS request to an alias that you expect to work. The following example makes a DNS request to an alias with a value of mytarget.boundary.dev:

$ nslookup mytarget.boundary.dev
;; Truncated, retrying in TCP mode.
Server:         fc00:a20a::d7bf:c059
Address:        fc00:a20a::d7bf:c059#53

Name:    mytarget.boundary.dev
Address: 100.84.164.9

You can tell two things from this:

The Client Agent is likely able to intercept the DNS request, because the server is a local IPv6 address in the ULA range. Both an IPv4 CGNAT range or IPv6 ULA range IP address are indications of this.
The Client Agent is able to identify mytarget.boundary.dev as an alias with a target that the requesting user is authorized to connect to, because it responded with a valid DNS response pointing to a local IPv4 address in the CGNAT range. Similarly to above, the IP address in the response may also be an IPv6 address in the ULA range.

If you do not see this kind of response, it may be that the alias you are trying to connect to doesn't exist, or your user is not authorized to connect to it. Double check that you are using the correct alias and that your user is authorized to connect to it.

Flush OS DNS cache

If you still do not see the expected behavior, it can be useful to flush the operating system's DNS cache. The exact steps depend on the operating system you use:

$ sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder

After you flush the DNS cache, try connecting to the alias again, or repeat the steps above.

If you are still not able to understand what is wrong, submit a support ticket. Include the boundary client-agent status output and the log file in the ticket.

Commonly seen errors

Refer to the following commonly seen errors for more information about their possible causes and resolutions.

nodename nor servname provided, or not known / No such host is known

This is a generic error for a failed DNS resolution. It can mean a number of different things:

The alias doesn't exist or is misspelled.
Your user isn't authenticated or doesn't have permission to connect to the target.
The Client Agent is not able to intercept DNS requests, it could shut down or paused.
The OS DNS cache is interfering with the operation of the Client Agent.
The Client Agent may not yet know about the alias. It takes around 2 minutes for the Client Agent to learn about new aliases.

Follow the troubleshooting steps above to resolve the issue.

failed to listen for DNS on either IPv4 or IPv6

This error happens when some other application on the local machine occupies the ports used by the Boundary Client Agent. The Client Agent requires access to port 53 for IPv4 and IPv6, both UDP and TCP. Diagnosing what causes the error differs per operating system:

As a privileged user, you can use the lsof program to find what applications are occupying a port, for example:

$ sudo lsof -nP | grep ":53"

If anything is occupying port 53, you may need to terminate the application before the Client Agent is able to start.

Applications that make use of the Apple Virtualization Framework are known to sometimes occupy this port under the name _mdnsresponder. If you have any virtualization software, you may need to turn it off before using the Client Agent.

Once you have identified which other software is using the port, you can stop it and try to start the Client Agent again.

failed to refresh alias cache: error="fetching resolvable aliases: error performing client request during List call"

This usually implies that there is a problem reaching the internet or the Boundary controller. The error is related to the periodic updating of aliases used by the Client Agent to know whether a DNS request matches an alias or not.

HashiCorp recommends that you pause the Client Agent and examine the status and logs for further errors:

$ boundary client-agent pause

Follow the troubleshooting steps to understand why the Client Agent is not able to reach the controller.

WARNING! Remote host indentification has changed! It is possible that someone is doing something nasty!

This error arises when you use an alias to connect to an SSH target after the first successful connection using that alias. The issue occurs because Boundary workers generate a new host key on every new SSH connection. You can safely ignore the warning using the StrictHostKeyChecking=no command line option:

$ ssh -o StrictHostKeyChecking=no targetalias.boundary.dev

You can also remove the existing server host key from the ~/.ssh/known_hosts file to avoid the error.

Conflicting software

Some software is known to cause conflicts with the Boundary Client Agent. The following sections are an incomplete list of potential conflicts and any available workarounds for issues.

Docker Desktop (MacOS)

Docker Desktop sometimes creates a local DNS listener that prevents the Client Agent from running. If you run Docker Desktop 4.26 or later, you must clear the Use kernel networking for UDP option. Otherwise, the Client Agent refuses to start.

Palo Alto Networking Global Protect VPN

If you are unable to establish a transparent session while using the Palo Alto Networking Global Protect VPN, you may need to explicitly specify a network interface and the upstream DNS server(s) to use.

By default, the Client Agent reads the primary network interface's DNS server configuration and uses that information to resolve domains that are not configured as aliases in Boundary. If the VPN configuration includes custom DNS servers, this information may not be available to the Client Agent, so you must explicitly specify the DNS server(s) to use.

To configure the DNS server(s) to use, use the override_upstream_dns_servers configuration option:

# The DNS servers must be specified as an IP, or an IP:Port.
# If no port is provided, port 53 is assumed.
# The order of the entries specifies the priority.
# We recommended providing both the VPN DNS servers
# and the default DNS servers, so that DNS requests can
# be resolved even when the VPN is not active.
override_upstream_dns_servers = [
   "10.0.0.1", # Example primary VPN DNS server
   "10.0.0.2", # Example secondary VPN DNS server
   "8.8.8.8",  # Fallback default DNS server
   "8.8.4.4:53", # Fallback default DNS server with a custom port
]

Note

The override_upstream_dns_servers is used for all non-Boundary DNS requests. If you only provide the VPN DNS servers, the Client Agent will not be able to resolve any DNS requests when the VPN is not active.

Primary network interfaces

By default, the Client Agent creates IPs on the primary network interface to serve its DNS server. Refer to the tabs below for possible conflicts for each supported operating system.

When you run the Client Agent alongside the PAN-GP VPN, the primary network interface will likely be set to a tun type interface, which the Client Agent cannot use for its IP addresses. You may see errors such as the following in the boundary-client-agent.log file or the boundary client-agent status command response:

[ERROR] macos.addIP: error adding ipv4 address: ifconfig: ioctl (SIOCAIFADDR): Destination address required

To work around the default tun interface, you must provide an explicit network interface using the interface_to_use configuration option. For example:

interface_to_use=en0

The interface_to_use option allows the Client Agent to create the IPs it needs to serve the DNS server and proxy traffic. You must restart the Client Agent for it to update its configuration with the new setting.

On Windows, the Client Agent may be able to create the IPs that it needs on the primary network interface, but it fails to establish any transparent sessions. You may see the following message:

[INFO] default route change detected, restarting

You must explicitly specify a network interface to use other than the primary one. You can list available network interfaces using the Powershell command Get-NetAdapter, or the older route print command. You must find the index of the interface you would normally use to connect to the internet. In this example, the interface index is 11:

PS C:\> Get-NetAdapter
Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
Ethernet                  Parallels VirtIO Ethernet Adapter            11 Up           00-1C-42-B3-F2-75        10 Gbps
Ethernet 2                PANGP Virtual Ethernet Adapter Secure        24 Up           02-50-41-00-00-01         2 Gbps

Alternatively, if you use route print, refer to the following example:

PS C:\> route print
===========================================================================
Interface List
 24...02 50 41 00 00 01 ......PANGP Virtual Ethernet Adapter Secure
 11...00 1c 42 b3 f2 75 ......Parallels VirtIO Ethernet Adapter
  1...........................Software Loopback Interface 1
===========================================================================

Your configuration should look like this:

interface_to_use=11

You must restart the Client Agent for it to update its configuration with the new setting.

Cloudflare WARP client

The Cloudflare WARP client uses a local DNS server to direct traffic. It has built-in checks to prevent it from being run alongside other software that uses the same mechanism. This includes the Boundary Client Agent. If you try to use the Client Agent with the Cloudflare WARP client, it may work, or you may see an error like this one:

Status: Unable to Connect
Error reason: DNS Proxy Failure
Error code: CF_DNS_PROXY_FAILURE
Error description: The WARP Agent must be the only process responsible for DNS resolution on the device. One or more processes are already bound to port 53: boundary-client-agent.
Learn more: https://cfl.re/CF_DNS_PROXY_FAILURE

You can still install both the Cloudflare WARP client and the Boundary Client Agent on the same machine. As long as you don't run both at the same time, they should work as expected.

Uninstall the Client Agent on Mac

If you used the Mac installer, you can run /Library/Application Support/HashiCorp/Boundary Uninstaller.app to uninstall Boundary. The uninstaller removes any installed components, including the Desktop client, CLI, and the Boundary Client Agent.

More information

Refer to the following topics for more information: