gNMI Telemetry: Dial-In vs Dial-Out for Networks - 夜莺博客

gNMI Telemetry: Dial-In vs Dial-Out for Networks

Streaming telemetry replaces polling with push: instead of asking a device for 10,000 counters every five minutes, the device sends the counters you care about as they change. The design decision that matters most is not the model or the collector - it is who initiates the connection. Dial-in and dial-out produce very different operational behaviour after a reboot, and mixing them up is how you end up with a monitoring gap nobody notices. This article compares the two models as documented for IOS XE and Junos, with the configuration each one needs.

Two Directions, Two Owners

  • Dial-in (dynamic): the collector is the client. It opens NETCONF or gNMI towards the device, subscribes with the ietf-yang-push model, and the device streams to that session. The subscription lives and dies with the session and leaves no trace in the running configuration.
  • Dial-out (configured, "gRPC tunnel"): the device is the client. Subscriptions are part of the configuration, with a fixed subscription ID and a destination, and the device reconnects to the collector after a reload or a stateful switchover.

The two names hide a deeper difference: who owns the session state. With dial-in, session state lives in the collector's process memory. Restart the collector and every subscription disappears; restart the device and the collector must notice the TCP teardown and re-establish. With dial-out, the subscription is a configuration object that a state machine on the device re-applies over and over. A device that reboots at 03:00 with dial-out configured is back on the collector before your first coffee; the same device with only dial-in subscriptions is silent until something re-subscribes it.

The Subscription Model Underneath the Two Names

Whichever direction you choose, the payload is described by the same frameworks, and understanding them makes the configuration options obvious.

  • gNMI (OpenConfig): defines a Subscribe RPC with three modes. ONCE returns the current state of the requested paths and then closes with a sync_response. POLL answers one snapshot per collector request. STREAM keeps pushing, and has three sub-modes: ON_CHANGE, SAMPLE (with a sample interval in nanoseconds), and TARGET_DEFINED, where the device decides the right cadence for each path.
  • NETCONF YANG-Push (RFC 8639 and RFC 8641): the same idea expressed as an establish-subscription RPC with a stream of yang-push, an XPath or subtree filter, and either a periodic period or an on-change trigger with dampening.
  • Subscription IDs: every accepted subscription returns an ID. On dial-in you use it to cancel cleanly; on dial-out the ID is something you choose in configuration (for example 101), which is why dial-out subscriptions are easy to audit but also easy to collide with someone else's numbering scheme.
  • Encodings: gNMI can carry protobuf, JSON_IETF (RFC 7951) or, on IOS XE dial-out, the compact Google-protobuf "KV-GPB" form configured as encode-kvgpb. KV-GPB is dramatically cheaper for the device to build and typically three to eight times smaller on the wire, which matters once you stream interface counters from hundreds of ports. JSON is the friendliest for debugging; use it in the lab and for small path sets.
  • Timestamps and freshness: gNMI updates carry device-side timestamps. Compare them against collector wall-clock time to detect a device whose counters are lagging; a delta that grows steadily is the earliest sign of a data-plane control path problem, long before an alert fires.

Behaviour You Must Design Around

Aspect Dial-In Dial-Out
Initiator Collector Device
Interface NETCONF, gNMI gRPC (gNMI over gRPC tunnel)
Survives device reload No - subscribers must re-subscribe Yes, reconnects automatically
Visible in running config No Yes, fixed subscription ID
Fan-out to several collectors One per session Configured receivers can be multiple
Access model Low-privilege users can subscribe ad hoc Change control applies to the config
Typical transport port TCP 830 (NETCONF), TCP 57400 (gNMI) TCP 57500 to the collector by convention
Escapes the management framework Only if the collector is well behaved No - a reboot is a controlled, logged event
Blast radius if misconfigured One session, one collector Every device in the change window

That last row is the one that catches teams out. A bad dial-in subscription costs you one collector session. A bad dial-out change pushed to a thousand switches simultaneously costs you the device CPU on a thousand switches simultaneously.

Which Model Does Your Platform Actually Support?

The feature names differ per vendor, and support has moved quickly across releases, so confirm against the release notes for your exact train before designing around a capability.

Platform Dial-in Dial-out
IOS XE (16.12 and 17.x) NETCONF yang-push; gNMI dial-in on later 17.x releases, listening on TCP 57400 Configured gRPC tunnel with telemetry ietf subscription
Junos gNMI dial-in via the extension-service gRPC server Junos Telemetry Interface, set services analytics
Arista EOS gNMI dial-in with OpenConfig and EOS-native paths gRPC dial-out, commonly through the OpenConfig streaming agent
SONiC gNMI server in the telemetry container Limited; usually consumed through the same gNMI endpoint

Configuring Dial-Out on IOS XE

telemetry ietf subscription 101
 encoding encode-kvgpb
 filter xpath /process-cpu-ios-xe-oper:cpu-usage/cpu-utilization/five-seconds
 source-address 192.0.2.10
 stream yang-push
 update-policy periodic 2000
 receiver ip address 192.0.2.50 57500 protocol grpc-tcp

The 2000 is milliseconds between updates, not seconds - a classic off-by-1000 mistake that leaves a collector flooded. Verify with:

show telemetry ietf subscription all
show telemetry ietf subscription 101 detail
show telemetry internal connection

Three habits make IOS XE dial-out survivable at scale. First, always set source-address to the address the collector will dial back into, and keep that address inside the management VRF; a loopback that moves is a subscription that silently stops arriving. Second, prefer on-change for anything that changes slowly (BGP peer state, hardware alarms) and reserve periodic sampling for counters that grow monotonically:

telemetry ietf subscription 102
 encoding encode-kvgpb
 filter xpath /interfaces/interface/state/counters
 source-address 192.0.2.10
 stream yang-push
 update-policy on-change
 receiver ip address 192.0.2.50 57500 protocol grpc-tcp
 receiver ip address 192.0.2.51 57500 protocol grpc-tcp

Third, remove subscriptions explicitly rather than editing them in place. Deleting and re-adding a subscription forces a clean re-establishment with the collector; editing the encoding or filter of a live subscription has historically produced receivers that keep streaming the old schema until the tunnel bounces.

no telemetry ietf subscription 102
show telemetry internal subscription-statistics

For encrypted transport, replace grpc-tcp with grpc-tls and pin the collector's certificate. Most production deployments start on an isolated management segment with plain gRPC and move to TLS once the certificate lifecycle is automated, which is a reasonable trade - but only because the management segment is not reachable from user networks.

Configuring Dial-Out on Junos

Junos calls the same concept the Junos Telemetry Interface and expresses it with three separate objects - a streaming server, an export profile, and a subscription that ties sensors to both:

set services analytics streaming-server COLLECTOR1 remote-address 192.0.2.50
set services analytics streaming-server COLLECTOR1 remote-port 57500
set services analytics export-profile PROF1 reporting-interval 10
set services analytics export-profile PROF1 format gpb
set services analytics sensor IFSTATS resource /junos/system/linecard/interface/
set services analytics subscription SUB1 local-address 192.0.2.10
set services analytics subscription SUB1 sensor IFSTATS
set services analytics subscription SUB1 export-profile PROF1
set services analytics subscription SUB1 streaming-server COLLECTOR1

Here the reporting interval is in seconds, the opposite convention to IOS XE - one more reason to document the unit next to every subscription in your repository. Verification is a three-level drill-down:

show services analytics streaming-server
show services analytics export-profile
show services analytics subscription
show agent sensors

The show agent sensors output is the useful one: it shows whether the sensor is actually running and incrementing, which separates "the device is not collecting" from "the device is collecting but not exporting".

Subscribing Dial-In with gNMI

# gnmic subscribe to interface counters, 10s, once the session is up
gnmic -a 192.0.2.10:57400 -u admin -p '***' --insecure subscribe \
  --path "/interfaces/interface[name=Ethernet1]/state/counters" \
  --mode stream --stream-mode sample --sample-interval 10s

Because the subscription is bound to the session, your collector must implement re-subscribe on reconnect; most tooling does, but you should test it by clearing the connection and confirming data resumes without manual intervention.

Before subscribing to anything at scale, ask the device what it is willing to do. The capability exchange returns the supported encodings and models, which tells you whether a path will work before you write a dashboard against it:

gnmic -a 192.0.2.10:57400 -u admin -p '***' --insecure capabilities
gnmic -a 192.0.2.10:57400 -u admin -p '***' --insecure path list /interfaces
gnmic -a 192.0.2.10:57400 -u admin -p '***' --insecure --encoding json_ietf \
  subscribe --path "/interfaces/interface[name=Ethernet1]/state/counters" --mode once

The --mode once form is the most underrated debugging tool in streaming telemetry. It performs a single synchronous read with the same data model and the same path as the live subscription, so if once returns data and the stream does not, the problem is in the streaming path, not in your path string.

gnmic -a 192.0.2.10:57400 -u admin -p '***' --insecure subscribe \
  --path "/network-instances/network-instance[name=default]/protocols/protocol[identifier=BGP][name=bgp]/bgp/neighbors/neighbor/state" \
  --mode stream --stream-mode on-change --encoding json_ietf

For NETCONF-based dial-in, the equivalent is a single RPC on the YANG-push stream, with XPath filtering and a periodic trigger in milleconds-scale precision:

<establish-subscription xmlns="urn:ietf:params:xml:ns:yang:ietf-subscribed-notifications">
  <stream>yang-push</stream>
  <stream-xpath-filter xmlns:cpu="http://cisco.com/ns/yang/Cisco-IOS-XE-process-cpu-oper">
    /cpu:cpu-usage/cpu:cpu-utilization/cpu:five-seconds
  </stream-xpath-filter>
  <periodic><period>1000</period></periodic>
</establish-subscription>

Sample Rates, Scale and Collector Sizing

Streaming telemetry is cheap per device and expensive per fleet, and the arithmetic is worth doing before the first rollout. Total updates per second on a collector is:

updates/sec = devices x subscriptions-per-device / interval-seconds

Two hundred switches, four subscriptions each, a ten second interval, is 80 updates per second. At roughly 400 bytes per KV-GPB update that is about 32 KB/s, or under 3 GB a day - trivial for a modern time-series database. The same fleet with JSON_IETF payloads is five to ten times that volume, and with a one second interval it is a hundred times the update rate. The lesson is that the encoding and the interval, not the device count, decide whether the collector survives.

Device-side cost is the constraint people forget. Every sampled path is a read against a data model that was often not designed as a high-rate data source, executed on the same control-plane CPU that runs BGP and the CLI:

show processes cpu sorted 5sec | exclude 0.00
show telemetry ietf subscription all | count

A practical budget: keep the total number of concurrent subscriptions per device in the low tens, keep high-rate sampling (one to five seconds) to a handful of paths, and use on-change everywhere the underlying object changes slowly. If a dashboard needs finer granularity than the device can produce, the answer is usually a derived metric on the collector, not another subscription on the switch.

Failure Modes and How to Debug Them

Symptom Likely cause Check
Dial-out configured, nothing arrives Collector not listening, or 57500 blocked by a firewall or ACL show telemetry internal connection, then test the port from the device's own subnet
Data stops after a device reboot Wrong source-address, or no route from that source to the collector show ip route, show telemetry ietf subscription all
Dial-in stream ends without error Session timeout, or the device cancelled the subscription under load Collector logs, subscription IDs returned at subscribe time
Values are stale but updating Device timestamps drifting, or a sensor that stopped incrementing Compare device timestamps against wall clock
Control-plane CPU jumps after rollout Too many paths at high rate, or a JSON encoding on a counter-heavy leaf show processes cpu sorted, switch to encode-kvgpb
Subscription missing from configuration review Someone used dial-in, which leaves no trace Compare show telemetry ietf subscription all against the saved configuration

The stale-values row deserves its own habit. Devices you cannot reach are obvious; devices that stream yesterday's counters look perfectly healthy on a dashboard. Add a freshness check - maximum age of the newest sample per device - as a first-class alert, not a Grafana annotation.

Security, AAA and the Management VRF

  • Transport: use grpc-tls with a pinned certificate once the certificate lifecycle is automated. Remember the device is the client in dial-out, so it must be able to validate the collector's name, which means a resolvable name or a matching SAN.
  • AAA: a dial-in subscriber needs read access to a large part of the data model. Create a dedicated collector account with the minimum role that still returns the paths you need, and authenticate it through TACACS+ or RADIUS so deprovisioning is one change and not a fleet-wide configuration push.
  • Management VRF: bind telemetry to the management plane. On IOS XE that means a source-address inside the management VRF; on Junos a local-address on the management interface. The point is that a data-plane outage - a failed uplink, a routing loop, a saturated spine - leaves telemetry flowing so you can see what happened.
  • Exposure: never let TCP 57400 or 57500 be reachable from user or guest networks. A gNMI endpoint with a weak credential is a full read of your device state, and a dial-out receiver port is a free denial-of-service target.
  • Change control: dial-out subscriptions are configuration, so they should travel through the same review and rollback process as routing changes. That is a feature, not a burden - it is why the "survives a reboot" property exists at all.

Choosing

  1. Greenfield monitoring at scale: dial-out. The device owns reliability and reconnects after a reboot, and subscriptions appear in configuration reviews.
  2. Interactive troubleshooting: dial-in. No config footprint, short-lived, perfect for a five-minute investigation.
  3. Both, usually: dial-out for the golden signals (CPU, interface counters, BGP state), dial-in for on-demand deep dives.

A reference architecture that works well is deliberately boring: one or two dial-out collectors per site, each receiving from the site's devices on the management segment; a message bus in front of the time-series database so collectors can restart without losing data; configuration for the subscriptions generated from a single source of truth so that every device receives the same set; and a lab device that always gets the new subscription first. Dial-in is exposed only to a troubleshooting workstation and to the network engineering group, never to automation.

Migration Path

Teams rarely start from nothing, so the migration usually looks like this. Inventory what is actually streamed today, because in most networks the honest answer is "nothing - we poll with SNMP and pretend it is telemetry". Stand up the collector and a bus, and validate with dial-in against two lab devices, which requires no configuration change and therefore no change window. Then move the golden signals to dial-out, device by device, with the subscription generated from the source of truth. Finally, keep SNMP polling for the metrics that have no streaming equivalent - optical power readings, environmental sensors, older platforms - and document the boundary so nobody believes the streaming pipeline is complete when half the estate is still polled every five minutes.

Deployment Checklist

  • Decide per signal: dial-out for golden signals, dial-in for deep dives - and write the decision down.
  • Document the unit of every interval in the source of truth (milliseconds on IOS XE dial-out, seconds on Junos profiles, nanoseconds in gNMI).
  • Pick an encoding deliberately: encode-kvgpb for volume, JSON for debugging.
  • Set source-address (or Junos local-address) to a stable management-VRF address.
  • Confirm the collector port is open in both directions and monitored, not just present in the ACL.
  • Create a read-only service account per collector and authenticate it centrally.
  • Test re-subscribe explicitly: clear the session and prove data resumes unaided.
  • Alert on data freshness per device, not only on device reachability.
  • Watch control-plane CPU for a week after each rate increase.
  • Keep a lab device that receives new subscriptions before the fleet does.

Related reading: NetFlow v9 and flexible NetFlow configuration, Prometheus SNMP exporter monitoring and NETCONF, RESTCONF and gNMI compared and sFlow vs NetFlow vs IPFIX.

原文链接:Cisco: Model Driven Telemetry white paper