stats/opentelemetry: add trace event for name resolution delay #8074

vinothkumarr227 · 2025-02-10T10:40:32Z

stats/opentelemetry: add trace event for name resolution delay.

RELEASE NOTES: None

codecov · 2025-02-10T10:50:12Z

Codecov Report

Attention: Patch coverage is 86.66667% with 4 lines in your changes missing coverage. Please review.

Project coverage is 82.24%. Comparing base (e0d191d) to head (797db0a).
Report is 23 commits behind head on master.

Files with missing lines	Patch %	Lines
clientconn.go	33.33%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8074      +/-   ##
==========================================
- Coverage   82.29%   82.24%   -0.06%     
==========================================
  Files         387      387              
  Lines       39065    38948     -117     
==========================================
- Hits        32150    32033     -117     
- Misses       5584     5594      +10     
+ Partials     1331     1321      -10

Files with missing lines	Coverage Δ
internal/resolver/config_selector.go	`100.00% <ø> (ø)`
stats/opentelemetry/client_metrics.go	`89.44% <100.00%> (+0.05%)`	⬆️
stats/opentelemetry/opentelemetry.go	`75.00% <ø> (ø)`
stats/opentelemetry/trace.go	`88.23% <100.00%> (+4.90%)`	⬆️
stream.go	`81.55% <100.00%> (-0.14%)`	⬇️
clientconn.go	`90.55% <33.33%> (-1.63%)`	⬇️

... and 42 files with indirect coverage changes

purnesh42H · 2025-02-19T07:29:30Z

clientconn.go

-func (cc *ClientConn) waitForResolvedAddrs(ctx context.Context) error {
+// context expires, whichever happens first.
+//
+// If the name resolution took longer than expected (indicating a delay before


nit: If the name resolution did not succeed in first attempt, it returns true indicating delay in name resolution completion. In all other cases, including name resolution failure and name resolution succeeding in first attempt, it returns false.

purnesh42H · 2025-02-19T07:30:44Z

stats/handlers.go

@@ -38,6 +38,8 @@ type RPCTagInfo struct {
 	// FailFast indicates if this RPC is failfast.
 	// This field is only valid on client side, it's always false on server side.
 	FailFast bool
+	// NameResolutionDelay indicates if the RPC was delayed due to address resolution.


// NameResolutionDelay indicates if there was a delay in the name resolution.
// This field is only valid on client side, it's always false on server side.

purnesh42H · 2025-02-19T07:31:48Z

stats/opentelemetry/opentelemetry.go

@@ -211,6 +211,8 @@ type attemptInfo struct {
 	countSentMsg        uint32
 	countRecvMsg        uint32
 	previousRPCAttempts uint32
+	// nameResolutionDelayed indicates if the RPC was delayed by address resolution.


same comment as above about docstring

purnesh42H · 2025-02-19T07:33:07Z

stream.go

@@ -212,9 +216,13 @@ func newClientStream(ctx context.Context, desc *StreamDesc, cc *ClientConn, meth
 	}
 	// Provide an opportunity for the first RPC to see the first service config
 	// provided by the resolver.
-	if err := cc.waitForResolvedAddrs(ctx); err != nil {
+	isDelayed, err := cc.waitForResolvedAddrs(ctx)


nit: isDelayed -> nameResDelayed/nameResolutionDelayed

purnesh42H · 2025-02-19T07:35:20Z

stream.go

 		return nil, err
 	}
+	if isDelayed {


i think its fine to always add the nameResolutionDelay key. It will be either true/false. That we don't have to check again and again.

purnesh42H · 2025-02-19T07:35:51Z

stream.go

@@ -416,8 +424,9 @@ func (cs *clientStream) newAttemptLocked(isTransparent bool) (*csAttempt, error)
 	method := cs.callHdr.Method
 	var beginTime time.Time
 	shs := cs.cc.dopts.copts.StatsHandlers
+	isDelayed, _ := ctx.Value(nameResolutionDelayKey).(bool)


same comment about naming the variable as above

purnesh42H · 2025-02-19T07:48:41Z

stream.go

 		return nil, err
 	}
+	if isDelayed {
+		ctx = context.WithValue(ctx, nameResolutionDelayKey, isDelayed)


instead of adding it to context, can't we add to the rpcInfo below at line 233? and then we can pass that rpcInfo to newClientStreamWithParams. We can also just pass the bool value

Adding something less scoped like this to context has several downsides like context pollution, ambiguity, potential for misuse. It is because context is passed to different parts of application or even beyond current application

Thank you for the suggestion. I’ve updated the code to move nameResolutionDelay to rpcInfo as recommended. rpcInfo is now passed to newClientStreamWithParams,

purnesh42H · 2025-02-19T07:55:49Z

stream.go

@@ -212,9 +216,13 @@ func newClientStream(ctx context.Context, desc *StreamDesc, cc *ClientConn, meth
 	}
 	// Provide an opportunity for the first RPC to see the first service config
 	// provided by the resolver.
-	if err := cc.waitForResolvedAddrs(ctx); err != nil {
+	isDelayed, err := cc.waitForResolvedAddrs(ctx)


returning a bool and error is not good practice for go. It breaks the established pattern of error handling in Go because returned bool indicates success/failure in general. Can we do something better? It might be fine if we can't but we can try to look for better ways.

I have an approach as follows:

Add a nameResolutionDelay field: Add a new nameResolutionDelay field to the ClientConn struct to store the delay state.
Modify waitForResolvedAddrs: Set the nameResolutionDelay field directly in ClientConn instead of returning a boolean.
Access in newAttemptLocked: Use the nameResolutionDelay field from ClientConn within newAttemptLocked.

i don't think we want to add the field to clientconn because clientconn is not restricted to only single rpc. Returning a struct sounds better but we don't have any other field apart from boolean field. Let's keep bool, err for now. But make sure docstring is updated to indicate the bool correctly.

purnesh42H · 2025-02-19T07:56:49Z

test/clientconn_test.go

+	t.Log("Created a ClientConn...")
+
+	// First RPC should fail because there are no addresses yet.
+	ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)


can you point out if there is any existing test which is enforcing the name resolution delay in this way by failing first rpc?

…few doc issues

purnesh42H · 2025-02-21T05:36:04Z

just to give you more context

The firstResolveEvent is a grpcsync.Event (a custom synchronization primitive). It's used to signal whether the name resolver has sent at least one address update. HasFired() is a method on this event that tells you if the signal has already been sent.
If firstResolveEvent.HasFired() returns true, it means the resolver has already provided addresses at least once. In this case, there is no need to wait, so we can immediately return nil to signal that resolution has already occurred in the past. This is the fast path indicating no delay in name resolution
case <-cc.firstResolveEvent.Done() is part of a select statement. It's waiting for a signal on the cc.firstResolveEvent.Done() channel. The select statement will block here until something is sent on the firstResolveEvent.Done() channel. When something is sent on this channel, it means the firstResolveEvent has been fired. This is the waiting path. If cc.firstResolveEvent.HasFired() was false in the previous step, the code enters the select block. This case is specifically waiting for the resolver to send the first address update. Until then, it will block. So, its indicated delayed name resolution because rpc was blocked

purnesh42H · 2025-02-21T05:38:06Z

Test Case 1: Fast Path (Line 699)

Setup:
Create a ClientConn with a manual resolver that immediately returns a set of addresses.
Make one RPC call (or call a function that uses waitForResolvedAddrs).
Verify:
The first RPC call should pass without blocking. Because the resolver returned addresses quickly.
Make another RPC call
When the second RPC call happens, the firstResolveEvent will already be fired.

Test Case 2: Waiting Path (Line 703)

Setup:
Create a ClientConn with a manual resolver that delays returning any addresses.
Start an RPC call (or call a function that uses waitForResolvedAddrs).
The RPC should block.
Verify:
The RPC should block (we can use a channel mechanism to block/unblock. There are examples in resolver tests).
After a timeout or a manual trigger, make the manual resolver return addresses.
The blocked RPC should now continue and complete.

vinothkumarr227 added 2 commits February 10, 2025 10:38

Added add trace event for name resolution delay

d773e44

small tweaks

126b42f

vinothkumarr227 added 15 commits February 10, 2025 10:52

Removed unused parameter

dad6621

Removed unused parameter

37e1803

Fixed the duplicate import issues

9ae5cc8

refactor: simplify tests, fix SubConn deprecation, and add docs

f9b00e9

refactor: add function documentation and update variable names

3ce7799

Add test for 'Name resolution completed with delay' event check

76dc143

small tweaks

82f3ede

small tweaks

2ec49a9

Fixed the test cases

3af52f4

Added add trace event for name resolution delay

c0affc5

small tweaks

323b663

small tweaks

b76ed24

small tweaks

bc99c93

Fixed the name resolutions delay event test issues

e86b142

small tweaks

27914a2

arjan-bal requested a review from purnesh42H February 17, 2025 07:16

arjan-bal assigned purnesh42H Feb 17, 2025

arjan-bal added this to the 1.71 Release milestone Feb 17, 2025

arjan-bal added Type: Feature New features or improvements in behavior Area: Observability Includes Stats, Tracing, Channelz, Healthz, Binlog, Reflection, Admin, GCP Observability labels Feb 17, 2025

purnesh42H reviewed Feb 19, 2025

View reviewed changes

purnesh42H assigned vinothkumarr227 and unassigned purnesh42H Feb 19, 2025

arjan-bal modified the milestones: 1.71 Release, 1.72 Release Feb 19, 2025

vinothkumarr227 added 2 commits February 19, 2025 13:09

Refactor the integration of nameResolutionDelay in RPCInfo and fix a …

63bb794

…few doc issues

small tweaks

a4b6281

vinothkumarr227 added 2 commits February 21, 2025 12:16

Fixed the test cases

89a2bfa

small tweaks

797db0a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stats/opentelemetry: add trace event for name resolution delay #8074

stats/opentelemetry: add trace event for name resolution delay #8074

vinothkumarr227 commented Feb 10, 2025 •

edited

Loading

codecov bot commented Feb 10, 2025 •

edited

Loading

purnesh42H Feb 19, 2025

vinothkumarr227 Feb 19, 2025

purnesh42H Feb 19, 2025

vinothkumarr227 Feb 19, 2025

purnesh42H Feb 19, 2025

vinothkumarr227 Feb 19, 2025

purnesh42H Feb 19, 2025

vinothkumarr227 Feb 19, 2025

purnesh42H Feb 19, 2025

vinothkumarr227 Feb 19, 2025

purnesh42H Feb 19, 2025

vinothkumarr227 Feb 19, 2025

purnesh42H Feb 19, 2025

vinothkumarr227 Feb 19, 2025

purnesh42H Feb 19, 2025

vinothkumarr227 Feb 19, 2025

purnesh42H Feb 21, 2025

purnesh42H Feb 19, 2025

purnesh42H commented Feb 21, 2025

purnesh42H commented Feb 21, 2025

stats/opentelemetry: add trace event for name resolution delay #8074

Are you sure you want to change the base?

stats/opentelemetry: add trace event for name resolution delay #8074

Conversation

vinothkumarr227 commented Feb 10, 2025 • edited Loading

codecov bot commented Feb 10, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

purnesh42H commented Feb 21, 2025

purnesh42H commented Feb 21, 2025

vinothkumarr227 commented Feb 10, 2025 •

edited

Loading

codecov bot commented Feb 10, 2025 •

edited

Loading