VStream API: allow keyspace-level heartbeats to be streamed #16593

rohit-nayak-ps · 2024-08-13T21:21:14Z

Description

This PR adds an option to the VTGate VStream API to allow heartbeats created by vttablet's heartbeat writer to be streamed as regular FieldEvents/RowEvents for the sidecar _vt.heartbeat table.

flags := &vtgatepb.VStreamFlags{
	StreamKeyspaceHeartbeats: true,
}

Note that the vttablet heartbeats have to be enabled either through the throttler (which is on by default) or explicitly turning on the heartbeats using, for example, --heartbeat_enable, --heartbeat_interval 1s and --heartbeat_on_demand_duration 0

Related Issue(s)

Fixes: Feature Request: Binlog Timestamp Watermarking for VStream #16477

Checklist

"Backport to:" labels have been added if this change should be back-ported to release branches
If this change is to be back-ported to previous releases, a justification is included in the PR description
Tests were added or are not required
Did the new or modified tests pass consistently locally and on CI?
Documentation was added or is not required

Signed-off-by: Rohit Nayak <[email protected]>

…l tables Signed-off-by: Rohit Nayak <[email protected]>

Signed-off-by: Rohit Nayak <[email protected]>

codecov · 2024-08-14T08:39:06Z

Codecov Report

Attention: Patch coverage is 90.12346% with 8 lines in your changes missing coverage. Please review.

Project coverage is 68.98%. Comparing base (cc68dd5) to head (8fb0240).
Report is 34 commits behind head on main.

Files	Patch %	Lines
go/vt/vttablet/tabletserver/vstreamer/vstreamer.go	94.33%	3 Missing ⚠️
go/vt/vtgate/vstream_manager.go	75.00%	2 Missing ⚠️
...et/tabletmanager/vreplication/replica_connector.go	0.00%	1 Missing ⚠️
go/vt/vttablet/tabletserver/binlog_watcher.go	0.00%	1 Missing ⚠️
go/vt/vttablet/tabletserver/tabletserver.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #16593      +/-   ##
==========================================
+ Coverage   68.85%   68.98%   +0.12%     
==========================================
  Files        1557     1562       +5     
  Lines      199891   200776     +885     
==========================================
+ Hits       137644   138505     +861     
- Misses      62247    62271      +24

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…nes and the ones specified separately Signed-off-by: Rohit Nayak <[email protected]>

Signed-off-by: Rohit Nayak <[email protected]>

proto/binlogdata.proto

proto/vtgate.proto

go/vt/vtgate/vstream_manager.go

go/vt/vttablet/tabletserver/vstreamer/vstreamer.go

go/test/endtoend/vreplication/vstream_test.go

proto/binlogdata.proto

go/vt/vtgate/vstream_manager.go

mattlord · 2024-08-19T18:11:29Z

go/vt/vttablet/tabletserver/tabletserver.go

 		return err
 	}
-	return tsv.vstreamer.Stream(ctx, request.Position, request.TableLastPKs, request.Filter, throttlerapp.VStreamerName, send)
+	return tsv.vstreamer.Stream(ctx, request.Position, request.TableLastPKs, request.Filter, throttlerapp.VStreamerName, send, request.Options)


Is it worth adding vstream options to support a variadic set of options (0-N) like we recently did for lock options and (workflow) server options? Then we don't need to pass this as nil in so many places: https://github.com/vitessio/vitess/blob/main/go/vt/vtctl/workflow/server_options.go

I prefer using strong typing unless functionally contraindicated. Especially for a grpc API.

While updating all usages can be tedious, it also helps review the parts of code where that API is being called: #16593 (comment)

mattlord

LGTM! I only had some minor comments. Let me know what you think and I'll come back to this quickly.

go/vt/vttablet/tabletserver/vstreamer/helper_event_test.go

mattlord · 2024-08-19T18:37:30Z

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

-func TestInternalTables(t *testing.T) {
+func TestOnlineDDLTables(t *testing.T) {


This is a clue that it might be better to explicitly call these sidecar[db]_tables rather than internal_tables. Although they're all internal tables so it's also fine as-is IMO.

yeah, I was also vacillating between internal and sidecar(db). internal is not precise because we could also term online ddl shadow tables and other artifacts as internal. I will let @deepthi also weigh in before changing it everywhere.

This seems to have been resolved.

@deepthi, the question was a broader one. Currently we use options/proto attributes like InternalTables, is_internal. Should we change that to SidecarDBTables and is_sidecardb_table?

We could either continue calling it Internal (keeping it generic, allowing for future specification of other tables than SidecarDB tables, like, I don't know, shadow tables in OnlineDDL for example) or lock it down to SidecarDB tables using that in the object naming instead.

internal is fine for now. Renaming proto fields is not difficult if we wanted to change it in the future.

mattlord · 2024-08-19T18:38:28Z

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

 	runCases(t, filter, testcases, startPos, nil)
 }

+// TestInternalTables tests streaming of sidecar db tables.


This comment for the function name is another clue that it might be better/clearer to call them SidecarDBTables rather than InternalTables.

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

Signed-off-by: Rohit Nayak <[email protected]>

deepthi · 2024-08-23T21:24:26Z

go/vt/vttablet/tabletserver/vstreamer/vstreamer_test.go

-func TestInternalTables(t *testing.T) {
+func TestOnlineDDLTables(t *testing.T) {


This seems to have been resolved.

go/vt/vttablet/tabletmanager/vreplication/external_connector.go

deepthi · 2024-08-23T21:35:28Z

go/vt/vttablet/tabletmanager/vreplication/replica_connector.go


 func (c *ReplicaConnector) VStream(ctx context.Context, startPos string, filter *binlogdatapb.Filter, send func([]*binlogdatapb.VEvent) error) error {
-	return c.vstreamer.Stream(ctx, startPos, nil, filter, throttlerapp.ReplicaConnectorName, send)
+	return c.vstreamer.Stream(ctx, startPos, nil, filter, throttlerapp.ReplicaConnectorName, send, nil)


Here again, based on how this is being used (only in PITR), we don't pass in the options.
Is it possible to replace usages of this connector with the tabletConnector?

Is it possible to replace usages of this connector with the tabletConnector?

I think so. Should be a separate PR though.

We should even think about deleting all the binlog server code in the existing PITR implementation. No one is using it.

go/vt/vttablet/tabletserver/vstreamer/vstreamer.go

Signed-off-by: Rohit Nayak <[email protected]>

mattlord

Nice work, @rohit-nayak-ps ! ❤️

…#16593) Signed-off-by: Rohit Nayak <[email protected]> Signed-off-by: Malcolm Akinje <[email protected]>

…#16593) Signed-off-by: Malcolm Akinje <[email protected]>

…#16593) (#620) * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) Signed-off-by: Malcolm Akinje <[email protected]> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <[email protected]> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <[email protected]> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <[email protected]> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <[email protected]> * fix import Signed-off-by: Tim Vaillancourt <[email protected]> * fix ineffassign Signed-off-by: Tim Vaillancourt <[email protected]> * missing import Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <[email protected]> * get count from backend Signed-off-by: Tim Vaillancourt <[email protected]> * rm unused map Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Tim Vaillancourt <[email protected]> Co-authored-by: Manan Gupta <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <[email protected]> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <[email protected]> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <[email protected]> * simplify Signed-off-by: Tim Vaillancourt <[email protected]> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <[email protected]> * use map of map Signed-off-by: Tim Vaillancourt <[email protected]> * capitalize Cell Signed-off-by: Tim Vaillancourt <[email protected]> * gofmt lint Signed-off-by: Tim Vaillancourt <[email protected]> * fix plural in names Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Malcolm Akinje <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Tim Vaillancourt <[email protected]> Signed-off-by: Tom Thornton <[email protected]> Signed-off-by: Malcolm Akinje <[email protected]> Co-authored-by: Tim Vaillancourt <[email protected]> Co-authored-by: Manan Gupta <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> Co-authored-by: Tom Thornton <[email protected]>

…d Journal Events (#585) * VTGate VStream: Ensure reasonable delivery time for reshard journal event (vitessio#16639) Signed-off-by: Malcolm Akinje <[email protected]> Signed-off-by: Malcolm Akinje <[email protected]> * Backport sqlparser patch for v15->v19 upgrade: 14763 Fix accepting bind variables in time related function calls (#590) * Fix accepting bind variables in time related function calls. (vitessio#14763) Signed-off-by: Manan Gupta <[email protected]> * fix test --------- Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * Upgrade vitess addons to 0.19.8 (#591) This upgrade allows us to control whether vtorc raises problems or not via an environment variable. Signed-off-by: Eduardo J. Ortega U. <[email protected]> * Use prefix in all vtorc check and recover logs (vitessio#17526) (#592) This is a backport of vitessio#17526 . Original PR description below: Description This is meant to make recovery actions more easily identified from the logs. See vitessio#17465 Signed-off-by: Eduardo J. Ortega U. <[email protected]> * `slack-19.0`: various backports for `vtorc`, part 2 (#596) * Ensure all topo read calls consider `--topo_read_concurrency` (vitessio#17276) Signed-off-by: Tim Vaillancourt <[email protected]> * Revert "add keyrange support for vtorc clusters_to_watch (#457)" This reverts commit 45c2199. * [release-19.0] `vtorc`: require topo for `Healthy: true` in `/debug/health` (vitessio#17129) (vitessio#17351) Signed-off-by: Tim Vaillancourt <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Tim Vaillancourt <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * `vtorc`: fetch all tablets from cells once + filter during refresh (vitessio#17388) Signed-off-by: Tim Vaillancourt <[email protected]> * Support KeyRange in `--clusters_to_watch` flag (vitessio#17604) Signed-off-by: Manan Gupta <[email protected]> * missing func Signed-off-by: Tim Vaillancourt <[email protected]> * Add api end point to print the current database state in VTOrc (vitessio#15485) Signed-off-by: Manan Gupta <[email protected]> --------- Signed-off-by: Tim Vaillancourt <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Manan Gupta <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * `slack-19.0`: `vtorc`: improve handling of partial cell topo results (#599) * `vtorc`: improve handling of partial cell topo results Signed-off-by: Tim Vaillancourt <[email protected]> * add unit test Signed-off-by: Tim Vaillancourt <[email protected]> * improve test Signed-off-by: Tim Vaillancourt <[email protected]> * add comments Signed-off-by: Tim Vaillancourt <[email protected]> * move sort to test Signed-off-by: Tim Vaillancourt <[email protected]> * goimports Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Tim Vaillancourt <[email protected]> * `slack-19.0`: skip tests that will fail on v15 downgrade testing (#605) Signed-off-by: Tim Vaillancourt <[email protected]> * `slack-19.0`: Add stats for shards watched by VTOrc (#606) * Add stats for shards watched by VTOrc Signed-off-by: Tim Vaillancourt <[email protected]> * Use len() in make --------- Signed-off-by: Tim Vaillancourt <[email protected]> * Add `GetServerStatus` RPC to use in PRS (vitessio#16022) (#607) Signed-off-by: Manan Gupta <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * backport/patch connection pool bug/perf fixes (#604) * [release-19.0] smartconnpool: do not allow connections to starve (vitessio#17675) (vitessio#17683) Signed-off-by: Dirkjan Bussink <[email protected]> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> * smartconnpool: Better handling for idle expiration (vitessio#17756) Signed-off-by: Vicent Marti <[email protected]> --------- Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Vicent Martí <[email protected]> Co-authored-by: Tim Vaillancourt <[email protected]> * pool: reopen connection closed by idle timeout (vitessio#17818) (#609) Signed-off-by: Harshit Gangal <[email protected]> Signed-off-by: Vicent Martí <[email protected]> Co-authored-by: Harshit Gangal <[email protected]> Co-authored-by: Vicent Martí <[email protected]> * VReplication: Support excluding lagging tablets and use this in vstream manager (vitessio#17835) (#612) * `slack-19.0`: backport v22 VTOrc optimizations, part 2 (#613) * `vtorc`: remove duplicate instance read from backend (vitessio#17834) Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: add index for `inst.ReadInstanceClusterAttributes` table scan Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Tim Vaillancourt <[email protected]> * Add stats for shards watched by VTOrc, purge stale shards (vitessio#17815) (#616) * --consolidator-query-waiter-cap to set the max number of waiter for consolidated query (vitessio#17244) (#614) Signed-off-by: Jun Wang <[email protected]> Signed-off-by: Tim Vaillancourt <[email protected]> Co-authored-by: jwang <[email protected]> Co-authored-by: Jun Wang <[email protected]> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <[email protected]> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <[email protected]> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <[email protected]> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <[email protected]> * fix import Signed-off-by: Tim Vaillancourt <[email protected]> * fix ineffassign Signed-off-by: Tim Vaillancourt <[email protected]> * missing import Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <[email protected]> * get count from backend Signed-off-by: Tim Vaillancourt <[email protected]> * rm unused map Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Tim Vaillancourt <[email protected]> Co-authored-by: Manan Gupta <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <[email protected]> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <[email protected]> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <[email protected]> * simplify Signed-off-by: Tim Vaillancourt <[email protected]> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <[email protected]> * use map of map Signed-off-by: Tim Vaillancourt <[email protected]> * capitalize Cell Signed-off-by: Tim Vaillancourt <[email protected]> * gofmt lint Signed-off-by: Tim Vaillancourt <[email protected]> * fix plural in names Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Tim Vaillancourt <[email protected]> * fix releasing the global read lock when mysqlshell backup fails (vitessio#17000) (#623) Signed-off-by: Renan Rangel <[email protected]> * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) (#620) * VStream API: allow keyspace-level heartbeats to be streamed (vitessio#16593) Signed-off-by: Malcolm Akinje <[email protected]> * `slack-19.0` backport v22 `vtorc` optimizations + stats, part 3 (#618) * Remove unused code in discovery queue creation (vitessio#17515) Signed-off-by: Manan Gupta <[email protected]> * vtorc: Cleanup unused code (vitessio#15508) Signed-off-by: Dirkjan Bussink <[email protected]> * `vtorc`: cleanup discover queue, add concurrency flag (vitessio#17825) Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: add tablets watched stats Signed-off-by: Tim Vaillancourt <[email protected]> * fix missing merge conflict update Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: skip unnecessary `inst.ReadTablet` in `logic.LockShard(...)` Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: use `errgroup` in keyspace/shard discovery Signed-off-by: Tim Vaillancourt <[email protected]> * fix import Signed-off-by: Tim Vaillancourt <[email protected]> * fix ineffassign Signed-off-by: Tim Vaillancourt <[email protected]> * missing import Signed-off-by: Tim Vaillancourt <[email protected]> * `vtorc`: add stats for discovery workers Signed-off-by: Tim Vaillancourt <[email protected]> * get count from backend Signed-off-by: Tim Vaillancourt <[email protected]> * rm unused map Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Tim Vaillancourt <[email protected]> Co-authored-by: Manan Gupta <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> * Bp pr 17558 pr 17858.slack19.0 (#615) * VReplication: Improve error handling in VTGate VStreams (vitessio#17558) Signed-off-by: Tom Thornton <[email protected]> * Backport vitessio#17858 --------- Signed-off-by: Tom Thornton <[email protected]> * `slack-19.0`: re-backport tweaks from vitessio#17911 (#621) * fix bug in reverse `if` Signed-off-by: Tim Vaillancourt <[email protected]> * simplify Signed-off-by: Tim Vaillancourt <[email protected]> * add `ReadTabletCountsByShard` test Signed-off-by: Tim Vaillancourt <[email protected]> * use map of map Signed-off-by: Tim Vaillancourt <[email protected]> * capitalize Cell Signed-off-by: Tim Vaillancourt <[email protected]> * gofmt lint Signed-off-by: Tim Vaillancourt <[email protected]> * fix plural in names Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Tim Vaillancourt <[email protected]> --------- Signed-off-by: Malcolm Akinje <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Tim Vaillancourt <[email protected]> Signed-off-by: Tom Thornton <[email protected]> Signed-off-by: Malcolm Akinje <[email protected]> Co-authored-by: Tim Vaillancourt <[email protected]> Co-authored-by: Manan Gupta <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> Co-authored-by: Tom Thornton <[email protected]> * Increase health check channel buffer (vitessio#17821) (#625) Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Malcolm Akinje <[email protected]> Co-authored-by: Manan Gupta <[email protected]> * VStream: Allow for automatic resume after Reshard across VStreams (vitessio#15393) (#627) Signed-off-by: Tanjin Xu <[email protected]> Co-authored-by: Matt Lord <[email protected]> --------- Signed-off-by: Malcolm Akinje <[email protected]> Signed-off-by: Malcolm Akinje <[email protected]> Signed-off-by: Manan Gupta <[email protected]> Signed-off-by: Eduardo J. Ortega U. <[email protected]> Signed-off-by: Tim Vaillancourt <[email protected]> Signed-off-by: Dirkjan Bussink <[email protected]> Signed-off-by: Vicent Marti <[email protected]> Signed-off-by: Harshit Gangal <[email protected]> Signed-off-by: Vicent Martí <[email protected]> Signed-off-by: Jun Wang <[email protected]> Signed-off-by: Tom Thornton <[email protected]> Signed-off-by: Renan Rangel <[email protected]> Signed-off-by: Tanjin Xu <[email protected]> Co-authored-by: Tanjin Xu <[email protected]> Co-authored-by: Manan Gupta <[email protected]> Co-authored-by: Eduardo J. Ortega U. <[email protected]> Co-authored-by: Tim Vaillancourt <[email protected]> Co-authored-by: vitess-bot[bot] <108069721+vitess-bot[bot]@users.noreply.github.com> Co-authored-by: Manan Gupta <[email protected]> Co-authored-by: Vicent Martí <[email protected]> Co-authored-by: Harshit Gangal <[email protected]> Co-authored-by: Tom Thornton <[email protected]> Co-authored-by: jwang <[email protected]> Co-authored-by: Jun Wang <[email protected]> Co-authored-by: Dirkjan Bussink <[email protected]> Co-authored-by: Renan Rangel <[email protected]> Co-authored-by: Matt Lord <[email protected]>

github-actions bot added this to the v21.0.0 milestone Aug 13, 2024

rohit-nayak-ps added Type: Enhancement Logical improvement (somewhere between a bug and feature) Component: VReplication Component: Throttler Component: VTGate labels Aug 13, 2024

rohit-nayak-ps added 11 commits August 14, 2024 10:17

Add VStream flag to enable heartbeats

78687fc

Signed-off-by: Rohit Nayak <[email protected]>

Add e2e test

657ccca

Signed-off-by: Rohit Nayak <[email protected]>

Always send heartbeat table events on

1085d56

Signed-off-by: Rohit Nayak <[email protected]>

Add IsInternal to FIELD/ROW events for sidecar tables

2af77cc

Signed-off-by: Rohit Nayak <[email protected]>

Pass options through to VStreamer

0a24b85

Signed-off-by: Rohit Nayak <[email protected]>

Fix unit tests for new Stream signature

6d55bb7

Signed-off-by: Rohit Nayak <[email protected]>

Minor refactor

f00e37c

Signed-off-by: Rohit Nayak <[email protected]>

Allow streaming of any internal tables. Add unit test. Update e2e test

25c92ef

Signed-off-by: Rohit Nayak <[email protected]>

Don't error out while dropping tables in defer to account for interna…

fa40b47

…l tables Signed-off-by: Rohit Nayak <[email protected]>

Only expose keyspace heartbeats in VTGate API

d516f8b

Signed-off-by: Rohit Nayak <[email protected]>

Fix vstream call for new signature in vdiff framework

731799a

Signed-off-by: Rohit Nayak <[email protected]>

rohit-nayak-ps force-pushed the rohit/vstream-vt-heartbeat branch from 68863d4 to 731799a Compare August 14, 2024 08:17

rohit-nayak-ps added 2 commits August 14, 2024 11:47

If streaming from a sidecar stream all tables, not just the special o…

22fade6

…nes and the ones specified separately Signed-off-by: Rohit Nayak <[email protected]>

Self-review

45edbb1

Signed-off-by: Rohit Nayak <[email protected]>

rohit-nayak-ps mentioned this pull request Aug 14, 2024

Feature Request: Binlog Timestamp Watermarking for VStream #16477

Closed

rohit-nayak-ps added 2 commits August 16, 2024 08:15

Self-review: e2e test case

ad21a8e

Signed-off-by: Rohit Nayak <[email protected]>

Self-review

5221811

Signed-off-by: Rohit Nayak <[email protected]>

rohit-nayak-ps marked this pull request as ready for review August 16, 2024 08:41

rohit-nayak-ps requested review from deepthi, derekperkins, harshit-gangal, mattlord, shlomi-noach and systay as code owners August 16, 2024 08:41

rohit-nayak-ps requested review from GuptaManan100, ajm188 and timvaillancourt as code owners August 16, 2024 08:41

rohit-nayak-ps mentioned this pull request Aug 18, 2024

VReplication: make flags workflow-specific and dynamically changeable #16583

Merged

5 tasks

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

proto/binlogdata.proto Show resolved Hide resolved

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

proto/vtgate.proto Show resolved Hide resolved

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

go/vt/vtgate/vstream_manager.go Outdated Show resolved Hide resolved

harshit-gangal reviewed Aug 19, 2024

View reviewed changes

go/vt/vttablet/tabletserver/vstreamer/vstreamer.go Outdated Show resolved Hide resolved

mattlord reviewed Aug 19, 2024

View reviewed changes

Address review comments

06e7b17

Signed-off-by: Rohit Nayak <[email protected]>

deepthi reviewed Aug 23, 2024

View reviewed changes

rohit-nayak-ps added 2 commits August 24, 2024 19:26

Address review comments

2f63e6a

Signed-off-by: Rohit Nayak <[email protected]>

Address review comments: change IsInternal to IsInternalTable

8fb0240

Signed-off-by: Rohit Nayak <[email protected]>

mattlord approved these changes Aug 26, 2024

View reviewed changes

deepthi approved these changes Aug 27, 2024

View reviewed changes

deepthi merged commit bc6fb1c into vitessio:main Aug 27, 2024

deepthi deleted the rohit/vstream-vt-heartbeat branch August 27, 2024 16:09

makinje16 mentioned this pull request Jan 16, 2025

VStream API [Backport]: Allow Keyspace-Level Heartbeats to be Streamed (#16593) slackhq/vitess#586

Closed

4 tasks

makinje16 pushed a commit to slackhq/vitess that referenced this pull request Mar 7, 2025

VStream API: allow keyspace-level heartbeats to be streamed (vitessio…

bd4696d

…#16593) Signed-off-by: Rohit Nayak <[email protected]> Signed-off-by: Malcolm Akinje <[email protected]>

tanjinx mentioned this pull request Mar 12, 2025

VStream API: allow keyspace-level heartbeats to be streamed (#16593) slackhq/vitess#620

Merged

5 tasks

makinje16 added a commit to slackhq/vitess that referenced this pull request Mar 13, 2025

VStream API: allow keyspace-level heartbeats to be streamed (vitessio…

936dffe

…#16593) Signed-off-by: Malcolm Akinje <[email protected]>

		func TestInternalTables(t *testing.T) {
		func TestOnlineDDLTables(t *testing.T) {

VStream API: allow keyspace-level heartbeats to be streamed #16593

VStream API: allow keyspace-level heartbeats to be streamed #16593

Uh oh!

Conversation

rohit-nayak-ps commented Aug 13, 2024 • edited by mattlord Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Checklist

Uh oh!

codecov bot commented Aug 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattlord left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohit-nayak-ps Aug 24, 2024 • edited by deepthi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattlord left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rohit-nayak-ps commented Aug 13, 2024 •

edited by mattlord

Loading

codecov bot commented Aug 14, 2024 •

edited

Loading

rohit-nayak-ps Aug 24, 2024 •

edited by deepthi

Loading