philipp/doran
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Initial commit through Cline Kanban

Browse source
This commit is contained in:
philipp 2026-03-28 13:04:10 +01:00
commit 20c3feb4d2
415 changed files with 26235 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
.env.example Normal file
View file

@ -0,0 +1,6 @@
NEAR_INTENTS_API_KEY=your_solver_jwt
NEAR_INTENTS_WS_URL=wss://solver-relay-v2.chaindefuser.com/ws
KAFKA_BROKERS=127.0.0.1:9092
KAFKA_CLIENT_ID=trading-system
KAFKA_TOPIC_NORM_SWAP_DEMAND=norm.swap_demand
KAFKA_CONSUMER_GROUP_DUMMY=dummy-reactor-v1

5
.gitignore vendored Normal file
View file

@ -0,0 +1,5 @@
.ant-colony/
.venv/
__pycache__/
*.pyc
.env

134
README.md Normal file
View file

@ -0,0 +1,134 @@
# near-intents-monitor
Minimal event-driven POC for the first trading-system component:
- **venue ingest**: NEAR Intents solver-bus quote flow
- **central bus**: Redpanda / Kafka-compatible broker
- **dummy reactor**: placeholder consumer for later trade-decision logic
## Architecture
```text
NEAR Intents WebSocket
|
v
src/apps/near-intents-ingest.mjs
|
+--> raw.near_intents.quote
|
+--> norm.swap_demand
|
v
src/apps/dummy-consumer.mjs
```
The ingest app connects to the NEAR Intents websocket, subscribes to `quote` and `quote_status`, normalizes quote demand, and publishes to a Kafka-compatible topic.
## Project structure
```text
src/
apps/
near-intents-ingest.mjs
dummy-consumer.mjs
bus/
kafka/
producer.mjs
consumer.mjs
core/
event-envelope.mjs
log.mjs
pair-filter.mjs
lib/
env.mjs
config.mjs
venues/
near-intents/
ingest.mjs
normalize.mjs
ws.mjs
```
## Environment
Create `.env` in repo root:
```env
NEAR_INTENTS_API_KEY=your_solver_jwt
NEAR_INTENTS_WS_URL=wss://solver-relay-v2.chaindefuser.com/ws
KAFKA_BROKERS=127.0.0.1:9092
KAFKA_CLIENT_ID=trading-system
KAFKA_TOPIC_NORM_SWAP_DEMAND=norm.swap_demand
KAFKA_CONSUMER_GROUP_DUMMY=dummy-reactor-v1
```
### Broker notes
- `KAFKA_BROKERS` accepts a comma-separated broker list.
- Redpanda works because the apps use the Kafka protocol via `kafkajs`.
- `src/lib/config.mjs` is the shared config loader for both app entrypoints.
- The ingest app publishes normalized quote-demand events to `norm.swap_demand` by default.
## Install
```bash
npm install
```
## Run
### Start NEAR Intents ingest
Use the package script:
```bash
npm run near-intents:ingest
```
Or run the app directly:
```bash
node src/apps/near-intents-ingest.mjs
```
Optional exact-pair filter:
```bash
npm run near-intents:ingest -- --pair 'asset_a->asset_b'
```
Example:
```bash
npm run near-intents:ingest -- --pair 'nep141:btc.omft.near->nep141:gnosis-0x420ca0f9b9b604ce0fd9c18ef134c705e5fa3430.omft.near'
```
The filter is direction-agnostic, so `asset_a->asset_b` also matches `asset_b->asset_a`.
### Start the dummy consumer
Use the package script:
```bash
npm run dummy-consumer
```
Or run the app directly:
```bash
node src/apps/dummy-consumer.mjs
```
The dummy consumer subscribes to `norm.swap_demand`, logs the observed pair and quote id, and stands in for a future decision engine.
## Scripts
- `npm run near-intents:ingest` — start the websocket ingest and publish to Kafka/Redpanda topics
- `npm run dummy-consumer` — consume normalized demand events
- `npm start` — legacy wrapper that forwards into the ingest app
## Notes
- This repo is now bus-first: venue intake and downstream reaction are decoupled through Kafka-compatible topics.
- `index.mjs` remains only as a compatibility launch wrapper; operational docs should prefer `src/apps/*` entrypoints and npm scripts.
- Older single-file, Python, or TUI-only runtime instructions are obsolete for this repository state.

198
docs/minimal-product.md Normal file
View file

@ -0,0 +1,198 @@
# Minimal product: NEAR Intents demand monitor
## Goal
Build the smallest useful event-driven product for crypto trading research:
- read **live user demand** from NEAR Intents
- publish demand into a **central Kafka/Redpanda-compatible bus**
- prove downstream consumption with a **dummy reactor**
- avoid dashboards, execution, wallets, storage, auth workflows beyond the required API key, strategy code, and generic infra beyond the message bus itself
## Why this is the right first slice
From the NEAR Intents docs, there are several possible data surfaces:
1. **Message Bus WebSocket `quote` subscription**
- Endpoint: `wss://solver-relay-v2.chaindefuser.com/ws`
- Real-time stream for quote requests
- Subscription request shape:
```json
{
"jsonrpc": "2.0",
"id": 1,
"method": "subscribe",
"params": ["quote"]
}
```
- Expected live frame shape is JSON-RPC-like but should be treated as flexible. The adapter should accept quote payloads when the useful fields appear either:
- directly under `params`
- directly under `result`
- or at the top level of the message body
- Fields of interest include:
- `quote_id` (or equivalent request identifier)
- `defuse_asset_identifier_in`
- `defuse_asset_identifier_out`
- `exact_amount_in` or `exact_amount_out`
- `min_deadline_ms`
- Subscription acknowledgements may also vary. They may arrive as an `id`-matched JSON-RPC response with a simple `result`, a structured `result`, or other non-quote control frame before the first quote event.
- This is the closest public signal to **current demand**.
2. **Message Bus JSON-RPC `publish_intent` / `get_status`**
- Endpoint: `https://solver-relay-v2.chaindefuser.com/rpc`
- Useful for posting intents or checking a known `intent_hash`
- Not a public firehose of all intents.
3. **Explorer API `/api/v0/transactions`**
- Historical and analytics friendly
- Requires JWT auth
- Better for history, not best for a minimal live monitor
4. **Verifier contract intent payloads**
- The on-chain swap expression is usually `token_diff`
- Important for understanding settlement semantics
- Not the easiest first live intake path for a lean bus-first system
## Product decision
The minimal product should monitor **WebSocket `quote` events** and route them through a bus-first runtime.
### Why
- closest live signal to user demand
- directly reflects what users are requesting from solvers
- enough to answer the first trading question: **what assets are being requested right now?**
- decouples venue intake from downstream analysis through Kafka-compatible topics
### Important implementation note
Current docs for the market-maker quickstart and live endpoint behavior indicate the Message Bus requires a **partner API key / JWT** in the `Authorization: Bearer ...` header.
That means the best path is still the quote stream, but live operation is partner-gated.
### Important caveat
A `quote` event is **pre-trade demand**, not guaranteed execution.
That is fine for v0. The purpose is demand sensing, not settlement accounting.
## Runtime shape
```text
NEAR Intents websocket
|
v
src/apps/near-intents-ingest.mjs
|
+--> raw.near_intents.quote
|
+--> norm.swap_demand
|
v
src/apps/dummy-consumer.mjs
```
### Runtime contracts
#### Ingest app
`src/apps/near-intents-ingest.mjs`:
- loads env
- parses optional `--pair 'asset_a->asset_b'`
- starts the NEAR Intents websocket adapter
- writes raw and normalized events to the configured broker
#### Dummy consumer
`src/apps/dummy-consumer.mjs`:
- subscribes to `norm.swap_demand`
- logs observed pair and quote id
- exists only to prove a downstream consumer contract
#### Bus config
Default env-driven topics and group ids:
- `KAFKA_TOPIC_RAW_NEAR_INTENTS_QUOTE=raw.near_intents.quote`
- `KAFKA_TOPIC_NORM_SWAP_DEMAND=norm.swap_demand`
- `KAFKA_CONSUMER_GROUP_DUMMY=dummy-reactor-v1`
Redpanda is a valid runtime target because the transport is Kafka-compatible.
## Internal model
Normalize each quote event into a thin bus envelope:
Top-level envelope fields:
- `venue`
- `source`
- `type`
- `eventId`
- `occurredAt`
- `ingestedAt`
- `assetIn`
- `assetOut`
- `raw`
- `quote`
Nested `quote` fields:
- `quoteId`
- `assetIn`
- `assetOut`
- `amountIn`
- `amountOut`
- `ttlMs`
Field extraction must remain tolerant to known upstream aliases, and normalization should continue to operate on the merged `metadata + data` payload shape from the Message Bus event.
The live adapter now intentionally accepts quote-like payloads from `params`, `result`, or the top-level message body, but only processes frames that actually look like quote data. Subscription acknowledgements and unrelated control frames should still be ignored.
## Filtering
The ingest runtime supports an optional exact-pair filter:
```bash
npm run near-intents:ingest -- --pair 'asset_a->asset_b'
```
The filter is direction-agnostic, so the reversed asset order is also accepted.
## Scope boundaries
### Must do
- connect to the websocket
- subscribe to `quote` and tolerate control frames
- normalize quote events into one compact model
- publish raw and normalized events to Kafka/Redpanda-compatible topics
- allow a downstream consumer to react to normalized events
- reconnect automatically on disconnect
- document `npm` and `node` entrypoints
### Must not do
- Python packaging or CLI guidance
- TUI-specific product requirements
- charts
- account details
- pnl
- routing internals
- market making controls
- execution buttons
- config panels
- speculative infra beyond the current bus and dummy consumer
## Path to success
1. Connect to WebSocket
2. Subscribe to `quote`
3. Normalize incoming events into one compact model
4. Publish raw envelopes to `raw.near_intents.quote`
5. Publish normalized envelopes to `norm.swap_demand`
6. Start a dummy consumer on the normalized topic
7. Reconnect automatically on disconnect
8. Only after this works, consider:
- `quote_status`-specific downstream handling
- historical replay via Explorer API
- token metadata enrichment
- filtering and alerts beyond `--pair`
## Packaging alignment
Current repository packaging and usage should stay aligned around the JavaScript runtime entrypoints:
- package scripts:
- `npm run near-intents:ingest`
- `npm run dummy-consumer`
- `npm start` as a compatibility wrapper
- direct app entrypoints:
- `node src/apps/near-intents-ingest.mjs`
- `node src/apps/dummy-consumer.mjs`
Documentation should treat the npm scripts and `src/apps/*` node entrypoints as canonical. Older single-file and Python/TUI instructions should remain removed to avoid runtime confusion.
## Sources
- NEAR Intents Message Bus WebSocket docs: `subscribe` with `quote` / `quote_status`
- NEAR Intents Message Bus RPC docs: `quote`, `publish_intent`, `get_status`
- Verifier contract docs: `token_diff` intent type
- Explorer API OpenAPI: authenticated historical transactions

144
docs/spec.md Normal file
View file

@ -0,0 +1,144 @@
# NEAR Intents demand monitor: bus-first source plan
## Why websocket quote requests are still the MVP demand signal
Public solver quote requests remain the closest thing to live demand because they appear when a user or integration asks the network for executable pricing. They are still the right upstream source, but the runtime architecture is now bus-first rather than terminal-first.
Why this source wins for a first monitor:
- **Most real-time:** quote requests arrive before settlement and usually before a completed trade is visible anywhere else.
- **Closer to intent formation:** they reflect active user demand, not just historical outcomes.
- **Operationally simple:** a single websocket feed can drive the ingest side without indexing chains, scraping dashboards, or correlating multiple APIs.
- **Good enough for ranking demand:** even if quotes do not always become fills, repeated quote flow is still a strong indicator of what users are currently trying to do.
## Tradeoffs vs other sources
### Solver websocket quote requests
Pros:
- lowest-latency view of current demand
- directly tied to solver workflow
- suitable for a streaming ingest adapter
- can be normalized into pair, size, and frequency metrics immediately
Cons:
- quote requests are **interest**, not guaranteed executed volume
- public access may still be rate-limited, undocumented, or require credentials depending on environment
- schema and availability may change faster than user-facing products
### Explorer
Explorer (`https://explorer.near-intents.org/`) is useful for validation and historical inspection, but it is usually a worse primary source for an MVP demand monitor.
Tradeoffs:
- better for human inspection than low-latency streaming
- likely shows processed/published activity instead of raw quote demand
- may lag the actual request path
- less convenient as a machine-first demand feed
### Status dashboard / published status
Status (`https://status.near-intents.org/posts/dashboard`) is useful for system health, not demand discovery.
Tradeoffs:
- tells us whether the platform is up, degraded, or incident-affected
- does **not** represent per-request user demand
- coarse and aggregated by design
### Published intents / settled outcomes
Published or completed intents are higher-confidence signals, but lower-fidelity for immediate demand sensing.
Tradeoffs:
- stronger evidence of actual execution
- misses abandoned demand and pre-trade discovery
- arrives later than quote traffic
- may require more indexing and entity correlation work
## Runtime architecture
```text
solver websocket quote stream
|
v
src/apps/near-intents-ingest.mjs
|
+--> raw.near_intents.quote
|
+--> norm.swap_demand
|
v
src/apps/dummy-consumer.mjs
```
### Responsibilities
#### `src/apps/near-intents-ingest.mjs`
- loads env from `.env`
- parses optional `--pair 'asset_a->asset_b'`
- connects to the NEAR Intents websocket
- subscribes to `quote` and `quote_status`
- publishes raw venue envelopes to `raw.near_intents.quote`
- publishes normalized swap-demand envelopes to `norm.swap_demand`
#### `src/apps/dummy-consumer.mjs`
- consumes normalized events from `norm.swap_demand`
- logs observed demand as a placeholder for later strategy logic
#### Kafka / Redpanda layer
- broker endpoint comes from `KAFKA_BROKERS`
- Redpanda is supported through Kafka protocol compatibility
- topics are configurable via env and default to:
- `raw.near_intents.quote`
- `norm.swap_demand`
## Assumptions and limitations
- The websocket is the best available **MVP** source, not a perfect truth source.
- Demand is approximated by quote requests, not by settled intents.
- Live endpoints require auth in practice; `NEAR_INTENTS_API_KEY` must be provided.
- Request schemas may evolve; the parser should tolerate missing fields.
- The current product is intentionally minimal: no database, no backfill, no reconciliation against chain state.
- The dummy consumer proves the decoupled flow but is not a strategy engine.
## Run instructions
Install:
```bash
npm install
```
Start ingest:
```bash
npm run near-intents:ingest
```
Direct node entrypoint:
```bash
node src/apps/near-intents-ingest.mjs
```
Run with exact-pair filtering:
```bash
npm run near-intents:ingest -- --pair 'asset_a->asset_b'
```
Start dummy consumer:
```bash
npm run dummy-consumer
```
Direct node entrypoint:
```bash
node src/apps/dummy-consumer.mjs
```
## Decision summary
For an MVP whose job is to answer "what are users asking for right now?", solver websocket quote requests are still the best first source because they are the most direct, timely, and stream-friendly signal. The implementation now routes that signal through Kafka/Redpanda topics so ingestion and downstream reaction can evolve independently.

1
index.mjs Normal file
View file

@ -0,0 +1 @@
import './src/apps/near-intents-ingest.mjs';

17
node_modules/.package-lock.json generated vendored Normal file
View file

@ -0,0 +1,17 @@
{
"name": "near-intents-monitor-poc",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"node_modules/kafkajs": {
"version": "2.2.4",
"resolved": "https://registry.npmjs.org/kafkajs/-/kafkajs-2.2.4.tgz",
"integrity": "sha512-j/YeapB1vfPT2iOIUn/vxdyKEuhuY2PxMBvf5JWux6iSaukAccrMtXEY/Lb7OvavDhOWME589bpLrEdnVHjfjA==",
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
}
}
}

754
node_modules/kafkajs/CHANGELOG.md generated vendored Normal file
View file

@ -0,0 +1,754 @@
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
## [2.2.4] - 2023-02-27
### Added
- Include `groupId` in debug log when failing to find group coordinator #1522
### Fixed
- Rejoin group after ILLEGAL_GENERATION error #1474
- Fix consumer getting stuck after very brief throttling #1532
- Prevent infinite crash loop when no brokers are available #1408
## [2.2.3] - 2022-11-21
### Fixed
- Fix regression in SASL/PLAIN authentication #1480
## [2.2.2] - 2022-10-18
### Fixed
- Fix compatibility with AWS MSK Serverless #1463
## [2.2.1] - 2022-10-13
### Fixed
- Fixed Typescript definitions for custom authentication mechanisms #1459
## [2.2.0] - 2022-08-16
### Added
- Add the ability to inject custom authentication mechanisms #1372
- Add admin methods `alterPartitionReassignments` & `listPartitionReassignments` #1419
### Fixed
- Fix deprecation warning when connecting to a broker over TLS via IP address #1425
- Improve consumer performance when subscribed to thousands of topics #1436
## [2.1.0] - 2022-06-28
### Added
- Add `pause` function to `eachMessage`/`eachBatch` to pause the current topic-partition #1364
- The `KafkaMessage` type is now a union between the pre-Kafka 0.10 message format and the current #1401
### Fixed
- Fix 100% CPU utilization when all brokers are unavailable #1402
- Fix persistent error when trying to produce after a topic authorization error #1385
- Fix error when aborting or committing an empty transaction #1388
- Don't re-process messages from a paused partition after breaking the consumption flow #1382
## [2.0.2] - 2022-05-31
### Fixed
- Fix `consumer.seek` when seeking on multiple partitions for the same topic #1378
## [2.0.1] - 2022-05-23
### Fixed
- Fix members leaving the group after not being assigned any partitions #1362
- Make `REPLICA_NOT_AVAILABLE` retriable #1351
- Document `admin.createTopics` respecting cluster default partitions number and replication factor #1360
## [2.0.0] - 2022-05-06
This is the first major version released in 4 years, and contains a few important breaking changes. **A [migration guide](https://kafka.js.org/docs/migration-guide-v2.0.0) has been prepared to help with the migration process.** Be sure to read it before upgrading from older versions of KafkaJS.
### Added
- Validate configEntries when creating topics #1309
- New `topics` argument for `consumer.subscribe` to subscribe to multiple topics #1313
- Support duplicate header keys #1132
### Removed
- **BREAKING:** Drop support for Node 10 and 12 #1333
- **BREAKING:** Remove deprecated enum `ResourceTypes` #1334
- **BREAKING:** Remove deprecated argument `topic` from `admin.fetchOffsets` #1335
- **BREAKING:** Remove deprecated method `getTopicMetadata` from admin client #1336
- **BREAKING:** Remove typo type `TopicPartitionOffsetAndMedata` #1338
- **BREAKING:** Remove deprecated error property originalError. Replaced by `cause` #1341
### Changed
- **BREAKING:** Change default partitioner to Java compatible #1339
- Improve consumer performance #1258
- **BREAKING:** Enforce request timeout by default #1337
- **BREAKING** Honor default replication factor and partition count when creating topics #1305
- Increase default authentication timeout to 10 seconds #1340
### Fixed
- Fix invalid sequence numbers when producing concurrently with idempotent producer #1050 #1172
- Fix correlation id and sequence number overflow #1310
- Fix consumer not restarting on retriable connection errors #1304
- Avoid endless sleep loop #1323
## [1.16.0] - 2022-02-09
### Added
- Allow manual heartbeating from inside `eachMessage` handler #1255
- Add `rebalancing` consumer event #1067 #1079
- Add overload typings for all event types #1202
- Return `configSource` in `admin.decribeConfigs` #1023
- Add `topics` property to `admin.fetchOffsets` to fetch offsets for multiple topics #992 #998
- Improve error output from `admin.createTopic` #1104
- Export Error classes #1254
- Validate `brokers` list contains strings #1284
- Throw error when failing to stop or disconnect consumer #960
### Changed
- Don't commit offsets from `consumer.seek` when `autoCommit` is `false` #1012
- Do not restart the consumer on non-retriable errors #1274
- Downgrade consumer rebalance error log to `warn` #1279
- Make default round-robin partitioner topic-aware #1112
### Fixed
- Fix `offset` type of `consumer.seek` #981
- Fix crash when used in Electron app built with electron-builder #984
- Improve performance of Fetch requests #985
- Fix crash when using topics with name of built-in Javascript functions #995
- Fix type of consumer constructor to require config object #1002
- Fix message type to allow `null` key #1037
- Respect `heartbeatInterval` when invoking `heartbeat` concurrently #1026
- Fix type of `timestamp` of `LoggerEntryContent` to be string #1082
- Fix return type of `admin.describeAcls` #1118
- Fix consumer getting stuck in `DISCONNECTING` state if in-flight requests time out during disconnect #1208
- Fix failed serialization of BigInts when logging #1234
- Fix crash when committing offsets for a topic before consumer initialization #1235
- Reauthenticate to all brokers on demand #1241
- Remove unnecessary warn log when calling `admin.deleteTopicRecords` with offset `-1` #1265
- Handle empty control batches #1256
- Send empty topic array as null when fetching metadata #1184
## [1.15.0] - 2020-11-24
### Added
- Initial work for static membership #888
- Add consumer instrumentation event: received unsubscribed topics #897
- Add option for `admin.fetchOffsets` to resolve the offsets #895
- Add ACL functions to admin client #697
- Add `admin.deleteTopicRecords` #905
- Emit `GROUP_JOIN` event on stale partition assignments #937
### Changed
- Added properties to error classes typescript types #900
- Make header value type definition possibly undefined #927
- Bump API versions for client-side throttling #933
- Add `UNKNOWN_TOPIC_OR_PARTITION` check for `addMultipleTargetTopics` #938
### Fixed
- Fix describe/alter broker configs (introduced `ConfigResourceTypes`) #898
- Fix record batch compression masking (fix ZSTD compression) #912
- Prevent inflight's correlation id collisions #926
- Fix ACL, ISocketFactory and SaslOptions type definitions #941 #959 #966
- Fix deadlock on the connection `onError` handler #944
- Fix deadlock on the connection `onTimeout ` handler #956
- Remove nested retriers from producer #962 (fixes #958 #950)
## [1.14.0] - 2020-09-21
### Added
- Support Produce v6 protocol #869
- Support Produce v7 protocol (support for ZSTD compression) #869
- Broker rediscovery with config.brokers parameter taking a callback function #854 #882
### Changed
- Remove long.js in favor of BigInt #663
- Remove allowExperimentalV011 flag #847
### Fixed
- Only commit offsets on eachMessage failures if autoCommit is enabled #866
- Fix consumer offsets not committed if consumer stop was invoked right after the batch process #874
- Remove brokers with closed connections from the brokers list #878
- Type improvements and fixes #877
## [1.13.0] - 2020-09-10
### Added
- Add listGroup method to admin interface #645
- Add describeCluster method to admin client #648
- Add createPartitions method to admin client #661
- Add deleteGroups method to admin client #646
- Add listTopics method to admin client #718
- Add describeGroups method to admin client #742
- Allow to handle consumer retry failure at the user level #643
- Support Fetch v8 protocol (including client-side throttling) #776
- Support Fetch v9 protocol #778
- Support Fetch v10 protocol #792
- Support Fetch v11 protocol #810
- Support JoinGroup v3 and v4 protocol #801
- Oauthbearer support #680
- Add new protocol errors #824
- Add versioning to docs #835
- Add fetch topic offsets by timestamp #604
- Support LOG_APPEND_TIME record timestamps #838
- Suppress JoinGroup V4+ response error log when memberId is empty #860
### Changed
- Replace fetch promise all with async generator #570
- Improve balance in the RoundRobinAssigner #635
- Add single requestTimeout runner instead of setTimeout per request #650
- Provide the subscribed topics to the protocol() function #545
- Only wait for the lock when there are enqueued batches #670
- Update consumer default retries to 5 #720 (related to #719)
- Simplify and speed up SeekOffsets #668
- Remove maxInFlight option from default retry and moved into Producer ad Consumer #754
- Use addMultipleTargetTopics instead of looping over multiple calls to addTargetTopic #748
- Only disconnect the consumer and producers if they were created #784
- Resolve socket requests without response immediately when they have been queued #785
- Move default request timeout from connection to cluster #739
- Simplify the BufferedAsyncIterator #671
- Ensure fair fetch response allocation across topic-partitions #859
### Fixed
- Type improvements and fixes #636 #664 #675 #722 #729 #758 #799 #813 #757 #749 #764 #828 #843 #839
- Remove invalid topics from targetTopics on INVALID_TOPIC_EXCEPTION #666
- Fix network buffering performance problems #669
- Delete the entry for the waiter when the timeout is reached #694
- Fix encoder instanceof issue with Encoder #685
- Fix default retry for consumer #719
- Runner#waitForConsumer uses consuming stop event instead of timers #724
- Fix unhandled rejections #714 #797
- Make Array shuffle test pass in Node >= 11.0.0 #740
- Use setImmediate when scheduling calls of scheduleFetch #752
- Improve offset commit handling #775
- Use the length from the message to pre-allocate the result array #771
- Fix application lock in case of errors before the connection is established #780
- Add isNaN check for concurrency limit #787
- Fix admin client createTopics timeout #800
- Avoid repeated costly copies of buffers when working with encoders #811
- Fix fall-back retry config for producer #851
## [1.12.0] - 2020-01-30
### Added
- Force refresh of metadata when topic is not present #556
- Expose ConsumerRunConfig type #615
- Randomize order of seed brokers #632
### Changed
- Support TLS SNI by default #512
- Changed typing of `logLevel` argument of `logCreator` #538
- Add type boolean in ssl KafkaConfig #557
- Allow logging Fetch response payload buffers #573
- Remove default null for logCreator #595
- Add error names to ensure error names work with webpack + uglify #602
- Merge TopicMessages by topic in producer sendBatch #626
### Fixed
- Skip control records without auto-resolve #511
- Handle empty member assignment #567
- Only fetch for partitions with initialized offsets #582
- Get correct next offset for compacted topics #577
- TS type definition for removing instrumentation event listeners #608
- Fixed IHeaders definition to accept plain strings #547
- Make TS type ProducerBatch fields optional #610
- Fix typings for logger getters #620
## [1.11.0] - 2019-09-30
### Added
- Add Typescript SASLMechanism type to definitions #477
- Allow SASL Connections to Periodically Re-Authenticate #496
### Changed
- Throw validation error when the broker list is empty #460
- Improve the encoder to avoid copying unnecessary bytes #471
- Throw an error on subscription changes for running consumers #470
- Default `throttle_time_ms` to 0 if missing in `ApiVersions` response #495
- Remove normalisation of the password when using SCRAM mechanism #505
### Fixed
- Fix built-in partitioners type definition error #455
- Detect replaced brokers on refresh metadata #457
- Make NodeJS REPL get correct `randomBytes()` #462
- Fix `IHeaders` type definition (from string to Buffer) #467
- Rename Typescript definitions from `ResourceType` to `ResourceTypes` #468
- Update Typescript definitions to make `configNames` optional in `ResourceTypes` #474
- Fix `transactionState.ABORTING` value #478
## [1.10.0] - 2019-07-31
### Added
- Allow the consumer to pause and resume individual partitions #417
- Add `consumer.commitOffsets` for manual committing #436
- Expose consumer paused partitions #444
### Changed
- Removed unnecessary async code #434
- Use SubscriptionState to track member assignment #429
### Fixed
- Improve type compatibility with @types/kafkajs #416
- Fix `fetchTopicMetadata` return type #433
## [1.9.3] - 2019-06-27
### Fixed
- Fix AWS-IAM mechanism name #411 #412
- Fix TypeScript types for topic subscription with RegExp #413
## [1.9.2] - 2019-06-26
### Fixed
- Fix typescript types for Logger, consumer pause and resume, eachMessage and EachBatch interfaces #409
## [1.9.1] - 2019-06-25
### Fixed
- Fix typescript types for SSL, SASL and batch #407
## [1.9.0] - 2019-06-25
### Added
- Add typescript declaration file #362 #385 #390
- Add `requestTimeout` to apiVersions #369
- Discard messages saw a seek operation during a fetch or batch processing #367
- Include fetched offset metadata retrieved with `admin.fetchOffsets` #389
- Allow offset metadata to be written as part of OffsetCommit requests #392
- Prevent the consumption of messages for topics paused while fetch is in-flight #397
- Add `AWS-IAM` SASL mechanism #402
- Add `batch.offsetLagLow` #405
### Changed
- Don't modify `global.crypto` #365
- Change log level about producer without metadata #382
- Update encoder to write arrays as single `Buffer.concat` where possible #394
### Fixed
- Log error message on connection errors #400
- Make sure runner has connected brokers and fresh metadata before it starts #404
## [1.8.1] - 2019-06-25
### Fixed
- Make sure runner has connected brokers and fresh metadata before it starts #404
## [1.8.0] - 2019-05-13
### Added
- Add partition-aware concurrent mode for `eachMessage` #332
- Add `JavaCompatiblePartitioner` #358
- Add `consumer.subscribe({ topic: RegExp })` #346
- Update supported protocols to latest of Kafka 1 #343 #347 #348
### Changed
- Add documentation link to `REBALANCE_IN_PROGRESS` error #341
### Fixed
- Fix crash on offline replicas in metadata v5 response #350
## [1.7.0] - 2019-04-12
### Fixed
- Improve compatibility with terserjs #338
### Added
- Add `admin#fetchTopicMetadata` #331
### Changed
- Deprecated `admin#getTopicMetadata` #331
- `admin#fetchTopicOffsets` returns the low and high watermarks #333
## [1.6.0] - 2019-04-01
### Added
- Allow providing a socketFactory on client creation #263
- Add fetchTopicOffsets method #314
## [1.5.2] - 2019-04-01
### Fixed
- Process a fixed number of lock releases per iteration on `lock#release` #323
### Changed
- Use the max between the default request timeout and the protocol override #318
- Only emit events if there are listeners #321
## [1.5.1] - 2019-03-14
### Fixed
- Handle `null` keys on isAbortMarker #312
### Changed
- Prevent subsequent calls to `consumer#run` to override the running consumer #305
- Improve browser compatibility #300
- Add custom `requestTimeout` for protocol fetch #310
- Make `requestTimeout` optional, the current implementation is behind the flag `enforceRequestTimeout` #313
## [1.5.0] - 2019-03-05
### Changed
- See `1.5.0-beta.X` versions
## [1.5.0-beta.4] - 2019-02-28
### Fixed
- Abort old transactions on protocol error `CONCURRENT_TRANSACTIONS` #299
## [1.5.0-beta.3] - 2019-02-20
### Fixed
- Missing default restart time on crashes due to retriable errors #283
- Add custom requestTimeout for JoinGroup v0 #293
- Fix propagation of custom retry configs #295
### Changed
- Allow calling `Producer.sendBatch` with empty list #287
- Encode non-buffer key as string by default #291
## [1.5.0-beta.2] - 2019-02-13
### Fixed
- Handle undefined message key when producing with 0.11 #247
- Fix consumer restart on find coordinator errors #253
- Crash consumer on codec not implemented error #256
- Fix error message on invalid username or password #270
- Restart consumer on crashes due to retriable error #269
- Remove deleted topics from the cluster target group #273
### Changed
- Change Node engine requirement to >=8.6.0 #250
- Don't include lockfile and vscode files in package #264
### Added
- Allow configuring log level at runtime #278
## [1.5.0-beta.1] - 2019-01-17
### Fixed
- Rolling upgrade from 0.10 to 0.11 causes unknown magic byte errors #246
### Changed
- Validate consumer groupId #244
### Added
- Expose network queue size event to consumers, producers and admin #245
## [1.5.0-beta.0] - 2019-01-08
### Changed
- Add transactional attributes to record batch #199
- Ignore control records #208
- Filter aborted messages on the consumer #223 #210 #228
- Make Round robin assigner forward `userdata` #231
### Added
- Protocol `FindCoordinator` v1 #189
- Protocol `InitProducerId` v0 #190
- Protocol `AddPartitionsToTxn` v0 #191
- Protocol `AddOffsetsToTxn` v0 #194
- Protocol `TxnOffCommit` v0 #195
- Protocol `EndTxn` v0 #198
- Protocol `ListOffsets` v1 and v2 #217 #209
- Accept max in-flight requests on the connection #216
- Idempotent producer #203
- Transactional producer #206
- Protocol `SASLAuthenticate` #229
- Add SendOffsets to consumer `eachBatch` #232
- Add network instrumentation events #233
- Allow users to provide offsets to the `commitOffsetsIfNecessary` #235
## [1.4.8] - 2019-02-18
### Fixed
- Handle undefined message key when producing with 0.11 #247
- Fix consumer restart on find coordinator errors #253
- Crash consumer on codec not implemented error #256
- Fix error message on invalid username or password #270
## [1.4.7] - 2019-01-17
### Fixed
- Rolling upgrade from 0.10 to 0.11 causes unknown magic byte errors #246
## [1.4.6] - 2018-12-03
### Fixed
- Always assign partitions based on subscribed topics #227
## [1.4.5] - 2018-11-28
### Fixed
- Fix crash in mitigation for receiving metadata for unsubscribed topics #221
### Added
- Add `CRASH` instrumentation event for the consumer #221
## [1.4.4] - 2018-10-29
### Fixed
- Protocol produce v3 wasn't filtering `undefined` timestamps and was sending timestamp 0 (`NaN` converted) for all messages #188
## [1.4.3] - 2018-10-22
### Changed
- Version `1.4.2` without test files
## [1.4.2] - 2018-10-22
### Changed
- Allow messages with a value of `null` to support tombstones #185
## [1.4.1] - 2018-10-17
### Fixed
- Decode multiple RecordBatch on protocol Fetch v4 #179
- Skip incomplete record batches #182
- Producer with `acks=0` never resolve #181
### Added
- Runtime flag for displaying buffers in debug output #176
- Add ZSTD to compression codecs and types #157
- Admin get topic metadata #174
### Changed
- Add description to lock instances #178
## [1.4.0] - 2018-10-09
### Fixed
- Potential offset loss when updating offsets for resolved partitions #124
- Refresh metadata on lock timeout #131
- Cleans up stale brokers on metadata refresh #131
- Force metadata refresh on `ECONNREFUSED` #134
- Handle API version not supported #135
- Handle v0.10 messages on v0.11 Fetch API #143
### Added
- Admin delete topics #117
- Update metadata api and allow to disable auto topic creation #118
- Use highest available API version #135 #146
- Admin describe and alter configs #138
- Validate message format in producer #142
- Consumers can detect that a topic was updated and force a rebalance #136
### Changed
- Improved stack trace for `KafkaJSNumberOfRetriesExceeded` #123
- Enable Kafka v0.11 API by default #141 (Can still be disabled with `allowExperimentalV011=false`)
- Replace event emitter Lock #154
- Add member assignment to `GROUP_JOIN` instrumentation event #136
## [1.3.1] - 2018-08-20
### Fixed
- Client logger accessor #106
- Producer v3 decode format #114
- Parsing multiple responses #115
- Fetch v4 for partial messages on record batch #116
### Added
- Connection instrumentation events #110
## [1.3.0] - 2018-08-06
### Fixed
- Skip unsubscribed topic assignment #86
- Refresh metadata when producing to a topic without metadata #87
- Discard messages with a lower offset than requested #100
### Added
- Add consumer auto commit policies #89
- Notify user when setting heartbeat interval to same or higher than session timeout #91
- Constantly refresh metadata based on `metadataMaxAge` #94
- New instrumentation events #95
- Expose loggers #97 #102
- Add offset management operations to the admin client #101
- Support to record batch compression #103
- Handle missing username/password during authentication #104
## [1.2.0] - 2018-07-02
### Fixed
- Make sure authentication handshake remains consistent event when `broker.connect` is called concurrently #81
### Added
- Add `producer.sendBatch` to produce to multiple topics at once #82
- Experimental support to native Kafka 0.11 (`allowExperimentalV011`) #61 #68 #75 #76 #78
- Enabled protocol `fetch` v3
## [1.1.0] - 2018-06-14
### Added
- Support to SASL SCRAM (`scram-sha-256` and `scram-sha-512`) #72
- Admin client with support to create topics #73
## [1.0.1] - 2018-05-18
### Fixed
- Prevent crash when re-producing after metadata refresh #62
## [1.0.0] - 2018-05-14
### Changed
- Updated readme
## [0.8.1] - 2018-04-10
### Fixed
- Throw unretriable error for incompatible message format #49
- Producer can't reconnect after broker disconnection #48
## [0.8.0] - 2018-04-05
### Removed
- Backwards compatibility with the old member assignment protocol (<= 0.6.x). __v0.7.x__ is the last version with support for both protocols (commit f965e91cf883bff74332e53a8c1d25c2af39e566)
### Fixed
- Only retry failed brokers #38
### Changed
- Use selected assigner on consumer sync #35
- Add validations to consumer subscribe and producer send #41
### Added
- Consumer pause and resume #41
- Allow `partitionAssigners` to be configured
- Allow auto-resolve to be disabled for eachBatch #44
## [0.7.1] - 2018-01-22
### Fixed
- Fix member assignment protocol #33
## [0.7.0] - 2018-01-19
### Fixed
- Fix retry on error for message handlers #30
### Changed
- Use decoder offset for validating response length #21
- Change log creator to improve interoperability #24
- Add support to `KAFKAJS_LOG_LEVEL` #24
- Improved assigner protocol #27 #29
### Added
- Add seek API to consumer #23
- Add experimental describe group to consumer #31
## [0.6.8] - 2017-12-27
### Fixed
- Only use seed brokers to bootstrap consumers (introduced a broker pool abstraction) #19
- Don't include the size of the size field when determining if the buffer has enough bytes to read the full response #20
## [0.6.7] - 2017-12-18
### Fixed
- Don't throw KafkaJSOffsetOutOfRange on every protocol error
## [0.6.6] - 2017-12-15
### Fixed
- Fix index out of range errors when decoding partial messages #11
- Don't rely on seed broker for finding group coordinator #13
## [0.6.5] - 2017-12-14
### Fixed
- Fix partial message decode #10
- Throw not implemented error for Snappy and LZ4 compression #10
- Don't crash when setting offsets to default #10
## [0.6.4] - 2017-12-13
### Added
- Add stack trace to connection error logs
### Changed
- Skip consumer callback error logs for kafkajs errors
### Fixed
- Use the connection timeout callback for socket timeout
- Check if still connected before writing to the socket
## [0.6.3] - 2017-12-11
### Added
- Add error logs for user errors in `eachMessage` and `eachBatch`
### Fixed
- Recover from rebalance in progress when starting the consumer
## [0.6.2] - 2017-12-11
### Added
- Add `logger: "kafkajs"` to log lines
## [0.6.1] - 2017-12-08
### Added
- Add latest Kafka error codes
### Fixed
- Add fallback error for unsupported error codes
## [0.6.0] - 2017-12-07
### Added
- Expose consumer group `isRunning` to `eachBatch`
- Add `offsetLag` to consumer batch
## [0.5.0] - 2017-12-04
### Added
- Add ability to subscribe to events (heartbeats and offsets commit) #4
## [0.4.1] - 2017-11-30
### Changed
- Add heartbeat after each message
- Update default heartbeat interval to a better value
### Fixed
- Accept all consumer properties from create consumer
## [0.4.0] - 2017-11-23
### Added
- Commit previously resolved offsets when `eachBatch` throws an error
- Commit previously resolved offsets when `eachMessage` throws an error
- Consumer group recover from offset out of range
## [0.3.2] - 2017-11-23
### Fixed
- Stop consuming messages when the consumer group is not running
## [0.3.1] - 2017-11-20
### Fixed
- NPM bundle (add .npmignore)
- Fix package.json reference to main file
## [0.3.0] - 2017-11-20
### Added
- Add cluster method to fetch offsets for a list of topics
- Add offset resolution to the consumer group
### Changed
- Rename protocol Offsets to ListOffsets
- Update cluster fetch topics offset to allow different configurations per topic
- Create different instances of the cluster for producers and consumers
### Fixed
- Fix the use of timestamp in the ListOffsets protocol
- Fix create topic script
- Prevent unnecessary reconnections on cluster connect
## [0.2.0] - 2017-11-13
### Added
- Expose consumer groups
- Message and MessageSet V0 and V1 decoder
- Protocol fetch V0, V1, V2 and V3
- Protocol find coordinator V0
- Protocol join group V0
- Protocol sync group V0
- Protocol leave group V0
- Protocol offsets V0
- Protocol offset commit V0, V1 and V2
- Protocol offset fetch V1 and V2
- Protocol heartbeat V0
- Support to compressed message sets in protocol fetch
- Add find coordinator group to cluster
- Set timestamp for compressed messages
- Travis integration
- Eslint and prettier
### Changed
- Throw an error when cluster findBroker doesn't find a broker
- Move `KafkaProtocolError` to errors file
- Convert encode, decode and parse to async functions
- Update producer to use async compressor
- Update fetch to use async decompressor
- Create a separate error class for SASL errors
- Accepts a list of brokers instead of host and port
- Move retry logic out of connection and create namespaces for the logger
- Retry when refresh metadata throws leader not available
### Fixed
- Fix broker maxWaitTime default value
- Take OS differences when asserting gzip results
- Clear the connection timeout when the connection ends or fails due to an error
## [0.1.2] - 2017-10-17
### Added
- Expose if the cluster is connected
### Fixed
- Reconnect the cluster if it is not connected when producing messages
- Throw error if metadata is still not loaded when finding the topic partition
- Refresh metadata in case it is not loaded
- Make connection throw a retriable error when not connected
## [0.1.1] - 2017-10-16
### Changed
- Only retry retriable errors
- Propagate clientId and connectionTimeout to Cluster
### Fixed
- Typo when loading the SASL Handshake protocol
## [0.1.0] - 2017-10-15
### Added
- Producer compatible with Kafka 0.10.x
- GZIP compression
- Plain, SSL and SASL_SSL implementations
- Published on Github

40
node_modules/kafkajs/CONTRIBUTING.md generated vendored Normal file
View file

@ -0,0 +1,40 @@
# Contributing to KafkaJS
Thank you for considering contributing to KafkaJS!
Reading and following these guidelines will help us make the contribution process easy and effective for everyone involved. It also communicates that you agree to respect the time of the developers managing and developing these open source projects.
## Getting Started
Contributions are made to this repo via Issues and Pull Requests (PRs). A few general guidelines that cover both:
- Search for existing issues and PRs before creating your own.
- The maintainers of KafkaJS are people that volunteer their time. We try to address issues and PRs in a timely manner, but cannot make any guarantees. Please don't @mention individual maintainers to try to get their attention.
### Issues
Issues should be used to report problems with the library, request a new feature, or to discuss potential changes before a PR is created. Please follow the issue template that's provided when you first create an issue in order to collect all the necessary information.
Issues are not a support channel. Please use [StackOverflow](https://stackoverflow.com/questions/tagged/kafkajs), [Slack](https://join.slack.com/t/kafkajs/shared_invite/zt-1ezd5395v-SOpTqYoYfRCyPKOkUggK0A) or other online resources instead. [Limited support from a maintainer](https://github.com/sponsors/Nevon?frequency=one-time&sponsor=Nevon) is available to sponsors.
If you find an issue that addresses the problem you're having, please add your own reproduction information to the existing issue rather than creating a new one. Adding a [reaction](https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) can also help be indicating to our maintainers that a particular problem is affecting more than just the reporter.
### Pull Requests
PRs are welcome and can be a quick way to get your fix or improvement out. If you've never contributed before, see [the contribution guidelines on our website](https://kafka.js.org/docs/contribution-guide) for practical information on how to get started.
In general, PRs should:
- Only fix/add the functionality in question **OR** address wide-spread whitespace/style issues, not both.
- Add tests for fixed or changed functionality.
- Address a single concern.
- Update the [Typescript type definitions](./types) if your change introduces any new or affects existing interfaces.
- Include documentation if it changes the functionality of the library. Our [documentation](https://kafka.js.org/docs/getting-started) is in the [`/docs`](./docs/) folder of the repo.
If your PR introduces a change in functionality or adds new functionality, always open an issue first to discuss your proposal before implementing it. This is especially crucial for breaking changes, which will almost always be rejected unless discussed first. For bug fixes this is not required, but still recommended.
Once a PR is merged and the master build is successful, a pre-release version of KafkaJS will be published to NPM in the [beta channel](https://www.npmjs.com/package/kafkajs/v/beta), which you can use until a there has been a stable release made containing your change.
## Getting Help
Join our [Slack community](https://join.slack.com/t/kafkajs/shared_invite/zt-1ezd5395v-SOpTqYoYfRCyPKOkUggK0A) if you have questions about the contribution process or otherwise want to get in touch.

24
node_modules/kafkajs/LICENSE generated vendored Normal file
View file

@ -0,0 +1,24 @@
The MIT License
Copyright (c) 2018 Túlio Ornelas (ornelas.tulio@gmail.com)
Permission is hereby granted, free of charge,
to any person obtaining a copy of this software and
associated documentation files (the "Software"), to
deal in the Software without restriction, including
without limitation the rights to use, copy, modify,
merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom
the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice
shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR
ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

178
node_modules/kafkajs/README.md generated vendored Normal file
View file

@ -0,0 +1,178 @@
[![npm version](https://img.shields.io/npm/v/kafkajs?color=%2344cc11&label=stable)](https://www.npmjs.com/package/kafkajs) [![npm pre-release version](https://img.shields.io/npm/v/kafkajs/beta?label=pre-release)](https://www.npmjs.com/package/kafkajs) [![Build Status](https://dev.azure.com/tulios/kafkajs/_apis/build/status/tulios.kafkajs?branchName=master)](https://dev.azure.com/tulios/kafkajs/_build/latest?definitionId=2&branchName=master) [![Slack Channel](https://join.slack.com/t/kafkajs/shared_invite/zt-1ezd5395v-SOpTqYoYfRCyPKOkUggK0Abadge.svg)](https://join.slack.com/t/kafkajs/shared_invite/zt-1ezd5395v-SOpTqYoYfRCyPKOkUggK0A)
<br />
<p align="center">
<a href="https://kafka.js.org">
<img src="https://raw.githubusercontent.com/tulios/kafkajs/master/logo/v2/kafkajs_circle.svg" alt="Logo" width="125" height="125">
</a>
<h3 align="center">KafkaJS</h3>
<p align="center">
A modern Apache Kafka® client for Node.js
<br />
<a href="https://kafka.js.org/"><strong>Get Started »</strong></a>
<br />
<br />
<a href="https://kafka.js.org/docs/getting-started" target="_blank">Read the Docs</a>
·
<a href="https://github.com/tulios/kafkajs/issues/new?assignees=&labels=&template=bug_report.md&title=">Report Bug</a>
·
<a href="https://github.com/tulios/kafkajs/issues/new?assignees=&labels=&template=feature_request.md&title=">Request Feature</a>
</p>
</p>
## Table of Contents
- [About the project](#about)
- [Sponsors](#sponsorship)
- [Features](#features)
- [Getting Started](#getting-started)
- [Usage](#usage)
- [Contributing](#contributing)
- [Help Wanted](#help-wanted)
- [Contact](#contact)
- [License](#license)
- [Acknowledgements](#acknowledgements)
## <a name="about"></a> About the Project
KafkaJS is a modern [Apache Kafka](https://kafka.apache.org/) client for Node.js. It is compatible with Kafka 0.10+ and offers native support for 0.11 features.
<small>KAFKA is a registered trademark of The Apache Software Foundation and has been licensed for use by KafkaJS. KafkaJS has no affiliation with and is not endorsed by The Apache Software Foundation.</small>
## <a name="sponsorship"></a> Sponsors ❤️
<p id="banner" align="center">
<table>
<tr>
<td>
<img src="https://raw.githubusercontent.com/tulios/kafkajs/master/logo/sponsors/upstash.png" width="220" height="185" align="left" />
<h3>Upstash: Serverless Kafka</h3>
<ul>
<li>True Serverless Kafka with per-request-pricing</li>
<li>Managed Apache Kafka, works with all Kafka clients</li>
<li>Built-in REST API designed for serverless and edge functions</li>
<li><b><a href="https://upstash.com/?utm_source=kafkajs">Start for free in 30 seconds!</a></b></li>
</ul>
<img width="1000" height="0">
</td>
</tr>
<tr>
<td>
<img src="https://raw.githubusercontent.com/tulios/kafkajs/master/logo/sponsors/kafkajs-devs.png" alt="Logo" width="220" height="185" align="left" />
<h3>Get help directly from a KafkaJS developer</h3>
<ul>
<li>Become a Github Sponsor to have a video call with a KafkaJS developer</li>
<li>Receive personalized support, validate ideas or accelerate your learning</li>
<li>Save time and get productive sooner, while supporting KafkaJS!</li>
<li><b><a href="https://github.com/sponsors/Nevon?frequency=one-time&sponsor=Nevon">See support options!</a></b></li>
</ul>
<img width="1000" height="0">
</td>
</tr>
</table>
</p>
*To become a sponsor, [reach out in our Slack community](https://join.slack.com/t/kafkajs/shared_invite/zt-1ezd5395v-SOpTqYoYfRCyPKOkUggK0A) to get in touch with one of the maintainers. Also consider becoming a Github Sponsor by following any of the links under "[Sponsor this project](https://github.com/tulios/kafkajs#sponsors)" in the sidebar.*
### <a name="features"></a> Features
* Producer
* Consumer groups with pause, resume, and seek
* Transactional support for producers and consumers
* Message headers
* GZIP compression
* Snappy, LZ4 and ZSTD compression through pluggable codecs
* Plain, SSL and SASL_SSL implementations
* Support for SCRAM-SHA-256 and SCRAM-SHA-512
* Support for AWS IAM authentication
* Admin client
### <a name="getting-started"></a> Getting Started
```sh
npm install kafkajs
# yarn add kafkajs
```
#### <a name="usage"></a> Usage
```javascript
const { Kafka } = require('kafkajs')
const kafka = new Kafka({
clientId: 'my-app',
brokers: ['kafka1:9092', 'kafka2:9092']
})
const producer = kafka.producer()
const consumer = kafka.consumer({ groupId: 'test-group' })
const run = async () => {
// Producing
await producer.connect()
await producer.send({
topic: 'test-topic',
messages: [
{ value: 'Hello KafkaJS user!' },
],
})
// Consuming
await consumer.connect()
await consumer.subscribe({ topic: 'test-topic', fromBeginning: true })
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
console.log({
partition,
offset: message.offset,
value: message.value.toString(),
})
},
})
}
run().catch(console.error)
```
Learn more about using [KafkaJS on the official site!](https://kafka.js.org)
- [Getting Started](https://kafka.js.org/docs/getting-started)
- [A Brief Intro to Kafka](https://kafka.js.org/docs/introduction)
- [Configuring KafkaJS](https://kafka.js.org/docs/configuration)
- [Example Producer](https://kafka.js.org/docs/producer-example)
- [Example Consumer](https://kafka.js.org/docs/consumer-example)
> _Read something on the website that didn't work with the latest stable version?_
[Check the pre-release versions](https://kafka.js.org/docs/pre-releases) - the website is updated on every merge to master.
## <a name="contributing"></a> Contributing
KafkaJS is an open-source project where development takes place in the open on GitHub. Although the project is maintained by a small group of dedicated volunteers, we are grateful to the community for bug fixes, feature development and other contributions.
See [Developing KafkaJS](https://kafka.js.org/docs/contribution-guide) for information on how to run and develop KafkaJS.
### <a name="help-wanted"></a> Help wanted 🤝
We welcome contributions to KafkaJS, but we also want to see a thriving third-party ecosystem. If you would like to create an open-source project that builds on top of KafkaJS, [please get in touch](https://join.slack.com/t/kafkajs/shared_invite/zt-1ezd5395v-SOpTqYoYfRCyPKOkUggK0A) and we'd be happy to provide feedback and support.
Here are some projects that we would like to build, but haven't yet been able to prioritize:
* [Dead Letter Queue](https://eng.uber.com/reliable-reprocessing/) - Automatically reprocess messages
* ✅ [Schema Registry](https://www.confluent.io/confluent-schema-registry/) - **[Now available!](https://www.npmjs.com/package/@kafkajs/confluent-schema-registry)** thanks to [@erikengervall](https://github.com/erikengervall)
* [Metrics](https://prometheus.io/) - Integrate with the [instrumentation events](https://kafka.js.org/docs/instrumentation-events) to expose commonly used metrics
### <a name="contact"></a> Contact 💬
[Join our Slack community](https://join.slack.com/t/kafkajs/shared_invite/zt-1ezd5395v-SOpTqYoYfRCyPKOkUggK0A)
## <a name="license"></a> License
See [LICENSE](https://github.com/tulios/kafkajs/blob/master/LICENSE) for more details.
### <a name="acknowledgements"></a> Acknowledgements
* Thanks to [Sebastian Norde](https://github.com/sebastiannorde) for the V1 logo ❤️
* Thanks to [Tracy (Tan Yun)](https://medium.com/@tanyuntracy) for the V2 logo ❤️
<small>Apache Kafka and Kafka are either registered trademarks or trademarks of The Apache Software Foundation in the United States and other countries. KafkaJS has no affiliation with the Apache Software Foundation.</small>

30
node_modules/kafkajs/index.js generated vendored Normal file
View file

@ -0,0 +1,30 @@
const Kafka = require('./src')
const PartitionAssigners = require('./src/consumer/assigners')
const AssignerProtocol = require('./src/consumer/assignerProtocol')
const Partitioners = require('./src/producer/partitioners')
const Compression = require('./src/protocol/message/compression')
const ConfigResourceTypes = require('./src/protocol/configResourceTypes')
const ConfigSource = require('./src/protocol/configSource')
const AclResourceTypes = require('./src/protocol/aclResourceTypes')
const AclOperationTypes = require('./src/protocol/aclOperationTypes')
const AclPermissionTypes = require('./src/protocol/aclPermissionTypes')
const ResourcePatternTypes = require('./src/protocol/resourcePatternTypes')
const { isRebalancing, isKafkaJSError, ...errors } = require('./src/errors')
const { LEVELS } = require('./src/loggers')
module.exports = {
Kafka,
PartitionAssigners,
AssignerProtocol,
Partitioners,
logLevel: LEVELS,
CompressionTypes: Compression.Types,
CompressionCodecs: Compression.Codecs,
ConfigResourceTypes,
AclResourceTypes,
AclOperationTypes,
AclPermissionTypes,
ResourcePatternTypes,
ConfigSource,
...errors,
}

84
node_modules/kafkajs/package.json generated vendored Normal file
View file

@ -0,0 +1,84 @@
{
"name": "kafkajs",
"version": "2.2.4",
"description": "A modern Apache Kafka client for node.js",
"author": "Tulio Ornelas <ornelas.tulio@gmail.com>",
"main": "index.js",
"types": "types/index.d.ts",
"license": "MIT",
"keywords": [
"kafka",
"sasl",
"scram"
],
"engines": {
"node": ">=14.0.0"
},
"repository": {
"type": "git",
"url": "https://github.com/tulios/kafkajs.git"
},
"bugs": {
"url": "https://github.com/tulios/kafkajs/issues"
},
"homepage": "https://kafka.js.org",
"scripts": {
"jest": "export KAFKA_VERSION=${KAFKA_VERSION:='2.4'} && NODE_ENV=test echo \"KAFKA_VERSION: ${KAFKA_VERSION}\" && KAFKAJS_DEBUG_PROTOCOL_BUFFERS=1 jest",
"test:local": "yarn jest --detectOpenHandles",
"test:debug": "NODE_ENV=test KAFKAJS_DEBUG_PROTOCOL_BUFFERS=1 node --inspect-brk $(yarn bin 2>/dev/null)/jest --detectOpenHandles --runInBand --watch",
"test:local:watch": "yarn test:local --watch",
"test": "yarn lint && JEST_JUNIT_OUTPUT_NAME=test-report.xml ./scripts/testWithKafka.sh 'yarn jest --ci --maxWorkers=4 --no-watchman --forceExit'",
"lint": "find . -path ./node_modules -prune -o -path ./coverage -prune -o -path ./website -prune -o -name '*.js' -print0 | xargs -0 eslint",
"format": "find . -path ./node_modules -prune -o -path ./coverage -prune -o -path ./website -prune -o -name '*.js' -print0 | xargs -0 prettier --write",
"precommit": "lint-staged",
"test:group:broker": "yarn jest --forceExit --testPathPattern 'src/broker/.*'",
"test:group:admin": "yarn jest --forceExit --testPathPattern 'src/admin/.*'",
"test:group:producer": "yarn jest --forceExit --testPathPattern 'src/producer/.*'",
"test:group:consumer": "yarn jest --forceExit --testPathPattern 'src/consumer/.*.spec.js'",
"test:group:others": "yarn jest --forceExit --testPathPattern 'src/(?!(broker|admin|producer|consumer)/).*'",
"test:group:oauthbearer": "OAUTHBEARER_ENABLED=1 yarn jest --forceExit src/producer/index.spec.js src/broker/__tests__/connect.spec.js src/consumer/__tests__/connection.spec.js src/broker/__tests__/disconnect.spec.js src/admin/__tests__/connection.spec.js src/broker/__tests__/reauthenticate.spec.js",
"test:group:broker:ci": "JEST_JUNIT_OUTPUT_NAME=test-report.xml ./scripts/testWithKafka.sh \"yarn test:group:broker --ci --maxWorkers=4 --no-watchman\"",
"test:group:admin:ci": "JEST_JUNIT_OUTPUT_NAME=test-report.xml ./scripts/testWithKafka.sh \"yarn test:group:admin --ci --maxWorkers=4 --no-watchman\"",
"test:group:producer:ci": "JEST_JUNIT_OUTPUT_NAME=test-report.xml ./scripts/testWithKafka.sh \"yarn test:group:producer --ci --maxWorkers=4 --no-watchman\"",
"test:group:consumer:ci": "JEST_JUNIT_OUTPUT_NAME=test-report.xml ./scripts/testWithKafka.sh \"yarn test:group:consumer --ci --maxWorkers=4 --no-watchman\"",
"test:group:others:ci": "JEST_JUNIT_OUTPUT_NAME=test-report.xml ./scripts/testWithKafka.sh \"yarn test:group:others --ci --maxWorkers=4 --no-watchman\"",
"test:group:oauthbearer:ci": "JEST_JUNIT_OUTPUT_NAME=test-report.xml COMPOSE_FILE='docker-compose.2_4_oauthbearer.yml' ./scripts/testWithKafka.sh \"yarn test:group:oauthbearer --ci --maxWorkers=4 --no-watchman\"",
"test:types": "tsc -p types/"
},
"devDependencies": {
"@types/jest": "^27.4.0",
"@types/node": "^12.0.8",
"@typescript-eslint/typescript-estree": "^1.10.2",
"eslint": "^6.8.0",
"eslint-config-prettier": "^6.0.0",
"eslint-config-standard": "^13.0.1",
"eslint-plugin-import": "^2.18.2",
"eslint-plugin-jest": "^26.1.0",
"eslint-plugin-node": "^11.0.0",
"eslint-plugin-prettier": "^3.1.0",
"eslint-plugin-promise": "^4.2.1",
"eslint-plugin-standard": "^4.0.0",
"execa": "^2.0.3",
"glob": "^7.1.4",
"husky": "^3.0.1",
"ip": "^1.1.5",
"jest": "^25.1.0",
"jest-circus": "^25.1.0",
"jest-extended": "^0.11.2",
"jest-junit": "^10.0.0",
"jsonwebtoken": "^9.0.0",
"lint-staged": "^9.2.0",
"mockdate": "^2.0.5",
"prettier": "^1.18.2",
"semver": "^6.2.0",
"typescript": "^3.8.3",
"uuid": "^3.3.2"
},
"dependencies": {},
"lint-staged": {
"*.js": [
"prettier --write",
"git add"
]
}
}

1606
node_modules/kafkajs/src/admin/index.js generated vendored Normal file
View file

File diff suppressed because it is too large Load diff

28
node_modules/kafkajs/src/admin/instrumentationEvents.js generated vendored Normal file
View file

@ -0,0 +1,28 @@
const swapObject = require('../utils/swapObject')
const networkEvents = require('../network/instrumentationEvents')
const InstrumentationEventType = require('../instrumentation/eventType')
const adminType = InstrumentationEventType('admin')
const events = {
CONNECT: adminType('connect'),
DISCONNECT: adminType('disconnect'),
REQUEST: adminType(networkEvents.NETWORK_REQUEST),
REQUEST_TIMEOUT: adminType(networkEvents.NETWORK_REQUEST_TIMEOUT),
REQUEST_QUEUE_SIZE: adminType(networkEvents.NETWORK_REQUEST_QUEUE_SIZE),
}
const wrappedEvents = {
[events.REQUEST]: networkEvents.NETWORK_REQUEST,
[events.REQUEST_TIMEOUT]: networkEvents.NETWORK_REQUEST_TIMEOUT,
[events.REQUEST_QUEUE_SIZE]: networkEvents.NETWORK_REQUEST_QUEUE_SIZE,
}
const reversedWrappedEvents = swapObject(wrappedEvents)
const unwrap = eventName => wrappedEvents[eventName] || eventName
const wrap = eventName => reversedWrappedEvents[eventName] || eventName
module.exports = {
events,
wrap,
unwrap,
}

913
node_modules/kafkajs/src/broker/index.js generated vendored Normal file
View file

@ -0,0 +1,913 @@
const Lock = require('../utils/lock')
const { Types: Compression } = require('../protocol/message/compression')
const { requests, lookup } = require('../protocol/requests')
const { KafkaJSNonRetriableError } = require('../errors')
const apiKeys = require('../protocol/requests/apiKeys')
const shuffle = require('../utils/shuffle')
const PRIVATE = {
SEND_REQUEST: Symbol('private:Broker:sendRequest'),
}
/** @type {import("../protocol/requests").Lookup} */
const notInitializedLookup = () => {
throw new Error('Broker not connected')
}
/**
* Each node in a Kafka cluster is called broker. This class contains
* the high-level operations a node can perform.
*
* @type {import("../../types").Broker}
*/
module.exports = class Broker {
/**
* @param {Object} options
* @param {import("../network/connectionPool")} options.connectionPool
* @param {import("../../types").Logger} options.logger
* @param {number} [options.nodeId]
* @param {import("../../types").ApiVersions} [options.versions=null] The object with all available versions and APIs
* supported by this cluster. The output of broker#apiVersions
* @param {number} [options.authenticationTimeout=10000]
* @param {boolean} [options.allowAutoTopicCreation=true] If this and the broker config 'auto.create.topics.enable'
* are true, topics that don't exist will be created when
* fetching metadata.
*/
constructor({
connectionPool,
logger,
nodeId = null,
versions = null,
authenticationTimeout = 10000,
allowAutoTopicCreation = true,
}) {
this.connectionPool = connectionPool
this.nodeId = nodeId
this.rootLogger = logger
this.logger = logger.namespace('Broker')
this.versions = versions
this.authenticationTimeout = authenticationTimeout
this.allowAutoTopicCreation = allowAutoTopicCreation
// The lock timeout has twice the connectionTimeout because the same timeout is used
// for the first apiVersions call
const lockTimeout = 2 * this.connectionPool.connectionTimeout + this.authenticationTimeout
this.brokerAddress = `${this.connectionPool.host}:${this.connectionPool.port}`
this.lock = new Lock({
timeout: lockTimeout,
description: `connect to broker ${this.brokerAddress}`,
})
this.lookupRequest = notInitializedLookup
}
/**
* @public
* @returns {boolean}
*/
isConnected() {
return this.connectionPool.sasl
? this.connectionPool.isConnected() && this.connectionPool.isAuthenticated()
: this.connectionPool.isConnected()
}
/**
* @public
* @returns {Promise}
*/
async connect() {
await this.lock.acquire()
try {
if (this.isConnected()) {
return
}
const connection = await this.connectionPool.getConnection()
if (!this.versions) {
this.versions = await this.apiVersions()
}
this.connectionPool.setVersions(this.versions)
this.lookupRequest = lookup(this.versions)
if (connection.getSupportAuthenticationProtocol() === null) {
let supportAuthenticationProtocol = false
try {
this.lookupRequest(apiKeys.SaslAuthenticate, requests.SaslAuthenticate)
supportAuthenticationProtocol = true
} catch (_) {
supportAuthenticationProtocol = false
}
this.connectionPool.setSupportAuthenticationProtocol(supportAuthenticationProtocol)
this.logger.debug(`Verified support for SaslAuthenticate`, {
broker: this.brokerAddress,
supportAuthenticationProtocol,
})
}
await connection.authenticate()
} finally {
await this.lock.release()
}
}
/**
* @public
* @returns {Promise}
*/
async disconnect() {
await this.connectionPool.destroy()
}
/**
* @public
* @returns {Promise<import("../../types").ApiVersions>}
*/
async apiVersions() {
let response
const availableVersions = requests.ApiVersions.versions
.map(Number)
.sort()
.reverse()
// Find the best version implemented by the server
for (const candidateVersion of availableVersions) {
try {
const apiVersions = requests.ApiVersions.protocol({ version: candidateVersion })
response = await this[PRIVATE.SEND_REQUEST]({
...apiVersions(),
requestTimeout: this.connectionPool.connectionTimeout,
})
break
} catch (e) {
if (e.type !== 'UNSUPPORTED_VERSION') {
throw e
}
}
}
if (!response) {
throw new KafkaJSNonRetriableError('API Versions not supported')
}
return response.apiVersions.reduce(
(obj, version) =>
Object.assign(obj, {
[version.apiKey]: {
minVersion: version.minVersion,
maxVersion: version.maxVersion,
},
}),
{}
)
}
/**
* @public
* @type {import("../../types").Broker['metadata']}
* @param {string[]} [topics=[]] An array of topics to fetch metadata for.
* If no topics are specified fetch metadata for all topics
*/
async metadata(topics = []) {
const metadata = this.lookupRequest(apiKeys.Metadata, requests.Metadata)
const shuffledTopics = shuffle(topics)
return await this[PRIVATE.SEND_REQUEST](
metadata({ topics: shuffledTopics, allowAutoTopicCreation: this.allowAutoTopicCreation })
)
}
/**
* @public
* @param {Object} request
* @param {Array} request.topicData An array of messages per topic and per partition, example:
* [
* {
* topic: 'test-topic-1',
* partitions: [
* {
* partition: 0,
* firstSequence: 0,
* messages: [
* { key: '1', value: 'A' },
* { key: '2', value: 'B' },
* ]
* },
* {
* partition: 1,
* firstSequence: 0,
* messages: [
* { key: '3', value: 'C' },
* ]
* }
* ]
* },
* {
* topic: 'test-topic-2',
* partitions: [
* {
* partition: 4,
* firstSequence: 0,
* messages: [
* { key: '32', value: 'E' },
* ]
* },
* ]
* },
* ]
* @param {number} [request.acks=-1] Control the number of required acks.
* -1 = all replicas must acknowledge
* 0 = no acknowledgments
* 1 = only waits for the leader to acknowledge
* @param {number} [request.timeout=30000] The time to await a response in ms
* @param {string} [request.transactionalId=null]
* @param {number} [request.producerId=-1] Broker assigned producerId
* @param {number} [request.producerEpoch=0] Broker assigned producerEpoch
* @param {import("../../types").CompressionTypes} [request.compression=CompressionTypes.None] Compression codec
* @returns {Promise}
*/
async produce({
topicData,
transactionalId,
producerId,
producerEpoch,
acks = -1,
timeout = 30000,
compression = Compression.None,
}) {
const produce = this.lookupRequest(apiKeys.Produce, requests.Produce)
return await this[PRIVATE.SEND_REQUEST](
produce({
acks,
timeout,
compression,
topicData,
transactionalId,
producerId,
producerEpoch,
})
)
}
/**
* @public
* @param {Object} request
* @param {number} [request.replicaId=-1] Broker id of the follower. For normal consumers, use -1
* @param {number} [request.isolationLevel=1] This setting controls the visibility of transactional records. Default READ_COMMITTED.
* @param {number} [request.maxWaitTime=5000] Maximum time in ms to wait for the response
* @param {number} [request.minBytes=1] Minimum bytes to accumulate in the response
* @param {number} [request.maxBytes=10485760] Maximum bytes to accumulate in the response. Note that this is
* not an absolute maximum, if the first message in the first non-empty
* partition of the fetch is larger than this value, the message will still
* be returned to ensure that progress can be made. Default 10MB.
* @param {Array} request.topics Topics to fetch
* [
* {
* topic: 'topic-name',
* partitions: [
* {
* partition: 0,
* fetchOffset: '4124',
* maxBytes: 2048
* }
* ]
* }
* ]
* @param {string} [request.rackId=''] A rack identifier for this client. This can be any string value which indicates where this
* client is physically located. It corresponds with the broker config `broker.rack`.
* @returns {Promise}
*/
async fetch({
replicaId,
isolationLevel,
maxWaitTime = 5000,
minBytes = 1,
maxBytes = 10485760,
topics,
rackId = '',
}) {
// TODO: validate topics not null/empty
const fetch = this.lookupRequest(apiKeys.Fetch, requests.Fetch)
// Shuffle topic-partitions to ensure fair response allocation across partitions (KIP-74)
const flattenedTopicPartitions = topics.reduce((topicPartitions, { topic, partitions }) => {
partitions.forEach(partition => {
topicPartitions.push({ topic, partition })
})
return topicPartitions
}, [])
const shuffledTopicPartitions = shuffle(flattenedTopicPartitions)
// Consecutive partitions for the same topic can be combined into a single `topic` entry
const consolidatedTopicPartitions = shuffledTopicPartitions.reduce(
(topicPartitions, { topic, partition }) => {
const last = topicPartitions[topicPartitions.length - 1]
if (last != null && last.topic === topic) {
topicPartitions[topicPartitions.length - 1].partitions.push(partition)
} else {
topicPartitions.push({ topic, partitions: [partition] })
}
return topicPartitions
},
[]
)
return await this[PRIVATE.SEND_REQUEST](
fetch({
replicaId,
isolationLevel,
maxWaitTime,
minBytes,
maxBytes,
topics: consolidatedTopicPartitions,
rackId,
})
)
}
/**
* @public
* @param {object} request
* @param {string} request.groupId The group id
* @param {number} request.groupGenerationId The generation of the group
* @param {string} request.memberId The member id assigned by the group coordinator
* @returns {Promise}
*/
async heartbeat({ groupId, groupGenerationId, memberId }) {
const heartbeat = this.lookupRequest(apiKeys.Heartbeat, requests.Heartbeat)
return await this[PRIVATE.SEND_REQUEST](heartbeat({ groupId, groupGenerationId, memberId }))
}
/**
* @public
* @param {object} request
* @param {string} request.groupId The unique group id
* @param {import("../protocol/coordinatorTypes").CoordinatorType} request.coordinatorType The type of coordinator to find
* @returns {Promise}
*/
async findGroupCoordinator({ groupId, coordinatorType }) {
// TODO: validate groupId, mandatory
const findCoordinator = this.lookupRequest(apiKeys.GroupCoordinator, requests.GroupCoordinator)
return await this[PRIVATE.SEND_REQUEST](findCoordinator({ groupId, coordinatorType }))
}
/**
* @public
* @param {object} request
* @param {string} request.groupId The unique group id
* @param {number} request.sessionTimeout The coordinator considers the consumer dead if it receives
* no heartbeat after this timeout in ms
* @param {number} request.rebalanceTimeout The maximum time that the coordinator will wait for each member
* to rejoin when rebalancing the group
* @param {string} [request.memberId=""] The assigned consumer id or an empty string for a new consumer
* @param {string} [request.protocolType="consumer"] Unique name for class of protocols implemented by group
* @param {Array} request.groupProtocols List of protocols that the member supports (assignment strategy)
* [{ name: 'AssignerName', metadata: '{"version": 1, "topics": []}' }]
* @returns {Promise}
*/
async joinGroup({
groupId,
sessionTimeout,
rebalanceTimeout,
memberId = '',
protocolType = 'consumer',
groupProtocols,
}) {
const joinGroup = this.lookupRequest(apiKeys.JoinGroup, requests.JoinGroup)
const makeRequest = (assignedMemberId = memberId) =>
this[PRIVATE.SEND_REQUEST](
joinGroup({
groupId,
sessionTimeout,
rebalanceTimeout,
memberId: assignedMemberId,
protocolType,
groupProtocols,
})
)
try {
return await makeRequest()
} catch (error) {
if (error.name === 'KafkaJSMemberIdRequired') {
return makeRequest(error.memberId)
}
throw error
}
}
/**
* @public
* @param {object} request
* @param {string} request.groupId
* @param {string} request.memberId
* @returns {Promise}
*/
async leaveGroup({ groupId, memberId }) {
const leaveGroup = this.lookupRequest(apiKeys.LeaveGroup, requests.LeaveGroup)
return await this[PRIVATE.SEND_REQUEST](leaveGroup({ groupId, memberId }))
}
/**
* @public
* @param {object} request
* @param {string} request.groupId
* @param {number} request.generationId
* @param {string} request.memberId
* @param {object} request.groupAssignment
* @returns {Promise}
*/
async syncGroup({ groupId, generationId, memberId, groupAssignment }) {
const syncGroup = this.lookupRequest(apiKeys.SyncGroup, requests.SyncGroup)
return await this[PRIVATE.SEND_REQUEST](
syncGroup({
groupId,
generationId,
memberId,
groupAssignment,
})
)
}
/**
* @public
* @param {object} request
* @param {number} request.replicaId=-1 Broker id of the follower. For normal consumers, use -1
* @param {number} request.isolationLevel=1 This setting controls the visibility of transactional records (default READ_COMMITTED, Kafka >0.11 only)
* @param {TopicPartitionOffset[]} request.topics e.g:
*
* @typedef {Object} TopicPartitionOffset
* @property {string} topic
* @property {PartitionOffset[]} partitions
*
* @typedef {Object} PartitionOffset
* @property {number} partition
* @property {number} [timestamp=-1]
*
*
* @returns {Promise}
*/
async listOffsets({ replicaId, isolationLevel, topics }) {
const listOffsets = this.lookupRequest(apiKeys.ListOffsets, requests.ListOffsets)
const result = await this[PRIVATE.SEND_REQUEST](
listOffsets({ replicaId, isolationLevel, topics })
)
// ListOffsets >= v1 will return a single `offset` rather than an array of `offsets` (ListOffsets V0).
// Normalize to just return `offset`.
for (const response of result.responses) {
response.partitions = response.partitions.map(({ offsets, ...partitionData }) => {
return offsets ? { ...partitionData, offset: offsets.pop() } : partitionData
})
}
return result
}
/**
* @public
* @param {object} request
* @param {string} request.groupId
* @param {number} request.groupGenerationId
* @param {string} request.memberId
* @param {number} [request.retentionTime=-1] -1 signals to the broker that its default configuration
* should be used.
* @param {object} request.topics Topics to commit offsets, e.g:
* [
* {
* topic: 'topic-name',
* partitions: [
* { partition: 0, offset: '11' }
* ]
* }
* ]
* @returns {Promise}
*/
async offsetCommit({ groupId, groupGenerationId, memberId, retentionTime, topics }) {
const offsetCommit = this.lookupRequest(apiKeys.OffsetCommit, requests.OffsetCommit)
return await this[PRIVATE.SEND_REQUEST](
offsetCommit({
groupId,
groupGenerationId,
memberId,
retentionTime,
topics,
})
)
}
/**
* @public
* @param {object} request
* @param {string} request.groupId
* @param {object} request.topics - If the topic array is null fetch offsets for all topics. e.g:
* [
* {
* topic: 'topic-name',
* partitions: [
* { partition: 0 }
* ]
* }
* ]
* @returns {Promise}
*/
async offsetFetch({ groupId, topics }) {
const offsetFetch = this.lookupRequest(apiKeys.OffsetFetch, requests.OffsetFetch)
return await this[PRIVATE.SEND_REQUEST](offsetFetch({ groupId, topics }))
}
/**
* @public
* @param {object} request
* @param {Array} request.groupIds
* @returns {Promise}
*/
async describeGroups({ groupIds }) {
const describeGroups = this.lookupRequest(apiKeys.DescribeGroups, requests.DescribeGroups)
return await this[PRIVATE.SEND_REQUEST](describeGroups({ groupIds }))
}
/**
* @public
* @param {object} request
* @param {Array} request.topics e.g:
* [
* {
* topic: 'topic-name',
* numPartitions: 1,
* replicationFactor: 1
* }
* ]
* @param {boolean} [request.validateOnly=false] If this is true, the request will be validated, but the topic
* won't be created
* @param {number} [request.timeout=5000] The time in ms to wait for a topic to be completely created
* on the controller node
* @returns {Promise}
*/
async createTopics({ topics, validateOnly = false, timeout = 5000 }) {
const createTopics = this.lookupRequest(apiKeys.CreateTopics, requests.CreateTopics)
return await this[PRIVATE.SEND_REQUEST](createTopics({ topics, validateOnly, timeout }))
}
/**
* @public
* @param {object} request
* @param {Array} request.topicPartitions e.g:
* [
* {
* topic: 'topic-name',
* count: 3,
* assignments: []
* }
* ]
* @param {boolean} [request.validateOnly=false] If this is true, the request will be validated, but the topic
* won't be created
* @param {number} [request.timeout=5000] The time in ms to wait for a topic to be completely created
* on the controller node
* @returns {Promise<void>}
*/
async createPartitions({ topicPartitions, validateOnly = false, timeout = 5000 }) {
const createPartitions = this.lookupRequest(apiKeys.CreatePartitions, requests.CreatePartitions)
return await this[PRIVATE.SEND_REQUEST](
createPartitions({ topicPartitions, validateOnly, timeout })
)
}
/**
* @public
* @param {object} request
* @param {string[]} request.topics An array of topics to be deleted
* @param {number} [request.timeout=5000] The time in ms to wait for a topic to be completely deleted on the
* controller node.
* @returns {Promise}
*/
async deleteTopics({ topics, timeout = 5000 }) {
const deleteTopics = this.lookupRequest(apiKeys.DeleteTopics, requests.DeleteTopics)
return await this[PRIVATE.SEND_REQUEST](deleteTopics({ topics, timeout }))
}
/**
* @public
* @param {object} request
* @param {import("../../types").ResourceConfigQuery[]} request.resources
* [{
* type: RESOURCE_TYPES.TOPIC,
* name: 'topic-name',
* configNames: ['compression.type', 'retention.ms']
* }]
* @param {boolean} [request.includeSynonyms=false]
* @returns {Promise}
*/
async describeConfigs({ resources, includeSynonyms = false }) {
const describeConfigs = this.lookupRequest(apiKeys.DescribeConfigs, requests.DescribeConfigs)
return await this[PRIVATE.SEND_REQUEST](describeConfigs({ resources, includeSynonyms }))
}
/**
* @public
* @param {object} request
* @param {import("../../types").IResourceConfig[]} request.resources
* [{
* type: RESOURCE_TYPES.TOPIC,
* name: 'topic-name',
* configEntries: [
* {
* name: 'cleanup.policy',
* value: 'compact'
* }
* ]
* }]
* @param {boolean} [request.validateOnly=false]
* @returns {Promise}
*/
async alterConfigs({ resources, validateOnly = false }) {
const alterConfigs = this.lookupRequest(apiKeys.AlterConfigs, requests.AlterConfigs)
return await this[PRIVATE.SEND_REQUEST](alterConfigs({ resources, validateOnly }))
}
/**
* Send an `InitProducerId` request to fetch a PID and bump the producer epoch.
*
* Request should be made to the transaction coordinator.
* @public
* @param {object} request
* @param {number} request.transactionTimeout The time in ms to wait for before aborting idle transactions
* @param {number} [request.transactionalId] The transactional id or null if the producer is not transactional
* @returns {Promise}
*/
async initProducerId({ transactionalId, transactionTimeout }) {
const initProducerId = this.lookupRequest(apiKeys.InitProducerId, requests.InitProducerId)
return await this[PRIVATE.SEND_REQUEST](initProducerId({ transactionalId, transactionTimeout }))
}
/**
* Send an `AddPartitionsToTxn` request to mark a TopicPartition as participating in the transaction.
*
* Request should be made to the transaction coordinator.
* @public
* @param {object} request
* @param {string} request.transactionalId The transactional id corresponding to the transaction.
* @param {number} request.producerId Current producer id in use by the transactional id.
* @param {number} request.producerEpoch Current epoch associated with the producer id.
* @param {object[]} request.topics e.g:
* [
* {
* topic: 'topic-name',
* partitions: [ 0, 1]
* }
* ]
* @returns {Promise}
*/
async addPartitionsToTxn({ transactionalId, producerId, producerEpoch, topics }) {
const addPartitionsToTxn = this.lookupRequest(
apiKeys.AddPartitionsToTxn,
requests.AddPartitionsToTxn
)
return await this[PRIVATE.SEND_REQUEST](
addPartitionsToTxn({ transactionalId, producerId, producerEpoch, topics })
)
}
/**
* Send an `AddOffsetsToTxn` request.
*
* Request should be made to the transaction coordinator.
* @public
* @param {object} request
* @param {string} request.transactionalId The transactional id corresponding to the transaction.
* @param {number} request.producerId Current producer id in use by the transactional id.
* @param {number} request.producerEpoch Current epoch associated with the producer id.
* @param {string} request.groupId The unique group identifier (for the consumer group)
* @returns {Promise}
*/
async addOffsetsToTxn({ transactionalId, producerId, producerEpoch, groupId }) {
const addOffsetsToTxn = this.lookupRequest(apiKeys.AddOffsetsToTxn, requests.AddOffsetsToTxn)
return await this[PRIVATE.SEND_REQUEST](
addOffsetsToTxn({ transactionalId, producerId, producerEpoch, groupId })
)
}
/**
* Send a `TxnOffsetCommit` request to persist the offsets in the `__consumer_offsets` topics.
*
* Request should be made to the consumer coordinator.
* @public
* @param {object} request
* @param {OffsetCommitTopic[]} request.topics
* @param {string} request.transactionalId The transactional id corresponding to the transaction.
* @param {string} request.groupId The unique group identifier (for the consumer group)
* @param {number} request.producerId Current producer id in use by the transactional id.
* @param {number} request.producerEpoch Current epoch associated with the producer id.
* @param {OffsetCommitTopic[]} request.topics
*
* @typedef {Object} OffsetCommitTopic
* @property {string} topic
* @property {OffsetCommitTopicPartition[]} partitions
*
* @typedef {Object} OffsetCommitTopicPartition
* @property {number} partition
* @property {number} offset
* @property {string} [metadata]
*
* @returns {Promise}
*/
async txnOffsetCommit({ transactionalId, groupId, producerId, producerEpoch, topics }) {
const txnOffsetCommit = this.lookupRequest(apiKeys.TxnOffsetCommit, requests.TxnOffsetCommit)
return await this[PRIVATE.SEND_REQUEST](
txnOffsetCommit({ transactionalId, groupId, producerId, producerEpoch, topics })
)
}
/**
* Send an `EndTxn` request to indicate transaction should be committed or aborted.
*
* Request should be made to the transaction coordinator.
* @public
* @param {object} request
* @param {string} request.transactionalId The transactional id corresponding to the transaction.
* @param {number} request.producerId Current producer id in use by the transactional id.
* @param {number} request.producerEpoch Current epoch associated with the producer id.
* @param {boolean} request.transactionResult The result of the transaction (false = ABORT, true = COMMIT)
* @returns {Promise}
*/
async endTxn({ transactionalId, producerId, producerEpoch, transactionResult }) {
const endTxn = this.lookupRequest(apiKeys.EndTxn, requests.EndTxn)
return await this[PRIVATE.SEND_REQUEST](
endTxn({ transactionalId, producerId, producerEpoch, transactionResult })
)
}
/**
* Send request for list of groups
* @public
* @returns {Promise}
*/
async listGroups() {
const listGroups = this.lookupRequest(apiKeys.ListGroups, requests.ListGroups)
return await this[PRIVATE.SEND_REQUEST](listGroups())
}
/**
* Send request to delete groups
* @param {string[]} groupIds
* @public
* @returns {Promise}
*/
async deleteGroups(groupIds) {
const deleteGroups = this.lookupRequest(apiKeys.DeleteGroups, requests.DeleteGroups)
return await this[PRIVATE.SEND_REQUEST](deleteGroups(groupIds))
}
/**
* Send request to delete records
* @public
* @param {object} request
* @param {TopicPartitionRecords[]} request.topics
* [
* {
* topic: 'my-topic-name',
* partitions: [
* { partition: 0, offset 2 },
* { partition: 1, offset 4 },
* ],
* }
* ]
* @returns {Promise<Array>} example:
* {
* throttleTime: 0
* [
* {
* topic: 'my-topic-name',
* partitions: [
* { partition: 0, lowWatermark: '2n', errorCode: 0 },
* { partition: 1, lowWatermark: '4n', errorCode: 0 },
* ],
* },
* ]
* }
*
* @typedef {object} TopicPartitionRecords
* @property {string} topic
* @property {PartitionRecord[]} partitions
*
* @typedef {object} PartitionRecord
* @property {number} partition
* @property {number} offset
*/
async deleteRecords({ topics }) {
const deleteRecords = this.lookupRequest(apiKeys.DeleteRecords, requests.DeleteRecords)
return await this[PRIVATE.SEND_REQUEST](deleteRecords({ topics }))
}
/**
* @public
* @param {object} request
* @param {import("../../types").AclEntry[]} request.acl e.g:
* [
* {
* resourceType: AclResourceTypes.TOPIC,
* resourceName: 'topic-name',
* resourcePatternType: ResourcePatternTypes.LITERAL,
* principal: 'User:bob',
* host: '*',
* operation: AclOperationTypes.ALL,
* permissionType: AclPermissionTypes.DENY,
* }
* ]
* @returns {Promise<void>}
*/
async createAcls({ acl }) {
const createAcls = this.lookupRequest(apiKeys.CreateAcls, requests.CreateAcls)
return await this[PRIVATE.SEND_REQUEST](createAcls({ creations: acl }))
}
/**
* @public
* @param {import("../../types").AclEntry} aclEntry
* @returns {Promise<void>}
*/
async describeAcls({
resourceType,
resourceName,
resourcePatternType,
principal,
host,
operation,
permissionType,
}) {
const describeAcls = this.lookupRequest(apiKeys.DescribeAcls, requests.DescribeAcls)
return await this[PRIVATE.SEND_REQUEST](
describeAcls({
resourceType,
resourceName,
resourcePatternType,
principal,
host,
operation,
permissionType,
})
)
}
/**
* @public
* @param {Object} request
* @param {import("../../types").AclEntry[]} request.filters
* @returns {Promise<void>}
*/
async deleteAcls({ filters }) {
const deleteAcls = this.lookupRequest(apiKeys.DeleteAcls, requests.DeleteAcls)
return await this[PRIVATE.SEND_REQUEST](deleteAcls({ filters }))
}
/**
* @public
* @param {Object} request
* @param {import("../../types").PartitionReassignment[]} request.topics
* @param {number} [request.timeout]
* @returns {Promise}
*/
async alterPartitionReassignments({ topics, timeout }) {
const alterPartitionReassignments = this.lookupRequest(
apiKeys.AlterPartitionReassignments,
requests.AlterPartitionReassignments
)
return await this[PRIVATE.SEND_REQUEST](alterPartitionReassignments({ topics, timeout }))
}
/**
* @public
* @param {Object} request
* @param {import("../../types").TopicPartitions[]} request.topics can be null
* @param {number} [request.timeout]
* @returns {Promise}
*/
async listPartitionReassignments({ topics = null, timeout }) {
const listPartitionReassignments = this.lookupRequest(
apiKeys.ListPartitionReassignments,
requests.ListPartitionReassignments
)
return await this[PRIVATE.SEND_REQUEST](listPartitionReassignments({ topics, timeout }))
}
/**
* @private
*/
async [PRIVATE.SEND_REQUEST](protocolRequest) {
try {
return await this.connectionPool.send(protocolRequest)
} catch (e) {
if (e.name === 'KafkaJSConnectionClosedError') {
await this.disconnect()
}
throw e
}
}
}

37
node_modules/kafkajs/src/broker/saslAuthenticator/awsIam.js generated vendored Normal file
View file

@ -0,0 +1,37 @@
const { request, response } = require('../../protocol/sasl/awsIam')
const { KafkaJSSASLAuthenticationError } = require('../../errors')
const awsIAMAuthenticatorProvider = sasl => ({ host, port, logger, saslAuthenticate }) => {
return {
authenticate: async () => {
if (!sasl.authorizationIdentity) {
throw new KafkaJSSASLAuthenticationError('SASL AWS-IAM: Missing authorizationIdentity')
}
if (!sasl.accessKeyId) {
throw new KafkaJSSASLAuthenticationError('SASL AWS-IAM: Missing accessKeyId')
}
if (!sasl.secretAccessKey) {
throw new KafkaJSSASLAuthenticationError('SASL AWS-IAM: Missing secretAccessKey')
}
if (!sasl.sessionToken) {
sasl.sessionToken = ''
}
const broker = `${host}:${port}`
try {
logger.debug('Authenticate with SASL AWS-IAM', { broker })
await saslAuthenticate({ request: request(sasl), response })
logger.debug('SASL AWS-IAM authentication successful', { broker })
} catch (e) {
const error = new KafkaJSSASLAuthenticationError(
`SASL AWS-IAM authentication failed: ${e.message}`
)
logger.error(error.message, { broker })
throw error
}
},
}
}
module.exports = awsIAMAuthenticatorProvider

82
node_modules/kafkajs/src/broker/saslAuthenticator/index.js generated vendored Normal file
View file

@ -0,0 +1,82 @@
const { requests, lookup } = require('../../protocol/requests')
const apiKeys = require('../../protocol/requests/apiKeys')
const plainAuthenticatorProvider = require('./plain')
const scram256AuthenticatorProvider = require('./scram256')
const scram512AuthenticatorProvider = require('./scram512')
const awsIAMAuthenticatorProvider = require('./awsIam')
const oauthBearerAuthenticatorProvider = require('./oauthBearer')
const { KafkaJSSASLAuthenticationError } = require('../../errors')
const BUILT_IN_AUTHENTICATION_PROVIDERS = {
AWS: awsIAMAuthenticatorProvider,
PLAIN: plainAuthenticatorProvider,
OAUTHBEARER: oauthBearerAuthenticatorProvider,
'SCRAM-SHA-256': scram256AuthenticatorProvider,
'SCRAM-SHA-512': scram512AuthenticatorProvider,
}
const UNLIMITED_SESSION_LIFETIME = '0'
module.exports = class SASLAuthenticator {
constructor(connection, logger, versions, supportAuthenticationProtocol) {
this.connection = connection
this.logger = logger
this.sessionLifetime = UNLIMITED_SESSION_LIFETIME
const lookupRequest = lookup(versions)
this.saslHandshake = lookupRequest(apiKeys.SaslHandshake, requests.SaslHandshake)
this.protocolAuthentication = supportAuthenticationProtocol
? lookupRequest(apiKeys.SaslAuthenticate, requests.SaslAuthenticate)
: null
}
async authenticate() {
const mechanism = this.connection.sasl.mechanism.toUpperCase()
const handshake = await this.connection.send(this.saslHandshake({ mechanism }))
if (!handshake.enabledMechanisms.includes(mechanism)) {
throw new KafkaJSSASLAuthenticationError(
`SASL ${mechanism} mechanism is not supported by the server`
)
}
const saslAuthenticate = async ({ request, response }) => {
if (this.protocolAuthentication) {
const requestAuthBytes = await request.encode()
const authResponse = await this.connection.send(
this.protocolAuthentication({ authBytes: requestAuthBytes })
)
// `0` is a string because `sessionLifetimeMs` is an int64 encoded as string.
// This is not present in SaslAuthenticateV0, so we default to `"0"`
this.sessionLifetime = authResponse.sessionLifetimeMs || UNLIMITED_SESSION_LIFETIME
if (!response) {
return
}
const { authBytes: responseAuthBytes } = authResponse
const payloadDecoded = await response.decode(responseAuthBytes)
return response.parse(payloadDecoded)
}
return this.connection.sendAuthRequest({ request, response })
}
if (
!this.connection.sasl.authenticationProvider &&
Object.keys(BUILT_IN_AUTHENTICATION_PROVIDERS).includes(mechanism)
) {
this.connection.sasl.authenticationProvider = BUILT_IN_AUTHENTICATION_PROVIDERS[mechanism](
this.connection.sasl
)
}
await this.connection.sasl
.authenticationProvider({
host: this.connection.host,
port: this.connection.port,
logger: this.logger.namespace(`SaslAuthenticator-${mechanism}`),
saslAuthenticate,
})
.authenticate()
}
}

50
node_modules/kafkajs/src/broker/saslAuthenticator/oauthBearer.js generated vendored Normal file
View file

@ -0,0 +1,50 @@
/**
* The sasl object must include a property named oauthBearerProvider, an
* async function that is used to return the OAuth bearer token.
*
* The OAuth bearer token must be an object with properties value and
* (optionally) extensions, that will be sent during the SASL/OAUTHBEARER
* request.
*
* The implementation of the oauthBearerProvider must take care that tokens are
* reused and refreshed when appropriate.
*/
const { request } = require('../../protocol/sasl/oauthBearer')
const { KafkaJSSASLAuthenticationError } = require('../../errors')
const oauthBearerAuthenticatorProvider = sasl => ({ host, port, logger, saslAuthenticate }) => {
return {
authenticate: async () => {
const { oauthBearerProvider } = sasl
if (oauthBearerProvider == null) {
throw new KafkaJSSASLAuthenticationError(
'SASL OAUTHBEARER: Missing OAuth bearer token provider'
)
}
const oauthBearerToken = await oauthBearerProvider()
if (oauthBearerToken.value == null) {
throw new KafkaJSSASLAuthenticationError('SASL OAUTHBEARER: Invalid OAuth bearer token')
}
const broker = `${host}:${port}`
try {
logger.debug('Authenticate with SASL OAUTHBEARER', { broker })
await saslAuthenticate({ request: await request(sasl, oauthBearerToken) })
logger.debug('SASL OAUTHBEARER authentication successful', { broker })
} catch (e) {
const error = new KafkaJSSASLAuthenticationError(
`SASL OAUTHBEARER authentication failed: ${e.message}`
)
logger.error(error.message, { broker })
throw error
}
},
}
}
module.exports = oauthBearerAuthenticatorProvider

28
node_modules/kafkajs/src/broker/saslAuthenticator/plain.js generated vendored Normal file
View file

@ -0,0 +1,28 @@
const { request, response } = require('../../protocol/sasl/plain')
const { KafkaJSSASLAuthenticationError } = require('../../errors')
const plainAuthenticatorProvider = sasl => ({ host, port, logger, saslAuthenticate }) => {
return {
authenticate: async () => {
if (sasl.username == null || sasl.password == null) {
throw new KafkaJSSASLAuthenticationError('SASL Plain: Invalid username or password')
}
const broker = `${host}:${port}`
try {
logger.debug('Authenticate with SASL PLAIN', { broker })
await saslAuthenticate({ request: request(sasl), response })
logger.debug('SASL PLAIN authentication successful', { broker })
} catch (e) {
const error = new KafkaJSSASLAuthenticationError(
`SASL PLAIN authentication failed: ${e.message}`
)
logger.error(error.message, { broker })
throw error
}
},
}
}
module.exports = plainAuthenticatorProvider

325
node_modules/kafkajs/src/broker/saslAuthenticator/scram.js generated vendored Normal file
View file

@ -0,0 +1,325 @@
const crypto = require('crypto')
const scram = require('../../protocol/sasl/scram')
const { KafkaJSSASLAuthenticationError, KafkaJSNonRetriableError } = require('../../errors')
const GS2_HEADER = 'n,,'
const EQUAL_SIGN_REGEX = /=/g
const COMMA_SIGN_REGEX = /,/g
const URLSAFE_BASE64_PLUS_REGEX = /\+/g
const URLSAFE_BASE64_SLASH_REGEX = /\//g
const URLSAFE_BASE64_TRAILING_EQUAL_REGEX = /=+$/
const HMAC_CLIENT_KEY = 'Client Key'
const HMAC_SERVER_KEY = 'Server Key'
const DIGESTS = {
SHA256: {
length: 32,
type: 'sha256',
minIterations: 4096,
},
SHA512: {
length: 64,
type: 'sha512',
minIterations: 4096,
},
}
const encode64 = str => Buffer.from(str).toString('base64')
class SCRAM {
/**
* From https://tools.ietf.org/html/rfc5802#section-5.1
*
* The characters ',' or '=' in usernames are sent as '=2C' and
* '=3D' respectively. If the server receives a username that
* contains '=' not followed by either '2C' or '3D', then the
* server MUST fail the authentication.
*
* @returns {String}
*/
static sanitizeString(str) {
return str.replace(EQUAL_SIGN_REGEX, '=3D').replace(COMMA_SIGN_REGEX, '=2C')
}
/**
* In cryptography, a nonce is an arbitrary number that can be used just once.
* It is similar in spirit to a nonce * word, hence the name. It is often a random or pseudo-random
* number issued in an authentication protocol to * ensure that old communications cannot be reused
* in replay attacks.
*
* @returns {String}
*/
static nonce() {
return crypto
.randomBytes(16)
.toString('base64')
.replace(URLSAFE_BASE64_PLUS_REGEX, '-') // make it url safe
.replace(URLSAFE_BASE64_SLASH_REGEX, '_')
.replace(URLSAFE_BASE64_TRAILING_EQUAL_REGEX, '')
.toString('ascii')
}
/**
* Hi() is, essentially, PBKDF2 [RFC2898] with HMAC() as the
* pseudorandom function (PRF) and with dkLen == output length of
* HMAC() == output length of H()
*
* @returns {Promise<Buffer>}
*/
static hi(password, salt, iterations, digestDefinition) {
return new Promise((resolve, reject) => {
crypto.pbkdf2(
password,
salt,
iterations,
digestDefinition.length,
digestDefinition.type,
(err, derivedKey) => (err ? reject(err) : resolve(derivedKey))
)
})
}
/**
* Apply the exclusive-or operation to combine the octet string
* on the left of this operator with the octet string on the right of
* this operator. The length of the output and each of the two
* inputs will be the same for this use
*
* @returns {Buffer}
*/
static xor(left, right) {
const bufferA = Buffer.from(left)
const bufferB = Buffer.from(right)
const length = Buffer.byteLength(bufferA)
if (length !== Buffer.byteLength(bufferB)) {
throw new KafkaJSNonRetriableError('Buffers must be of the same length')
}
const result = []
for (let i = 0; i < length; i++) {
result.push(bufferA[i] ^ bufferB[i])
}
return Buffer.from(result)
}
/**
* @param {SASLOptions} sasl
* @param {Logger} logger
* @param {Function} saslAuthenticate
* @param {DigestDefinition} digestDefinition
*/
constructor(sasl, host, port, logger, saslAuthenticate, digestDefinition) {
this.sasl = sasl
this.host = host
this.port = port
this.logger = logger
this.saslAuthenticate = saslAuthenticate
this.digestDefinition = digestDefinition
const digestType = digestDefinition.type.toUpperCase()
this.PREFIX = `SASL SCRAM ${digestType} authentication`
this.currentNonce = SCRAM.nonce()
}
async authenticate() {
const { PREFIX } = this
const broker = `${this.host}:${this.port}`
if (this.sasl.username == null || this.sasl.password == null) {
throw new KafkaJSSASLAuthenticationError(`${this.PREFIX}: Invalid username or password`)
}
try {
this.logger.debug('Exchanging first client message', { broker })
const clientMessageResponse = await this.sendClientFirstMessage()
this.logger.debug('Sending final message', { broker })
const finalResponse = await this.sendClientFinalMessage(clientMessageResponse)
if (finalResponse.e) {
throw new Error(finalResponse.e)
}
const serverKey = await this.serverKey(clientMessageResponse)
const serverSignature = this.serverSignature(serverKey, clientMessageResponse)
if (finalResponse.v !== serverSignature) {
throw new Error('Invalid server signature in server final message')
}
this.logger.debug(`${PREFIX} successful`, { broker })
} catch (e) {
const error = new KafkaJSSASLAuthenticationError(`${PREFIX} failed: ${e.message}`)
this.logger.error(error.message, { broker })
throw error
}
}
/**
* @private
*/
async sendClientFirstMessage() {
const clientFirstMessage = `${GS2_HEADER}${this.firstMessageBare()}`
const request = scram.firstMessage.request({ clientFirstMessage })
const response = scram.firstMessage.response
return this.saslAuthenticate({
request,
response,
})
}
/**
* @private
*/
async sendClientFinalMessage(clientMessageResponse) {
const { PREFIX } = this
const iterations = parseInt(clientMessageResponse.i, 10)
const { minIterations } = this.digestDefinition
if (!clientMessageResponse.r.startsWith(this.currentNonce)) {
throw new KafkaJSSASLAuthenticationError(
`${PREFIX} failed: Invalid server nonce, it does not start with the client nonce`
)
}
if (iterations < minIterations) {
throw new KafkaJSSASLAuthenticationError(
`${PREFIX} failed: Requested iterations ${iterations} is less than the minimum ${minIterations}`
)
}
const finalMessageWithoutProof = this.finalMessageWithoutProof(clientMessageResponse)
const clientProof = await this.clientProof(clientMessageResponse)
const finalMessage = `${finalMessageWithoutProof},p=${clientProof}`
const request = scram.finalMessage.request({ finalMessage })
const response = scram.finalMessage.response
return this.saslAuthenticate({
request,
response,
})
}
/**
* @private
*/
async clientProof(clientMessageResponse) {
const clientKey = await this.clientKey(clientMessageResponse)
const storedKey = this.H(clientKey)
const clientSignature = this.clientSignature(storedKey, clientMessageResponse)
return encode64(SCRAM.xor(clientKey, clientSignature))
}
/**
* @private
*/
async clientKey(clientMessageResponse) {
const saltedPassword = await this.saltPassword(clientMessageResponse)
return this.HMAC(saltedPassword, HMAC_CLIENT_KEY)
}
/**
* @private
*/
async serverKey(clientMessageResponse) {
const saltedPassword = await this.saltPassword(clientMessageResponse)
return this.HMAC(saltedPassword, HMAC_SERVER_KEY)
}
/**
* @private
*/
clientSignature(storedKey, clientMessageResponse) {
return this.HMAC(storedKey, this.authMessage(clientMessageResponse))
}
/**
* @private
*/
serverSignature(serverKey, clientMessageResponse) {
return encode64(this.HMAC(serverKey, this.authMessage(clientMessageResponse)))
}
/**
* @private
*/
authMessage(clientMessageResponse) {
return [
this.firstMessageBare(),
clientMessageResponse.original,
this.finalMessageWithoutProof(clientMessageResponse),
].join(',')
}
/**
* @private
*/
async saltPassword(clientMessageResponse) {
const salt = Buffer.from(clientMessageResponse.s, 'base64')
const iterations = parseInt(clientMessageResponse.i, 10)
return SCRAM.hi(this.encodedPassword(), salt, iterations, this.digestDefinition)
}
/**
* @private
*/
firstMessageBare() {
return `n=${this.encodedUsername()},r=${this.currentNonce}`
}
/**
* @private
*/
finalMessageWithoutProof(clientMessageResponse) {
const rnonce = clientMessageResponse.r
return `c=${encode64(GS2_HEADER)},r=${rnonce}`
}
/**
* @private
*/
encodedUsername() {
const { username } = this.sasl
return SCRAM.sanitizeString(username).toString('utf-8')
}
/**
* @private
*/
encodedPassword() {
const { password } = this.sasl
return password.toString('utf-8')
}
/**
* @private
*/
H(data) {
return crypto
.createHash(this.digestDefinition.type)
.update(data)
.digest()
}
/**
* @private
*/
HMAC(key, data) {
return crypto
.createHmac(this.digestDefinition.type, key)
.update(data)
.digest()
}
}
module.exports = {
DIGESTS,
SCRAM,
}

10
node_modules/kafkajs/src/broker/saslAuthenticator/scram256.js generated vendored Normal file
View file

@ -0,0 +1,10 @@
const { SCRAM, DIGESTS } = require('./scram')
const scram256AuthenticatorProvider = sasl => ({ host, port, logger, saslAuthenticate }) => {
const scram = new SCRAM(sasl, host, port, logger, saslAuthenticate, DIGESTS.SHA256)
return {
authenticate: async () => await scram.authenticate(),
}
}
module.exports = scram256AuthenticatorProvider

10
node_modules/kafkajs/src/broker/saslAuthenticator/scram512.js generated vendored Normal file
View file

@ -0,0 +1,10 @@
const { SCRAM, DIGESTS } = require('./scram')
const scram512AuthenticatorProvider = sasl => ({ host, port, logger, saslAuthenticate }) => {
const scram = new SCRAM(sasl, host, port, logger, saslAuthenticate, DIGESTS.SHA512)
return {
authenticate: async () => await scram.authenticate(),
}
}
module.exports = scram512AuthenticatorProvider

349
node_modules/kafkajs/src/cluster/brokerPool.js generated vendored Normal file
View file

@ -0,0 +1,349 @@
const Broker = require('../broker')
const createRetry = require('../retry')
const shuffle = require('../utils/shuffle')
const arrayDiff = require('../utils/arrayDiff')
const { KafkaJSBrokerNotFound, KafkaJSProtocolError } = require('../errors')
const { keys, assign, values } = Object
const hasBrokerBeenReplaced = (broker, { host, port, rack }) =>
broker.connectionPool.host !== host ||
broker.connectionPool.port !== port ||
broker.connectionPool.rack !== rack
module.exports = class BrokerPool {
/**
* @param {object} options
* @param {import("./connectionPoolBuilder").ConnectionPoolBuilder} options.connectionPoolBuilder
* @param {import("../../types").Logger} options.logger
* @param {import("../../types").RetryOptions} [options.retry]
* @param {boolean} [options.allowAutoTopicCreation]
* @param {number} [options.authenticationTimeout]
* @param {number} [options.metadataMaxAge]
*/
constructor({
connectionPoolBuilder,
logger,
retry,
allowAutoTopicCreation,
authenticationTimeout,
metadataMaxAge,
}) {
this.rootLogger = logger
this.connectionPoolBuilder = connectionPoolBuilder
this.metadataMaxAge = metadataMaxAge || 0
this.logger = logger.namespace('BrokerPool')
this.retrier = createRetry(assign({}, retry))
this.createBroker = options =>
new Broker({
allowAutoTopicCreation,
authenticationTimeout,
...options,
})
this.brokers = {}
/** @type {Broker | undefined} */
this.seedBroker = undefined
/** @type {import("../../types").BrokerMetadata | null} */
this.metadata = null
this.metadataExpireAt = null
this.versions = null
}
/**
* @public
* @returns {Boolean}
*/
hasConnectedBrokers() {
const brokers = values(this.brokers)
return (
!!brokers.find(broker => broker.isConnected()) ||
(this.seedBroker ? this.seedBroker.isConnected() : false)
)
}
async createSeedBroker() {
if (this.seedBroker) {
await this.seedBroker.disconnect()
}
const connectionPool = await this.connectionPoolBuilder.build()
this.seedBroker = this.createBroker({
connectionPool,
logger: this.rootLogger,
})
}
/**
* @public
* @returns {Promise<void>}
*/
async connect() {
if (this.hasConnectedBrokers()) {
return
}
if (!this.seedBroker) {
await this.createSeedBroker()
}
return this.retrier(async (bail, retryCount, retryTime) => {
try {
await this.seedBroker.connect()
this.versions = this.seedBroker.versions
} catch (e) {
if (e.name === 'KafkaJSConnectionError' || e.type === 'ILLEGAL_SASL_STATE') {
// Connection builder will always rotate the seed broker
await this.createSeedBroker()
this.logger.error(
`Failed to connect to seed broker, trying another broker from the list: ${e.message}`,
{ retryCount, retryTime }
)
} else {
this.logger.error(e.message, { retryCount, retryTime })
}
if (e.retriable) throw e
bail(e)
}
})
}
/**
* @public
* @returns {Promise}
*/
async disconnect() {
this.seedBroker && (await this.seedBroker.disconnect())
await Promise.all(values(this.brokers).map(broker => broker.disconnect()))
this.brokers = {}
this.metadata = null
this.versions = null
}
/**
* @public
* @param {Object} destination
* @param {string} destination.host
* @param {number} destination.port
*/
removeBroker({ host, port }) {
const removedBroker = values(this.brokers).find(
broker => broker.connectionPool.host === host && broker.connectionPool.port === port
)
if (removedBroker) {
delete this.brokers[removedBroker.nodeId]
this.metadataExpireAt = null
if (this.seedBroker.nodeId === removedBroker.nodeId) {
this.seedBroker = shuffle(values(this.brokers))[0]
}
}
}
/**
* @public
* @param {Array<String>} topics
* @returns {Promise<null>}
*/
async refreshMetadata(topics) {
const broker = await this.findConnectedBroker()
const { host: seedHost, port: seedPort } = this.seedBroker.connectionPool
return this.retrier(async (bail, retryCount, retryTime) => {
try {
this.metadata = await broker.metadata(topics)
this.metadataExpireAt = Date.now() + this.metadataMaxAge
const replacedBrokers = []
this.brokers = await this.metadata.brokers.reduce(
async (resultPromise, { nodeId, host, port, rack }) => {
const result = await resultPromise
if (result[nodeId]) {
if (!hasBrokerBeenReplaced(result[nodeId], { host, port, rack })) {
return result
}
replacedBrokers.push(result[nodeId])
}
if (host === seedHost && port === seedPort) {
this.seedBroker.nodeId = nodeId
this.seedBroker.connectionPool.rack = rack
return assign(result, {
[nodeId]: this.seedBroker,
})
}
return assign(result, {
[nodeId]: this.createBroker({
logger: this.rootLogger,
versions: this.versions,
connectionPool: await this.connectionPoolBuilder.build({ host, port, rack }),
nodeId,
}),
})
},
this.brokers
)
const freshBrokerIds = this.metadata.brokers.map(({ nodeId }) => `${nodeId}`).sort()
const currentBrokerIds = keys(this.brokers).sort()
const unusedBrokerIds = arrayDiff(currentBrokerIds, freshBrokerIds)
const brokerDisconnects = unusedBrokerIds.map(nodeId => {
const broker = this.brokers[nodeId]
return broker.disconnect().then(() => {
delete this.brokers[nodeId]
})
})
const replacedBrokersDisconnects = replacedBrokers.map(broker => broker.disconnect())
await Promise.all([...brokerDisconnects, ...replacedBrokersDisconnects])
} catch (e) {
if (e.type === 'LEADER_NOT_AVAILABLE') {
throw e
}
bail(e)
}
})
}
/**
* Only refreshes metadata if the data is stale according to the `metadataMaxAge` param or does not contain information about the provided topics
*
* @public
* @param {Array<String>} topics
* @returns {Promise<null>}
*/
async refreshMetadataIfNecessary(topics) {
const shouldRefresh =
this.metadata == null ||
this.metadataExpireAt == null ||
Date.now() > this.metadataExpireAt ||
!topics.every(topic =>
this.metadata.topicMetadata.some(topicMetadata => topicMetadata.topic === topic)
)
if (shouldRefresh) {
return this.refreshMetadata(topics)
}
}
/** @type {() => string[]} */
getNodeIds() {
return keys(this.brokers)
}
/**
* @public
* @param {object} options
* @param {string} options.nodeId
* @returns {Promise<Broker>}
*/
async findBroker({ nodeId }) {
const broker = this.brokers[nodeId]
if (!broker) {
throw new KafkaJSBrokerNotFound(`Broker ${nodeId} not found in the cached metadata`)
}
await this.connectBroker(broker)
return broker
}
/**
* @public
* @param {(params: { nodeId: string, broker: Broker }) => Promise<T>} callback
* @returns {Promise<T>}
* @template T
*/
async withBroker(callback) {
const brokers = shuffle(keys(this.brokers))
if (brokers.length === 0) {
throw new KafkaJSBrokerNotFound('No brokers in the broker pool')
}
for (const nodeId of brokers) {
const broker = await this.findBroker({ nodeId })
try {
return await callback({ nodeId, broker })
} catch (e) {}
}
return null
}
/**
* @public
* @returns {Promise<Broker>}
*/
async findConnectedBroker() {
const nodeIds = shuffle(keys(this.brokers))
const connectedBrokerId = nodeIds.find(nodeId => this.brokers[nodeId].isConnected())
if (connectedBrokerId) {
return await this.findBroker({ nodeId: connectedBrokerId })
}
// Cycle through the nodes until one connects
for (const nodeId of nodeIds) {
try {
return await this.findBroker({ nodeId })
} catch (e) {}
}
// Failed to connect to all known brokers, metadata might be old
await this.connect()
return this.seedBroker
}
/**
* @private
* @param {Broker} broker
* @returns {Promise<null>}
*/
async connectBroker(broker) {
if (broker.isConnected()) {
return
}
return this.retrier(async (bail, retryCount, retryTime) => {
try {
await broker.connect()
} catch (e) {
if (e.name === 'KafkaJSConnectionError' || e.type === 'ILLEGAL_SASL_STATE') {
await broker.disconnect()
}
// To avoid reconnecting to an unavailable host, we bail on connection errors
// and refresh metadata on a higher level before reconnecting
if (e.name === 'KafkaJSConnectionError') {
return bail(e)
}
if (e.type === 'ILLEGAL_SASL_STATE') {
// Rebuild the connection pool since it can't recover from illegal SASL state
broker.connectionPool = await this.connectionPoolBuilder.build({
host: broker.connectionPool.host,
port: broker.connectionPool.port,
rack: broker.connectionPool.rack,
})
this.logger.error(`Failed to connect to broker, reconnecting`, { retryCount, retryTime })
throw new KafkaJSProtocolError(e, { retriable: true })
}
if (e.retriable) throw e
this.logger.error(e, { retryCount, retryTime, stack: e.stack })
bail(e)
}
})
}
}

117
node_modules/kafkajs/src/cluster/connectionPoolBuilder.js generated vendored Normal file
View file

@ -0,0 +1,117 @@
const { KafkaJSConnectionError, KafkaJSNonRetriableError } = require('../errors')
const ConnectionPool = require('../network/connectionPool')
/**
* @typedef {Object} ConnectionPoolBuilder
* @property {(destination?: { host?: string, port?: number, rack?: string }) => Promise<ConnectionPool>} build
*/
/**
* @param {Object} options
* @param {import("../../types").ISocketFactory} [options.socketFactory]
* @param {string[]|(() => string[])} options.brokers
* @param {Object} [options.ssl]
* @param {Object} [options.sasl]
* @param {string} options.clientId
* @param {number} options.requestTimeout
* @param {boolean} [options.enforceRequestTimeout]
* @param {number} [options.connectionTimeout]
* @param {number} [options.maxInFlightRequests]
* @param {import("../../types").RetryOptions} [options.retry]
* @param {import("../../types").Logger} options.logger
* @param {import("../instrumentation/emitter")} [options.instrumentationEmitter]
* @param {number} [options.reauthenticationThreshold]
* @returns {ConnectionPoolBuilder}
*/
module.exports = ({
socketFactory,
brokers,
ssl,
sasl,
clientId,
requestTimeout,
enforceRequestTimeout,
connectionTimeout,
maxInFlightRequests,
logger,
instrumentationEmitter = null,
reauthenticationThreshold,
}) => {
let index = 0
const isValidBroker = broker => {
return broker && typeof broker === 'string' && broker.length > 0
}
const validateBrokers = brokers => {
if (!brokers) {
throw new KafkaJSNonRetriableError(`Failed to connect: brokers should not be null`)
}
if (Array.isArray(brokers)) {
if (!brokers.length) {
throw new KafkaJSNonRetriableError(`Failed to connect: brokers array is empty`)
}
brokers.forEach((broker, index) => {
if (!isValidBroker(broker)) {
throw new KafkaJSNonRetriableError(
`Failed to connect: broker at index ${index} is invalid "${typeof broker}"`
)
}
})
}
}
const getBrokers = async () => {
let list
if (typeof brokers === 'function') {
try {
list = await brokers()
} catch (e) {
const wrappedError = new KafkaJSConnectionError(
`Failed to connect: "config.brokers" threw: ${e.message}`
)
wrappedError.stack = `${wrappedError.name}\n Caused by: ${e.stack}`
throw wrappedError
}
} else {
list = brokers
}
validateBrokers(list)
return list
}
return {
build: async ({ host, port, rack } = {}) => {
if (!host) {
const list = await getBrokers()
const randomBroker = list[index++ % list.length]
host = randomBroker.split(':')[0]
port = Number(randomBroker.split(':')[1])
}
return new ConnectionPool({
host,
port,
rack,
sasl,
ssl,
clientId,
socketFactory,
connectionTimeout,
requestTimeout,
enforceRequestTimeout,
maxInFlightRequests,
instrumentationEmitter,
logger,
reauthenticationThreshold,
})
},
}
}

539
node_modules/kafkajs/src/cluster/index.js generated vendored Normal file
View file

@ -0,0 +1,539 @@
const BrokerPool = require('./brokerPool')
const Lock = require('../utils/lock')
const sharedPromiseTo = require('../utils/sharedPromiseTo')
const createRetry = require('../retry')
const connectionPoolBuilder = require('./connectionPoolBuilder')
const { EARLIEST_OFFSET, LATEST_OFFSET } = require('../constants')
const {
KafkaJSError,
KafkaJSBrokerNotFound,
KafkaJSMetadataNotLoaded,
KafkaJSTopicMetadataNotLoaded,
KafkaJSGroupCoordinatorNotFound,
} = require('../errors')
const COORDINATOR_TYPES = require('../protocol/coordinatorTypes')
const { keys } = Object
const mergeTopics = (obj, { topic, partitions }) => ({
...obj,
[topic]: [...(obj[topic] || []), ...partitions],
})
const PRIVATE = {
CONNECT: Symbol('private:Cluster:connect'),
REFRESH_METADATA: Symbol('private:Cluster:refreshMetadata'),
REFRESH_METADATA_IF_NECESSARY: Symbol('private:Cluster:refreshMetadataIfNecessary'),
FIND_CONTROLLER_BROKER: Symbol('private:Cluster:findControllerBroker'),
}
module.exports = class Cluster {
/**
* @param {Object} options
* @param {Array<string>} options.brokers example: ['127.0.0.1:9092', '127.0.0.1:9094']
* @param {Object} options.ssl
* @param {Object} options.sasl
* @param {string} options.clientId
* @param {number} options.connectionTimeout - in milliseconds
* @param {number} options.authenticationTimeout - in milliseconds
* @param {number} options.reauthenticationThreshold - in milliseconds
* @param {number} [options.requestTimeout=30000] - in milliseconds
* @param {boolean} [options.enforceRequestTimeout]
* @param {number} options.metadataMaxAge - in milliseconds
* @param {boolean} options.allowAutoTopicCreation
* @param {number} options.maxInFlightRequests
* @param {number} options.isolationLevel
* @param {import("../../types").RetryOptions} options.retry
* @param {import("../../types").Logger} options.logger
* @param {import("../../types").ISocketFactory} options.socketFactory
* @param {Map} [options.offsets]
* @param {import("../instrumentation/emitter")} [options.instrumentationEmitter=null]
*/
constructor({
logger: rootLogger,
socketFactory,
brokers,
ssl,
sasl,
clientId,
connectionTimeout,
authenticationTimeout,
reauthenticationThreshold,
requestTimeout = 30000,
enforceRequestTimeout,
metadataMaxAge,
retry,
allowAutoTopicCreation,
maxInFlightRequests,
isolationLevel,
instrumentationEmitter = null,
offsets = new Map(),
}) {
this.rootLogger = rootLogger
this.logger = rootLogger.namespace('Cluster')
this.retrier = createRetry(retry)
this.connectionPoolBuilder = connectionPoolBuilder({
logger: rootLogger,
instrumentationEmitter,
socketFactory,
brokers,
ssl,
sasl,
clientId,
connectionTimeout,
requestTimeout,
enforceRequestTimeout,
maxInFlightRequests,
reauthenticationThreshold,
})
this.targetTopics = new Set()
this.mutatingTargetTopics = new Lock({
description: `updating target topics`,
timeout: requestTimeout,
})
this.isolationLevel = isolationLevel
this.brokerPool = new BrokerPool({
connectionPoolBuilder: this.connectionPoolBuilder,
logger: this.rootLogger,
retry,
allowAutoTopicCreation,
authenticationTimeout,
metadataMaxAge,
})
this.committedOffsetsByGroup = offsets
this[PRIVATE.CONNECT] = sharedPromiseTo(async () => {
return await this.brokerPool.connect()
})
this[PRIVATE.REFRESH_METADATA] = sharedPromiseTo(async () => {
return await this.brokerPool.refreshMetadata(Array.from(this.targetTopics))
})
this[PRIVATE.REFRESH_METADATA_IF_NECESSARY] = sharedPromiseTo(async () => {
return await this.brokerPool.refreshMetadataIfNecessary(Array.from(this.targetTopics))
})
this[PRIVATE.FIND_CONTROLLER_BROKER] = sharedPromiseTo(async () => {
const { metadata } = this.brokerPool
if (!metadata || metadata.controllerId == null) {
throw new KafkaJSMetadataNotLoaded('Topic metadata not loaded')
}
const broker = await this.findBroker({ nodeId: metadata.controllerId })
if (!broker) {
throw new KafkaJSBrokerNotFound(
`Controller broker with id ${metadata.controllerId} not found in the cached metadata`
)
}
return broker
})
}
isConnected() {
return this.brokerPool.hasConnectedBrokers()
}
/**
* @public
* @returns {Promise<void>}
*/
async connect() {
await this[PRIVATE.CONNECT]()
}
/**
* @public
* @returns {Promise<void>}
*/
async disconnect() {
await this.brokerPool.disconnect()
}
/**
* @public
* @param {object} destination
* @param {String} destination.host
* @param {Number} destination.port
*/
removeBroker({ host, port }) {
this.brokerPool.removeBroker({ host, port })
}
/**
* @public
* @returns {Promise<void>}
*/
async refreshMetadata() {
await this[PRIVATE.REFRESH_METADATA]()
}
/**
* @public
* @returns {Promise<void>}
*/
async refreshMetadataIfNecessary() {
await this[PRIVATE.REFRESH_METADATA_IF_NECESSARY]()
}
/**
* @public
* @returns {Promise<import("../../types").BrokerMetadata>}
*/
async metadata({ topics = [] } = {}) {
return this.retrier(async (bail, retryCount, retryTime) => {
try {
await this.brokerPool.refreshMetadataIfNecessary(topics)
return this.brokerPool.withBroker(async ({ broker }) => broker.metadata(topics))
} catch (e) {
if (e.type === 'LEADER_NOT_AVAILABLE') {
throw e
}
bail(e)
}
})
}
/**
* @public
* @param {string} topic
* @return {Promise}
*/
async addTargetTopic(topic) {
return this.addMultipleTargetTopics([topic])
}
/**
* @public
* @param {string[]} topics
* @return {Promise}
*/
async addMultipleTargetTopics(topics) {
await this.mutatingTargetTopics.acquire()
try {
const previousSize = this.targetTopics.size
const previousTopics = new Set(this.targetTopics)
for (const topic of topics) {
this.targetTopics.add(topic)
}
const hasChanged = previousSize !== this.targetTopics.size || !this.brokerPool.metadata
if (hasChanged) {
try {
await this.refreshMetadata()
} catch (e) {
if (
e.type === 'INVALID_TOPIC_EXCEPTION' ||
e.type === 'UNKNOWN_TOPIC_OR_PARTITION' ||
e.type === 'TOPIC_AUTHORIZATION_FAILED'
) {
this.targetTopics = previousTopics
}
throw e
}
}
} finally {
await this.mutatingTargetTopics.release()
}
}
/** @type {() => string[]} */
getNodeIds() {
return this.brokerPool.getNodeIds()
}
/**
* @public
* @param {object} options
* @param {string} options.nodeId
* @returns {Promise<import("../../types").Broker>}
*/
async findBroker({ nodeId }) {
try {
return await this.brokerPool.findBroker({ nodeId })
} catch (e) {
// The client probably has stale metadata
if (
e.name === 'KafkaJSBrokerNotFound' ||
e.name === 'KafkaJSLockTimeout' ||
e.name === 'KafkaJSConnectionError'
) {
await this.refreshMetadata()
}
throw e
}
}
/**
* @public
* @returns {Promise<import("../../types").Broker>}
*/
async findControllerBroker() {
return await this[PRIVATE.FIND_CONTROLLER_BROKER]()
}
/**
* @public
* @param {string} topic
* @returns {import("../../types").PartitionMetadata[]} Example:
* [{
* isr: [2],
* leader: 2,
* partitionErrorCode: 0,
* partitionId: 0,
* replicas: [2],
* }]
*/
findTopicPartitionMetadata(topic) {
const { metadata } = this.brokerPool
if (!metadata || !metadata.topicMetadata) {
throw new KafkaJSTopicMetadataNotLoaded('Topic metadata not loaded', { topic })
}
const topicMetadata = metadata.topicMetadata.find(t => t.topic === topic)
return topicMetadata ? topicMetadata.partitionMetadata : []
}
/**
* @public
* @param {string} topic
* @param {(number|string)[]} partitions
* @returns {Object} Object with leader and partitions. For partitions 0 and 5
* the result could be:
* { '0': [0], '2': [5] }
*
* where the key is the nodeId.
*/
findLeaderForPartitions(topic, partitions) {
const partitionMetadata = this.findTopicPartitionMetadata(topic)
return partitions.reduce((result, id) => {
const partitionId = parseInt(id, 10)
const metadata = partitionMetadata.find(p => p.partitionId === partitionId)
if (!metadata) {
return result
}
if (metadata.leader === null || metadata.leader === undefined) {
throw new KafkaJSError('Invalid partition metadata', { topic, partitionId, metadata })
}
const { leader } = metadata
const current = result[leader] || []
return { ...result, [leader]: [...current, partitionId] }
}, {})
}
/**
* @public
* @param {object} params
* @param {string} params.groupId
* @param {import("../protocol/coordinatorTypes").CoordinatorType} [params.coordinatorType=0]
* @returns {Promise<import("../../types").Broker>}
*/
async findGroupCoordinator({ groupId, coordinatorType = COORDINATOR_TYPES.GROUP }) {
return this.retrier(async (bail, retryCount, retryTime) => {
try {
const { coordinator } = await this.findGroupCoordinatorMetadata({
groupId,
coordinatorType,
})
return await this.findBroker({ nodeId: coordinator.nodeId })
} catch (e) {
// A new broker can join the cluster before we have the chance
// to refresh metadata
if (e.name === 'KafkaJSBrokerNotFound' || e.type === 'GROUP_COORDINATOR_NOT_AVAILABLE') {
this.logger.debug(`${e.message}, refreshing metadata and trying again...`, {
groupId,
retryCount,
retryTime,
})
await this.refreshMetadata()
throw e
}
if (e.code === 'ECONNREFUSED') {
// During maintenance the current coordinator can go down; findBroker will
// refresh metadata and re-throw the error. findGroupCoordinator has to re-throw
// the error to go through the retry cycle.
throw e
}
bail(e)
}
})
}
/**
* @public
* @param {object} params
* @param {string} params.groupId
* @param {import("../protocol/coordinatorTypes").CoordinatorType} [params.coordinatorType=0]
* @returns {Promise<Object>}
*/
async findGroupCoordinatorMetadata({ groupId, coordinatorType }) {
const brokerMetadata = await this.brokerPool.withBroker(async ({ nodeId, broker }) => {
return await this.retrier(async (bail, retryCount, retryTime) => {
try {
const brokerMetadata = await broker.findGroupCoordinator({ groupId, coordinatorType })
this.logger.debug('Found group coordinator', {
broker: brokerMetadata.host,
nodeId: brokerMetadata.coordinator.nodeId,
})
return brokerMetadata
} catch (e) {
this.logger.debug('Tried to find group coordinator', {
nodeId,
groupId,
error: e,
})
if (e.type === 'GROUP_COORDINATOR_NOT_AVAILABLE') {
this.logger.debug('Group coordinator not available, retrying...', {
nodeId,
retryCount,
retryTime,
})
throw e
}
bail(e)
}
})
})
if (brokerMetadata) {
return brokerMetadata
}
throw new KafkaJSGroupCoordinatorNotFound('Failed to find group coordinator')
}
/**
* @param {object} topicConfiguration
* @returns {number}
*/
defaultOffset({ fromBeginning }) {
return fromBeginning ? EARLIEST_OFFSET : LATEST_OFFSET
}
/**
* @public
* @param {Array<Object>} topics
* [
* {
* topic: 'my-topic-name',
* partitions: [{ partition: 0 }],
* fromBeginning: false
* }
* ]
* @returns {Promise<import("../../types").TopicOffsets[]>} example:
* [
* {
* topic: 'my-topic-name',
* partitions: [
* { partition: 0, offset: '1' },
* { partition: 1, offset: '2' },
* { partition: 2, offset: '1' },
* ],
* },
* ]
*/
async fetchTopicsOffset(topics) {
const partitionsPerBroker = {}
const topicConfigurations = {}
const addDefaultOffset = topic => partition => {
const { timestamp } = topicConfigurations[topic]
return { ...partition, timestamp }
}
// Index all topics and partitions per leader (nodeId)
for (const topicData of topics) {
const { topic, partitions, fromBeginning, fromTimestamp } = topicData
const partitionsPerLeader = this.findLeaderForPartitions(
topic,
partitions.map(p => p.partition)
)
const timestamp =
fromTimestamp != null ? fromTimestamp : this.defaultOffset({ fromBeginning })
topicConfigurations[topic] = { timestamp }
keys(partitionsPerLeader).forEach(nodeId => {
partitionsPerBroker[nodeId] = partitionsPerBroker[nodeId] || {}
partitionsPerBroker[nodeId][topic] = partitions.filter(p =>
partitionsPerLeader[nodeId].includes(p.partition)
)
})
}
// Create a list of requests to fetch the offset of all partitions
const requests = keys(partitionsPerBroker).map(async nodeId => {
const broker = await this.findBroker({ nodeId })
const partitions = partitionsPerBroker[nodeId]
const { responses: topicOffsets } = await broker.listOffsets({
isolationLevel: this.isolationLevel,
topics: keys(partitions).map(topic => ({
topic,
partitions: partitions[topic].map(addDefaultOffset(topic)),
})),
})
return topicOffsets
})
// Execute all requests, merge and normalize the responses
const responses = await Promise.all(requests)
const partitionsPerTopic = responses.flat().reduce(mergeTopics, {})
return keys(partitionsPerTopic).map(topic => ({
topic,
partitions: partitionsPerTopic[topic].map(({ partition, offset }) => ({
partition,
offset,
})),
}))
}
/**
* Retrieve the object mapping for committed offsets for a single consumer group
* @param {object} options
* @param {string} options.groupId
* @returns {Object}
*/
committedOffsets({ groupId }) {
if (!this.committedOffsetsByGroup.has(groupId)) {
this.committedOffsetsByGroup.set(groupId, {})
}
return this.committedOffsetsByGroup.get(groupId)
}
/**
* Mark offset as committed for a single consumer group's topic-partition
* @param {object} options
* @param {string} options.groupId
* @param {string} options.topic
* @param {string|number} options.partition
* @param {string} options.offset
*/
markOffsetAsCommitted({ groupId, topic, partition, offset }) {
const committedOffsets = this.committedOffsets({ groupId })
committedOffsets[topic] = committedOffsets[topic] || {}
committedOffsets[topic][partition] = offset
}
}

9
node_modules/kafkajs/src/constants.js generated vendored Normal file
View file

@ -0,0 +1,9 @@
const EARLIEST_OFFSET = -2
const LATEST_OFFSET = -1
const INT_32_MAX_VALUE = Math.pow(2, 31) - 1
module.exports = {
EARLIEST_OFFSET,
LATEST_OFFSET,
INT_32_MAX_VALUE,
}

87
node_modules/kafkajs/src/consumer/assignerProtocol.js generated vendored Normal file
View file

@ -0,0 +1,87 @@
const Encoder = require('../protocol/encoder')
const Decoder = require('../protocol/decoder')
const MemberMetadata = {
/**
* @param {Object} metadata
* @param {number} metadata.version
* @param {Array<string>} metadata.topics
* @param {Buffer} [metadata.userData=Buffer.alloc(0)]
*
* @returns Buffer
*/
encode({ version, topics, userData = Buffer.alloc(0) }) {
return new Encoder()
.writeInt16(version)
.writeArray(topics)
.writeBytes(userData).buffer
},
/**
* @param {Buffer} buffer
* @returns {Object}
*/
decode(buffer) {
const decoder = new Decoder(buffer)
return {
version: decoder.readInt16(),
topics: decoder.readArray(d => d.readString()),
userData: decoder.readBytes(),
}
},
}
const MemberAssignment = {
/**
* @param {object} options
* @param {number} options.version
* @param {Object<String,Array>} options.assignment, example:
* {
* 'topic-A': [0, 2, 4, 6],
* 'topic-B': [0, 2],
* }
* @param {Buffer} [options.userData=Buffer.alloc(0)]
*
* @returns Buffer
*/
encode({ version, assignment, userData = Buffer.alloc(0) }) {
return new Encoder()
.writeInt16(version)
.writeArray(
Object.keys(assignment).map(topic =>
new Encoder().writeString(topic).writeArray(assignment[topic])
)
)
.writeBytes(userData).buffer
},
/**
* @param {Buffer} buffer
* @returns {Object|null}
*/
decode(buffer) {
const decoder = new Decoder(buffer)
const decodePartitions = d => d.readInt32()
const decodeAssignment = d => ({
topic: d.readString(),
partitions: d.readArray(decodePartitions),
})
const indexAssignment = (obj, { topic, partitions }) =>
Object.assign(obj, { [topic]: partitions })
if (!decoder.canReadInt16()) {
return null
}
return {
version: decoder.readInt16(),
assignment: decoder.readArray(decodeAssignment).reduce(indexAssignment, {}),
userData: decoder.readBytes(),
}
},
}
module.exports = {
MemberMetadata,
MemberAssignment,
}

5
node_modules/kafkajs/src/consumer/assigners/index.js generated vendored Normal file
View file

@ -0,0 +1,5 @@
const roundRobin = require('./roundRobinAssigner')
module.exports = {
roundRobin,
}

81
node_modules/kafkajs/src/consumer/assigners/roundRobinAssigner/index.js generated vendored Normal file
View file

@ -0,0 +1,81 @@
const { MemberMetadata, MemberAssignment } = require('../../assignerProtocol')
/**
* RoundRobinAssigner
* @type {import('types').PartitionAssigner}
*/
module.exports = ({ cluster }) => ({
name: 'RoundRobinAssigner',
version: 0,
/**
* Assign the topics to the provided members.
*
* The members array contains information about each member, `memberMetadata` is the result of the
* `protocol` operation.
*
* @param {object} group
* @param {import('types').GroupMember[]} group.members array of members, e.g:
[{ memberId: 'test-5f93f5a3', memberMetadata: Buffer }]
* @param {string[]} group.topics
* @returns {Promise<import('types').GroupMemberAssignment[]>} object partitions per topic per member, e.g:
* [
* {
* memberId: 'test-5f93f5a3',
* memberAssignment: {
* 'topic-A': [0, 2, 4, 6],
* 'topic-B': [1],
* },
* },
* {
* memberId: 'test-3d3d5341',
* memberAssignment: {
* 'topic-A': [1, 3, 5],
* 'topic-B': [0, 2],
* },
* }
* ]
*/
async assign({ members, topics }) {
const membersCount = members.length
const sortedMembers = members.map(({ memberId }) => memberId).sort()
const assignment = {}
const topicsPartitions = topics.flatMap(topic => {
const partitionMetadata = cluster.findTopicPartitionMetadata(topic)
return partitionMetadata.map(m => ({ topic: topic, partitionId: m.partitionId }))
})
topicsPartitions.forEach((topicPartition, i) => {
const assignee = sortedMembers[i % membersCount]
if (!assignment[assignee]) {
assignment[assignee] = Object.create(null)
}
if (!assignment[assignee][topicPartition.topic]) {
assignment[assignee][topicPartition.topic] = []
}
assignment[assignee][topicPartition.topic].push(topicPartition.partitionId)
})
return Object.keys(assignment).map(memberId => ({
memberId,
memberAssignment: MemberAssignment.encode({
version: this.version,
assignment: assignment[memberId],
}),
}))
},
protocol({ topics }) {
return {
name: this.name,
metadata: MemberMetadata.encode({
version: this.version,
topics,
}),
}
},
})

112
node_modules/kafkajs/src/consumer/batch.js generated vendored Normal file
View file

@ -0,0 +1,112 @@
const Long = require('../utils/long')
const filterAbortedMessages = require('./filterAbortedMessages')
/**
* A batch collects messages returned from a single fetch call.
*
* A batch could contain _multiple_ Kafka RecordBatches.
*/
module.exports = class Batch {
constructor(topic, fetchedOffset, partitionData) {
this.fetchedOffset = fetchedOffset
const longFetchedOffset = Long.fromValue(this.fetchedOffset)
const { abortedTransactions, messages } = partitionData
this.topic = topic
this.partition = partitionData.partition
this.highWatermark = partitionData.highWatermark
this.rawMessages = messages
// Apparently fetch can return different offsets than the target offset provided to the fetch API.
// Discard messages that are not in the requested offset
// https://github.com/apache/kafka/blob/bf237fa7c576bd141d78fdea9f17f65ea269c290/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L912
this.messagesWithinOffset = this.rawMessages.filter(message =>
Long.fromValue(message.offset).gte(longFetchedOffset)
)
// 1. Don't expose aborted messages
// 2. Don't expose control records
// @see https://kafka.apache.org/documentation/#controlbatch
this.messages = filterAbortedMessages({
messages: this.messagesWithinOffset,
abortedTransactions,
}).filter(message => !message.isControlRecord)
}
isEmpty() {
return this.messages.length === 0
}
isEmptyIncludingFiltered() {
return this.messagesWithinOffset.length === 0
}
/**
* If the batch contained raw messages (i.e was not truly empty) but all messages were filtered out due to
* log compaction, control records or other reasons
*/
isEmptyDueToFiltering() {
return this.isEmpty() && this.rawMessages.length > 0
}
isEmptyControlRecord() {
return (
this.isEmpty() && this.messagesWithinOffset.some(({ isControlRecord }) => isControlRecord)
)
}
/**
* With compressed messages, it's possible for the returned messages to have offsets smaller than the starting offset.
* These messages will be filtered out (i.e. they are not even included in this.messagesWithinOffset)
* If these are the only messages, the batch will appear as an empty batch.
*
* isEmpty() and isEmptyIncludingFiltered() will always return true if the batch is empty,
* but this method will only return true if the batch is empty due to log compacted messages.
*
* @returns boolean True if the batch is empty, because of log compacted messages in the partition.
*/
isEmptyDueToLogCompactedMessages() {
const hasMessages = this.rawMessages.length > 0
return hasMessages && this.isEmptyIncludingFiltered()
}
firstOffset() {
return this.isEmptyIncludingFiltered() ? null : this.messagesWithinOffset[0].offset
}
lastOffset() {
if (this.isEmptyDueToLogCompactedMessages()) {
return this.fetchedOffset
}
if (this.isEmptyIncludingFiltered()) {
return Long.fromValue(this.highWatermark)
.add(-1)
.toString()
}
return this.messagesWithinOffset[this.messagesWithinOffset.length - 1].offset
}
/**
* Returns the lag based on the last offset in the batch (also known as "high")
*/
offsetLag() {
const lastOffsetOfPartition = Long.fromValue(this.highWatermark).add(-1)
const lastConsumedOffset = Long.fromValue(this.lastOffset())
return lastOffsetOfPartition.add(lastConsumedOffset.multiply(-1)).toString()
}
/**
* Returns the lag based on the first offset in the batch
*/
offsetLagLow() {
if (this.isEmptyIncludingFiltered()) {
return '0'
}
const lastOffsetOfPartition = Long.fromValue(this.highWatermark).add(-1)
const firstConsumedOffset = Long.fromValue(this.firstOffset())
return lastOffsetOfPartition.add(firstConsumedOffset.multiply(-1)).toString()
}
}

759
node_modules/kafkajs/src/consumer/consumerGroup.js generated vendored Normal file
View file

@ -0,0 +1,759 @@
const sleep = require('../utils/sleep')
const websiteUrl = require('../utils/websiteUrl')
const arrayDiff = require('../utils/arrayDiff')
const createRetry = require('../retry')
const sharedPromiseTo = require('../utils/sharedPromiseTo')
const OffsetManager = require('./offsetManager')
const Batch = require('./batch')
const SeekOffsets = require('./seekOffsets')
const SubscriptionState = require('./subscriptionState')
const {
events: { GROUP_JOIN, HEARTBEAT, CONNECT, RECEIVED_UNSUBSCRIBED_TOPICS },
} = require('./instrumentationEvents')
const { MemberAssignment } = require('./assignerProtocol')
const {
KafkaJSError,
KafkaJSNonRetriableError,
KafkaJSStaleTopicMetadataAssignment,
isRebalancing,
} = require('../errors')
const { keys } = Object
const STALE_METADATA_ERRORS = [
'LEADER_NOT_AVAILABLE',
// Fetch before v9 uses NOT_LEADER_FOR_PARTITION
'NOT_LEADER_FOR_PARTITION',
// Fetch after v9 uses {FENCED,UNKNOWN}_LEADER_EPOCH
'FENCED_LEADER_EPOCH',
'UNKNOWN_LEADER_EPOCH',
'UNKNOWN_TOPIC_OR_PARTITION',
]
const PRIVATE = {
JOIN: Symbol('private:ConsumerGroup:join'),
SYNC: Symbol('private:ConsumerGroup:sync'),
SHARED_HEARTBEAT: Symbol('private:ConsumerGroup:sharedHeartbeat'),
}
module.exports = class ConsumerGroup {
/**
* @param {object} options
* @param {import('../../types').RetryOptions} options.retry
* @param {import('../../types').Cluster} options.cluster
* @param {string} options.groupId
* @param {string[]} options.topics
* @param {Record<string, { fromBeginning?: boolean }>} options.topicConfigurations
* @param {import('../../types').Logger} options.logger
* @param {import('../instrumentation/emitter')} options.instrumentationEmitter
* @param {import('../../types').Assigner[]} options.assigners
* @param {number} options.sessionTimeout
* @param {number} options.rebalanceTimeout
* @param {number} options.maxBytesPerPartition
* @param {number} options.minBytes
* @param {number} options.maxBytes
* @param {number} options.maxWaitTimeInMs
* @param {boolean} options.autoCommit
* @param {number} options.autoCommitInterval
* @param {number} options.autoCommitThreshold
* @param {number} options.isolationLevel
* @param {string} options.rackId
* @param {number} options.metadataMaxAge
*/
constructor({
retry,
cluster,
groupId,
topics,
topicConfigurations,
logger,
instrumentationEmitter,
assigners,
sessionTimeout,
rebalanceTimeout,
maxBytesPerPartition,
minBytes,
maxBytes,
maxWaitTimeInMs,
autoCommit,
autoCommitInterval,
autoCommitThreshold,
isolationLevel,
rackId,
metadataMaxAge,
}) {
/** @type {import("../../types").Cluster} */
this.cluster = cluster
this.groupId = groupId
this.topics = topics
this.topicsSubscribed = topics
this.topicConfigurations = topicConfigurations
this.logger = logger.namespace('ConsumerGroup')
this.instrumentationEmitter = instrumentationEmitter
this.retrier = createRetry(Object.assign({}, retry))
this.assigners = assigners
this.sessionTimeout = sessionTimeout
this.rebalanceTimeout = rebalanceTimeout
this.maxBytesPerPartition = maxBytesPerPartition
this.minBytes = minBytes
this.maxBytes = maxBytes
this.maxWaitTime = maxWaitTimeInMs
this.autoCommit = autoCommit
this.autoCommitInterval = autoCommitInterval
this.autoCommitThreshold = autoCommitThreshold
this.isolationLevel = isolationLevel
this.rackId = rackId
this.metadataMaxAge = metadataMaxAge
this.seekOffset = new SeekOffsets()
this.coordinator = null
this.generationId = null
this.leaderId = null
this.memberId = null
this.members = null
this.groupProtocol = null
this.partitionsPerSubscribedTopic = null
/**
* Preferred read replica per topic and partition
*
* Each of the partitions tracks the preferred read replica (`nodeId`) and a timestamp
* until when that preference is valid.
*
* @type {{[topicName: string]: {[partition: number]: {nodeId: number, expireAt: number}}}}
*/
this.preferredReadReplicasPerTopicPartition = {}
this.offsetManager = null
this.subscriptionState = new SubscriptionState()
this.lastRequest = Date.now()
this[PRIVATE.SHARED_HEARTBEAT] = sharedPromiseTo(async ({ interval }) => {
const { groupId, generationId, memberId } = this
const now = Date.now()
if (memberId && now >= this.lastRequest + interval) {
const payload = {
groupId,
memberId,
groupGenerationId: generationId,
}
await this.coordinator.heartbeat(payload)
this.instrumentationEmitter.emit(HEARTBEAT, payload)
this.lastRequest = Date.now()
}
})
}
isLeader() {
return this.leaderId && this.memberId === this.leaderId
}
getNodeIds() {
return this.cluster.getNodeIds()
}
async connect() {
await this.cluster.connect()
this.instrumentationEmitter.emit(CONNECT)
await this.cluster.refreshMetadataIfNecessary()
}
async [PRIVATE.JOIN]() {
const { groupId, sessionTimeout, rebalanceTimeout } = this
this.coordinator = await this.cluster.findGroupCoordinator({ groupId })
const groupData = await this.coordinator.joinGroup({
groupId,
sessionTimeout,
rebalanceTimeout,
memberId: this.memberId || '',
groupProtocols: this.assigners.map(assigner =>
assigner.protocol({
topics: this.topicsSubscribed,
})
),
})
this.generationId = groupData.generationId
this.leaderId = groupData.leaderId
this.memberId = groupData.memberId
this.members = groupData.members
this.groupProtocol = groupData.groupProtocol
}
async leave() {
const { groupId, memberId } = this
if (memberId) {
await this.coordinator.leaveGroup({ groupId, memberId })
this.memberId = null
}
}
async [PRIVATE.SYNC]() {
let assignment = []
const {
groupId,
generationId,
memberId,
members,
groupProtocol,
topics,
topicsSubscribed,
coordinator,
} = this
if (this.isLeader()) {
this.logger.debug('Chosen as group leader', { groupId, generationId, memberId, topics })
const assigner = this.assigners.find(({ name }) => name === groupProtocol)
if (!assigner) {
throw new KafkaJSNonRetriableError(
`Unsupported partition assigner "${groupProtocol}", the assigner wasn't found in the assigners list`
)
}
await this.cluster.refreshMetadata()
assignment = await assigner.assign({ members, topics: topicsSubscribed })
this.logger.debug('Group assignment', {
groupId,
generationId,
groupProtocol,
assignment,
topics: topicsSubscribed,
})
}
// Keep track of the partitions for the subscribed topics
this.partitionsPerSubscribedTopic = this.generatePartitionsPerSubscribedTopic()
const { memberAssignment } = await this.coordinator.syncGroup({
groupId,
generationId,
memberId,
groupAssignment: assignment,
})
const decodedMemberAssignment = MemberAssignment.decode(memberAssignment)
const decodedAssignment =
decodedMemberAssignment != null ? decodedMemberAssignment.assignment : {}
this.logger.debug('Received assignment', {
groupId,
generationId,
memberId,
memberAssignment: decodedAssignment,
})
const assignedTopics = keys(decodedAssignment)
const topicsNotSubscribed = arrayDiff(assignedTopics, topicsSubscribed)
if (topicsNotSubscribed.length > 0) {
const payload = {
groupId,
generationId,
memberId,
assignedTopics,
topicsSubscribed,
topicsNotSubscribed,
}
this.instrumentationEmitter.emit(RECEIVED_UNSUBSCRIBED_TOPICS, payload)
this.logger.warn('Consumer group received unsubscribed topics', {
...payload,
helpUrl: websiteUrl(
'docs/faq',
'why-am-i-receiving-messages-for-topics-i-m-not-subscribed-to'
),
})
}
// Remove unsubscribed topics from the list
const safeAssignment = arrayDiff(assignedTopics, topicsNotSubscribed)
const currentMemberAssignment = safeAssignment.map(topic => ({
topic,
partitions: decodedAssignment[topic],
}))
// Check if the consumer is aware of all assigned partitions
for (const assignment of currentMemberAssignment) {
const { topic, partitions: assignedPartitions } = assignment
const knownPartitions = this.partitionsPerSubscribedTopic.get(topic)
const isAwareOfAllAssignedPartitions = assignedPartitions.every(partition =>
knownPartitions.includes(partition)
)
if (!isAwareOfAllAssignedPartitions) {
this.logger.warn('Consumer is not aware of all assigned partitions, refreshing metadata', {
groupId,
generationId,
memberId,
topic,
knownPartitions,
assignedPartitions,
})
// If the consumer is not aware of all assigned partitions, refresh metadata
// and update the list of partitions per subscribed topic. It's enough to perform
// this operation once since refresh metadata will update metadata for all topics
await this.cluster.refreshMetadata()
this.partitionsPerSubscribedTopic = this.generatePartitionsPerSubscribedTopic()
break
}
}
this.topics = currentMemberAssignment.map(({ topic }) => topic)
this.subscriptionState.assign(currentMemberAssignment)
this.offsetManager = new OffsetManager({
cluster: this.cluster,
topicConfigurations: this.topicConfigurations,
instrumentationEmitter: this.instrumentationEmitter,
memberAssignment: currentMemberAssignment.reduce(
(partitionsByTopic, { topic, partitions }) => ({
...partitionsByTopic,
[topic]: partitions,
}),
{}
),
autoCommit: this.autoCommit,
autoCommitInterval: this.autoCommitInterval,
autoCommitThreshold: this.autoCommitThreshold,
coordinator,
groupId,
generationId,
memberId,
})
}
joinAndSync() {
const startJoin = Date.now()
return this.retrier(async bail => {
try {
await this[PRIVATE.JOIN]()
await this[PRIVATE.SYNC]()
const memberAssignment = this.assigned().reduce(
(result, { topic, partitions }) => ({ ...result, [topic]: partitions }),
{}
)
const payload = {
groupId: this.groupId,
memberId: this.memberId,
leaderId: this.leaderId,
isLeader: this.isLeader(),
memberAssignment,
groupProtocol: this.groupProtocol,
duration: Date.now() - startJoin,
}
this.instrumentationEmitter.emit(GROUP_JOIN, payload)
this.logger.info('Consumer has joined the group', payload)
} catch (e) {
if (isRebalancing(e)) {
// Rebalance in progress isn't a retriable protocol error since the consumer
// has to go through find coordinator and join again before it can
// actually retry the operation. We wrap the original error in a retriable error
// here instead in order to restart the join + sync sequence using the retrier.
throw new KafkaJSError(e)
}
if (e.type === 'UNKNOWN_MEMBER_ID') {
this.memberId = null
throw new KafkaJSError(e)
}
bail(e)
}
})
}
/**
* @param {import("../../types").TopicPartition} topicPartition
*/
resetOffset({ topic, partition }) {
this.offsetManager.resetOffset({ topic, partition })
}
/**
* @param {import("../../types").TopicPartitionOffset} topicPartitionOffset
*/
resolveOffset({ topic, partition, offset }) {
this.offsetManager.resolveOffset({ topic, partition, offset })
}
/**
* Update the consumer offset for the given topic/partition. This will be used
* on the next fetch. If this API is invoked for the same topic/partition more
* than once, the latest offset will be used on the next fetch.
*
* @param {import("../../types").TopicPartitionOffset} topicPartitionOffset
*/
seek({ topic, partition, offset }) {
this.seekOffset.set(topic, partition, offset)
}
pause(topicPartitions) {
this.logger.info(`Pausing fetching from ${topicPartitions.length} topics`, {
topicPartitions,
})
this.subscriptionState.pause(topicPartitions)
}
resume(topicPartitions) {
this.logger.info(`Resuming fetching from ${topicPartitions.length} topics`, {
topicPartitions,
})
this.subscriptionState.resume(topicPartitions)
}
assigned() {
return this.subscriptionState.assigned()
}
paused() {
return this.subscriptionState.paused()
}
/**
* @param {string} topic
* @param {string} partition
* @returns {boolean} whether the specified topic-partition are paused or not
*/
isPaused(topic, partition) {
return this.subscriptionState.isPaused(topic, partition)
}
async commitOffsetsIfNecessary() {
await this.offsetManager.commitOffsetsIfNecessary()
}
async commitOffsets(offsets) {
await this.offsetManager.commitOffsets(offsets)
}
uncommittedOffsets() {
return this.offsetManager.uncommittedOffsets()
}
async heartbeat({ interval }) {
return this[PRIVATE.SHARED_HEARTBEAT]({ interval })
}
async fetch(nodeId) {
try {
await this.cluster.refreshMetadataIfNecessary()
this.checkForStaleAssignment()
let topicPartitions = this.subscriptionState.assigned()
topicPartitions = this.filterPartitionsByNode(nodeId, topicPartitions)
await this.seekOffsets(topicPartitions)
const committedOffsets = this.offsetManager.committedOffsets()
const activeTopicPartitions = this.getActiveTopicPartitions()
const requests = topicPartitions
.map(({ topic, partitions }) => ({
topic,
partitions: partitions
.filter(
partition =>
/**
* When recovering from OffsetOutOfRange, each partition can recover
* concurrently, which invalidates resolved and committed offsets as part
* of the recovery mechanism (see OffsetManager.clearOffsets). In concurrent
* scenarios this can initiate a new fetch with invalid offsets.
*
* This was further highlighted by https://github.com/tulios/kafkajs/pull/570,
* which increased concurrency, making this more likely to happen.
*
* This is solved by only making requests for partitions with initialized offsets.
*
* See the following pull request which explains the context of the problem:
* @issue https://github.com/tulios/kafkajs/pull/578
*/
committedOffsets[topic][partition] != null &&
activeTopicPartitions[topic].has(partition)
)
.map(partition => ({
partition,
fetchOffset: this.offsetManager.nextOffset(topic, partition).toString(),
maxBytes: this.maxBytesPerPartition,
})),
}))
.filter(({ partitions }) => partitions.length)
if (!requests.length) {
await sleep(this.maxWaitTime)
return []
}
const broker = await this.cluster.findBroker({ nodeId })
const { responses } = await broker.fetch({
maxWaitTime: this.maxWaitTime,
minBytes: this.minBytes,
maxBytes: this.maxBytes,
isolationLevel: this.isolationLevel,
topics: requests,
rackId: this.rackId,
})
return responses.flatMap(({ topicName, partitions }) => {
const topicRequestData = requests.find(({ topic }) => topic === topicName)
let preferredReadReplicas = this.preferredReadReplicasPerTopicPartition[topicName]
if (!preferredReadReplicas) {
this.preferredReadReplicasPerTopicPartition[topicName] = preferredReadReplicas = {}
}
return partitions
.filter(
({ partition }) =>
!this.seekOffset.has(topicName, partition) &&
!this.subscriptionState.isPaused(topicName, partition)
)
.map(partitionData => {
const { partition, preferredReadReplica } = partitionData
if (preferredReadReplica != null && preferredReadReplica !== -1) {
const { nodeId: currentPreferredReadReplica } = preferredReadReplicas[partition] || {}
if (currentPreferredReadReplica !== preferredReadReplica) {
this.logger.info(`Preferred read replica is now ${preferredReadReplica}`, {
groupId: this.groupId,
memberId: this.memberId,
topic: topicName,
partition,
})
}
preferredReadReplicas[partition] = {
nodeId: preferredReadReplica,
expireAt: Date.now() + this.metadataMaxAge,
}
}
const partitionRequestData = topicRequestData.partitions.find(
({ partition }) => partition === partitionData.partition
)
const fetchedOffset = partitionRequestData.fetchOffset
return new Batch(topicName, fetchedOffset, partitionData)
})
})
} catch (e) {
await this.recoverFromFetch(e)
return []
}
}
async recoverFromFetch(e) {
if (STALE_METADATA_ERRORS.includes(e.type) || e.name === 'KafkaJSTopicMetadataNotLoaded') {
this.logger.debug('Stale cluster metadata, refreshing...', {
groupId: this.groupId,
memberId: this.memberId,
error: e.message,
})
await this.cluster.refreshMetadata()
await this.joinAndSync()
return
}
if (e.name === 'KafkaJSStaleTopicMetadataAssignment') {
this.logger.warn(`${e.message}, resync group`, {
groupId: this.groupId,
memberId: this.memberId,
topic: e.topic,
unknownPartitions: e.unknownPartitions,
})
await this.joinAndSync()
return
}
if (e.name === 'KafkaJSOffsetOutOfRange') {
await this.recoverFromOffsetOutOfRange(e)
return
}
if (e.name === 'KafkaJSConnectionClosedError') {
this.cluster.removeBroker({ host: e.host, port: e.port })
return
}
if (e.name === 'KafkaJSBrokerNotFound' || e.name === 'KafkaJSConnectionClosedError') {
this.logger.debug(`${e.message}, refreshing metadata and retrying...`)
await this.cluster.refreshMetadata()
return
}
throw e
}
async recoverFromOffsetOutOfRange(e) {
// If we are fetching from a follower try with the leader before resetting offsets
const preferredReadReplicas = this.preferredReadReplicasPerTopicPartition[e.topic]
if (preferredReadReplicas && typeof preferredReadReplicas[e.partition] === 'number') {
this.logger.info('Offset out of range while fetching from follower, retrying with leader', {
topic: e.topic,
partition: e.partition,
groupId: this.groupId,
memberId: this.memberId,
})
delete preferredReadReplicas[e.partition]
} else {
this.logger.error('Offset out of range, resetting to default offset', {
topic: e.topic,
partition: e.partition,
groupId: this.groupId,
memberId: this.memberId,
})
await this.offsetManager.setDefaultOffset({
topic: e.topic,
partition: e.partition,
})
}
}
generatePartitionsPerSubscribedTopic() {
const map = new Map()
for (const topic of this.topicsSubscribed) {
const partitions = this.cluster
.findTopicPartitionMetadata(topic)
.map(m => m.partitionId)
.sort()
map.set(topic, partitions)
}
return map
}
checkForStaleAssignment() {
if (!this.partitionsPerSubscribedTopic) {
return
}
const newPartitionsPerSubscribedTopic = this.generatePartitionsPerSubscribedTopic()
for (const [topic, partitions] of newPartitionsPerSubscribedTopic) {
const diff = arrayDiff(partitions, this.partitionsPerSubscribedTopic.get(topic))
if (diff.length > 0) {
throw new KafkaJSStaleTopicMetadataAssignment('Topic has been updated', {
topic,
unknownPartitions: diff,
})
}
}
}
async seekOffsets(topicPartitions) {
for (const { topic, partitions } of topicPartitions) {
for (const partition of partitions) {
const seekEntry = this.seekOffset.pop(topic, partition)
if (!seekEntry) {
continue
}
this.logger.debug('Seek offset', {
groupId: this.groupId,
memberId: this.memberId,
seek: seekEntry,
})
await this.offsetManager.seek(seekEntry)
}
}
await this.offsetManager.resolveOffsets()
}
hasSeekOffset({ topic, partition }) {
return this.seekOffset.has(topic, partition)
}
/**
* For each of the partitions find the best nodeId to read it from
*
* @param {string} topic
* @param {number[]} partitions
* @returns {{[nodeId: number]: number[]}} per-node assignment of partitions
* @see Cluster~findLeaderForPartitions
*/
// Invariant: The resulting object has each partition referenced exactly once
findReadReplicaForPartitions(topic, partitions) {
const partitionMetadata = this.cluster.findTopicPartitionMetadata(topic)
const preferredReadReplicas = this.preferredReadReplicasPerTopicPartition[topic]
return partitions.reduce((result, id) => {
const partitionId = parseInt(id, 10)
const metadata = partitionMetadata.find(p => p.partitionId === partitionId)
if (!metadata) {
return result
}
if (metadata.leader == null) {
throw new KafkaJSError('Invalid partition metadata', { topic, partitionId, metadata })
}
// Pick the preferred replica if there is one, and it isn't known to be offline, otherwise the leader.
let nodeId = metadata.leader
if (preferredReadReplicas) {
const { nodeId: preferredReadReplica, expireAt } = preferredReadReplicas[partitionId] || {}
if (Date.now() >= expireAt) {
this.logger.debug('Preferred read replica information has expired, using leader', {
topic,
partitionId,
groupId: this.groupId,
memberId: this.memberId,
preferredReadReplica,
leader: metadata.leader,
})
// Drop the entry
delete preferredReadReplicas[partitionId]
} else if (preferredReadReplica != null) {
// Valid entry, check whether it is not offline
// Note that we don't delete the preference here, and rather hope that eventually that replica comes online again
const offlineReplicas = metadata.offlineReplicas
if (Array.isArray(offlineReplicas) && offlineReplicas.includes(nodeId)) {
this.logger.debug('Preferred read replica is offline, using leader', {
topic,
partitionId,
groupId: this.groupId,
memberId: this.memberId,
preferredReadReplica,
leader: metadata.leader,
})
} else {
nodeId = preferredReadReplica
}
}
}
const current = result[nodeId] || []
return { ...result, [nodeId]: [...current, partitionId] }
}, {})
}
filterPartitionsByNode(nodeId, topicPartitions) {
return topicPartitions.map(({ topic, partitions }) => ({
topic,
partitions: this.findReadReplicaForPartitions(topic, partitions)[nodeId] || [],
}))
}
getActiveTopicPartitions() {
const activeSubscriptionState = this.subscriptionState.active()
const activeTopicPartitions = {}
activeSubscriptionState.forEach(({ topic, partitions }) => {
activeTopicPartitions[topic] = new Set(partitions)
})
return activeTopicPartitions
}
}

99
node_modules/kafkajs/src/consumer/fetchManager.js generated vendored Normal file
View file

@ -0,0 +1,99 @@
const seq = require('../utils/seq')
const createFetcher = require('./fetcher')
const createWorker = require('./worker')
const createWorkerQueue = require('./workerQueue')
const { KafkaJSFetcherRebalanceError, KafkaJSNoBrokerAvailableError } = require('../errors')
/** @typedef {ReturnType<typeof createFetchManager>} FetchManager */
/**
* @param {object} options
* @param {import('../../types').Logger} options.logger
* @param {() => number[]} options.getNodeIds
* @param {(nodeId: number) => Promise<import('../../types').Batch[]>} options.fetch
* @param {import('./worker').Handler<T>} options.handler
* @param {number} [options.concurrency]
* @template T
*/
const createFetchManager = ({
logger: rootLogger,
getNodeIds,
fetch,
handler,
concurrency = 1,
}) => {
const logger = rootLogger.namespace('FetchManager')
const workers = seq(concurrency, workerId => createWorker({ handler, workerId }))
const workerQueue = createWorkerQueue({ workers })
let fetchers = []
const getFetchers = () => fetchers
const createFetchers = () => {
const nodeIds = getNodeIds()
const partitionAssignments = new Map()
if (nodeIds.length === 0) {
throw new KafkaJSNoBrokerAvailableError()
}
const validateShouldRebalance = () => {
const current = getNodeIds()
const hasChanged =
nodeIds.length !== current.length || nodeIds.some(nodeId => !current.includes(nodeId))
if (hasChanged && current.length !== 0) {
throw new KafkaJSFetcherRebalanceError()
}
}
const fetchers = nodeIds.map(nodeId =>
createFetcher({
nodeId,
workerQueue,
partitionAssignments,
fetch: async nodeId => {
validateShouldRebalance()
return fetch(nodeId)
},
logger,
})
)
logger.debug(`Created ${fetchers.length} fetchers`, { nodeIds, concurrency })
return fetchers
}
const start = async () => {
logger.debug('Starting...')
while (true) {
fetchers = createFetchers()
try {
await Promise.all(fetchers.map(fetcher => fetcher.start()))
} catch (error) {
await stop()
if (error instanceof KafkaJSFetcherRebalanceError) {
logger.debug('Rebalancing fetchers...')
continue
}
throw error
}
break
}
}
const stop = async () => {
logger.debug('Stopping fetchers...')
await Promise.all(fetchers.map(fetcher => fetcher.stop()))
logger.debug('Stopped fetchers')
}
return { start, stop, getFetchers }
}
module.exports = createFetchManager

86
node_modules/kafkajs/src/consumer/fetcher.js generated vendored Normal file
View file

@ -0,0 +1,86 @@
const EventEmitter = require('events')
/**
* Fetches data from all assigned nodes, waits for workerQueue to drain and repeats.
*
* @param {object} options
* @param {number} options.nodeId
* @param {import('./workerQueue').WorkerQueue} options.workerQueue
* @param {Map<string, string[]>} options.partitionAssignments
* @param {(nodeId: number) => Promise<T[]>} options.fetch
* @param {import('../../types').Logger} options.logger
* @template T
*/
const createFetcher = ({
nodeId,
workerQueue,
partitionAssignments,
fetch,
logger: rootLogger,
}) => {
const logger = rootLogger.namespace(`Fetcher ${nodeId}`)
const emitter = new EventEmitter()
let isRunning = false
const getWorkerQueue = () => workerQueue
const assignmentKey = ({ topic, partition }) => `${topic}|${partition}`
const getAssignedFetcher = batch => partitionAssignments.get(assignmentKey(batch))
const assignTopicPartition = batch => partitionAssignments.set(assignmentKey(batch), nodeId)
const unassignTopicPartition = batch => partitionAssignments.delete(assignmentKey(batch))
const filterUnassignedBatches = batches =>
batches.filter(batch => {
const assignedFetcher = getAssignedFetcher(batch)
if (assignedFetcher != null && assignedFetcher !== nodeId) {
logger.info(
'Filtering out batch due to partition already being processed by another fetcher',
{
topic: batch.topic,
partition: batch.partition,
assignedFetcher: assignedFetcher,
fetcher: nodeId,
}
)
return false
}
return true
})
const start = async () => {
if (isRunning) return
isRunning = true
while (isRunning) {
try {
const batches = await fetch(nodeId)
if (isRunning) {
const availableBatches = filterUnassignedBatches(batches)
if (availableBatches.length > 0) {
availableBatches.forEach(assignTopicPartition)
try {
await workerQueue.push(...availableBatches)
} finally {
availableBatches.forEach(unassignTopicPartition)
}
}
}
} catch (error) {
isRunning = false
emitter.emit('end')
throw error
}
}
emitter.emit('end')
}
const stop = async () => {
if (!isRunning) return
isRunning = false
await new Promise(resolve => emitter.once('end', () => resolve()))
}
return { start, stop, getWorkerQueue }
}
module.exports = createFetcher

64
node_modules/kafkajs/src/consumer/filterAbortedMessages.js generated vendored Normal file
View file

@ -0,0 +1,64 @@
const Long = require('../utils/long')
const ABORTED_MESSAGE_KEY = Buffer.from([0, 0, 0, 0])
const isAbortMarker = ({ key }) => {
// Handle null/undefined keys.
if (!key) return false
// Cast key to buffer defensively
return Buffer.from(key).equals(ABORTED_MESSAGE_KEY)
}
/**
* Remove messages marked as aborted according to the aborted transactions list.
*
* Start of an aborted transaction is determined by message offset.
* End of an aborted transaction is determined by control messages.
* @param {Message[]} messages
* @param {Transaction[]} [abortedTransactions]
* @returns {Message[]} Messages which did not participate in an aborted transaction
*
* @typedef {object} Message
* @param {Buffer} key
* @param {lastOffset} key Int64
* @param {RecordBatch} batchContext
*
* @typedef {object} Transaction
* @param {string} firstOffset Int64
* @param {string} producerId Int64
*
* @typedef {object} RecordBatch
* @param {string} producerId Int64
* @param {boolean} inTransaction
*/
module.exports = ({ messages, abortedTransactions }) => {
const currentAbortedTransactions = new Map()
if (!abortedTransactions || !abortedTransactions.length) {
return messages
}
const remainingAbortedTransactions = [...abortedTransactions]
return messages.filter(message => {
// If the message offset is GTE the first offset of the next aborted transaction
// then we have stepped into an aborted transaction.
if (
remainingAbortedTransactions.length &&
Long.fromValue(message.offset).gte(remainingAbortedTransactions[0].firstOffset)
) {
const { producerId } = remainingAbortedTransactions.shift()
currentAbortedTransactions.set(producerId, true)
}
const { producerId, inTransaction } = message.batchContext
if (isAbortMarker(message)) {
// Transaction is over, we no longer need to ignore messages from this producer
currentAbortedTransactions.delete(producerId)
} else if (currentAbortedTransactions.has(producerId) && inTransaction) {
return false
}
return true
})
}

522
node_modules/kafkajs/src/consumer/index.js generated vendored Normal file
View file

@ -0,0 +1,522 @@
const Long = require('../utils/long')
const createRetry = require('../retry')
const { initialRetryTime } = require('../retry/defaults')
const ConsumerGroup = require('./consumerGroup')
const Runner = require('./runner')
const { events, wrap: wrapEvent, unwrap: unwrapEvent } = require('./instrumentationEvents')
const InstrumentationEventEmitter = require('../instrumentation/emitter')
const { KafkaJSNonRetriableError } = require('../errors')
const { roundRobin } = require('./assigners')
const { EARLIEST_OFFSET, LATEST_OFFSET } = require('../constants')
const ISOLATION_LEVEL = require('../protocol/isolationLevel')
const sharedPromiseTo = require('../utils/sharedPromiseTo')
const { keys, values } = Object
const { CONNECT, DISCONNECT, STOP, CRASH } = events
const eventNames = values(events)
const eventKeys = keys(events)
.map(key => `consumer.events.${key}`)
.join(', ')
const specialOffsets = [
Long.fromValue(EARLIEST_OFFSET).toString(),
Long.fromValue(LATEST_OFFSET).toString(),
]
/**
* @param {Object} params
* @param {import("../../types").Cluster} params.cluster
* @param {String} params.groupId
* @param {import('../../types').RetryOptions} [params.retry]
* @param {import('../../types').Logger} params.logger
* @param {import('../../types').PartitionAssigner[]} [params.partitionAssigners]
* @param {number} [params.sessionTimeout]
* @param {number} [params.rebalanceTimeout]
* @param {number} [params.heartbeatInterval]
* @param {number} [params.maxBytesPerPartition]
* @param {number} [params.minBytes]
* @param {number} [params.maxBytes]
* @param {number} [params.maxWaitTimeInMs]
* @param {number} [params.isolationLevel]
* @param {string} [params.rackId]
* @param {InstrumentationEventEmitter} [params.instrumentationEmitter]
* @param {number} params.metadataMaxAge
*
* @returns {import("../../types").Consumer}
*/
module.exports = ({
cluster,
groupId,
retry,
logger: rootLogger,
partitionAssigners = [roundRobin],
sessionTimeout = 30000,
rebalanceTimeout = 60000,
heartbeatInterval = 3000,
maxBytesPerPartition = 1048576, // 1MB
minBytes = 1,
maxBytes = 10485760, // 10MB
maxWaitTimeInMs = 5000,
isolationLevel = ISOLATION_LEVEL.READ_COMMITTED,
rackId = '',
instrumentationEmitter: rootInstrumentationEmitter,
metadataMaxAge,
}) => {
if (!groupId) {
throw new KafkaJSNonRetriableError('Consumer groupId must be a non-empty string.')
}
const logger = rootLogger.namespace('Consumer')
const instrumentationEmitter = rootInstrumentationEmitter || new InstrumentationEventEmitter()
const assigners = partitionAssigners.map(createAssigner =>
createAssigner({ groupId, logger, cluster })
)
/** @type {Record<string, { fromBeginning?: boolean }>} */
const topics = {}
let runner = null
/** @type {ConsumerGroup} */
let consumerGroup = null
let restartTimeout = null
if (heartbeatInterval >= sessionTimeout) {
throw new KafkaJSNonRetriableError(
`Consumer heartbeatInterval (${heartbeatInterval}) must be lower than sessionTimeout (${sessionTimeout}). It is recommended to set heartbeatInterval to approximately a third of the sessionTimeout.`
)
}
/** @type {import("../../types").Consumer["connect"]} */
const connect = async () => {
await cluster.connect()
instrumentationEmitter.emit(CONNECT)
}
/** @type {import("../../types").Consumer["disconnect"]} */
const disconnect = async () => {
try {
await stop()
logger.debug('consumer has stopped, disconnecting', { groupId })
await cluster.disconnect()
instrumentationEmitter.emit(DISCONNECT)
} catch (e) {
logger.error(`Caught error when disconnecting the consumer: ${e.message}`, {
stack: e.stack,
groupId,
})
throw e
}
}
/** @type {import("../../types").Consumer["stop"]} */
const stop = sharedPromiseTo(async () => {
try {
if (runner) {
await runner.stop()
runner = null
consumerGroup = null
instrumentationEmitter.emit(STOP)
}
clearTimeout(restartTimeout)
logger.info('Stopped', { groupId })
} catch (e) {
logger.error(`Caught error when stopping the consumer: ${e.message}`, {
stack: e.stack,
groupId,
})
throw e
}
})
/** @type {import("../../types").Consumer["subscribe"]} */
const subscribe = async ({ topic, topics: subscriptionTopics, fromBeginning = false }) => {
if (consumerGroup) {
throw new KafkaJSNonRetriableError('Cannot subscribe to topic while consumer is running')
}
if (!topic && !subscriptionTopics) {
throw new KafkaJSNonRetriableError('Missing required argument "topics"')
}
if (subscriptionTopics != null && !Array.isArray(subscriptionTopics)) {
throw new KafkaJSNonRetriableError('Argument "topics" must be an array')
}
const subscriptions = subscriptionTopics || [topic]
for (const subscription of subscriptions) {
if (typeof subscription !== 'string' && !(subscription instanceof RegExp)) {
throw new KafkaJSNonRetriableError(
`Invalid topic ${subscription} (${typeof subscription}), the topic name has to be a String or a RegExp`
)
}
}
const hasRegexSubscriptions = subscriptions.some(subscription => subscription instanceof RegExp)
const metadata = hasRegexSubscriptions ? await cluster.metadata() : undefined
const topicsToSubscribe = []
for (const subscription of subscriptions) {
const isRegExp = subscription instanceof RegExp
if (isRegExp) {
const topicRegExp = subscription
const matchedTopics = metadata.topicMetadata
.map(({ topic: topicName }) => topicName)
.filter(topicName => topicRegExp.test(topicName))
logger.debug('Subscription based on RegExp', {
groupId,
topicRegExp: topicRegExp.toString(),
matchedTopics,
})
topicsToSubscribe.push(...matchedTopics)
} else {
topicsToSubscribe.push(subscription)
}
}
for (const t of topicsToSubscribe) {
topics[t] = { fromBeginning }
}
await cluster.addMultipleTargetTopics(topicsToSubscribe)
}
/** @type {import("../../types").Consumer["run"]} */
const run = async ({
autoCommit = true,
autoCommitInterval = null,
autoCommitThreshold = null,
eachBatchAutoResolve = true,
partitionsConsumedConcurrently: concurrency = 1,
eachBatch = null,
eachMessage = null,
} = {}) => {
if (consumerGroup) {
logger.warn('consumer#run was called, but the consumer is already running', { groupId })
return
}
const start = async onCrash => {
logger.info('Starting', { groupId })
consumerGroup = new ConsumerGroup({
logger: rootLogger,
topics: keys(topics),
topicConfigurations: topics,
retry,
cluster,
groupId,
assigners,
sessionTimeout,
rebalanceTimeout,
maxBytesPerPartition,
minBytes,
maxBytes,
maxWaitTimeInMs,
instrumentationEmitter,
isolationLevel,
rackId,
metadataMaxAge,
autoCommit,
autoCommitInterval,
autoCommitThreshold,
})
runner = new Runner({
logger: rootLogger,
consumerGroup,
instrumentationEmitter,
heartbeatInterval,
retry,
autoCommit,
eachBatchAutoResolve,
eachBatch,
eachMessage,
onCrash,
concurrency,
})
await runner.start()
}
const onCrash = async e => {
logger.error(`Crash: ${e.name}: ${e.message}`, {
groupId,
retryCount: e.retryCount,
stack: e.stack,
})
if (e.name === 'KafkaJSConnectionClosedError') {
cluster.removeBroker({ host: e.host, port: e.port })
}
await disconnect()
const getOriginalCause = error => {
if (error.cause) {
return getOriginalCause(error.cause)
}
return error
}
const isErrorRetriable =
e.name === 'KafkaJSNumberOfRetriesExceeded' || getOriginalCause(e).retriable === true
const shouldRestart =
isErrorRetriable &&
(!retry ||
!retry.restartOnFailure ||
(await retry.restartOnFailure(e).catch(error => {
logger.error(
'Caught error when invoking user-provided "restartOnFailure" callback. Defaulting to restarting.',
{
error: error.message || error,
cause: e.message || e,
groupId,
}
)
return true
})))
instrumentationEmitter.emit(CRASH, {
error: e,
groupId,
restart: shouldRestart,
})
if (shouldRestart) {
const retryTime = e.retryTime || (retry && retry.initialRetryTime) || initialRetryTime
logger.error(`Restarting the consumer in ${retryTime}ms`, {
retryCount: e.retryCount,
retryTime,
groupId,
})
restartTimeout = setTimeout(() => start(onCrash), retryTime)
}
}
await start(onCrash)
}
/** @type {import("../../types").Consumer["on"]} */
const on = (eventName, listener) => {
if (!eventNames.includes(eventName)) {
throw new KafkaJSNonRetriableError(`Event name should be one of ${eventKeys}`)
}
return instrumentationEmitter.addListener(unwrapEvent(eventName), event => {
event.type = wrapEvent(event.type)
Promise.resolve(listener(event)).catch(e => {
logger.error(`Failed to execute listener: ${e.message}`, {
eventName,
stack: e.stack,
})
})
})
}
/**
* @type {import("../../types").Consumer["commitOffsets"]}
* @param topicPartitions
* Example: [{ topic: 'topic-name', partition: 0, offset: '1', metadata: 'event-id-3' }]
*/
const commitOffsets = async (topicPartitions = []) => {
const commitsByTopic = topicPartitions.reduce(
(payload, { topic, partition, offset, metadata = null }) => {
if (!topic) {
throw new KafkaJSNonRetriableError(`Invalid topic ${topic}`)
}
if (isNaN(partition)) {
throw new KafkaJSNonRetriableError(
`Invalid partition, expected a number received ${partition}`
)
}
let commitOffset
try {
commitOffset = Long.fromValue(offset)
} catch (_) {
throw new KafkaJSNonRetriableError(`Invalid offset, expected a long received ${offset}`)
}
if (commitOffset.lessThan(0)) {
throw new KafkaJSNonRetriableError('Offset must not be a negative number')
}
if (metadata !== null && typeof metadata !== 'string') {
throw new KafkaJSNonRetriableError(
`Invalid offset metadata, expected string or null, received ${metadata}`
)
}
const topicCommits = payload[topic] || []
topicCommits.push({ partition, offset: commitOffset, metadata })
return { ...payload, [topic]: topicCommits }
},
{}
)
if (!consumerGroup) {
throw new KafkaJSNonRetriableError(
'Consumer group was not initialized, consumer#run must be called first'
)
}
const topics = Object.keys(commitsByTopic)
return runner.commitOffsets({
topics: topics.map(topic => {
return {
topic,
partitions: commitsByTopic[topic],
}
}),
})
}
/** @type {import("../../types").Consumer["seek"]} */
const seek = ({ topic, partition, offset }) => {
if (!topic) {
throw new KafkaJSNonRetriableError(`Invalid topic ${topic}`)
}
if (isNaN(partition)) {
throw new KafkaJSNonRetriableError(
`Invalid partition, expected a number received ${partition}`
)
}
let seekOffset
try {
seekOffset = Long.fromValue(offset)
} catch (_) {
throw new KafkaJSNonRetriableError(`Invalid offset, expected a long received ${offset}`)
}
if (seekOffset.lessThan(0) && !specialOffsets.includes(seekOffset.toString())) {
throw new KafkaJSNonRetriableError('Offset must not be a negative number')
}
if (!consumerGroup) {
throw new KafkaJSNonRetriableError(
'Consumer group was not initialized, consumer#run must be called first'
)
}
consumerGroup.seek({ topic, partition, offset: seekOffset.toString() })
}
/** @type {import("../../types").Consumer["describeGroup"]} */
const describeGroup = async () => {
const coordinator = await cluster.findGroupCoordinator({ groupId })
const retrier = createRetry(retry)
return retrier(async () => {
const { groups } = await coordinator.describeGroups({ groupIds: [groupId] })
return groups.find(group => group.groupId === groupId)
})
}
/**
* @type {import("../../types").Consumer["pause"]}
* @param topicPartitions
* Example: [{ topic: 'topic-name', partitions: [1, 2] }]
*/
const pause = (topicPartitions = []) => {
for (const topicPartition of topicPartitions) {
if (!topicPartition || !topicPartition.topic) {
throw new KafkaJSNonRetriableError(
`Invalid topic ${(topicPartition && topicPartition.topic) || topicPartition}`
)
} else if (
typeof topicPartition.partitions !== 'undefined' &&
(!Array.isArray(topicPartition.partitions) || topicPartition.partitions.some(isNaN))
) {
throw new KafkaJSNonRetriableError(
`Array of valid partitions required to pause specific partitions instead of ${topicPartition.partitions}`
)
}
}
if (!consumerGroup) {
throw new KafkaJSNonRetriableError(
'Consumer group was not initialized, consumer#run must be called first'
)
}
consumerGroup.pause(topicPartitions)
}
/**
* Returns the list of topic partitions paused on this consumer
*
* @type {import("../../types").Consumer["paused"]}
*/
const paused = () => {
if (!consumerGroup) {
return []
}
return consumerGroup.paused()
}
/**
* @type {import("../../types").Consumer["resume"]}
* @param topicPartitions
* Example: [{ topic: 'topic-name', partitions: [1, 2] }]
*/
const resume = (topicPartitions = []) => {
for (const topicPartition of topicPartitions) {
if (!topicPartition || !topicPartition.topic) {
throw new KafkaJSNonRetriableError(
`Invalid topic ${(topicPartition && topicPartition.topic) || topicPartition}`
)
} else if (
typeof topicPartition.partitions !== 'undefined' &&
(!Array.isArray(topicPartition.partitions) || topicPartition.partitions.some(isNaN))
) {
throw new KafkaJSNonRetriableError(
`Array of valid partitions required to resume specific partitions instead of ${topicPartition.partitions}`
)
}
}
if (!consumerGroup) {
throw new KafkaJSNonRetriableError(
'Consumer group was not initialized, consumer#run must be called first'
)
}
consumerGroup.resume(topicPartitions)
}
/**
* @return {Object} logger
*/
const getLogger = () => logger
return {
connect,
disconnect,
subscribe,
stop,
run,
commitOffsets,
seek,
describeGroup,
pause,
paused,
resume,
on,
events,
logger: getLogger,
}
}

40
node_modules/kafkajs/src/consumer/instrumentationEvents.js generated vendored Normal file
View file

@ -0,0 +1,40 @@
const swapObject = require('../utils/swapObject')
const InstrumentationEventType = require('../instrumentation/eventType')
const networkEvents = require('../network/instrumentationEvents')
const consumerType = InstrumentationEventType('consumer')
/** @type {import('types').ConsumerEvents} */
const events = {
HEARTBEAT: consumerType('heartbeat'),
COMMIT_OFFSETS: consumerType('commit_offsets'),
GROUP_JOIN: consumerType('group_join'),
FETCH: consumerType('fetch'),
FETCH_START: consumerType('fetch_start'),
START_BATCH_PROCESS: consumerType('start_batch_process'),
END_BATCH_PROCESS: consumerType('end_batch_process'),
CONNECT: consumerType('connect'),
DISCONNECT: consumerType('disconnect'),
STOP: consumerType('stop'),
CRASH: consumerType('crash'),
REBALANCING: consumerType('rebalancing'),
RECEIVED_UNSUBSCRIBED_TOPICS: consumerType('received_unsubscribed_topics'),
REQUEST: consumerType(networkEvents.NETWORK_REQUEST),
REQUEST_TIMEOUT: consumerType(networkEvents.NETWORK_REQUEST_TIMEOUT),
REQUEST_QUEUE_SIZE: consumerType(networkEvents.NETWORK_REQUEST_QUEUE_SIZE),
}
const wrappedEvents = {
[events.REQUEST]: networkEvents.NETWORK_REQUEST,
[events.REQUEST_TIMEOUT]: networkEvents.NETWORK_REQUEST_TIMEOUT,
[events.REQUEST_QUEUE_SIZE]: networkEvents.NETWORK_REQUEST_QUEUE_SIZE,
}
const reversedWrappedEvents = swapObject(wrappedEvents)
const unwrap = eventName => wrappedEvents[eventName] || eventName
const wrap = eventName => reversedWrappedEvents[eventName] || eventName
module.exports = {
events,
wrap,
unwrap,
}

384
node_modules/kafkajs/src/consumer/offsetManager/index.js generated vendored Normal file
View file

@ -0,0 +1,384 @@
const Long = require('../../utils/long')
const isInvalidOffset = require('./isInvalidOffset')
const initializeConsumerOffsets = require('./initializeConsumerOffsets')
const {
events: { COMMIT_OFFSETS },
} = require('../instrumentationEvents')
const { keys, assign } = Object
const indexTopics = topics => topics.reduce((obj, topic) => assign(obj, { [topic]: {} }), {})
const PRIVATE = {
COMMITTED_OFFSETS: Symbol('private:OffsetManager:committedOffsets'),
}
module.exports = class OffsetManager {
/**
* @param {Object} options
* @param {import("../../../types").Cluster} options.cluster
* @param {import("../../../types").Broker} options.coordinator
* @param {import("../../../types").IMemberAssignment} options.memberAssignment
* @param {boolean} options.autoCommit
* @param {number | null} options.autoCommitInterval
* @param {number | null} options.autoCommitThreshold
* @param {{[topic: string]: { fromBeginning: boolean }}} options.topicConfigurations
* @param {import("../../instrumentation/emitter")} options.instrumentationEmitter
* @param {string} options.groupId
* @param {number} options.generationId
* @param {string} options.memberId
*/
constructor({
cluster,
coordinator,
memberAssignment,
autoCommit,
autoCommitInterval,
autoCommitThreshold,
topicConfigurations,
instrumentationEmitter,
groupId,
generationId,
memberId,
}) {
this.cluster = cluster
this.coordinator = coordinator
// memberAssignment format:
// {
// 'topic1': [0, 1, 2, 3],
// 'topic2': [0, 1, 2, 3, 4, 5],
// }
this.memberAssignment = memberAssignment
this.topicConfigurations = topicConfigurations
this.instrumentationEmitter = instrumentationEmitter
this.groupId = groupId
this.generationId = generationId
this.memberId = memberId
this.autoCommit = autoCommit
this.autoCommitInterval = autoCommitInterval
this.autoCommitThreshold = autoCommitThreshold
this.lastCommit = Date.now()
this.topics = keys(memberAssignment)
this.clearAllOffsets()
}
/**
* @param {string} topic
* @param {number} partition
* @returns {Long}
*/
nextOffset(topic, partition) {
if (!this.resolvedOffsets[topic][partition]) {
this.resolvedOffsets[topic][partition] = this.committedOffsets()[topic][partition]
}
let offset = this.resolvedOffsets[topic][partition]
if (isInvalidOffset(offset)) {
offset = '0'
}
return Long.fromValue(offset)
}
/**
* @returns {Promise<import("../../../types").Broker>}
*/
async getCoordinator() {
if (!this.coordinator.isConnected()) {
this.coordinator = await this.cluster.findBroker(this.coordinator)
}
return this.coordinator
}
/**
* @param {import("../../../types").TopicPartition} topicPartition
*/
resetOffset({ topic, partition }) {
this.resolvedOffsets[topic][partition] = this.committedOffsets()[topic][partition]
}
/**
* @param {import("../../../types").TopicPartitionOffset} topicPartitionOffset
*/
resolveOffset({ topic, partition, offset }) {
this.resolvedOffsets[topic][partition] = Long.fromValue(offset)
.add(1)
.toString()
}
/**
* @returns {Long}
*/
countResolvedOffsets() {
const committedOffsets = this.committedOffsets()
const subtractOffsets = (resolvedOffset, committedOffset) => {
const resolvedOffsetLong = Long.fromValue(resolvedOffset)
return isInvalidOffset(committedOffset)
? resolvedOffsetLong
: resolvedOffsetLong.subtract(Long.fromValue(committedOffset))
}
const subtractPartitionOffsets = (resolvedTopicOffsets, committedTopicOffsets) =>
keys(resolvedTopicOffsets).map(partition =>
subtractOffsets(resolvedTopicOffsets[partition], committedTopicOffsets[partition])
)
const subtractTopicOffsets = topic =>
subtractPartitionOffsets(this.resolvedOffsets[topic], committedOffsets[topic])
const offsetsDiff = this.topics.flatMap(subtractTopicOffsets)
return offsetsDiff.reduce((sum, offset) => sum.add(offset), Long.fromValue(0))
}
/**
* @param {import("../../../types").TopicPartition} topicPartition
*/
async setDefaultOffset({ topic, partition }) {
const { groupId, generationId, memberId } = this
const defaultOffset = this.cluster.defaultOffset(this.topicConfigurations[topic])
const coordinator = await this.getCoordinator()
await coordinator.offsetCommit({
groupId,
memberId,
groupGenerationId: generationId,
topics: [
{
topic,
partitions: [{ partition, offset: defaultOffset }],
},
],
})
this.clearOffsets({ topic, partition })
}
/**
* Commit the given offset to the topic/partition. If the consumer isn't assigned to the given
* topic/partition this method will be a NO-OP.
*
* @param {import("../../../types").TopicPartitionOffset} topicPartitionOffset
*/
async seek({ topic, partition, offset }) {
if (!this.memberAssignment[topic] || !this.memberAssignment[topic].includes(partition)) {
return
}
if (!this.autoCommit) {
this.resolveOffset({
topic,
partition,
offset: Long.fromValue(offset)
.subtract(1)
.toString(),
})
return
}
const { groupId, generationId, memberId } = this
const coordinator = await this.getCoordinator()
await coordinator.offsetCommit({
groupId,
memberId,
groupGenerationId: generationId,
topics: [
{
topic,
partitions: [{ partition, offset }],
},
],
})
this.clearOffsets({ topic, partition })
}
async commitOffsetsIfNecessary() {
const now = Date.now()
const timeoutReached =
this.autoCommitInterval != null && now >= this.lastCommit + this.autoCommitInterval
const thresholdReached =
this.autoCommitThreshold != null &&
this.countResolvedOffsets().gte(Long.fromValue(this.autoCommitThreshold))
if (timeoutReached || thresholdReached) {
return this.commitOffsets()
}
}
/**
* Return all locally resolved offsets which are not marked as committed, by topic-partition.
* @returns {import('../../../types').OffsetsByTopicPartition}
*/
uncommittedOffsets() {
const offsets = topic => keys(this.resolvedOffsets[topic])
const emptyPartitions = ({ partitions }) => partitions.length > 0
const toPartitions = topic => partition => ({
partition,
offset: this.resolvedOffsets[topic][partition],
})
const changedOffsets = topic => ({ partition, offset }) => {
return (
offset !== this.committedOffsets()[topic][partition] &&
Long.fromValue(offset).greaterThanOrEqual(0)
)
}
// Select and format updated partitions
const topicsWithPartitionsToCommit = this.topics
.map(topic => ({
topic,
partitions: offsets(topic)
.map(toPartitions(topic))
.filter(changedOffsets(topic)),
}))
.filter(emptyPartitions)
return { topics: topicsWithPartitionsToCommit }
}
async commitOffsets(offsets = {}) {
const { groupId, generationId, memberId } = this
const { topics = this.uncommittedOffsets().topics } = offsets
if (topics.length === 0) {
this.lastCommit = Date.now()
return
}
const payload = {
groupId,
memberId,
groupGenerationId: generationId,
topics,
}
try {
const coordinator = await this.getCoordinator()
await coordinator.offsetCommit(payload)
this.instrumentationEmitter.emit(COMMIT_OFFSETS, payload)
// Update local reference of committed offsets
topics.forEach(({ topic, partitions }) => {
const updatedOffsets = partitions.reduce(
(obj, { partition, offset }) => assign(obj, { [partition]: offset }),
{}
)
this[PRIVATE.COMMITTED_OFFSETS][topic] = assign(
{},
this.committedOffsets()[topic],
updatedOffsets
)
})
this.lastCommit = Date.now()
} catch (e) {
// metadata is stale, the coordinator has changed due to a restart or
// broker reassignment
if (e.type === 'NOT_COORDINATOR_FOR_GROUP') {
await this.cluster.refreshMetadata()
}
throw e
}
}
async resolveOffsets() {
const { groupId } = this
const invalidOffset = topic => partition => {
return isInvalidOffset(this.committedOffsets()[topic][partition])
}
const pendingPartitions = this.topics
.map(topic => ({
topic,
partitions: this.memberAssignment[topic]
.filter(invalidOffset(topic))
.map(partition => ({ partition })),
}))
.filter(t => t.partitions.length > 0)
if (pendingPartitions.length === 0) {
return
}
const coordinator = await this.getCoordinator()
const { responses: consumerOffsets } = await coordinator.offsetFetch({
groupId,
topics: pendingPartitions,
})
const unresolvedPartitions = consumerOffsets.map(({ topic, partitions }) =>
assign(
{
topic,
partitions: partitions
.filter(({ offset }) => isInvalidOffset(offset))
.map(({ partition }) => assign({ partition })),
},
this.topicConfigurations[topic]
)
)
const indexPartitions = (obj, { partition, offset }) => {
return assign(obj, { [partition]: offset })
}
const hasUnresolvedPartitions = () => unresolvedPartitions.some(t => t.partitions.length > 0)
let offsets = consumerOffsets
if (hasUnresolvedPartitions()) {
const topicOffsets = await this.cluster.fetchTopicsOffset(unresolvedPartitions)
offsets = initializeConsumerOffsets(consumerOffsets, topicOffsets)
}
offsets.forEach(({ topic, partitions }) => {
this.committedOffsets()[topic] = partitions.reduce(indexPartitions, {
...this.committedOffsets()[topic],
})
})
}
/**
* @private
* @param {import("../../../types").TopicPartition} topicPartition
*/
clearOffsets({ topic, partition }) {
delete this.committedOffsets()[topic][partition]
delete this.resolvedOffsets[topic][partition]
}
/**
* @private
*/
clearAllOffsets() {
const committedOffsets = this.committedOffsets()
for (const topic in committedOffsets) {
delete committedOffsets[topic]
}
for (const topic of this.topics) {
committedOffsets[topic] = {}
}
this.resolvedOffsets = indexTopics(this.topics)
}
committedOffsets() {
if (!this[PRIVATE.COMMITTED_OFFSETS]) {
this[PRIVATE.COMMITTED_OFFSETS] = this.groupId
? this.cluster.committedOffsets({ groupId: this.groupId })
: {}
}
return this[PRIVATE.COMMITTED_OFFSETS]
}
}

26
node_modules/kafkajs/src/consumer/offsetManager/initializeConsumerOffsets.js generated vendored Normal file
View file

@ -0,0 +1,26 @@
const isInvalidOffset = require('./isInvalidOffset')
const { keys, assign } = Object
const indexPartitions = (obj, { partition, offset }) => assign(obj, { [partition]: offset })
const indexTopics = (obj, { topic, partitions }) =>
assign(obj, { [topic]: partitions.reduce(indexPartitions, {}) })
module.exports = (consumerOffsets, topicOffsets) => {
const indexedConsumerOffsets = consumerOffsets.reduce(indexTopics, {})
const indexedTopicOffsets = topicOffsets.reduce(indexTopics, {})
return keys(indexedConsumerOffsets).map(topic => {
const partitions = indexedConsumerOffsets[topic]
return {
topic,
partitions: keys(partitions).map(partition => {
const offset = partitions[partition]
const resolvedOffset = isInvalidOffset(offset)
? indexedTopicOffsets[topic][partition]
: offset
return { partition: Number(partition), offset: resolvedOffset }
}),
}
})
}

3
node_modules/kafkajs/src/consumer/offsetManager/isInvalidOffset.js generated vendored Normal file
View file

@ -0,0 +1,3 @@
const Long = require('../../utils/long')
module.exports = offset => (!offset && offset !== 0) || Long.fromValue(offset).isNegative()

518
node_modules/kafkajs/src/consumer/runner.js generated vendored Normal file
View file

@ -0,0 +1,518 @@
const { EventEmitter } = require('events')
const Long = require('../utils/long')
const createRetry = require('../retry')
const { isKafkaJSError, isRebalancing } = require('../errors')
const {
events: { FETCH, FETCH_START, START_BATCH_PROCESS, END_BATCH_PROCESS, REBALANCING },
} = require('./instrumentationEvents')
const createFetchManager = require('./fetchManager')
const isSameOffset = (offsetA, offsetB) => Long.fromValue(offsetA).equals(Long.fromValue(offsetB))
const CONSUMING_START = 'consuming-start'
const CONSUMING_STOP = 'consuming-stop'
module.exports = class Runner extends EventEmitter {
/**
* @param {object} options
* @param {import("../../types").Logger} options.logger
* @param {import("./consumerGroup")} options.consumerGroup
* @param {import("../instrumentation/emitter")} options.instrumentationEmitter
* @param {boolean} [options.eachBatchAutoResolve=true]
* @param {number} options.concurrency
* @param {(payload: import("../../types").EachBatchPayload) => Promise<void>} [options.eachBatch]
* @param {(payload: import("../../types").EachMessagePayload) => Promise<void>} [options.eachMessage]
* @param {number} [options.heartbeatInterval]
* @param {(reason: Error) => void} options.onCrash
* @param {import("../../types").RetryOptions} [options.retry]
* @param {boolean} [options.autoCommit=true]
*/
constructor({
logger,
consumerGroup,
instrumentationEmitter,
eachBatchAutoResolve = true,
concurrency,
eachBatch,
eachMessage,
heartbeatInterval,
onCrash,
retry,
autoCommit = true,
}) {
super()
this.logger = logger.namespace('Runner')
this.consumerGroup = consumerGroup
this.instrumentationEmitter = instrumentationEmitter
this.eachBatchAutoResolve = eachBatchAutoResolve
this.eachBatch = eachBatch
this.eachMessage = eachMessage
this.heartbeatInterval = heartbeatInterval
this.retrier = createRetry(Object.assign({}, retry))
this.onCrash = onCrash
this.autoCommit = autoCommit
this.fetchManager = createFetchManager({
logger: this.logger,
getNodeIds: () => this.consumerGroup.getNodeIds(),
fetch: nodeId => this.fetch(nodeId),
handler: batch => this.handleBatch(batch),
concurrency,
})
this.running = false
this.consuming = false
}
get consuming() {
return this._consuming
}
set consuming(value) {
if (this._consuming !== value) {
this._consuming = value
this.emit(value ? CONSUMING_START : CONSUMING_STOP)
}
}
async start() {
if (this.running) {
return
}
try {
await this.consumerGroup.connect()
await this.consumerGroup.joinAndSync()
} catch (e) {
return this.onCrash(e)
}
this.running = true
this.scheduleFetchManager()
}
scheduleFetchManager() {
if (!this.running) {
this.consuming = false
this.logger.info('consumer not running, exiting', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
})
return
}
this.consuming = true
this.retrier(async (bail, retryCount, retryTime) => {
if (!this.running) {
return
}
try {
await this.fetchManager.start()
} catch (e) {
if (isRebalancing(e)) {
this.logger.warn('The group is rebalancing, re-joining', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
error: e.message,
})
this.instrumentationEmitter.emit(REBALANCING, {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
})
await this.consumerGroup.joinAndSync()
return
}
if (e.type === 'UNKNOWN_MEMBER_ID') {
this.logger.error('The coordinator is not aware of this member, re-joining the group', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
error: e.message,
})
this.consumerGroup.memberId = null
await this.consumerGroup.joinAndSync()
return
}
if (e.name === 'KafkaJSNotImplemented') {
return bail(e)
}
if (e.name === 'KafkaJSNoBrokerAvailableError') {
return bail(e)
}
this.logger.debug('Error while scheduling fetch manager, trying again...', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
error: e.message,
stack: e.stack,
retryCount,
retryTime,
})
throw e
}
})
.then(() => {
this.scheduleFetchManager()
})
.catch(e => {
this.onCrash(e)
this.consuming = false
this.running = false
})
}
async stop() {
if (!this.running) {
return
}
this.logger.debug('stop consumer group', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
})
this.running = false
try {
await this.fetchManager.stop()
await this.waitForConsumer()
await this.consumerGroup.leave()
} catch (e) {}
}
waitForConsumer() {
return new Promise(resolve => {
if (!this.consuming) {
return resolve()
}
this.logger.debug('waiting for consumer to finish...', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
})
this.once(CONSUMING_STOP, () => resolve())
})
}
async heartbeat() {
try {
await this.consumerGroup.heartbeat({ interval: this.heartbeatInterval })
} catch (e) {
if (isRebalancing(e)) {
await this.autoCommitOffsets()
}
throw e
}
}
async processEachMessage(batch) {
const { topic, partition } = batch
const pause = () => {
this.consumerGroup.pause([{ topic, partitions: [partition] }])
return () => this.consumerGroup.resume([{ topic, partitions: [partition] }])
}
for (const message of batch.messages) {
if (!this.running || this.consumerGroup.hasSeekOffset({ topic, partition })) {
break
}
try {
await this.eachMessage({
topic,
partition,
message,
heartbeat: () => this.heartbeat(),
pause,
})
} catch (e) {
if (!isKafkaJSError(e)) {
this.logger.error(`Error when calling eachMessage`, {
topic,
partition,
offset: message.offset,
stack: e.stack,
error: e,
})
}
// In case of errors, commit the previously consumed offsets unless autoCommit is disabled
await this.autoCommitOffsets()
throw e
}
this.consumerGroup.resolveOffset({ topic, partition, offset: message.offset })
await this.heartbeat()
await this.autoCommitOffsetsIfNecessary()
if (this.consumerGroup.isPaused(topic, partition)) {
break
}
}
}
async processEachBatch(batch) {
const { topic, partition } = batch
const lastFilteredMessage = batch.messages[batch.messages.length - 1]
const pause = () => {
this.consumerGroup.pause([{ topic, partitions: [partition] }])
return () => this.consumerGroup.resume([{ topic, partitions: [partition] }])
}
try {
await this.eachBatch({
batch,
resolveOffset: offset => {
/**
* The transactional producer generates a control record after committing the transaction.
* The control record is the last record on the RecordBatch, and it is filtered before it
* reaches the eachBatch callback. When disabling auto-resolve, the user-land code won't
* be able to resolve the control record offset, since it never reaches the callback,
* causing stuck consumers as the consumer will never move the offset marker.
*
* When the last offset of the batch is resolved, we should automatically resolve
* the control record offset as this entry doesn't have any meaning to the user-land code,
* and won't interfere with the stream processing.
*
* @see https://github.com/apache/kafka/blob/9aa660786e46c1efbf5605a6a69136a1dac6edb9/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L1499-L1505
*/
const offsetToResolve =
lastFilteredMessage && isSameOffset(offset, lastFilteredMessage.offset)
? batch.lastOffset()
: offset
this.consumerGroup.resolveOffset({ topic, partition, offset: offsetToResolve })
},
heartbeat: () => this.heartbeat(),
/**
* Pause consumption for the current topic-partition being processed
*/
pause,
/**
* Commit offsets if provided. Otherwise commit most recent resolved offsets
* if the autoCommit conditions are met.
*
* @param {import('../../types').OffsetsByTopicPartition} [offsets] Optional.
*/
commitOffsetsIfNecessary: async offsets => {
return offsets
? this.consumerGroup.commitOffsets(offsets)
: this.consumerGroup.commitOffsetsIfNecessary()
},
uncommittedOffsets: () => this.consumerGroup.uncommittedOffsets(),
isRunning: () => this.running,
isStale: () => this.consumerGroup.hasSeekOffset({ topic, partition }),
})
} catch (e) {
if (!isKafkaJSError(e)) {
this.logger.error(`Error when calling eachBatch`, {
topic,
partition,
offset: batch.firstOffset(),
stack: e.stack,
error: e,
})
}
// eachBatch has a special resolveOffset which can be used
// to keep track of the messages
await this.autoCommitOffsets()
throw e
}
// resolveOffset for the last offset can be disabled to allow the users of eachBatch to
// stop their consumers without resolving unprocessed offsets (issues/18)
if (this.eachBatchAutoResolve) {
this.consumerGroup.resolveOffset({ topic, partition, offset: batch.lastOffset() })
}
}
async fetch(nodeId) {
if (!this.running) {
this.logger.debug('consumer not running, exiting', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
})
return []
}
const startFetch = Date.now()
this.instrumentationEmitter.emit(FETCH_START, { nodeId })
const batches = await this.consumerGroup.fetch(nodeId)
this.instrumentationEmitter.emit(FETCH, {
/**
* PR #570 removed support for the number of batches in this instrumentation event;
* The new implementation uses an async generation to deliver the batches, which makes
* this number impossible to get. The number is set to 0 to keep the event backward
* compatible until we bump KafkaJS to version 2, following the end of node 8 LTS.
*
* @since 2019-11-29
*/
numberOfBatches: 0,
duration: Date.now() - startFetch,
nodeId,
})
if (batches.length === 0) {
await this.heartbeat()
}
return batches
}
async handleBatch(batch) {
if (!this.running) {
this.logger.debug('consumer not running, exiting', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
})
return
}
/** @param {import('./batch')} batch */
const onBatch = async batch => {
const startBatchProcess = Date.now()
const payload = {
topic: batch.topic,
partition: batch.partition,
highWatermark: batch.highWatermark,
offsetLag: batch.offsetLag(),
/**
* @since 2019-06-24 (>= 1.8.0)
*
* offsetLag returns the lag based on the latest offset in the batch, to
* keep the event backward compatible we just introduced "offsetLagLow"
* which calculates the lag based on the first offset in the batch
*/
offsetLagLow: batch.offsetLagLow(),
batchSize: batch.messages.length,
firstOffset: batch.firstOffset(),
lastOffset: batch.lastOffset(),
}
/**
* If the batch contained only control records or only aborted messages then we still
* need to resolve and auto-commit to ensure the consumer can move forward.
*
* We also need to emit batch instrumentation events to allow any listeners keeping
* track of offsets to know about the latest point of consumption.
*
* Added in #1256
*
* @see https://github.com/apache/kafka/blob/9aa660786e46c1efbf5605a6a69136a1dac6edb9/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L1499-L1505
*/
if (batch.isEmptyDueToFiltering()) {
this.instrumentationEmitter.emit(START_BATCH_PROCESS, payload)
this.consumerGroup.resolveOffset({
topic: batch.topic,
partition: batch.partition,
offset: batch.lastOffset(),
})
await this.autoCommitOffsetsIfNecessary()
this.instrumentationEmitter.emit(END_BATCH_PROCESS, {
...payload,
duration: Date.now() - startBatchProcess,
})
await this.heartbeat()
return
}
if (batch.isEmpty()) {
await this.heartbeat()
return
}
this.instrumentationEmitter.emit(START_BATCH_PROCESS, payload)
if (this.eachMessage) {
await this.processEachMessage(batch)
} else if (this.eachBatch) {
await this.processEachBatch(batch)
}
this.instrumentationEmitter.emit(END_BATCH_PROCESS, {
...payload,
duration: Date.now() - startBatchProcess,
})
await this.autoCommitOffsets()
await this.heartbeat()
}
await onBatch(batch)
}
autoCommitOffsets() {
if (this.autoCommit) {
return this.consumerGroup.commitOffsets()
}
}
autoCommitOffsetsIfNecessary() {
if (this.autoCommit) {
return this.consumerGroup.commitOffsetsIfNecessary()
}
}
commitOffsets(offsets) {
if (!this.running) {
this.logger.debug('consumer not running, exiting', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
offsets,
})
return
}
return this.retrier(async (bail, retryCount, retryTime) => {
try {
await this.consumerGroup.commitOffsets(offsets)
} catch (e) {
if (!this.running) {
this.logger.debug('consumer not running, exiting', {
error: e.message,
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
offsets,
})
return
}
if (e.name === 'KafkaJSNotImplemented') {
return bail(e)
}
this.logger.debug('Error while committing offsets, trying again...', {
groupId: this.consumerGroup.groupId,
memberId: this.consumerGroup.memberId,
error: e.message,
stack: e.stack,
retryCount,
retryTime,
offsets,
})
throw e
}
})
}
}

27
node_modules/kafkajs/src/consumer/seekOffsets.js generated vendored Normal file
View file

@ -0,0 +1,27 @@
module.exports = class SeekOffsets extends Map {
getKey(topic, partition) {
return JSON.stringify([topic, partition])
}
set(topic, partition, offset) {
const key = this.getKey(topic, partition)
super.set(key, offset)
}
has(topic, partition) {
const key = this.getKey(topic, partition)
return super.has(key)
}
pop(topic, partition) {
if (this.size === 0 || !this.has(topic, partition)) {
return
}
const key = this.getKey(topic, partition)
const offset = this.get(key)
this.delete(key)
return { topic, partition, offset }
}
}

123
node_modules/kafkajs/src/consumer/subscriptionState.js generated vendored Normal file
View file

@ -0,0 +1,123 @@
const createState = topic => ({
topic,
paused: new Set(),
pauseAll: false,
resumed: new Set(),
})
module.exports = class SubscriptionState {
constructor() {
this.assignedPartitionsByTopic = {}
this.subscriptionStatesByTopic = {}
}
/**
* Replace the current assignment with a new set of assignments
*
* @param {Array<TopicPartitions>} topicPartitions Example: [{ topic: 'topic-name', partitions: [1, 2] }]
*/
assign(topicPartitions = []) {
this.assignedPartitionsByTopic = topicPartitions.reduce(
(assigned, { topic, partitions = [] }) => {
return { ...assigned, [topic]: { topic, partitions } }
},
{}
)
}
/**
* @param {Array<TopicPartitions>} topicPartitions Example: [{ topic: 'topic-name', partitions: [1, 2] }]
*/
pause(topicPartitions = []) {
topicPartitions.forEach(({ topic, partitions }) => {
const state = this.subscriptionStatesByTopic[topic] || createState(topic)
if (typeof partitions === 'undefined') {
state.paused.clear()
state.resumed.clear()
state.pauseAll = true
} else if (Array.isArray(partitions)) {
partitions.forEach(partition => {
state.paused.add(partition)
state.resumed.delete(partition)
})
state.pauseAll = false
}
this.subscriptionStatesByTopic[topic] = state
})
}
/**
* @param {Array<TopicPartitions>} topicPartitions Example: [{ topic: 'topic-name', partitions: [1, 2] }]
*/
resume(topicPartitions = []) {
topicPartitions.forEach(({ topic, partitions }) => {
const state = this.subscriptionStatesByTopic[topic] || createState(topic)
if (typeof partitions === 'undefined') {
state.paused.clear()
state.resumed.clear()
state.pauseAll = false
} else if (Array.isArray(partitions)) {
partitions.forEach(partition => {
state.paused.delete(partition)
if (state.pauseAll) {
state.resumed.add(partition)
}
})
}
this.subscriptionStatesByTopic[topic] = state
})
}
/**
* @returns {Array<import("../../types").TopicPartitions>} topicPartitions
* Example: [{ topic: 'topic-name', partitions: [1, 2] }]
*/
assigned() {
return Object.values(this.assignedPartitionsByTopic).map(({ topic, partitions }) => ({
topic,
partitions: partitions.sort(),
}))
}
/**
* @returns {Array<import("../../types").TopicPartitions>} topicPartitions
* Example: [{ topic: 'topic-name', partitions: [1, 2] }]
*/
active() {
return Object.values(this.assignedPartitionsByTopic).map(({ topic, partitions }) => ({
topic,
partitions: partitions.filter(partition => !this.isPaused(topic, partition)).sort(),
}))
}
/**
* @returns {Array<import("../../types").TopicPartitions>} topicPartitions
* Example: [{ topic: 'topic-name', partitions: [1, 2] }]
*/
paused() {
return Object.values(this.assignedPartitionsByTopic)
.map(({ topic, partitions }) => ({
topic,
partitions: partitions.filter(partition => this.isPaused(topic, partition)).sort(),
}))
.filter(({ partitions }) => partitions.length !== 0)
}
isPaused(topic, partition) {
const state = this.subscriptionStatesByTopic[topic]
if (!state) {
return false
}
const partitionResumed = state.resumed.has(partition)
const partitionPaused = state.paused.has(partition)
return (state.pauseAll && !partitionResumed) || partitionPaused
}
}

40
node_modules/kafkajs/src/consumer/worker.js generated vendored Normal file
View file

@ -0,0 +1,40 @@
/**
* @typedef {(batch: T, metadata: { workerId: number }) => Promise<void>} Handler
* @template T
*
* @typedef {ReturnType<typeof createWorker>} Worker
*/
const sharedPromiseTo = require('../utils/sharedPromiseTo')
/**
* @param {{ handler: Handler<T>, workerId: number }} options
* @template T
*/
const createWorker = ({ handler, workerId }) => {
/**
* Takes batches from next() until it returns undefined.
*
* @param {{ next: () => { batch: T, resolve: () => void, reject: (e: Error) => void } | undefined }} param0
* @returns {Promise<void>}
*/
const run = sharedPromiseTo(async ({ next }) => {
while (true) {
const item = next()
if (!item) break
const { batch, resolve, reject } = item
try {
await handler(batch, { workerId })
resolve()
} catch (error) {
reject(error)
}
}
})
return { run }
}
module.exports = createWorker

40
node_modules/kafkajs/src/consumer/workerQueue.js generated vendored Normal file
View file

@ -0,0 +1,40 @@
/**
* @typedef {ReturnType<typeof createWorkerQueue>} WorkerQueue
*/
/**
* @param {object} options
* @param {import('./worker').Worker<T>[]} options.workers
* @template T
*/
const createWorkerQueue = ({ workers }) => {
/** @type {{ batch: T, resolve: (value?: any) => void, reject: (e: Error) => void}[]} */
const queue = []
const getWorkers = () => workers
/**
* Waits until workers have processed all batches in the queue.
*
* @param {...T} batches
* @returns {Promise<void>}
*/
const push = async (...batches) => {
const promises = batches.map(
batch => new Promise((resolve, reject) => queue.push({ batch, resolve, reject }))
)
workers.forEach(worker => worker.run({ next: () => queue.shift() }))
const results = await Promise.allSettled(promises)
const rejected = results.find(result => result.status === 'rejected')
if (rejected) {
// @ts-ignore
throw rejected.reason
}
}
return { push, getWorkers }
}
module.exports = createWorkerQueue

4
node_modules/kafkajs/src/env.js generated vendored Normal file
View file

@ -0,0 +1,4 @@
module.exports = () => ({
KAFKAJS_DEBUG_PROTOCOL_BUFFERS: process.env.KAFKAJS_DEBUG_PROTOCOL_BUFFERS,
KAFKAJS_DEBUG_EXTENDED_PROTOCOL_BUFFERS: process.env.KAFKAJS_DEBUG_EXTENDED_PROTOCOL_BUFFERS,
})

309
node_modules/kafkajs/src/errors.js generated vendored Normal file
View file

@ -0,0 +1,309 @@
const pkgJson = require('../package.json')
const { bugs } = pkgJson
class KafkaJSError extends Error {
constructor(e, { retriable = true, cause } = {}) {
super(e, { cause })
Error.captureStackTrace(this, this.constructor)
this.message = e.message || e
this.name = 'KafkaJSError'
this.retriable = retriable
this.helpUrl = e.helpUrl
this.cause = cause
}
}
class KafkaJSNonRetriableError extends KafkaJSError {
constructor(e, { cause } = {}) {
super(e, { retriable: false, cause })
this.name = 'KafkaJSNonRetriableError'
}
}
class KafkaJSProtocolError extends KafkaJSError {
constructor(e, { retriable = e.retriable } = {}) {
super(e, { retriable })
this.type = e.type
this.code = e.code
this.name = 'KafkaJSProtocolError'
}
}
class KafkaJSOffsetOutOfRange extends KafkaJSProtocolError {
constructor(e, { topic, partition }) {
super(e)
this.topic = topic
this.partition = partition
this.name = 'KafkaJSOffsetOutOfRange'
}
}
class KafkaJSMemberIdRequired extends KafkaJSProtocolError {
constructor(e, { memberId }) {
super(e)
this.memberId = memberId
this.name = 'KafkaJSMemberIdRequired'
}
}
class KafkaJSNumberOfRetriesExceeded extends KafkaJSNonRetriableError {
constructor(e, { retryCount, retryTime }) {
super(e, { cause: e })
this.stack = `${this.name}\n Caused by: ${e.stack}`
this.retryCount = retryCount
this.retryTime = retryTime
this.name = 'KafkaJSNumberOfRetriesExceeded'
}
}
class KafkaJSConnectionError extends KafkaJSError {
/**
* @param {string} e
* @param {object} options
* @param {string} [options.broker]
* @param {string} [options.code]
*/
constructor(e, { broker, code } = {}) {
super(e)
this.broker = broker
this.code = code
this.name = 'KafkaJSConnectionError'
}
}
class KafkaJSConnectionClosedError extends KafkaJSConnectionError {
constructor(e, { host, port } = {}) {
super(e, { broker: `${host}:${port}` })
this.host = host
this.port = port
this.name = 'KafkaJSConnectionClosedError'
}
}
class KafkaJSRequestTimeoutError extends KafkaJSError {
constructor(e, { broker, correlationId, createdAt, sentAt, pendingDuration } = {}) {
super(e)
this.broker = broker
this.correlationId = correlationId
this.createdAt = createdAt
this.sentAt = sentAt
this.pendingDuration = pendingDuration
this.name = 'KafkaJSRequestTimeoutError'
}
}
class KafkaJSMetadataNotLoaded extends KafkaJSError {
constructor() {
super(...arguments)
this.name = 'KafkaJSMetadataNotLoaded'
}
}
class KafkaJSTopicMetadataNotLoaded extends KafkaJSMetadataNotLoaded {
constructor(e, { topic } = {}) {
super(e)
this.topic = topic
this.name = 'KafkaJSTopicMetadataNotLoaded'
}
}
class KafkaJSStaleTopicMetadataAssignment extends KafkaJSError {
constructor(e, { topic, unknownPartitions } = {}) {
super(e)
this.topic = topic
this.unknownPartitions = unknownPartitions
this.name = 'KafkaJSStaleTopicMetadataAssignment'
}
}
class KafkaJSDeleteGroupsError extends KafkaJSError {
constructor(e, groups = []) {
super(e)
this.groups = groups
this.name = 'KafkaJSDeleteGroupsError'
}
}
class KafkaJSServerDoesNotSupportApiKey extends KafkaJSNonRetriableError {
constructor(e, { apiKey, apiName } = {}) {
super(e)
this.apiKey = apiKey
this.apiName = apiName
this.name = 'KafkaJSServerDoesNotSupportApiKey'
}
}
class KafkaJSBrokerNotFound extends KafkaJSError {
constructor() {
super(...arguments)
this.name = 'KafkaJSBrokerNotFound'
}
}
class KafkaJSPartialMessageError extends KafkaJSNonRetriableError {
constructor() {
super(...arguments)
this.name = 'KafkaJSPartialMessageError'
}
}
class KafkaJSSASLAuthenticationError extends KafkaJSNonRetriableError {
constructor() {
super(...arguments)
this.name = 'KafkaJSSASLAuthenticationError'
}
}
class KafkaJSGroupCoordinatorNotFound extends KafkaJSNonRetriableError {
constructor() {
super(...arguments)
this.name = 'KafkaJSGroupCoordinatorNotFound'
}
}
class KafkaJSNotImplemented extends KafkaJSNonRetriableError {
constructor() {
super(...arguments)
this.name = 'KafkaJSNotImplemented'
}
}
class KafkaJSTimeout extends KafkaJSNonRetriableError {
constructor() {
super(...arguments)
this.name = 'KafkaJSTimeout'
}
}
class KafkaJSLockTimeout extends KafkaJSTimeout {
constructor() {
super(...arguments)
this.name = 'KafkaJSLockTimeout'
}
}
class KafkaJSUnsupportedMagicByteInMessageSet extends KafkaJSNonRetriableError {
constructor() {
super(...arguments)
this.name = 'KafkaJSUnsupportedMagicByteInMessageSet'
}
}
class KafkaJSDeleteTopicRecordsError extends KafkaJSError {
constructor({ partitions }) {
/*
* This error is retriable if all the errors were retriable
*/
const retriable = partitions
.filter(({ error }) => error != null)
.every(({ error }) => error.retriable === true)
super('Error while deleting records', { retriable })
this.name = 'KafkaJSDeleteTopicRecordsError'
this.partitions = partitions
}
}
const issueUrl = bugs ? bugs.url : null
class KafkaJSInvariantViolation extends KafkaJSNonRetriableError {
constructor(e) {
const message = e.message || e
super(`Invariant violated: ${message}. This is likely a bug and should be reported.`)
this.name = 'KafkaJSInvariantViolation'
if (issueUrl !== null) {
const issueTitle = encodeURIComponent(`Invariant violation: ${message}`)
this.helpUrl = `${issueUrl}/new?assignees=&labels=bug&template=bug_report.md&title=${issueTitle}`
}
}
}
class KafkaJSInvalidVarIntError extends KafkaJSNonRetriableError {
constructor() {
super(...arguments)
this.name = 'KafkaJSNonRetriableError'
}
}
class KafkaJSInvalidLongError extends KafkaJSNonRetriableError {
constructor() {
super(...arguments)
this.name = 'KafkaJSNonRetriableError'
}
}
class KafkaJSCreateTopicError extends KafkaJSProtocolError {
constructor(e, topicName) {
super(e)
this.topic = topicName
this.name = 'KafkaJSCreateTopicError'
}
}
class KafkaJSAlterPartitionReassignmentsError extends KafkaJSProtocolError {
constructor(e, topicName, partition) {
super(e)
this.topic = topicName
this.partition = partition
this.name = 'KafkaJSAlterPartitionReassignmentsError'
}
}
class KafkaJSAggregateError extends Error {
constructor(message, errors) {
super(message)
this.errors = errors
this.name = 'KafkaJSAggregateError'
}
}
class KafkaJSFetcherRebalanceError extends Error {}
class KafkaJSNoBrokerAvailableError extends KafkaJSError {
constructor() {
super('No broker available')
this.name = 'KafkaJSNoBrokerAvailableError'
}
}
const isRebalancing = e =>
e.type === 'REBALANCE_IN_PROGRESS' ||
e.type === 'NOT_COORDINATOR_FOR_GROUP' ||
e.type === 'ILLEGAL_GENERATION'
const isKafkaJSError = e => e instanceof KafkaJSError
module.exports = {
KafkaJSError,
KafkaJSNonRetriableError,
KafkaJSPartialMessageError,
KafkaJSBrokerNotFound,
KafkaJSProtocolError,
KafkaJSConnectionError,
KafkaJSConnectionClosedError,
KafkaJSRequestTimeoutError,
KafkaJSSASLAuthenticationError,
KafkaJSNumberOfRetriesExceeded,
KafkaJSOffsetOutOfRange,
KafkaJSMemberIdRequired,
KafkaJSGroupCoordinatorNotFound,
KafkaJSNotImplemented,
KafkaJSMetadataNotLoaded,
KafkaJSTopicMetadataNotLoaded,
KafkaJSStaleTopicMetadataAssignment,
KafkaJSDeleteGroupsError,
KafkaJSTimeout,
KafkaJSLockTimeout,
KafkaJSServerDoesNotSupportApiKey,
KafkaJSUnsupportedMagicByteInMessageSet,
KafkaJSDeleteTopicRecordsError,
KafkaJSInvariantViolation,
KafkaJSInvalidVarIntError,
KafkaJSInvalidLongError,
KafkaJSCreateTopicError,
KafkaJSAggregateError,
KafkaJSFetcherRebalanceError,
KafkaJSNoBrokerAvailableError,
KafkaJSAlterPartitionReassignmentsError,
isRebalancing,
isKafkaJSError,
}

212
node_modules/kafkajs/src/index.js generated vendored Normal file
View file

@ -0,0 +1,212 @@
const {
createLogger,
LEVELS: { INFO },
} = require('./loggers')
const InstrumentationEventEmitter = require('./instrumentation/emitter')
const LoggerConsole = require('./loggers/console')
const Cluster = require('./cluster')
const createProducer = require('./producer')
const createConsumer = require('./consumer')
const createAdmin = require('./admin')
const ISOLATION_LEVEL = require('./protocol/isolationLevel')
const defaultSocketFactory = require('./network/socketFactory')
const once = require('./utils/once')
const websiteUrl = require('./utils/websiteUrl')
const PRIVATE = {
CREATE_CLUSTER: Symbol('private:Kafka:createCluster'),
CLUSTER_RETRY: Symbol('private:Kafka:clusterRetry'),
LOGGER: Symbol('private:Kafka:logger'),
OFFSETS: Symbol('private:Kafka:offsets'),
}
const DEFAULT_METADATA_MAX_AGE = 300000
const warnOfDefaultPartitioner = once(logger => {
if (process.env.KAFKAJS_NO_PARTITIONER_WARNING == null) {
logger.warn(
`KafkaJS v2.0.0 switched default partitioner. To retain the same partitioning behavior as in previous versions, create the producer with the option "createPartitioner: Partitioners.LegacyPartitioner". See the migration guide at ${websiteUrl(
'docs/migration-guide-v2.0.0',
'producer-new-default-partitioner'
)} for details. Silence this warning by setting the environment variable "KAFKAJS_NO_PARTITIONER_WARNING=1"`
)
}
})
module.exports = class Client {
/**
* @param {Object} options
* @param {Array<string>} options.brokers example: ['127.0.0.1:9092', '127.0.0.1:9094']
* @param {Object} options.ssl
* @param {Object} options.sasl
* @param {string} options.clientId
* @param {number} [options.connectionTimeout=1000] - in milliseconds
* @param {number} options.authenticationTimeout - in milliseconds
* @param {number} options.reauthenticationThreshold - in milliseconds
* @param {number} [options.requestTimeout=30000] - in milliseconds
* @param {boolean} [options.enforceRequestTimeout]
* @param {import("../types").RetryOptions} [options.retry]
* @param {import("../types").ISocketFactory} [options.socketFactory]
*/
constructor({
brokers,
ssl,
sasl,
clientId,
connectionTimeout = 1000,
authenticationTimeout,
reauthenticationThreshold,
requestTimeout,
enforceRequestTimeout = true,
retry,
socketFactory = defaultSocketFactory(),
logLevel = INFO,
logCreator = LoggerConsole,
}) {
this[PRIVATE.OFFSETS] = new Map()
this[PRIVATE.LOGGER] = createLogger({ level: logLevel, logCreator })
this[PRIVATE.CLUSTER_RETRY] = retry
this[PRIVATE.CREATE_CLUSTER] = ({
metadataMaxAge,
allowAutoTopicCreation = true,
maxInFlightRequests = null,
instrumentationEmitter = null,
isolationLevel,
}) =>
new Cluster({
logger: this[PRIVATE.LOGGER],
retry: this[PRIVATE.CLUSTER_RETRY],
offsets: this[PRIVATE.OFFSETS],
socketFactory,
brokers,
ssl,
sasl,
clientId,
connectionTimeout,
authenticationTimeout,
reauthenticationThreshold,
requestTimeout,
enforceRequestTimeout,
metadataMaxAge,
instrumentationEmitter,
allowAutoTopicCreation,
maxInFlightRequests,
isolationLevel,
})
}
/**
* @public
*/
producer({
createPartitioner,
retry,
metadataMaxAge = DEFAULT_METADATA_MAX_AGE,
allowAutoTopicCreation,
idempotent,
transactionalId,
transactionTimeout,
maxInFlightRequests,
} = {}) {
const instrumentationEmitter = new InstrumentationEventEmitter()
const cluster = this[PRIVATE.CREATE_CLUSTER]({
metadataMaxAge,
allowAutoTopicCreation,
maxInFlightRequests,
instrumentationEmitter,
})
if (createPartitioner == null) {
warnOfDefaultPartitioner(this[PRIVATE.LOGGER])
}
return createProducer({
retry: { ...this[PRIVATE.CLUSTER_RETRY], ...retry },
logger: this[PRIVATE.LOGGER],
cluster,
createPartitioner,
idempotent,
transactionalId,
transactionTimeout,
instrumentationEmitter,
})
}
/**
* @public
*/
consumer({
groupId,
partitionAssigners,
metadataMaxAge = DEFAULT_METADATA_MAX_AGE,
sessionTimeout,
rebalanceTimeout,
heartbeatInterval,
maxBytesPerPartition,
minBytes,
maxBytes,
maxWaitTimeInMs,
retry = { retries: 5 },
allowAutoTopicCreation,
maxInFlightRequests,
readUncommitted = false,
rackId = '',
} = {}) {
const isolationLevel = readUncommitted
? ISOLATION_LEVEL.READ_UNCOMMITTED
: ISOLATION_LEVEL.READ_COMMITTED
const instrumentationEmitter = new InstrumentationEventEmitter()
const cluster = this[PRIVATE.CREATE_CLUSTER]({
metadataMaxAge,
allowAutoTopicCreation,
maxInFlightRequests,
isolationLevel,
instrumentationEmitter,
})
return createConsumer({
retry: { ...this[PRIVATE.CLUSTER_RETRY], ...retry },
logger: this[PRIVATE.LOGGER],
cluster,
groupId,
partitionAssigners,
sessionTimeout,
rebalanceTimeout,
heartbeatInterval,
maxBytesPerPartition,
minBytes,
maxBytes,
maxWaitTimeInMs,
isolationLevel,
instrumentationEmitter,
rackId,
metadataMaxAge,
})
}
/**
* @public
*/
admin({ retry } = {}) {
const instrumentationEmitter = new InstrumentationEventEmitter()
const cluster = this[PRIVATE.CREATE_CLUSTER]({
allowAutoTopicCreation: false,
instrumentationEmitter,
})
return createAdmin({
retry: { ...this[PRIVATE.CLUSTER_RETRY], ...retry },
logger: this[PRIVATE.LOGGER],
instrumentationEmitter,
cluster,
})
}
/**
* @public
*/
logger() {
return this[PRIVATE.LOGGER]
}
}

34
node_modules/kafkajs/src/instrumentation/emitter.js generated vendored Normal file
View file

@ -0,0 +1,34 @@
const { EventEmitter } = require('events')
const InstrumentationEvent = require('./event')
const { KafkaJSError } = require('../errors')
module.exports = class InstrumentationEventEmitter {
constructor() {
this.emitter = new EventEmitter()
}
/**
* @param {string} eventName
* @param {Object} payload
*/
emit(eventName, payload) {
if (!eventName) {
throw new KafkaJSError('Invalid event name', { retriable: false })
}
if (this.emitter.listenerCount(eventName) > 0) {
const event = new InstrumentationEvent(eventName, payload)
this.emitter.emit(eventName, event)
}
}
/**
* @param {string} eventName
* @param {(...args: any[]) => void} listener
* @returns {import("../../types").RemoveInstrumentationEventListener<string>} removeListener
*/
addListener(eventName, listener) {
this.emitter.addListener(eventName, listener)
return () => this.emitter.removeListener(eventName, listener)
}
}

23
node_modules/kafkajs/src/instrumentation/event.js generated vendored Normal file
View file

@ -0,0 +1,23 @@
let id = 0
const nextId = () => {
if (id === Number.MAX_VALUE) {
id = 0
}
return id++
}
class InstrumentationEvent {
/**
* @param {String} type
* @param {Object} payload
*/
constructor(type, payload) {
this.id = nextId()
this.type = type
this.timestamp = Date.now()
this.payload = payload
}
}
module.exports = InstrumentationEvent

2
node_modules/kafkajs/src/instrumentation/eventType.js generated vendored Normal file
View file

@ -0,0 +1,2 @@
/** @type {<T1 extends string>(namespace: T1) => <T2 extends string>(type: T2) => `${T1}.${T2}`} */
module.exports = namespace => type => `${namespace}.${type}`

21
node_modules/kafkajs/src/loggers/console.js generated vendored Normal file
View file

@ -0,0 +1,21 @@
const { LEVELS: logLevel } = require('./index')
module.exports = () => ({ namespace, level, label, log }) => {
const prefix = namespace ? `[${namespace}] ` : ''
const message = JSON.stringify(
Object.assign({ level: label }, log, {
message: `${prefix}${log.message}`,
})
)
switch (level) {
case logLevel.INFO:
return console.info(message)
case logLevel.ERROR:
return console.error(message)
case logLevel.WARN:
return console.warn(message)
case logLevel.DEBUG:
return console.log(message)
}
}

68
node_modules/kafkajs/src/loggers/index.js generated vendored Normal file
View file

@ -0,0 +1,68 @@
const { assign } = Object
const LEVELS = {
NOTHING: 0,
ERROR: 1,
WARN: 2,
INFO: 4,
DEBUG: 5,
}
const createLevel = (label, level, currentLevel, namespace, logFunction) => (
message,
extra = {}
) => {
if (level > currentLevel()) return
logFunction({
namespace,
level,
label,
log: assign(
{
timestamp: new Date().toISOString(),
logger: 'kafkajs',
message,
},
extra
),
})
}
const evaluateLogLevel = logLevel => {
const envLogLevel = (process.env.KAFKAJS_LOG_LEVEL || '').toUpperCase()
return LEVELS[envLogLevel] == null ? logLevel : LEVELS[envLogLevel]
}
const createLogger = ({ level = LEVELS.INFO, logCreator } = {}) => {
let logLevel = evaluateLogLevel(level)
const logFunction = logCreator(logLevel)
const createNamespace = (namespace, logLevel = null) => {
const namespaceLogLevel = evaluateLogLevel(logLevel)
return createLogFunctions(namespace, namespaceLogLevel)
}
const createLogFunctions = (namespace, namespaceLogLevel = null) => {
const currentLogLevel = () => (namespaceLogLevel == null ? logLevel : namespaceLogLevel)
const logger = {
info: createLevel('INFO', LEVELS.INFO, currentLogLevel, namespace, logFunction),
error: createLevel('ERROR', LEVELS.ERROR, currentLogLevel, namespace, logFunction),
warn: createLevel('WARN', LEVELS.WARN, currentLogLevel, namespace, logFunction),
debug: createLevel('DEBUG', LEVELS.DEBUG, currentLogLevel, namespace, logFunction),
}
return assign(logger, {
namespace: createNamespace,
setLogLevel: newLevel => {
logLevel = newLevel
},
})
}
return createLogFunctions()
}
module.exports = {
LEVELS,
createLogger,
}

543
node_modules/kafkajs/src/network/connection.js generated vendored Normal file
View file

@ -0,0 +1,543 @@
const createSocket = require('./socket')
const createRequest = require('../protocol/request')
const Decoder = require('../protocol/decoder')
const { KafkaJSConnectionError, KafkaJSConnectionClosedError } = require('../errors')
const { INT_32_MAX_VALUE } = require('../constants')
const getEnv = require('../env')
const RequestQueue = require('./requestQueue')
const { CONNECTION_STATUS, CONNECTED_STATUS } = require('./connectionStatus')
const sharedPromiseTo = require('../utils/sharedPromiseTo')
const Long = require('../utils/long')
const SASLAuthenticator = require('../broker/saslAuthenticator')
const apiKeys = require('../protocol/requests/apiKeys')
const requestInfo = ({ apiName, apiKey, apiVersion }) =>
`${apiName}(key: ${apiKey}, version: ${apiVersion})`
/**
* @param request - request from protocol
* @returns {boolean}
*/
const isAuthenticatedRequest = request => {
return ![apiKeys.ApiVersions, apiKeys.SaslHandshake, apiKeys.SaslAuthenticate].includes(
request.apiKey
)
}
const PRIVATE = {
SHOULD_REAUTHENTICATE: Symbol('private:Connection:shouldReauthenticate'),
AUTHENTICATE: Symbol('private:Connection:authenticate'),
}
module.exports = class Connection {
/**
* @param {Object} options
* @param {string} options.host
* @param {number} options.port
* @param {import("../../types").Logger} options.logger
* @param {import("../../types").ISocketFactory} options.socketFactory
* @param {string} [options.clientId='kafkajs']
* @param {number} options.requestTimeout The maximum amount of time the client will wait for the response of a request,
* in milliseconds
* @param {string} [options.rack=null]
* @param {Object} [options.ssl=null] Options for the TLS Secure Context. It accepts all options,
* usually "cert", "key" and "ca". More information at
* https://nodejs.org/api/tls.html#tls_tls_createsecurecontext_options
* @param {Object} [options.sasl=null] Attributes used for SASL authentication. Options based on the
* key "mechanism". Connection is not actively using the SASL attributes
* but acting as a data object for this information
* @param {number} [options.reauthenticationThreshold=10000]
* @param {number} options.connectionTimeout The connection timeout, in milliseconds
* @param {boolean} [options.enforceRequestTimeout]
* @param {number} [options.maxInFlightRequests=null] The maximum number of unacknowledged requests on a connection before
* enqueuing
* @param {import("../instrumentation/emitter")} [options.instrumentationEmitter=null]
*/
constructor({
host,
port,
logger,
socketFactory,
requestTimeout,
reauthenticationThreshold = 10000,
rack = null,
ssl = null,
sasl = null,
clientId = 'kafkajs',
connectionTimeout,
enforceRequestTimeout = true,
maxInFlightRequests = null,
instrumentationEmitter = null,
}) {
this.host = host
this.port = port
this.rack = rack
this.clientId = clientId
this.broker = `${this.host}:${this.port}`
this.logger = logger.namespace('Connection')
this.socketFactory = socketFactory
this.ssl = ssl
this.sasl = sasl
this.requestTimeout = requestTimeout
this.connectionTimeout = connectionTimeout
this.reauthenticationThreshold = reauthenticationThreshold
this.bytesBuffered = 0
this.bytesNeeded = Decoder.int32Size()
this.chunks = []
this.connectionStatus = CONNECTION_STATUS.DISCONNECTED
this.correlationId = 0
this.requestQueue = new RequestQueue({
instrumentationEmitter,
maxInFlightRequests,
requestTimeout,
enforceRequestTimeout,
clientId,
broker: this.broker,
logger: logger.namespace('RequestQueue'),
isConnected: () => this.isConnected(),
})
this.versions = null
this.authHandlers = null
this.authExpectResponse = false
const log = level => (message, extra = {}) => {
const logFn = this.logger[level]
logFn(message, { broker: this.broker, clientId, ...extra })
}
this.logDebug = log('debug')
this.logError = log('error')
const env = getEnv()
this.shouldLogBuffers = env.KAFKAJS_DEBUG_PROTOCOL_BUFFERS === '1'
this.shouldLogFetchBuffer =
this.shouldLogBuffers && env.KAFKAJS_DEBUG_EXTENDED_PROTOCOL_BUFFERS === '1'
this.authenticatedAt = null
this.sessionLifetime = Long.ZERO
this.supportAuthenticationProtocol = null
/**
* @private
* @returns {Promise}
*/
this[PRIVATE.AUTHENTICATE] = sharedPromiseTo(async () => {
if (this.sasl && !this.isAuthenticated()) {
const authenticator = new SASLAuthenticator(
this,
this.logger,
this.versions,
this.supportAuthenticationProtocol
)
await authenticator.authenticate()
this.authenticatedAt = process.hrtime()
this.sessionLifetime = Long.fromValue(authenticator.sessionLifetime)
}
})
}
getSupportAuthenticationProtocol() {
return this.supportAuthenticationProtocol
}
setSupportAuthenticationProtocol(isSupported) {
this.supportAuthenticationProtocol = isSupported
}
setVersions(versions) {
this.versions = versions
}
isConnected() {
return CONNECTED_STATUS.includes(this.connectionStatus)
}
/**
* @public
* @returns {Promise}
*/
connect() {
return new Promise((resolve, reject) => {
if (this.isConnected()) {
return resolve(true)
}
this.authenticatedAt = null
let timeoutId
const onConnect = () => {
clearTimeout(timeoutId)
this.connectionStatus = CONNECTION_STATUS.CONNECTED
this.requestQueue.scheduleRequestTimeoutCheck()
resolve(true)
}
const onData = data => {
this.processData(data)
}
const onEnd = async () => {
clearTimeout(timeoutId)
const wasConnected = this.isConnected()
if (this.authHandlers) {
this.authHandlers.onError()
} else if (wasConnected) {
this.logDebug('Kafka server has closed connection')
this.rejectRequests(
new KafkaJSConnectionClosedError('Closed connection', {
host: this.host,
port: this.port,
})
)
}
await this.disconnect()
}
const onError = async e => {
clearTimeout(timeoutId)
const error = new KafkaJSConnectionError(`Connection error: ${e.message}`, {
broker: `${this.host}:${this.port}`,
code: e.code,
})
this.logError(error.message, { stack: e.stack })
this.rejectRequests(error)
await this.disconnect()
reject(error)
}
const onTimeout = async () => {
const error = new KafkaJSConnectionError('Connection timeout', {
broker: `${this.host}:${this.port}`,
})
this.logError(error.message)
this.rejectRequests(error)
await this.disconnect()
reject(error)
}
this.logDebug(`Connecting`, {
ssl: !!this.ssl,
sasl: !!this.sasl,
})
try {
timeoutId = setTimeout(onTimeout, this.connectionTimeout)
this.socket = createSocket({
socketFactory: this.socketFactory,
host: this.host,
port: this.port,
ssl: this.ssl,
onConnect,
onData,
onEnd,
onError,
onTimeout,
})
} catch (e) {
clearTimeout(timeoutId)
reject(
new KafkaJSConnectionError(`Failed to connect: ${e.message}`, {
broker: `${this.host}:${this.port}`,
})
)
}
})
}
/**
* @public
* @returns {Promise}
*/
async disconnect() {
this.authenticatedAt = null
this.connectionStatus = CONNECTION_STATUS.DISCONNECTING
this.logDebug('disconnecting...')
await this.requestQueue.waitForPendingRequests()
this.requestQueue.destroy()
if (this.socket) {
this.socket.end()
this.socket.unref()
}
this.connectionStatus = CONNECTION_STATUS.DISCONNECTED
this.logDebug('disconnected')
return true
}
/**
* @public
* @returns {boolean}
*/
isAuthenticated() {
return this.authenticatedAt != null && !this[PRIVATE.SHOULD_REAUTHENTICATE]()
}
/***
* @private
*/
[PRIVATE.SHOULD_REAUTHENTICATE]() {
if (this.sessionLifetime.equals(Long.ZERO)) {
return false
}
if (this.authenticatedAt == null) {
return true
}
const [secondsSince, remainingNanosSince] = process.hrtime(this.authenticatedAt)
const millisSince = Long.fromValue(secondsSince)
.multiply(1000)
.add(Long.fromValue(remainingNanosSince).divide(1000000))
const reauthenticateAt = millisSince.add(this.reauthenticationThreshold)
return reauthenticateAt.greaterThanOrEqual(this.sessionLifetime)
}
/** @public */
async authenticate() {
await this[PRIVATE.AUTHENTICATE]()
}
/**
* @public
* @returns {Promise}
*/
sendAuthRequest({ request, response }) {
this.authExpectResponse = !!response
/**
* TODO: rewrite removing the async promise executor
*/
/* eslint-disable no-async-promise-executor */
return new Promise(async (resolve, reject) => {
this.authHandlers = {
onSuccess: rawData => {
this.authHandlers = null
this.authExpectResponse = false
response
.decode(rawData)
.then(data => response.parse(data))
.then(resolve)
.catch(reject)
},
onError: () => {
this.authHandlers = null
this.authExpectResponse = false
reject(
new KafkaJSConnectionError('Connection closed by the server', {
broker: `${this.host}:${this.port}`,
})
)
},
}
try {
const requestPayload = await request.encode()
this.failIfNotConnected()
this.socket.write(requestPayload, 'binary')
} catch (e) {
reject(e)
}
})
}
/**
* @public
* @param {object} protocol
* @param {object} protocol.request It is defined by the protocol and consists of an object with "apiKey",
* "apiVersion", "apiName" and an "encode" function. The encode function
* must return an instance of Encoder
*
* @param {object} protocol.response It is defined by the protocol and consists of an object with two functions:
* "decode" and "parse"
*
* @param {number} [protocol.requestTimeout=null] Override for the default requestTimeout
* @param {boolean} [protocol.logResponseError=true] Whether to log errors
* @returns {Promise<data>} where data is the return of "response#parse"
*/
async send({ request, response, requestTimeout = null, logResponseError = true }) {
if (!this.isAuthenticated() && isAuthenticatedRequest(request)) {
await this[PRIVATE.AUTHENTICATE]()
}
this.failIfNotConnected()
const expectResponse = !request.expectResponse || request.expectResponse()
const sendRequest = async () => {
const { clientId } = this
const correlationId = this.nextCorrelationId()
const requestPayload = await createRequest({ request, correlationId, clientId })
const { apiKey, apiName, apiVersion } = request
this.logDebug(`Request ${requestInfo(request)}`, {
correlationId,
expectResponse,
size: Buffer.byteLength(requestPayload.buffer),
})
return new Promise((resolve, reject) => {
try {
this.failIfNotConnected()
const entry = { apiKey, apiName, apiVersion, correlationId, resolve, reject }
this.requestQueue.push({
entry,
expectResponse,
requestTimeout,
sendRequest: () => {
this.socket.write(requestPayload.buffer, 'binary')
},
})
} catch (e) {
reject(e)
}
})
}
const { correlationId, size, entry, payload } = await sendRequest()
if (!expectResponse) {
return
}
try {
const payloadDecoded = await response.decode(payload)
/**
* @see KIP-219
* If the response indicates that the client-side needs to throttle, do that.
*/
this.requestQueue.maybeThrottle(payloadDecoded.clientSideThrottleTime)
const data = await response.parse(payloadDecoded)
const isFetchApi = entry.apiName === 'Fetch'
this.logDebug(`Response ${requestInfo(entry)}`, {
correlationId,
size,
data: isFetchApi && !this.shouldLogFetchBuffer ? '[filtered]' : data,
})
return data
} catch (e) {
if (logResponseError) {
this.logError(`Response ${requestInfo(entry)}`, {
error: e.message,
correlationId,
size,
})
}
const isBuffer = Buffer.isBuffer(payload)
this.logDebug(`Response ${requestInfo(entry)}`, {
error: e.message,
correlationId,
payload:
isBuffer && !this.shouldLogBuffers ? { type: 'Buffer', data: '[filtered]' } : payload,
})
throw e
}
}
/**
* @private
*/
failIfNotConnected() {
if (!this.isConnected()) {
throw new KafkaJSConnectionError('Not connected', {
broker: `${this.host}:${this.port}`,
})
}
}
/**
* @private
*/
nextCorrelationId() {
if (this.correlationId >= INT_32_MAX_VALUE) {
this.correlationId = 0
}
return this.correlationId++
}
/**
* @private
*/
processData(rawData) {
if (this.authHandlers && !this.authExpectResponse) {
return this.authHandlers.onSuccess(rawData)
}
// Accumulate the new chunk
this.chunks.push(rawData)
this.bytesBuffered += Buffer.byteLength(rawData)
// Process data if there are enough bytes to read the expected response size,
// otherwise keep buffering
while (this.bytesNeeded <= this.bytesBuffered) {
const buffer = this.chunks.length > 1 ? Buffer.concat(this.chunks) : this.chunks[0]
const decoder = new Decoder(buffer)
const expectedResponseSize = decoder.readInt32()
// Return early if not enough bytes to read the full response
if (!decoder.canReadBytes(expectedResponseSize)) {
this.chunks = [buffer]
this.bytesBuffered = Buffer.byteLength(buffer)
this.bytesNeeded = Decoder.int32Size() + expectedResponseSize
return
}
const response = new Decoder(decoder.readBytes(expectedResponseSize))
// Reset the buffered chunks as the rest of the bytes
const remainderBuffer = decoder.readAll()
this.chunks = [remainderBuffer]
this.bytesBuffered = Buffer.byteLength(remainderBuffer)
this.bytesNeeded = Decoder.int32Size()
if (this.authHandlers) {
const rawResponseSize = Decoder.int32Size() + expectedResponseSize
const rawResponseBuffer = buffer.slice(0, rawResponseSize)
return this.authHandlers.onSuccess(rawResponseBuffer)
}
const correlationId = response.readInt32()
const payload = response.readAll()
this.requestQueue.fulfillRequest({
size: expectedResponseSize,
correlationId,
payload,
})
}
}
/**
* @private
*/
rejectRequests(error) {
this.requestQueue.rejectAll(error)
}
}

65
node_modules/kafkajs/src/network/connectionPool.js generated vendored Normal file
View file

@ -0,0 +1,65 @@
const apiKeys = require('../protocol/requests/apiKeys')
const Connection = require('./connection')
module.exports = class ConnectionPool {
/**
* @param {ConstructorParameters<typeof Connection>[0]} options
*/
constructor(options) {
this.logger = options.logger.namespace('ConnectionPool')
this.connectionTimeout = options.connectionTimeout
this.host = options.host
this.port = options.port
this.rack = options.rack
this.ssl = options.ssl
this.sasl = options.sasl
this.clientId = options.clientId
this.socketFactory = options.socketFactory
this.pool = new Array(2).fill().map(() => new Connection(options))
}
isConnected() {
return this.pool.some(c => c.isConnected())
}
isAuthenticated() {
return this.pool.some(c => c.isAuthenticated())
}
setSupportAuthenticationProtocol(isSupported) {
this.map(c => c.setSupportAuthenticationProtocol(isSupported))
}
setVersions(versions) {
this.map(c => c.setVersions(versions))
}
map(callback) {
return this.pool.map(c => callback(c))
}
async send(protocolRequest) {
const connection = await this.getConnectionByRequest(protocolRequest)
return connection.send(protocolRequest)
}
getConnectionByRequest({ request: { apiKey } }) {
const index = { [apiKeys.Fetch]: 1 }[apiKey] || 0
return this.getConnection(index)
}
async getConnection(index = 0) {
const connection = this.pool[index]
if (!connection.isConnected()) {
await connection.connect()
}
return connection
}
async destroy() {
await Promise.all(this.map(c => c.disconnect()))
}
}

12
node_modules/kafkajs/src/network/connectionStatus.js generated vendored Normal file
View file

@ -0,0 +1,12 @@
const CONNECTION_STATUS = {
CONNECTED: 'connected',
DISCONNECTING: 'disconnecting',
DISCONNECTED: 'disconnected',
}
const CONNECTED_STATUS = [CONNECTION_STATUS.CONNECTED, CONNECTION_STATUS.DISCONNECTING]
module.exports = {
CONNECTION_STATUS,
CONNECTED_STATUS,
}

8
node_modules/kafkajs/src/network/instrumentationEvents.js generated vendored Normal file
View file

@ -0,0 +1,8 @@
const InstrumentationEventType = require('../instrumentation/eventType')
const eventType = InstrumentationEventType('network')
module.exports = {
NETWORK_REQUEST: eventType('request'),
NETWORK_REQUEST_TIMEOUT: eventType('request_timeout'),
NETWORK_REQUEST_QUEUE_SIZE: eventType('request_queue_size'),
}

323
node_modules/kafkajs/src/network/requestQueue/index.js generated vendored Normal file
View file

@ -0,0 +1,323 @@
const { EventEmitter } = require('events')
const SocketRequest = require('./socketRequest')
const events = require('../instrumentationEvents')
const { KafkaJSInvariantViolation } = require('../../errors')
const PRIVATE = {
EMIT_QUEUE_SIZE_EVENT: Symbol('private:RequestQueue:emitQueueSizeEvent'),
EMIT_REQUEST_QUEUE_EMPTY: Symbol('private:RequestQueue:emitQueueEmpty'),
}
const REQUEST_QUEUE_EMPTY = 'requestQueueEmpty'
const CHECK_PENDING_REQUESTS_INTERVAL = 10
module.exports = class RequestQueue extends EventEmitter {
/**
* @param {Object} options
* @param {number} options.maxInFlightRequests
* @param {number} options.requestTimeout
* @param {boolean} options.enforceRequestTimeout
* @param {string} options.clientId
* @param {string} options.broker
* @param {import("../../../types").Logger} options.logger
* @param {import("../../instrumentation/emitter")} [options.instrumentationEmitter=null]
* @param {() => boolean} [options.isConnected]
*/
constructor({
instrumentationEmitter = null,
maxInFlightRequests,
requestTimeout,
enforceRequestTimeout,
clientId,
broker,
logger,
isConnected = () => true,
}) {
super()
this.instrumentationEmitter = instrumentationEmitter
this.maxInFlightRequests = maxInFlightRequests
this.requestTimeout = requestTimeout
this.enforceRequestTimeout = enforceRequestTimeout
this.clientId = clientId
this.broker = broker
this.logger = logger
this.isConnected = isConnected
this.inflight = new Map()
this.pending = []
/**
* Until when this request queue is throttled and shouldn't send requests
*
* The value represents the timestamp of the end of the throttling in ms-since-epoch. If the value
* is smaller than the current timestamp no throttling is active.
*
* @type {number}
*/
this.throttledUntil = -1
/**
* Timeout id if we have scheduled a check for pending requests due to client-side throttling
*
* @type {null|NodeJS.Timeout}
*/
this.throttleCheckTimeoutId = null
this[PRIVATE.EMIT_REQUEST_QUEUE_EMPTY] = () => {
if (this.pending.length === 0 && this.inflight.size === 0) {
this.emit(REQUEST_QUEUE_EMPTY)
}
}
this[PRIVATE.EMIT_QUEUE_SIZE_EVENT] = () => {
instrumentationEmitter &&
instrumentationEmitter.emit(events.NETWORK_REQUEST_QUEUE_SIZE, {
broker: this.broker,
clientId: this.clientId,
queueSize: this.pending.length,
})
this[PRIVATE.EMIT_REQUEST_QUEUE_EMPTY]()
}
}
/**
* @public
*/
scheduleRequestTimeoutCheck() {
if (this.enforceRequestTimeout) {
this.destroy()
this.requestTimeoutIntervalId = setInterval(() => {
this.inflight.forEach(request => {
if (Date.now() - request.sentAt > request.requestTimeout) {
request.timeoutRequest()
}
})
if (!this.isConnected()) {
this.destroy()
}
}, Math.min(this.requestTimeout, 100))
}
}
maybeThrottle(clientSideThrottleTime) {
if (clientSideThrottleTime !== null && clientSideThrottleTime > 0) {
this.logger.debug(`Client side throttling in effect for ${clientSideThrottleTime}ms`)
const minimumThrottledUntil = Date.now() + clientSideThrottleTime
this.throttledUntil = Math.max(minimumThrottledUntil, this.throttledUntil)
}
}
createSocketRequest(pushedRequest) {
const { correlationId } = pushedRequest.entry
const defaultRequestTimeout = this.requestTimeout
const customRequestTimeout = pushedRequest.requestTimeout
// Some protocol requests have custom request timeouts (e.g JoinGroup, Fetch, etc). The custom
// timeouts are influenced by user configurations, which can be lower than the default requestTimeout
const requestTimeout = Math.max(defaultRequestTimeout, customRequestTimeout || 0)
const socketRequest = new SocketRequest({
entry: pushedRequest.entry,
expectResponse: pushedRequest.expectResponse,
broker: this.broker,
clientId: this.clientId,
instrumentationEmitter: this.instrumentationEmitter,
requestTimeout,
send: () => {
if (this.inflight.has(correlationId)) {
throw new KafkaJSInvariantViolation('Correlation id already exists')
}
this.inflight.set(correlationId, socketRequest)
pushedRequest.sendRequest()
},
timeout: () => {
this.inflight.delete(correlationId)
this.checkPendingRequests()
// Try to emit REQUEST_QUEUE_EMPTY. Otherwise, waitForPendingRequests may stuck forever
this[PRIVATE.EMIT_REQUEST_QUEUE_EMPTY]()
},
})
return socketRequest
}
/**
* @typedef {Object} PushedRequest
* @property {import("./socketRequest").RequestEntry} entry
* @property {boolean} expectResponse
* @property {Function} sendRequest
* @property {number} [requestTimeout]
*
* @public
* @param {PushedRequest} pushedRequest
*/
push(pushedRequest) {
const { correlationId } = pushedRequest.entry
const socketRequest = this.createSocketRequest(pushedRequest)
if (this.canSendSocketRequestImmediately()) {
this.sendSocketRequest(socketRequest)
return
}
this.pending.push(socketRequest)
this.scheduleCheckPendingRequests()
this.logger.debug(`Request enqueued`, {
clientId: this.clientId,
broker: this.broker,
correlationId,
})
this[PRIVATE.EMIT_QUEUE_SIZE_EVENT]()
}
/**
* @param {SocketRequest} socketRequest
*/
sendSocketRequest(socketRequest) {
socketRequest.send()
if (!socketRequest.expectResponse) {
this.logger.debug(`Request does not expect a response, resolving immediately`, {
clientId: this.clientId,
broker: this.broker,
correlationId: socketRequest.correlationId,
})
this.inflight.delete(socketRequest.correlationId)
socketRequest.completed({ size: 0, payload: null })
}
}
/**
* @public
* @param {object} response
* @param {number} response.correlationId
* @param {Buffer} response.payload
* @param {number} response.size
*/
fulfillRequest({ correlationId, payload, size }) {
const socketRequest = this.inflight.get(correlationId)
this.inflight.delete(correlationId)
this.checkPendingRequests()
if (socketRequest) {
socketRequest.completed({ size, payload })
} else {
this.logger.warn(`Response without match`, {
clientId: this.clientId,
broker: this.broker,
correlationId,
})
}
this[PRIVATE.EMIT_REQUEST_QUEUE_EMPTY]()
}
/**
* @public
* @param {Error} error
*/
rejectAll(error) {
const requests = [...this.inflight.values(), ...this.pending]
for (const socketRequest of requests) {
socketRequest.rejected(error)
this.inflight.delete(socketRequest.correlationId)
}
this.pending = []
this.inflight.clear()
this[PRIVATE.EMIT_QUEUE_SIZE_EVENT]()
}
/**
* @public
*/
waitForPendingRequests() {
return new Promise(resolve => {
if (this.pending.length === 0 && this.inflight.size === 0) {
return resolve()
}
this.logger.debug('Waiting for pending requests', {
clientId: this.clientId,
broker: this.broker,
currentInflightRequests: this.inflight.size,
currentPendingQueueSize: this.pending.length,
})
this.once(REQUEST_QUEUE_EMPTY, () => resolve())
})
}
/**
* @public
*/
destroy() {
clearInterval(this.requestTimeoutIntervalId)
clearTimeout(this.throttleCheckTimeoutId)
this.throttleCheckTimeoutId = null
}
canSendSocketRequestImmediately() {
const shouldEnqueue =
(this.maxInFlightRequests != null && this.inflight.size >= this.maxInFlightRequests) ||
this.throttledUntil > Date.now()
return !shouldEnqueue
}
/**
* Check and process pending requests either now or in the future
*
* This function will send out as many pending requests as possible taking throttling and
* in-flight limits into account.
*/
checkPendingRequests() {
while (this.pending.length > 0 && this.canSendSocketRequestImmediately()) {
const pendingRequest = this.pending.shift() // first in first out
this.sendSocketRequest(pendingRequest)
this.logger.debug(`Consumed pending request`, {
clientId: this.clientId,
broker: this.broker,
correlationId: pendingRequest.correlationId,
pendingDuration: pendingRequest.pendingDuration,
currentPendingQueueSize: this.pending.length,
})
this[PRIVATE.EMIT_QUEUE_SIZE_EVENT]()
}
this.scheduleCheckPendingRequests()
}
/**
* Ensure that pending requests will be checked in the future
*
* If there is a client-side throttling in place this will ensure that we will check
* the pending request queue eventually.
*/
scheduleCheckPendingRequests() {
// If we're throttled: Schedule checkPendingRequests when the throttle
// should be resolved. If there is already something scheduled we assume that that
// will be fine, and potentially fix up a new timeout if needed at that time.
// Note that if we're merely "overloaded" by having too many inflight requests
// we will anyways check the queue when one of them gets fulfilled.
let scheduleAt = this.throttledUntil - Date.now()
if (!this.throttleCheckTimeoutId) {
if (this.pending.length > 0) {
scheduleAt = scheduleAt > 0 ? scheduleAt : CHECK_PENDING_REQUESTS_INTERVAL
}
this.throttleCheckTimeoutId = setTimeout(() => {
this.throttleCheckTimeoutId = null
this.checkPendingRequests()
}, scheduleAt)
}
}
}

168
node_modules/kafkajs/src/network/requestQueue/socketRequest.js generated vendored Normal file
View file

@ -0,0 +1,168 @@
const { KafkaJSRequestTimeoutError, KafkaJSNonRetriableError } = require('../../errors')
const events = require('../instrumentationEvents')
const PRIVATE = {
STATE: Symbol('private:SocketRequest:state'),
EMIT_EVENT: Symbol('private:SocketRequest:emitEvent'),
}
const REQUEST_STATE = {
PENDING: Symbol('PENDING'),
SENT: Symbol('SENT'),
COMPLETED: Symbol('COMPLETED'),
REJECTED: Symbol('REJECTED'),
}
/**
* SocketRequest abstracts the life cycle of a socket request, making it easier to track
* request durations and to have individual timeouts per request.
*
* @typedef {Object} SocketRequest
* @property {number} createdAt
* @property {number} sentAt
* @property {number} pendingDuration
* @property {number} duration
* @property {number} requestTimeout
* @property {string} broker
* @property {string} clientId
* @property {RequestEntry} entry
* @property {boolean} expectResponse
* @property {Function} send
* @property {Function} timeout
*
* @typedef {Object} RequestEntry
* @property {string} apiKey
* @property {string} apiName
* @property {number} apiVersion
* @property {number} correlationId
* @property {Function} resolve
* @property {Function} reject
*/
module.exports = class SocketRequest {
/**
* @param {Object} options
* @param {number} options.requestTimeout
* @param {string} options.broker - e.g: 127.0.0.1:9092
* @param {string} options.clientId
* @param {RequestEntry} options.entry
* @param {boolean} options.expectResponse
* @param {Function} options.send
* @param {() => void} options.timeout
* @param {import("../../instrumentation/emitter")} [options.instrumentationEmitter=null]
*/
constructor({
requestTimeout,
broker,
clientId,
entry,
expectResponse,
send,
timeout,
instrumentationEmitter = null,
}) {
this.createdAt = Date.now()
this.requestTimeout = requestTimeout
this.broker = broker
this.clientId = clientId
this.entry = entry
this.correlationId = entry.correlationId
this.expectResponse = expectResponse
this.sendRequest = send
this.timeoutHandler = timeout
this.sentAt = null
this.duration = null
this.pendingDuration = null
this[PRIVATE.STATE] = REQUEST_STATE.PENDING
this[PRIVATE.EMIT_EVENT] = (eventName, payload) =>
instrumentationEmitter && instrumentationEmitter.emit(eventName, payload)
}
send() {
this.throwIfInvalidState({
accepted: [REQUEST_STATE.PENDING],
next: REQUEST_STATE.SENT,
})
this.sendRequest()
this.sentAt = Date.now()
this.pendingDuration = this.sentAt - this.createdAt
this[PRIVATE.STATE] = REQUEST_STATE.SENT
}
timeoutRequest() {
const { apiName, apiKey, apiVersion } = this.entry
const requestInfo = `${apiName}(key: ${apiKey}, version: ${apiVersion})`
const eventData = {
broker: this.broker,
clientId: this.clientId,
correlationId: this.correlationId,
createdAt: this.createdAt,
sentAt: this.sentAt,
pendingDuration: this.pendingDuration,
}
this.timeoutHandler()
this.rejected(new KafkaJSRequestTimeoutError(`Request ${requestInfo} timed out`, eventData))
this[PRIVATE.EMIT_EVENT](events.NETWORK_REQUEST_TIMEOUT, {
...eventData,
apiName,
apiKey,
apiVersion,
})
}
completed({ size, payload }) {
this.throwIfInvalidState({
accepted: [REQUEST_STATE.SENT],
next: REQUEST_STATE.COMPLETED,
})
const { entry, correlationId, broker, clientId, createdAt, sentAt, pendingDuration } = this
this[PRIVATE.STATE] = REQUEST_STATE.COMPLETED
this.duration = Date.now() - this.sentAt
entry.resolve({ correlationId, entry, size, payload })
this[PRIVATE.EMIT_EVENT](events.NETWORK_REQUEST, {
broker,
clientId,
correlationId,
size,
createdAt,
sentAt,
pendingDuration,
duration: this.duration,
apiName: entry.apiName,
apiKey: entry.apiKey,
apiVersion: entry.apiVersion,
})
}
rejected(error) {
this.throwIfInvalidState({
accepted: [REQUEST_STATE.PENDING, REQUEST_STATE.SENT],
next: REQUEST_STATE.REJECTED,
})
this[PRIVATE.STATE] = REQUEST_STATE.REJECTED
this.duration = Date.now() - this.sentAt
this.entry.reject(error)
}
/**
* @private
*/
throwIfInvalidState({ accepted, next }) {
if (accepted.includes(this[PRIVATE.STATE])) {
return
}
const current = this[PRIVATE.STATE].toString()
throw new KafkaJSNonRetriableError(
`Invalid state, can't transition from ${current} to ${next.toString()}`
)
}
}

32
node_modules/kafkajs/src/network/socket.js generated vendored Normal file
View file

@ -0,0 +1,32 @@
/**
* @param {Object} options
* @param {import("../../types").ISocketFactory} options.socketFactory
* @param {string} options.host
* @param {number} options.port
* @param {Object} options.ssl
* @param {() => void} options.onConnect
* @param {(data: Buffer) => void} options.onData
* @param {() => void} options.onEnd
* @param {(err: Error) => void} options.onError
* @param {() => void} options.onTimeout
*/
module.exports = ({
socketFactory,
host,
port,
ssl,
onConnect,
onData,
onEnd,
onError,
onTimeout,
}) => {
const socket = socketFactory({ host, port, ssl, onConnect })
socket.on('data', onData)
socket.on('end', onEnd)
socket.on('error', onError)
socket.on('timeout', onTimeout)
return socket
}

22
node_modules/kafkajs/src/network/socketFactory.js generated vendored Normal file
View file

@ -0,0 +1,22 @@
const KEEP_ALIVE_DELAY = 60000 // in ms
/**
* @returns {import("../../types").ISocketFactory}
*/
module.exports = () => {
const net = require('net')
const tls = require('tls')
return ({ host, port, ssl, onConnect }) => {
const socket = ssl
? tls.connect(
Object.assign({ host, port }, !net.isIP(host) ? { servername: host } : {}, ssl),
onConnect
)
: net.connect({ host, port }, onConnect)
socket.setKeepAlive(true, KEEP_ALIVE_DELAY)
return socket
}
}

11
node_modules/kafkajs/src/producer/createTopicData.js generated vendored Normal file
View file

@ -0,0 +1,11 @@
module.exports = topicDataForBroker => {
return topicDataForBroker.map(
({ topic, partitions, messagesPerPartition, sequencePerPartition }) => ({
topic,
partitions: partitions.map(partition => ({
partition,
messages: messagesPerPartition[partition],
})),
})
)
}

464
node_modules/kafkajs/src/producer/eosManager/index.js generated vendored Normal file
View file

@ -0,0 +1,464 @@
const createRetry = require('../../retry')
const Lock = require('../../utils/lock')
const { KafkaJSNonRetriableError } = require('../../errors')
const COORDINATOR_TYPES = require('../../protocol/coordinatorTypes')
const createStateMachine = require('./transactionStateMachine')
const { INT_32_MAX_VALUE } = require('../../constants')
const assert = require('assert')
const STATES = require('./transactionStates')
const NO_PRODUCER_ID = -1
const SEQUENCE_START = 0
const INIT_PRODUCER_RETRIABLE_PROTOCOL_ERRORS = [
'NOT_COORDINATOR_FOR_GROUP',
'GROUP_COORDINATOR_NOT_AVAILABLE',
'GROUP_LOAD_IN_PROGRESS',
/**
* The producer might have crashed and never committed the transaction; retry the
* request so Kafka can abort the current transaction
* @see https://github.com/apache/kafka/blob/201da0542726472d954080d54bc585b111aaf86f/clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionManager.java#L1001-L1002
*/
'CONCURRENT_TRANSACTIONS',
]
const COMMIT_RETRIABLE_PROTOCOL_ERRORS = [
'UNKNOWN_TOPIC_OR_PARTITION',
'COORDINATOR_LOAD_IN_PROGRESS',
]
const COMMIT_STALE_COORDINATOR_PROTOCOL_ERRORS = ['COORDINATOR_NOT_AVAILABLE', 'NOT_COORDINATOR']
/**
* @typedef {Object} EosManager
*/
/**
* Manage behavior for an idempotent producer and transactions.
*
* @returns {EosManager}
*/
module.exports = ({
logger,
cluster,
transactionTimeout = 60000,
transactional,
transactionalId,
}) => {
if (transactional && !transactionalId) {
throw new KafkaJSNonRetriableError('Cannot manage transactions without a transactionalId')
}
const retrier = createRetry(cluster.retry)
/**
* Current producer ID
*/
let producerId = NO_PRODUCER_ID
/**
* Current producer epoch
*/
let producerEpoch = 0
/**
* Idempotent production requires that the producer track the sequence number of messages.
*
* Sequences are sent with every Record Batch and tracked per Topic-Partition
*/
let producerSequence = {}
/**
* Idempotent production requires a mutex lock per broker to serialize requests with sequence number handling
*/
let brokerMutexLocks = {}
/**
* Topic partitions already participating in the transaction
*/
let transactionTopicPartitions = {}
/**
* Offsets have been added to the transaction
*/
let hasOffsetsAddedToTransaction = false
const stateMachine = createStateMachine({ logger })
stateMachine.on('transition', ({ to }) => {
if (to === STATES.READY) {
transactionTopicPartitions = {}
hasOffsetsAddedToTransaction = false
}
})
const findTransactionCoordinator = () => {
return cluster.findGroupCoordinator({
groupId: transactionalId,
coordinatorType: COORDINATOR_TYPES.TRANSACTION,
})
}
const transactionalGuard = () => {
if (!transactional) {
throw new KafkaJSNonRetriableError('Method unavailable if non-transactional')
}
}
/**
* A transaction is ongoing when offsets or partitions added to it
*
* @returns {boolean}
*/
const isOngoing = () => {
return (
hasOffsetsAddedToTransaction ||
Object.entries(transactionTopicPartitions).some(([, partitions]) => {
return Object.entries(partitions).some(
([, isPartitionAddedToTransaction]) => isPartitionAddedToTransaction
)
})
)
}
const eosManager = stateMachine.createGuarded(
{
/**
* Get the current producer id
* @returns {number}
*/
getProducerId() {
return producerId
},
/**
* Get the current producer epoch
* @returns {number}
*/
getProducerEpoch() {
return producerEpoch
},
getTransactionalId() {
return transactionalId
},
/**
* Initialize the idempotent producer by making an `InitProducerId` request.
* Overwrites any existing state in this transaction manager
*/
async initProducerId() {
return retrier(async (bail, retryCount, retryTime) => {
try {
await cluster.refreshMetadataIfNecessary()
// If non-transactional we can request the PID from any broker
const broker = await (transactional
? findTransactionCoordinator()
: cluster.findControllerBroker())
const result = await broker.initProducerId({
transactionalId: transactional ? transactionalId : undefined,
transactionTimeout,
})
stateMachine.transitionTo(STATES.READY)
producerId = result.producerId
producerEpoch = result.producerEpoch
producerSequence = {}
brokerMutexLocks = {}
logger.debug('Initialized producer id & epoch', { producerId, producerEpoch })
} catch (e) {
if (INIT_PRODUCER_RETRIABLE_PROTOCOL_ERRORS.includes(e.type)) {
if (e.type === 'CONCURRENT_TRANSACTIONS') {
logger.debug('There is an ongoing transaction on this transactionId, retrying', {
error: e.message,
stack: e.stack,
transactionalId,
retryCount,
retryTime,
})
}
throw e
}
bail(e)
}
})
},
/**
* Get the current sequence for a given Topic-Partition. Defaults to 0.
*
* @param {string} topic
* @param {string} partition
* @returns {number}
*/
getSequence(topic, partition) {
if (!eosManager.isInitialized()) {
return SEQUENCE_START
}
producerSequence[topic] = producerSequence[topic] || {}
producerSequence[topic][partition] = producerSequence[topic][partition] || SEQUENCE_START
return producerSequence[topic][partition]
},
/**
* Update the sequence for a given Topic-Partition.
*
* Do nothing if not yet initialized (not idempotent)
* @param {string} topic
* @param {string} partition
* @param {number} increment
*/
updateSequence(topic, partition, increment) {
if (!eosManager.isInitialized()) {
return
}
const previous = eosManager.getSequence(topic, partition)
let sequence = previous + increment
// Sequence is defined as Int32 in the Record Batch,
// so theoretically should need to rotate here
if (sequence >= INT_32_MAX_VALUE) {
logger.debug(
`Sequence for ${topic} ${partition} exceeds max value (${sequence}). Rotating to 0.`
)
sequence = 0
}
producerSequence[topic][partition] = sequence
},
/**
* Begin a transaction
*/
beginTransaction() {
transactionalGuard()
stateMachine.transitionTo(STATES.TRANSACTING)
},
/**
* Add partitions to a transaction if they are not already marked as participating.
*
* Should be called prior to sending any messages during a transaction
* @param {TopicData[]} topicData
*
* @typedef {Object} TopicData
* @property {string} topic
* @property {object[]} partitions
* @property {number} partitions[].partition
*/
async addPartitionsToTransaction(topicData) {
transactionalGuard()
const newTopicPartitions = {}
topicData.forEach(({ topic, partitions }) => {
transactionTopicPartitions[topic] = transactionTopicPartitions[topic] || {}
partitions.forEach(({ partition }) => {
if (!transactionTopicPartitions[topic][partition]) {
newTopicPartitions[topic] = newTopicPartitions[topic] || []
newTopicPartitions[topic].push(partition)
}
})
})
const topics = Object.keys(newTopicPartitions).map(topic => ({
topic,
partitions: newTopicPartitions[topic],
}))
if (topics.length) {
const broker = await findTransactionCoordinator()
await broker.addPartitionsToTxn({ transactionalId, producerId, producerEpoch, topics })
}
topics.forEach(({ topic, partitions }) => {
partitions.forEach(partition => {
transactionTopicPartitions[topic][partition] = true
})
})
},
/**
* Commit the ongoing transaction
*/
async commit() {
transactionalGuard()
stateMachine.transitionTo(STATES.COMMITTING)
if (!isOngoing()) {
logger.debug('No partitions or offsets registered, not sending EndTxn')
stateMachine.transitionTo(STATES.READY)
return
}
const broker = await findTransactionCoordinator()
await broker.endTxn({
producerId,
producerEpoch,
transactionalId,
transactionResult: true,
})
stateMachine.transitionTo(STATES.READY)
},
/**
* Abort the ongoing transaction
*/
async abort() {
transactionalGuard()
stateMachine.transitionTo(STATES.ABORTING)
if (!isOngoing()) {
logger.debug('No partitions or offsets registered, not sending EndTxn')
stateMachine.transitionTo(STATES.READY)
return
}
const broker = await findTransactionCoordinator()
await broker.endTxn({
producerId,
producerEpoch,
transactionalId,
transactionResult: false,
})
stateMachine.transitionTo(STATES.READY)
},
/**
* Whether the producer id has already been initialized
*/
isInitialized() {
return producerId !== NO_PRODUCER_ID
},
isTransactional() {
return transactional
},
isInTransaction() {
return stateMachine.state() === STATES.TRANSACTING
},
async acquireBrokerLock(broker) {
if (this.isInitialized()) {
brokerMutexLocks[broker.nodeId] =
brokerMutexLocks[broker.nodeId] || new Lock({ timeout: 0xffff })
await brokerMutexLocks[broker.nodeId].acquire()
}
},
releaseBrokerLock(broker) {
if (this.isInitialized()) brokerMutexLocks[broker.nodeId].release()
},
/**
* Mark the provided offsets as participating in the transaction for the given consumer group.
*
* This allows us to commit an offset as consumed only if the transaction passes.
* @param {string} consumerGroupId The unique group identifier
* @param {OffsetCommitTopic[]} topics The unique group identifier
* @returns {Promise}
*
* @typedef {Object} OffsetCommitTopic
* @property {string} topic
* @property {OffsetCommitTopicPartition[]} partitions
*
* @typedef {Object} OffsetCommitTopicPartition
* @property {number} partition
* @property {number} offset
*/
async sendOffsets({ consumerGroupId, topics }) {
assert(consumerGroupId, 'Missing consumerGroupId')
assert(topics, 'Missing offset topics')
const transactionCoordinator = await findTransactionCoordinator()
// Do we need to add offsets if we've already done so for this consumer group?
await transactionCoordinator.addOffsetsToTxn({
transactionalId,
producerId,
producerEpoch,
groupId: consumerGroupId,
})
hasOffsetsAddedToTransaction = true
let groupCoordinator = await cluster.findGroupCoordinator({
groupId: consumerGroupId,
coordinatorType: COORDINATOR_TYPES.GROUP,
})
return retrier(async (bail, retryCount, retryTime) => {
try {
await groupCoordinator.txnOffsetCommit({
transactionalId,
producerId,
producerEpoch,
groupId: consumerGroupId,
topics,
})
} catch (e) {
if (COMMIT_RETRIABLE_PROTOCOL_ERRORS.includes(e.type)) {
logger.debug('Group coordinator is not ready yet, retrying', {
error: e.message,
stack: e.stack,
transactionalId,
retryCount,
retryTime,
})
throw e
}
if (
COMMIT_STALE_COORDINATOR_PROTOCOL_ERRORS.includes(e.type) ||
e.code === 'ECONNREFUSED'
) {
logger.debug(
'Invalid group coordinator, finding new group coordinator and retrying',
{
error: e.message,
stack: e.stack,
transactionalId,
retryCount,
retryTime,
}
)
groupCoordinator = await cluster.findGroupCoordinator({
groupId: consumerGroupId,
coordinatorType: COORDINATOR_TYPES.GROUP,
})
throw e
}
bail(e)
}
})
},
},
/**
* Transaction state guards
*/
{
initProducerId: { legalStates: [STATES.UNINITIALIZED, STATES.READY] },
beginTransaction: { legalStates: [STATES.READY], async: false },
addPartitionsToTransaction: { legalStates: [STATES.TRANSACTING] },
sendOffsets: { legalStates: [STATES.TRANSACTING] },
commit: { legalStates: [STATES.TRANSACTING] },
abort: { legalStates: [STATES.TRANSACTING] },
}
)
return eosManager
}

79
node_modules/kafkajs/src/producer/eosManager/transactionStateMachine.js generated vendored Normal file
View file

@ -0,0 +1,79 @@
const { EventEmitter } = require('events')
const { KafkaJSNonRetriableError } = require('../../errors')
const STATES = require('./transactionStates')
const VALID_STATE_TRANSITIONS = {
[STATES.UNINITIALIZED]: [STATES.READY],
[STATES.READY]: [STATES.READY, STATES.TRANSACTING],
[STATES.TRANSACTING]: [STATES.COMMITTING, STATES.ABORTING],
[STATES.COMMITTING]: [STATES.READY],
[STATES.ABORTING]: [STATES.READY],
}
module.exports = ({ logger, initialState = STATES.UNINITIALIZED }) => {
let currentState = initialState
const guard = (object, method, { legalStates, async: isAsync = true }) => {
if (!object[method]) {
throw new KafkaJSNonRetriableError(`Cannot add guard on missing method "${method}"`)
}
return (...args) => {
const fn = object[method]
if (!legalStates.includes(currentState)) {
const error = new KafkaJSNonRetriableError(
`Transaction state exception: Cannot call "${method}" in state "${currentState}"`
)
if (isAsync) {
return Promise.reject(error)
} else {
throw error
}
}
return fn.apply(object, args)
}
}
const stateMachine = Object.assign(new EventEmitter(), {
/**
* Create a clone of "object" where we ensure state machine is in correct state
* prior to calling any of the configured methods
* @param {Object} object The object whose methods we will guard
* @param {Object} methodStateMapping Keys are method names on "object"
* @param {string[]} methodStateMapping.legalStates Legal states for this method
* @param {boolean=true} methodStateMapping.async Whether this method is async (throw vs reject)
*/
createGuarded(object, methodStateMapping) {
const guardedMethods = Object.keys(methodStateMapping).reduce((guards, method) => {
guards[method] = guard(object, method, methodStateMapping[method])
return guards
}, {})
return { ...object, ...guardedMethods }
},
/**
* Transition safely to a new state
*/
transitionTo(state) {
logger.debug(`Transaction state transition ${currentState} --> ${state}`)
if (!VALID_STATE_TRANSITIONS[currentState].includes(state)) {
throw new KafkaJSNonRetriableError(
`Transaction state exception: Invalid transition ${currentState} --> ${state}`
)
}
stateMachine.emit('transition', { to: state, from: currentState })
currentState = state
},
state() {
return currentState
},
})
return stateMachine
}

7
node_modules/kafkajs/src/producer/eosManager/transactionStates.js generated vendored Normal file
View file

@ -0,0 +1,7 @@
module.exports = {
UNINITIALIZED: 'UNINITIALIZED',
READY: 'READY',
TRANSACTING: 'TRANSACTING',
COMMITTING: 'COMMITTING',
ABORTING: 'ABORTING',
}

11
node_modules/kafkajs/src/producer/groupMessagesPerPartition.js generated vendored Normal file
View file

@ -0,0 +1,11 @@
module.exports = ({ topic, partitionMetadata, messages, partitioner }) => {
if (partitionMetadata.length === 0) {
return {}
}
return messages.reduce((result, message) => {
const partition = partitioner({ topic, partitionMetadata, message })
const current = result[partition] || []
return Object.assign(result, { [partition]: [...current, message] })
}, {})
}

246
node_modules/kafkajs/src/producer/index.js generated vendored Normal file
View file

@ -0,0 +1,246 @@
const createRetry = require('../retry')
const { CONNECTION_STATUS } = require('../network/connectionStatus')
const { DefaultPartitioner } = require('./partitioners/')
const InstrumentationEventEmitter = require('../instrumentation/emitter')
const createEosManager = require('./eosManager')
const createMessageProducer = require('./messageProducer')
const { events, wrap: wrapEvent, unwrap: unwrapEvent } = require('./instrumentationEvents')
const { KafkaJSNonRetriableError } = require('../errors')
const { values, keys } = Object
const eventNames = values(events)
const eventKeys = keys(events)
.map(key => `producer.events.${key}`)
.join(', ')
const { CONNECT, DISCONNECT } = events
/**
*
* @param {Object} params
* @param {import('../../types').Cluster} params.cluster
* @param {import('../../types').Logger} params.logger
* @param {import('../../types').ICustomPartitioner} [params.createPartitioner]
* @param {import('../../types').RetryOptions} [params.retry]
* @param {boolean} [params.idempotent]
* @param {string} [params.transactionalId]
* @param {number} [params.transactionTimeout]
* @param {InstrumentationEventEmitter} [params.instrumentationEmitter]
*
* @returns {import('../../types').Producer}
*/
module.exports = ({
cluster,
logger: rootLogger,
createPartitioner = DefaultPartitioner,
retry,
idempotent = false,
transactionalId,
transactionTimeout,
instrumentationEmitter: rootInstrumentationEmitter,
}) => {
let connectionStatus = CONNECTION_STATUS.DISCONNECTED
retry = retry || { retries: idempotent ? Number.MAX_SAFE_INTEGER : 5 }
if (idempotent && retry.retries < 1) {
throw new KafkaJSNonRetriableError(
'Idempotent producer must allow retries to protect against transient errors'
)
}
const logger = rootLogger.namespace('Producer')
if (idempotent && retry.retries < Number.MAX_SAFE_INTEGER) {
logger.warn('Limiting retries for the idempotent producer may invalidate EoS guarantees')
}
const partitioner = createPartitioner()
const retrier = createRetry(Object.assign({}, cluster.retry, retry))
const instrumentationEmitter = rootInstrumentationEmitter || new InstrumentationEventEmitter()
const idempotentEosManager = createEosManager({
logger,
cluster,
transactionTimeout,
transactional: false,
transactionalId,
})
const { send, sendBatch } = createMessageProducer({
logger,
cluster,
partitioner,
eosManager: idempotentEosManager,
idempotent,
retrier,
getConnectionStatus: () => connectionStatus,
})
let transactionalEosManager
/** @type {import("../../types").Producer["on"]} */
const on = (eventName, listener) => {
if (!eventNames.includes(eventName)) {
throw new KafkaJSNonRetriableError(`Event name should be one of ${eventKeys}`)
}
return instrumentationEmitter.addListener(unwrapEvent(eventName), event => {
event.type = wrapEvent(event.type)
Promise.resolve(listener(event)).catch(e => {
logger.error(`Failed to execute listener: ${e.message}`, {
eventName,
stack: e.stack,
})
})
})
}
/**
* Begin a transaction. The returned object contains methods to send messages
* to the transaction and end the transaction by committing or aborting.
*
* Only messages sent on the transaction object will participate in the transaction.
*
* Calling any of the transactional methods after the transaction has ended
* will raise an exception (use `isActive` to ascertain if ended).
* @returns {Promise<Transaction>}
*
* @typedef {Object} Transaction
* @property {Function} send Identical to the producer "send" method
* @property {Function} sendBatch Identical to the producer "sendBatch" method
* @property {Function} abort Abort the transaction
* @property {Function} commit Commit the transaction
* @property {Function} isActive Whether the transaction is active
*/
const transaction = async () => {
if (!transactionalId) {
throw new KafkaJSNonRetriableError('Must provide transactional id for transactional producer')
}
let transactionDidEnd = false
transactionalEosManager =
transactionalEosManager ||
createEosManager({
logger,
cluster,
transactionTimeout,
transactional: true,
transactionalId,
})
if (transactionalEosManager.isInTransaction()) {
throw new KafkaJSNonRetriableError(
'There is already an ongoing transaction for this producer. Please end the transaction before beginning another.'
)
}
// We only initialize the producer id once
if (!transactionalEosManager.isInitialized()) {
await transactionalEosManager.initProducerId()
}
transactionalEosManager.beginTransaction()
const { send: sendTxn, sendBatch: sendBatchTxn } = createMessageProducer({
logger,
cluster,
partitioner,
retrier,
eosManager: transactionalEosManager,
idempotent: true,
getConnectionStatus: () => connectionStatus,
})
const isActive = () => transactionalEosManager.isInTransaction() && !transactionDidEnd
const transactionGuard = fn => (...args) => {
if (!isActive()) {
return Promise.reject(
new KafkaJSNonRetriableError('Cannot continue to use transaction once ended')
)
}
return fn(...args)
}
return {
sendBatch: transactionGuard(sendBatchTxn),
send: transactionGuard(sendTxn),
/**
* Abort the ongoing transaction.
*
* @throws {KafkaJSNonRetriableError} If transaction has ended
*/
abort: transactionGuard(async () => {
await transactionalEosManager.abort()
transactionDidEnd = true
}),
/**
* Commit the ongoing transaction.
*
* @throws {KafkaJSNonRetriableError} If transaction has ended
*/
commit: transactionGuard(async () => {
await transactionalEosManager.commit()
transactionDidEnd = true
}),
/**
* Sends a list of specified offsets to the consumer group coordinator, and also marks those offsets as part of the current transaction.
*
* @throws {KafkaJSNonRetriableError} If transaction has ended
*/
sendOffsets: transactionGuard(async ({ consumerGroupId, topics }) => {
await transactionalEosManager.sendOffsets({ consumerGroupId, topics })
for (const topicOffsets of topics) {
const { topic, partitions } = topicOffsets
for (const { partition, offset } of partitions) {
cluster.markOffsetAsCommitted({
groupId: consumerGroupId,
topic,
partition,
offset,
})
}
}
}),
isActive,
}
}
/**
* @returns {Object} logger
*/
const getLogger = () => logger
return {
/**
* @returns {Promise}
*/
connect: async () => {
await cluster.connect()
connectionStatus = CONNECTION_STATUS.CONNECTED
instrumentationEmitter.emit(CONNECT)
if (idempotent && !idempotentEosManager.isInitialized()) {
await idempotentEosManager.initProducerId()
}
},
/**
* @return {Promise}
*/
disconnect: async () => {
connectionStatus = CONNECTION_STATUS.DISCONNECTING
await cluster.disconnect()
connectionStatus = CONNECTION_STATUS.DISCONNECTED
instrumentationEmitter.emit(DISCONNECT)
},
isIdempotent: () => {
return idempotent
},
events,
on,
send,
sendBatch,
transaction,
logger: getLogger,
}
}

28
node_modules/kafkajs/src/producer/instrumentationEvents.js generated vendored Normal file
View file

@ -0,0 +1,28 @@
const swapObject = require('../utils/swapObject')
const networkEvents = require('../network/instrumentationEvents')
const InstrumentationEventType = require('../instrumentation/eventType')
const producerType = InstrumentationEventType('producer')
const events = {
CONNECT: producerType('connect'),
DISCONNECT: producerType('disconnect'),
REQUEST: producerType(networkEvents.NETWORK_REQUEST),
REQUEST_TIMEOUT: producerType(networkEvents.NETWORK_REQUEST_TIMEOUT),
REQUEST_QUEUE_SIZE: producerType(networkEvents.NETWORK_REQUEST_QUEUE_SIZE),
}
const wrappedEvents = {
[events.REQUEST]: networkEvents.NETWORK_REQUEST,
[events.REQUEST_TIMEOUT]: networkEvents.NETWORK_REQUEST_TIMEOUT,
[events.REQUEST_QUEUE_SIZE]: networkEvents.NETWORK_REQUEST_QUEUE_SIZE,
}
const reversedWrappedEvents = swapObject(wrappedEvents)
const unwrap = eventName => wrappedEvents[eventName] || eventName
const wrap = eventName => reversedWrappedEvents[eventName] || eventName
module.exports = {
events,
wrap,
unwrap,
}

132
node_modules/kafkajs/src/producer/messageProducer.js generated vendored Normal file
View file

@ -0,0 +1,132 @@
const createSendMessages = require('./sendMessages')
const { KafkaJSError, KafkaJSNonRetriableError } = require('../errors')
const { CONNECTION_STATUS } = require('../network/connectionStatus')
module.exports = ({
logger,
cluster,
partitioner,
eosManager,
idempotent,
retrier,
getConnectionStatus,
}) => {
const sendMessages = createSendMessages({
logger,
cluster,
retrier,
partitioner,
eosManager,
})
const validateConnectionStatus = () => {
const connectionStatus = getConnectionStatus()
switch (connectionStatus) {
case CONNECTION_STATUS.DISCONNECTING:
throw new KafkaJSNonRetriableError(
`The producer is disconnecting; therefore, it can't safely accept messages anymore`
)
case CONNECTION_STATUS.DISCONNECTED:
throw new KafkaJSError('The producer is disconnected')
}
}
/**
* @typedef {Object} TopicMessages
* @property {string} topic
* @property {Array} messages An array of objects with "key" and "value", example:
* [{ key: 'my-key', value: 'my-value'}]
*
* @typedef {Object} SendBatchRequest
* @property {Array<TopicMessages>} topicMessages
* @property {number} [acks=-1] Control the number of required acks.
* -1 = all replicas must acknowledge
* 0 = no acknowledgments
* 1 = only waits for the leader to acknowledge
*
* @property {number} [timeout=30000] The time to await a response in ms
* @property {Compression.Types} [compression=Compression.Types.None] Compression codec
*
* @param {SendBatchRequest}
* @returns {Promise}
*/
const sendBatch = async ({ acks = -1, timeout, compression, topicMessages = [] }) => {
if (topicMessages.some(({ topic }) => !topic)) {
throw new KafkaJSNonRetriableError(`Invalid topic`)
}
if (idempotent && acks !== -1) {
throw new KafkaJSNonRetriableError(
`Not requiring ack for all messages invalidates the idempotent producer's EoS guarantees`
)
}
for (const { topic, messages } of topicMessages) {
if (!messages) {
throw new KafkaJSNonRetriableError(
`Invalid messages array [${messages}] for topic "${topic}"`
)
}
const messageWithoutValue = messages.find(message => message.value === undefined)
if (messageWithoutValue) {
throw new KafkaJSNonRetriableError(
`Invalid message without value for topic "${topic}": ${JSON.stringify(
messageWithoutValue
)}`
)
}
}
validateConnectionStatus()
const mergedTopicMessages = topicMessages.reduce((merged, { topic, messages }) => {
const index = merged.findIndex(({ topic: mergedTopic }) => topic === mergedTopic)
if (index === -1) {
merged.push({ topic, messages })
} else {
merged[index].messages = [...merged[index].messages, ...messages]
}
return merged
}, [])
return await sendMessages({
acks,
timeout,
compression,
topicMessages: mergedTopicMessages,
})
}
/**
* @param {ProduceRequest} ProduceRequest
* @returns {Promise}
*
* @typedef {Object} ProduceRequest
* @property {string} topic
* @property {Array} messages An array of objects with "key" and "value", example:
* [{ key: 'my-key', value: 'my-value'}]
* @property {number} [acks=-1] Control the number of required acks.
* -1 = all replicas must acknowledge
* 0 = no acknowledgments
* 1 = only waits for the leader to acknowledge
* @property {number} [timeout=30000] The time to await a response in ms
* @property {Compression.Types} [compression=Compression.Types.None] Compression codec
*/
const send = async ({ acks, timeout, compression, topic, messages }) => {
const topicMessage = { topic, messages }
return sendBatch({
acks,
timeout,
compression,
topicMessages: [topicMessage],
})
}
return {
send,
sendBatch,
}
}

4
node_modules/kafkajs/src/producer/partitioners/default/index.js generated vendored Normal file
View file

@ -0,0 +1,4 @@
const murmur2 = require('./murmur2')
const createDefaultPartitioner = require('../legacy/partitioner')
module.exports = createDefaultPartitioner(murmur2)

53
node_modules/kafkajs/src/producer/partitioners/default/murmur2.js generated vendored Normal file
View file

@ -0,0 +1,53 @@
/* eslint-disable */
const Long = require('../../../utils/long')
// Based on the kafka client 0.10.2 murmur2 implementation
// https://github.com/apache/kafka/blob/0.10.2/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L364
const SEED = Long.fromValue(0x9747b28c)
// 'm' and 'r' are mixing constants generated offline.
// They're not really 'magic', they just happen to work well.
const M = Long.fromValue(0x5bd1e995)
const R = Long.fromValue(24)
module.exports = key => {
const data = Buffer.isBuffer(key) ? key : Buffer.from(String(key))
const length = data.length
// Initialize the hash to a random value
let h = Long.fromValue(SEED.xor(length))
let length4 = Math.floor(length / 4)
for (let i = 0; i < length4; i++) {
const i4 = i * 4
let k =
(data[i4 + 0] & 0xff) +
((data[i4 + 1] & 0xff) << 8) +
((data[i4 + 2] & 0xff) << 16) +
((data[i4 + 3] & 0xff) << 24)
k = Long.fromValue(k)
k = k.multiply(M)
k = k.xor(k.toInt() >>> R)
k = Long.fromValue(k).multiply(M)
h = h.multiply(M)
h = h.xor(k)
}
// Handle the last few bytes of the input array
switch (length % 4) {
case 3:
h = h.xor((data[(length & ~3) + 2] & 0xff) << 16)
case 2:
h = h.xor((data[(length & ~3) + 1] & 0xff) << 8)
case 1:
h = h.xor(data[length & ~3] & 0xff)
h = h.multiply(M)
}
h = h.xor(h.toInt() >>> 13)
h = h.multiply(M)
h = h.xor(h.toInt() >>> 15)
return h.toInt()
}

14
node_modules/kafkajs/src/producer/partitioners/index.js generated vendored Normal file
View file

@ -0,0 +1,14 @@
const DefaultPartitioner = require('./default')
const LegacyPartitioner = require('./legacy')
module.exports = {
DefaultPartitioner,
LegacyPartitioner,
/**
* @deprecated Use DefaultPartitioner instead
*
* The JavaCompatiblePartitioner was renamed DefaultPartitioner
* and made to be the default in 2.0.0.
*/
JavaCompatiblePartitioner: DefaultPartitioner,
}

4
node_modules/kafkajs/src/producer/partitioners/legacy/index.js generated vendored Normal file
View file

@ -0,0 +1,4 @@
const murmur2 = require('./murmur2')
const createLegacyPartitioner = require('./partitioner')
module.exports = createLegacyPartitioner(murmur2)

51
node_modules/kafkajs/src/producer/partitioners/legacy/murmur2.js generated vendored Normal file
View file

@ -0,0 +1,51 @@
/* eslint-disable */
// Based on the kafka client 0.10.2 murmur2 implementation
// https://github.com/apache/kafka/blob/0.10.2/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L364
const SEED = 0x9747b28c
// 'm' and 'r' are mixing constants generated offline.
// They're not really 'magic', they just happen to work well.
const M = 0x5bd1e995
const R = 24
module.exports = key => {
const data = Buffer.isBuffer(key) ? key : Buffer.from(String(key))
const length = data.length
// Initialize the hash to a random value
let h = SEED ^ length
let length4 = length / 4
for (let i = 0; i < length4; i++) {
const i4 = i * 4
let k =
(data[i4 + 0] & 0xff) +
((data[i4 + 1] & 0xff) << 8) +
((data[i4 + 2] & 0xff) << 16) +
((data[i4 + 3] & 0xff) << 24)
k *= M
k ^= k >>> R
k *= M
h *= M
h ^= k
}
// Handle the last few bytes of the input array
switch (length % 4) {
case 3:
h ^= (data[(length & ~3) + 2] & 0xff) << 16
case 2:
h ^= (data[(length & ~3) + 1] & 0xff) << 8
case 1:
h ^= data[length & ~3] & 0xff
h *= M
}
h ^= h >>> 13
h *= M
h ^= h >>> 15
return h
}

47
node_modules/kafkajs/src/producer/partitioners/legacy/partitioner.js generated vendored Normal file
View file

@ -0,0 +1,47 @@
const randomBytes = require('./randomBytes')
// Based on the java client 0.10.2
// https://github.com/apache/kafka/blob/0.10.2/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java
/**
* A cheap way to deterministically convert a number to a positive value. When the input is
* positive, the original value is returned. When the input number is negative, the returned
* positive value is the original value bit AND against 0x7fffffff which is not its absolutely
* value.
*/
const toPositive = x => x & 0x7fffffff
/**
* The default partitioning strategy:
* - If a partition is specified in the message, use it
* - If no partition is specified but a key is present choose a partition based on a hash of the key
* - If no partition or key is present choose a partition in a round-robin fashion
*/
module.exports = murmur2 => () => {
const counters = {}
return ({ topic, partitionMetadata, message }) => {
if (!(topic in counters)) {
counters[topic] = randomBytes(32).readUInt32BE(0)
}
const numPartitions = partitionMetadata.length
const availablePartitions = partitionMetadata.filter(p => p.leader >= 0)
const numAvailablePartitions = availablePartitions.length
if (message.partition !== null && message.partition !== undefined) {
return message.partition
}
if (message.key !== null && message.key !== undefined) {
return toPositive(murmur2(message.key)) % numPartitions
}
if (numAvailablePartitions > 0) {
const i = toPositive(++counters[topic]) % numAvailablePartitions
return availablePartitions[i].partitionId
}
// no partitions are available, give a non-available partition
return toPositive(++counters[topic]) % numPartitions
}
}

31
node_modules/kafkajs/src/producer/partitioners/legacy/randomBytes.js generated vendored Normal file
View file

@ -0,0 +1,31 @@
const { KafkaJSNonRetriableError } = require('../../../errors')
const toNodeCompatible = crypto => ({
randomBytes: size => crypto.getRandomValues(Buffer.allocUnsafe(size)),
})
let cryptoImplementation = null
if (global && global.crypto) {
cryptoImplementation =
global.crypto.randomBytes === undefined ? toNodeCompatible(global.crypto) : global.crypto
} else if (global && global.msCrypto) {
cryptoImplementation = toNodeCompatible(global.msCrypto)
} else if (global && !global.crypto) {
cryptoImplementation = require('crypto')
}
const MAX_BYTES = 65536
module.exports = size => {
if (size > MAX_BYTES) {
throw new KafkaJSNonRetriableError(
`Byte length (${size}) exceeds the max number of bytes of entropy available (${MAX_BYTES})`
)
}
if (!cryptoImplementation) {
throw new KafkaJSNonRetriableError('No available crypto implementation')
}
return cryptoImplementation.randomBytes(size)
}

4
node_modules/kafkajs/src/producer/responseSerializer.js generated vendored Normal file
View file

@ -0,0 +1,4 @@
module.exports = ({ topics }) =>
topics.flatMap(({ topicName, partitions }) =>
partitions.map(partition => ({ topicName, ...partition }))
)

170
node_modules/kafkajs/src/producer/sendMessages.js generated vendored Normal file
View file

@ -0,0 +1,170 @@
const { KafkaJSMetadataNotLoaded } = require('../errors')
const { staleMetadata } = require('../protocol/error')
const groupMessagesPerPartition = require('./groupMessagesPerPartition')
const createTopicData = require('./createTopicData')
const responseSerializer = require('./responseSerializer')
const { keys } = Object
/**
* @param {Object} options
* @param {import("../../types").Logger} options.logger
* @param {import("../../types").Cluster} options.cluster
* @param {ReturnType<import("../../types").ICustomPartitioner>} options.partitioner
* @param {import("./eosManager").EosManager} options.eosManager
* @param {import("../retry").Retrier} options.retrier
*/
module.exports = ({ logger, cluster, partitioner, eosManager, retrier }) => {
return async ({ acks, timeout, compression, topicMessages }) => {
/** @type {Map<import("../../types").Broker, any[]>} */
const responsePerBroker = new Map()
/** @param {Map<import("../../types").Broker, any[]>} responsePerBroker */
const createProducerRequests = async responsePerBroker => {
const topicMetadata = new Map()
await cluster.refreshMetadataIfNecessary()
for (const { topic, messages } of topicMessages) {
const partitionMetadata = cluster.findTopicPartitionMetadata(topic)
if (partitionMetadata.length === 0) {
logger.debug('Producing to topic without metadata', {
topic,
targetTopics: Array.from(cluster.targetTopics),
})
throw new KafkaJSMetadataNotLoaded('Producing to topic without metadata')
}
const messagesPerPartition = groupMessagesPerPartition({
topic,
partitionMetadata,
messages,
partitioner,
})
const partitions = keys(messagesPerPartition)
const partitionsPerLeader = cluster.findLeaderForPartitions(topic, partitions)
const leaders = keys(partitionsPerLeader)
topicMetadata.set(topic, {
partitionsPerLeader,
messagesPerPartition,
})
for (const nodeId of leaders) {
const broker = await cluster.findBroker({ nodeId })
if (!responsePerBroker.has(broker)) {
responsePerBroker.set(broker, null)
}
}
}
const brokers = Array.from(responsePerBroker.keys())
const brokersWithoutResponse = brokers.filter(broker => !responsePerBroker.get(broker))
return brokersWithoutResponse.map(async broker => {
const entries = Array.from(topicMetadata.entries())
const topicDataForBroker = entries
.filter(([_, { partitionsPerLeader }]) => !!partitionsPerLeader[broker.nodeId])
.map(([topic, { partitionsPerLeader, messagesPerPartition, sequencePerPartition }]) => ({
topic,
partitions: partitionsPerLeader[broker.nodeId],
messagesPerPartition,
}))
const topicData = createTopicData(topicDataForBroker)
await eosManager.acquireBrokerLock(broker)
try {
if (eosManager.isTransactional()) {
await eosManager.addPartitionsToTransaction(topicData)
}
topicData.forEach(({ topic, partitions }) => {
partitions.forEach(entry => {
entry['firstSequence'] = eosManager.getSequence(topic, entry.partition)
eosManager.updateSequence(topic, entry.partition, entry.messages.length)
})
})
let response
try {
response = await broker.produce({
transactionalId: eosManager.isTransactional()
? eosManager.getTransactionalId()
: undefined,
producerId: eosManager.getProducerId(),
producerEpoch: eosManager.getProducerEpoch(),
acks,
timeout,
compression,
topicData,
})
} catch (e) {
topicData.forEach(({ topic, partitions }) => {
partitions.forEach(entry => {
eosManager.updateSequence(topic, entry.partition, -entry.messages.length)
})
})
throw e
}
const expectResponse = acks !== 0
const formattedResponse = expectResponse ? responseSerializer(response) : []
responsePerBroker.set(broker, formattedResponse)
} catch (e) {
responsePerBroker.delete(broker)
throw e
} finally {
await eosManager.releaseBrokerLock(broker)
}
})
}
return retrier(async (bail, retryCount, retryTime) => {
const topics = topicMessages.map(({ topic }) => topic)
await cluster.addMultipleTargetTopics(topics)
try {
const requests = await createProducerRequests(responsePerBroker)
await Promise.all(requests)
return Array.from(responsePerBroker.values()).flat()
} catch (e) {
if (e.name === 'KafkaJSConnectionClosedError') {
cluster.removeBroker({ host: e.host, port: e.port })
}
if (!cluster.isConnected()) {
logger.debug(`Cluster has disconnected, reconnecting: ${e.message}`, {
retryCount,
retryTime,
})
await cluster.connect()
await cluster.refreshMetadata()
throw e
}
// This is necessary in case the metadata is stale and the number of partitions
// for this topic has increased in the meantime
if (
staleMetadata(e) ||
e.name === 'KafkaJSMetadataNotLoaded' ||
e.name === 'KafkaJSConnectionError' ||
e.name === 'KafkaJSConnectionClosedError' ||
(e.name === 'KafkaJSProtocolError' && e.retriable)
) {
logger.error(`Failed to send messages: ${e.message}`, { retryCount, retryTime })
await cluster.refreshMetadata()
throw e
}
logger.error(`${e.message}`, { retryCount, retryTime })
if (e.retriable) throw e
bail(e)
}
})
}
}

65
node_modules/kafkajs/src/protocol/aclOperationTypes.js generated vendored Normal file
View file

@ -0,0 +1,65 @@
// From:
// https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/acl/AclOperation.java#L44
/**
* @typedef {number} ACLOperationTypes
*
* Enum for ACL Operations Types
* @readonly
* @enum {ACLOperationTypes}
*/
module.exports = {
/**
* Represents any AclOperation which this client cannot understand, perhaps because this
* client is too old.
*/
UNKNOWN: 0,
/**
* In a filter, matches any AclOperation.
*/
ANY: 1,
/**
* ALL operation.
*/
ALL: 2,
/**
* READ operation.
*/
READ: 3,
/**
* WRITE operation.
*/
WRITE: 4,
/**
* CREATE operation.
*/
CREATE: 5,
/**
* DELETE operation.
*/
DELETE: 6,
/**
* ALTER operation.
*/
ALTER: 7,
/**
* DESCRIBE operation.
*/
DESCRIBE: 8,
/**
* CLUSTER_ACTION operation.
*/
CLUSTER_ACTION: 9,
/**
* DESCRIBE_CONFIGS operation.
*/
DESCRIBE_CONFIGS: 10,
/**
* ALTER_CONFIGS operation.
*/
ALTER_CONFIGS: 11,
/**
* IDEMPOTENT_WRITE operation.
*/
IDEMPOTENT_WRITE: 12,
}

29
node_modules/kafkajs/src/protocol/aclPermissionTypes.js generated vendored Normal file
View file

@ -0,0 +1,29 @@
// From:
// https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/acl/AclPermissionType.java/#L31
/**
* @typedef {number} ACLPermissionTypes
*
* Enum for Permission Types
* @readonly
* @enum {ACLPermissionTypes}
*/
module.exports = {
/**
* Represents any AclPermissionType which this client cannot understand,
* perhaps because this client is too old.
*/
UNKNOWN: 0,
/**
* In a filter, matches any AclPermissionType.
*/
ANY: 1,
/**
* Disallows access.
*/
DENY: 2,
/**
* Grants access.
*/
ALLOW: 3,
}

42
node_modules/kafkajs/src/protocol/aclResourceTypes.js generated vendored Normal file
View file

@ -0,0 +1,42 @@
/**
* @see https://github.com/apache/kafka/blob/a15387f34d142684859c2a57fcbef25edcdce25a/clients/src/main/java/org/apache/kafka/common/resource/ResourceType.java#L25-L31
* @typedef {number} ACLResourceTypes
*
* Enum for ACL Resource Types
* @readonly
* @enum {ACLResourceTypes}
*/
module.exports = {
/**
* Represents any ResourceType which this client cannot understand,
* perhaps because this client is too old.
*/
UNKNOWN: 0,
/**
* In a filter, matches any ResourceType.
*/
ANY: 1,
/**
* A Kafka topic.
* @see http://kafka.apache.org/documentation/#topicconfigs
*/
TOPIC: 2,
/**
* A consumer group.
* @see http://kafka.apache.org/documentation/#consumerconfigs
*/
GROUP: 3,
/**
* The cluster as a whole.
*/
CLUSTER: 4,
/**
* A transactional ID.
*/
TRANSACTIONAL_ID: 5,
/**
* A token ID.
*/
DELEGATION_TOKEN: 6,
}

9
node_modules/kafkajs/src/protocol/configResourceTypes.js generated vendored Normal file
View file

@ -0,0 +1,9 @@
/**
* @see https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigResource.java
*/
module.exports = {
UNKNOWN: 0,
TOPIC: 2,
BROKER: 4,
BROKER_LOGGER: 8,
}

12
node_modules/kafkajs/src/protocol/configSource.js generated vendored Normal file
View file

@ -0,0 +1,12 @@
/**
* @see https://github.com/apache/kafka/blob/1f240ce1793cab09e1c4823e17436d2b030df2bc/clients/src/main/java/org/apache/kafka/common/requests/DescribeConfigsResponse.java#L115-L122
*/
module.exports = {
UNKNOWN: 0,
TOPIC_CONFIG: 1,
DYNAMIC_BROKER_CONFIG: 2,
DYNAMIC_DEFAULT_BROKER_CONFIG: 3,
STATIC_BROKER_CONFIG: 4,
DEFAULT_CONFIG: 5,
DYNAMIC_BROKER_LOGGER_CONFIG: 6,
}

12
node_modules/kafkajs/src/protocol/coordinatorTypes.js generated vendored Normal file
View file

@ -0,0 +1,12 @@
// From: https://kafka.apache.org/protocol.html#The_Messages_FindCoordinator
/**
* @typedef {number} CoordinatorType
*
* Enum for the types of coordinator to find.
* @enum {CoordinatorType}
*/
module.exports = {
GROUP: 0,
TRANSACTION: 1,
}

270
node_modules/kafkajs/src/protocol/crc32.js generated vendored Normal file
View file

@ -0,0 +1,270 @@
// Based on https://github.com/brianloveswords/buffer-crc32/blob/master/index.js
var CRC_TABLE = new Int32Array([
0x00000000,
0x77073096,
0xee0e612c,
0x990951ba,
0x076dc419,
0x706af48f,
0xe963a535,
0x9e6495a3,
0x0edb8832,
0x79dcb8a4,
0xe0d5e91e,
0x97d2d988,
0x09b64c2b,
0x7eb17cbd,
0xe7b82d07,
0x90bf1d91,
0x1db71064,
0x6ab020f2,
0xf3b97148,
0x84be41de,
0x1adad47d,
0x6ddde4eb,
0xf4d4b551,
0x83d385c7,
0x136c9856,
0x646ba8c0,
0xfd62f97a,
0x8a65c9ec,
0x14015c4f,
0x63066cd9,
0xfa0f3d63,
0x8d080df5,
0x3b6e20c8,
0x4c69105e,
0xd56041e4,
0xa2677172,
0x3c03e4d1,
0x4b04d447,
0xd20d85fd,
0xa50ab56b,
0x35b5a8fa,
0x42b2986c,
0xdbbbc9d6,
0xacbcf940,
0x32d86ce3,
0x45df5c75,
0xdcd60dcf,
0xabd13d59,
0x26d930ac,
0x51de003a,
0xc8d75180,
0xbfd06116,
0x21b4f4b5,
0x56b3c423,
0xcfba9599,
0xb8bda50f,
0x2802b89e,
0x5f058808,
0xc60cd9b2,
0xb10be924,
0x2f6f7c87,
0x58684c11,
0xc1611dab,
0xb6662d3d,
0x76dc4190,
0x01db7106,
0x98d220bc,
0xefd5102a,
0x71b18589,
0x06b6b51f,
0x9fbfe4a5,
0xe8b8d433,
0x7807c9a2,
0x0f00f934,
0x9609a88e,
0xe10e9818,
0x7f6a0dbb,
0x086d3d2d,
0x91646c97,
0xe6635c01,
0x6b6b51f4,
0x1c6c6162,
0x856530d8,
0xf262004e,
0x6c0695ed,
0x1b01a57b,
0x8208f4c1,
0xf50fc457,
0x65b0d9c6,
0x12b7e950,
0x8bbeb8ea,
0xfcb9887c,
0x62dd1ddf,
0x15da2d49,
0x8cd37cf3,
0xfbd44c65,
0x4db26158,
0x3ab551ce,
0xa3bc0074,
0xd4bb30e2,
0x4adfa541,
0x3dd895d7,
0xa4d1c46d,
0xd3d6f4fb,
0x4369e96a,
0x346ed9fc,
0xad678846,
0xda60b8d0,
0x44042d73,
0x33031de5,
0xaa0a4c5f,
0xdd0d7cc9,
0x5005713c,
0x270241aa,
0xbe0b1010,
0xc90c2086,
0x5768b525,
0x206f85b3,
0xb966d409,
0xce61e49f,
0x5edef90e,
0x29d9c998,
0xb0d09822,
0xc7d7a8b4,
0x59b33d17,
0x2eb40d81,
0xb7bd5c3b,
0xc0ba6cad,
0xedb88320,
0x9abfb3b6,
0x03b6e20c,
0x74b1d29a,
0xead54739,
0x9dd277af,
0x04db2615,
0x73dc1683,
0xe3630b12,
0x94643b84,
0x0d6d6a3e,
0x7a6a5aa8,
0xe40ecf0b,
0x9309ff9d,
0x0a00ae27,
0x7d079eb1,
0xf00f9344,
0x8708a3d2,
0x1e01f268,
0x6906c2fe,
0xf762575d,
0x806567cb,
0x196c3671,
0x6e6b06e7,
0xfed41b76,
0x89d32be0,
0x10da7a5a,
0x67dd4acc,
0xf9b9df6f,
0x8ebeeff9,
0x17b7be43,
0x60b08ed5,
0xd6d6a3e8,
0xa1d1937e,
0x38d8c2c4,
0x4fdff252,
0xd1bb67f1,
0xa6bc5767,
0x3fb506dd,
0x48b2364b,
0xd80d2bda,
0xaf0a1b4c,
0x36034af6,
0x41047a60,
0xdf60efc3,
0xa867df55,
0x316e8eef,
0x4669be79,
0xcb61b38c,
0xbc66831a,
0x256fd2a0,
0x5268e236,
0xcc0c7795,
0xbb0b4703,
0x220216b9,
0x5505262f,
0xc5ba3bbe,
0xb2bd0b28,
0x2bb45a92,
0x5cb36a04,
0xc2d7ffa7,
0xb5d0cf31,
0x2cd99e8b,
0x5bdeae1d,
0x9b64c2b0,
0xec63f226,
0x756aa39c,
0x026d930a,
0x9c0906a9,
0xeb0e363f,
0x72076785,
0x05005713,
0x95bf4a82,
0xe2b87a14,
0x7bb12bae,
0x0cb61b38,
0x92d28e9b,
0xe5d5be0d,
0x7cdcefb7,
0x0bdbdf21,
0x86d3d2d4,
0xf1d4e242,
0x68ddb3f8,
0x1fda836e,
0x81be16cd,
0xf6b9265b,
0x6fb077e1,
0x18b74777,
0x88085ae6,
0xff0f6a70,
0x66063bca,
0x11010b5c,
0x8f659eff,
0xf862ae69,
0x616bffd3,
0x166ccf45,
0xa00ae278,
0xd70dd2ee,
0x4e048354,
0x3903b3c2,
0xa7672661,
0xd06016f7,
0x4969474d,
0x3e6e77db,
0xaed16a4a,
0xd9d65adc,
0x40df0b66,
0x37d83bf0,
0xa9bcae53,
0xdebb9ec5,
0x47b2cf7f,
0x30b5ffe9,
0xbdbdf21c,
0xcabac28a,
0x53b39330,
0x24b4a3a6,
0xbad03605,
0xcdd70693,
0x54de5729,
0x23d967bf,
0xb3667a2e,
0xc4614ab8,
0x5d681b02,
0x2a6f2b94,
0xb40bbe37,
0xc30c8ea1,
0x5a05df1b,
0x2d02ef8d,
])
module.exports = encoder => {
const { buffer } = encoder
const l = buffer.length
let crc = -1
for (let n = 0; n < l; n++) {
crc = CRC_TABLE[(crc ^ buffer[n]) & 0xff] ^ (crc >>> 8)
}
return crc ^ -1
}

309
node_modules/kafkajs/src/protocol/decoder.js generated vendored Normal file
View file

@ -0,0 +1,309 @@
const { KafkaJSInvalidVarIntError, KafkaJSInvalidLongError } = require('../errors')
const Long = require('../utils/long')
const INT8_SIZE = 1
const INT16_SIZE = 2
const INT32_SIZE = 4
const INT64_SIZE = 8
const DOUBLE_SIZE = 8
const MOST_SIGNIFICANT_BIT = 0x80 // 128
const OTHER_BITS = 0x7f // 127
module.exports = class Decoder {
static int32Size() {
return INT32_SIZE
}
static decodeZigZag(value) {
return (value >>> 1) ^ -(value & 1)
}
static decodeZigZag64(longValue) {
return longValue.shiftRightUnsigned(1).xor(longValue.and(Long.fromInt(1)).negate())
}
constructor(buffer) {
this.buffer = buffer
this.offset = 0
}
readInt8() {
const value = this.buffer.readInt8(this.offset)
this.offset += INT8_SIZE
return value
}
canReadInt16() {
return this.canReadBytes(INT16_SIZE)
}
readInt16() {
const value = this.buffer.readInt16BE(this.offset)
this.offset += INT16_SIZE
return value
}
canReadInt32() {
return this.canReadBytes(INT32_SIZE)
}
readInt32() {
const value = this.buffer.readInt32BE(this.offset)
this.offset += INT32_SIZE
return value
}
canReadInt64() {
return this.canReadBytes(INT64_SIZE)
}
readInt64() {
const first = this.buffer[this.offset]
const last = this.buffer[this.offset + 7]
const low =
(first << 24) + // Overflow
this.buffer[this.offset + 1] * 2 ** 16 +
this.buffer[this.offset + 2] * 2 ** 8 +
this.buffer[this.offset + 3]
const high =
this.buffer[this.offset + 4] * 2 ** 24 +
this.buffer[this.offset + 5] * 2 ** 16 +
this.buffer[this.offset + 6] * 2 ** 8 +
last
this.offset += INT64_SIZE
return (BigInt(low) << 32n) + BigInt(high)
}
readDouble() {
const value = this.buffer.readDoubleBE(this.offset)
this.offset += DOUBLE_SIZE
return value
}
readString() {
const byteLength = this.readInt16()
if (byteLength === -1) {
return null
}
const stringBuffer = this.buffer.slice(this.offset, this.offset + byteLength)
const value = stringBuffer.toString('utf8')
this.offset += byteLength
return value
}
readVarIntString() {
const byteLength = this.readVarInt()
if (byteLength === -1) {
return null
}
const stringBuffer = this.buffer.slice(this.offset, this.offset + byteLength)
const value = stringBuffer.toString('utf8')
this.offset += byteLength
return value
}
readUVarIntString() {
const byteLength = this.readUVarInt()
if (byteLength === 0) {
return null
}
const stringBuffer = this.buffer.slice(this.offset, this.offset + byteLength - 1)
const value = stringBuffer.toString('utf8')
this.offset += byteLength - 1
return value
}
canReadBytes(length) {
return Buffer.byteLength(this.buffer) - this.offset >= length
}
readBytes(byteLength = this.readInt32()) {
if (byteLength === -1) {
return null
}
const stringBuffer = this.buffer.slice(this.offset, this.offset + byteLength)
this.offset += byteLength
return stringBuffer
}
readVarIntBytes() {
const byteLength = this.readVarInt()
if (byteLength === -1) {
return null
}
const stringBuffer = this.buffer.slice(this.offset, this.offset + byteLength)
this.offset += byteLength
return stringBuffer
}
readUVarIntBytes() {
const byteLength = this.readUVarInt()
if (byteLength === 0) {
return null
}
const stringBuffer = this.buffer.slice(this.offset, this.offset + byteLength)
this.offset += byteLength - 1
return stringBuffer
}
readBoolean() {
return this.readInt8() === 1
}
readAll() {
const result = this.buffer.slice(this.offset)
this.offset += Buffer.byteLength(this.buffer)
return result
}
readArray(reader) {
const length = this.readInt32()
if (length === -1) {
return []
}
const array = new Array(length)
for (let i = 0; i < length; i++) {
array[i] = reader(this)
}
return array
}
readVarIntArray(reader) {
const length = this.readVarInt()
if (length === -1) {
return []
}
const array = new Array(length)
for (let i = 0; i < length; i++) {
array[i] = reader(this)
}
return array
}
/* According to the protocol type documentation: https://kafka.apache.org/protocol#protocol_types,
a compact array with length zero is a null array. An array with length 1 is an empty array. */
readUVarIntArray(reader) {
const length = this.readUVarInt()
if (length === 0) {
return null
}
const array = new Array(length - 1)
for (let i = 0; i < length - 1; i++) {
array[i] = reader(this)
}
return array
}
async readArrayAsync(reader) {
const length = this.readInt32()
if (length === -1) {
return []
}
const array = new Array(length)
for (let i = 0; i < length; i++) {
array[i] = await reader(this)
}
return array
}
readVarInt() {
let currentByte
let result = 0
let i = 0
do {
currentByte = this.buffer[this.offset++]
result += (currentByte & OTHER_BITS) << i
i += 7
} while (currentByte >= MOST_SIGNIFICANT_BIT)
return Decoder.decodeZigZag(result)
}
// By default JavaScript's numbers are of type float64, performing bitwise operations converts the numbers to a signed 32-bit integer
// Unsigned Right Shift Operator >>> ensures the returned value is an unsigned 32-bit integer
readUVarInt() {
let currentByte
let result = 0
let i = 0
while (((currentByte = this.buffer[this.offset++]) & MOST_SIGNIFICANT_BIT) !== 0) {
result |= (currentByte & OTHER_BITS) << i
i += 7
if (i > 28) {
throw new KafkaJSInvalidVarIntError('Invalid VarInt, must contain 5 bytes or less')
}
}
result |= currentByte << i
return result >>> 0
}
readTaggedFields() {
const numberOfTaggedFields = this.readUVarInt()
if (numberOfTaggedFields === 0) {
return null
}
const taggedFields = {}
for (let i = 0; i < numberOfTaggedFields; i++) {
// Right now this will read tag, the field length, and then length number of bytes for the field value skipping over the tag
this.readUVarInt()
this.readUVarIntBytes()
}
return taggedFields
}
readVarLong() {
let currentByte
let result = Long.fromInt(0)
let i = 0
do {
if (i > 63) {
throw new KafkaJSInvalidLongError('Invalid Long, must contain 9 bytes or less')
}
currentByte = this.buffer[this.offset++]
result = result.add(Long.fromInt(currentByte & OTHER_BITS).shiftLeft(i))
i += 7
} while (currentByte >= MOST_SIGNIFICANT_BIT)
return Decoder.decodeZigZag64(result)
}
slice(size) {
return new Decoder(this.buffer.slice(this.offset, this.offset + size))
}
forward(size) {
this.offset += size
}
}

405
node_modules/kafkajs/src/protocol/encoder.js generated vendored Normal file
View file

@ -0,0 +1,405 @@
const Long = require('../utils/long')
const INT8_SIZE = 1
const INT16_SIZE = 2
const INT32_SIZE = 4
const INT64_SIZE = 8
const DOUBLE_SIZE = 8
const MOST_SIGNIFICANT_BIT = 0x80 // 128
const OTHER_BITS = 0x7f // 127
const UNSIGNED_INT32_MAX_NUMBER = 0xffffff80
const UNSIGNED_INT64_MAX_NUMBER = 0xffffffffffffff80n
module.exports = class Encoder {
static encodeZigZag(value) {
return (value << 1) ^ (value >> 31)
}
static encodeZigZag64(value) {
const longValue = Long.fromValue(value)
return longValue.shiftLeft(1).xor(longValue.shiftRight(63))
}
static sizeOfVarInt(value) {
let encodedValue = this.encodeZigZag(value)
let bytes = 1
while ((encodedValue & UNSIGNED_INT32_MAX_NUMBER) !== 0) {
bytes += 1
encodedValue >>>= 7
}
return bytes
}
static sizeOfVarLong(value) {
let longValue = Encoder.encodeZigZag64(value)
let bytes = 1
while (longValue.and(UNSIGNED_INT64_MAX_NUMBER).notEquals(Long.fromInt(0))) {
bytes += 1
longValue = longValue.shiftRightUnsigned(7)
}
return bytes
}
static sizeOfVarIntBytes(value) {
const size = value == null ? -1 : Buffer.byteLength(value)
if (size < 0) {
return Encoder.sizeOfVarInt(-1)
}
return Encoder.sizeOfVarInt(size) + size
}
static nextPowerOfTwo(value) {
return 1 << (31 - Math.clz32(value) + 1)
}
/**
* Construct a new encoder with the given initial size
*
* @param {number} [initialSize] initial size
*/
constructor(initialSize = 511) {
this.buf = Buffer.alloc(Encoder.nextPowerOfTwo(initialSize))
this.offset = 0
}
/**
* @param {Buffer} buffer
*/
writeBufferInternal(buffer) {
const bufferLength = buffer.length
this.ensureAvailable(bufferLength)
buffer.copy(this.buf, this.offset, 0)
this.offset += bufferLength
}
ensureAvailable(length) {
if (this.offset + length > this.buf.length) {
const newLength = Encoder.nextPowerOfTwo(this.offset + length)
const newBuffer = Buffer.alloc(newLength)
this.buf.copy(newBuffer, 0, 0, this.offset)
this.buf = newBuffer
}
}
get buffer() {
return this.buf.slice(0, this.offset)
}
writeInt8(value) {
this.ensureAvailable(INT8_SIZE)
this.buf.writeInt8(value, this.offset)
this.offset += INT8_SIZE
return this
}
writeInt16(value) {
this.ensureAvailable(INT16_SIZE)
this.buf.writeInt16BE(value, this.offset)
this.offset += INT16_SIZE
return this
}
writeInt32(value) {
this.ensureAvailable(INT32_SIZE)
this.buf.writeInt32BE(value, this.offset)
this.offset += INT32_SIZE
return this
}
writeUInt32(value) {
this.ensureAvailable(INT32_SIZE)
this.buf.writeUInt32BE(value, this.offset)
this.offset += INT32_SIZE
return this
}
writeInt64(value) {
this.ensureAvailable(INT64_SIZE)
const longValue = Long.fromValue(value)
this.buf.writeInt32BE(longValue.getHighBits(), this.offset)
this.buf.writeInt32BE(longValue.getLowBits(), this.offset + INT32_SIZE)
this.offset += INT64_SIZE
return this
}
writeDouble(value) {
this.ensureAvailable(DOUBLE_SIZE)
this.buf.writeDoubleBE(value, this.offset)
this.offset += DOUBLE_SIZE
return this
}
writeBoolean(value) {
value ? this.writeInt8(1) : this.writeInt8(0)
return this
}
writeString(value) {
if (value == null) {
this.writeInt16(-1)
return this
}
const byteLength = Buffer.byteLength(value, 'utf8')
this.ensureAvailable(INT16_SIZE + byteLength)
this.writeInt16(byteLength)
this.buf.write(value, this.offset, byteLength, 'utf8')
this.offset += byteLength
return this
}
writeVarIntString(value) {
if (value == null) {
this.writeVarInt(-1)
return this
}
const byteLength = Buffer.byteLength(value, 'utf8')
this.writeVarInt(byteLength)
this.ensureAvailable(byteLength)
this.buf.write(value, this.offset, byteLength, 'utf8')
this.offset += byteLength
return this
}
writeUVarIntString(value) {
if (value == null) {
this.writeUVarInt(0)
return this
}
const byteLength = Buffer.byteLength(value, 'utf8')
this.writeUVarInt(byteLength + 1)
this.ensureAvailable(byteLength)
this.buf.write(value, this.offset, byteLength, 'utf8')
this.offset += byteLength
return this
}
writeBytes(value) {
if (value == null) {
this.writeInt32(-1)
return this
}
if (Buffer.isBuffer(value)) {
// raw bytes
this.ensureAvailable(INT32_SIZE + value.length)
this.writeInt32(value.length)
this.writeBufferInternal(value)
} else {
const valueToWrite = String(value)
const byteLength = Buffer.byteLength(valueToWrite, 'utf8')
this.ensureAvailable(INT32_SIZE + byteLength)
this.writeInt32(byteLength)
this.buf.write(valueToWrite, this.offset, byteLength, 'utf8')
this.offset += byteLength
}
return this
}
writeVarIntBytes(value) {
if (value == null) {
this.writeVarInt(-1)
return this
}
if (Buffer.isBuffer(value)) {
// raw bytes
this.writeVarInt(value.length)
this.writeBufferInternal(value)
} else {
const valueToWrite = String(value)
const byteLength = Buffer.byteLength(valueToWrite, 'utf8')
this.writeVarInt(byteLength)
this.ensureAvailable(byteLength)
this.buf.write(valueToWrite, this.offset, byteLength, 'utf8')
this.offset += byteLength
}
return this
}
writeUVarIntBytes(value) {
if (value == null) {
this.writeVarInt(0)
return this
}
if (Buffer.isBuffer(value)) {
// raw bytes
this.writeUVarInt(value.length + 1)
this.writeBufferInternal(value)
} else {
const valueToWrite = String(value)
const byteLength = Buffer.byteLength(valueToWrite, 'utf8')
this.writeUVarInt(byteLength + 1)
this.ensureAvailable(byteLength)
this.buf.write(valueToWrite, this.offset, byteLength, 'utf8')
this.offset += byteLength
}
return this
}
writeEncoder(value) {
if (value == null || !Buffer.isBuffer(value.buf)) {
throw new Error('value should be an instance of Encoder')
}
this.writeBufferInternal(value.buffer)
return this
}
writeEncoderArray(value) {
if (!Array.isArray(value) || value.some(v => v == null || !Buffer.isBuffer(v.buf))) {
throw new Error('all values should be an instance of Encoder[]')
}
value.forEach(v => {
this.writeBufferInternal(v.buffer)
})
return this
}
writeBuffer(value) {
if (!Buffer.isBuffer(value)) {
throw new Error('value should be an instance of Buffer')
}
this.writeBufferInternal(value)
return this
}
/**
* @param {any[]} array
* @param {'int32'|'number'|'string'|'object'} [type]
*/
writeNullableArray(array, type) {
// A null value is encoded with length of -1 and there are no following bytes
// On the context of this library, empty array and null are the same thing
const length = array.length !== 0 ? array.length : -1
this.writeArray(array, type, length)
return this
}
/**
* @param {any[]} array
* @param {'int32'|'number'|'string'|'object'} [type]
* @param {number} [length]
*/
writeArray(array, type, length) {
const arrayLength = length == null ? array.length : length
this.writeInt32(arrayLength)
if (type !== undefined) {
switch (type) {
case 'int32':
case 'number':
array.forEach(value => this.writeInt32(value))
break
case 'string':
array.forEach(value => this.writeString(value))
break
case 'object':
this.writeEncoderArray(array)
break
}
} else {
array.forEach(value => {
switch (typeof value) {
case 'number':
this.writeInt32(value)
break
case 'string':
this.writeString(value)
break
case 'object':
this.writeEncoder(value)
break
}
})
}
return this
}
writeVarIntArray(array, type) {
if (type === 'object') {
this.writeVarInt(array.length)
this.writeEncoderArray(array)
} else {
const objectArray = array.filter(v => typeof v === 'object')
this.writeVarInt(objectArray.length)
this.writeEncoderArray(objectArray)
}
return this
}
writeUVarIntArray(array, type) {
if (type === 'object') {
this.writeUVarInt(array.length + 1)
this.writeEncoderArray(array)
} else if (array === null) {
this.writeUVarInt(0)
} else {
const objectArray = array.filter(v => typeof v === 'object')
this.writeUVarInt(objectArray.length + 1)
this.writeEncoderArray(objectArray)
}
return this
}
// Based on:
// https://en.wikipedia.org/wiki/LEB128 Using LEB128 format similar to VLQ.
// https://github.com/addthis/stream-lib/blob/master/src/main/java/com/clearspring/analytics/util/Varint.java#L106
writeVarInt(value) {
return this.writeUVarInt(Encoder.encodeZigZag(value))
}
writeUVarInt(value) {
const byteArray = []
while ((value & UNSIGNED_INT32_MAX_NUMBER) !== 0) {
byteArray.push((value & OTHER_BITS) | MOST_SIGNIFICANT_BIT)
value >>>= 7
}
byteArray.push(value & OTHER_BITS)
this.writeBufferInternal(Buffer.from(byteArray))
return this
}
writeVarLong(value) {
const byteArray = []
let longValue = Encoder.encodeZigZag64(value)
while (longValue.and(UNSIGNED_INT64_MAX_NUMBER).notEquals(Long.fromInt(0))) {
byteArray.push(
longValue
.and(OTHER_BITS)
.or(MOST_SIGNIFICANT_BIT)
.toInt()
)
longValue = longValue.shiftRightUnsigned(7)
}
byteArray.push(longValue.toInt())
this.writeBufferInternal(Buffer.from(byteArray))
return this
}
size() {
// We can use the offset here directly, because we anyways will not re-encode the buffer when writing
return this.offset
}
toJSON() {
return this.buffer.toJSON()
}
}

601
node_modules/kafkajs/src/protocol/error.js generated vendored Normal file
View file

@ -0,0 +1,601 @@
const { KafkaJSProtocolError } = require('../errors')
const websiteUrl = require('../utils/websiteUrl')
const errorCodes = [
{
type: 'UNKNOWN',
code: -1,
retriable: false,
message: 'The server experienced an unexpected error when processing the request',
},
{
type: 'OFFSET_OUT_OF_RANGE',
code: 1,
retriable: false,
message: 'The requested offset is not within the range of offsets maintained by the server',
},
{
type: 'CORRUPT_MESSAGE',
code: 2,
retriable: true,
message:
'This message has failed its CRC checksum, exceeds the valid size, or is otherwise corrupt',
},
{
type: 'UNKNOWN_TOPIC_OR_PARTITION',
code: 3,
retriable: true,
message: 'This server does not host this topic-partition',
},
{
type: 'INVALID_FETCH_SIZE',
code: 4,
retriable: false,
message: 'The requested fetch size is invalid',
},
{
type: 'LEADER_NOT_AVAILABLE',
code: 5,
retriable: true,
message:
'There is no leader for this topic-partition as we are in the middle of a leadership election',
},
{
type: 'NOT_LEADER_FOR_PARTITION',
code: 6,
retriable: true,
message: 'This server is not the leader for that topic-partition',
},
{
type: 'REQUEST_TIMED_OUT',
code: 7,
retriable: true,
message: 'The request timed out',
},
{
type: 'BROKER_NOT_AVAILABLE',
code: 8,
retriable: false,
message: 'The broker is not available',
},
{
type: 'REPLICA_NOT_AVAILABLE',
code: 9,
retriable: true,
message: 'The replica is not available for the requested topic-partition',
},
{
type: 'MESSAGE_TOO_LARGE',
code: 10,
retriable: false,
message:
'The request included a message larger than the max message size the server will accept',
},
{
type: 'STALE_CONTROLLER_EPOCH',
code: 11,
retriable: false,
message: 'The controller moved to another broker',
},
{
type: 'OFFSET_METADATA_TOO_LARGE',
code: 12,
retriable: false,
message: 'The metadata field of the offset request was too large',
},
{
type: 'NETWORK_EXCEPTION',
code: 13,
retriable: true,
message: 'The server disconnected before a response was received',
},
{
type: 'GROUP_LOAD_IN_PROGRESS',
code: 14,
retriable: true,
message: "The coordinator is loading and hence can't process requests for this group",
},
{
type: 'GROUP_COORDINATOR_NOT_AVAILABLE',
code: 15,
retriable: true,
message: 'The group coordinator is not available',
},
{
type: 'NOT_COORDINATOR_FOR_GROUP',
code: 16,
retriable: true,
message: 'This is not the correct coordinator for this group',
},
{
type: 'INVALID_TOPIC_EXCEPTION',
code: 17,
retriable: false,
message: 'The request attempted to perform an operation on an invalid topic',
},
{
type: 'RECORD_LIST_TOO_LARGE',
code: 18,
retriable: false,
message:
'The request included message batch larger than the configured segment size on the server',
},
{
type: 'NOT_ENOUGH_REPLICAS',
code: 19,
retriable: true,
message: 'Messages are rejected since there are fewer in-sync replicas than required',
},
{
type: 'NOT_ENOUGH_REPLICAS_AFTER_APPEND',
code: 20,
retriable: true,
message: 'Messages are written to the log, but to fewer in-sync replicas than required',
},
{
type: 'INVALID_REQUIRED_ACKS',
code: 21,
retriable: false,
message: 'Produce request specified an invalid value for required acks',
},
{
type: 'ILLEGAL_GENERATION',
code: 22,
retriable: false,
message: 'Specified group generation id is not valid',
},
{
type: 'INCONSISTENT_GROUP_PROTOCOL',
code: 23,
retriable: false,
message:
"The group member's supported protocols are incompatible with those of existing members",
},
{
type: 'INVALID_GROUP_ID',
code: 24,
retriable: false,
message: 'The configured groupId is invalid',
},
{
type: 'UNKNOWN_MEMBER_ID',
code: 25,
retriable: false,
message: 'The coordinator is not aware of this member',
},
{
type: 'INVALID_SESSION_TIMEOUT',
code: 26,
retriable: false,
message:
'The session timeout is not within the range allowed by the broker (as configured by group.min.session.timeout.ms and group.max.session.timeout.ms)',
},
{
type: 'REBALANCE_IN_PROGRESS',
code: 27,
retriable: false,
message: 'The group is rebalancing, so a rejoin is needed',
helpUrl: websiteUrl('docs/faq', 'what-does-it-mean-to-get-rebalance-in-progress-errors'),
},
{
type: 'INVALID_COMMIT_OFFSET_SIZE',
code: 28,
retriable: false,
message: 'The committing offset data size is not valid',
},
{
type: 'TOPIC_AUTHORIZATION_FAILED',
code: 29,
retriable: false,
message: 'Not authorized to access topics: [Topic authorization failed]',
},
{
type: 'GROUP_AUTHORIZATION_FAILED',
code: 30,
retriable: false,
message: 'Not authorized to access group: Group authorization failed',
},
{
type: 'CLUSTER_AUTHORIZATION_FAILED',
code: 31,
retriable: false,
message: 'Cluster authorization failed',
},
{
type: 'INVALID_TIMESTAMP',
code: 32,
retriable: false,
message: 'The timestamp of the message is out of acceptable range',
},
{
type: 'UNSUPPORTED_SASL_MECHANISM',
code: 33,
retriable: false,
message: 'The broker does not support the requested SASL mechanism',
},
{
type: 'ILLEGAL_SASL_STATE',
code: 34,
retriable: false,
message: 'Request is not valid given the current SASL state',
},
{
type: 'UNSUPPORTED_VERSION',
code: 35,
retriable: false,
message: 'The version of API is not supported',
},
{
type: 'TOPIC_ALREADY_EXISTS',
code: 36,
retriable: false,
message: 'Topic with this name already exists',
},
{
type: 'INVALID_PARTITIONS',
code: 37,
retriable: false,
message: 'Number of partitions is invalid',
},
{
type: 'INVALID_REPLICATION_FACTOR',
code: 38,
retriable: false,
message: 'Replication-factor is invalid',
},
{
type: 'INVALID_REPLICA_ASSIGNMENT',
code: 39,
retriable: false,
message: 'Replica assignment is invalid',
},
{
type: 'INVALID_CONFIG',
code: 40,
retriable: false,
message: 'Configuration is invalid',
},
{
type: 'NOT_CONTROLLER',
code: 41,
retriable: true,
message: 'This is not the correct controller for this cluster',
},
{
type: 'INVALID_REQUEST',
code: 42,
retriable: false,
message:
'This most likely occurs because of a request being malformed by the client library or the message was sent to an incompatible broker. See the broker logs for more details',
},
{
type: 'UNSUPPORTED_FOR_MESSAGE_FORMAT',
code: 43,
retriable: false,
message: 'The message format version on the broker does not support the request',
},
{
type: 'POLICY_VIOLATION',
code: 44,
retriable: false,
message: 'Request parameters do not satisfy the configured policy',
},
{
type: 'OUT_OF_ORDER_SEQUENCE_NUMBER',
code: 45,
retriable: false,
message: 'The broker received an out of order sequence number',
},
{
type: 'DUPLICATE_SEQUENCE_NUMBER',
code: 46,
retriable: false,
message: 'The broker received a duplicate sequence number',
},
{
type: 'INVALID_PRODUCER_EPOCH',
code: 47,
retriable: false,
message:
"Producer attempted an operation with an old epoch. Either there is a newer producer with the same transactionalId, or the producer's transaction has been expired by the broker",
},
{
type: 'INVALID_TXN_STATE',
code: 48,
retriable: false,
message: 'The producer attempted a transactional operation in an invalid state',
},
{
type: 'INVALID_PRODUCER_ID_MAPPING',
code: 49,
retriable: false,
message:
'The producer attempted to use a producer id which is not currently assigned to its transactional id',
},
{
type: 'INVALID_TRANSACTION_TIMEOUT',
code: 50,
retriable: false,
message:
'The transaction timeout is larger than the maximum value allowed by the broker (as configured by max.transaction.timeout.ms)',
},
{
type: 'CONCURRENT_TRANSACTIONS',
code: 51,
/**
* The concurrent transactions error has "retriable" set to false on the protocol documentation (https://kafka.apache.org/protocol.html#protocol_error_codes)
* but the server expects the clients to retry. PR #223
* @see https://github.com/apache/kafka/blob/12f310d50e7f5b1c18c4f61a119a6cd830da3bc0/core/src/main/scala/kafka/coordinator/transaction/TransactionCoordinator.scala#L153
*/
retriable: true,
message:
'The producer attempted to update a transaction while another concurrent operation on the same transaction was ongoing',
},
{
type: 'TRANSACTION_COORDINATOR_FENCED',
code: 52,
retriable: false,
message:
'Indicates that the transaction coordinator sending a WriteTxnMarker is no longer the current coordinator for a given producer',
},
{
type: 'TRANSACTIONAL_ID_AUTHORIZATION_FAILED',
code: 53,
retriable: false,
message: 'Transactional Id authorization failed',
},
{
type: 'SECURITY_DISABLED',
code: 54,
retriable: false,
message: 'Security features are disabled',
},
{
type: 'OPERATION_NOT_ATTEMPTED',
code: 55,
retriable: false,
message:
'The broker did not attempt to execute this operation. This may happen for batched RPCs where some operations in the batch failed, causing the broker to respond without trying the rest',
},
{
type: 'KAFKA_STORAGE_ERROR',
code: 56,
retriable: true,
message: 'Disk error when trying to access log file on the disk',
},
{
type: 'LOG_DIR_NOT_FOUND',
code: 57,
retriable: false,
message: 'The user-specified log directory is not found in the broker config',
},
{
type: 'SASL_AUTHENTICATION_FAILED',
code: 58,
retriable: false,
message: 'SASL Authentication failed',
helpUrl: websiteUrl('docs/configuration', 'sasl'),
},
{
type: 'UNKNOWN_PRODUCER_ID',
code: 59,
retriable: false,
message:
"This exception is raised by the broker if it could not locate the producer metadata associated with the producerId in question. This could happen if, for instance, the producer's records were deleted because their retention time had elapsed. Once the last records of the producerId are removed, the producer's metadata is removed from the broker, and future appends by the producer will return this exception",
},
{
type: 'REASSIGNMENT_IN_PROGRESS',
code: 60,
retriable: false,
message: 'A partition reassignment is in progress',
},
{
type: 'DELEGATION_TOKEN_AUTH_DISABLED',
code: 61,
retriable: false,
message: 'Delegation Token feature is not enabled',
},
{
type: 'DELEGATION_TOKEN_NOT_FOUND',
code: 62,
retriable: false,
message: 'Delegation Token is not found on server',
},
{
type: 'DELEGATION_TOKEN_OWNER_MISMATCH',
code: 63,
retriable: false,
message: 'Specified Principal is not valid Owner/Renewer',
},
{
type: 'DELEGATION_TOKEN_REQUEST_NOT_ALLOWED',
code: 64,
retriable: false,
message:
'Delegation Token requests are not allowed on PLAINTEXT/1-way SSL channels and on delegation token authenticated channels',
},
{
type: 'DELEGATION_TOKEN_AUTHORIZATION_FAILED',
code: 65,
retriable: false,
message: 'Delegation Token authorization failed',
},
{
type: 'DELEGATION_TOKEN_EXPIRED',
code: 66,
retriable: false,
message: 'Delegation Token is expired',
},
{
type: 'INVALID_PRINCIPAL_TYPE',
code: 67,
retriable: false,
message: 'Supplied principalType is not supported',
},
{
type: 'NON_EMPTY_GROUP',
code: 68,
retriable: false,
message: 'The group is not empty',
},
{
type: 'GROUP_ID_NOT_FOUND',
code: 69,
retriable: false,
message: 'The group id was not found',
},
{
type: 'FETCH_SESSION_ID_NOT_FOUND',
code: 70,
retriable: true,
message: 'The fetch session ID was not found',
},
{
type: 'INVALID_FETCH_SESSION_EPOCH',
code: 71,
retriable: true,
message: 'The fetch session epoch is invalid',
},
{
type: 'LISTENER_NOT_FOUND',
code: 72,
retriable: true,
message:
'There is no listener on the leader broker that matches the listener on which metadata request was processed',
},
{
type: 'TOPIC_DELETION_DISABLED',
code: 73,
retriable: false,
message: 'Topic deletion is disabled',
},
{
type: 'FENCED_LEADER_EPOCH',
code: 74,
retriable: true,
message: 'The leader epoch in the request is older than the epoch on the broker',
},
{
type: 'UNKNOWN_LEADER_EPOCH',
code: 75,
retriable: true,
message: 'The leader epoch in the request is newer than the epoch on the broker',
},
{
type: 'UNSUPPORTED_COMPRESSION_TYPE',
code: 76,
retriable: false,
message: 'The requesting client does not support the compression type of given partition',
},
{
type: 'STALE_BROKER_EPOCH',
code: 77,
retriable: false,
message: 'Broker epoch has changed',
},
{
type: 'OFFSET_NOT_AVAILABLE',
code: 78,
retriable: true,
message:
'The leader high watermark has not caught up from a recent leader election so the offsets cannot be guaranteed to be monotonically increasing',
},
{
type: 'MEMBER_ID_REQUIRED',
code: 79,
retriable: false,
message:
'The group member needs to have a valid member id before actually entering a consumer group',
},
{
type: 'PREFERRED_LEADER_NOT_AVAILABLE',
code: 80,
retriable: true,
message: 'The preferred leader was not available',
},
{
type: 'GROUP_MAX_SIZE_REACHED',
code: 81,
retriable: false,
message:
'The consumer group has reached its max size. It already has the configured maximum number of members',
},
{
type: 'FENCED_INSTANCE_ID',
code: 82,
retriable: false,
message:
'The broker rejected this static consumer since another consumer with the same group instance id has registered with a different member id',
},
{
type: 'ELIGIBLE_LEADERS_NOT_AVAILABLE',
code: 83,
retriable: true,
message: 'Eligible topic partition leaders are not available',
},
{
type: 'ELECTION_NOT_NEEDED',
code: 84,
retriable: true,
message: 'Leader election not needed for topic partition',
},
{
type: 'NO_REASSIGNMENT_IN_PROGRESS',
code: 85,
retriable: false,
message: 'No partition reassignment is in progress',
},
{
type: 'GROUP_SUBSCRIBED_TO_TOPIC',
code: 86,
retriable: false,
message:
'Deleting offsets of a topic is forbidden while the consumer group is actively subscribed to it',
},
{
type: 'INVALID_RECORD',
code: 87,
retriable: false,
message: 'This record has failed the validation on broker and hence be rejected',
},
{
type: 'UNSTABLE_OFFSET_COMMIT',
code: 88,
retriable: true,
message: 'There are unstable offsets that need to be cleared',
},
]
const unknownErrorCode = errorCode => ({
type: 'KAFKAJS_UNKNOWN_ERROR_CODE',
code: -99,
retriable: false,
message: `Unknown error code ${errorCode}`,
})
const SUCCESS_CODE = 0
const UNSUPPORTED_VERSION_CODE = 35
const failure = code => code !== SUCCESS_CODE
const createErrorFromCode = code => {
return new KafkaJSProtocolError(errorCodes.find(e => e.code === code) || unknownErrorCode(code))
}
const failIfVersionNotSupported = code => {
if (code === UNSUPPORTED_VERSION_CODE) {
throw createErrorFromCode(UNSUPPORTED_VERSION_CODE)
}
}
const staleMetadata = e =>
['UNKNOWN_TOPIC_OR_PARTITION', 'LEADER_NOT_AVAILABLE', 'NOT_LEADER_FOR_PARTITION'].includes(
e.type
)
module.exports = {
failure,
errorCodes,
createErrorFromCode,
failIfVersionNotSupported,
staleMetadata,
}

15
node_modules/kafkajs/src/protocol/isolationLevel.js generated vendored Normal file
View file

@ -0,0 +1,15 @@
/**
* Enum for isolation levels
* @readonly
* @enum {number}
*/
module.exports = {
// Makes all records visible
READ_UNCOMMITTED: 0,
// non-transactional and COMMITTED transactional records are visible. It returns all data
// from offsets smaller than the current LSO (last stable offset), and enables the inclusion of
// the list of aborted transactions in the result, which allows consumers to discard ABORTED
// transactional records
READ_COMMITTED: 1,
}

23
node_modules/kafkajs/src/protocol/message/compression/gzip.js generated vendored Normal file
View file

@ -0,0 +1,23 @@
const { promisify } = require('util')
const zlib = require('zlib')
const gzip = promisify(zlib.gzip)
const unzip = promisify(zlib.unzip)
module.exports = {
/**
* @param {Encoder} encoder
* @returns {Promise}
*/
async compress(encoder) {
return await gzip(encoder.buffer)
},
/**
* @param {Buffer} buffer
* @returns {Promise}
*/
async decompress(buffer) {
return await unzip(buffer)
},
}

38
node_modules/kafkajs/src/protocol/message/compression/index.js generated vendored Normal file
View file

@ -0,0 +1,38 @@
const { KafkaJSNotImplemented } = require('../../../errors')
const COMPRESSION_CODEC_MASK = 0x07
const Types = {
None: 0,
GZIP: 1,
Snappy: 2,
LZ4: 3,
ZSTD: 4,
}
const Codecs = {
[Types.GZIP]: () => require('./gzip'),
[Types.Snappy]: () => {
throw new KafkaJSNotImplemented('Snappy compression not implemented')
},
[Types.LZ4]: () => {
throw new KafkaJSNotImplemented('LZ4 compression not implemented')
},
[Types.ZSTD]: () => {
throw new KafkaJSNotImplemented('ZSTD compression not implemented')
},
}
const lookupCodec = type => (Codecs[type] ? Codecs[type]() : null)
const lookupCodecByAttributes = attributes => {
const codec = Codecs[attributes & COMPRESSION_CODEC_MASK]
return codec ? codec() : null
}
module.exports = {
Types,
Codecs,
lookupCodec,
lookupCodecByAttributes,
COMPRESSION_CODEC_MASK,
}

37
node_modules/kafkajs/src/protocol/message/decoder.js generated vendored Normal file
View file

@ -0,0 +1,37 @@
const {
KafkaJSPartialMessageError,
KafkaJSUnsupportedMagicByteInMessageSet,
} = require('../../errors')
const V0Decoder = require('./v0/decoder')
const V1Decoder = require('./v1/decoder')
const decodeMessage = (decoder, magicByte) => {
switch (magicByte) {
case 0:
return V0Decoder(decoder)
case 1:
return V1Decoder(decoder)
default:
throw new KafkaJSUnsupportedMagicByteInMessageSet(
`Unsupported MessageSet message version, magic byte: ${magicByte}`
)
}
}
module.exports = (offset, size, decoder) => {
// Don't decrement decoder.offset because slice is already considering the current
// offset of the decoder
const remainingBytes = Buffer.byteLength(decoder.slice(size).buffer)
if (remainingBytes < size) {
throw new KafkaJSPartialMessageError(
`Tried to decode a partial message: remainingBytes(${remainingBytes}) < messageSize(${size})`
)
}
const crc = decoder.readInt32()
const magicByte = decoder.readInt8()
const message = decodeMessage(decoder, magicByte)
return Object.assign({ offset, size, crc, magicByte }, message)
}

6
node_modules/kafkajs/src/protocol/message/index.js generated vendored Normal file
View file

@ -0,0 +1,6 @@
const versions = {
0: require('./v0'),
1: require('./v1'),
}
module.exports = ({ version = 0 }) => versions[version]

5
node_modules/kafkajs/src/protocol/message/v0/decoder.js generated vendored Normal file
View file

@ -0,0 +1,5 @@
module.exports = decoder => ({
attributes: decoder.readInt8(),
key: decoder.readBytes(),
value: decoder.readBytes(),
})

24
node_modules/kafkajs/src/protocol/message/v0/index.js generated vendored Normal file
View file

@ -0,0 +1,24 @@
const Encoder = require('../../encoder')
const crc32 = require('../../crc32')
const { Types: Compression, COMPRESSION_CODEC_MASK } = require('../compression')
/**
* v0
* Message => Crc MagicByte Attributes Key Value
* Crc => int32
* MagicByte => int8
* Attributes => int8
* Key => bytes
* Value => bytes
*/
module.exports = ({ compression = Compression.None, key, value }) => {
const content = new Encoder()
.writeInt8(0) // magicByte
.writeInt8(compression & COMPRESSION_CODEC_MASK)
.writeBytes(key)
.writeBytes(value)
const crc = crc32(content)
return new Encoder().writeInt32(crc).writeEncoder(content)
}

6
node_modules/kafkajs/src/protocol/message/v1/decoder.js generated vendored Normal file
View file

@ -0,0 +1,6 @@
module.exports = decoder => ({
attributes: decoder.readInt8(),
timestamp: decoder.readInt64().toString(),
key: decoder.readBytes(),
value: decoder.readBytes(),
})

26
node_modules/kafkajs/src/protocol/message/v1/index.js generated vendored Normal file
View file

@ -0,0 +1,26 @@
const Encoder = require('../../encoder')
const crc32 = require('../../crc32')
const { Types: Compression, COMPRESSION_CODEC_MASK } = require('../compression')
/**
* v1 (supported since 0.10.0)
* Message => Crc MagicByte Attributes Key Value
* Crc => int32
* MagicByte => int8
* Attributes => int8
* Timestamp => int64
* Key => bytes
* Value => bytes
*/
module.exports = ({ compression = Compression.None, timestamp = Date.now(), key, value }) => {
const content = new Encoder()
.writeInt8(1) // magicByte
.writeInt8(compression & COMPRESSION_CODEC_MASK)
.writeInt64(timestamp)
.writeBytes(key)
.writeBytes(value)
const crc = crc32(content)
return new Encoder().writeInt32(crc).writeEncoder(content)
}

91
node_modules/kafkajs/src/protocol/messageSet/decoder.js generated vendored Normal file
View file

@ -0,0 +1,91 @@
const Long = require('../../utils/long')
const Decoder = require('../decoder')
const MessageDecoder = require('../message/decoder')
const { lookupCodecByAttributes } = require('../message/compression')
const { KafkaJSPartialMessageError } = require('../../errors')
/**
* MessageSet => [Offset MessageSize Message]
* Offset => int64
* MessageSize => int32
* Message => Bytes
*/
module.exports = async (primaryDecoder, size = null) => {
const messages = []
const messageSetSize = size || primaryDecoder.readInt32()
const messageSetDecoder = primaryDecoder.slice(messageSetSize)
while (messageSetDecoder.offset < messageSetSize) {
try {
const message = EntryDecoder(messageSetDecoder)
const codec = lookupCodecByAttributes(message.attributes)
if (codec) {
const buffer = await codec.decompress(message.value)
messages.push(...EntriesDecoder(new Decoder(buffer), message))
} else {
messages.push(message)
}
} catch (e) {
if (e.name === 'KafkaJSPartialMessageError') {
// We tried to decode a partial message, it means that minBytes
// is probably too low
break
}
if (e.name === 'KafkaJSUnsupportedMagicByteInMessageSet') {
// Received a MessageSet and a RecordBatch on the same response, the cluster is probably
// upgrading the message format from 0.10 to 0.11. Stop processing this message set to
// receive the full record batch on the next request
break
}
throw e
}
}
primaryDecoder.forward(messageSetSize)
return messages
}
const EntriesDecoder = (decoder, compressedMessage) => {
const messages = []
while (decoder.offset < decoder.buffer.length) {
messages.push(EntryDecoder(decoder))
}
if (compressedMessage.magicByte > 0 && compressedMessage.offset >= 0) {
const compressedOffset = Long.fromValue(compressedMessage.offset)
const lastMessageOffset = Long.fromValue(messages[messages.length - 1].offset)
const baseOffset = compressedOffset - lastMessageOffset
for (const message of messages) {
message.offset = Long.fromValue(message.offset)
.add(baseOffset)
.toString()
}
}
return messages
}
const EntryDecoder = decoder => {
if (!decoder.canReadInt64()) {
throw new KafkaJSPartialMessageError(
`Tried to decode a partial message: There isn't enough bytes to read the offset`
)
}
const offset = decoder.readInt64().toString()
if (!decoder.canReadInt32()) {
throw new KafkaJSPartialMessageError(
`Tried to decode a partial message: There isn't enough bytes to read the message size`
)
}
const size = decoder.readInt32()
return MessageDecoder(offset, size, decoder)
}

41
node_modules/kafkajs/src/protocol/messageSet/index.js generated vendored Normal file
View file

@ -0,0 +1,41 @@
const Encoder = require('../encoder')
const MessageProtocol = require('../message')
const { Types } = require('../message/compression')
/**
* MessageSet => [Offset MessageSize Message]
* Offset => int64
* MessageSize => int32
* Message => Bytes
*/
/**
* [
* { key: "<value>", value: "<value>" },
* { key: "<value>", value: "<value>" },
* ]
*/
module.exports = ({ messageVersion = 0, compression, entries }) => {
const isCompressed = compression !== Types.None
const Message = MessageProtocol({ version: messageVersion })
const encoder = new Encoder()
// Messages in a message set are __not__ encoded as an array.
// They are written in sequence.
// https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-Messagesets
entries.forEach((entry, i) => {
const message = Message(entry)
// This is the offset used in kafka as the log sequence number.
// When the producer is sending non compressed messages, it can set the offsets to anything
// When the producer is sending compressed messages, to avoid server side recompression, each compressed message
// should have offset starting from 0 and increasing by one for each inner message in the compressed message
encoder.writeInt64(isCompressed ? i : -1)
encoder.writeInt32(message.size())
encoder.writeEncoder(message)
})
return encoder
}

85
node_modules/kafkajs/src/protocol/recordBatch/crc32C/crc32C.js generated vendored Normal file
View file

@ -0,0 +1,85 @@
/**
* A javascript implementation of the CRC32 checksum that uses
* the CRC32-C polynomial, the same polynomial used by iSCSI
*
* also known as CRC32 Castagnoli
* based on: https://github.com/ashi009/node-fast-crc32c/blob/master/impls/js_crc32c.js
*/
const crc32C = buffer => {
let crc = 0 ^ -1
for (let i = 0; i < buffer.length; i++) {
crc = T[(crc ^ buffer[i]) & 0xff] ^ (crc >>> 8)
}
return (crc ^ -1) >>> 0
}
module.exports = crc32C
// prettier-ignore
var T = new Int32Array([
0x00000000, 0xf26b8303, 0xe13b70f7, 0x1350f3f4,
0xc79a971f, 0x35f1141c, 0x26a1e7e8, 0xd4ca64eb,
0x8ad958cf, 0x78b2dbcc, 0x6be22838, 0x9989ab3b,
0x4d43cfd0, 0xbf284cd3, 0xac78bf27, 0x5e133c24,
0x105ec76f, 0xe235446c, 0xf165b798, 0x030e349b,
0xd7c45070, 0x25afd373, 0x36ff2087, 0xc494a384,
0x9a879fa0, 0x68ec1ca3, 0x7bbcef57, 0x89d76c54,
0x5d1d08bf, 0xaf768bbc, 0xbc267848, 0x4e4dfb4b,
0x20bd8ede, 0xd2d60ddd, 0xc186fe29, 0x33ed7d2a,
0xe72719c1, 0x154c9ac2, 0x061c6936, 0xf477ea35,
0xaa64d611, 0x580f5512, 0x4b5fa6e6, 0xb93425e5,
0x6dfe410e, 0x9f95c20d, 0x8cc531f9, 0x7eaeb2fa,
0x30e349b1, 0xc288cab2, 0xd1d83946, 0x23b3ba45,
0xf779deae, 0x05125dad, 0x1642ae59, 0xe4292d5a,
0xba3a117e, 0x4851927d, 0x5b016189, 0xa96ae28a,
0x7da08661, 0x8fcb0562, 0x9c9bf696, 0x6ef07595,
0x417b1dbc, 0xb3109ebf, 0xa0406d4b, 0x522bee48,
0x86e18aa3, 0x748a09a0, 0x67dafa54, 0x95b17957,
0xcba24573, 0x39c9c670, 0x2a993584, 0xd8f2b687,
0x0c38d26c, 0xfe53516f, 0xed03a29b, 0x1f682198,
0x5125dad3, 0xa34e59d0, 0xb01eaa24, 0x42752927,
0x96bf4dcc, 0x64d4cecf, 0x77843d3b, 0x85efbe38,
0xdbfc821c, 0x2997011f, 0x3ac7f2eb, 0xc8ac71e8,
0x1c661503, 0xee0d9600, 0xfd5d65f4, 0x0f36e6f7,
0x61c69362, 0x93ad1061, 0x80fde395, 0x72966096,
0xa65c047d, 0x5437877e, 0x4767748a, 0xb50cf789,
0xeb1fcbad, 0x197448ae, 0x0a24bb5a, 0xf84f3859,
0x2c855cb2, 0xdeeedfb1, 0xcdbe2c45, 0x3fd5af46,
0x7198540d, 0x83f3d70e, 0x90a324fa, 0x62c8a7f9,
0xb602c312, 0x44694011, 0x5739b3e5, 0xa55230e6,
0xfb410cc2, 0x092a8fc1, 0x1a7a7c35, 0xe811ff36,
0x3cdb9bdd, 0xceb018de, 0xdde0eb2a, 0x2f8b6829,
0x82f63b78, 0x709db87b, 0x63cd4b8f, 0x91a6c88c,
0x456cac67, 0xb7072f64, 0xa457dc90, 0x563c5f93,
0x082f63b7, 0xfa44e0b4, 0xe9141340, 0x1b7f9043,
0xcfb5f4a8, 0x3dde77ab, 0x2e8e845f, 0xdce5075c,
0x92a8fc17, 0x60c37f14, 0x73938ce0, 0x81f80fe3,
0x55326b08, 0xa759e80b, 0xb4091bff, 0x466298fc,
0x1871a4d8, 0xea1a27db, 0xf94ad42f, 0x0b21572c,
0xdfeb33c7, 0x2d80b0c4, 0x3ed04330, 0xccbbc033,
0xa24bb5a6, 0x502036a5, 0x4370c551, 0xb11b4652,
0x65d122b9, 0x97baa1ba, 0x84ea524e, 0x7681d14d,
0x2892ed69, 0xdaf96e6a, 0xc9a99d9e, 0x3bc21e9d,
0xef087a76, 0x1d63f975, 0x0e330a81, 0xfc588982,
0xb21572c9, 0x407ef1ca, 0x532e023e, 0xa145813d,
0x758fe5d6, 0x87e466d5, 0x94b49521, 0x66df1622,
0x38cc2a06, 0xcaa7a905, 0xd9f75af1, 0x2b9cd9f2,
0xff56bd19, 0x0d3d3e1a, 0x1e6dcdee, 0xec064eed,
0xc38d26c4, 0x31e6a5c7, 0x22b65633, 0xd0ddd530,
0x0417b1db, 0xf67c32d8, 0xe52cc12c, 0x1747422f,
0x49547e0b, 0xbb3ffd08, 0xa86f0efc, 0x5a048dff,
0x8ecee914, 0x7ca56a17, 0x6ff599e3, 0x9d9e1ae0,
0xd3d3e1ab, 0x21b862a8, 0x32e8915c, 0xc083125f,
0x144976b4, 0xe622f5b7, 0xf5720643, 0x07198540,
0x590ab964, 0xab613a67, 0xb831c993, 0x4a5a4a90,
0x9e902e7b, 0x6cfbad78, 0x7fab5e8c, 0x8dc0dd8f,
0xe330a81a, 0x115b2b19, 0x020bd8ed, 0xf0605bee,
0x24aa3f05, 0xd6c1bc06, 0xc5914ff2, 0x37faccf1,
0x69e9f0d5, 0x9b8273d6, 0x88d28022, 0x7ab90321,
0xae7367ca, 0x5c18e4c9, 0x4f48173d, 0xbd23943e,
0xf36e6f75, 0x0105ec76, 0x12551f82, 0xe03e9c81,
0x34f4f86a, 0xc69f7b69, 0xd5cf889d, 0x27a40b9e,
0x79b737ba, 0x8bdcb4b9, 0x988c474d, 0x6ae7c44e,
0xbe2da0a5, 0x4c4623a6, 0x5f16d052, 0xad7d5351
]);

Some files were not shown because too many files have changed in this diff Show more