Back to catalog
Human reviewedData & analysis

Skill dossier

spark-engineer

Source declaration

Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.

The author's original description, preserved in English. It describes intent, not verified performance.

Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.

datareferences

Research summary

Human reviewed

Skill Atlas editorial review · editorial-preview-1.0.0

Guides Apache Spark development: DataFrame pipelines, partitioning, joins, skew handling, caching and structured streaming, with Spark UI validation.

Review scope

Deterministic static analysis of collected SKILL.md and supporting text/code. Machine-assisted candidates and findings await human editorial review; no dynamic behavior or effect is assessed.

Observable capabilities

  • Analyzes data volume, transformations, latency needs and cluster resources.
  • Designs the pipeline, choosing DataFrame versus RDD and broadcast opportunities.
  • Tunes shuffle partitions, skew and joins, and defines explicit schemas.
  • Validates partition counts and checks the Spark UI for spill before finishing.

Limitations & scope

  • Cannot observe the Spark UI itself; the user must supply metrics.
  • Numeric guidance (200-1000 partitions per core, 200MB broadcast) is stated without sources.
  • This review did not run any Spark code.

Relevant for

  • Data engineers building ETL pipelines
  • Teams debugging slow Spark jobs
  • Engineers writing structured streaming

What ships with the skill

The file inventory comes from the collected source at the recorded commit. Paths and sizes are structural observations, not a measure of quality.

files
6
Markdown files
6
scripts
0
references
5
Size
86.1 KB
Structural complexity
Low

File inventory

PathTypeBytes
skills/spark-engineer/references/partitioning-caching.mdReference14.5 KB
skills/spark-engineer/references/performance-tuning.mdReference15.5 KB
skills/spark-engineer/references/rdd-operations.mdReference16 KB
skills/spark-engineer/references/spark-sql-dataframes.mdReference13.5 KB
skills/spark-engineer/references/streaming-patterns.mdReference20 KB
skills/spark-engineer/SKILL.mdInstruction6.6 KB

SKILL.md frontmatter

{
  "name": "spark-engineer",
  "description": "Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines, or big data workloads. Invoke to write DataFrame transformations, optimize Spark SQL queries, implement RDD pipelines, tune shuffle operations, configure executor memory, process .parquet files, handle data partitioning, or build structured streaming analytics.",
  "license": "MIT",
  "metadata": {
    "author": "https://github.com/Jeffallan",
    "company": "https://synergetic.solutions",
    "version": "1.1.0",
    "domain": "data-ml",
    "triggers": "Apache Spark, PySpark, Spark SQL, distributed computing, big data, DataFrame API, RDD, Spark Streaming, structured streaming, data partitioning, Spark performance, cluster computing, data processing pipeline",
    "role": "expert",
    "scope": "implementation",
    "output-format": "code",
    "related-skills": "python-pro, sql-pro, devops-engineer"
  }
}

A reading of the operating sequence

This workflow reconstructs the apparent sequence from source material. Unless controlled evaluation is explicitly recorded, it is provisional—not an observed trace.

  1. 01
    Input

    Analyze requirements

    Understand data volume, transformations, latency requirements and cluster resources.

  2. 02
    Action

    Implement pipeline

    Write Spark code with optimized transformations, appropriate caching and error handling.

  3. 03
    Decision

    Validate or re-optimize

    Check Spark UI for spill; if spill or skew is found, return to the optimize step.

  4. 04
    Output

    Deliver Spark solution

    Provide code, configuration recommendations, partitioning strategy and monitoring guidance.

Potential risk surfaces

Static findings identify points to inspect before installation. They do not establish that a skill is safe, malicious, or effective.

Items warrant review

50 deterministic lexical finding(s) require review across the collected text. Findings can occur in documentation and do not prove execution or exploitability; binary-only files were not scanned.

Network accessObserved
File writesObserved
Credential useNot observed
Note

File modification is described or implemented

46 occurrences

This line contains a possible file-write or redirection operation. Check destination paths and overwrite behavior; the static review did not write any file. This rule matched 46 location(s) in the collected text; 3 representative location(s) are cited. Matching text may be documentation, and the analyzer did not execute it.

Medium

Network access is mentioned

4 occurrences

A URL or network client appears in this source line. Review destinations, data sent, and trust boundaries; static text cannot establish actual network activity. This rule matched 4 location(s) in the collected text; 3 representative location(s) are cited. Matching text may be documentation, and the analyzer did not execute it.

Editorial security note

Check storage paths and overwrite modes in generated jobs, since the example writes with mode("overwrite") to a bucket.

Where context can be spent

Discovery, activation, and resource layers have different meanings. Exact values appear only when the official token-counting method was run with a named model.

Exact token counts unavailable

All token values, including per-file inventory, are null until the pipeline performs an official exact count with an explicit model.

No token measurement was recorded with the required official method. This dossier does not substitute character counts, file sizes, or estimates.

Controlled evaluation

No controlled evaluation

This skill has only the evidence identified above. No task success, token efficiency, latency, or quality advantage is claimed.

Not tested

Use with the source in view

Install command
git clone https://github.com/Jeffallan/claude-skills.git cd claude-skills # Copy this skill directory into one of: # ~/.claude/skills/spark-engineer # .claude/skills/spark-engineer
  1. Open the pinned source at https://github.com/Jeffallan/claude-skills/blob/67c1961b180dc92cf08983344627109e983e8f51/skills/spark-engineer/SKILL.md.
  2. Review the license, prerequisites and all bundled resources.
  3. Install the skill into a personal or project-level Claude Code skills directory.
  4. Restart or refresh Claude Code so the skill can be discovered.
Verify installation

Check that SKILL.md for spark-engineer is discoverable at the destination, then ask Claude Code to use it on a small test task.

Prerequisites

  • Cannot observe the Spark UI itself; the user must supply metrics.
  • Numeric guidance (200-1000 partitions per core, 200MB broadcast) is stated without sources.
  • This review did not run any Spark code.

Example prompts

Representative task

Use the spark-engineer skill to optimize this PySpark job that joins a 2 TB events table with a small product dimension and groups by user_id. Check for skew, set an explicit schema and recommend shuffle partitions.

Inspect before use

Read the spark-engineer skill files at https://github.com/Jeffallan/claude-skills/blob/67c1961b180dc92cf08983344627109e983e8f51/skills/spark-engineer/SKILL.md. Explain its workflow, required tools, permissions and the main risks before I decide whether to install it.

A decision, with its limits attached

Strengths

  • Concrete patterns for broadcast joins, salting and caching.
  • Validation loop returns to optimization if spill or skew is found.

Considerations

  • Advice depends on cluster and data specifics.
  • Example S3 paths are placeholders.
Research conclusion

A practical Spark tuning guide whose numbers are starting points to measure against your own workload.

Traceable by construction

The dossier is a time-bound reading of a public artifact, not an endorsement. Use the source link to inspect the exact upstream material.

Jeffallanclaude-skillsskills/spark-engineer
Source type
Community
Commit
67c1961b180dc92cf08983344627109e983e8f51
Collected
20 Nov 2026
Last checked
20 Nov 2026
Upstream license
MIT
Repository stars
12,464
Content basis

Deterministic lexical parsing, YAML frontmatter validation, file inventory, source-described workflow extraction, and heuristic provisional classification of the supplied files only. Generated by machine; no LLM, execution, dynamic testing, or human editorial approval.