sfn-glue-sam: Replace deprecated Python shell 3.6 with 3.9#3130
sfn-glue-sam: Replace deprecated Python shell 3.6 with 3.9#3130kakakakakku wants to merge 3 commits into
Conversation
| MaxConcurrentRuns: 1 | ||
| DefaultArguments: | ||
| "--extra-py-files": !Sub "s3://${ArtifactsBucket}/glue/libs/awswrangler-2.12.0-py3-none-any.whl" | ||
| "--library-set": "analytics" |
There was a problem hiding this comment.
note: The bundled awswrangler-2.12.0 whl is from 2021 and there's no real need to bundle it individually anymore. We can use --library-set: analytics option 👍
See https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html
| "--extra-py-files": !Sub "s3://${ArtifactsBucket}/glue/libs/awswrangler-2.12.0-py3-none-any.whl" | ||
| "--library-set": "analytics" | ||
| "--config_path": !Sub "${ETLParametersPath}" | ||
| GlueVersion: "1.0" |
There was a problem hiding this comment.
note: GlueVersion is ignored for Python shell jobs.
You don't need to specify the version of AWS Glue since the parameter --glue-version doesn't apply for AWS Glue shell jobs. Any version specified will be ignored.
https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html
There was a problem hiding this comment.
| The SAM template deploys: | ||
| * A Step Functions State Machine | ||
| * An EventBridge rule that triggers the Step Functions every 2 days | ||
| * An EventBridge rule that triggers the Step Functions every 2 days (disabled by default) |
There was a problem hiding this comment.
note: The schedule is Enabled: False in stack.yaml, so it doesn't trigger automatically after deploy.
| 1. Copy glue job script and libs: | ||
| 1. Manually create two Amazon S3 buckets: | ||
| - `ARTIFACTS_BUCKET` — stores the AWS Glue job script. | ||
| - `DATA_BUCKET` — stores the source data for the AWS Glue job. |
There was a problem hiding this comment.
note: These two S3 buckets are not created by the stack. They need to exist before deployment. The original README didn't mention this step at all, so first-time users would likely hit a deploy error without knowing why. Added explicit creation commands with a short note on each bucket's role😀
| Command: | ||
| Name: pythonshell | ||
| PythonVersion: "3" | ||
| PythonVersion: "3.9" |
There was a problem hiding this comment.
3 participants
Issue #, if available:
N/A
Description of changes:
Hi😀 Thanks for the useful patterns!
This pattern was last updated about 5 years ago and currently doesn't work because Python shell 3.6 reached end of support on March 1, 2026. I made the following updates to bring it back to a working state.
Check
sam deploycompleted successfully and works good.$ aws s3 mb s3://kakakakakku-sfn-glue-sam-artifacts make_bucket: kakakakakku-sfn-glue-sam-artifacts $ aws s3 mb s3://kakakakakku-sfn-glue-sam-data make_bucket: kakakakakku-sfn-glue-sam-data $ aws s3 cp code/glue s3://kakakakakku-sfn-glue-sam-artifacts/glue --recursive $ sam build --template code/cloudformation/stack.yaml $ sam deploy --guided CloudFormation outputs from deployed stack ----------------------------------------------------------------------------------------------------------------------------- Outputs ----------------------------------------------------------------------------------------------------------------------------- Key GlueJobName Description - Value Feature Engineering Key ETLStepFunctionsARN Description - Value arn:aws:states:ap-northeast-1:000000000000:stateMachine:ETLStepFunctions-9pMndnlymAiz ----------------------------------------------------------------------------------------------------------------------------- Successfully created/updated stack - sfn-glue-sam in ap-northeast-1Thank you😀
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.