{"owner":"apache","repo":"hadoop","hasSkills":true,"totalSkillsCount":8,"totalTokensCount":55820,"categories":["anthropic-skill","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":["hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/encryption.md","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committer_architecture.md","hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md","hadoop-tools/hadoop-resourceestimator/README.md","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md"],"skills":{"hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md":"\n\n# Working with IAM Assumed Roles\n\n\n\nAWS [\"IAM Assumed Roles\"](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)\nallows applications to change the AWS role with which to authenticate with AWS services.\nThe assumed roles can have different rights from the main user login.\n\nThe S3A connector supports assumed roles for authentication with AWS.\nA full set of login credentials must be provided, which will be used\nto obtain the assumed role and refresh it regularly.\nBy using per-filesystem configuration, it is possible to use different\nassumed roles for different buckets.\n\n*IAM Assumed Roles are unlikely to be supported by third-party systems\nsupporting the S3 APIs.*\n\n## Using IAM Assumed Roles\n\n### Before You Begin\n\nThis document assumes you know about IAM Assumed roles, what they\nare, how to configure their policies, etc.\n\n* You need a role to assume, and know its \"ARN\".\n* You need a pair of long-lived IAM User credentials, not the root account set.\n* Have the AWS CLI installed, and test that it works there.\n* Give the role access to S3, and, if using S3Guard, to DynamoDB.\n* For working with data encrypted with SSE-KMS, the role must\nhave access to the appropriate KMS keys.\n\nTrying to learn how IAM Assumed Roles work by debugging stack traces from\nthe S3A client is \"suboptimal\".\n\n### How the S3A connector support IAM Assumed Roles.\n\nTo use assumed roles, the client must be configured to use the\n*Assumed Role Credential Provider*, `org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider`,\nin the configuration option `fs.s3a.aws.credentials.provider`.\n\nThis AWS Credential provider will read in the `fs.s3a.assumed.role` options needed to connect to the\nSecurity Token Service [Assumed Role API](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html),\nfirst authenticating with the full credentials, then assuming the specific role\nspecified. It will then refresh this login at the configured rate of\n`fs.s3a.assumed.role.session.duration`\n\nTo authenticate with the [AWS STS service](https://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html)\nboth for the initial credential retrieval\nand for background refreshes, a different credential provider must be\ncreated, one which uses long-lived credentials (secret keys, environment variables).\nShort lived credentials (e.g other session tokens, EC2 instance credentials) cannot be used.\n\nA list of providers can be set in `s.s3a.assumed.role.credentials.provider`;\nif unset the standard `BasicAWSCredentialsProvider` credential provider is used,\nwhich uses `fs.s3a.access.key` and `fs.s3a.secret.key`.\n\nNote: although you can list other AWS credential providers in to the\nAssumed Role Credential Provider, it can only cause confusion.\n\n### Configuring Assumed Roles\n\nTo use assumed roles, the S3A client credentials provider must be set to\nthe `AssumedRoleCredentialProvider`, and `fs.s3a.assumed.role.arn` to\nthe previously created ARN.\n\n```xml\n\n fs.s3a.aws.credentials.provider\n org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n\n\n\n fs.s3a.assumed.role.arn\n arn:aws:iam::90066806600238:role/s3-restricted\n\n```\n\nThe STS service itself needs the caller to be authenticated, *which can\nonly be done with a set of long-lived credentials*.\nThis means the normal `fs.s3a.access.key` and `fs.s3a.secret.key`\npair, environment variables, or some other supplier of long-lived secrets.\n\nThe default is the `fs.s3a.access.key` and `fs.s3a.secret.key` pair.\nIf you wish to use a different authentication mechanism, set it in the property\n`fs.s3a.assumed.role.credentials.provider`.\n\n```xml\n\n fs.s3a.assumed.role.credentials.provider\n org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\n\n```\n\nRequirements for long-lived credentials notwithstanding, this option takes the\nsame values as `fs.s3a.aws.credentials.provider`.\n\nThe safest way to manage AWS secrets is via\n[Hadoop Credential Providers](index.html#hadoop_credential_providers).\n\n### Assumed Role Configuration Options\n\nHere are the full set of configuration options.\n\n```xml\n\n fs.s3a.assumed.role.arn\n \n \n AWS ARN for the role to be assumed.\n Required if the fs.s3a.aws.credentials.provider contains\n org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider\n \n\n\n\n fs.s3a.assumed.role.session.name\n \n \n Session name for the assumed role, must be valid characters according to\n the AWS APIs.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n If not set, one is generated from the current Hadoop/Kerberos username.\n \n\n\n\n fs.s3a.assumed.role.policy\n \n \n JSON policy to apply to the role.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.session.duration\n 30m\n \n Duration of assumed roles before a refresh is attempted.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n Range: 15m to 1h\n \n\n\n\n fs.s3a.assumed.role.sts.endpoint\n \n \n AWS Security Token Service Endpoint. If unset, uses the default endpoint.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.sts.endpoint.region\n us-west-1\n \n AWS Security Token Service Endpoint's region;\n Needed if fs.s3a.assumed.role.sts.endpoint points to an endpoint\n other than the default one and the v4 signature is used.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.credentials.provider\n org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\n \n List of credential providers to authenticate with the STS endpoint and\n retrieve short-lived role credentials.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n If unset, uses \"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\".\n \n\n```\n\n## Restricting S3A operations through AWS Policies\n\nThe S3A client needs to be granted specific permissions in order\nto work with a bucket.\nHere is a non-normative list of the permissions which must be granted\nfor FileSystem operations to work.\n\n*Disclaimer* The specific set of actions which the S3A connector needs\nwill change over time.\n\nAs more operations are added to the S3A connector, and as the\nmeans by which existing operations are implemented change, the\nAWS actions which are required by the client will change.\n\nThese lists represent the minimum actions to which the client's principal\nmust have in order to work with a bucket.\n\n\n### Read Access Permissions\n\nPermissions which must be granted when reading from a bucket:\n\n\n```\ns3:Get*\ns3:ListBucket\n```\n\nWhen using S3Guard, the client needs the appropriate\nDynamoDB access permissions\n\nTo use SSE-KMS encryption, the client needs the\nSSE-KMS Permissions to access the\nKMS key(s).\n\n### Write Access Permissions\n\nThese permissions must all be granted for write access:\n\n```\ns3:Get*\ns3:Delete*\ns3:Put*\ns3:ListBucket\ns3:ListBucketMultipartUploads\ns3:AbortMultipartUpload\n```\n\n### SSE-KMS Permissions\n\nWhen to read data encrypted using SSE-KMS, the client must have\n `kms:Decrypt` permission for the specific key a file was encrypted with.\n\n```\nkms:Decrypt\n```\n\nTo write data using SSE-KMS, the client must have all the following permissions.\n\n```\nkms:Decrypt\nkms:GenerateDataKey\n```\n\nThis includes renaming: renamed files are encrypted with the encryption key\nof the current S3A client; it must decrypt the source file first.\n\nIf the caller doesn't have these permissions, the operation will fail with an\n`AccessDeniedException`: the S3 Store does not provide the specifics of\nthe cause of the failure.\n\n### S3Guard Permissions\n\nTo use S3Guard, all clients must have a subset of the\n[AWS DynamoDB Permissions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/api-permissions-reference.html).\n\nTo work with buckets protected with S3Guard, the client must have\nall the following rights on the DynamoDB Table used to protect that bucket.\n\n```\ndynamodb:BatchGetItem\ndynamodb:BatchWriteItem\ndynamodb:DeleteItem\ndynamodb:DescribeTable\ndynamodb:GetItem\ndynamodb:PutItem\ndynamodb:Query\ndynamodb:UpdateItem\n```\n\nThis is true, *even if the client only has read access to the data*.\n\nFor the `hadoop s3guard` table management commands, _extra_ permissions are required:\n\n```\ndynamodb:CreateTable\ndynamodb:DescribeLimits\ndynamodb:DeleteTable\ndynamodb:Scan\ndynamodb:TagResource\ndynamodb:UntagResource\ndynamodb:UpdateTable\n```\n\nWithout these permissions, tables cannot be created, destroyed or have their IO capacity\nchanged through the `s3guard set-capacity` call.\nThe `dynamodb:Scan` permission is needed for `s3guard prune`\n\nThe `dynamodb:CreateTable` permission is needed by a client it tries to\ncreate the DynamoDB table on startup, that is\n`fs.s3a.s3guard.ddb.table.create` is `true` and the table does not already exist.\n\n### Mixed Permissions in a single S3 Bucket\n\nMixing permissions down the \"directory tree\" is limited\nonly to the extent of supporting writeable directories under\nread-only parent paths.\n\n*Disclaimer:* When a client lacks write access up the entire\ndirectory tree, there are no guarantees of consistent filesystem\nviews or operations.\n\nParticular troublespots are \"directory markers\" and\nfailures of non-atomic operations, particularly `rename()` and `delete()`.\n\nA directory marker such as `/users/` will not be deleted if the user `alice`\ncreates a directory `/users/alice` *and* she only has access to `/users/alice`.\n\nWhen a path or directory is deleted, the parent directory may not exist afterwards.\nIn the example above, if `alice` deletes `/users/alice` and there are no\nother entries under `/users/alice`, then the directory marker `/users/` cannot\nbe created. The directory `/users` will not exist in listings,\n`getFileStatus(\"/users\")` or similar.\n\nRename will fail if it cannot delete the items it has just copied, that is\n`rename(read-only-source, writeable-dest)` will fail —but only after\nperforming the COPY of the data.\nEven though the operation failed, for a single file copy, the destination\nfile will exist.\nFor a directory copy, only a partial copy of the source data may take place\nbefore the permission failure is raised.\n\n\n*S3Guard*: if [S3Guard](s3guard.html) is used to manage the directory listings,\nthen after partial failures of rename/copy the DynamoDB tables can get out of sync.\n\n### Example: Read access to the base, R/W to the path underneath\n\nThis example has the base bucket read only, and a directory underneath,\n`/users/alice/` granted full R/W access.\n\n```json\n{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"4\",\n \"Effect\" : \"Allow\",\n \"Action\" : [\n \"s3:ListBucket\",\n \"s3:ListBucketMultipartUploads\",\n \"s3:Get*\"\n ],\n \"Resource\" : \"arn:aws:s3:::example-bucket/*\"\n }, {\n \"Sid\" : \"5\",\n \"Effect\" : \"Allow\",\n \"Action\" : [\n \"s3:Get*\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:AbortMultipartUpload\",\n \"s3:ListMultipartUploadParts\" ],\n \"Resource\" : [\n \"arn:aws:s3:::example-bucket/users/alice/*\",\n \"arn:aws:s3:::example-bucket/users/alice\",\n \"arn:aws:s3:::example-bucket/users/alice/\"\n ]\n } ]\n}\n```\n\nNote how three resources are provided to represent the path `/users/alice`\n\n| Path | Matches |\n|-------|----------|\n| `/users/alice` | Any file `alice` created under `/users` |\n| `/users/alice/` | The directory marker `alice/` created under `/users` |\n| `/users/alice/*` | All files and directories under the path `/users/alice` |\n\nNote that the resource `arn:aws:s3:::example-bucket/users/alice*` cannot\nbe used to refer to all of these paths, because it would also cover\nadjacent paths like `/users/alice2` and `/users/alicebob`.\n\n\n## Troubleshooting Assumed Roles\n\n1. Make sure the role works and the user trying to enter it can do so from AWS\nthe command line before trying to use the S3A client.\n1. Try to access the S3 bucket with reads and writes from the AWS CLI.\n1. With the Hadoop configuration set too use the role,\n try to read data from the `hadoop fs` CLI:\n`hadoop fs -ls -p s3a://bucket/`\n1. With the hadoop CLI, try to create a new directory with a request such as\n`hadoop fs -mkdirs -p s3a://bucket/path/p1/`\n\n\n### IOException: \"Unset property fs.s3a.assumed.role.arn\"\n\nThe Assumed Role Credential Provider is enabled, but `fs.s3a.assumed.role.arn` is unset.\n\n```\njava.io.IOException: Unset property fs.s3a.assumed.role.arn\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:76)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n```\n\n### \"Not authorized to perform sts:AssumeRole\"\n\nThis can arise if the role ARN set in `fs.s3a.assumed.role.arn` is invalid\nor one to which the caller has no access.\n\n```\njava.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Not authorized to perform sts:AssumeRole (Service: AWSSecurityTokenService; Status Code: 403;\n Error Code: AccessDenied; Request ID: aad4e59a-f4b0-11e7-8c78-f36aaa9457f6):AccessDenied\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n```\n\n### \"Roles may not be assumed by root accounts\"\n\nYou can't assume a role with the root account of an AWS account;\nyou need to create a new user and give it the permission to change into\nthe role.\n\n```\njava.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;\n Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2):\n No AWS Credentials provided by AssumedRoleCredentialProvider :\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService;\n Status Code: 403; Error Code: AccessDenied; Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n ... 22 more\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;\n Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n```\n\n### \"Assume Role session duration should be in the range of 15min - 1Hr\"\n\nThe value of `fs.s3a.assumed.role.session.duration` is out of range.\n\n```\njava.lang.IllegalArgumentException: Assume Role session duration should be in the range of 15min\n- 1Hr\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$Builder.withRoleSessionDurationSeconds(STSAssumeRoleSessionCredentialsProvider.java:437)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:86)\n```\n\n\n### `MalformedPolicyDocumentException` \"The policy is not in the valid JSON format\"\n\n\nThe policy set in `fs.s3a.assumed.role.policy` is not valid according to the\nAWS specification of Role Policies.\n\n```\nrg.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n The policy is not in the valid JSON format. (Service: AWSSecurityTokenService; Status Code: 400;\n Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c):\n MalformedPolicyDocument: The policy is not in the valid JSON format.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\nCaused by: com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n The policy is not in the valid JSON format.\n (Service: AWSSecurityTokenService; Status Code: 400;\n Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 19 more\n```\n\n### `MalformedPolicyDocumentException` \"Syntax errors in policy\"\n\nThe policy set in `fs.s3a.assumed.role.policy` is not valid JSON.\n\n```\norg.apache.hadoop.fs.s3a.AWSBadRequestException:\nInstantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n Syntax errors in policy. (Service: AWSSecurityTokenService;\n Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45):\n MalformedPolicyDocument: Syntax errors in policy.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 19 more\n```\n\n### `IOException`: \"AssumedRoleCredentialProvider cannot be in fs.s3a.assumed.role.credentials.provider\"\n\nYou can't use the Assumed Role Credential Provider as the provider in\n`fs.s3a.assumed.role.credentials.provider`.\n\n```\njava.io.IOException: AssumedRoleCredentialProvider cannot be in fs.s3a.assumed.role.credentials.provider\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:86)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n```\n\n### `AWSBadRequestException`: \"not a valid key=value pair\"\n\n\nThere's an space or other typo in the `fs.s3a.access.key` or `fs.s3a.secret.key` values used for the\ninner authentication which is breaking signature creation.\n\n```\n org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header:\n 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code:\n IncompleteSignature; Request ID: c4a8841d-f556-11e7-99f9-af005a829416):IncompleteSignature:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign)\n in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date,\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature;\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign)\n in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date,\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature;\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 25 more\n```\n\n### `AccessDeniedException/InvalidClientTokenId`: \"The security token included in the request is invalid\"\n\nThe credentials used to authenticate with the AWS Security Token Service are invalid.\n\n```\n[ERROR] Failures:\n[ERROR] java.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n The security token included in the request is invalid.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId;\n Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7):InvalidClientTokenId\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\nThe security token included in the request is invalid.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId;\n Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 25 more\n```\n\n### `AWSSecurityTokenServiceExceptiond`: \"Member must satisfy regular expression pattern: `[\\w+=,.@-]*`\"\n\n\nThe session name, as set in `fs.s3a.assumed.role.session.name` must match the wildcard `[\\w+=,.@-]*`.\n\nIf the property is unset, it is extracted from the current username and then sanitized to\nmatch these constraints.\nIf set explicitly, it must be valid.\n\n```\norg.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint: Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n Request ID: 7c437acb-f55d-11e7-9ad8-3b5e4f701c20):ValidationError:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint: Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint:\n Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:135)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:124)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 26 more\n```\n\n\n### `java.nio.file.AccessDeniedException` within a FileSystem API call\n\nIf an operation fails with an `AccessDeniedException`, then the role does not have\nthe permission for the S3 Operation invoked during the call.\n\n```\njava.nio.file.AccessDeniedException: s3a://bucket/readonlyDir:\n rename(s3a://bucket/readonlyDir, s3a://bucket/renameDest)\n on s3a://bucket/readonlyDir:\n com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied\n (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1;\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=),\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=:AccessDenied\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:216)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:143)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:853)\n ...\nCaused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied\n (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1;\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=),\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229)\n at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176)\n at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2066)\n at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2052)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObject$7(S3AFileSystem.java:1338)\n at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:314)\n at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:280)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObject(S3AFileSystem.java:1334)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1657)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:1046)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:851)\n```\n\nThis is the policy restriction behaving as intended: the caller is trying to\nperform an action which is forbidden.\n\n1. If a policy has been set in `fs.s3a.assumed.role.policy` then it must declare *all*\npermissions which the caller is allowed to perform. The existing role policies\nact as an outer constraint on what the caller can perform, but are not inherited.\n\n1. If the policy for a bucket is set up with complex rules on different paths,\ncheck the path for the operation.\n\n1. The policy may have omitted one or more actions which are required.\nMake sure that all the read and write permissions are allowed for any bucket/path\nto which data is being written to, and read permissions for all\nbuckets read from.\n\nIf the bucket is using SSE-KMS to encrypt data:\n\n1. The caller must have the `kms:Decrypt` permission to read the data.\n1. The caller needs `kms:Decrypt` and `kms:GenerateDataKey`.\n\nWithout permissions, the request fails *and there is no explicit message indicating\nthat this is an encryption-key issue*.\n\n### `AccessDeniedException` + `AmazonDynamoDBException`\n\n```\njava.nio.file.AccessDeniedException: bucket1:\n com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:\n User: arn:aws:sts::980678866538:assumed-role/s3guard-test-role/test is not authorized to perform:\n dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-1:980678866538:table/bucket1\n (Service: AmazonDynamoDBv2; Status Code: 400;\n```\n\nThe caller is trying to access an S3 bucket which uses S3Guard, but the caller\nlacks the relevant DynamoDB access permissions.\n\nThe `dynamodb:DescribeTable` operation is the first one used in S3Guard to access,\nthe DynamoDB table, so it is often the first to fail. It can be a sign\nthat the role has no permissions at all to access the table named in the exception,\nor just that this specific permission has been omitted.\n\nIf the role policy requested for the assumed role didn't ask for any DynamoDB\npermissions, this is where all attempts to work with a S3Guarded bucket will\nfail. Check the value of `fs.s3a.assumed.role.policy`\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/encryption.md":"\n\n\n# Working with Encrypted S3 Data\n\n\n\n\n## Introduction\n\nThe S3A filesystem client supports Amazon S3's Server Side Encryption\nfor at-rest data encryption.\nYou should to read up on the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html)\nfor S3 Server Side Encryption for up to date information on the encryption mechanisms.\n\n\n\nWhen configuring an encryption method in the `core-site.xml`, this will apply cluster wide.\nAny new file written will be encrypted with this encryption configuration.\nWhen the S3A client reads a file, S3 will attempt to decrypt it using the mechanism\nand keys with which the file was encrypted.\n\n* It is **NOT** advised to mix and match encryption types in a bucket\n* It is much simpler and safer to encrypt with just one type and key per bucket.\n* You can use AWS bucket policies to mandate encryption rules for a bucket.\n* You can use S3A per-bucket configuration to ensure that S3A clients use encryption\npolicies consistent with the mandated rules.\n* You can use S3 Default Encryption to encrypt data without needing to\nset anything in the client.\n* Changing the encryption options on the client does not change how existing\nfiles were encrypted, except when the files are renamed.\n* For all mechanisms other than SSE-C, clients do not need any configuration\noptions set in order to read encrypted data: it is all automatically handled\nin S3 itself.\n\n## How data is encrypted\n\nAWS S3 supports server-side encryption inside the storage system itself.\nWhen an S3 client uploading data requests data to be encrypted, then an encryption key is used\nto encrypt the data as it saved to S3. It remains encrypted on S3 until deleted:\nclients cannot change the encryption attributes of an object once uploaded.\n\nThe Amazon AWS SDK also offers client-side encryption, in which all the encoding\nand decoding of data is performed on the client. This is *not* supported by\nthe S3A client.\n\nThe server-side \"SSE\" encryption is performed with symmetric AES256 encryption;\nS3 offers different mechanisms for actually defining the key to use.\n\n\nThere are four key management mechanisms, which in order of simplicity of use,\nare:\n\n* S3 Default Encryption\n* SSE-S3: an AES256 key is generated in S3, and saved alongside the data.\n* SSE-KMS: an AES256 key is generated in S3, and encrypted with a secret key provided\nby Amazon's Key Management Service, a key referenced by name in the uploading client.\n* SSE-C : the client specifies an actual base64 encoded AES-256 key to be used\nto encrypt and decrypt the data.\n\n\n## S3 Default Encryption\n\nThis feature allows the administrators of the AWS account to set the \"default\"\nencryption policy on a bucket -the encryption to use if the client does\nnot explicitly declare an encryption algorithm.\n\n[S3 Default Encryption for S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html)\n\nThis supports SSE-S3 and SSE-KMS.\n\nThere is no need to set anything up in the client: do it in the AWS console.\n\n\n## SSE-S3 Amazon S3-Managed Encryption Keys\n\nIn SSE-S3, all keys and secrets are managed inside S3. This is the simplest encryption mechanism.\nThere is no extra cost for storing data with this option.\n\n\n### Enabling SSE-S3\n\nTo write S3-SSE encrypted files, the value of\n`fs.s3a.server-side-encryption-algorithm` must be set to that of\nthe encryption mechanism used in `core-site`; currently only `AES256` is supported.\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n AES256\n\n```\n\nOnce set, all new data will be stored encrypted. There is no need to set this property when downloading data — the data will be automatically decrypted when read using\nthe Amazon S3-managed key.\n\nTo learn more, refer to\n[Protecting Data Using Server-Side Encryption with Amazon S3-Managed Encryption Keys (SSE-S3) in AWS documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html).\n\n\n### SSE-KMS: Amazon S3-KMS Managed Encryption Keys\n\n\nAmazon offers a pay-per-use key management service, [AWS KMS](https://aws.amazon.com/documentation/kms/).\nThis service can be used to encrypt data on S3 by defining \"customer master keys\", CMKs,\nwhich can be centrally managed and assigned to specific roles and IAM accounts.\n\nThe AWS KMS [can be used encrypt data on S3uploaded data](http://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html).\n\n> The AWS KMS service is **not** related to the Key Management Service built into Hadoop (*Hadoop KMS*). The *Hadoop KMS* primarily focuses on\n managing keys for *HDFS Transparent Encryption*. Similarly, HDFS encryption is unrelated to S3 data encryption.\n\nWhen uploading data encrypted with SSE-KMS, the sequence is as follows.\n\n1. The S3A client must declare a specific CMK in the property `fs.s3a.server-side-encryption.key`, or leave\nit blank to use the default configured for that region.\n\n1. The S3A client uploads all the data as normal, now including encryption information.\n\n1. The S3 service encrypts the data with a symmetric key unique to the new object.\n\n1. The S3 service retrieves the chosen CMK key from the KMS service, and, if the user has\nthe right to use it, uses it to encrypt the object-specific key.\n\n\nWhen downloading SSE-KMS encrypted data, the sequence is as follows\n\n1. The S3A client issues an HTTP GET request to read the data.\n1. S3 sees that the data was encrypted with SSE-KMS, and looks up the specific key in the KMS service\n1. If and only if the requesting user has been granted permission to use the CMS key does\nthe KMS service provide S3 with the key.\n1. As a result, S3 will only decode the data if the user has been granted access to the key.\n\n\nKMS keys can be managed by an organization's administrators in AWS, including\nhaving access permissions assigned and removed from specific users, groups, and IAM roles.\nOnly those \"principals\" with granted rights to a key may access it,\nhence only they may encrypt data with the key, *and decrypt data encrypted with it*.\nThis allows KMS to be used to provide a cryptographically secure access control mechanism for data stores on S3.\n\n\nEach KMS server is region specific, and accordingly, so is each CMK configured.\nA CMK defined in one region cannot be used with an S3 bucket in a different region.\n\n\nNotes\n\n* Callers are charged for every use of a key, both for encrypting the data in uploads\n and for decrypting it when reading it back.\n* Random-access IO on files may result in multiple GET requests of an object during a read\nsequence (especially for columnar data), so may require more than one key retrieval to process a single file,\n* The KMS service is throttled: too many requests may cause requests to fail.\n* As well as incurring charges, heavy I/O *may* reach IO limits for a customer. If those limits are reached,\nthey can be increased through the AWS console.\n\n\n### Enabling SSE-KMS\n\nTo enable SSE-KMS, the property `fs.s3a.server-side-encryption-algorithm` must be set to `SSE-KMS` in `core-site`:\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n SSE-KMS\n\n```\n\nThe ID of the specific key used to encrypt the data should also be set in the property `fs.s3a.server-side-encryption.key`:\n\n```xml\n\n fs.s3a.server-side-encryption.key\n arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n```\n\nOrganizations may define a default key in the Amazon KMS; if a default key is set,\nthen it will be used whenever SSE-KMS encryption is chosen and the value of `fs.s3a.server-side-encryption.key` is empty.\n\n### the S3A `fs.s3a.encryption.key` key only affects created files\n\nWith SSE-KMS, the S3A client option `fs.s3a.server-side-encryption.key` sets the\nkey to be used when new files are created. When reading files, this key,\nand indeed the value of `fs.s3a.server-side-encryption-algorithme` is ignored:\nS3 will attempt to retrieve the key and decrypt the file based on the create-time settings.\n\nThis means that\n\n* There's no need to configure any client simply reading data.\n* It is possible for a client to read data encrypted with one KMS key, and\nwrite it with another.\n\n\n## SSE-C: Server side encryption with a client-supplied key.\n\nIn SSE-C, the client supplies the secret key needed to read and write data.\nEvery client trying to read or write data must be configured with the same\nsecret key.\n\n\nSSE-C integration with Hadoop is still stabilizing; issues related to it are still surfacing.\nIt is already clear that SSE-C with a common key must be used exclusively within\na bucket if it is to be used at all. This is the only way to ensure that path and\ndirectory listings do not fail with \"Bad Request\" errors.\n\n### Enabling SSE-C\n\nTo use SSE-C, the configuration option `fs.s3a.server-side-encryption-algorithm`\nmust be set to `SSE-C`, and a base-64 encoding of the key placed in\n`fs.s3a.server-side-encryption.key`.\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n SSE-C\n\n\n\n fs.s3a.server-side-encryption.key\n SGVscCwgSSdtIHRyYXBwZWQgaW5zaWRlIGEgYmFzZS02NC1jb2RlYyE=\n\n```\n\nAll clients must share this same key.\n\n### The `fs.s3a.encryption.key` value is used to read and write data\n\nWith SSE-C, the S3A client option `fs.s3a.server-side-encryption.key` sets the\nkey to be used for both reading *and* writing data.\n\nWhen reading any file written with SSE-C, the same key must be set\nin the property `fs.s3a.server-side-encryption.key`.\n\nThis is unlike SSE-S3 and SSE-KMS, where the information needed to\ndecode data is kept in AWS infrastructure.\n\n\n### SSE-C Warning\n\nYou need to fully understand how SSE-C works in the S3\nenvironment before using this encryption type. Please refer to the Server Side\nEncryption documentation available from AWS. SSE-C is only recommended for\nadvanced users with advanced encryption use cases. Failure to properly manage\nencryption keys can cause data loss. Currently, the AWS S3 API(and thus S3A)\nonly supports one encryption key and cannot support decrypting objects during\nmoves under a previous key to a new destination. It is **NOT** advised to use\nmultiple encryption keys in a bucket, and is recommended to use one key per\nbucket and to not change this key. This is due to when a request is made to S3,\nthe actual encryption key must be provided to decrypt the object and access the\nmetadata. Since only one encryption key can be provided at a time, S3A will not\npass the correct encryption key to decrypt the data.\n\n\n## Encryption best practises\n\n\n### Mandate encryption through policies\n\nBecause it is up to the clients to enable encryption on new objects, all clients\nmust be correctly configured in order to guarantee that data is encrypted.\n\n\nTo mandate that all data uploaded to a bucket is encrypted,\nyou can set a [bucket policy](https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/)\ndeclaring that clients must provide encryption information with all data uploaded.\n\n\n* Mandating an encryption mechanism on newly uploaded data does not encrypt existing data; existing data will retain whatever encryption (if any) applied at the time of creation*\n\nHere is a policy to mandate `SSE-S3/AES265` encryption on all data uploaded to a bucket. This covers uploads as well as the copy operations which take place when file/directory rename operations are mimicked.\n\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Id\": \"EncryptionPolicy\",\n \"Statement\": [\n {\n \"Sid\": \"RequireEncryptionHeaderOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"Null\": {\n \"s3:x-amz-server-side-encryption\": true\n }\n }\n },\n {\n \"Sid\": \"RequireAESEncryptionOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"StringNotEquals\": {\n \"s3:x-amz-server-side-encryption\": \"AES256\"\n }\n }\n }\n ]\n}\n```\n\nTo use SSE-KMS, a different restriction must be defined:\n\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Id\": \"EncryptionPolicy\",\n \"Statement\": [\n {\n \"Sid\": \"RequireEncryptionHeaderOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"Null\": {\n \"s3:x-amz-server-side-encryption\": true\n }\n }\n },\n {\n \"Sid\": \"RequireKMSEncryptionOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"StringNotEquals\": {\n \"s3:x-amz-server-side-encryption\": \"SSE-KMS\"\n }\n }\n }\n ]\n}\n```\n\nTo use one of these policies:\n\n1. Replace `BUCKET` with the specific name of the bucket being secured.\n1. Locate the bucket in the AWS console [S3 section](https://console.aws.amazon.com/s3/home).\n1. Select the \"Permissions\" tab.\n1. Select the \"Bucket Policy\" tab in the permissions section.\n1. Paste the edited policy into the form.\n1. Save the policy.\n\n### Use S3a per-bucket configuration to control encryption settings\n\nIn an organisation which has embraced S3 encryption, different buckets inevitably have\ndifferent encryption policies, such as different keys for SSE-KMS encryption.\nIn particular, as different keys need to be named for different regions, unless\nyou rely on the administrator-managed \"default\" key for each S3 region, you\nwill need unique keys.\n\nS3A's per-bucket configuration enables this.\n\n\nHere, for example, are settings for a bucket in London, `london-stats`:\n\n\n```xml\n\n fs.s3a.bucket.london-stats.server-side-encryption-algorithm\n AES256\n\n```\n\nThis requests SSE-S; if matched with a bucket policy then all data will\nbe encrypted as it is uploaded.\n\n\nA different bucket can use a different policy\n(here SSE-KMS) and, when necessary, declare a key.\n\nHere is an example bucket in S3 Ireland, which uses SSE-KMS and\na KMS key hosted in the AWS-KMS service in the same region.\n\n\n```xml\n\n fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm\n SSE-KMS\n\n\n\n fs.s3a.bucket.ireland-dev.server-side-encryption.key\n arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n\n```\n\nAgain the appropriate bucket policy can be used to guarantee that all callers\nwill use SSE-KMS; they can even mandate the name of the key used to encrypt\nthe data, so guaranteeing that access to thee data can be read by everyone\ngranted access to that key, and nobody without access to it.\n\n\n### Use rename() to encrypt files with new keys\n\nThe encryption of an object is set when it is uploaded. If you want to encrypt\nan unencrypted file, or change the SEE-KMS key of a file, the only way to do\nso is by copying the object.\n\nHow can you do that from Hadoop? With `rename()`.\n\nThe S3A client mimics a real filesystem's' rename operation by copying all the\nsource files to the destination paths, then deleting the old ones.\n\nNote: this does not work for SSE-C, because you cannot set a different key\nfor reading as for writing, and you must supply that key for reading. There\nyou need to copy one bucket to a different bucket, one with a different key.\nUse `distCp`for this, with per-bucket encryption policies.\n\n\n## Troubleshooting Encryption\n\nThe [troubleshooting](./troubleshooting_s3a.html) document covers\nstack traces which may surface when working with encrypted data.\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md":"\n\n# Committing work to S3 with the \"S3A Committers\"\n\n\n\nThis page covers the S3A Committers, which can commit work directly\nto an S3 object store.\n\nThese committers are designed to solve a fundamental problem which\nthe standard committers of work cannot do to S3: consistent, high performance,\nand reliable commitment of output to S3.\n\nFor details on their internal design, see\n[S3A Committers: Architecture and Implementation](./committer_architecture.html).\n\n\n## Introduction: The Commit Problem\n\n\nApache Hadoop MapReduce (and behind the scenes, Apache Spark) often write\nthe output of their work to filesystems\n\nNormally, Hadoop uses the `FileOutputFormatCommitter` to manage the\npromotion of files created in a single task attempt to the final output of\na query. This is done in a way to handle failures of tasks and jobs, and to\nsupport speculative execution. It does that by listing directories and renaming\ntheir content into the final destination when tasks and then jobs are committed.\n\nThis has some key requirement of the underlying filesystem:\n\n1. When you list a directory, you see all the files which have been created in it,\nand no files which are not in it (i.e. have been deleted).\n1. When you rename a directory, it is an `O(1)` atomic transaction. No other\nprocess across the cluster may rename a file or directory to the same path.\nIf the rename fails for any reason, either the data is at the original location,\nor it is at the destination, -in which case the rename actually succeeded.\n\n**The S3 object store and the `s3a://` filesystem client cannot meet these requirements.*\n\n1. Amazon S3 has inconsistent directory listings unless S3Guard is enabled.\n1. The S3A mimics `rename()` by copying files and then deleting the originals.\nThis can fail partway through, and there is nothing to prevent any other process\nin the cluster attempting a rename at the same time.\n\nAs a result,\n\n* Files my not be listed, hence not renamed into place.\n* Deleted files may still be discovered, confusing the rename process to the point\nof failure.\n* If a rename fails, the data is left in an unknown state.\n* If more than one process attempts to commit work simultaneously, the output\ndirectory may contain the results of both processes: it is no longer an exclusive\noperation.\n*. While S3Guard may deliver the listing consistency, commit time is still\nproportional to the amount of data created. It still can't handle task failure.\n\n**Using the \"classic\" `FileOutputCommmitter` to commit work to Amazon S3 risks\nloss or corruption of generated data**\n\n\nTo address these problems there is now explicit support in the `hadop-aws`\nmodule for committing work to Amazon S3 via the S3A filesystem client,\n*the S3A Committers*\n\n\nFor safe, as well as high-performance output of work to S3,\nwe need use \"a committer\" explicitly written to work with S3, treating it as\nan object store with special features.\n\n\n### Background : Hadoop's \"Commit Protocol\"\n\nHow exactly is work written to its final destination? That is accomplished by\na \"commit protocol\" between the workers and the job manager.\n\nThis protocol is implemented in Hadoop MapReduce, with a similar but extended\nversion in Apache Spark:\n\n1. A \"Job\" is the entire query, with inputs to outputs\n1. The \"Job Manager\" is the process in charge of choreographing the execution\nof the job. It may perform some of the actual computation too.\n1. The job has \"workers\", which are processes which work the actual data\nand write the results.\n1. Workers execute \"Tasks\", which are fractions of the job, a job whose\ninput has been *partitioned* into units of work which can be executed independently.\n1. The Job Manager directs workers to execute \"tasks\", usually trying to schedule\nthe work close to the data (if the filesystem provides locality information).\n1. Workers can fail: the Job manager needs to detect this and reschedule their active tasks.\n1. Workers can also become separated from the Job Manager, a \"network partition\".\nIt is (provably) impossible for the Job Manager to distinguish a running-but-unreachable\nworker from a failed one.\n1. The output of a failed task must not be visible; this is to avoid its\ndata getting into the final output.\n1. Multiple workers can be instructed to evaluate the same partition of the work;\nthis \"speculation\" delivers speedup as it can address the \"straggler problem\".\nWhen multiple workers are working on the same data, only one worker is allowed\nto write the final output.\n1. The entire job may fail (often from the failure of the Job Manager (MR Master, Spark Driver, ...)).\n1, The network may partition, with workers isolated from each other or\nthe process managing the entire commit.\n1. Restarted jobs may recover from a failure by reusing the output of all\ncompleted tasks (MapReduce with the \"v1\" algorithm), or just by rerunning everything\n(The \"v2\" algorithm and Spark).\n\n\nWhat is \"the commit protocol\" then? It is the requirements on workers as to\nwhen their data is made visible, where, for a filesystem, \"visible\" means \"can\nbe seen in the destination directory of the query.\"\n\n* There is a destination directory of work, \"the output directory.\"\n* The final output of tasks must be in this directory *or paths underneath it*.\n* The intermediate output of a task must not be visible in the destination directory.\nThat is: they must not write directly to the destination.\n* The final output of a task *may* be visible under the destination.\n* The Job Manager makes the decision as to whether a task's data is to be \"committed\",\nbe it directly to the final directory or to some intermediate store..\n* Individual workers communicate with the Job manager to manage the commit process:\nwhether the output is to be *committed* or *aborted*\n* When a worker commits the output of a task, it somehow promotes its intermediate work to becoming\nfinal.\n* When a worker aborts a task's output, that output must not become visible\n(i.e. it is not committed).\n* Jobs themselves may be committed/aborted (the nature of \"when\" is not covered here).\n* After a Job is committed, all its work must be visible.\n* And a file `_SUCCESS` may be written to the output directory.\n* After a Job is aborted, all its intermediate data is lost.\n* Jobs may also fail. When restarted, the successor job must be able to clean up\nall the intermediate and committed work of its predecessor(s).\n* Task and Job processes measure the intervals between communications with their\nApplication Master and YARN respectively.\nWhen the interval has grown too large they must conclude\nthat the network has partitioned and that they must abort their work.\n\n\nThat's \"essentially\" it. When working with HDFS and similar filesystems,\ndirectory `rename()` is the mechanism used to commit the work of tasks and\njobs.\n* Tasks write data to task attempt directories under the directory `_temporary`\nunderneath the final destination directory.\n* When a task is committed, these files are renamed to the destination directory\n(v2 algorithm) or a job attempt directory under `_temporary` (v1 algorithm).\n* When a job is committed, for the v2 algorithm the `_SUCCESS` file is created,\nand the `_temporary` deleted.\n* For the v1 algorithm, when a job is committed, all the tasks committed under\nthe job attempt directory will have their output renamed into the destination\ndirectory.\n* The v2 algorithm recovers from failure by deleting the destination directory\nand restarting the job.\n* The v1 algorithm recovers from failure by discovering all committed tasks\nwhose output is in the job attempt directory, *and only rerunning all uncommitted tasks*.\n\n\nNone of this algorithm works safely or swiftly when working with \"raw\" AWS S3 storage:\n* Directory listing can be inconsistent: the tasks and jobs may not list all work to\nbe committed.\n* Renames go from being fast, atomic operations to slow operations which can fail partway through.\n\nThis then is the problem which the S3A committers address:\n\n*How to safely and reliably commit work to Amazon S3 or compatible object store*\n\n\n## Meet the S3A Commmitters\n\nSince Hadoop 3.1, the S3A FileSystem has been accompanied by classes\ndesigned to integrate with the Hadoop and Spark job commit protocols, classes\nwhich interact with the S3A filesystem to reliably commit work work to S3:\n*The S3A Committers*\n\nThe underlying architecture of this process is very complex, and\ncovered in [the committer architecture documentation](./committer_architecture.html).\n\nThe key concept to know of is S3's \"Multipart Upload\" mechanism. This allows\nan S3 client to write data to S3 in multiple HTTP POST requests, only completing\nthe write operation with a final POST to complete the upload; this final POST\nconsisting of a short list of the etags of the uploaded blocks.\nThis multipart upload mechanism is already automatically used when writing large\namounts of data to S3; an implementation detail of the S3A output stream.\n\nThe S3A committers make explicit use of this multipart upload (\"MPU\") mechanism:\n\n1. The individual *tasks* in a job write their data to S3 as POST operations\nwithin multipart uploads, yet do not issue the final POST to complete the upload.\n1. The multipart uploads are committed in the job commit process.\n\nThere are two different S3A committer types, *staging*\nand *magic*. The committers primarily vary in how data is written during task execution,\nhow the pending commit information is passed to the job manager, and in how\nconflict with existing files is resolved.\n\n\n| feature | staging | magic |\n|--------|---------|---|\n| task output destination | local disk | S3A *without completing the write* |\n| task commit process | upload data from disk to S3 | list all pending uploads on s3 and write details to job attempt directory |\n| task abort process | delete local disk data | list all pending uploads and abort them |\n| job commit | list & complete pending uploads | list & complete pending uploads |\n\nThe other metric is \"maturity\". There, the fact that the Staging committers\nare based on Netflix's production code counts in its favor.\n\n\n### The Staging Committer\n\nThis is based on work from Netflix. It \"stages\" data into the local filesystem.\nIt also requires the cluster to have HDFS, so that\n\nTasks write to URLs with `file://` schemas. When a task is committed,\nits files are listed, uploaded to S3 as incompleted Multipart Uploads.\nThe information needed to complete the uploads is saved to HDFS where\nit is committed through the standard \"v1\" commit algorithm.\n\nWhen the Job is committed, the Job Manager reads the lists of pending writes from its\nHDFS Job destination directory and completes those uploads.\n\nCancelling a task is straightforward: the local directory is deleted with\nits staged data. Cancelling a job is achieved by reading in the lists of\npending writes from the HDFS job attempt directory, and aborting those\nuploads. For extra safety, all outstanding multipart writes to the destination directory\nare aborted.\n\nThe staging committer comes in two slightly different forms, with slightly\ndifferent conflict resolution policies:\n\n\n* **Directory**: the entire directory tree of data is written or overwritten,\nas normal.\n\n* **Partitioned**: special handling of partitioned directory trees of the form\n`YEAR=2017/MONTH=09/DAY=19`: conflict resolution is limited to the partitions\nbeing updated.\n\n\nThe Partitioned Committer is intended to allow jobs updating a partitioned\ndirectory tree to restrict the conflict resolution to only those partition\ndirectories containing new data. It is intended for use with Apache Spark\nonly.\n\n\n## Conflict Resolution in the Staging Committers\n\nThe Staging committers offer the ability to replace the conflict policy\nof the execution engine with policy designed to work with the tree of data.\nThis is based on the experience and needs of Netflix, where efficiently adding\nnew data to an existing partitioned directory tree is a common operation.\n\n```xml\n\n fs.s3a.committer.staging.conflict-mode\n fail\n \n Staging committer conflict resolution policy: {@value}.\n Supported: fail, append, replace.\n \n\n```\n\n**replace** : when the job is committed (and not before), delete files in\ndirectories into which new data will be written.\n\n**fail**: when there are existing files in the destination, fail the job.\n\n**append**: Add new data to the directories at the destination; overwriting\nany with the same name. Reliable use requires unique names for generated files,\nwhich the committers generate\nby default.\n\nThe difference between the two staging committers are as follows:\n\nThe Directory Committer uses the entire directory tree for conflict resolution.\nIf any file exists at the destination it will fail in job setup; if the resolution\nmechanism is \"replace\" then all existing files will be deleted.\n\nThe partitioned committer calculates the partitions into which files are added,\nthe final directories in the tree, and uses that in its conflict resolution\nprocess:\n\n\n**replace** : delete all data in the destination partition before committing\nthe new files.\n\n**fail**: fail if there is data in the destination partition, ignoring the state\nof any parallel partitions.\n\n**append**: add the new data.\n\nIt's intended for use in Apache Spark Dataset operations, rather\nthan Hadoop's original MapReduce engine, and only in jobs\nwhere adding new data to an existing dataset is the desired goal.\n\nPrerequisites for successful work\n\n1. The output is written into partitions via `PARTITIONED BY` or `partitionedBy()`\ninstructions.\n2. There is no data written directly to the root path (all files there are\nignored; it's implicitly \"append\").\n\nHere's an example in Spark, assuming that `sourceDataset` is a dataset\nwhose columns include \"year\" and \"month\":\n\n```scala\nsourceDataset\n .write\n .partitionBy(\"year\", \"month\")\n .mode(SaveMode.Append)\n .opt(\"fs.s3a.committer.name\", \"partitioned\")\n .opt(\"fs.s3a.committer.staging.conflict-mode\", \"replace\")\n .format(\"orc\")\n .save(\"s3a://examples/statistics\")\n```\n\n\n### The Magic Committer\n\nThe \"Magic\" committer does its work through \"magic\" in the filesystem:\nattempts to write to specific \"magic\" paths are interpreted as writes\nto a parent directory *which are not to be completed*. When the output stream\nis closed, the information needed to complete the write is saved in the magic\ndirectory. The task committer saves the list of these to a directory for the\njob committers use, or, if aborting, lists the pending writes and aborts them.\n\nThe job committer reads in the list of pending commits, and commits them as\nthe Staging Committer does.\n\nCompared to the Staging Committer, the Magic Committer offers faster write\ntimes: output is uploaded to S3 as it is written, rather than in the\ntask commit.\n\nHowever, it has extra requirements of the filesystem\n\n1. It requires a consistent object store, which for Amazon S3,\nmeans that [S3Guard](./s3guard.html) must be enabled. For third-party stores,\nconsult the documentation.\n1. The S3A client must be configured to recognize interactions\nwith the magic directories and treat them specially.\n\n\nIt's also not been field tested to the extent of Netflix's committer; consider\nit the least mature of the committers.\n\n\n#### Which Committer to Use?\n\n1. If you want to create or update existing partitioned data trees in Spark, use thee\nPartitioned Committer. Make sure you have enough hard disk capacity for all staged data.\nDo not use it in other situations.\n\n1. If you know that your object store is consistent, or that the processes\nwriting data use S3Guard, use the Magic Committer for higher performance\nwriting of large amounts of data.\n\n1. Otherwise: use the directory committer, making sure you have enough\nhard disk capacity for all staged data.\n\nPut differently: start with the Directory Committer.\n\n## Switching to an S3A Committer\n\nTo use an S3A committer, the property `mapreduce.outputcommitter.factory.scheme.s3a`\nmust be set to the S3A committer factory, `org.apache.hadoop.fs.s3a.commit.staging.S3ACommitterFactory`.\nThis is done in `mapred-default.xml`\n\n```xml\n\n mapreduce.outputcommitter.factory.scheme.s3a\n org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory\n \n The committer factory to use when writing data to S3A filesystems.\n \n\n```\n\nWhat is missing is an explicit choice of committer to use in the property\n`fs.s3a.committer.name`; so the classic (and unsafe) file committer is used.\n\n| `fs.s3a.committer.name` | Committer |\n|--------|---------|\n| `directory` | directory staging committer |\n| `partitioned` | partition staging committer (for use in Spark only) |\n| `magic` | the \"magic\" committer |\n| `file` | the original and unsafe File committer; (default) |\n\n\n\n## Using the Directory and Partitioned Staging Committers\n\nGenerated files are initially written to a local directory underneath one of the temporary\ndirectories listed in `fs.s3a.buffer.dir`.\n\n\nThe staging committer needs a path in the cluster filesystem\n(e.g. HDFS). This must be declared in `fs.s3a.committer.staging.tmp.path`.\n\nTemporary files are saved in HDFS (or other cluster filesystem) under the path\n`${fs.s3a.committer.staging.tmp.path}/${user}` where `user` is the name of the user running the job.\nThe default value of `fs.s3a.committer.staging.tmp.path` is `tmp/staging`,\nWhich will be converted at run time to a path under the current user's home directory,\nessentially `~/tmp/staging`\n so the temporary directory\n\nThe application attempt ID is used to create a unique path under this directory,\nresulting in a path `~/tmp/staging/${user}/${application-attempt-id}/` under which\nsummary data of each task's pending commits are managed using the standard\n`FileOutputFormat` committer.\n\nWhen a task is committed the data is uploaded under the destination directory.\nThe policy of how to react if the destination exists is defined by\nthe `fs.s3a.committer.staging.conflict-mode` setting.\n\n| `fs.s3a.committer.staging.conflict-mode` | Meaning |\n| -----------------------------------------|---------|\n| `fail` | Fail if the destination directory exists |\n| `replace` | Delete all existing files before committing the new data |\n| `append` | Add the new files to the existing directory tree |\n\n\n## The \"Partitioned\" Staging Committer\n\nThis committer an extension of the \"Directory\" committer which has a special conflict resolution\npolicy designed to support operations which insert new data into a directory tree structured\nusing Hive's partitioning strategy: different levels of the tree represent different columns.\n\nFor example, log data could be partitioned by `YEAR` and then by `MONTH`, with different\nentries underneath.\n\n```\nlogs/YEAR=2017/MONTH=01/\n log-20170101.avro\n log-20170102.avro\n ...\n log-20170131.avro\n\nlogs/YEAR=2017/MONTH=02/\n log-20170201.avro\n log-20170202.avro\n ...\n log-20170227.avro\n\nlogs/YEAR=2017/MONTH=03/\nlogs/YEAR=2017/MONTH=04/\n```\n\nA partitioned structure like this allows for queries using Hive or Spark to filter out\nfiles which do not contain relevant data.\n\nWhat the partitioned committer does is, where the tooling permits, allows callers\nto add data to an existing partitioned layout*.\n\nMore specifically, it does this by having a conflict resolution options which\nonly act on individual partitions, rather than across the entire output tree.\n\n| `fs.s3a.committer.staging.conflict-mode` | Meaning |\n| -----------------------------------------|---------|\n| `fail` | Fail if the destination partition(s) exist |\n| `replace` | Delete the existing data partitions before committing the new data |\n| `append` | Add the new data to the existing partitions |\n\n\nAs an example, if a job was writing the file\n`logs/YEAR=2017/MONTH=02/log-20170228.avro`, then with a policy of `fail`,\nthe job would fail. With a policy of `replace`, then entire directory\n`logs/YEAR=2017/MONTH=02/` would be deleted before the new file `log-20170228.avro`\nwas written. With the policy of `append`, the new file would be added to\nthe existing set of files.\n\n\n### Notes\n\n1. A deep partition tree can itself be a performance problem in S3 and the s3a client,\nor, more specifically. a problem with applications which use recursive directory tree\nwalks to work with data.\n\n1. The outcome if you have more than one job trying simultaneously to write data\nto the same destination with any policy other than \"append\" is undefined.\n\n1. In the `append` operation, there is no check for conflict with file names.\nIf, in the example above, the file `log-20170228.avro` already existed,\nit would be overridden. Set `fs.s3a.committer.staging.unique-filenames` to `true`\nto ensure that a UUID is included in every filename to avoid this.\n\n\n## Using the Magic committer\n\nThis is less mature than the Staging Committer, but promises higher\nperformance.\n\n### FileSystem client setup\n\n1. Use a *consistent* S3 object store. For Amazon S3, this means enabling\n[S3Guard](./s3guard.html). For S3-compatible filesystems, consult the filesystem\ndocumentation to see if it is consistent, hence compatible \"out of the box\".\n1. Turn the magic on by `fs.s3a.committer.magic.enabled\"`\n\n```xml\n\n fs.s3a.committer.magic.enabled\n \n Enable support in the filesystem for the S3 \"Magic\" committer.\n \n true\n\n```\n\n*Do not use the Magic Committer on an inconsistent S3 object store. For\nAmazon S3, that means S3Guard must *always* be enabled.\n\n\n### Enabling the committer\n\n```xml\n\n fs.s3a.committer.name\n magic\n\n\n```\n\nConflict management is left to the execution engine itself.\n\n## Committer Configuration Options\n\n\n| Option | Magic | Directory | Partitioned | Meaning | Default |\n|--------|-------|-----------|-------------|---------|---------|\n| `mapreduce.fileoutputcommitter.marksuccessfuljobs` | X | X | X | Write a `_SUCCESS` file at the end of each job | `true` |\n| `fs.s3a.committer.threads` | X | X | X | Number of threads in committers for parallel operations on files. | 8 |\n| `fs.s3a.committer.staging.conflict-mode` | | X | X | Conflict resolution: `fail`, `abort` or `overwrite`| `fail` |\n| `fs.s3a.committer.staging.unique-filenames` | | X | X | Generate unique filenames | `true` |\n\n| `fs.s3a.committer.magic.enabled` | X | | | Enable \"magic committer\" support in the filesystem | `false` |\n\n\n\n\n| Option | Magic | Directory | Partitioned | Meaning | Default |\n|--------|-------|-----------|-------------|---------|---------|\n| `fs.s3a.buffer.dir` | X | X | X | Local filesystem directory for data being written and/or staged. | |\n| `fs.s3a.committer.staging.tmp.path` | | X | X | Path in the cluster filesystem for temporary data | `tmp/staging` |\n\n\n```xml\n\n fs.s3a.committer.name\n file\n \n Committer to create for output to S3A, one of:\n \"file\", \"directory\", \"partitioned\", \"magic\".\n \n\n\n\n fs.s3a.committer.magic.enabled\n false\n \n Enable support in the filesystem for the S3 \"Magic\" committer.\n When working with AWS S3, S3Guard must be enabled for the destination\n bucket, as consistent metadata listings are required.\n \n\n\n\n fs.s3a.committer.threads\n 8\n \n Number of threads in committers for parallel operations on files\n (upload, commit, abort, delete...)\n \n\n\n\n fs.s3a.committer.staging.tmp.path\n tmp/staging\n \n Path in the cluster filesystem for temporary data.\n This is for HDFS, not the local filesystem.\n It is only for the summary data of each file, not the actual\n data being committed.\n Using an unqualified path guarantees that the full path will be\n generated relative to the home directory of the user creating the job,\n hence private (assuming home directory permissions are secure).\n \n\n\n\n fs.s3a.committer.staging.unique-filenames\n true\n \n Option for final files to have a unique name through job attempt info,\n or the value of fs.s3a.committer.staging.uuid\n When writing data with the \"append\" conflict option, this guarantees\n that new data will not overwrite any existing data.\n \n\n\n\n fs.s3a.committer.staging.conflict-mode\n fail\n \n Staging committer conflict resolution policy.\n Supported: \"fail\", \"append\", \"replace\".\n \n\n\n\n s.s3a.committer.staging.abort.pending.uploads\n true\n \n Should the staging committers abort all pending uploads to the destination\n directory?\n\n Changing this if more than one partitioned committer is\n writing to the same destination tree simultaneously; otherwise\n the first job to complete will cancel all outstanding uploads from the\n others. However, it may lead to leaked outstanding uploads from failed\n tasks. If disabled, configure the bucket lifecycle to remove uploads\n after a time period, and/or set up a workflow to explicitly delete\n entries. Otherwise there is a risk that uncommitted uploads may run up\n bills.\n \n\n\n\n mapreduce.outputcommitter.factory.scheme.s3a\n org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory\n \n The committer factory to use when writing data to S3A filesystems.\n If mapreduce.outputcommitter.factory.class is set, it will\n override this property.\n\n (This property is set in mapred-default.xml)\n \n\n\n```\n\n\n## Troubleshooting\n\n### `Filesystem does not have support for 'magic' committer`\n\n```\norg.apache.hadoop.fs.s3a.commit.PathCommitException: `s3a://landsat-pds': Filesystem does not have support for 'magic' committer enabled\nin configuration option fs.s3a.committer.magic.enabled\n```\n\nThe Job is configured to use the magic committer, but the S3A bucket has not been explicitly\ndeclared as supporting it.\n\nThe destination bucket **must** be declared as supporting the magic committer.\n\nThis can be done for those buckets which are known to be consistent, either\nbecause [S3Guard](s3guard.html) is used to provide consistency,\nor because the S3-compatible filesystem is known to be strongly consistent.\n\n```xml\n\n fs.s3a.bucket.landsat-pds.committer.magic.enabled\n true\n\n```\n\n*IMPORTANT*: only enable the magic committer against object stores which\noffer consistent listings. By default, Amazon S3 does not do this -which is\nwhy the option `fs.s3a.committer.magic.enabled` is disabled by default.\n\n\nTip: you can verify that a bucket supports the magic committer through the\n`hadoop s3guard bucket-info` command:\n\n\n```\n> hadoop s3guard bucket-info -magic s3a://landsat-pds/\n\nFilesystem s3a://landsat-pds\nLocation: us-west-2\nFilesystem s3a://landsat-pds is not using S3Guard\nThe \"magic\" committer is not supported\n\nS3A Client\n Endpoint: fs.s3a.endpoint=(unset)\n Encryption: fs.s3a.server-side-encryption-algorithm=none\n Input seek policy: fs.s3a.experimental.input.fadvise=normal\n2017-09-27 19:18:57,917 INFO util.ExitUtil: Exiting with status 46: 46: The magic committer is not enabled for s3a://landsat-pds\n```\n\n## Error message: \"File being created has a magic path, but the filesystem has magic file support disabled:\n\nA file is being written to a path which is used for \"magic\" files,\nfiles which are actually written to a different destination than their stated path\n*but the filesystem doesn't support \"magic\" files*\n\nThis message should not appear through the committer itself —it will\nfail with the error message in the previous section, but may arise\nif other applications are attempting to create files under the path `/__magic/`.\n\nMake sure the filesystem meets the requirements of the magic committer\n(a consistent S3A filesystem through S3Guard or the S3 service itself),\nand set the `fs.s3a.committer.magic.enabled` flag to indicate that magic file\nwrites are supported.\n\n\n### `FileOutputCommitter` appears to be still used (from logs or delays in commits)\n\nThe Staging committers use the original `FileOutputCommitter` to manage\nthe propagation of commit information: do not worry if it the logs show `FileOutputCommitter`\nwork with data in the cluster filesystem (e.g. HDFS).\n\nOne way to make sure that the `FileOutputCommitter` is not being used to write\nthe data to S3 is to set the option `mapreduce.fileoutputcommitter.algorithm.version`\nto a value such as \"10\". Because the only supported algorithms are \"1\" and \"2\",\nany erroneously created `FileOutputCommitter` will raise an exception in its constructor\nwhen instantiated:\n\n```\njava.io.IOException: Only 1 or 2 algorithm version is supported\nat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:130)\nat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:104)\nat org.apache.parquet.hadoop.ParquetOutputCommitter.(ParquetOutputCommitter.java:42)\nat org.apache.parquet.hadoop.ParquetOutputFormat.getOutputCommitter(ParquetOutputFormat.java:395)\nat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupCommitter(HadoopMapReduceCommitProtocol.scala:67)\nat com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol.setupCommitter(PathOutputCommitProtocol.scala:62)\nat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:124)\nat com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol.setupJob(PathOutputCommitProtocol.scala:152)\nat org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:175)\nat org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:145)\n```\n\nWhile that will not make the problem go away, it will at least make\nthe failure happen at the start of a job.\n\n(Setting this option will not interfere with the Staging Committers' use of HDFS,\nas it explicitly sets the algorithm to \"2\" for that part of its work).\n\nThe other way to check which committer to use is to examine the `_SUCCESS` file.\nIf it is 0-bytes long, the classic `FileOutputCommitter` committed the job.\nThe S3A committers all write a non-empty JSON file; the `committer` field lists\nthe committer used.\n\n\n*Common causes*\n\n1. The property `fs.s3a.committer.name` is set to \"file\". Fix: change.\n1. The job has overridden the property `mapreduce.outputcommitter.factory.class`\nwith a new factory class for all committers. This takes priority over\nall committers registered for the s3a:// schema.\n1. The property `mapreduce.outputcommitter.factory.scheme.s3a` is unset.\n1. The output format has overridden `FileOutputFormat.getOutputCommitter()`\nand is returning its own committer -one which is a subclass of `FileOutputCommitter`.\n\nThat final cause. *the output format is returning its own committer*, is not\neasily fixed; it may be that the custom committer performs critical work\nduring its lifecycle, and contains assumptions about the state of the written\ndata during task and job commit (i.e. it is in the destination filesystem).\nConsult with the authors/maintainers of the output format\nto see whether it would be possible to integrate with the new committer factory\nmechanism and object-store-specific commit algorithms.\n\nParquet is a special case here: its committer does no extra work\nother than add the option to read all newly-created files then write a schema\nsummary. The Spark integration has explicit handling for Parquet to enable it\nto support the new committers, removing this (slow on S3) option.\n\nIf you have subclassed `FileOutputCommitter` and want to move to the\nfactory model, please get in touch.\n\n\n## Job/Task fails with PathExistsException: Destination path exists and committer conflict resolution mode is \"fail\"\n\nThis surfaces when either of two conditions are met.\n\n1. The Directory committer is used with `fs.s3a.committer.staging.conflict-mode` set to\n`fail` and the output/destination directory exists.\nThe job will fail in the driver during job setup.\n1. The Partitioned Committer is used with `fs.s3a.committer.staging.conflict-mode` set to\n`fail` and one of the partitions. The specific task(s) generating conflicting data will fail\nduring task commit, which will cause the entire job to fail.\n\nIf you are trying to write data and want write conflicts to be rejected, this is the correct\nbehavior: there was data at the destination so the job was aborted.\n\n## Staging committer task fails with IOException: No space left on device\n\nThere's not enough space on the local hard disk (real or virtual)\nto store all the uncommitted data of the active tasks on that host.\nBecause the staging committers write all output to the local disk\nand only upload the data on task commits, enough local temporary\nstorage is needed to store all output generated by all uncommitted\ntasks running on the single host. Small EC2 VMs may run out of disk.\n\n1. Make sure that `fs.s3a.buffer.dir` includes a temporary directory on\nevery available hard disk; this spreads load better.\n\n1. Add more disk space. In EC2: request instances with more local storage.\nThere is no need for EMR storage; this is just for temporary data.\n\n1. Purge the directories listed in `fs.s3a.buffer.dir` of old data.\nFailed tasks may not clean up all old files.\n\n1. Reduce the number of worker threads/process in the host.\n\n1. Consider partitioning the job into more tasks. This *may* result in more tasks\ngenerating less data each.\n\n1. Use the magic committer. This only needs enough disk storage to buffer\nblocks of the currently being written file during their upload process,\nso can use a lot less disk space.\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committer_architecture.md":"\n\n# S3A Committers: Architecture and Implementation\n\n\n\nThis document covers the architecture and implementation details of the S3A committers.\n\nFor information on using the committers, see [the S3A Committers](./committer.html).\n\n\n## Problem: Efficient, reliable commits of work to consistent S3 buckets\n\n\nThe standard commit algorithms (the `FileOutputCommitter` and its v1 and v2 algorithms)\nrely on directory rename being an `O(1)` atomic operation: callers output their\nwork to temporary directories in the destination filesystem, then\nrename these directories to the final destination as way of committing work.\nThis is the perfect solution for committing work against any filesystem with\nconsistent listing operations and where the `FileSystem.rename()` command\nis an atomic `O(1)` operation.\n\nUsing rename allows individual tasks to work in temporary directories, with the\nrename as the atomic operation can be used to explicitly commit tasks and\nultimately the entire job. Because the cost of the rename is low, it can be\nperformed during task and job commits with minimal delays. Note that HDFS\nwill lock the namenode metadata during the rename operation, so all rename() calls\nwill be serialized. However, as they only update the metadata of two directory\nentries, the duration of the lock is low.\n\nIn contrast to a \"real\" filesystem, Amazon's S3A object store, similar to\nmost others, does not support `rename()` at all. A hash operation on the filename\ndetermines the location of of the data —there is no separate metadata to change.\nTo mimic renaming, the Hadoop S3A client has to copy the data to a new object\nwith the destination filename, then delete the original entry. This copy\ncan be executed server-side, but as it does not complete until the in-cluster\ncopy has completed, it takes time proportional to the amount of data.\n\nThe rename overhead is the most visible issue, but it is not the most dangerous.\nThat is the fact that path listings have no consistency guarantees, and may\nlag the addition or deletion of files.\nIf files are not listed, the commit operation will *not* copy them, and\nso they will not appear in the final output.\n\nThe solution to this problem is closely coupled to the S3 protocol itself:\ndelayed completion of multi-part PUT operations\n\nThat is: tasks write all data as multipart uploads, *but delay the final\ncommit action until until the final, single job commit action.* Only that\ndata committed in the job commit action will be made visible; work from speculative\nand failed tasks will not be instantiated. As there is no rename, there is no\ndelay while data is copied from a temporary directory to the final directory.\nThe duration of the commit will be the time needed to determine which commit operations\nto construct, and to execute them.\n\n\n## Terminology\n\n* *Job*: a potentially parallelized query/operation to execute. The execution\nof a job: the division of work into tasks and the management of their completion,\nis generally executed in a single process.\n\nThe output of a Job is made visible to other stages in a larger operation\nsequence or other applications if the job *completes successfully*.\n\n* *Job Driver*. Not sure quite what term to use here. Whatever process schedules\ntask execution, tracks success/failures and, determines when all the work has been\nprocessed and then commits the output. It may also determine that a job\nhas failed and cannot be recovered, in which case the job is aborted.\nIn MR and Tez, this is inside the YARN application master.\nIn Spark it is the driver, which can run in the AM, the YARN client, or other\nplaces (e.g Livy?).\n\n* *Final directory*: the directory into which the output of a job is placed\nso as to be visible.\n\n* *Task* a single operation within a job, on a single process, one which generates\none or more files.\nAfter a successful job completion, the data MUST be visible in the final directory.\nA task completes successfully if it generates all the output it expects to without\nfailing in some way (error in processing; network/process failure).\n\n* *Job Context* an instance of the class `org.apache.hadoop.mapreduce.JobContext`,\nwhich provides a read-only view of the Job for the Job Driver and tasks.\n\n* *Task Attempt Context* an instance of the class\n`org.apache.hadoop.mapreduce.TaskAttemptContext extends JobContext, Progressable`,\nwhich provides operations for tasks, such as getting and setting status,\nprogress and counter values.\n\n* *Task Working Directory*: a directory for exclusive access by a single task,\ninto which uncommitted work may be placed.\n\n* *Task Commit* The act of taking the output of a task, as found in the\nTask Working Directory, and making it visible in the final directory.\nThis is traditionally implemented via a `FileSystem.rename()` call.\n\n It is useful to differentiate between a *task-side commit*: an operation performed\n in the task process after its work, and a *driver-side task commit*, in which\n the Job driver performs the commit operation. Any task-side commit work will\n be performed across the cluster, and may take place off the critical part for\n job execution. However, unless the commit protocol requires all tasks to await\n a signal from the job driver, task-side commits cannot instantiate their output\n in the final directory. They may be used to promote the output of a successful\n task into a state ready for the job commit, addressing speculative execution\n and failures.\n\n* *Job Commit* The act of taking all successfully completed tasks of a job,\nand committing them. This process is generally non-atomic; as it is often\na serialized operation at the end of a job, its performance can be a bottleneck.\n\n* *Task Abort* To cancel a task such that its data is not committed.\n\n* *Job Abort* To cancel all work in a job: no task's work is committed.\n\n* *Speculative Task Execution/ \"Speculation\"* Running multiple tasks against the same\ninput dataset in parallel, with the first task which completes being the one\nwhich is considered successful. Its output SHALL be committed; the other task\nSHALL be aborted. There's a requirement that a task can be executed in parallel,\nand that the output of a task MUST NOT BE visible until the job is committed,\nat the behest of the Job driver. There is the expectation that the output\nSHOULD BE the same on each task, though that MAY NOT be the case. What matters\nis if any instance of a speculative task is committed, the output MUST BE\nconsidered valid.\n\nThere is an expectation that the Job Driver and tasks can communicate: if a task\nperforms any operations itself during the task commit phase, it shall only do\nthis when instructed by the Job Driver. Similarly, if a task is unable to\ncommunicate its final status to the Job Driver, it MUST NOT commit is work.\nThis is very important when working with S3, as some network partitions could\nisolate a task from the Job Driver, while the task retains access to S3.\n\n## The execution workflow\n\n\n**setup**:\n\n* A job is created, assigned a Job ID (YARN?).\n* For each attempt, an attempt ID is created, to build the job attempt ID.\n* `Driver`: a `JobContext` is created/configured\n* A committer instance is instantiated with the `JobContext`; `setupJob()` invoked.\n\n\n## The `FileOutputCommitter`\n\nThe standard commit protocols are implemented in\n `org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter`.\n\nThere are two algorithms, the \"v1\" designed to address failures and restarts\nof the MapReduce application master. The V2 algorithm cannot recover from failure\nexcept by re-executing the entire job. It does, however, propagate all its work\nto the output directory in the task commit. When working with object stores which\nmimic `rename()` by copy and delete, it is more efficient due to the reduced\nnumber of listings, copies and deletes, and, because these are executed in\ntask commits, eliminates the final `O(data)` pause at the end of all work.\nIt is still highly inefficient, but the inefficiencies are less visible as\nlarge pauses in execution.\n\n\nNotes\n\n* The v1 algorithm was implemented to handle MapReduce AM restarts, it was\n not used in Hadoop 1.x, whose JobTracker could not recover from failures.\n Historically then, it is the second version of a file commit algorithm.\n* Because the renames are considered to be fast, there is no logging\nof renames being in progress, or their duration.\n* Speculative execution is supported by having every task attempt write its\nuncommitted data to a task attempt directory. When a task is ready to commit,\nit must notify that job driver that it is ready to commit -the job driver\nwill then commit or abort the task.\n\n## Hadoop MR Commit algorithm \"1\"\n\n\nThe \"v1\" commit algorithm is the default commit algorithm in Hadoop 2.x;\nit was implemented as part of [MAPREDUCE-2702](https://issues.apache.org/jira/browse/MAPREDUCE-2702).\n\nThis algorithm is designed to handle a failure and restart of the Job driver,\nwith the restarted job driver only rerunning the incomplete tasks; the\noutput of the completed tasks is recovered for commitment when the restarted\njob completes.\n\nThere is a cost: the time to commit by recursive listing all files in all\ncommitted task directories, and renaming this.\n\nAs this is performed sequentially, time to commit is `O(files)`, which is\ngenerally `O(tasks)`.\n\n\n\n```python\n#Job Attempt Path is `$dest/_temporary/$appAttemptId/`\n\njobAttemptPath = '$dest/_temporary/$appAttemptId/'\n\n# Task Attempt Path is `$dest/_temporary/$appAttemptId/_temporary/$taskAttemptID`\ntaskAttemptPath = '$dest/_temporary/$appAttemptId/_temporary/$taskAttemptID'\n\n#Task committed path is `$dest/_temporary/$appAttemptId/$taskAttemptID`\ntaskCommittedPath = '$dest/_temporary/$appAttemptId/$taskAttemptID'\n```\n\nTasks write in/under the task attempt path.\n\n### Job Setup\n\n```python\nfs.mkdir(jobAttemptPath)\n```\n\n### Task Setup\n\nNone: directories are created on demand.\n\n\n### Task Commit\n\nRename task attempt path to task committed path.\n\n```python\ndef needsTaskCommit(fs, jobAttemptPath, taskAttemptPath, dest):\n return fs.exists(taskAttemptPath)\n\n\ndef commitTask(fs, jobAttemptPath, taskAttemptPath, dest):\n if fs.exists(taskAttemptPath) :\n fs.delete(taskCommittedPath, recursive=True)\n fs.rename(taskAttemptPath, taskCommittedPath)\n```\n\nOn a genuine filesystem this is an `O(1)` directory rename.\n\nOn an object store with a mimiced rename, it is `O(data)` for the copy,\nalong with overhead for listing and deleting all files (For S3, that's\n`(1 + files/500)` lists, and the same number of delete calls.\n\n\n### Task Abort\n\nDelete task attempt path.\n\n```python\ndef abortTask(fs, jobAttemptPath, taskAttemptPath, dest):\n fs.delete(taskAttemptPath, recursive=True)\n```\n\nOn a genuine filesystem this is an `O(1)` operation. On an object store,\nproportional to the time to list and delete files, usually in batches.\n\n\n### Job Commit\n\nMerge all files/directories in all task committed paths into final destination path.\nOptionally; create 0-byte `_SUCCESS` file in destination path.\n\n```python\ndef commitJob(fs, jobAttemptDir, dest):\n for committedTask in fs.listFiles(jobAttemptDir):\n mergePathsV1(fs, committedTask, dest)\n fs.touch(\"$dest/_SUCCESS\")\n```\n\n(See below for details on `mergePaths()`)\n\n\nA failure during job commit cannot be recovered from except by re-executing\nthe entire query:\n\n```python\ndef isCommitJobRepeatable() :\n return False\n```\n\nAccordingly, it is a failure point in the protocol. With a low number of files\nand fast rename/list algorithms, the window of vulnerability is low. At\nscale, the vulnerability increases. It could actually be reduced through\nparallel execution of the renaming of of committed tasks.\n\n\n### Job Abort\n\nDelete all data under job attempt path.\n\n```python\ndef abortJob(fs, jobAttemptDir, dest):\n fs.delete(jobAttemptDir, recursive = True)\n```\n\n### Job Cleanup\n\n```python\ndef cleanupJob(fs, dest):\n fs.delete('$dest/_temporary', recursive = True)\n```\n\n\n### Job Recovery Before `commitJob()`\n\nFor all committers, the recovery process takes place in the application\nmaster.\n1. The job history file of the previous attempt is loaded and scanned\nto determine which tasks were recorded as having succeeded.\n1. For each successful task, the job committer has its `recoverTask()` method\ninvoked with a `TaskAttemptContext` built from the previous attempt's details.\n1. If the method does not raise an exception, it is considered to have been\nrecovered, and not to be re-executed.\n1. All other tasks are queued for execution.\n\nFor the v1 committer, task recovery is straightforward.\nThe directory of the committed task from the previous attempt is\nmoved under the directory of the current application attempt.\n\n```python\ndef recoverTask(tac):\n oldAttemptId = appAttemptId - 1\n fs.rename('$dest/_temporary/oldAttemptId/${tac.taskId}',\n '$dest/_temporary/appAttemptId/${tac.taskId}')\n```\n\nThis significantly improves time to recover from Job driver (here MR AM) failure.\nThe only lost work is that of all tasks in progress -those which had generated\ndata but were not yet committed.\n\nOnly the failure of the job driver requires a job restart, not an individual\ntask. Therefore the probability of this happening is independent of the number\nof tasks executed in parallel, instead simply due to the duration of the query.\n\nThe longer the task, the higher the risk of failure, the more value there is\nin recovering the work in progress.\n\nFast queries not only have a lower risk of failure, they can recover from\nfailure simply by rerunning the entire job. This is implicitly the strategy\nin Spark, which does not attempt to recover any in-progress jobs. The faster\nyour queries, the simpler your recovery strategy needs to be.\n\n### Job Recovery During `commitJob()`\n\nThis is not possible; a failure during job commit requires the entire job\nto be re-executed after cleaning up the destination directory.\n\n### `mergePaths(FileSystem fs, FileStatus src, Path dest)` Algorithm\n\n`mergePaths()` is the core algorithm to merge data; it is somewhat confusing\nas the implementation mixes the strategies for both algorithms across\ntwo co-recursive routines, `mergePaths()` and `renameOrMerge()`.\n\n\nHere the two algorithms have been split, and one of the co-recursive methods\ninlined.\n\n```python\ndef mergePathsV1(fs, src, dest) :\n if fs.exists(dest) :\n toStat = fs.getFileStatus(dest)\n else:\n toStat = None\n\n if src.isFile :\n if not toStat is None :\n fs.delete(dest, recursive = True)\n fs.rename(src.getPath, dest)\n else :\n # src is directory, choose action on dest type\n if not toStat is None :\n if not toStat.isDirectory :\n # Destination exists and is not a directory\n fs.delete(dest)\n fs.rename(src.getPath(), dest)\n else :\n # Destination exists and is a directory\n # merge all children under destination directory\n for child in fs.listStatus(src.getPath) :\n mergePathsV1(fs, child, dest + child.getName)\n else :\n # destination does not exist\n fs.rename(src.getPath(), dest)\n```\n\n## The v2 Commit Algorithm\n\n\nThe v2 algorithm directly commits task output into the destination directory.\nIt is essentially a re-implementation of the Hadoop 1.x commit algorithm.\n\n1. During execution, intermediate data becomes visible.\n1. On a failure, all output must be deleted and the job restarted.\n\nIt implements `mergePaths` differently, as shown below.\n\n```python\ndef mergePathsV2(fs, src, dest) :\n if fs.exists(dest) :\n toStat = fs.getFileStatus(dest)\n else:\n toStat = None\n\n if src.isFile :\n if not toStat is None :\n fs.delete(dest, recursive = True)\n fs.rename(src.getPath, dest)\n else :\n # destination is directory, choose action on source type\n if src.isDirectory :\n if not toStat is None :\n if not toStat.isDirectory :\n # Destination exists and is not a directory\n fs.delete(dest)\n fs.mkdirs(dest) #\n for child in fs.listStatus(src.getPath) : # HERE\n mergePathsV2(fs, child, dest + child.getName) #\n\n else :\n # Destination exists and is a directory\n # merge all children under destination directory\n for child in fs.listStatus(src.getPath) :\n mergePathsV2(fs, child, dest + child.getName)\n else :\n # destination does not exist\n fs.mkdirs(dest) #\n for child in fs.listStatus(src.getPath) : # HERE\n mergePathsV2(fs, child, dest + child.getName) #\n```\n\nBoth recurse down any source directory tree, and commit single files\nby renaming the files.\n\nA a key difference is that the v1 algorithm commits a source directory to\nvia a directory rename, which is traditionally an `O(1)` operation.\n\nIn contrast, the v2 algorithm lists all direct children of a source directory\nand recursively calls `mergePath()` on them, ultimately renaming the individual\nfiles. As such, the number of renames it performs equals the number of source\n*files*, rather than the number of source *directories*; the number of directory\nlistings being `O(depth(src))` , where `depth(path)` is a function returning the\ndepth of directories under the given path.\n\nOn a normal filesystem, the v2 merge algorithm is potentially more expensive\nthan the v1 algorithm. However, as the merging only takes place in task commit,\nit is potentially less of a bottleneck in the entire execution process.\n\nOn an object store, it is suboptimal not just from its expectation that `rename()`\nis an `O(1)` operation, but from its expectation that a recursive tree walk is\nan efficient way to enumerate and act on a tree of data. If the algorithm was\nswitched to using `FileSystem.listFiles(path, recursive)` for a single call to\nenumerate all children under a path, then the listing operation would be significantly\nfaster, at least on a deep or wide tree. However, for any realistic dataset,\nthe size of the output files is likely to be the main cause of delays. That\nis, if the cost of `mergePathsV2` is `O(depth(src)) + O(data))`, then\ngenerally the `O(data)` value will be more significant than the `depth(src)`.\n\nThere is one key exception: tests which work on small amounts of data yet try\nto generate realistic output directory structures. In these tests the cost\nof listing directories and calling `getFileStatus()` could exceed that of the copy\ncalls. This is why small-scale tests of the commit algorithms against object stores\nmust be considered significantly misleading.\n\n### v2 Task Commit\n\nRename task attempt path to task committed path.\n\n```python\ndef needsTaskCommit(fs, jobAttemptPath, taskAttemptPath, dest):\n return fs.exists(taskAttemptPath)\n\ndef commitTask(fs, jobAttemptPath, taskAttemptPath, dest):\n if fs.exists(taskAttemptPath) :\n mergePathsV2(fs. taskAttemptPath, dest)\n```\n\n### v2 Task Abort\n\nDelete task attempt path.\n\n```python\ndef abortTask(fs, jobAttemptPath, taskAttemptPath, dest):\n fs.delete(taskAttemptPath, recursive=True)\n```\n\nCost: `O(1)` for normal filesystems, `O(files)` for object stores.\n\n\n### v2 Job Commit\n\nAs all the task output is already completed, all that is needed is\nto touch the `_SUCCESS` marker.\n\n```python\ndef commitJob(fs, jobAttemptDir, dest):\n fs.touch(\"$dest/_SUCCESS\")\n```\nCost: `O(1)`\n\nA failure during job abort is implicitly repeatable\n\n```python\ndef isCommitJobRepeatable() :\n return True\n```\n\n### v2 Job Abort\n\nDelete all data under job attempt path.\n\n```python\ndef abortJob(fs, jobAttemptDir, dest):\n fs.delete(jobAttemptDir, recursive=True)\n```\n\nCost: `O(1)` for normal filesystems, `O(files)` for object stores.\n\n### v2 Task Recovery\n\nAs no data is written to the destination directory, a task can be cleaned up\nby deleting the task attempt directory.\n\n### v2 Job Recovery Before `commitJob()`\n\n\nBecause the data has been renamed into the destination directory, all tasks\nrecorded as having being committed have no recovery needed at all:\n\n```python\ndef recoverTask(tac):\n```\n\nAll active and queued tasks are scheduled for execution.\n\nThere is a weakness here, the same one on a failure during `commitTask()`:\nit is only safe to repeat a task which failed during that commit operation\nif the name of all generated files are constant across all task attempts.\n\nIf the Job AM fails while a task attempt has been instructed to commit,\nand that commit is not recorded as having completed, the state of that\nin-progress task is unknown...really it isn't be safe to recover the\njob at this point.\n\n\n### v2 Job Recovery During `commitJob()`\n\nThis is straightforward: `commitJob()` is re-invoked.\n\n## How MapReduce uses the committer in a task\n\nMapReduce runs each Mapper or Reducer in its own container; it\ngets its own process. Implicitly, this also means that each task gets\nits own instance of every filesystem.\n\nThe work is choreographed in `org.apache.hadoop.mapred.Task`, for which there\nare specific subclassing for the Map (`MapTask`) and reduce (`ReduceTask`).\nThere are also cleanup tasks whose role is simpler: clean things up,\nand a Job setup task which runs `OutputCommittre.setupJob`. That is: even\nthe job setup phase is run in a Task.\n\nMapTask uses the committer to write the output of all the mappers into\nthe filesystem, ready for the reducers. Each partition writes its data\nas a `MapFile`, which is actually two `SequenceFile` files in a directory,:\nthe `data` file of all Key-Value\noutput, and `index` which contains an index of some of the keys in the file.\n\nThis is all written to the local filesystem.\n\nThe `ReduceTask` does the final write to the destination filesystem.\n\nBecause the Map phase uses a committer to commits intermediate work,\nany plug in committer supplied to a process\nthrough any extension mechanism *must* work with the output generated by a mapper.\nThe staging committers do only if unique filenames are disabled, but as they\nand the magic committers are only meant to be used for the final output of work,\nit is somewhat moot. What is important is to be able to use different committers\nfor the map phase as the final reduce. This is implicit if separate committers\nare defined for different filesystems: the new committer can be defined for\nthe final destination FS, while `file://` can retain the default\n`FileOutputCommitter`.\n\n### Task Setup\n\n`Task.initialize()`: read in the configuration, instantiate the `JobContextImpl`\nand `TaskAttemptContextImpl` instances bonded to the current job & task.\n\n### Task Ccommit\n\nAfter the work is completed, `Task.done()` is invoked. Which is essentially the following\ncodepath:\n\n```java\nif (committer.needsTaskCommit(taskContext)) {\n // get permission to commit from AM\n int retries = MAX_RETRIES;\n while(true) {\n try {\n umbilical.commitPending(taskId, taskStatus);\n break;\n } catch(IOException ie) {\n if (--retries == 0) {\n // FAIL WITHOUT CALLING ABORT\n System.exit(67);\n }\n }\n }\n // commit the work\n try {\n committer.commitTask(taskContext);\n } catch (IOException iee) {\n // failure: abort\n try {\n committer.abortTask(taskContext);\n } catch (IOException ioe) {\n LOG.warn(\"Failure cleaning up: \" +\n StringUtils.stringifyException(ioe));\n }\n throw iee;\n }\n}\n```\n\nThat is: if and only if there is data to write, the Task requests clearance\nto do so from the AM. This ensures that speculative work is not committed if\nanother task has written it, *and that even non-speculative work is not committed\nif the task has lost contact with the AM*.\n\nThat is: task output is not committed if the Job's AM has failed or\nis in a separate network partition from the Task.\n\nIf permission is granted, the commit call is invoked.\nIf this doesn't work, `abortTask()` is called; all\nfailures there are logged and swallowed.\n\n\nThis method actually appears to be limited in that it does not guarantee\nthat `committer.abortTask()` is always called: if the `umbilical.commitPending()` calls\nfail repeatedly and the Task process aborts, `committer.abortTask()` is not called.\nIf this is due to a network partition, we can hope that the AM invokes\n`committer.abortTask()`, and that if it is an AM failure then a restarted AM can\nclean up its previous attempts. For the classic FileOutputCommitter, listing and\ndeleting the previous attempt's data is straightforward. However, for S3 committers\nusing Multipart Upload as the means of uploading uncommitted data, it is critical\nto ensure that pending uploads are always aborted. This can be done by\n\n* Making sure that all task-side failure branches in `Task.done()` call `committer.abortTask()`.\n* Having job commit & abort cleaning up all pending multipart writes to the same directory\ntree. That is: require that no other jobs are writing to the same tree, and so\nlist all pending operations and cancel them.\n* Add a CLI command to verify that there are no commits under a path; returning\na non-zero exit code if there are. This can be used in testing.\n* Add a CLI command to purge commits under a path. This can be used in workflows,\nif felt necessary, and in test runs to isolate the output of different tests.\n* Document a recommended purge-pending-commit timeout for the system\n* Add probes in the integration tests to verify that pending uploads are never\npresent after a write.\n\n\n\n### How MapReduce uses the committer in the Application Master\n\nThe AM uses a committer to set up and commit the job. To support failure\nand recovery of the AM, `OutputCommitter.isRecoverySupported()` is used\nto declare whether all the output of successful tasks can be used in the final\njob, or the entire job needs to be reset and repeated.\n`OutputCommitter.isCommitJobRepeatable()` addresses the other question:\ncan a committer recover from a failure of the commit process itself.\n\nA committer is created in the Application Master, and handed to an instance\nof `org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler`,\nwhich then manages the job-level lifecycle.\n\nIf the MRv1 API is used, the committer is chosen from the value of\n`\"mapred.output.committer.class\"`; in the MRv2 API the output format\nis instantiated, then asked for a committer using a task and task attempt ID of\n0. A committer is obtained from the output format via a call to\n`Committer.getOutputCommitter(taskContext)`, again using the task attempt context\nwith the (job, 0, 0) task and attempt IDs. That is: even for the job committers,\na task context is always passed in to the `OutputFormat` when requesting a committer.\n*It is critical for all implementations of `OutputCommitter.abortTask()` to\nbe able to execute from the AM, rather than the container and host running\nthe task. Furthermore, all information needed for the abort (paths, filesystem instances\n&c) *must* be retrieved from the `TaskAttemptContext` passed to the method,\nrather than relying on fields initiated from the context passed to the constructor.\n\n\n#### AM: Job setup: `OutputCommitter.setupJob()`\n\nThis is initiated in `org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.StartTransition`.\nIt is queued for asynchronous execution in `org.apache.hadoop.mapreduce.v2.app.MRAppMaster.startJobs()`,\nwhich is invoked when the service is started. Thus: the job is set up when the\nAM is started.\n\n\n#### AM: Job Commit: `OutputCommitter.commitJob()`\n\nIn the \"default\"/cluster filesystem, the `CommitterEventHandler` uses data in\nthe staging area defined in `yarn.app.mapreduce.am.staging-dir`\n(default `/tmp/hadoop-yarn/staging/$user/.staging`), in a subdirectory\nnamed from the job ID.\n\nThree paths are built up for using the filesystem for tracking the state\nof the commit process, with a goal of making the commit operation recoverable,\nwhere supported.\n\n\n| name | role |\n|------|--------|\n| `COMMIT_STARTED` | mark the commencement of the job commit |\n| `COMMIT_SUCCESS` | mark the successful completion of the job commit |\n| `COMMIT_FAIL` | mark the failure of the job commit |\n\nThese markers are used to manage job restart/failure of they happen during\nthe job commit itself.\n\nWhen an AM starts up, it looks in its staging area for these files. as a way\nto determine the previous state of the job. If there are no `COMMIT_` marker files,\nthe job is considered not to have attempted to commit itself yet.\n\n\nThe presence of `COMMIT_SUCCESS` or `COMMIT_FAIL` are taken as evidence\nthat the previous job completed successfully or unsuccessfully; the AM\nthen completes with a success/failure error code, without attempting to rerun\nthe job.\n\nIf `COMMIT_STARTED` exists but not either of the completion markers, then,\nif the committer declares that its job commit operation is repeatable\n(`Committer.isCommitJobRepeatable(jobContext) == true`), then an attempt\nis made to recommit the job, deleting the `COMMIT_STARTED` and commencing\nthe commit process again.\n\nThese `COMMIT_STARTED` files are simply 0-byte files, but are created\nwith the overwrite bit only set to true if the job commit is considered\nrepeatable:\n\n```java\nprivate void touchz(Path p, boolean overwrite) throws IOException {\n fs.create(p, overwrite).close();\n}\n```\nThat is: the atomicity of the `create(path, overwrite=false)` on the cluster\nfilesystem is used to guarantee that only one process will attempt to commit\na specific job.\n\n```java\nboolean commitJobIsRepeatable = committer.isCommitJobRepeatable(\n event.getJobContext());\ntry {\n touchz(startCommitFile, commitJobIsRepeatable);\n waitForValidCommitWindow();\n committer.commitJob(event.getJobContext());\n touchz(endCommitSuccessFile, commitJobIsRepeatable);\n} catch (Exception e) {\n touchz(endCommitFailureFile, commitJobIsRepeatable);\n}\n```\n\nThe `waitForValidCommitWindow()` operation is important: it declares that\nthe committer must not commit unless there has been communication with the YARN\nResource Manager with in `yarn.app.mapreduce.am.job.committer.commit-window` milliseconds\n(default: 10,000). It does this by waiting until the next heartbeat it received.\nThere's a possible bug here: if the interval is set too small the thread may\npermanently spin waiting a callback within the window. Ignoring that, this algorithm\nguarantees that\n\n1. As only one call can create a file with `overwrite=false`,\n only one process's attempt to commit a non-repeatable job will proceed\n\n1. Only a process with contact with the YARN within the configured window\n may commit a job.\n\n1. If the AM is partitioned from the rest of the network, provided that its clock\nis monotonically increasing at the same rate as the rest of the cluster, then\nthe rest of the cluster can be confident that\n`yarn.app.mapreduce.am.ob.committer.commit-window` milliseconds after the AM\nsuccessfully heartbeated to the YARN RM, then the output of this job attempt\nwill *never* be committed. This permits 1 job to run simultaneously, but\nhelps ensure that only one of them will attempt to commit.\n\n1. Provided YARN heartbeats are only sent to the AM which successfully created\nthe `COMMIT_STARTED` file, it will initiate the commit operation.\n\nPossible issues with this algorithm:\n\n* The `COMMIT_STARTED` file is created before waiting to get a heartbeat. It\nmay be that this AM has lost contact with YARN, but doesn't know it yet. When the\nYARN liveness protocols eventually time out, the AM will correctly terminate,\nbut as the `COMMIT_STARTED` file has been created at this point, no other launched\nAM will be able to commit.\n\n* If two committers attempt to create `COMMIT_STARTED` files on a no-repeatable\ncommit, one will succeed, wait for a heartbeat then attempt a (possibly slow) commit.\nThe second committer will fail, and will *immediately* create a `COMMIT_FAILED` file.\nAs a result, the state of the staging area will imply that the commit has failed,\nwhen really it is in progress, and that only the second process failed.\n\nIt would seem safer to address this through\n1. Waiting for a heartbeat before creating the `COMMIT_STARTED` file.\n1. Maybe: not creating the `COMMIT_FAILED` file if the failure happens when\ntrying to create the `COMMIT_STARTED` file. That is: only a process which\nsuccessfully created the `COMMIT_STARTED` file may indicate that a commit has failed.\n\n\n### AM: Cancelling job commit\n\nThe thread performing the commit is interrupted; the `CommitterEventHandler`\nawaits for it to finish. (set in `yarn.app.mapreduce.am.job.committer.cancel-timeout`\nas milliseconds).\n\n### AM: Task Abort\n\nThe AM may call the `OutputCommitter.taskAbort()` with a task attempt context,\nwhen handling the failure/loss of a container. That is: on container failure,\nthe task abort operation is executed in the AM, using the AM's committer.\nThis avoids the need to create a new container, and means that the \"best-effort\"\ntask abort does cope with container failures.\n\nA partition between the AM and the task container means that this AM-executed\ntask abort may take place while a task in the partitioned container is still\nexecuting. Unless output writing operations will fail after the abort operation,\nthe partitioned task may not become aware of the partition until it's own task\ncommit sequence in `Task.done()`, when `talkToAMTGetPermissionToCommit()`\n\n# Requirements of an S3A Committer\n\n1. Support an eventually consistent S3 object store as a reliable direct\ndestination of work through the S3A filesystem client.\n1. Efficient: implies no rename, and a minimal amount of delay in the job driver's\ntask and job commit phases,\n1. Support task failure and speculation.\n1. Can be used by existing code: Hadoop MapReduce, Spark, Hive.\n1. Retrofittable to existing subclasses of FileOutputFormat and/or compatible\nwith committers which expect a specific FileOutputFormat.\n1. Clean up uncommitted data from all task attempts, all previous attempts of\nthe job, and any previous incompleted jobs.\n1. Security: not to permit privilege escalation from other users with\nwrite access to the same file system(s).\n\n\n## Features of S3 and the S3A Client\n\n\nA core problem is that\n[object stores are not filesystems](../../../hadoop-project-dist/hadoop-common/filesystem/introduction.html);\nhow `rename()` has been emulated in the S3A client means that both the existing\nMR committer algorithms have significant performance problems.\n\n1. Single-object renames are implemented as a copy and delete sequence.\n1. COPY is atomic, but overwrites cannot be prevented.\n1. Amazon S3 is eventually consistent on listings, deletes and updates.\n1. Amazon S3 has create consistency, however, the negative response of a HEAD/GET\nperformed on a path before an object was created can be cached, unintentionally\ncreating a create inconsistency. The S3A client library does perform such a check,\non `create()` and `rename()` to check the state of the destination path, and\nso, whether the operation is permitted.\n1. Multi-object renames are sequential or parallel single object COPY+DELETE operations:\nnon atomic, `O(data)` and, on failure, can leave the filesystem in an unknown\nstate.\n1. There is a PUT operation, capable of uploading 5GB of data in one HTTP request.\n1. The PUT operation is atomic, but there is no PUT-no-overwrite option.\n1. There is a multipart POST/PUT sequence for uploading larger amounts of data\nin a sequence of PUT requests.\n\n\nThe Hadoop S3A Filesystem client supports PUT and multipart PUT for uploading\ndata, with the `S3ABlockOutputStream` of HADOOP-13560 uploading written data\nas parts of a multipart PUT once the threshold set in the configuration\nparameter `fs.s3a.multipart.size` (default: 100MB).\n\n[S3Guard](./s3guard.html) adds an option of consistent view of the filesystem\nto all processes using the shared DynamoDB table as the authoritative store of\nmetadata. Some S3-compatible object stores are fully consistent; the\nproposed algorithm is designed to work with such object stores without the\nneed for any DynamoDB tables.\n\n## Related work: Spark's `DirectOutputCommitter`\n\nOne implementation to look at is the\n[`DirectOutputCommitter` of Spark 1.6](https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/DirectParquetOutputCommitter.scala).\n\nThis implements a zero rename commit by subclassing the `ParquetOutputCommitter`\nand then\n\n1. Returning the destination directory as the task working directory.\n1. Subclassing all the task commit/abort operations to be no-ops.\n\nWith the working directory as the destination directory, there is no need\nto move/rename the task output on a successful commit. However, it is flawed.\nThere is no notion of \"committing\" or \"aborting\" a task, hence no ability to\nhandle speculative execution or failures. This is why the committer\nwas removed from Spark 2 [SPARK-10063](https://issues.apache.org/jira/browse/SPARK-10063)\n\nThere is also the issue that work-in-progress data is visible; this may or may\nnot be a problem.\n\n## Related work: IBM's \"Stocator\" committer\n\nIBM's [Stocator](https://github.com/SparkTC/stocator) can transform indirect\nwrites of V1/V2 committers into direct writes to the destination directory.\n\nHpw does it do this? It's a special Hadoop `FileSystem` implementation which\nrecognizes writes to `_temporary` paths and translate them to writes to the\nbase directory. As well as translating the write operation, it also supports\na `getFileStatus()` call on the original path, returning details on the file\nat the final destination. This allows for committing applications to verify\nthe creation/existence/size of the written files (in contrast to the magic\ncommitter covered below).\n\nThe FS targets Openstack Swift, though other object stores are supportable through\ndifferent backends.\n\nThis solution is innovative in that it appears to deliver the same semantics\n(and hence failure modes) as the Spark Direct OutputCommitter, but which\ndoes not need any change in either Spark *or* the Hadoop committers. In contrast,\nthe committers proposed here combines changing the Hadoop MR committers for\nease of pluggability, and offers a new committer exclusively for S3, one\nstrongly dependent upon and tightly integrated with the S3A Filesystem.\n\nThe simplicity of the Stocator committer is something to appreciate.\n\n## Background: The S3 multi-part PUT mechanism\n\nIn the [S3 REST API](http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html),\nmultipart uploads allow clients to upload a series of \"Parts\" of a file,\nthen commit the upload with a final call.\n\n1. Caller initiates a multipart request, including the destination bucket, key\nand metadata.\n\n POST bucket.s3.aws.com/path?uploads\n\n An `UploadId` is returned\n\n1. Caller uploads one or more parts.\n\n PUT bucket.s3.aws.com/path?partNumber=PartNumber&uploadId=UploadId\n\n The part number is used to declare the ordering of the PUTs; they\n can be uploaded in parallel and out of order.\n All parts *excluding the final part* must be 5MB or larger.\n Every upload completes with an etag returned\n\n1. Caller completes the operation\n\n POST /ObjectName?uploadId=UploadId\n \n (number)(Tag)\n ...\n \n\n This final call lists the etags of all uploaded parts and the actual ordering\n of the parts within the object.\n\nThe completion operation is apparently `O(1)`; presumably the PUT requests\nhave already uploaded the data to the server(s) which will eventually be\nserving up the data for the final path. All that is needed to complete\nthe upload is to construct an object by linking together the files in\nthe server's local filesystem and update an entry the index table of the\nobject store.\n\nIn the S3A client, all PUT calls in the sequence and the final commit are\ninitiated by the same process. *This does not have to be the case*.\nIt is that fact, that a different process may perform different parts\nof the upload, which make this algorithm viable.\n\n\n## The Netfix \"Staging\" committer\n\nRyan Blue, of Netflix, has submitted an alternate committer, one which has a\nnumber of appealing features\n\n* Doesn't have any requirements of the destination object store,\n* Known to work.\n\nThe final point is not to be underestimated, es not even\na need for a consistency layer.\n* Overall a simpler design. Especially given the need to\nbe resilient to the various failure modes which may arise.\n\n\nThe committer writes task outputs to a temporary directory on the local FS.\nTask outputs are directed to the local FS by `getTaskAttemptPath` and `getWorkPath`.\nOn task commit, the committer enumerates files in the task attempt directory (ignoring hidden files).\nEach file is uploaded to S3 using the [multi-part upload API](http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html),\n\nThe information needed to commit the upload is saved into HDFS and committed\nvia that protocol: when the job commits, the pending uploads of the successful\ntasks are all committed.\n\n\n### Commit logic\n\nThe core algorithm is as follows:\n\n1. The destination directory for output (e.g. `FileOutputFormat` and subclasses)\nis a local `file://` reference.\n1. Task commit initiates the multipart PUT to the destination object store.\n1. A list of every pending PUT for task is persisted to a single file\nwithin a consistent, cluster-wide filesystem. For Netflix, that is HDFS.\n1. The Standard `FileOutputCommitter` (algorithm 1) is used to manage the commit/abort of these\nfiles. That is: it copies only those lists of files to commit from successful tasks\ninto a (transient) job commit directory.\n1. The S3 job committer reads the pending file list for every task committed\nin HDFS, and completes those put requests.\n\nBy using `FileOutputCommmitter` to manage the propagation of the lists of files\nto commit, the existing commit algorithm implicitly becomes that defining which\nfiles will be committed at the end of the job.\n\n\nThe Netflix contribution has Hadoop `OutputCommitter` implementations for S3.\n\nThere are 3 main classes:\n* `S3MultipartOutputCommitter` is a base committer class that handles commit logic. This should not be used directly.\n* `S3DirectoryOutputCommitter` for writing unpartitioned data to S3 with conflict resolution.\n* `S3PartitionedOutputCommitter` for writing partitioned data to S3 with conflict resolution.\n\nCallers should use `S3DirectoryOutputCommitter` for single-directory outputs,\nor `S3PartitionedOutputCommitter` for partitioned data.\n\n\nThese S3 committers work by writing task outputs to a temporary directory on the local FS.\nTask outputs are directed to the local FS by `getTaskAttemptPath` and `getWorkPath`.\n\n\n### Conflict Resolution\n\nThe single-directory and partitioned committers handle conflict resolution by\nchecking whether target paths exist in S3 before uploading any data.\nThere are three conflict resolution modes, controlled by setting `fs.s3a.committer.staging.conflict-mode`:\n\n* `fail`: Fail a task if an output directory or partition already exists. (Default)\n* `append`: Upload data files without checking whether directories or partitions already exist.\n* `replace`: If an output directory exists, delete it so the new data replaces the current content.\n\nThe partitioned committer enforces the conflict mode when a conflict is detected with output data, not before the job runs.\nConflict resolution differs from an output mode because it does not enforce the mode when there is no conflict.\nFor example, overwriting a partition should remove all sub-partitions and data it contains, whether or not new output is created.\nConflict resolution will only replace partitions that have output data.\n\nWhen the conflict mode is `replace`, conflicting directories are removed during\njob commit. Data is only deleted if all tasks have completed successfully.\n\nA UUID that identifies a write is added to filenames that are uploaded to S3.\nThis allows rolling back data from jobs that fail during job commit (see failure cases below) and avoids\nfile-level conflicts when appending data to existing directories.\n\n\n*Note* the checks for existence are made via `S3AFileSystem.getFileStatus()` requests of the destination paths.\nUnless the view of the S3 store is consistent, it may be that a newly-deleted object\nis still discovered in the probe, so a commit fail, even when there is no longer any actual conflict.\n\n### Performance\n\nCompared to the previous proposal, henceforth the \"magic\" committer, this\ncommitter, the \"staging committer\", adds the extra overhead of uploading\neach file at the end of every task. This is an `O(data)` operation; it can be\nparallelized, but is bounded by the bandwidth from compute node to S3, as\nwell as the write/IOP capacity of the destination shard of S3. If many tasks\ncomplete at or near the same time, there may be a peak of bandwidth load\nslowing down the upload.\n\nTime to commit will be the same, and, given the Netflix committer has already\nimplemented the parallelization logic here, a time of `O(files/threads)`.\n\n### Resilience\n\nThere's already a lot of code in the task and job commits to handle failure.\n\nAny failure in a commit triggers a best-effort abort/revert of the commit\nactions for a task/job.\n\nTask commits delegate to the `FileOutputCommitter` to ensure that only one task's\noutput reaches the job commit.\n\nSimilarly, if a task is aborted, temporary output on the local FS is removed.\n\nIf a task dies while the committer is running, it is possible for data to be\neft on the local FS or as unfinished parts in S3.\nUnfinished upload parts in S3 are not visible to table readers and are cleaned\nup following the rules in the target bucket's life-cycle policy.\n\nFailures during job commit are handled by deleting any files that have already\nbeen completed and aborting the remaining uploads.\nBecause uploads are completed individually, the files that are deleted were visible to readers.\n\nIf the process dies while the job committer is running, there are two possible failures:\n\n1. Some directories that would be replaced have been deleted, but no new data is visible.\n2. Some new data is visible but is not complete, and all replaced directories have been removed.\n Only complete files are visible.\n\nIf the process dies during job commit, cleaning up is a manual process.\nFile names include a UUID for each write so that files can be identified and removed.\n\n\n**Failure during task execution**\n\nAll data is written to local temporary files; these need to be cleaned up.\n\nThe job must ensure that the local (pending) data is purged.\n\n**Failure during task commit**\n\n\nA process failure during the upload process will result in the\nlist of pending multipart PUTs to *not* be persisted to the cluster filesystem.\nThis window is smaller than the entire task execution, but still potentially\nsignificant, at least for large uploads.\n\nPer-file persistence, or incremental overwrites of the upload list may\nreduce the problems here, but there would still be a small risk of\nan outstanding multipart upload not being recorded\n\n**Explicit Task abort before task commit**\n\nTask will delete all local data; no uploads will be initiated.\n\n**Failure to communicate with S3 during data upload**\n\nIf an upload fails, tasks will:\n\n* Retry using the retry policies implemented in the S3AFileSystem classes\nand the AWS libraries.\n* Eventually: attempt to abort outstanding multipart uploads already initiated.\n* Remove temporary files on the local FS.\n\n\n**Explicit Job Abort**\n\nAll in-progress tasks are aborted and cleaned up. The pending commit data\nof all completed tasks can be loaded, outstanding multipart PUT requests aborted.\n\n\nThis is done by\n\n1. Listing all local files, with a best effort read attempt followed by\nan abort of all successfully read files.\n1. List and abort all pending multipart uploads.\n\nBecause of action #2, action #1 is superfluous. It is retained so as to leave\nopen the option of making action #2 a configurable option -which would be\nrequired to handle the use case of >1 partitioned commit running simultaneously/\n\n**Job Driver failure before Job Commit**\n\nBecause the local data is managed with the v1 commit algorithm, the\nsecond attempt of the job will recover all the outstanding commit data\nof the first attempt; those tasks will not be rerun.\n\nThis also ensures that on a job abort, the individual tasks' .pendingset\nfiles can be read and used to initiate the abort of those uploads.\nThat is: a recovered job can clean up the pending writes of the previous job\n\nIf the query engine does not support multiple job attempts, then the\npending commit data will not be recovered; an explicit abort operation will\nneed to be initiated (we will add a CLI command for this), or the S3 bucket\nmust be configured to automatically delete the pending request.\n\n**Job Driver failure during Job Commit**\n\nThose uploads already executed by a failed job commit will persist; those\nyet to execute will remain outstanding.\n\nThe committer currently declares itself as non-recoverable, but that\nmay not actually hold, as the recovery process could be one of:\n\n1. Enumerate all job commits from the .pendingset files (*:= Commits*).\n1. List all outstanding uploads under the destination path (*:= Outstandings*)..\n1. List all written files (for better consistency, via a GET call on the known\nfilenames)\n1. Identify all files which are yet to be completed (*Commits - Written*)/\n1. Verify that the set of pend","hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md":"\n\n# Hadoop-Aliyun module: Integration with Aliyun Web Services\n\n\n\n## Overview\n\nThe `hadoop-aliyun` module provides support for Aliyun integration with\n[Aliyun Object Storage Service (Aliyun OSS)](https://www.aliyun.com/product/oss).\nThe generated JAR file, `hadoop-aliyun.jar` also declares a transitive\ndependency on all external artifacts which are needed for this support — enabling\ndownstream applications to easily use this support.\n\nTo make it part of Apache Hadoop's default classpath, simply make sure\nthat HADOOP_OPTIONAL_TOOLS in hadoop-env.sh has 'hadoop-aliyun' in the list.\n\n### Features\n\n* Read and write data stored in Aliyun OSS.\n* Present a hierarchical file system view by implementing the standard Hadoop\n[`FileSystem`](../api/org/apache/hadoop/fs/FileSystem.html) interface.\n* Can act as a source of data in a MapReduce job, or a sink.\n\n### Warning #1: Object Stores are not filesystems.\n\nAliyun OSS is an example of \"an object store\". In order to achieve scalability\nand especially high availability, Aliyun OSS has relaxed some of the constraints\nwhich classic \"POSIX\" filesystems promise.\n\n\n\nSpecifically\n\n1. Atomic operations: `delete()` and `rename()` are implemented by recursive\nfile-by-file operations. They take time at least proportional to the number of files,\nduring which time partial updates may be visible. `delete()` and `rename()`\ncan not guarantee atomicity. If the operations are interrupted, the filesystem\nis left in an intermediate state.\n2. File owner and group are persisted, but the permissions model is not enforced.\nAuthorization occurs at the level of the entire Aliyun account via\n[Aliyun Resource Access Management (Aliyun RAM)](https://www.aliyun.com/product/ram).\n3. Directory last access time is not tracked.\n4. The append operation is not supported.\n\n### Warning #2: Directory last access time is not tracked,\nFeatures of Hadoop relying on this can have unexpected behaviour. E.g. the\nAggregatedLogDeletionService of YARN will not remove the appropriate log files.\n\n### Warning #3: Your Aliyun credentials are valuable\n\nYour Aliyun credentials not only pay for services, they offer read and write\naccess to the data. Anyone with the account can not only read your datasets\n— they can delete them.\n\nDo not inadvertently share these credentials through means such as\n1. Checking in to SCM any configuration files containing the secrets.\n2. Logging them to a console, as they invariably end up being seen.\n3. Defining filesystem URIs with the credentials in the URL, such as\n`oss://accessKeyId:accessKeySecret@directory/file`. They will end up in\nlogs and error messages.\n4. Including the secrets in bug reports.\n\nIf you do any of these: change your credentials immediately!\n\n### Warning #4: The Aliyun OSS client provided by Aliyun E-MapReduce are different from this implementation\n\nSpecifically: on Aliyun E-MapReduce, `oss://` is also supported but with\na different implementation. If you are using Aliyun E-MapReduce,\nfollow these instructions —and be aware that all issues related to Aliyun\nOSS integration in E-MapReduce can only be addressed by Aliyun themselves:\nplease raise your issues with them.\n\n## OSS\n\n### Authentication properties\n\n \n fs.oss.accessKeyId\n Aliyun access key ID\n \n\n \n fs.oss.accessKeySecret\n Aliyun access key secret\n \n\n \n fs.oss.credentials.provider\n \n Class name of a credentials provider that implements\n com.aliyun.oss.common.auth.CredentialsProvider. Omit if using access/secret keys\n or another authentication mechanism. The specified class must provide an\n accessible constructor accepting java.net.URI and\n org.apache.hadoop.conf.Configuration, or an accessible default constructor.\n \n \n\n### Other properties\n\n \n fs.oss.endpoint\n Aliyun OSS endpoint to connect to. An up-to-date list is\n provided in the Aliyun OSS Documentation.\n \n \n\n \n fs.oss.assumed.role.arn\n \n Role ARN for the role to be assumed.\n Required if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.sts.endpoint\n \n STS Token Service endpoint.\n Required if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.session.name\n \n \n Session name for the assumed role, must be valid characters\n according to Aliyun API. It is optional, will be generated by\n oss java sdk if it is empty.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.session.duration\n \n \n Duration of assumed roles before it is expired. Default is 30 minutes.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.sts.expiredFactor\n \n \n Sts token will be refreshed after (expiredFactor * duration) seconds.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.proxy.host\n Hostname of the (optinal) proxy server for Aliyun OSS connection\n \n\n \n fs.oss.proxy.port\n Proxy server port\n \n\n \n fs.oss.proxy.username\n Username for authenticating with proxy server\n \n\n \n fs.oss.proxy.password\n Password for authenticating with proxy server.\n \n\n \n fs.oss.proxy.domain\n Domain for authenticating with proxy server.\n \n\n \n fs.oss.proxy.workstation\n Workstation for authenticating with proxy server.\n \n\n \n fs.oss.attempts.maximum\n 20\n How many times we should retry commands on transient errors.\n \n\n \n fs.oss.connection.establish.timeout\n 50000\n Connection setup timeout in milliseconds.\n \n\n \n fs.oss.connection.timeout\n 200000\n Socket connection timeout in milliseconds.\n \n\n \n fs.oss.paging.maximum\n 1000\n How many keys to request from Aliyun OSS when doing directory listings at a time.\n \n \n\n \n fs.oss.multipart.upload.size\n 10485760\n Size of each of multipart pieces in bytes.\n \n\n \n fs.oss.multipart.upload.threshold\n 20971520\n Minimum size in bytes before we start a multipart uploads or copy.\n \n\n \n fs.oss.multipart.download.size\n 102400/value>\n Size in bytes in each request from ALiyun OSS.\n \n\n \n fs.oss.buffer.dir\n Comma separated list of directories to buffer OSS data before uploading to Aliyun OSS\n \n\n \n fs.oss.acl.default\n \n Set a canned ACL for bucket. Value may be private, public-read, public-read-write.\n \n \n\n \n fs.oss.server-side-encryption-algorithm\n \n Specify a server-side encryption algorithm for oss: file system.\n Unset by default, and the only other currently allowable value is AES256.\n \n \n\n \n fs.oss.connection.maximum\n 32\n Number of simultaneous connections to oss.\n \n\n \n fs.oss.connection.secure.enabled\n true\n Connect to oss over ssl or not, true by default.\n \n\n## Testing the hadoop-aliyun Module\n\nTo test `oss://` filesystem client, two files which pass in authentication\ndetails to the test runner are needed.\n\n1. `auth-keys.xml`\n2. `core-site.xml`\n\nThose two configuration files must be put into\n`hadoop-tools/hadoop-aliyun/src/test/resources`.\n\n### `core-site.xml`\n\nThis file pre-exists and sources the configurations created in `auth-keys.xml`.\n\nFor most cases, no modification is needed, unless a specific, non-default property\nneeds to be set during the testing.\n\n### `auth-keys.xml`\n\nThis file triggers the testing of Aliyun OSS module. Without this file,\n*none of the tests in this module will be executed*\n\nIt contains the access key Id/secret and proxy information that are needed to\nconnect to Aliyun OSS, and an OSS bucket URL should be also provided.\n\n1. `test.fs.oss.name` : the URL of the bucket for Aliyun OSS tests\n\nThe contents of the bucket will be cleaned during the testing process, so\ndo not use the bucket for any purpose other than testing.\n\n### Run Hadoop contract tests\nCreate file `contract-test-options.xml` under `/test/resources`. If a\nspecific file `fs.contract.test.fs.oss` test path is not defined, those\ntests will be skipped. Credentials are also needed to run any of those\ntests, they can be copied from `auth-keys.xml` or through direct\nXInclude inclusion. Here is an example of `contract-test-options.xml`:\n\n \n \n \n\n \n\n \n fs.contract.test.fs.oss\n oss://spark-tests\n \n\n \n fs.oss.impl\n org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem\n \n\n \n fs.oss.endpoint\n oss-cn-hangzhou.aliyuncs.com\n \n\n \n fs.oss.buffer.dir\n /tmp/oss\n \n\n \n fs.oss.multipart.download.size\n 102400\n \n \n","hadoop-tools/hadoop-resourceestimator/README.md":"Resource Estimator Service\n==========================\n\nResource Estimator Service can parse the history logs of production jobs, extract their resource consumption skylines in the past runs and predict their resource requirements for the new run.\n\n## Current Status\n\n * Support [Hadoop YARN ResourceManager](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) logs.\n * In-memory store for parsed history resource skyline and estimation.\n * A [Linear Programming](https://github.com/optimatika/ojAlgo) based estimator.\n * Provides REST interface to parse logs, query history store and estimations.\n\n## Upcoming features\n\n * UI to query history and edit and save estimations.\n * Persisent store implementation for store (either DB based or distributed key-value like HBase).\n * Integrate directly with the [Hadoop YARN Reservation System](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ReservationSystem.html) to make a recurring reservation based on the estimated resources.\n\nRefer to the [design document](https://issues.apache.org/jira/secure/attachment/12886714/ResourceEstimator-design-v1.pdf) for more details.\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md":"\n\n# Maximizing Performance when working with the S3A Connector\n\n\n\n\n## Introduction\n\nS3 is slower to work with than HDFS, even on virtual clusters running on\nAmazon EC2.\n\nThat's because its a very different system, as you can see:\n\n\n| Feature | HDFS | S3 through the S3A connector |\n|---------|------|------------------------------|\n| communication | RPC | HTTP GET/PUT/HEAD/LIST/COPY requests |\n| data locality | local storage | remote S3 servers |\n| replication | multiple datanodes | asynchronous after upload |\n| consistency | consistent data and listings | eventual consistent for listings, deletes and updates |\n| bandwidth | best: local IO, worst: datacenter network | bandwidth between servers and S3 |\n| latency | low | high, especially for \"low cost\" directory operations |\n| rename | fast, atomic | slow faked rename through COPY & DELETE|\n| delete | fast, atomic | fast for a file, slow & non-atomic for directories |\n| writing| incremental | in blocks; not visible until the writer is closed |\n| reading | seek() is fast | seek() is slow and expensive |\n| IOPs | limited only by hardware | callers are throttled to shards in an s3 bucket |\n| Security | Posix user+group; ACLs | AWS Roles and policies |\n\nFrom a performance perspective, key points to remember are:\n\n* S3 throttles bucket access across all callers: adding workers can make things worse.\n* EC2 VMs have network IO throttled based on the VM type.\n* Directory rename and copy operations take *much* longer the more objects and data there is.\nThe slow performance of `rename()` surfaces during the commit phase of jobs,\napplications like `DistCP`, and elsewhere.\n* seek() calls when reading a file can force new HTTP requests.\nThis can make reading columnar Parquet/ORC data expensive.\n\nOverall, although the S3A connector makes S3 look like a file system,\nit isn't, and some attempts to preserve the metaphor are \"aggressively suboptimal\".\n\nTo make most efficient use of S3, care is needed.\n\n## Speeding up directory listing operations through S3Guard\n\n[S3Guard](s3guard.html) provides significant speedups for operations which\nlist files a lot. This includes the setup of all queries against data:\nMapReduce, Hive and Spark, as well as DistCP.\n\n\nExperiment with using it to see what speedup it delivers.\n\n\n## Improving data input performance through fadvise\n\nThe S3A Filesystem client supports the notion of input policies, similar\nto that of the Posix `fadvise()` API call. This tunes the behavior of the S3A\nclient to optimise HTTP GET requests for the different use cases.\n\n### fadvise `sequential`\n\nRead through the file, possibly with some short forward seeks.\n\nThe whole document is requested in a single HTTP request; forward seeks\nwithin the readahead range are supported by skipping over the intermediate\ndata.\n\nThis delivers maximum sequential throughput —but with very expensive\nbackward seeks.\n\nApplications reading a file in bulk (DistCP, any copy operations) should use\nsequential access, as should those reading data from gzipped `.gz` files.\nBecause the \"normal\" fadvise policy starts off in sequential IO mode,\nthere is rarely any need to explicit request this policy.\n\n### fadvise `random`\n\nOptimised for random IO, specifically the Hadoop `PositionedReadable`\noperations —though `seek(offset); read(byte_buffer)` also benefits.\n\nRather than ask for the whole file, the range of the HTTP request is\nset to that that of the length of data desired in the `read` operation\n(Rounded up to the readahead value set in `setReadahead()` if necessary).\n\nBy reducing the cost of closing existing HTTP requests, this is\nhighly efficient for file IO accessing a binary file\nthrough a series of `PositionedReadable.read()` and `PositionedReadable.readFully()`\ncalls. Sequential reading of a file is expensive, as now many HTTP requests must\nbe made to read through the file: there's a delay between each GET operation.\n\n\nRandom IO is best for IO with seek-heavy characteristics:\n\n* Data is read using the `PositionedReadable` API.\n* Long distance (many MB) forward seeks\n* Backward seeks as likely as forward seeks.\n* Little or no use of single character `read()` calls or small `read(buffer)`\ncalls.\n* Applications running close to the S3 data store. That is: in EC2 VMs in\nthe same datacenter as the S3 instance.\n\nThe desired fadvise policy must be set in the configuration option\n`fs.s3a.experimental.input.fadvise` when the filesystem instance is created.\nThat is: it can only be set on a per-filesystem basis, not on a per-file-read\nbasis.\n\n```xml\n\n fs.s3a.experimental.input.fadvise\n random\n \n Policy for reading files.\n Values: 'random', 'sequential' or 'normal'\n \n\n```\n\n[HDFS-2744](https://issues.apache.org/jira/browse/HDFS-2744),\n*Extend FSDataInputStream to allow fadvise* proposes adding a public API\nto set fadvise policies on input streams. Once implemented,\nthis will become the supported mechanism used for configuring the input IO policy.\n\n### fadvise `normal` (default)\n\nThe `normal` policy starts off reading a file in `sequential` mode,\nbut if the caller seeks backwards in the stream, it switches from\nsequential to `random`.\n\nThis policy essentially recognizes the initial read pattern of columnar\nstorage formats (e.g. Apache ORC and Apache Parquet), which seek to the end\nof a file, read in index data and then seek backwards to selectively read\ncolumns. The first seeks may be be expensive compared to the random policy,\nhowever the overall process is much less expensive than either sequentially\nreading through a file with the `random` policy, or reading columnar data\nwith the `sequential` policy.\n\n\n## Committing Work in MapReduce and Spark\n\nHadoop MapReduce, Apache Hive and Apache Spark all write their work\nto HDFS and similar filesystems.\nWhen using S3 as a destination, this is slow because of the way `rename()`\nis mimicked with copy and delete.\n\nIf committing output takes a long time, it is because you are using the standard\n`FileOutputCommitter`. If you are doing this on any S3 endpoint which lacks\nlist consistency (Amazon S3 without [S3Guard](s3guard.html)), this committer\nis at risk of losing data!\n\n*Your problem may appear to be performance, but that is a symptom\nof the underlying problem: the way S3A fakes rename operations means that\nthe rename cannot be safely be used in output-commit algorithms.*\n\nFix: Use one of the dedicated [S3A Committers](committers.md).\n\n## Options to Tune\n\n### Thread and connection pool sizes.\n\nEach S3A client interacting with a single bucket, as a single user, has its\nown dedicated pool of open HTTP 1.1 connections alongside a pool of threads used\nfor upload and copy operations.\nThe default pool sizes are intended to strike a balance between performance\nand memory/thread use.\n\nYou can have a larger pool of (reused) HTTP connections and threads\nfor parallel IO (especially uploads) by setting the properties\n\n\n| property | meaning | default |\n|----------|---------|---------|\n| `fs.s3a.threads.max`| Threads in the AWS transfer manager| 10 |\n| `fs.s3a.connection.maximum`| Maximum number of HTTP connections | 10|\n\nWe recommend using larger values for processes which perform\na lot of IO: `DistCp`, Spark Workers and similar.\n\n```xml\n\n fs.s3a.threads.max\n 20\n\n\n fs.s3a.connection.maximum\n 20\n\n```\n\nBe aware, however, that processes which perform many parallel queries\nmay consume large amounts of resources if each query is working with\na different set of s3 buckets, or are acting on behalf of different users.\n\n### For large data uploads, tune the block size: `fs.s3a.block.size`\n\nWhen uploading data, it is uploaded in blocks set by the option\n`fs.s3a.block.size`; default value \"32M\" for 32 Megabytes.\n\nIf a larger value is used, then more data is buffered before the upload\nbegins:\n\n```xml\n\n fs.s3a.block.size\n 128M\n\n```\n\nThis means that fewer PUT/POST requests are made of S3 to upload data,\nwhich reduces the likelihood that S3 will throttle the client(s)\n\n### Maybe: Buffer Write Data in Memory\n\nWhen large files are being uploaded, blocks are saved to disk and then\nqueued for uploading, with multiple threads uploading different blocks\nin parallel.\n\nThe blocks can be buffered in memory by setting the option\n`fs.s3a.fast.upload.buffer` to `bytebuffer`, or, for on-heap storage\n`array`.\n\n1. Switching to in memory-IO reduces disk IO, and can be faster if the bandwidth\nto the S3 store is so high that the disk IO becomes the bottleneck.\nThis can have a tangible benefit when working with on-premise S3-compatible\nobject stores with very high bandwidth to servers.\n\nIt is very easy to run out of memory when buffering to it; the option\n`fs.s3a.fast.upload.active.blocks\"` exists to tune how many active blocks\na single output stream writing to S3 may have queued at a time.\n\nAs the size of each buffered block is determined by the value of `fs.s3a.block.size`,\nthe larger the block size, the more likely you will run out of memory.\n\n## DistCP\n\nDistCP can be slow, especially if the parameters and options for the operation\nare not tuned for working with S3.\n\nTo exacerbate the issue, DistCP invariably puts heavy load against the\nbucket being worked with, which will cause S3 to throttle requests.\nIt will throttle: directory operations, uploads of new data, and delete operations,\namongst other things\n\n### DistCP: Options to Tune\n\n* `-numListstatusThreads ` : set to something higher than the default (1).\n* `-bandwidth ` : use to limit the upload bandwidth per worker\n* `-m ` : limit the number of mappers, hence the load on the S3 bucket.\n\nAdding more maps with the `-m` option does not guarantee better performance;\nit may just increase the amount of throttling which takes place.\nA smaller number of maps with a higher bandwidth per map can be more efficient.\n\n### DistCP: Options to Avoid.\n\nDistCp's `-atomic` option copies up data into a directory, then renames\nit into place, which is the where the copy takes place. This is a performance\nkiller.\n\n* Do not use the `-atomic` option.\n* The `-append` operation is not supported on S3; avoid.\n* `-p` S3 does not have a POSIX-style permission model; this will fail.\n\n\n### DistCP: Parameters to Tune\n\n1. As discussed [earlier](#pooling), use large values for\n`fs.s3a.threads.max` and `fs.s3a.connection.maximum`.\n\n1. Make sure that the bucket is using `sequential` or `normal` fadvise seek policies,\nthat is, `fs.s3a.experimental.fadvise` is not set to `random`\n\n1. Perform listings in parallel by setting `-numListstatusThreads`\nto a higher number. Make sure that `fs.s3a.connection.maximum`\nis equal to or greater than the value used.\n\n1. If using `-delete`, set `fs.trash.interval` to 0 to avoid the deleted\nobjects from being copied to a trash directory.\n\n*DO NOT* switch `fs.s3a.fast.upload.buffer` to buffer in memory.\nIf one distcp mapper runs out of memory it will fail,\nand that runs the risk of failing the entire job.\nIt is safer to keep the default value, `disk`.\n\nWhat is potentially useful is uploading in bigger blocks; this is more\nefficient in terms of HTTP connection use, and reduce the IOP rate against\nthe S3 bucket/shard.\n\n```xml\n\n fs.s3a.threads.max\n 20\n\n\n\n fs.s3a.connection.maximum\n 30\n \n Make greater than both fs.s3a.threads.max and -numListstatusThreads\n \n\n\n\n fs.s3a.experimental.fadvise\n normal\n\n\n\n fs.s3a.block.size\n 128M\n\n\n\n fs.s3a.fast.upload.buffer\n disk\n\n\n\n fs.trash.interval\n 0\n\n```\n\n## hadoop shell commands `fs -rm`\n\nThe `hadoop fs -rm` command can rename the file under `.Trash` rather than\ndeleting it. Use `-skipTrash` to eliminate that step.\n\n\nThis can be set in the property `fs.trash.interval`; while the default is 0,\nmost HDFS deployments have it set to a non-zero value to reduce the risk of\ndata loss.\n\n```xml\n\n fs.trash.interval\n 0\n\n```\n\n\n## Improving S3 load-balancing behavior\n\nAmazon S3 uses a set of front-end servers to provide access to the underlying data.\nThe choice of which front-end server to use is handled via load-balancing DNS\nservice: when the IP address of an S3 bucket is looked up, the choice of which\nIP address to return to the client is made based on the the current load\nof the front-end servers.\n\nOver time, the load across the front-end changes, so those servers considered\n\"lightly loaded\" will change. If the DNS value is cached for any length of time,\nyour application may end up talking to an overloaded server. Or, in the case\nof failures, trying to talk to a server that is no longer there.\n\nAnd by default, for historical security reasons in the era of applets,\nthe DNS TTL of a JVM is \"infinity\".\n\nTo work with AWS better, set the DNS time-to-live of an application which\nworks with S3 to something lower.\nSee [AWS documentation](http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-jvm-ttl.html).\n\n## Troubleshooting network performance\n\nAn example of this is covered in [HADOOP-13871](https://issues.apache.org/jira/browse/HADOOP-13871).\n\n1. For public data, use `curl`:\n\n curl -O https://landsat-pds.s3.amazonaws.com/scene_list.gz\n1. Use `nettop` to monitor a processes connections.\n\n\n## Throttling\n\nWhen many requests are made of a specific S3 bucket (or shard inside it),\nS3 will respond with a 503 \"throttled\" response.\nThrottling can be recovered from, provided overall load decreases.\nFurthermore, because it is sent before any changes are made to the object store,\nis inherently idempotent. For this reason, the client will always attempt to\nretry throttled requests.\n\nThe limit of the number of times a throttled request can be retried,\nand the exponential interval increase between attempts, can be configured\nindependently of the other retry limits.\n\n```xml\n\n fs.s3a.retry.throttle.limit\n 20\n \n Number of times to retry any throttled request.\n \n\n\n\n fs.s3a.retry.throttle.interval\n 500ms\n \n Interval between retry attempts on throttled requests.\n \n\n```\n\nIf a client is failing due to `AWSServiceThrottledException` failures,\nincreasing the interval and limit *may* address this. However, it\nit is a sign of AWS services being overloaded by the sheer number of clients\nand rate of requests. Spreading data across different buckets, and/or using\na more balanced directory structure may be beneficial.\nConsult [the AWS documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).\n\nReading or writing data encrypted with SSE-KMS forces S3 to make calls of\nthe AWS KMS Key Management Service, which comes with its own\n[Request Rate Limits](http://docs.aws.amazon.com/kms/latest/developerguide/limits.html).\nThese default to 1200/second for an account, across all keys and all uses of\nthem, which, for S3 means: across all buckets with data encrypted with SSE-KMS.\n\n### Tips to Keep Throttling down\n\nIf you are seeing a lot of throttling responses on a large scale\noperation like a `distcp` copy, *reduce* the number of processes trying\nto work with the bucket (for distcp: reduce the number of mappers with the\n`-m` option).\n\nIf you are reading or writing lists of files, if you can randomize\nthe list so they are not processed in a simple sorted order, you may\nreduce load on a specific shard of S3 data, so potentially increase throughput.\n\nAn S3 Bucket is throttled by requests coming from all\nsimultaneous clients. Different applications and jobs may interfere with\neach other: consider that when troubleshooting.\nPartitioning data into different buckets may help isolate load here.\n\nIf you are using data encrypted with SSE-KMS, then the\nwill also apply: these are stricter than the S3 numbers.\nIf you believe that you are reaching these limits, you may be able to\nget them increased.\nConsult [the KMS Rate Limit documentation](http://docs.aws.amazon.com/kms/latest/developerguide/limits.html).\n\n### S3Guard and Throttling\n\n\nS3Guard uses DynamoDB for directory and file lookups;\nit is rate limited to the amount of (guaranteed) IO purchased for a\ntable.\n\nTo see the allocated capacity of a bucket, the `hadoop s3guard bucket-info s3a://bucket`\ncommand will print out the allocated capacity.\n\n\nIf significant throttling events/rate is observed here, the pre-allocated\nIOPs can be increased with the `hadoop s3guard set-capacity` command, or\nthrough the AWS Console. Throttling events in S3Guard are noted in logs, and\nalso in the S3A metrics `s3guard_metadatastore_throttle_rate` and\n`s3guard_metadatastore_throttled`.\n\nIf you are using DistCP for a large backup to/from a S3Guarded bucket, it is\nactually possible to increase the capacity for the duration of the operation.\n\n\n## Best Practises for Code\n\nHere are some best practises if you are writing applications to work with\nS3 or any other object store through the Hadoop APIs.\n\nUse `listFiles(path, recursive)` over `listStatus(path)`.\nThe recursive `listFiles()` call can enumerate all dependents of a path\nin a single LIST call, irrespective of how deep the path is.\nIn contrast, any directory tree-walk implemented in the client is issuing\nmultiple HTTP requests to scan each directory, all the way down.\n\nCache the outcome of `getFileStats()`, rather than repeatedly ask for it.\nThat includes using `isFile()`, `isDirectory()`, which are simply wrappers\naround `getFileStatus()`.\n\nDon't immediately look for a file with a `getFileStatus()` or listing call\nafter creating it, or try to read it immediately.\nThis is where eventual consistency problems surface: the data may not yet be visible.\n\nRely on `FileNotFoundException` being raised if the source of an operation is\nmissing, rather than implementing your own probe for the file before\nconditionally calling the operation.\n\n### `rename()`\n\nAvoid any algorithm which uploads data into a temporary file and then uses\n`rename()` to commit it into place with a final path.\nOn HDFS this offers a fast commit operation.\nWith S3, Wasb and other object stores, you can write straight to the destination,\nknowing that the file isn't visible until you close the write: the write itself\nis atomic.\n\nThe `rename()` operation may return `false` if the source is missing; this\nis a weakness in the API. Consider a check before calling rename, and if/when\na new rename() call is made public, switch to it.\n\n\n### `delete(path, recursive)`\n\nKeep in mind that `delete(path, recursive)` is a no-op if the path does not exist, so\nthere's no need to have a check for the path existing before you call it.\n\n`delete()` is often used as a cleanup operation.\nWith an object store this is slow, and may cause problems if the caller\nexpects an immediate response. For example, a thread may block so long\nthat other liveness checks start to fail.\nConsider spawning off an executor thread to do these background cleanup operations.\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md":"\n\n# Testing the S3A filesystem client and its features, including S3Guard\n\n\n\nThis module includes both unit tests, which can run in isolation without\nconnecting to the S3 service, and integration tests, which require a working\nconnection to S3 to interact with a bucket. Unit test suites follow the naming\nconvention `Test*.java`. Integration tests follow the naming convention\n`ITest*.java`.\n\nDue to eventual consistency, integration tests may fail without reason.\nTransient failures, which no longer occur upon rerunning the test, should thus\nbe ignored.\n\n## Policy for submitting patches which affect the `hadoop-aws` module.\n\nThe Apache Jenkins infrastructure does not run any S3 integration tests,\ndue to the need to keep credentials secure.\n\n### The submitter of any patch is required to run all the integration tests and declare which S3 region/implementation they used.\n\nThis is important: **patches which do not include this declaration will be ignored**\n\nThis policy has proven to be the only mechanism to guarantee full regression\ntesting of code changes. Why the declaration of region? Two reasons\n\n1. It helps us identify regressions which only surface against specific endpoints\nor third-party implementations of the S3 protocol.\n1. It forces the submitters to be more honest about their testing. It's easy\nto lie, \"yes, I tested this\". To say \"yes, I tested this against S3 US-west\"\nis a more specific lie and harder to make. And, if you get caught out: you\nlose all credibility with the project.\n\nYou don't need to test from a VM within the AWS infrastructure; with the\n`-Dparallel=tests` option the non-scale tests complete in under ten minutes.\nBecause the tests clean up after themselves, they are also designed to be low\ncost. It's neither hard nor expensive to run the tests; if you can't,\nthere's no guarantee your patch works. The reviewers have enough to do, and\ndon't have the time to do these tests, especially as every failure will simply\nmake for a slow iterative development.\n\nPlease: run the tests. And if you don't, we are sorry for declining your\npatch, but we have to.\n\n\n### What if there's an intermittent failure of a test?\n\nSome of the tests do fail intermittently, especially in parallel runs.\nIf this happens, try to run the test on its own to see if the test succeeds.\n\nIf it still fails, include this fact in your declaration. We know some tests\nare intermittently unreliable.\n\n### What if the tests are timing out or failing over my network connection?\n\nThe tests and the S3A client are designed to be configurable for different\ntimeouts. If you are seeing problems and this configuration isn't working,\nthat's a sign of the configuration mechanism isn't complete. If it's happening\nin the production code, that could be a sign of a problem which may surface\nover long-haul connections. Please help us identify and fix these problems\n— especially as you are the one best placed to verify the fixes work.\n\n## Setting up the tests\n\nTo integration test the S3* filesystem clients, you need to provide\n`auth-keys.xml` which passes in authentication details to the test runner.\n\nIt is a Hadoop XML configuration file, which must be placed into\n`hadoop-tools/hadoop-aws/src/test/resources`.\n\n### File `core-site.xml`\n\nThis file pre-exists and sources the configurations created\nunder `auth-keys.xml`.\n\nFor most purposes you will not need to edit this file unless you\nneed to apply a specific, non-default property change during the tests.\n\n### File `auth-keys.xml`\n\nThe presence of this file triggers the testing of the S3 classes.\n\nWithout this file, *none of the integration tests in this module will be\nexecuted*.\n\nThe XML file must contain all the ID/key information needed to connect\neach of the filesystem clients to the object stores, and a URL for\neach filesystem for its testing.\n\n1. `test.fs.s3a.name` : the URL of the bucket for S3a tests\n1. `fs.contract.test.fs.s3a` : the URL of the bucket for S3a filesystem contract tests\n\n\nThe contents of the bucket will be destroyed during the test process:\ndo not use the bucket for any purpose other than testing. Furthermore, for\ns3a, all in-progress multi-part uploads to the bucket will be aborted at the\nstart of a test (by forcing `fs.s3a.multipart.purge=true`) to clean up the\ntemporary state of previously failed tests.\n\nExample:\n\n```xml\n\n\n \n test.fs.s3a.name\n s3a://test-aws-s3a/\n \n\n \n fs.contract.test.fs.s3a\n ${test.fs.s3a.name}\n \n\n \n fs.s3a.access.key\n AWS access key ID. Omit for IAM role-based authentication.\n DONOTCOMMITTHISKEYTOSCM\n \n\n \n fs.s3a.secret.key\n AWS secret key. Omit for IAM role-based authentication.\n DONOTEVERSHARETHISSECRETKEY!\n \n\n \n test.sts.endpoint\n Specific endpoint to use for STS requests.\n sts.amazonaws.com\n \n\n\n```\n\n### Configuring S3a Encryption\n\nFor S3a encryption tests to run correctly, the\n`fs.s3a.server-side-encryption.key` must be configured in the s3a contract xml\nfile with a AWS KMS encryption key arn as this value is different for each AWS\nKMS.\n\nExample:\n\n```xml\n\n fs.s3a.server-side-encryption.key\n arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n```\n\nYou can also force all the tests to run with a specific SSE encryption method\nby configuring the property `fs.s3a.server-side-encryption-algorithm` in the s3a\ncontract file.\n\n## Running the Tests\n\nAfter completing the configuration, execute the test run through Maven.\n\n```bash\nmvn clean verify\n```\n\nIt's also possible to execute multiple test suites in parallel by passing the\n`parallel-tests` property on the command line. The tests spend most of their\ntime blocked on network I/O with the S3 service, so running in parallel tends to\ncomplete full test runs faster.\n\n```bash\nmvn -Dparallel-tests clean verify\n```\n\nSome tests must run with exclusive access to the S3 bucket, so even with the\n`parallel-tests` property, several test suites will run in serial in a separate\nMaven execution step after the parallel tests.\n\nBy default, `parallel-tests` runs 4 test suites concurrently. This can be tuned\nby passing the `testsThreadCount` property.\n\n```bash\nmvn -Dparallel-tests -DtestsThreadCount=8 clean verify\n```\n\nTo run just unit tests, which do not require S3 connectivity or AWS credentials,\nuse any of the above invocations, but switch the goal to `test` instead of\n`verify`.\n\n```bash\nmvn clean test\n\nmvn -Dparallel-tests clean test\n\nmvn -Dparallel-tests -DtestsThreadCount=8 clean test\n```\n\nTo run only a specific named subset of tests, pass the `test` property for unit\ntests or the `it.test` property for integration tests.\n\n```bash\nmvn clean test -Dtest=TestS3AInputPolicies\n\nmvn clean verify -Dit.test=ITestS3AFileContextStatistics -Dtest=none\n\nmvn clean verify -Dtest=TestS3A* -Dit.test=ITestS3A*\n```\n\nNote that when running a specific subset of tests, the patterns passed in `test`\nand `it.test` override the configuration of which tests need to run in isolation\nin a separate serial phase (mentioned above). This can cause unpredictable\nresults, so the recommendation is to avoid passing `parallel-tests` in\ncombination with `test` or `it.test`. If you know that you are specifying only\ntests that can run safely in parallel, then it will work. For wide patterns,\nlike `ITestS3A*` shown above, it may cause unpredictable test failures.\n\n### Testing against different regions\n\nS3A can connect to different regions —the tests support this. Simply\ndefine the target region in `auth-keys.xml`.\n\n```xml\n\n fs.s3a.endpoint\n s3.eu-central-1.amazonaws.com\n\n```\nThis is used for all tests expect for scale tests using a Public CSV.gz file\n(see below)\n\n### CSV Data Tests\n\nThe `TestS3AInputStreamPerformance` tests require read access to a multi-MB\ntext file. The default file for these tests is one published by amazon,\n[s3a://landsat-pds.s3.amazonaws.com/scene_list.gz](http://landsat-pds.s3.amazonaws.com/scene_list.gz).\nThis is a gzipped CSV index of other files which amazon serves for open use.\n\nThe path to this object is set in the option `fs.s3a.scale.test.csvfile`,\n\n```xml\n\n fs.s3a.scale.test.csvfile\n s3a://landsat-pds/scene_list.gz\n\n```\n\n1. If the option is not overridden, the default value is used. This\nis hosted in Amazon's US-east datacenter.\n1. If `fs.s3a.scale.test.csvfile` is empty, tests which require it will be skipped.\n1. If the data cannot be read for any reason then the test will fail.\n1. If the property is set to a different path, then that data must be readable\nand \"sufficiently\" large.\n\n(the reason the space or newline is needed is to add \"an empty entry\"; an empty\n`` would be considered undefined and pick up the default)\n\nOf using a test file in an S3 region requiring a different endpoint value\nset in `fs.s3a.endpoint`, a bucket-specific endpoint must be defined.\nFor the default test dataset, hosted in the `landsat-pds` bucket, this is:\n\n```xml\n\n fs.s3a.bucket.landsat-pds.endpoint\n s3.amazonaws.com\n The endpoint for s3a://landsat-pds URLs\n\n```\n\n## Viewing Integration Test Reports\n\n\nIntegration test results and logs are stored in `target/failsafe-reports/`.\nAn HTML report can be generated during site generation, or with the `surefire-report`\nplugin:\n\n```bash\nmvn surefire-report:failsafe-report-only\n```\n## Scale Tests\n\nThere are a set of tests designed to measure the scalability and performance\nat scale of the S3A tests, *Scale Tests*. Tests include: creating\nand traversing directory trees, uploading large files, renaming them,\ndeleting them, seeking through the files, performing random IO, and others.\nThis makes them a foundational part of the benchmarking.\n\nBy their very nature they are slow. And, as their execution time is often\nlimited by bandwidth between the computer running the tests and the S3 endpoint,\nparallel execution does not speed these tests up.\n\n***Note: Running scale tests with -Ds3guard and -Ddynamo requires that\nyou use a private, testing-only DynamoDB table.*** The tests do disruptive\nthings such as deleting metadata and setting the provisioned throughput\nto very low values.\n\n### Enabling the Scale Tests\n\nThe tests are enabled if the `scale` property is set in the maven build\nthis can be done regardless of whether or not the parallel test profile\nis used\n\n```bash\nmvn verify -Dscale\n\nmvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8\n```\n\nThe most bandwidth intensive tests (those which upload data) always run\nsequentially; those which are slow due to HTTPS setup costs or server-side\nactions are included in the set of parallelized tests.\n\n\n### Tuning scale options from Maven\n\n\nSome of the tests can be tuned from the maven build or from the\nconfiguration file used to run the tests.\n\n```bash\nmvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dfs.s3a.scale.test.huge.filesize=128M\n```\n\nThe algorithm is\n\n1. The value is queried from the configuration file, using a default value if\nit is not set.\n1. The value is queried from the JVM System Properties, where it is passed\ndown by maven.\n1. If the system property is null, an empty string, or it has the value `unset`,\nthen the configuration value is used. The `unset` option is used to\n[work round a quirk in maven property propagation](http://stackoverflow.com/questions/7773134/null-versus-empty-arguments-in-maven).\n\nOnly a few properties can be set this way; more will be added.\n\n| Property | Meaning |\n|-----------|-------------|\n| `fs.s3a.scale.test.timeout`| Timeout in seconds for scale tests |\n| `fs.s3a.scale.test.huge.filesize`| Size for huge file uploads |\n| `fs.s3a.scale.test.huge.huge.partitionsize`| Size for partitions in huge file uploads |\n\nThe file and partition sizes are numeric values with a k/m/g/t/p suffix depending\non the desired size. For example: 128M, 128m, 2G, 2G, 4T or even 1P.\n\n### Scale test configuration options\n\nSome scale tests perform multiple operations (such as creating many directories).\n\nThe exact number of operations to perform is configurable in the option\n`scale.test.operation.count`\n\n```xml\n\n scale.test.operation.count\n 10\n\n```\n\nLarger values generate more load, and are recommended when testing locally,\nor in batch runs.\n\nSmaller values results in faster test runs, especially when the object\nstore is a long way away.\n\nOperations which work on directories have a separate option: this controls\nthe width and depth of tests creating recursive directories. Larger\nvalues create exponentially more directories, with consequent performance\nimpact.\n\n```xml\n\n scale.test.directory.count\n 2\n\n```\n\nDistCp tests targeting S3A support a configurable file size. The default is\n10 MB, but the configuration value is expressed in KB so that it can be tuned\nsmaller to achieve faster test runs.\n\n```xml\n\n scale.test.distcp.file.size.kb\n 10240\n\n```\n\nS3A specific scale test properties are\n\n*`fs.s3a.scale.test.huge.filesize`: size in MB for \"Huge file tests\".*\n\nThe Huge File tests validate S3A's ability to handle large files —the property\n`fs.s3a.scale.test.huge.filesize` declares the file size to use.\n\n```xml\n\n fs.s3a.scale.test.huge.filesize\n 200M\n\n```\n\nAmazon S3 handles files larger than 5GB differently than smaller ones.\nSetting the huge filesize to a number greater than that) validates support\nfor huge files.\n\n```xml\n\n fs.s3a.scale.test.huge.filesize\n 6G\n\n```\n\nTests at this scale are slow: they are best executed from hosts running in\nthe cloud infrastructure where the S3 endpoint is based.\nOtherwise, set a large timeout in `fs.s3a.scale.test.timeout`\n\n```xml\n\n fs.s3a.scale.test.timeout\n 432000\n\n```\n\nThe tests are executed in an order to only clean up created files after\nthe end of all the tests. If the tests are interrupted, the test data will remain.\n\n\n## Testing against non AWS S3 endpoints.\n\nThe S3A filesystem is designed to work with storage endpoints which implement\nthe S3 protocols to the extent that the amazon S3 SDK is capable of talking\nto it. We encourage testing against other filesystems and submissions of patches\nwhich address issues. In particular, we encourage testing of Hadoop release\ncandidates, as these third-party endpoints get even less testing than the\nS3 endpoint itself.\n\n\n### Disabling the encryption tests\n\nIf the endpoint doesn't support server-side-encryption, these will fail. They\ncan be turned off.\n\n```xml\n\n test.fs.s3a.encryption.enabled\n false\n\n```\n\nEncryption is only used for those specific test suites with `Encryption` in\ntheir classname.\n\n### Configuring the CSV file read tests**\n\nTo test on alternate infrastructures supporting\nthe same APIs, the option `fs.s3a.scale.test.csvfile` must either be\nset to \" \", or an object of at least 10MB is uploaded to the object store, and\nthe `fs.s3a.scale.test.csvfile` option set to its path.\n\n```xml\n\n fs.s3a.scale.test.csvfile\n \n\n```\n\n(yes, the space is necessary. The Hadoop `Configuration` class treats an empty\nvalue as \"do not override the default\").\n\n\n### Testing Session Credentials\n\nThe test `TestS3ATemporaryCredentials` requests a set of temporary\ncredentials from the STS service, then uses them to authenticate with S3.\n\nIf an S3 implementation does not support STS, then the functional test\ncases must be disabled:\n\n```xml\n\n test.fs.s3a.sts.enabled\n false\n\n```\nThese tests request a temporary set of credentials from the STS service endpoint.\nAn alternate endpoint may be defined in `test.fs.s3a.sts.endpoint`.\n\n```xml\n\n test.fs.s3a.sts.endpoint\n https://sts.example.org/\n\n```\nThe default is \"\"; meaning \"use the amazon default value\".\n\n\n## Debugging Test failures\n\nLogging at debug level is the standard way to provide more diagnostics output;\nafter setting this rerun the tests\n\n```properties\nlog4j.logger.org.apache.hadoop.fs.s3a=DEBUG\n```\n\nThere are also some logging options for debug logging of the AWS client\n```properties\nlog4j.logger.com.amazonaws=DEBUG\nlog4j.logger.com.amazonaws.http.conn.ssl=INFO\nlog4j.logger.com.amazonaws.internal=INFO\n```\n\nThere is also the option of enabling logging on a bucket; this could perhaps\nbe used to diagnose problems from that end. This isn't something actively\nused, but remains an option. If you are forced to debug this way, consider\nsetting the `fs.s3a.user.agent.prefix` to a unique prefix for a specific\ntest run, which will enable the specific log entries to be more easily\nlocated.\n\n## Adding new tests\n\nNew tests are always welcome. Bear in mind that we need to keep costs\nand test time down, which is done by\n\n* Not duplicating tests.\n* Being efficient in your use of Hadoop API calls.\n* Isolating large/slow tests into the \"scale\" test group.\n* Designing all tests to execute in parallel (where possible).\n* Adding new probes and predicates into existing tests, albeit carefully.\n\n*No duplication*: if an operation is tested elsewhere, don't repeat it. This\napplies as much for metadata operations as it does for bulk IO. If a new\ntest case is added which completely obsoletes an existing test, it is OK\nto cut the previous one —after showing that coverage is not worsened.\n\n*Efficient*: prefer the `getFileStatus()` and examining the results, rather than\ncall to `exists()`, `isFile()`, etc.\n\n*Isolating Scale tests*. Any S3A test doing large amounts of IO MUST extend the\nclass `S3AScaleTestBase`, so only running if `scale` is defined on a build,\nsupporting test timeouts configurable by the user. Scale tests should also\nsupport configurability as to the actual size of objects/number of operations,\nso that behavior at different scale can be verified.\n\n*Designed for parallel execution*. A key need here is for each test suite to work\non isolated parts of the filesystem. Subclasses of `AbstractS3ATestBase`\nSHOULD use the `path()` method, with a base path of the test suite name, to\nbuild isolated paths. Tests MUST NOT assume that they have exclusive access\nto a bucket.\n\n*Extending existing tests where appropriate*. This recommendation goes\nagainst normal testing best practise of \"test one thing per method\".\nBecause it is so slow to create directory trees or upload large files, we do\nnot have that luxury. All the tests against real S3 endpoints are integration\ntests where sharing test setup and teardown saves time and money.\n\nA standard way to do this is to extend existing tests with some extra predicates,\nrather than write new tests. When doing this, make sure that the new predicates\nfail with meaningful diagnostics, so any new problems can be easily debugged\nfrom test logs.\n\n\n## Requirements of new Tests\n\n\nThis is what we expect from new tests; they're an extension of the normal\nHadoop requirements, based on the need to work with remote servers whose\nuse requires the presence of secret credentials, where tests may be slow,\nand where finding out why something failed from nothing but the test output\nis critical.\n\n### Subclasses Existing Shared Base Classes\n\nExtend `AbstractS3ATestBase` or `AbstractSTestS3AHugeFiles` unless justifiable.\nThese set things up for testing against the object stores, provide good threadnames,\nhelp generate isolated paths, and for `AbstractSTestS3AHugeFiles` subclasses,\nonly run if `-Dscale` is set.\n\nKey features of `AbstractS3ATestBase`\n\n* `getFileSystem()` returns the S3A Filesystem bonded to the contract test Filesystem\ndefined in `fs.s3a.contract.test`\n* will automatically skip all tests if that URL is unset.\n* Extends `AbstractFSContractTestBase` and `Assert` for all their methods.\n\nHaving shared base classes may help reduce future maintenance too. Please\nuse them/\n\n### Secure\n\nDon't ever log credentials. The credential tests go out of their way to\nnot provide meaningful logs or assertion messages precisely to avoid this.\n\n### Efficient of Time and Money\n\nThis means efficient in test setup/teardown, and, ideally, making use of\nexisting public datasets to save setup time and tester cost.\n\nStrategies of particular note are:\n\n1. `ITestS3ADirectoryPerformance`: a single test case sets up the directory\ntree then performs different list operations, measuring the time taken.\n1. `AbstractSTestS3AHugeFiles`: marks the test suite as\n`@FixMethodOrder(MethodSorters.NAME_ASCENDING)` then orders the test cases such\nthat each test case expects the previous test to have completed (here: uploaded a file,\nrenamed a file, ...). This provides for independent tests in the reports, yet still\npermits an ordered sequence of operations. Do note the use of `Assume.assume()`\nto detect when the preconditions for a single test case are not met, hence,\nthe tests become skipped, rather than fail with a trace which is really a false alarm.\n\nThe ordered test case mechanism of `AbstractSTestS3AHugeFiles` is probably\nthe most elegant way of chaining test setup/teardown.\n\nRegarding reusing existing data, we tend to use the landsat archive of\nAWS US-East for our testing of input stream operations. This doesn't work\nagainst other regions, or with third party S3 implementations. Thus the\nURL can be overridden for testing elsewhere.\n\n\n### Works With Other S3 Endpoints\n\nDon't assume AWS S3 US-East only, do allow for working with external S3 implementations.\nThose may be behind the latest S3 API features, not support encryption, session\nAPIs, etc.\n\nThey won't have the same CSV test files as some of the input tests rely on.\nLook at `ITestS3AInputStreamPerformance` to see how tests can be written\nto support the declaration of a specific large test file on alternate filesystems.\n\n\n### Works Over Long-haul Links\n\nAs well as making file size and operation counts scalable, this includes\nmaking test timeouts adequate. The Scale tests make this configurable; it's\nhard coded to ten minutes in `AbstractS3ATestBase()`; subclasses can\nchange this by overriding `getTestTimeoutMillis()`.\n\nEqually importantly: support proxies, as some testers need them.\n\n\n### Provides Diagnostics and timing information\n\n1. Give threads useful names.\n1. Create logs, log things. Know that the `S3AFileSystem` and its input\nand output streams *all* provide useful statistics in their {{toString()}}\ncalls; logging them is useful on its own.\n1. you can use `AbstractS3ATestBase.describe(format-stringm, args)` here.; it\nadds some newlines so as to be easier to spot.\n1. Use `ContractTestUtils.NanoTimer` to measure the duration of operations,\nand log the output.\n\n### Fails Meaningfully\n\nThe `ContractTestUtils` class contains a whole set of assertions for making\nstatements about the expected state of a filesystem, e.g.\n`assertPathExists(FS, path)`, `assertPathDoesNotExists(FS, path)`, and others.\nThese do their best to provide meaningful diagnostics on failures (e.g. directory\nlistings, file status, ...), so help make failures easier to understand.\n\nAt the very least, do not use `assertTrue()` or `assertFalse()` without\nincluding error messages.\n\n### Sets up its filesystem and checks for those settings\n\nTests can overrun `createConfiguration()` to add new options to the configuration\nfile for the S3A Filesystem instance used in their tests.\n\nHowever, filesystem caching may mean that a test suite may get a cached\ninstance created with an different configuration. For tests which don't need\nspecific configurations caching is good: it reduces test setup time.\n\nFor those tests which do need unique options (encryption, magic files),\nthings can break, and they will do so in hard-to-replicate ways.\n\nUse `S3ATestUtils.disableFilesystemCaching(conf)` to disable caching when\nmodifying the config. As an example from `AbstractTestS3AEncryption`:\n\n```java\n@Override\nprotected Configuration createConfiguration() {\n Configuration conf = super.createConfiguration();\n S3ATestUtils.disableFilesystemCaching(conf);\n conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM,\n getSSEAlgorithm().getMethod());\n return conf;\n}\n```\n\nThen verify in the setup method or test cases that their filesystem actually has\nthe desired feature (`fs.getConf().getProperty(...)`). This not only\ncatches filesystem reuse problems, it catches the situation where the\nfilesystem configuration in `auth-keys.xml` has explicit per-bucket settings\nwhich override the test suite's general option settings.\n\n### Cleans Up Afterwards\n\nKeeps costs down.\n\n1. Do not only cleanup if a test case completes successfully; test suite\nteardown must do it.\n1. That teardown code must check for the filesystem and other fields being\nnull before the cleanup. Why? If test setup fails, the teardown methods still\nget called.\n\n### Works Reliably\n\nWe really appreciate this — you will too.\n\n### Runs in parallel unless this is unworkable.\n\nTests must be designed to run in parallel with other tests, all working\nwith the same shared S3 bucket. This means\n\n* Uses relative and JVM-fork-unique paths provided by the method\n `AbstractFSContractTestBase.path(String filepath)`.\n* Doesn't manipulate the root directory or make assertions about its contents\n(for example: delete its contents and assert that it is now empty).\n* Doesn't have a specific requirement of all active clients of the bucket\n(example: SSE-C tests which require all files, even directory markers,\nto be encrypted with the same key).\n* Doesn't use so much bandwidth that all other tests will be starved of IO and\nstart timing out (e.g. the scale tests).\n\nTests such as these can only be run as sequential tests. When adding one,\nexclude it in the POM file. from the parallel failsafe run and add to the\nsequential one afterwards. The IO heavy ones must also be subclasses of\n`S3AScaleTestBase` and so only run if the system/maven property\n`fs.s3a.scale.test.enabled` is true.\n\n## Individual test cases can be run in an IDE\n\nThis is invaluable for debugging test failures.\n\n### Keeping AWS Costs down\n\nMost of the base S3 tests are designed to use public AWS data\n(the landsat-pds bucket) for read IO, so you don't have to pay for bytes\ndownloaded or long term storage costs. The scale tests do work with more data\nso will cost more as well as generally take more time to execute.\n\nYou are however billed for\n\n1. Data left in S3 after test runs.\n2. DynamoDB capacity reserved by S3Guard tables.\n3. HTTP operations on files (HEAD, LIST, GET).\n4. In-progress multipart uploads from bulk IO or S3A committer tests.\n5. Encryption/decryption using AWS KMS keys.\n\nThe GET/decrypt costs are incurred on each partial read of a file,\nso random IO can cost more than sequential IO; the speedup of queries with\ncolumnar data usually justifies this.\n\nThe DynamoDB costs come from the number of entries stores and the allocated capacity.\n\nHow to keep costs down\n\n* Don't run the scale tests with large datasets; keep `fs.s3a.scale.test.huge.filesize` unset, or a few MB (minimum: 5).\n* Remove all files in the filesystem. The root tests usually do this, but\nit can be manually done:\n\n hadoop fs -rm -r -f -skipTrash s3a://test-bucket/\\*\n* Abort all outstanding uploads:\n\n hadoop s3guard uploads -abort -force s3a://test-bucket/\n* If you don't need it, destroy the S3Guard DDB table.\n\n hadoop s3guard destroy s3a://hwdev-steve-ireland-new/\n\nThe S3Guard tests will automatically create the Dynamo DB table in runs with\n`-Ds3guard -Ddynamodb` set; default capacity of these buckets\ntests is very small; it keeps costs down at the expense of IO performance\nand, for test runs in or near the S3/DDB stores, throttling events.\n\nIf you want to manage capacity, use `s3guard set-capacity` to increase it\n(performance) or decrease it (costs).\nFor remote `hadoop-aws` test runs, the read/write capacities of \"10\" each should suffice;\nincrease it if parallel test run logs warn of throttling.\n\nTip: for agility, use DynamoDB autoscaling, setting the minimum to something very low (e.g 5 units), the maximum to the largest amount you are willing to pay.\nThis will automatically reduce capacity when you are not running tests against\nthe bucket, slowly increase it over multiple test runs, if the load justifies it.\n\n## Tips\n\n### How to keep your credentials really safe\n\nAlthough the `auth-keys.xml` file is marked as ignored in git and subversion,\nit is still in your source tree, and there's always that risk that it may\ncreep out.\n\nYou can avoid this by keeping your keys outside the source tree and\nusing an absolute XInclude reference to it.\n\n```xml\n\n\n \n\n\n```\n\n# Failure Injection\n\n**Warning do not enable any type of failure injection in production. The\nfollowing settings are for testing only.**\n\nOne of the challenges with S3A integration tests is the fact that S3 is an\neventually-consistent storage system. In practice, we rarely see delays in\nvisibility of recently created objects both in listings (`listStatus()`) and\nwhen getting a single file's metadata (`getFileStatus()`). Since this behavior\nis rare and non-deterministic, thorough integration testing is challenging.\n\nTo address this, S3A supports a shim layer on top of the `AmazonS3Client`\nclass which artificially delays certain paths from appearing in listings.\nThis is implemented in the class `InconsistentAmazonS3Client`.\n\n## Simulating List Inconsistencies\n\n### Enabling the InconsistentAmazonS3CClient\n\nThere are two ways of enabling the `InconsistentAmazonS3Client`: at\nconfig-time, or programmatically. For an example of programmatic test usage,\nsee `ITestS3GuardListConsistency`.\n\nTo enable the fault-injecting client via configuration, switch the\nS3A client to use the \"Inconsistent S3 Client Factory\" when connecting to\nS3:\n\n```xml\n\n fs.s3a.s3.client.factory.impl\n org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory\n\n```\n\nThe inconsistent client works by:\n\n1. Choosing which objects will be \"inconsistent\" at the time the object is\ncreated or deleted.\n2. When `listObjects()` is called, any keys that we have marked as\ninconsistent above will not be returned in the results (until the\nconfigured delay has elapsed). Similarly, deleted items may be *added* to\nmissing results to delay the visibility of the delete.\n\nThere are two ways of choosing which keys (filenames) will be affected: By\nsubstring, and by random probability.\n\n```xml\n\n fs.s3a.failinject.inconsistency.key.substring\n DELAY_LISTING_ME\n\n\n\n fs.s3a.failinject.inconsistency.probability\n 1.0\n\n```\n\nBy default, any object which has the substring \"DELAY_LISTING_ME\" in its key\nwill subject to delayed visibility. For example, the path\n`s3a://my-bucket/test/DELAY_LISTING_ME/file.txt` would match this condition.\nTo match all keys use the value \"\\*\" (a single asterisk). This is a special\nvalue: *We don't support arbitrary wildcards.*\n\nThe default probability of delaying an object is 1.0. This means that *all*\nkeys that match the substring will get delayed visibility. Note that we take\nthe logical *and* of the two conditions (substring matches *and* probability\nrandom chance occurs). Here are some example configurations:\n\n```\n| substring | probability | behavior |\n|-----------|-------------|--------------------------------------------|\n| | 0.001 | An empty tag in .xml config will |\n| | | be interpreted as unset and revert to the |\n| | | default value, \"DELAY_LISTING_ME\" |\n| | | |\n| * | 0.001 | 1/1000 chance of *any* key being delayed. |\n| | | |\n| delay | 0.01 | 1/100 chance of any key containing \"delay\" |\n| | | |\n| delay | 1.0 | All keys containing substring \"delay\" .. |\n```\n\nYou can also configure how long you want the delay in visibility to last.\nThe default is 5000 milliseconds (five seconds).\n\n```xml\n\n fs.s3a.failinject.inconsistency.msec\n 5000\n\n```\n\nFuture versions of this client will introduce new failure modes,\nwith simulation of S3 throttling exceptions the next feature under\ndevelopment.\n\n### Limitations of Inconsistency Injection\n\nAlthough `InconsistentAmazonS3Client` can delay the visibility of an object\nor parent directory, it does not prevent the key of that object from\nappearing in all prefix searches. For example, if we create the following\nobject with the default configuration above, in an otherwise empty bucket:\n\n```\ns3a://bucket/a/b/c/DELAY_LISTING_ME\n```\n\nThen the following paths will still be visible as directories (ignoring\npossible real-world inconsistencies):\n\n```\ns3a://bucket/a\ns3a://bucket/a/b\n```\n\nWhereas `getFileStatus()` on the following *will* be subject to delayed\nvisibility (`FileNotFoundException` until delay has elapsed):\n\n```\ns3a://bucket/a/b/c\ns3a://bucket/a/b/c/DELAY_LISTING_ME\n```\n\nIn real-life S3 inconsistency, however, we expect that all the above paths\n(including `a` and `b`) will be subject to delayed visibility.\n\n### Using the `InconsistentAmazonS3CClient` in downstream integration tests\n\nThe inconsistent client is shipped in the `hadoop-aws` JAR, so it can\nbe used in applications which work with S3 to see how they handle\ninconsistent directory listings.\n\n## Testing S3Guard\n\n[S3Guard](./s3guard.html) is an extension to S3A which adds consistent metadata\nlistings to the S3A client. As it is part of S3A, it also needs to be tested.\n\nThe basic strategy for testing S3Guard correctness consists of:\n\n1. MetadataStore Contract tests.\n\n The MetadataStore contract tests are inspired by the Hadoop FileSystem and\n `FileContext` contract tests. Each implementation of the `MetadataStore` interface\n subclasses the `MetadataStoreTestBase` class and customizes it to initialize\n their MetadataStore. This test ensures that the different implementations\n all satisfy the semantics of the MetadataStore API.\n\n2. Running existing S3A unit and integration tests with S3Guard enabled.\n\n You can run the S3A integration tests on top of S3Guard by configuring your\n `MetadataStore` in your\n `hadoop-tools/hadoop-aws/src/test/resources/core-site.xml` or\n `hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml` files.\n Next run the S3A integration tests as outlined in the *Running the Tests* section\n of the [S3A documentation](./index.html)\n\n3. Running fault-injection tests that test S3Guard's consistency features.\n\n The `ITestS3GuardListConsistency` uses failure injection to ensure\n that list consistency logic is correct even when the underlying storage is\n eventually consistent.\n\n The integration test adds a shim above the Amazon S3 Client layer that injects\n delays in object visibility.\n\n All of these tests will be run if you follow the steps listed in step 2 above.\n\n No charges are incurred for using this store, and its consistency\n guarantees are that of the underlying object store instance. \n\n## Testing S3A with S3Guard Enabled\n\nAll the S3A tests which work with a private repository can be configured to\nrun with S3Guard by using the `s3guard` profile. When set, this will run\nall the tests with local memory for the metadata set to \"non-authoritative\" mode.\n\n```bash\nmvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard\n```\n\nWhen the `s3guard` profile is enabled, following profiles can be specified:\n\n* `dynamo`: use an AWS-hosted DynamoDB table; creating the table if it does\n not exist. You will have to pay the bills for DynamoDB web service.\n* `dynamodblocal`: use an in-memory DynamoDBLocal server instead of real AWS\n DynamoDB web service; launch the server and creating the table.\n You won't be charged bills for using DynamoDB in test. As it runs in-JVM,\n the table isn't shared across other tests running in parallel.\n* `non-auth`: treat the S3Guard metadata as authoritative.\n\n```bash\nmvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Ddynamo -Dauth\n```\n\nWhen experimenting with options, it is usually best to run a single test suite\nat a time until the operations appear to be working.\n\n```bash\nmvn -T 1C verify -Dtest=skip -Dit.test=ITestS3AMiscOperations -Ds3guard -Ddynamo\n```\n\n### Notes\n\n1. If the `s3guard` profile is not set, then the S3Guard properties are those\nof the test configuration set in `contract-test-options.xml` or `auth-keys.xml`\n\nIf the `s3guard` profile *is* set,\n1. The S3Guard options from maven (the dynamo and authoritative flags)\n overwrite any previously set in the configuration files.\n1. DynamoDB will be configured to create any missing tables.\n\n\n### Scale Testing MetadataStore Directly\n\nThere are some scale tests that exercise Metadata Store implementations\ndirectly. These ensure that S3Guard is are robust to things like DynamoDB\nthrottling, and compare performance for different implementations. These\nare included in the scale tests executed when `-Dscale` is passed to\nthe maven command line.\n\nThe two S3Guard scale tests are `ITestDynamoDBMetadataStoreScale` and\n`ITestLocalMetadataStoreScale`.\n\nTo run these tests, your DynamoDB table needs to be of limited capacity;\nthe values in `ITestDynamoDBMetadataStoreScale` currently require a read capacity\nof 10 or less. a write capacity of 15 or more.\n\nThe following settings allow us to run `ITestDynamoDBMetadataStoreScale` with\nartificially low read and write capacity provisioned, so we can judge the\neffects of being throttled by the DynamoDB service:\n\n```xml\n\n scale.test.operation.count\n 10\n\n\n scale.test.directory.count\n 3\n\n\n fs.s3a.scale.test.enabled\n true\n\n\n fs.s3a.s3guard.ddb.table\n my-scale-test\n\n\n fs.s3a.s3guard.ddb.table.create\n true\n\n\n fs.s3a.s3guard.ddb.table.capacity.read\n 5\n\n\n fs.s3a.s3guard.ddb.table.capacity.write\n 5\n\n```\n\nThese tests verify that the invoked operations can trigger retries in the\nS3Guard code, rather than just in the AWS SDK level, so showing that if\nSDK operations fail, they get retried. They also verify that the filesystem\nstatistics are updated to record that throttling took place.\n\n*Do not panic if these tests fail to detect throttling!*\n\nThese tests are unreliable as they need certain conditions to be met\nto repeatedly fail:\n\n1. You must have a low-enough latency connection to the DynamoDB store that,\nfor the capacity allocated, you can overload it.\n1. The AWS Console can give you a view of what is happening here.\n1. Running a single test on its own is less likely to trigger an overload\nthan trying to run the whole test suite.\n1. And running the test suite more than once, back-to-back, can also help\noverload the cluster.\n1. Stepping through with a debugger will reduce load, so may not trigger\nfailures.\n\nIf the tests fail, it *probably* just means you aren't putting enough load\non the table.\n\nThese tests do not verify that the entire set of DynamoDB calls made\nduring the use of a S3Guarded S3A filesystem are wrapped by retry logic.\n\n*The best way to verify resilience is to run the entire `hadoop-aws` test suite,\nor even a real application, with throttling enabled.\n\n### Testing only: Local Metadata Store\n\nThere is an in-memory Metadata Store for testing.\n\n```xml\n\n fs.s3a.metadatastore.impl\n org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore\n\n```\n\nThis is not for use in production.\n\n## Testing Assumed Roles\n\nTests for the AWS Assumed Role credential provider require an assumed\nrole to request.\n\nIf this role is not set, the tests which require it will be skipped.\n\nTo run the tests in `ITestAssumeRole`, you need:\n\n1. A role in your AWS account will full read and write access rights to\nthe S3 bucket used in the tests, and ideally DynamoDB, for S3Guard.\nIf your bucket is set up by default to use S3Guard, the role must have access\nto that service.\n\n1. Your IAM User to have the permissions to adopt that role.\n\n1. The role ARN must be set in `fs.s3a.assumed.role.arn`.\n\n```xml\n\n fs.s3a.assumed.role.arn\n arn:aws:iam::9878543210123:role/role-s3-restricted\n\n```\n\nThe tests assume the role with different subsets of permissions and verify\nthat the S3A client (mostly) works when the caller has only write access\nto part of the directory tree.\n\nYou can also run the entire test suite in an assumed role, a more\nthorough test, by switching to the credentials provider.\n\n```xml\n\n fs.s3a.aws.credentials.provider\n org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n\n```\n\nThe usual credentials needed to log in to the bucket will be used, but now\nthe credentials used to interact with S3 and DynamoDB will be temporary\nrole credentials, rather than the full credentials.\n\n## Qualifying an AWS SDK Update\n\nUpdating the AWS SDK is something which does need to be done regularly,\nbut is rarely without complications, major or minor.\n\nAssume that the version of the SDK will remain constant for an X.Y release,\nexcluding security fixes, so it's good to have an update before each release\n— as long as that update works doesn't trigger any regressions.\n\n\n1. Don't make this a last minute action.\n1. The upgrade patch should focus purely on the SDK update, so it can be cherry\npicked and reverted easily.\n1. Do not mix in an SDK update with any other piece of work, for the same reason.\n1. Plan for an afternoon's work, including before/after testing, log analysis\nand any manual tests.\n1. Make sure all the integration tests are running (including s3guard, ARN, encryption, scale)\n *before you start the upgrade*.\n1. Create a JIRA for updating the SDK. Don't include the version (yet),\nas it may take a couple of SDK updates before it is ready.\n1. Identify the latest AWS SDK [available for download](https://aws.amazon.com/sdk-for-java/).\n1. Create a private git branch of trunk for JIRA, and in\n `hadoop-project/pom.xml` update the `aws-java-sdk.version` to the new SDK version.\n1. Do a clean build and rerun all the `hadoop-aws` tests, with and without the `-Ds3guard -Ddynamodb` options.\n This includes the `-Pscale` set, with a role defined for the assumed role tests.\n in `fs.s3a.assumed.role.arn` for testing assumed roles,\n and `fs.s3a.server-side-encryption.key` for encryption, for full coverage.\n If you can, scale up the scale tests.\n1. Create the site with `mvn site -DskipTests`; look in `target/site` for the report.\n1. Review *every single `-output.txt` file in `hadoop-tools/hadoop-aws/target/failsafe-reports`,\n paying particular attention to\n `org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance-output.txt`,\n as that is where changes in stream close/abort logic will surface.\n1. Run `mvn install` to install the artifacts, then in\n `hadoop-cloud-storage-project/hadoop-cloud-storage` run\n `mvn dependency:tree -Dverbose > target/dependencies.txt`.\n Examine the `target/dependencies.txt` file to verify that no new\n artifacts have unintentionally been declared as dependencies\n of the shaded `aws-java-sdk-bundle` artifact.\n\n### Basic command line regression testing\n\nWe need a run through of the CLI to see if there have been changes there\nwhich cause problems, especially whether new log messages have surfaced,\nor whether some packaging change breaks that CLI\n\nFrom the root of the project, create a command line release `mvn package -Pdist -DskipTests -Dmaven.javadoc.skip=true -DskipShade`;\n\n1. Change into the `hadoop/dist/target/hadoop-x.y.z-SNAPSHOT` dir.\n1. Copy a `core-site.xml` file into `etc/hadoop`.\n1. Set the `HADOOP_OPTIONAL_TOOLS` env var on the command line or `~/.hadoop-env`.\n\n```bash\nexport HADOOP_OPTIONAL_TOOLS=\"hadoop-aws\"\n```\n\nRun some basic s3guard commands as well as file operations.\n\n```bash\nexport BUCKET=s3a://example-bucket-name\n\nbin/hadoop s3guard bucket-info $BUCKET\nbin/hadoop s3guard set-capacity $BUCKET\nbin/hadoop s3guard set-capacity -read 15 -write 15 $BUCKET\nbin/hadoop s3guard uploads $BUCKET\nbin/hadoop s3guard diff $BUCKET/\nbin/hadoop s3guard prune -minutes 10 $BUCKET/\nbin/hadoop s3guard import $BUCKET/\nbin/hadoop fs -ls $BUCKET/\nbin/hadoop fs -ls $BUCKET/file\nbin/hadoop fs -rm -R -f $BUCKET/dir-no-trailing\nbin/hadoop fs -rm -R -f $BUCKET/dir-trailing/\nbin/hadoop fs -rm $BUCKET/\nbin/hadoop fs -touchz $BUCKET/file\n# expect I/O error as root dir is not empty\nbin/hadoop fs -rm -r $BUCKET/\nbin/hadoop fs -rm -r $BUCKET/\\*\n# now success\nbin/hadoop fs -rm -r $BUCKET/\n\nbin/hadoop fs -mkdir $BUCKET/dir-no-trailing\n# fails with S3Guard\nbin/hadoop fs -mkdir $BUCKET/dir-trailing/\nbin/hadoop fs -touchz $BUCKET/file\nbin/hadoop fs -ls $BUCKET/\nbin/hadoop fs -mv $BUCKET/file $BUCKET/file2\n# expect \"No such file or directory\"\nbin/hadoop fs -stat $BUCKET/file\nbin/hadoop fs -stat $BUCKET/file2\nbin/hadoop fs -mv $BUCKET/file2 $BUCKET/dir-no-trailing\nbin/hadoop fs -stat $BUCKET/dir-no-trailing/file2\n# treated the same as the file stat\nbin/hadoop fs -stat $BUCKET/dir-no-trailing/file2/\nbin/hadoop fs -ls $BUCKET/dir-no-trailing/file2/\nbin/hadoop fs -ls $BUCKET/dir-no-trailing\n# expect a \"0\" here:\nbin/hadoop fs -test -d $BUCKET/dir-no-trailing && echo $?\n# expect a \"1\" here:\nbin/hadoop fs -test -d $BUCKET/dir-no-trailing/file2 && echo $?\n# will return NONE unless bucket has checksums enabled\nbin/hadoop fs -checksum $BUCKET/dir-no-trailing/file2\n# expect \"etag\" + a long string\nbin/hadoop fs -D fs.s3a.etag.checksum.enabled=true -checksum $BUCKET/dir-no-trailing/file2\n```\n\n### Other tests\n\n* Whatever applications you have which use S3A: build and run them before the upgrade,\nThen see if complete successfully in roughly the same time once the upgrade is applied.\n* Test any third-party endpoints you have access to.\n* Try different regions (especially a v4 only region), and encryption settings.\n* Any performance tests you have can identify slowdowns, which can be a sign\n of changed behavior in the SDK (especially on stream reads and writes).\n* If you can, try to test in an environment where a proxy is needed to talk\nto AWS services.\n* Try and get other people, especially anyone with their own endpoints,\n apps or different deployment environments, to run their own tests.\n\n### Dealing with Deprecated APIs and New Features\n\nA Jenkins run should tell you if there are new deprecations.\nIf so, you should think about how to deal with them.\n\nMoving to methods and APIs which weren't in the previous SDK release makes it\nharder to roll back if there is a problem; but there may be good reasons\nfor the deprecation.\n\nAt the same time, there may be good reasons for staying with the old code.\n\n* AWS have embraced the builder pattern for new operations; note that objects\nconstructed this way often have their (existing) setter methods disabled; this\nmay break existing code.\n* New versions of S3 calls (list v2, bucket existence checks, bulk operations)\nmay be better than the previous HTTP operations & APIs, but they may not work with\nthird-party endpoints, so can only be adopted if made optional, which then\nadds a new configuration option (with docs, testing, ...). A change like that\nmust be done in its own patch, with its new tests which compare the old\nvs new operations.\n\n### Committing the patch\n\nWhen the patch is committed: update the JIRA to the version number actually\nused; use that title in the commit message.\n\nBe prepared to roll-back, re-iterate or code your way out of a regression.\n\nThere may be some problem which surfaces with wider use, which can get\nfixed in a new AWS release, "},"files":{"hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md":"\n\n# Working with IAM Assumed Roles\n\n\n\nAWS [\"IAM Assumed Roles\"](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)\nallows applications to change the AWS role with which to authenticate with AWS services.\nThe assumed roles can have different rights from the main user login.\n\nThe S3A connector supports assumed roles for authentication with AWS.\nA full set of login credentials must be provided, which will be used\nto obtain the assumed role and refresh it regularly.\nBy using per-filesystem configuration, it is possible to use different\nassumed roles for different buckets.\n\n*IAM Assumed Roles are unlikely to be supported by third-party systems\nsupporting the S3 APIs.*\n\n## Using IAM Assumed Roles\n\n### Before You Begin\n\nThis document assumes you know about IAM Assumed roles, what they\nare, how to configure their policies, etc.\n\n* You need a role to assume, and know its \"ARN\".\n* You need a pair of long-lived IAM User credentials, not the root account set.\n* Have the AWS CLI installed, and test that it works there.\n* Give the role access to S3, and, if using S3Guard, to DynamoDB.\n* For working with data encrypted with SSE-KMS, the role must\nhave access to the appropriate KMS keys.\n\nTrying to learn how IAM Assumed Roles work by debugging stack traces from\nthe S3A client is \"suboptimal\".\n\n### How the S3A connector support IAM Assumed Roles.\n\nTo use assumed roles, the client must be configured to use the\n*Assumed Role Credential Provider*, `org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider`,\nin the configuration option `fs.s3a.aws.credentials.provider`.\n\nThis AWS Credential provider will read in the `fs.s3a.assumed.role` options needed to connect to the\nSecurity Token Service [Assumed Role API](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html),\nfirst authenticating with the full credentials, then assuming the specific role\nspecified. It will then refresh this login at the configured rate of\n`fs.s3a.assumed.role.session.duration`\n\nTo authenticate with the [AWS STS service](https://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html)\nboth for the initial credential retrieval\nand for background refreshes, a different credential provider must be\ncreated, one which uses long-lived credentials (secret keys, environment variables).\nShort lived credentials (e.g other session tokens, EC2 instance credentials) cannot be used.\n\nA list of providers can be set in `s.s3a.assumed.role.credentials.provider`;\nif unset the standard `BasicAWSCredentialsProvider` credential provider is used,\nwhich uses `fs.s3a.access.key` and `fs.s3a.secret.key`.\n\nNote: although you can list other AWS credential providers in to the\nAssumed Role Credential Provider, it can only cause confusion.\n\n### Configuring Assumed Roles\n\nTo use assumed roles, the S3A client credentials provider must be set to\nthe `AssumedRoleCredentialProvider`, and `fs.s3a.assumed.role.arn` to\nthe previously created ARN.\n\n```xml\n\n fs.s3a.aws.credentials.provider\n org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n\n\n\n fs.s3a.assumed.role.arn\n arn:aws:iam::90066806600238:role/s3-restricted\n\n```\n\nThe STS service itself needs the caller to be authenticated, *which can\nonly be done with a set of long-lived credentials*.\nThis means the normal `fs.s3a.access.key` and `fs.s3a.secret.key`\npair, environment variables, or some other supplier of long-lived secrets.\n\nThe default is the `fs.s3a.access.key` and `fs.s3a.secret.key` pair.\nIf you wish to use a different authentication mechanism, set it in the property\n`fs.s3a.assumed.role.credentials.provider`.\n\n```xml\n\n fs.s3a.assumed.role.credentials.provider\n org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\n\n```\n\nRequirements for long-lived credentials notwithstanding, this option takes the\nsame values as `fs.s3a.aws.credentials.provider`.\n\nThe safest way to manage AWS secrets is via\n[Hadoop Credential Providers](index.html#hadoop_credential_providers).\n\n### Assumed Role Configuration Options\n\nHere are the full set of configuration options.\n\n```xml\n\n fs.s3a.assumed.role.arn\n \n \n AWS ARN for the role to be assumed.\n Required if the fs.s3a.aws.credentials.provider contains\n org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider\n \n\n\n\n fs.s3a.assumed.role.session.name\n \n \n Session name for the assumed role, must be valid characters according to\n the AWS APIs.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n If not set, one is generated from the current Hadoop/Kerberos username.\n \n\n\n\n fs.s3a.assumed.role.policy\n \n \n JSON policy to apply to the role.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.session.duration\n 30m\n \n Duration of assumed roles before a refresh is attempted.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n Range: 15m to 1h\n \n\n\n\n fs.s3a.assumed.role.sts.endpoint\n \n \n AWS Security Token Service Endpoint. If unset, uses the default endpoint.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.sts.endpoint.region\n us-west-1\n \n AWS Security Token Service Endpoint's region;\n Needed if fs.s3a.assumed.role.sts.endpoint points to an endpoint\n other than the default one and the v4 signature is used.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.credentials.provider\n org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\n \n List of credential providers to authenticate with the STS endpoint and\n retrieve short-lived role credentials.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n If unset, uses \"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\".\n \n\n```\n\n## Restricting S3A operations through AWS Policies\n\nThe S3A client needs to be granted specific permissions in order\nto work with a bucket.\nHere is a non-normative list of the permissions which must be granted\nfor FileSystem operations to work.\n\n*Disclaimer* The specific set of actions which the S3A connector needs\nwill change over time.\n\nAs more operations are added to the S3A connector, and as the\nmeans by which existing operations are implemented change, the\nAWS actions which are required by the client will change.\n\nThese lists represent the minimum actions to which the client's principal\nmust have in order to work with a bucket.\n\n\n### Read Access Permissions\n\nPermissions which must be granted when reading from a bucket:\n\n\n```\ns3:Get*\ns3:ListBucket\n```\n\nWhen using S3Guard, the client needs the appropriate\nDynamoDB access permissions\n\nTo use SSE-KMS encryption, the client needs the\nSSE-KMS Permissions to access the\nKMS key(s).\n\n### Write Access Permissions\n\nThese permissions must all be granted for write access:\n\n```\ns3:Get*\ns3:Delete*\ns3:Put*\ns3:ListBucket\ns3:ListBucketMultipartUploads\ns3:AbortMultipartUpload\n```\n\n### SSE-KMS Permissions\n\nWhen to read data encrypted using SSE-KMS, the client must have\n `kms:Decrypt` permission for the specific key a file was encrypted with.\n\n```\nkms:Decrypt\n```\n\nTo write data using SSE-KMS, the client must have all the following permissions.\n\n```\nkms:Decrypt\nkms:GenerateDataKey\n```\n\nThis includes renaming: renamed files are encrypted with the encryption key\nof the current S3A client; it must decrypt the source file first.\n\nIf the caller doesn't have these permissions, the operation will fail with an\n`AccessDeniedException`: the S3 Store does not provide the specifics of\nthe cause of the failure.\n\n### S3Guard Permissions\n\nTo use S3Guard, all clients must have a subset of the\n[AWS DynamoDB Permissions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/api-permissions-reference.html).\n\nTo work with buckets protected with S3Guard, the client must have\nall the following rights on the DynamoDB Table used to protect that bucket.\n\n```\ndynamodb:BatchGetItem\ndynamodb:BatchWriteItem\ndynamodb:DeleteItem\ndynamodb:DescribeTable\ndynamodb:GetItem\ndynamodb:PutItem\ndynamodb:Query\ndynamodb:UpdateItem\n```\n\nThis is true, *even if the client only has read access to the data*.\n\nFor the `hadoop s3guard` table management commands, _extra_ permissions are required:\n\n```\ndynamodb:CreateTable\ndynamodb:DescribeLimits\ndynamodb:DeleteTable\ndynamodb:Scan\ndynamodb:TagResource\ndynamodb:UntagResource\ndynamodb:UpdateTable\n```\n\nWithout these permissions, tables cannot be created, destroyed or have their IO capacity\nchanged through the `s3guard set-capacity` call.\nThe `dynamodb:Scan` permission is needed for `s3guard prune`\n\nThe `dynamodb:CreateTable` permission is needed by a client it tries to\ncreate the DynamoDB table on startup, that is\n`fs.s3a.s3guard.ddb.table.create` is `true` and the table does not already exist.\n\n### Mixed Permissions in a single S3 Bucket\n\nMixing permissions down the \"directory tree\" is limited\nonly to the extent of supporting writeable directories under\nread-only parent paths.\n\n*Disclaimer:* When a client lacks write access up the entire\ndirectory tree, there are no guarantees of consistent filesystem\nviews or operations.\n\nParticular troublespots are \"directory markers\" and\nfailures of non-atomic operations, particularly `rename()` and `delete()`.\n\nA directory marker such as `/users/` will not be deleted if the user `alice`\ncreates a directory `/users/alice` *and* she only has access to `/users/alice`.\n\nWhen a path or directory is deleted, the parent directory may not exist afterwards.\nIn the example above, if `alice` deletes `/users/alice` and there are no\nother entries under `/users/alice`, then the directory marker `/users/` cannot\nbe created. The directory `/users` will not exist in listings,\n`getFileStatus(\"/users\")` or similar.\n\nRename will fail if it cannot delete the items it has just copied, that is\n`rename(read-only-source, writeable-dest)` will fail —but only after\nperforming the COPY of the data.\nEven though the operation failed, for a single file copy, the destination\nfile will exist.\nFor a directory copy, only a partial copy of the source data may take place\nbefore the permission failure is raised.\n\n\n*S3Guard*: if [S3Guard](s3guard.html) is used to manage the directory listings,\nthen after partial failures of rename/copy the DynamoDB tables can get out of sync.\n\n### Example: Read access to the base, R/W to the path underneath\n\nThis example has the base bucket read only, and a directory underneath,\n`/users/alice/` granted full R/W access.\n\n```json\n{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"4\",\n \"Effect\" : \"Allow\",\n \"Action\" : [\n \"s3:ListBucket\",\n \"s3:ListBucketMultipartUploads\",\n \"s3:Get*\"\n ],\n \"Resource\" : \"arn:aws:s3:::example-bucket/*\"\n }, {\n \"Sid\" : \"5\",\n \"Effect\" : \"Allow\",\n \"Action\" : [\n \"s3:Get*\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:AbortMultipartUpload\",\n \"s3:ListMultipartUploadParts\" ],\n \"Resource\" : [\n \"arn:aws:s3:::example-bucket/users/alice/*\",\n \"arn:aws:s3:::example-bucket/users/alice\",\n \"arn:aws:s3:::example-bucket/users/alice/\"\n ]\n } ]\n}\n```\n\nNote how three resources are provided to represent the path `/users/alice`\n\n| Path | Matches |\n|-------|----------|\n| `/users/alice` | Any file `alice` created under `/users` |\n| `/users/alice/` | The directory marker `alice/` created under `/users` |\n| `/users/alice/*` | All files and directories under the path `/users/alice` |\n\nNote that the resource `arn:aws:s3:::example-bucket/users/alice*` cannot\nbe used to refer to all of these paths, because it would also cover\nadjacent paths like `/users/alice2` and `/users/alicebob`.\n\n\n## Troubleshooting Assumed Roles\n\n1. Make sure the role works and the user trying to enter it can do so from AWS\nthe command line before trying to use the S3A client.\n1. Try to access the S3 bucket with reads and writes from the AWS CLI.\n1. With the Hadoop configuration set too use the role,\n try to read data from the `hadoop fs` CLI:\n`hadoop fs -ls -p s3a://bucket/`\n1. With the hadoop CLI, try to create a new directory with a request such as\n`hadoop fs -mkdirs -p s3a://bucket/path/p1/`\n\n\n### IOException: \"Unset property fs.s3a.assumed.role.arn\"\n\nThe Assumed Role Credential Provider is enabled, but `fs.s3a.assumed.role.arn` is unset.\n\n```\njava.io.IOException: Unset property fs.s3a.assumed.role.arn\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:76)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n```\n\n### \"Not authorized to perform sts:AssumeRole\"\n\nThis can arise if the role ARN set in `fs.s3a.assumed.role.arn` is invalid\nor one to which the caller has no access.\n\n```\njava.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Not authorized to perform sts:AssumeRole (Service: AWSSecurityTokenService; Status Code: 403;\n Error Code: AccessDenied; Request ID: aad4e59a-f4b0-11e7-8c78-f36aaa9457f6):AccessDenied\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n```\n\n### \"Roles may not be assumed by root accounts\"\n\nYou can't assume a role with the root account of an AWS account;\nyou need to create a new user and give it the permission to change into\nthe role.\n\n```\njava.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;\n Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2):\n No AWS Credentials provided by AssumedRoleCredentialProvider :\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService;\n Status Code: 403; Error Code: AccessDenied; Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n ... 22 more\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;\n Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n```\n\n### \"Assume Role session duration should be in the range of 15min - 1Hr\"\n\nThe value of `fs.s3a.assumed.role.session.duration` is out of range.\n\n```\njava.lang.IllegalArgumentException: Assume Role session duration should be in the range of 15min\n- 1Hr\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$Builder.withRoleSessionDurationSeconds(STSAssumeRoleSessionCredentialsProvider.java:437)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:86)\n```\n\n\n### `MalformedPolicyDocumentException` \"The policy is not in the valid JSON format\"\n\n\nThe policy set in `fs.s3a.assumed.role.policy` is not valid according to the\nAWS specification of Role Policies.\n\n```\nrg.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n The policy is not in the valid JSON format. (Service: AWSSecurityTokenService; Status Code: 400;\n Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c):\n MalformedPolicyDocument: The policy is not in the valid JSON format.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\nCaused by: com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n The policy is not in the valid JSON format.\n (Service: AWSSecurityTokenService; Status Code: 400;\n Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 19 more\n```\n\n### `MalformedPolicyDocumentException` \"Syntax errors in policy\"\n\nThe policy set in `fs.s3a.assumed.role.policy` is not valid JSON.\n\n```\norg.apache.hadoop.fs.s3a.AWSBadRequestException:\nInstantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n Syntax errors in policy. (Service: AWSSecurityTokenService;\n Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45):\n MalformedPolicyDocument: Syntax errors in policy.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 19 more\n```\n\n### `IOException`: \"AssumedRoleCredentialProvider cannot be in fs.s3a.assumed.role.credentials.provider\"\n\nYou can't use the Assumed Role Credential Provider as the provider in\n`fs.s3a.assumed.role.credentials.provider`.\n\n```\njava.io.IOException: AssumedRoleCredentialProvider cannot be in fs.s3a.assumed.role.credentials.provider\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:86)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n```\n\n### `AWSBadRequestException`: \"not a valid key=value pair\"\n\n\nThere's an space or other typo in the `fs.s3a.access.key` or `fs.s3a.secret.key` values used for the\ninner authentication which is breaking signature creation.\n\n```\n org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header:\n 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code:\n IncompleteSignature; Request ID: c4a8841d-f556-11e7-99f9-af005a829416):IncompleteSignature:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign)\n in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date,\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature;\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign)\n in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date,\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature;\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 25 more\n```\n\n### `AccessDeniedException/InvalidClientTokenId`: \"The security token included in the request is invalid\"\n\nThe credentials used to authenticate with the AWS Security Token Service are invalid.\n\n```\n[ERROR] Failures:\n[ERROR] java.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n The security token included in the request is invalid.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId;\n Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7):InvalidClientTokenId\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\nThe security token included in the request is invalid.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId;\n Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 25 more\n```\n\n### `AWSSecurityTokenServiceExceptiond`: \"Member must satisfy regular expression pattern: `[\\w+=,.@-]*`\"\n\n\nThe session name, as set in `fs.s3a.assumed.role.session.name` must match the wildcard `[\\w+=,.@-]*`.\n\nIf the property is unset, it is extracted from the current username and then sanitized to\nmatch these constraints.\nIf set explicitly, it must be valid.\n\n```\norg.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint: Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n Request ID: 7c437acb-f55d-11e7-9ad8-3b5e4f701c20):ValidationError:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint: Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint:\n Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:135)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:124)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 26 more\n```\n\n\n### `java.nio.file.AccessDeniedException` within a FileSystem API call\n\nIf an operation fails with an `AccessDeniedException`, then the role does not have\nthe permission for the S3 Operation invoked during the call.\n\n```\njava.nio.file.AccessDeniedException: s3a://bucket/readonlyDir:\n rename(s3a://bucket/readonlyDir, s3a://bucket/renameDest)\n on s3a://bucket/readonlyDir:\n com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied\n (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1;\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=),\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=:AccessDenied\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:216)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:143)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:853)\n ...\nCaused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied\n (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1;\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=),\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229)\n at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176)\n at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2066)\n at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2052)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObject$7(S3AFileSystem.java:1338)\n at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:314)\n at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:280)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObject(S3AFileSystem.java:1334)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1657)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:1046)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:851)\n```\n\nThis is the policy restriction behaving as intended: the caller is trying to\nperform an action which is forbidden.\n\n1. If a policy has been set in `fs.s3a.assumed.role.policy` then it must declare *all*\npermissions which the caller is allowed to perform. The existing role policies\nact as an outer constraint on what the caller can perform, but are not inherited.\n\n1. If the policy for a bucket is set up with complex rules on different paths,\ncheck the path for the operation.\n\n1. The policy may have omitted one or more actions which are required.\nMake sure that all the read and write permissions are allowed for any bucket/path\nto which data is being written to, and read permissions for all\nbuckets read from.\n\nIf the bucket is using SSE-KMS to encrypt data:\n\n1. The caller must have the `kms:Decrypt` permission to read the data.\n1. The caller needs `kms:Decrypt` and `kms:GenerateDataKey`.\n\nWithout permissions, the request fails *and there is no explicit message indicating\nthat this is an encryption-key issue*.\n\n### `AccessDeniedException` + `AmazonDynamoDBException`\n\n```\njava.nio.file.AccessDeniedException: bucket1:\n com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:\n User: arn:aws:sts::980678866538:assumed-role/s3guard-test-role/test is not authorized to perform:\n dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-1:980678866538:table/bucket1\n (Service: AmazonDynamoDBv2; Status Code: 400;\n```\n\nThe caller is trying to access an S3 bucket which uses S3Guard, but the caller\nlacks the relevant DynamoDB access permissions.\n\nThe `dynamodb:DescribeTable` operation is the first one used in S3Guard to access,\nthe DynamoDB table, so it is often the first to fail. It can be a sign\nthat the role has no permissions at all to access the table named in the exception,\nor just that this specific permission has been omitted.\n\nIf the role policy requested for the assumed role didn't ask for any DynamoDB\npermissions, this is where all attempts to work with a S3Guarded bucket will\nfail. Check the value of `fs.s3a.assumed.role.policy`\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/encryption.md":"\n\n\n# Working with Encrypted S3 Data\n\n\n\n\n## Introduction\n\nThe S3A filesystem client supports Amazon S3's Server Side Encryption\nfor at-rest data encryption.\nYou should to read up on the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html)\nfor S3 Server Side Encryption for up to date information on the encryption mechanisms.\n\n\n\nWhen configuring an encryption method in the `core-site.xml`, this will apply cluster wide.\nAny new file written will be encrypted with this encryption configuration.\nWhen the S3A client reads a file, S3 will attempt to decrypt it using the mechanism\nand keys with which the file was encrypted.\n\n* It is **NOT** advised to mix and match encryption types in a bucket\n* It is much simpler and safer to encrypt with just one type and key per bucket.\n* You can use AWS bucket policies to mandate encryption rules for a bucket.\n* You can use S3A per-bucket configuration to ensure that S3A clients use encryption\npolicies consistent with the mandated rules.\n* You can use S3 Default Encryption to encrypt data without needing to\nset anything in the client.\n* Changing the encryption options on the client does not change how existing\nfiles were encrypted, except when the files are renamed.\n* For all mechanisms other than SSE-C, clients do not need any configuration\noptions set in order to read encrypted data: it is all automatically handled\nin S3 itself.\n\n## How data is encrypted\n\nAWS S3 supports server-side encryption inside the storage system itself.\nWhen an S3 client uploading data requests data to be encrypted, then an encryption key is used\nto encrypt the data as it saved to S3. It remains encrypted on S3 until deleted:\nclients cannot change the encryption attributes of an object once uploaded.\n\nThe Amazon AWS SDK also offers client-side encryption, in which all the encoding\nand decoding of data is performed on the client. This is *not* supported by\nthe S3A client.\n\nThe server-side \"SSE\" encryption is performed with symmetric AES256 encryption;\nS3 offers different mechanisms for actually defining the key to use.\n\n\nThere are four key management mechanisms, which in order of simplicity of use,\nare:\n\n* S3 Default Encryption\n* SSE-S3: an AES256 key is generated in S3, and saved alongside the data.\n* SSE-KMS: an AES256 key is generated in S3, and encrypted with a secret key provided\nby Amazon's Key Management Service, a key referenced by name in the uploading client.\n* SSE-C : the client specifies an actual base64 encoded AES-256 key to be used\nto encrypt and decrypt the data.\n\n\n## S3 Default Encryption\n\nThis feature allows the administrators of the AWS account to set the \"default\"\nencryption policy on a bucket -the encryption to use if the client does\nnot explicitly declare an encryption algorithm.\n\n[S3 Default Encryption for S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html)\n\nThis supports SSE-S3 and SSE-KMS.\n\nThere is no need to set anything up in the client: do it in the AWS console.\n\n\n## SSE-S3 Amazon S3-Managed Encryption Keys\n\nIn SSE-S3, all keys and secrets are managed inside S3. This is the simplest encryption mechanism.\nThere is no extra cost for storing data with this option.\n\n\n### Enabling SSE-S3\n\nTo write S3-SSE encrypted files, the value of\n`fs.s3a.server-side-encryption-algorithm` must be set to that of\nthe encryption mechanism used in `core-site`; currently only `AES256` is supported.\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n AES256\n\n```\n\nOnce set, all new data will be stored encrypted. There is no need to set this property when downloading data — the data will be automatically decrypted when read using\nthe Amazon S3-managed key.\n\nTo learn more, refer to\n[Protecting Data Using Server-Side Encryption with Amazon S3-Managed Encryption Keys (SSE-S3) in AWS documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html).\n\n\n### SSE-KMS: Amazon S3-KMS Managed Encryption Keys\n\n\nAmazon offers a pay-per-use key management service, [AWS KMS](https://aws.amazon.com/documentation/kms/).\nThis service can be used to encrypt data on S3 by defining \"customer master keys\", CMKs,\nwhich can be centrally managed and assigned to specific roles and IAM accounts.\n\nThe AWS KMS [can be used encrypt data on S3uploaded data](http://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html).\n\n> The AWS KMS service is **not** related to the Key Management Service built into Hadoop (*Hadoop KMS*). The *Hadoop KMS* primarily focuses on\n managing keys for *HDFS Transparent Encryption*. Similarly, HDFS encryption is unrelated to S3 data encryption.\n\nWhen uploading data encrypted with SSE-KMS, the sequence is as follows.\n\n1. The S3A client must declare a specific CMK in the property `fs.s3a.server-side-encryption.key`, or leave\nit blank to use the default configured for that region.\n\n1. The S3A client uploads all the data as normal, now including encryption information.\n\n1. The S3 service encrypts the data with a symmetric key unique to the new object.\n\n1. The S3 service retrieves the chosen CMK key from the KMS service, and, if the user has\nthe right to use it, uses it to encrypt the object-specific key.\n\n\nWhen downloading SSE-KMS encrypted data, the sequence is as follows\n\n1. The S3A client issues an HTTP GET request to read the data.\n1. S3 sees that the data was encrypted with SSE-KMS, and looks up the specific key in the KMS service\n1. If and only if the requesting user has been granted permission to use the CMS key does\nthe KMS service provide S3 with the key.\n1. As a result, S3 will only decode the data if the user has been granted access to the key.\n\n\nKMS keys can be managed by an organization's administrators in AWS, including\nhaving access permissions assigned and removed from specific users, groups, and IAM roles.\nOnly those \"principals\" with granted rights to a key may access it,\nhence only they may encrypt data with the key, *and decrypt data encrypted with it*.\nThis allows KMS to be used to provide a cryptographically secure access control mechanism for data stores on S3.\n\n\nEach KMS server is region specific, and accordingly, so is each CMK configured.\nA CMK defined in one region cannot be used with an S3 bucket in a different region.\n\n\nNotes\n\n* Callers are charged for every use of a key, both for encrypting the data in uploads\n and for decrypting it when reading it back.\n* Random-access IO on files may result in multiple GET requests of an object during a read\nsequence (especially for columnar data), so may require more than one key retrieval to process a single file,\n* The KMS service is throttled: too many requests may cause requests to fail.\n* As well as incurring charges, heavy I/O *may* reach IO limits for a customer. If those limits are reached,\nthey can be increased through the AWS console.\n\n\n### Enabling SSE-KMS\n\nTo enable SSE-KMS, the property `fs.s3a.server-side-encryption-algorithm` must be set to `SSE-KMS` in `core-site`:\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n SSE-KMS\n\n```\n\nThe ID of the specific key used to encrypt the data should also be set in the property `fs.s3a.server-side-encryption.key`:\n\n```xml\n\n fs.s3a.server-side-encryption.key\n arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n```\n\nOrganizations may define a default key in the Amazon KMS; if a default key is set,\nthen it will be used whenever SSE-KMS encryption is chosen and the value of `fs.s3a.server-side-encryption.key` is empty.\n\n### the S3A `fs.s3a.encryption.key` key only affects created files\n\nWith SSE-KMS, the S3A client option `fs.s3a.server-side-encryption.key` sets the\nkey to be used when new files are created. When reading files, this key,\nand indeed the value of `fs.s3a.server-side-encryption-algorithme` is ignored:\nS3 will attempt to retrieve the key and decrypt the file based on the create-time settings.\n\nThis means that\n\n* There's no need to configure any client simply reading data.\n* It is possible for a client to read data encrypted with one KMS key, and\nwrite it with another.\n\n\n## SSE-C: Server side encryption with a client-supplied key.\n\nIn SSE-C, the client supplies the secret key needed to read and write data.\nEvery client trying to read or write data must be configured with the same\nsecret key.\n\n\nSSE-C integration with Hadoop is still stabilizing; issues related to it are still surfacing.\nIt is already clear that SSE-C with a common key must be used exclusively within\na bucket if it is to be used at all. This is the only way to ensure that path and\ndirectory listings do not fail with \"Bad Request\" errors.\n\n### Enabling SSE-C\n\nTo use SSE-C, the configuration option `fs.s3a.server-side-encryption-algorithm`\nmust be set to `SSE-C`, and a base-64 encoding of the key placed in\n`fs.s3a.server-side-encryption.key`.\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n SSE-C\n\n\n\n fs.s3a.server-side-encryption.key\n SGVscCwgSSdtIHRyYXBwZWQgaW5zaWRlIGEgYmFzZS02NC1jb2RlYyE=\n\n```\n\nAll clients must share this same key.\n\n### The `fs.s3a.encryption.key` value is used to read and write data\n\nWith SSE-C, the S3A client option `fs.s3a.server-side-encryption.key` sets the\nkey to be used for both reading *and* writing data.\n\nWhen reading any file written with SSE-C, the same key must be set\nin the property `fs.s3a.server-side-encryption.key`.\n\nThis is unlike SSE-S3 and SSE-KMS, where the information needed to\ndecode data is kept in AWS infrastructure.\n\n\n### SSE-C Warning\n\nYou need to fully understand how SSE-C works in the S3\nenvironment before using this encryption type. Please refer to the Server Side\nEncryption documentation available from AWS. SSE-C is only recommended for\nadvanced users with advanced encryption use cases. Failure to properly manage\nencryption keys can cause data loss. Currently, the AWS S3 API(and thus S3A)\nonly supports one encryption key and cannot support decrypting objects during\nmoves under a previous key to a new destination. It is **NOT** advised to use\nmultiple encryption keys in a bucket, and is recommended to use one key per\nbucket and to not change this key. This is due to when a request is made to S3,\nthe actual encryption key must be provided to decrypt the object and access the\nmetadata. Since only one encryption key can be provided at a time, S3A will not\npass the correct encryption key to decrypt the data.\n\n\n## Encryption best practises\n\n\n### Mandate encryption through policies\n\nBecause it is up to the clients to enable encryption on new objects, all clients\nmust be correctly configured in order to guarantee that data is encrypted.\n\n\nTo mandate that all data uploaded to a bucket is encrypted,\nyou can set a [bucket policy](https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/)\ndeclaring that clients must provide encryption information with all data uploaded.\n\n\n* Mandating an encryption mechanism on newly uploaded data does not encrypt existing data; existing data will retain whatever encryption (if any) applied at the time of creation*\n\nHere is a policy to mandate `SSE-S3/AES265` encryption on all data uploaded to a bucket. This covers uploads as well as the copy operations which take place when file/directory rename operations are mimicked.\n\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Id\": \"EncryptionPolicy\",\n \"Statement\": [\n {\n \"Sid\": \"RequireEncryptionHeaderOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"Null\": {\n \"s3:x-amz-server-side-encryption\": true\n }\n }\n },\n {\n \"Sid\": \"RequireAESEncryptionOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"StringNotEquals\": {\n \"s3:x-amz-server-side-encryption\": \"AES256\"\n }\n }\n }\n ]\n}\n```\n\nTo use SSE-KMS, a different restriction must be defined:\n\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Id\": \"EncryptionPolicy\",\n \"Statement\": [\n {\n \"Sid\": \"RequireEncryptionHeaderOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"Null\": {\n \"s3:x-amz-server-side-encryption\": true\n }\n }\n },\n {\n \"Sid\": \"RequireKMSEncryptionOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"StringNotEquals\": {\n \"s3:x-amz-server-side-encryption\": \"SSE-KMS\"\n }\n }\n }\n ]\n}\n```\n\nTo use one of these policies:\n\n1. Replace `BUCKET` with the specific name of the bucket being secured.\n1. Locate the bucket in the AWS console [S3 section](https://console.aws.amazon.com/s3/home).\n1. Select the \"Permissions\" tab.\n1. Select the \"Bucket Policy\" tab in the permissions section.\n1. Paste the edited policy into the form.\n1. Save the policy.\n\n### Use S3a per-bucket configuration to control encryption settings\n\nIn an organisation which has embraced S3 encryption, different buckets inevitably have\ndifferent encryption policies, such as different keys for SSE-KMS encryption.\nIn particular, as different keys need to be named for different regions, unless\nyou rely on the administrator-managed \"default\" key for each S3 region, you\nwill need unique keys.\n\nS3A's per-bucket configuration enables this.\n\n\nHere, for example, are settings for a bucket in London, `london-stats`:\n\n\n```xml\n\n fs.s3a.bucket.london-stats.server-side-encryption-algorithm\n AES256\n\n```\n\nThis requests SSE-S; if matched with a bucket policy then all data will\nbe encrypted as it is uploaded.\n\n\nA different bucket can use a different policy\n(here SSE-KMS) and, when necessary, declare a key.\n\nHere is an example bucket in S3 Ireland, which uses SSE-KMS and\na KMS key hosted in the AWS-KMS service in the same region.\n\n\n```xml\n\n fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm\n SSE-KMS\n\n\n\n fs.s3a.bucket.ireland-dev.server-side-encryption.key\n arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n\n```\n\nAgain the appropriate bucket policy can be used to guarantee that all callers\nwill use SSE-KMS; they can even mandate the name of the key used to encrypt\nthe data, so guaranteeing that access to thee data can be read by everyone\ngranted access to that key, and nobody without access to it.\n\n\n### Use rename() to encrypt files with new keys\n\nThe encryption of an object is set when it is uploaded. If you want to encrypt\nan unencrypted file, or change the SEE-KMS key of a file, the only way to do\nso is by copying the object.\n\nHow can you do that from Hadoop? With `rename()`.\n\nThe S3A client mimics a real filesystem's' rename operation by copying all the\nsource files to the destination paths, then deleting the old ones.\n\nNote: this does not work for SSE-C, because you cannot set a different key\nfor reading as for writing, and you must supply that key for reading. There\nyou need to copy one bucket to a different bucket, one with a different key.\nUse `distCp`for this, with per-bucket encryption policies.\n\n\n## Troubleshooting Encryption\n\nThe [troubleshooting](./troubleshooting_s3a.html) document covers\nstack traces which may surface when working with encrypted data.\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md":"\n\n# Committing work to S3 with the \"S3A Committers\"\n\n\n\nThis page covers the S3A Committers, which can commit work directly\nto an S3 object store.\n\nThese committers are designed to solve a fundamental problem which\nthe standard committers of work cannot do to S3: consistent, high performance,\nand reliable commitment of output to S3.\n\nFor details on their internal design, see\n[S3A Committers: Architecture and Implementation](./committer_architecture.html).\n\n\n## Introduction: The Commit Problem\n\n\nApache Hadoop MapReduce (and behind the scenes, Apache Spark) often write\nthe output of their work to filesystems\n\nNormally, Hadoop uses the `FileOutputFormatCommitter` to manage the\npromotion of files created in a single task attempt to the final output of\na query. This is done in a way to handle failures of tasks and jobs, and to\nsupport speculative execution. It does that by listing directories and renaming\ntheir content into the final destination when tasks and then jobs are committed.\n\nThis has some key requirement of the underlying filesystem:\n\n1. When you list a directory, you see all the files which have been created in it,\nand no files which are not in it (i.e. have been deleted).\n1. When you rename a directory, it is an `O(1)` atomic transaction. No other\nprocess across the cluster may rename a file or directory to the same path.\nIf the rename fails for any reason, either the data is at the original location,\nor it is at the destination, -in which case the rename actually succeeded.\n\n**The S3 object store and the `s3a://` filesystem client cannot meet these requirements.*\n\n1. Amazon S3 has inconsistent directory listings unless S3Guard is enabled.\n1. The S3A mimics `rename()` by copying files and then deleting the originals.\nThis can fail partway through, and there is nothing to prevent any other process\nin the cluster attempting a rename at the same time.\n\nAs a result,\n\n* Files my not be listed, hence not renamed into place.\n* Deleted files may still be discovered, confusing the rename process to the point\nof failure.\n* If a rename fails, the data is left in an unknown state.\n* If more than one process attempts to commit work simultaneously, the output\ndirectory may contain the results of both processes: it is no longer an exclusive\noperation.\n*. While S3Guard may deliver the listing consistency, commit time is still\nproportional to the amount of data created. It still can't handle task failure.\n\n**Using the \"classic\" `FileOutputCommmitter` to commit work to Amazon S3 risks\nloss or corruption of generated data**\n\n\nTo address these problems there is now explicit support in the `hadop-aws`\nmodule for committing work to Amazon S3 via the S3A filesystem client,\n*the S3A Committers*\n\n\nFor safe, as well as high-performance output of work to S3,\nwe need use \"a committer\" explicitly written to work with S3, treating it as\nan object store with special features.\n\n\n### Background : Hadoop's \"Commit Protocol\"\n\nHow exactly is work written to its final destination? That is accomplished by\na \"commit protocol\" between the workers and the job manager.\n\nThis protocol is implemented in Hadoop MapReduce, with a similar but extended\nversion in Apache Spark:\n\n1. A \"Job\" is the entire query, with inputs to outputs\n1. The \"Job Manager\" is the process in charge of choreographing the execution\nof the job. It may perform some of the actual computation too.\n1. The job has \"workers\", which are processes which work the actual data\nand write the results.\n1. Workers execute \"Tasks\", which are fractions of the job, a job whose\ninput has been *partitioned* into units of work which can be executed independently.\n1. The Job Manager directs workers to execute \"tasks\", usually trying to schedule\nthe work close to the data (if the filesystem provides locality information).\n1. Workers can fail: the Job manager needs to detect this and reschedule their active tasks.\n1. Workers can also become separated from the Job Manager, a \"network partition\".\nIt is (provably) impossible for the Job Manager to distinguish a running-but-unreachable\nworker from a failed one.\n1. The output of a failed task must not be visible; this is to avoid its\ndata getting into the final output.\n1. Multiple workers can be instructed to evaluate the same partition of the work;\nthis \"speculation\" delivers speedup as it can address the \"straggler problem\".\nWhen multiple workers are working on the same data, only one worker is allowed\nto write the final output.\n1. The entire job may fail (often from the failure of the Job Manager (MR Master, Spark Driver, ...)).\n1, The network may partition, with workers isolated from each other or\nthe process managing the entire commit.\n1. Restarted jobs may recover from a failure by reusing the output of all\ncompleted tasks (MapReduce with the \"v1\" algorithm), or just by rerunning everything\n(The \"v2\" algorithm and Spark).\n\n\nWhat is \"the commit protocol\" then? It is the requirements on workers as to\nwhen their data is made visible, where, for a filesystem, \"visible\" means \"can\nbe seen in the destination directory of the query.\"\n\n* There is a destination directory of work, \"the output directory.\"\n* The final output of tasks must be in this directory *or paths underneath it*.\n* The intermediate output of a task must not be visible in the destination directory.\nThat is: they must not write directly to the destination.\n* The final output of a task *may* be visible under the destination.\n* The Job Manager makes the decision as to whether a task's data is to be \"committed\",\nbe it directly to the final directory or to some intermediate store..\n* Individual workers communicate with the Job manager to manage the commit process:\nwhether the output is to be *committed* or *aborted*\n* When a worker commits the output of a task, it somehow promotes its intermediate work to becoming\nfinal.\n* When a worker aborts a task's output, that output must not become visible\n(i.e. it is not committed).\n* Jobs themselves may be committed/aborted (the nature of \"when\" is not covered here).\n* After a Job is committed, all its work must be visible.\n* And a file `_SUCCESS` may be written to the output directory.\n* After a Job is aborted, all its intermediate data is lost.\n* Jobs may also fail. When restarted, the successor job must be able to clean up\nall the intermediate and committed work of its predecessor(s).\n* Task and Job processes measure the intervals between communications with their\nApplication Master and YARN respectively.\nWhen the interval has grown too large they must conclude\nthat the network has partitioned and that they must abort their work.\n\n\nThat's \"essentially\" it. When working with HDFS and similar filesystems,\ndirectory `rename()` is the mechanism used to commit the work of tasks and\njobs.\n* Tasks write data to task attempt directories under the directory `_temporary`\nunderneath the final destination directory.\n* When a task is committed, these files are renamed to the destination directory\n(v2 algorithm) or a job attempt directory under `_temporary` (v1 algorithm).\n* When a job is committed, for the v2 algorithm the `_SUCCESS` file is created,\nand the `_temporary` deleted.\n* For the v1 algorithm, when a job is committed, all the tasks committed under\nthe job attempt directory will have their output renamed into the destination\ndirectory.\n* The v2 algorithm recovers from failure by deleting the destination directory\nand restarting the job.\n* The v1 algorithm recovers from failure by discovering all committed tasks\nwhose output is in the job attempt directory, *and only rerunning all uncommitted tasks*.\n\n\nNone of this algorithm works safely or swiftly when working with \"raw\" AWS S3 storage:\n* Directory listing can be inconsistent: the tasks and jobs may not list all work to\nbe committed.\n* Renames go from being fast, atomic operations to slow operations which can fail partway through.\n\nThis then is the problem which the S3A committers address:\n\n*How to safely and reliably commit work to Amazon S3 or compatible object store*\n\n\n## Meet the S3A Commmitters\n\nSince Hadoop 3.1, the S3A FileSystem has been accompanied by classes\ndesigned to integrate with the Hadoop and Spark job commit protocols, classes\nwhich interact with the S3A filesystem to reliably commit work work to S3:\n*The S3A Committers*\n\nThe underlying architecture of this process is very complex, and\ncovered in [the committer architecture documentation](./committer_architecture.html).\n\nThe key concept to know of is S3's \"Multipart Upload\" mechanism. This allows\nan S3 client to write data to S3 in multiple HTTP POST requests, only completing\nthe write operation with a final POST to complete the upload; this final POST\nconsisting of a short list of the etags of the uploaded blocks.\nThis multipart upload mechanism is already automatically used when writing large\namounts of data to S3; an implementation detail of the S3A output stream.\n\nThe S3A committers make explicit use of this multipart upload (\"MPU\") mechanism:\n\n1. The individual *tasks* in a job write their data to S3 as POST operations\nwithin multipart uploads, yet do not issue the final POST to complete the upload.\n1. The multipart uploads are committed in the job commit process.\n\nThere are two different S3A committer types, *staging*\nand *magic*. The committers primarily vary in how data is written during task execution,\nhow the pending commit information is passed to the job manager, and in how\nconflict with existing files is resolved.\n\n\n| feature | staging | magic |\n|--------|---------|---|\n| task output destination | local disk | S3A *without completing the write* |\n| task commit process | upload data from disk to S3 | list all pending uploads on s3 and write details to job attempt directory |\n| task abort process | delete local disk data | list all pending uploads and abort them |\n| job commit | list & complete pending uploads | list & complete pending uploads |\n\nThe other metric is \"maturity\". There, the fact that the Staging committers\nare based on Netflix's production code counts in its favor.\n\n\n### The Staging Committer\n\nThis is based on work from Netflix. It \"stages\" data into the local filesystem.\nIt also requires the cluster to have HDFS, so that\n\nTasks write to URLs with `file://` schemas. When a task is committed,\nits files are listed, uploaded to S3 as incompleted Multipart Uploads.\nThe information needed to complete the uploads is saved to HDFS where\nit is committed through the standard \"v1\" commit algorithm.\n\nWhen the Job is committed, the Job Manager reads the lists of pending writes from its\nHDFS Job destination directory and completes those uploads.\n\nCancelling a task is straightforward: the local directory is deleted with\nits staged data. Cancelling a job is achieved by reading in the lists of\npending writes from the HDFS job attempt directory, and aborting those\nuploads. For extra safety, all outstanding multipart writes to the destination directory\nare aborted.\n\nThe staging committer comes in two slightly different forms, with slightly\ndifferent conflict resolution policies:\n\n\n* **Directory**: the entire directory tree of data is written or overwritten,\nas normal.\n\n* **Partitioned**: special handling of partitioned directory trees of the form\n`YEAR=2017/MONTH=09/DAY=19`: conflict resolution is limited to the partitions\nbeing updated.\n\n\nThe Partitioned Committer is intended to allow jobs updating a partitioned\ndirectory tree to restrict the conflict resolution to only those partition\ndirectories containing new data. It is intended for use with Apache Spark\nonly.\n\n\n## Conflict Resolution in the Staging Committers\n\nThe Staging committers offer the ability to replace the conflict policy\nof the execution engine with policy designed to work with the tree of data.\nThis is based on the experience and needs of Netflix, where efficiently adding\nnew data to an existing partitioned directory tree is a common operation.\n\n```xml\n\n fs.s3a.committer.staging.conflict-mode\n fail\n \n Staging committer conflict resolution policy: {@value}.\n Supported: fail, append, replace.\n \n\n```\n\n**replace** : when the job is committed (and not before), delete files in\ndirectories into which new data will be written.\n\n**fail**: when there are existing files in the destination, fail the job.\n\n**append**: Add new data to the directories at the destination; overwriting\nany with the same name. Reliable use requires unique names for generated files,\nwhich the committers generate\nby default.\n\nThe difference between the two staging committers are as follows:\n\nThe Directory Committer uses the entire directory tree for conflict resolution.\nIf any file exists at the destination it will fail in job setup; if the resolution\nmechanism is \"replace\" then all existing files will be deleted.\n\nThe partitioned committer calculates the partitions into which files are added,\nthe final directories in the tree, and uses that in its conflict resolution\nprocess:\n\n\n**replace** : delete all data in the destination partition before committing\nthe new files.\n\n**fail**: fail if there is data in the destination partition, ignoring the state\nof any parallel partitions.\n\n**append**: add the new data.\n\nIt's intended for use in Apache Spark Dataset operations, rather\nthan Hadoop's original MapReduce engine, and only in jobs\nwhere adding new data to an existing dataset is the desired goal.\n\nPrerequisites for successful work\n\n1. The output is written into partitions via `PARTITIONED BY` or `partitionedBy()`\ninstructions.\n2. There is no data written directly to the root path (all files there are\nignored; it's implicitly \"append\").\n\nHere's an example in Spark, assuming that `sourceDataset` is a dataset\nwhose columns include \"year\" and \"month\":\n\n```scala\nsourceDataset\n .write\n .partitionBy(\"year\", \"month\")\n .mode(SaveMode.Append)\n .opt(\"fs.s3a.committer.name\", \"partitioned\")\n .opt(\"fs.s3a.committer.staging.conflict-mode\", \"replace\")\n .format(\"orc\")\n .save(\"s3a://examples/statistics\")\n```\n\n\n### The Magic Committer\n\nThe \"Magic\" committer does its work through \"magic\" in the filesystem:\nattempts to write to specific \"magic\" paths are interpreted as writes\nto a parent directory *which are not to be completed*. When the output stream\nis closed, the information needed to complete the write is saved in the magic\ndirectory. The task committer saves the list of these to a directory for the\njob committers use, or, if aborting, lists the pending writes and aborts them.\n\nThe job committer reads in the list of pending commits, and commits them as\nthe Staging Committer does.\n\nCompared to the Staging Committer, the Magic Committer offers faster write\ntimes: output is uploaded to S3 as it is written, rather than in the\ntask commit.\n\nHowever, it has extra requirements of the filesystem\n\n1. It requires a consistent object store, which for Amazon S3,\nmeans that [S3Guard](./s3guard.html) must be enabled. For third-party stores,\nconsult the documentation.\n1. The S3A client must be configured to recognize interactions\nwith the magic directories and treat them specially.\n\n\nIt's also not been field tested to the extent of Netflix's committer; consider\nit the least mature of the committers.\n\n\n#### Which Committer to Use?\n\n1. If you want to create or update existing partitioned data trees in Spark, use thee\nPartitioned Committer. Make sure you have enough hard disk capacity for all staged data.\nDo not use it in other situations.\n\n1. If you know that your object store is consistent, or that the processes\nwriting data use S3Guard, use the Magic Committer for higher performance\nwriting of large amounts of data.\n\n1. Otherwise: use the directory committer, making sure you have enough\nhard disk capacity for all staged data.\n\nPut differently: start with the Directory Committer.\n\n## Switching to an S3A Committer\n\nTo use an S3A committer, the property `mapreduce.outputcommitter.factory.scheme.s3a`\nmust be set to the S3A committer factory, `org.apache.hadoop.fs.s3a.commit.staging.S3ACommitterFactory`.\nThis is done in `mapred-default.xml`\n\n```xml\n\n mapreduce.outputcommitter.factory.scheme.s3a\n org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory\n \n The committer factory to use when writing data to S3A filesystems.\n \n\n```\n\nWhat is missing is an explicit choice of committer to use in the property\n`fs.s3a.committer.name`; so the classic (and unsafe) file committer is used.\n\n| `fs.s3a.committer.name` | Committer |\n|--------|---------|\n| `directory` | directory staging committer |\n| `partitioned` | partition staging committer (for use in Spark only) |\n| `magic` | the \"magic\" committer |\n| `file` | the original and unsafe File committer; (default) |\n\n\n\n## Using the Directory and Partitioned Staging Committers\n\nGenerated files are initially written to a local directory underneath one of the temporary\ndirectories listed in `fs.s3a.buffer.dir`.\n\n\nThe staging committer needs a path in the cluster filesystem\n(e.g. HDFS). This must be declared in `fs.s3a.committer.staging.tmp.path`.\n\nTemporary files are saved in HDFS (or other cluster filesystem) under the path\n`${fs.s3a.committer.staging.tmp.path}/${user}` where `user` is the name of the user running the job.\nThe default value of `fs.s3a.committer.staging.tmp.path` is `tmp/staging`,\nWhich will be converted at run time to a path under the current user's home directory,\nessentially `~/tmp/staging`\n so the temporary directory\n\nThe application attempt ID is used to create a unique path under this directory,\nresulting in a path `~/tmp/staging/${user}/${application-attempt-id}/` under which\nsummary data of each task's pending commits are managed using the standard\n`FileOutputFormat` committer.\n\nWhen a task is committed the data is uploaded under the destination directory.\nThe policy of how to react if the destination exists is defined by\nthe `fs.s3a.committer.staging.conflict-mode` setting.\n\n| `fs.s3a.committer.staging.conflict-mode` | Meaning |\n| -----------------------------------------|---------|\n| `fail` | Fail if the destination directory exists |\n| `replace` | Delete all existing files before committing the new data |\n| `append` | Add the new files to the existing directory tree |\n\n\n## The \"Partitioned\" Staging Committer\n\nThis committer an extension of the \"Directory\" committer which has a special conflict resolution\npolicy designed to support operations which insert new data into a directory tree structured\nusing Hive's partitioning strategy: different levels of the tree represent different columns.\n\nFor example, log data could be partitioned by `YEAR` and then by `MONTH`, with different\nentries underneath.\n\n```\nlogs/YEAR=2017/MONTH=01/\n log-20170101.avro\n log-20170102.avro\n ...\n log-20170131.avro\n\nlogs/YEAR=2017/MONTH=02/\n log-20170201.avro\n log-20170202.avro\n ...\n log-20170227.avro\n\nlogs/YEAR=2017/MONTH=03/\nlogs/YEAR=2017/MONTH=04/\n```\n\nA partitioned structure like this allows for queries using Hive or Spark to filter out\nfiles which do not contain relevant data.\n\nWhat the partitioned committer does is, where the tooling permits, allows callers\nto add data to an existing partitioned layout*.\n\nMore specifically, it does this by having a conflict resolution options which\nonly act on individual partitions, rather than across the entire output tree.\n\n| `fs.s3a.committer.staging.conflict-mode` | Meaning |\n| -----------------------------------------|---------|\n| `fail` | Fail if the destination partition(s) exist |\n| `replace` | Delete the existing data partitions before committing the new data |\n| `append` | Add the new data to the existing partitions |\n\n\nAs an example, if a job was writing the file\n`logs/YEAR=2017/MONTH=02/log-20170228.avro`, then with a policy of `fail`,\nthe job would fail. With a policy of `replace`, then entire directory\n`logs/YEAR=2017/MONTH=02/` would be deleted before the new file `log-20170228.avro`\nwas written. With the policy of `append`, the new file would be added to\nthe existing set of files.\n\n\n### Notes\n\n1. A deep partition tree can itself be a performance problem in S3 and the s3a client,\nor, more specifically. a problem with applications which use recursive directory tree\nwalks to work with data.\n\n1. The outcome if you have more than one job trying simultaneously to write data\nto the same destination with any policy other than \"append\" is undefined.\n\n1. In the `append` operation, there is no check for conflict with file names.\nIf, in the example above, the file `log-20170228.avro` already existed,\nit would be overridden. Set `fs.s3a.committer.staging.unique-filenames` to `true`\nto ensure that a UUID is included in every filename to avoid this.\n\n\n## Using the Magic committer\n\nThis is less mature than the Staging Committer, but promises higher\nperformance.\n\n### FileSystem client setup\n\n1. Use a *consistent* S3 object store. For Amazon S3, this means enabling\n[S3Guard](./s3guard.html). For S3-compatible filesystems, consult the filesystem\ndocumentation to see if it is consistent, hence compatible \"out of the box\".\n1. Turn the magic on by `fs.s3a.committer.magic.enabled\"`\n\n```xml\n\n fs.s3a.committer.magic.enabled\n \n Enable support in the filesystem for the S3 \"Magic\" committer.\n \n true\n\n```\n\n*Do not use the Magic Committer on an inconsistent S3 object store. For\nAmazon S3, that means S3Guard must *always* be enabled.\n\n\n### Enabling the committer\n\n```xml\n\n fs.s3a.committer.name\n magic\n\n\n```\n\nConflict management is left to the execution engine itself.\n\n## Committer Configuration Options\n\n\n| Option | Magic | Directory | Partitioned | Meaning | Default |\n|--------|-------|-----------|-------------|---------|---------|\n| `mapreduce.fileoutputcommitter.marksuccessfuljobs` | X | X | X | Write a `_SUCCESS` file at the end of each job | `true` |\n| `fs.s3a.committer.threads` | X | X | X | Number of threads in committers for parallel operations on files. | 8 |\n| `fs.s3a.committer.staging.conflict-mode` | | X | X | Conflict resolution: `fail`, `abort` or `overwrite`| `fail` |\n| `fs.s3a.committer.staging.unique-filenames` | | X | X | Generate unique filenames | `true` |\n\n| `fs.s3a.committer.magic.enabled` | X | | | Enable \"magic committer\" support in the filesystem | `false` |\n\n\n\n\n| Option | Magic | Directory | Partitioned | Meaning | Default |\n|--------|-------|-----------|-------------|---------|---------|\n| `fs.s3a.buffer.dir` | X | X | X | Local filesystem directory for data being written and/or staged. | |\n| `fs.s3a.committer.staging.tmp.path` | | X | X | Path in the cluster filesystem for temporary data | `tmp/staging` |\n\n\n```xml\n\n fs.s3a.committer.name\n file\n \n Committer to create for output to S3A, one of:\n \"file\", \"directory\", \"partitioned\", \"magic\".\n \n\n\n\n fs.s3a.committer.magic.enabled\n false\n \n Enable support in the filesystem for the S3 \"Magic\" committer.\n When working with AWS S3, S3Guard must be enabled for the destination\n bucket, as consistent metadata listings are required.\n \n\n\n\n fs.s3a.committer.threads\n 8\n \n Number of threads in committers for parallel operations on files\n (upload, commit, abort, delete...)\n \n\n\n\n fs.s3a.committer.staging.tmp.path\n tmp/staging\n \n Path in the cluster filesystem for temporary data.\n This is for HDFS, not the local filesystem.\n It is only for the summary data of each file, not the actual\n data being committed.\n Using an unqualified path guarantees that the full path will be\n generated relative to the home directory of the user creating the job,\n hence private (assuming home directory permissions are secure).\n \n\n\n\n fs.s3a.committer.staging.unique-filenames\n true\n \n Option for final files to have a unique name through job attempt info,\n or the value of fs.s3a.committer.staging.uuid\n When writing data with the \"append\" conflict option, this guarantees\n that new data will not overwrite any existing data.\n \n\n\n\n fs.s3a.committer.staging.conflict-mode\n fail\n \n Staging committer conflict resolution policy.\n Supported: \"fail\", \"append\", \"replace\".\n \n\n\n\n s.s3a.committer.staging.abort.pending.uploads\n true\n \n Should the staging committers abort all pending uploads to the destination\n directory?\n\n Changing this if more than one partitioned committer is\n writing to the same destination tree simultaneously; otherwise\n the first job to complete will cancel all outstanding uploads from the\n others. However, it may lead to leaked outstanding uploads from failed\n tasks. If disabled, configure the bucket lifecycle to remove uploads\n after a time period, and/or set up a workflow to explicitly delete\n entries. Otherwise there is a risk that uncommitted uploads may run up\n bills.\n \n\n\n\n mapreduce.outputcommitter.factory.scheme.s3a\n org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory\n \n The committer factory to use when writing data to S3A filesystems.\n If mapreduce.outputcommitter.factory.class is set, it will\n override this property.\n\n (This property is set in mapred-default.xml)\n \n\n\n```\n\n\n## Troubleshooting\n\n### `Filesystem does not have support for 'magic' committer`\n\n```\norg.apache.hadoop.fs.s3a.commit.PathCommitException: `s3a://landsat-pds': Filesystem does not have support for 'magic' committer enabled\nin configuration option fs.s3a.committer.magic.enabled\n```\n\nThe Job is configured to use the magic committer, but the S3A bucket has not been explicitly\ndeclared as supporting it.\n\nThe destination bucket **must** be declared as supporting the magic committer.\n\nThis can be done for those buckets which are known to be consistent, either\nbecause [S3Guard](s3guard.html) is used to provide consistency,\nor because the S3-compatible filesystem is known to be strongly consistent.\n\n```xml\n\n fs.s3a.bucket.landsat-pds.committer.magic.enabled\n true\n\n```\n\n*IMPORTANT*: only enable the magic committer against object stores which\noffer consistent listings. By default, Amazon S3 does not do this -which is\nwhy the option `fs.s3a.committer.magic.enabled` is disabled by default.\n\n\nTip: you can verify that a bucket supports the magic committer through the\n`hadoop s3guard bucket-info` command:\n\n\n```\n> hadoop s3guard bucket-info -magic s3a://landsat-pds/\n\nFilesystem s3a://landsat-pds\nLocation: us-west-2\nFilesystem s3a://landsat-pds is not using S3Guard\nThe \"magic\" committer is not supported\n\nS3A Client\n Endpoint: fs.s3a.endpoint=(unset)\n Encryption: fs.s3a.server-side-encryption-algorithm=none\n Input seek policy: fs.s3a.experimental.input.fadvise=normal\n2017-09-27 19:18:57,917 INFO util.ExitUtil: Exiting with status 46: 46: The magic committer is not enabled for s3a://landsat-pds\n```\n\n## Error message: \"File being created has a magic path, but the filesystem has magic file support disabled:\n\nA file is being written to a path which is used for \"magic\" files,\nfiles which are actually written to a different destination than their stated path\n*but the filesystem doesn't support \"magic\" files*\n\nThis message should not appear through the committer itself —it will\nfail with the error message in the previous section, but may arise\nif other applications are attempting to create files under the path `/__magic/`.\n\nMake sure the filesystem meets the requirements of the magic committer\n(a consistent S3A filesystem through S3Guard or the S3 service itself),\nand set the `fs.s3a.committer.magic.enabled` flag to indicate that magic file\nwrites are supported.\n\n\n### `FileOutputCommitter` appears to be still used (from logs or delays in commits)\n\nThe Staging committers use the original `FileOutputCommitter` to manage\nthe propagation of commit information: do not worry if it the logs show `FileOutputCommitter`\nwork with data in the cluster filesystem (e.g. HDFS).\n\nOne way to make sure that the `FileOutputCommitter` is not being used to write\nthe data to S3 is to set the option `mapreduce.fileoutputcommitter.algorithm.version`\nto a value such as \"10\". Because the only supported algorithms are \"1\" and \"2\",\nany erroneously created `FileOutputCommitter` will raise an exception in its constructor\nwhen instantiated:\n\n```\njava.io.IOException: Only 1 or 2 algorithm version is supported\nat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:130)\nat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:104)\nat org.apache.parquet.hadoop.ParquetOutputCommitter.(ParquetOutputCommitter.java:42)\nat org.apache.parquet.hadoop.ParquetOutputFormat.getOutputCommitter(ParquetOutputFormat.java:395)\nat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupCommitter(HadoopMapReduceCommitProtocol.scala:67)\nat com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol.setupCommitter(PathOutputCommitProtocol.scala:62)\nat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:124)\nat com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol.setupJob(PathOutputCommitProtocol.scala:152)\nat org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:175)\nat org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:145)\n```\n\nWhile that will not make the problem go away, it will at least make\nthe failure happen at the start of a job.\n\n(Setting this option will not interfere with the Staging Committers' use of HDFS,\nas it explicitly sets the algorithm to \"2\" for that part of its work).\n\nThe other way to check which committer to use is to examine the `_SUCCESS` file.\nIf it is 0-bytes long, the classic `FileOutputCommitter` committed the job.\nThe S3A committers all write a non-empty JSON file; the `committer` field lists\nthe committer used.\n\n\n*Common causes*\n\n1. The property `fs.s3a.committer.name` is set to \"file\". Fix: change.\n1. The job has overridden the property `mapreduce.outputcommitter.factory.class`\nwith a new factory class for all committers. This takes priority over\nall committers registered for the s3a:// schema.\n1. The property `mapreduce.outputcommitter.factory.scheme.s3a` is unset.\n1. The output format has overridden `FileOutputFormat.getOutputCommitter()`\nand is returning its own committer -one which is a subclass of `FileOutputCommitter`.\n\nThat final cause. *the output format is returning its own committer*, is not\neasily fixed; it may be that the custom committer performs critical work\nduring its lifecycle, and contains assumptions about the state of the written\ndata during task and job commit (i.e. it is in the destination filesystem).\nConsult with the authors/maintainers of the output format\nto see whether it would be possible to integrate with the new committer factory\nmechanism and object-store-specific commit algorithms.\n\nParquet is a special case here: its committer does no extra work\nother than add the option to read all newly-created files then write a schema\nsummary. The Spark integration has explicit handling for Parquet to enable it\nto support the new committers, removing this (slow on S3) option.\n\nIf you have subclassed `FileOutputCommitter` and want to move to the\nfactory model, please get in touch.\n\n\n## Job/Task fails with PathExistsException: Destination path exists and committer conflict resolution mode is \"fail\"\n\nThis surfaces when either of two conditions are met.\n\n1. The Directory committer is used with `fs.s3a.committer.staging.conflict-mode` set to\n`fail` and the output/destination directory exists.\nThe job will fail in the driver during job setup.\n1. The Partitioned Committer is used with `fs.s3a.committer.staging.conflict-mode` set to\n`fail` and one of the partitions. The specific task(s) generating conflicting data will fail\nduring task commit, which will cause the entire job to fail.\n\nIf you are trying to write data and want write conflicts to be rejected, this is the correct\nbehavior: there was data at the destination so the job was aborted.\n\n## Staging committer task fails with IOException: No space left on device\n\nThere's not enough space on the local hard disk (real or virtual)\nto store all the uncommitted data of the active tasks on that host.\nBecause the staging committers write all output to the local disk\nand only upload the data on task commits, enough local temporary\nstorage is needed to store all output generated by all uncommitted\ntasks running on the single host. Small EC2 VMs may run out of disk.\n\n1. Make sure that `fs.s3a.buffer.dir` includes a temporary directory on\nevery available hard disk; this spreads load better.\n\n1. Add more disk space. In EC2: request instances with more local storage.\nThere is no need for EMR storage; this is just for temporary data.\n\n1. Purge the directories listed in `fs.s3a.buffer.dir` of old data.\nFailed tasks may not clean up all old files.\n\n1. Reduce the number of worker threads/process in the host.\n\n1. Consider partitioning the job into more tasks. This *may* result in more tasks\ngenerating less data each.\n\n1. Use the magic committer. This only needs enough disk storage to buffer\nblocks of the currently being written file during their upload process,\nso can use a lot less disk space.\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committer_architecture.md":"\n\n# S3A Committers: Architecture and Implementation\n\n\n\nThis document covers the architecture and implementation details of the S3A committers.\n\nFor information on using the committers, see [the S3A Committers](./committer.html).\n\n\n## Problem: Efficient, reliable commits of work to consistent S3 buckets\n\n\nThe standard commit algorithms (the `FileOutputCommitter` and its v1 and v2 algorithms)\nrely on directory rename being an `O(1)` atomic operation: callers output their\nwork to temporary directories in the destination filesystem, then\nrename these directories to the final destination as way of committing work.\nThis is the perfect solution for committing work against any filesystem with\nconsistent listing operations and where the `FileSystem.rename()` command\nis an atomic `O(1)` operation.\n\nUsing rename allows individual tasks to work in temporary directories, with the\nrename as the atomic operation can be used to explicitly commit tasks and\nultimately the entire job. Because the cost of the rename is low, it can be\nperformed during task and job commits with minimal delays. Note that HDFS\nwill lock the namenode metadata during the rename operation, so all rename() calls\nwill be serialized. However, as they only update the metadata of two directory\nentries, the duration of the lock is low.\n\nIn contrast to a \"real\" filesystem, Amazon's S3A object store, similar to\nmost others, does not support `rename()` at all. A hash operation on the filename\ndetermines the location of of the data —there is no separate metadata to change.\nTo mimic renaming, the Hadoop S3A client has to copy the data to a new object\nwith the destination filename, then delete the original entry. This copy\ncan be executed server-side, but as it does not complete until the in-cluster\ncopy has completed, it takes time proportional to the amount of data.\n\nThe rename overhead is the most visible issue, but it is not the most dangerous.\nThat is the fact that path listings have no consistency guarantees, and may\nlag the addition or deletion of files.\nIf files are not listed, the commit operation will *not* copy them, and\nso they will not appear in the final output.\n\nThe solution to this problem is closely coupled to the S3 protocol itself:\ndelayed completion of multi-part PUT operations\n\nThat is: tasks write all data as multipart uploads, *but delay the final\ncommit action until until the final, single job commit action.* Only that\ndata committed in the job commit action will be made visible; work from speculative\nand failed tasks will not be instantiated. As there is no rename, there is no\ndelay while data is copied from a temporary directory to the final directory.\nThe duration of the commit will be the time needed to determine which commit operations\nto construct, and to execute them.\n\n\n## Terminology\n\n* *Job*: a potentially parallelized query/operation to execute. The execution\nof a job: the division of work into tasks and the management of their completion,\nis generally executed in a single process.\n\nThe output of a Job is made visible to other stages in a larger operation\nsequence or other applications if the job *completes successfully*.\n\n* *Job Driver*. Not sure quite what term to use here. Whatever process schedules\ntask execution, tracks success/failures and, determines when all the work has been\nprocessed and then commits the output. It may also determine that a job\nhas failed and cannot be recovered, in which case the job is aborted.\nIn MR and Tez, this is inside the YARN application master.\nIn Spark it is the driver, which can run in the AM, the YARN client, or other\nplaces (e.g Livy?).\n\n* *Final directory*: the directory into which the output of a job is placed\nso as to be visible.\n\n* *Task* a single operation within a job, on a single process, one which generates\none or more files.\nAfter a successful job completion, the data MUST be visible in the final directory.\nA task completes successfully if it generates all the output it expects to without\nfailing in some way (error in processing; network/process failure).\n\n* *Job Context* an instance of the class `org.apache.hadoop.mapreduce.JobContext`,\nwhich provides a read-only view of the Job for the Job Driver and tasks.\n\n* *Task Attempt Context* an instance of the class\n`org.apache.hadoop.mapreduce.TaskAttemptContext extends JobContext, Progressable`,\nwhich provides operations for tasks, such as getting and setting status,\nprogress and counter values.\n\n* *Task Working Directory*: a directory for exclusive access by a single task,\ninto which uncommitted work may be placed.\n\n* *Task Commit* The act of taking the output of a task, as found in the\nTask Working Directory, and making it visible in the final directory.\nThis is traditionally implemented via a `FileSystem.rename()` call.\n\n It is useful to differentiate between a *task-side commit*: an operation performed\n in the task process after its work, and a *driver-side task commit*, in which\n the Job driver performs the commit operation. Any task-side commit work will\n be performed across the cluster, and may take place off the critical part for\n job execution. However, unless the commit protocol requires all tasks to await\n a signal from the job driver, task-side commits cannot instantiate their output\n in the final directory. They may be used to promote the output of a successful\n task into a state ready for the job commit, addressing speculative execution\n and failures.\n\n* *Job Commit* The act of taking all successfully completed tasks of a job,\nand committing them. This process is generally non-atomic; as it is often\na serialized operation at the end of a job, its performance can be a bottleneck.\n\n* *Task Abort* To cancel a task such that its data is not committed.\n\n* *Job Abort* To cancel all work in a job: no task's work is committed.\n\n* *Speculative Task Execution/ \"Speculation\"* Running multiple tasks against the same\ninput dataset in parallel, with the first task which completes being the one\nwhich is considered successful. Its output SHALL be committed; the other task\nSHALL be aborted. There's a requirement that a task can be executed in parallel,\nand that the output of a task MUST NOT BE visible until the job is committed,\nat the behest of the Job driver. There is the expectation that the output\nSHOULD BE the same on each task, though that MAY NOT be the case. What matters\nis if any instance of a speculative task is committed, the output MUST BE\nconsidered valid.\n\nThere is an expectation that the Job Driver and tasks can communicate: if a task\nperforms any operations itself during the task commit phase, it shall only do\nthis when instructed by the Job Driver. Similarly, if a task is unable to\ncommunicate its final status to the Job Driver, it MUST NOT commit is work.\nThis is very important when working with S3, as some network partitions could\nisolate a task from the Job Driver, while the task retains access to S3.\n\n## The execution workflow\n\n\n**setup**:\n\n* A job is created, assigned a Job ID (YARN?).\n* For each attempt, an attempt ID is created, to build the job attempt ID.\n* `Driver`: a `JobContext` is created/configured\n* A committer instance is instantiated with the `JobContext`; `setupJob()` invoked.\n\n\n## The `FileOutputCommitter`\n\nThe standard commit protocols are implemented in\n `org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter`.\n\nThere are two algorithms, the \"v1\" designed to address failures and restarts\nof the MapReduce application master. The V2 algorithm cannot recover from failure\nexcept by re-executing the entire job. It does, however, propagate all its work\nto the output directory in the task commit. When working with object stores which\nmimic `rename()` by copy and delete, it is more efficient due to the reduced\nnumber of listings, copies and deletes, and, because these are executed in\ntask commits, eliminates the final `O(data)` pause at the end of all work.\nIt is still highly inefficient, but the inefficiencies are less visible as\nlarge pauses in execution.\n\n\nNotes\n\n* The v1 algorithm was implemented to handle MapReduce AM restarts, it was\n not used in Hadoop 1.x, whose JobTracker could not recover from failures.\n Historically then, it is the second version of a file commit algorithm.\n* Because the renames are considered to be fast, there is no logging\nof renames being in progress, or their duration.\n* Speculative execution is supported by having every task attempt write its\nuncommitted data to a task attempt directory. When a task is ready to commit,\nit must notify that job driver that it is ready to commit -the job driver\nwill then commit or abort the task.\n\n## Hadoop MR Commit algorithm \"1\"\n\n\nThe \"v1\" commit algorithm is the default commit algorithm in Hadoop 2.x;\nit was implemented as part of [MAPREDUCE-2702](https://issues.apache.org/jira/browse/MAPREDUCE-2702).\n\nThis algorithm is designed to handle a failure and restart of the Job driver,\nwith the restarted job driver only rerunning the incomplete tasks; the\noutput of the completed tasks is recovered for commitment when the restarted\njob completes.\n\nThere is a cost: the time to commit by recursive listing all files in all\ncommitted task directories, and renaming this.\n\nAs this is performed sequentially, time to commit is `O(files)`, which is\ngenerally `O(tasks)`.\n\n\n\n```python\n#Job Attempt Path is `$dest/_temporary/$appAttemptId/`\n\njobAttemptPath = '$dest/_temporary/$appAttemptId/'\n\n# Task Attempt Path is `$dest/_temporary/$appAttemptId/_temporary/$taskAttemptID`\ntaskAttemptPath = '$dest/_temporary/$appAttemptId/_temporary/$taskAttemptID'\n\n#Task committed path is `$dest/_temporary/$appAttemptId/$taskAttemptID`\ntaskCommittedPath = '$dest/_temporary/$appAttemptId/$taskAttemptID'\n```\n\nTasks write in/under the task attempt path.\n\n### Job Setup\n\n```python\nfs.mkdir(jobAttemptPath)\n```\n\n### Task Setup\n\nNone: directories are created on demand.\n\n\n### Task Commit\n\nRename task attempt path to task committed path.\n\n```python\ndef needsTaskCommit(fs, jobAttemptPath, taskAttemptPath, dest):\n return fs.exists(taskAttemptPath)\n\n\ndef commitTask(fs, jobAttemptPath, taskAttemptPath, dest):\n if fs.exists(taskAttemptPath) :\n fs.delete(taskCommittedPath, recursive=True)\n fs.rename(taskAttemptPath, taskCommittedPath)\n```\n\nOn a genuine filesystem this is an `O(1)` directory rename.\n\nOn an object store with a mimiced rename, it is `O(data)` for the copy,\nalong with overhead for listing and deleting all files (For S3, that's\n`(1 + files/500)` lists, and the same number of delete calls.\n\n\n### Task Abort\n\nDelete task attempt path.\n\n```python\ndef abortTask(fs, jobAttemptPath, taskAttemptPath, dest):\n fs.delete(taskAttemptPath, recursive=True)\n```\n\nOn a genuine filesystem this is an `O(1)` operation. On an object store,\nproportional to the time to list and delete files, usually in batches.\n\n\n### Job Commit\n\nMerge all files/directories in all task committed paths into final destination path.\nOptionally; create 0-byte `_SUCCESS` file in destination path.\n\n```python\ndef commitJob(fs, jobAttemptDir, dest):\n for committedTask in fs.listFiles(jobAttemptDir):\n mergePathsV1(fs, committedTask, dest)\n fs.touch(\"$dest/_SUCCESS\")\n```\n\n(See below for details on `mergePaths()`)\n\n\nA failure during job commit cannot be recovered from except by re-executing\nthe entire query:\n\n```python\ndef isCommitJobRepeatable() :\n return False\n```\n\nAccordingly, it is a failure point in the protocol. With a low number of files\nand fast rename/list algorithms, the window of vulnerability is low. At\nscale, the vulnerability increases. It could actually be reduced through\nparallel execution of the renaming of of committed tasks.\n\n\n### Job Abort\n\nDelete all data under job attempt path.\n\n```python\ndef abortJob(fs, jobAttemptDir, dest):\n fs.delete(jobAttemptDir, recursive = True)\n```\n\n### Job Cleanup\n\n```python\ndef cleanupJob(fs, dest):\n fs.delete('$dest/_temporary', recursive = True)\n```\n\n\n### Job Recovery Before `commitJob()`\n\nFor all committers, the recovery process takes place in the application\nmaster.\n1. The job history file of the previous attempt is loaded and scanned\nto determine which tasks were recorded as having succeeded.\n1. For each successful task, the job committer has its `recoverTask()` method\ninvoked with a `TaskAttemptContext` built from the previous attempt's details.\n1. If the method does not raise an exception, it is considered to have been\nrecovered, and not to be re-executed.\n1. All other tasks are queued for execution.\n\nFor the v1 committer, task recovery is straightforward.\nThe directory of the committed task from the previous attempt is\nmoved under the directory of the current application attempt.\n\n```python\ndef recoverTask(tac):\n oldAttemptId = appAttemptId - 1\n fs.rename('$dest/_temporary/oldAttemptId/${tac.taskId}',\n '$dest/_temporary/appAttemptId/${tac.taskId}')\n```\n\nThis significantly improves time to recover from Job driver (here MR AM) failure.\nThe only lost work is that of all tasks in progress -those which had generated\ndata but were not yet committed.\n\nOnly the failure of the job driver requires a job restart, not an individual\ntask. Therefore the probability of this happening is independent of the number\nof tasks executed in parallel, instead simply due to the duration of the query.\n\nThe longer the task, the higher the risk of failure, the more value there is\nin recovering the work in progress.\n\nFast queries not only have a lower risk of failure, they can recover from\nfailure simply by rerunning the entire job. This is implicitly the strategy\nin Spark, which does not attempt to recover any in-progress jobs. The faster\nyour queries, the simpler your recovery strategy needs to be.\n\n### Job Recovery During `commitJob()`\n\nThis is not possible; a failure during job commit requires the entire job\nto be re-executed after cleaning up the destination directory.\n\n### `mergePaths(FileSystem fs, FileStatus src, Path dest)` Algorithm\n\n`mergePaths()` is the core algorithm to merge data; it is somewhat confusing\nas the implementation mixes the strategies for both algorithms across\ntwo co-recursive routines, `mergePaths()` and `renameOrMerge()`.\n\n\nHere the two algorithms have been split, and one of the co-recursive methods\ninlined.\n\n```python\ndef mergePathsV1(fs, src, dest) :\n if fs.exists(dest) :\n toStat = fs.getFileStatus(dest)\n else:\n toStat = None\n\n if src.isFile :\n if not toStat is None :\n fs.delete(dest, recursive = True)\n fs.rename(src.getPath, dest)\n else :\n # src is directory, choose action on dest type\n if not toStat is None :\n if not toStat.isDirectory :\n # Destination exists and is not a directory\n fs.delete(dest)\n fs.rename(src.getPath(), dest)\n else :\n # Destination exists and is a directory\n # merge all children under destination directory\n for child in fs.listStatus(src.getPath) :\n mergePathsV1(fs, child, dest + child.getName)\n else :\n # destination does not exist\n fs.rename(src.getPath(), dest)\n```\n\n## The v2 Commit Algorithm\n\n\nThe v2 algorithm directly commits task output into the destination directory.\nIt is essentially a re-implementation of the Hadoop 1.x commit algorithm.\n\n1. During execution, intermediate data becomes visible.\n1. On a failure, all output must be deleted and the job restarted.\n\nIt implements `mergePaths` differently, as shown below.\n\n```python\ndef mergePathsV2(fs, src, dest) :\n if fs.exists(dest) :\n toStat = fs.getFileStatus(dest)\n else:\n toStat = None\n\n if src.isFile :\n if not toStat is None :\n fs.delete(dest, recursive = True)\n fs.rename(src.getPath, dest)\n else :\n # destination is directory, choose action on source type\n if src.isDirectory :\n if not toStat is None :\n if not toStat.isDirectory :\n # Destination exists and is not a directory\n fs.delete(dest)\n fs.mkdirs(dest) #\n for child in fs.listStatus(src.getPath) : # HERE\n mergePathsV2(fs, child, dest + child.getName) #\n\n else :\n # Destination exists and is a directory\n # merge all children under destination directory\n for child in fs.listStatus(src.getPath) :\n mergePathsV2(fs, child, dest + child.getName)\n else :\n # destination does not exist\n fs.mkdirs(dest) #\n for child in fs.listStatus(src.getPath) : # HERE\n mergePathsV2(fs, child, dest + child.getName) #\n```\n\nBoth recurse down any source directory tree, and commit single files\nby renaming the files.\n\nA a key difference is that the v1 algorithm commits a source directory to\nvia a directory rename, which is traditionally an `O(1)` operation.\n\nIn contrast, the v2 algorithm lists all direct children of a source directory\nand recursively calls `mergePath()` on them, ultimately renaming the individual\nfiles. As such, the number of renames it performs equals the number of source\n*files*, rather than the number of source *directories*; the number of directory\nlistings being `O(depth(src))` , where `depth(path)` is a function returning the\ndepth of directories under the given path.\n\nOn a normal filesystem, the v2 merge algorithm is potentially more expensive\nthan the v1 algorithm. However, as the merging only takes place in task commit,\nit is potentially less of a bottleneck in the entire execution process.\n\nOn an object store, it is suboptimal not just from its expectation that `rename()`\nis an `O(1)` operation, but from its expectation that a recursive tree walk is\nan efficient way to enumerate and act on a tree of data. If the algorithm was\nswitched to using `FileSystem.listFiles(path, recursive)` for a single call to\nenumerate all children under a path, then the listing operation would be significantly\nfaster, at least on a deep or wide tree. However, for any realistic dataset,\nthe size of the output files is likely to be the main cause of delays. That\nis, if the cost of `mergePathsV2` is `O(depth(src)) + O(data))`, then\ngenerally the `O(data)` value will be more significant than the `depth(src)`.\n\nThere is one key exception: tests which work on small amounts of data yet try\nto generate realistic output directory structures. In these tests the cost\nof listing directories and calling `getFileStatus()` could exceed that of the copy\ncalls. This is why small-scale tests of the commit algorithms against object stores\nmust be considered significantly misleading.\n\n### v2 Task Commit\n\nRename task attempt path to task committed path.\n\n```python\ndef needsTaskCommit(fs, jobAttemptPath, taskAttemptPath, dest):\n return fs.exists(taskAttemptPath)\n\ndef commitTask(fs, jobAttemptPath, taskAttemptPath, dest):\n if fs.exists(taskAttemptPath) :\n mergePathsV2(fs. taskAttemptPath, dest)\n```\n\n### v2 Task Abort\n\nDelete task attempt path.\n\n```python\ndef abortTask(fs, jobAttemptPath, taskAttemptPath, dest):\n fs.delete(taskAttemptPath, recursive=True)\n```\n\nCost: `O(1)` for normal filesystems, `O(files)` for object stores.\n\n\n### v2 Job Commit\n\nAs all the task output is already completed, all that is needed is\nto touch the `_SUCCESS` marker.\n\n```python\ndef commitJob(fs, jobAttemptDir, dest):\n fs.touch(\"$dest/_SUCCESS\")\n```\nCost: `O(1)`\n\nA failure during job abort is implicitly repeatable\n\n```python\ndef isCommitJobRepeatable() :\n return True\n```\n\n### v2 Job Abort\n\nDelete all data under job attempt path.\n\n```python\ndef abortJob(fs, jobAttemptDir, dest):\n fs.delete(jobAttemptDir, recursive=True)\n```\n\nCost: `O(1)` for normal filesystems, `O(files)` for object stores.\n\n### v2 Task Recovery\n\nAs no data is written to the destination directory, a task can be cleaned up\nby deleting the task attempt directory.\n\n### v2 Job Recovery Before `commitJob()`\n\n\nBecause the data has been renamed into the destination directory, all tasks\nrecorded as having being committed have no recovery needed at all:\n\n```python\ndef recoverTask(tac):\n```\n\nAll active and queued tasks are scheduled for execution.\n\nThere is a weakness here, the same one on a failure during `commitTask()`:\nit is only safe to repeat a task which failed during that commit operation\nif the name of all generated files are constant across all task attempts.\n\nIf the Job AM fails while a task attempt has been instructed to commit,\nand that commit is not recorded as having completed, the state of that\nin-progress task is unknown...really it isn't be safe to recover the\njob at this point.\n\n\n### v2 Job Recovery During `commitJob()`\n\nThis is straightforward: `commitJob()` is re-invoked.\n\n## How MapReduce uses the committer in a task\n\nMapReduce runs each Mapper or Reducer in its own container; it\ngets its own process. Implicitly, this also means that each task gets\nits own instance of every filesystem.\n\nThe work is choreographed in `org.apache.hadoop.mapred.Task`, for which there\nare specific subclassing for the Map (`MapTask`) and reduce (`ReduceTask`).\nThere are also cleanup tasks whose role is simpler: clean things up,\nand a Job setup task which runs `OutputCommittre.setupJob`. That is: even\nthe job setup phase is run in a Task.\n\nMapTask uses the committer to write the output of all the mappers into\nthe filesystem, ready for the reducers. Each partition writes its data\nas a `MapFile`, which is actually two `SequenceFile` files in a directory,:\nthe `data` file of all Key-Value\noutput, and `index` which contains an index of some of the keys in the file.\n\nThis is all written to the local filesystem.\n\nThe `ReduceTask` does the final write to the destination filesystem.\n\nBecause the Map phase uses a committer to commits intermediate work,\nany plug in committer supplied to a process\nthrough any extension mechanism *must* work with the output generated by a mapper.\nThe staging committers do only if unique filenames are disabled, but as they\nand the magic committers are only meant to be used for the final output of work,\nit is somewhat moot. What is important is to be able to use different committers\nfor the map phase as the final reduce. This is implicit if separate committers\nare defined for different filesystems: the new committer can be defined for\nthe final destination FS, while `file://` can retain the default\n`FileOutputCommitter`.\n\n### Task Setup\n\n`Task.initialize()`: read in the configuration, instantiate the `JobContextImpl`\nand `TaskAttemptContextImpl` instances bonded to the current job & task.\n\n### Task Ccommit\n\nAfter the work is completed, `Task.done()` is invoked. Which is essentially the following\ncodepath:\n\n```java\nif (committer.needsTaskCommit(taskContext)) {\n // get permission to commit from AM\n int retries = MAX_RETRIES;\n while(true) {\n try {\n umbilical.commitPending(taskId, taskStatus);\n break;\n } catch(IOException ie) {\n if (--retries == 0) {\n // FAIL WITHOUT CALLING ABORT\n System.exit(67);\n }\n }\n }\n // commit the work\n try {\n committer.commitTask(taskContext);\n } catch (IOException iee) {\n // failure: abort\n try {\n committer.abortTask(taskContext);\n } catch (IOException ioe) {\n LOG.warn(\"Failure cleaning up: \" +\n StringUtils.stringifyException(ioe));\n }\n throw iee;\n }\n}\n```\n\nThat is: if and only if there is data to write, the Task requests clearance\nto do so from the AM. This ensures that speculative work is not committed if\nanother task has written it, *and that even non-speculative work is not committed\nif the task has lost contact with the AM*.\n\nThat is: task output is not committed if the Job's AM has failed or\nis in a separate network partition from the Task.\n\nIf permission is granted, the commit call is invoked.\nIf this doesn't work, `abortTask()` is called; all\nfailures there are logged and swallowed.\n\n\nThis method actually appears to be limited in that it does not guarantee\nthat `committer.abortTask()` is always called: if the `umbilical.commitPending()` calls\nfail repeatedly and the Task process aborts, `committer.abortTask()` is not called.\nIf this is due to a network partition, we can hope that the AM invokes\n`committer.abortTask()`, and that if it is an AM failure then a restarted AM can\nclean up its previous attempts. For the classic FileOutputCommitter, listing and\ndeleting the previous attempt's data is straightforward. However, for S3 committers\nusing Multipart Upload as the means of uploading uncommitted data, it is critical\nto ensure that pending uploads are always aborted. This can be done by\n\n* Making sure that all task-side failure branches in `Task.done()` call `committer.abortTask()`.\n* Having job commit & abort cleaning up all pending multipart writes to the same directory\ntree. That is: require that no other jobs are writing to the same tree, and so\nlist all pending operations and cancel them.\n* Add a CLI command to verify that there are no commits under a path; returning\na non-zero exit code if there are. This can be used in testing.\n* Add a CLI command to purge commits under a path. This can be used in workflows,\nif felt necessary, and in test runs to isolate the output of different tests.\n* Document a recommended purge-pending-commit timeout for the system\n* Add probes in the integration tests to verify that pending uploads are never\npresent after a write.\n\n\n\n### How MapReduce uses the committer in the Application Master\n\nThe AM uses a committer to set up and commit the job. To support failure\nand recovery of the AM, `OutputCommitter.isRecoverySupported()` is used\nto declare whether all the output of successful tasks can be used in the final\njob, or the entire job needs to be reset and repeated.\n`OutputCommitter.isCommitJobRepeatable()` addresses the other question:\ncan a committer recover from a failure of the commit process itself.\n\nA committer is created in the Application Master, and handed to an instance\nof `org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler`,\nwhich then manages the job-level lifecycle.\n\nIf the MRv1 API is used, the committer is chosen from the value of\n`\"mapred.output.committer.class\"`; in the MRv2 API the output format\nis instantiated, then asked for a committer using a task and task attempt ID of\n0. A committer is obtained from the output format via a call to\n`Committer.getOutputCommitter(taskContext)`, again using the task attempt context\nwith the (job, 0, 0) task and attempt IDs. That is: even for the job committers,\na task context is always passed in to the `OutputFormat` when requesting a committer.\n*It is critical for all implementations of `OutputCommitter.abortTask()` to\nbe able to execute from the AM, rather than the container and host running\nthe task. Furthermore, all information needed for the abort (paths, filesystem instances\n&c) *must* be retrieved from the `TaskAttemptContext` passed to the method,\nrather than relying on fields initiated from the context passed to the constructor.\n\n\n#### AM: Job setup: `OutputCommitter.setupJob()`\n\nThis is initiated in `org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.StartTransition`.\nIt is queued for asynchronous execution in `org.apache.hadoop.mapreduce.v2.app.MRAppMaster.startJobs()`,\nwhich is invoked when the service is started. Thus: the job is set up when the\nAM is started.\n\n\n#### AM: Job Commit: `OutputCommitter.commitJob()`\n\nIn the \"default\"/cluster filesystem, the `CommitterEventHandler` uses data in\nthe staging area defined in `yarn.app.mapreduce.am.staging-dir`\n(default `/tmp/hadoop-yarn/staging/$user/.staging`), in a subdirectory\nnamed from the job ID.\n\nThree paths are built up for using the filesystem for tracking the state\nof the commit process, with a goal of making the commit operation recoverable,\nwhere supported.\n\n\n| name | role |\n|------|--------|\n| `COMMIT_STARTED` | mark the commencement of the job commit |\n| `COMMIT_SUCCESS` | mark the successful completion of the job commit |\n| `COMMIT_FAIL` | mark the failure of the job commit |\n\nThese markers are used to manage job restart/failure of they happen during\nthe job commit itself.\n\nWhen an AM starts up, it looks in its staging area for these files. as a way\nto determine the previous state of the job. If there are no `COMMIT_` marker files,\nthe job is considered not to have attempted to commit itself yet.\n\n\nThe presence of `COMMIT_SUCCESS` or `COMMIT_FAIL` are taken as evidence\nthat the previous job completed successfully or unsuccessfully; the AM\nthen completes with a success/failure error code, without attempting to rerun\nthe job.\n\nIf `COMMIT_STARTED` exists but not either of the completion markers, then,\nif the committer declares that its job commit operation is repeatable\n(`Committer.isCommitJobRepeatable(jobContext) == true`), then an attempt\nis made to recommit the job, deleting the `COMMIT_STARTED` and commencing\nthe commit process again.\n\nThese `COMMIT_STARTED` files are simply 0-byte files, but are created\nwith the overwrite bit only set to true if the job commit is considered\nrepeatable:\n\n```java\nprivate void touchz(Path p, boolean overwrite) throws IOException {\n fs.create(p, overwrite).close();\n}\n```\nThat is: the atomicity of the `create(path, overwrite=false)` on the cluster\nfilesystem is used to guarantee that only one process will attempt to commit\na specific job.\n\n```java\nboolean commitJobIsRepeatable = committer.isCommitJobRepeatable(\n event.getJobContext());\ntry {\n touchz(startCommitFile, commitJobIsRepeatable);\n waitForValidCommitWindow();\n committer.commitJob(event.getJobContext());\n touchz(endCommitSuccessFile, commitJobIsRepeatable);\n} catch (Exception e) {\n touchz(endCommitFailureFile, commitJobIsRepeatable);\n}\n```\n\nThe `waitForValidCommitWindow()` operation is important: it declares that\nthe committer must not commit unless there has been communication with the YARN\nResource Manager with in `yarn.app.mapreduce.am.job.committer.commit-window` milliseconds\n(default: 10,000). It does this by waiting until the next heartbeat it received.\nThere's a possible bug here: if the interval is set too small the thread may\npermanently spin waiting a callback within the window. Ignoring that, this algorithm\nguarantees that\n\n1. As only one call can create a file with `overwrite=false`,\n only one process's attempt to commit a non-repeatable job will proceed\n\n1. Only a process with contact with the YARN within the configured window\n may commit a job.\n\n1. If the AM is partitioned from the rest of the network, provided that its clock\nis monotonically increasing at the same rate as the rest of the cluster, then\nthe rest of the cluster can be confident that\n`yarn.app.mapreduce.am.ob.committer.commit-window` milliseconds after the AM\nsuccessfully heartbeated to the YARN RM, then the output of this job attempt\nwill *never* be committed. This permits 1 job to run simultaneously, but\nhelps ensure that only one of them will attempt to commit.\n\n1. Provided YARN heartbeats are only sent to the AM which successfully created\nthe `COMMIT_STARTED` file, it will initiate the commit operation.\n\nPossible issues with this algorithm:\n\n* The `COMMIT_STARTED` file is created before waiting to get a heartbeat. It\nmay be that this AM has lost contact with YARN, but doesn't know it yet. When the\nYARN liveness protocols eventually time out, the AM will correctly terminate,\nbut as the `COMMIT_STARTED` file has been created at this point, no other launched\nAM will be able to commit.\n\n* If two committers attempt to create `COMMIT_STARTED` files on a no-repeatable\ncommit, one will succeed, wait for a heartbeat then attempt a (possibly slow) commit.\nThe second committer will fail, and will *immediately* create a `COMMIT_FAILED` file.\nAs a result, the state of the staging area will imply that the commit has failed,\nwhen really it is in progress, and that only the second process failed.\n\nIt would seem safer to address this through\n1. Waiting for a heartbeat before creating the `COMMIT_STARTED` file.\n1. Maybe: not creating the `COMMIT_FAILED` file if the failure happens when\ntrying to create the `COMMIT_STARTED` file. That is: only a process which\nsuccessfully created the `COMMIT_STARTED` file may indicate that a commit has failed.\n\n\n### AM: Cancelling job commit\n\nThe thread performing the commit is interrupted; the `CommitterEventHandler`\nawaits for it to finish. (set in `yarn.app.mapreduce.am.job.committer.cancel-timeout`\nas milliseconds).\n\n### AM: Task Abort\n\nThe AM may call the `OutputCommitter.taskAbort()` with a task attempt context,\nwhen handling the failure/loss of a container. That is: on container failure,\nthe task abort operation is executed in the AM, using the AM's committer.\nThis avoids the need to create a new container, and means that the \"best-effort\"\ntask abort does cope with container failures.\n\nA partition between the AM and the task container means that this AM-executed\ntask abort may take place while a task in the partitioned container is still\nexecuting. Unless output writing operations will fail after the abort operation,\nthe partitioned task may not become aware of the partition until it's own task\ncommit sequence in `Task.done()`, when `talkToAMTGetPermissionToCommit()`\n\n# Requirements of an S3A Committer\n\n1. Support an eventually consistent S3 object store as a reliable direct\ndestination of work through the S3A filesystem client.\n1. Efficient: implies no rename, and a minimal amount of delay in the job driver's\ntask and job commit phases,\n1. Support task failure and speculation.\n1. Can be used by existing code: Hadoop MapReduce, Spark, Hive.\n1. Retrofittable to existing subclasses of FileOutputFormat and/or compatible\nwith committers which expect a specific FileOutputFormat.\n1. Clean up uncommitted data from all task attempts, all previous attempts of\nthe job, and any previous incompleted jobs.\n1. Security: not to permit privilege escalation from other users with\nwrite access to the same file system(s).\n\n\n## Features of S3 and the S3A Client\n\n\nA core problem is that\n[object stores are not filesystems](../../../hadoop-project-dist/hadoop-common/filesystem/introduction.html);\nhow `rename()` has been emulated in the S3A client means that both the existing\nMR committer algorithms have significant performance problems.\n\n1. Single-object renames are implemented as a copy and delete sequence.\n1. COPY is atomic, but overwrites cannot be prevented.\n1. Amazon S3 is eventually consistent on listings, deletes and updates.\n1. Amazon S3 has create consistency, however, the negative response of a HEAD/GET\nperformed on a path before an object was created can be cached, unintentionally\ncreating a create inconsistency. The S3A client library does perform such a check,\non `create()` and `rename()` to check the state of the destination path, and\nso, whether the operation is permitted.\n1. Multi-object renames are sequential or parallel single object COPY+DELETE operations:\nnon atomic, `O(data)` and, on failure, can leave the filesystem in an unknown\nstate.\n1. There is a PUT operation, capable of uploading 5GB of data in one HTTP request.\n1. The PUT operation is atomic, but there is no PUT-no-overwrite option.\n1. There is a multipart POST/PUT sequence for uploading larger amounts of data\nin a sequence of PUT requests.\n\n\nThe Hadoop S3A Filesystem client supports PUT and multipart PUT for uploading\ndata, with the `S3ABlockOutputStream` of HADOOP-13560 uploading written data\nas parts of a multipart PUT once the threshold set in the configuration\nparameter `fs.s3a.multipart.size` (default: 100MB).\n\n[S3Guard](./s3guard.html) adds an option of consistent view of the filesystem\nto all processes using the shared DynamoDB table as the authoritative store of\nmetadata. Some S3-compatible object stores are fully consistent; the\nproposed algorithm is designed to work with such object stores without the\nneed for any DynamoDB tables.\n\n## Related work: Spark's `DirectOutputCommitter`\n\nOne implementation to look at is the\n[`DirectOutputCommitter` of Spark 1.6](https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/DirectParquetOutputCommitter.scala).\n\nThis implements a zero rename commit by subclassing the `ParquetOutputCommitter`\nand then\n\n1. Returning the destination directory as the task working directory.\n1. Subclassing all the task commit/abort operations to be no-ops.\n\nWith the working directory as the destination directory, there is no need\nto move/rename the task output on a successful commit. However, it is flawed.\nThere is no notion of \"committing\" or \"aborting\" a task, hence no ability to\nhandle speculative execution or failures. This is why the committer\nwas removed from Spark 2 [SPARK-10063](https://issues.apache.org/jira/browse/SPARK-10063)\n\nThere is also the issue that work-in-progress data is visible; this may or may\nnot be a problem.\n\n## Related work: IBM's \"Stocator\" committer\n\nIBM's [Stocator](https://github.com/SparkTC/stocator) can transform indirect\nwrites of V1/V2 committers into direct writes to the destination directory.\n\nHpw does it do this? It's a special Hadoop `FileSystem` implementation which\nrecognizes writes to `_temporary` paths and translate them to writes to the\nbase directory. As well as translating the write operation, it also supports\na `getFileStatus()` call on the original path, returning details on the file\nat the final destination. This allows for committing applications to verify\nthe creation/existence/size of the written files (in contrast to the magic\ncommitter covered below).\n\nThe FS targets Openstack Swift, though other object stores are supportable through\ndifferent backends.\n\nThis solution is innovative in that it appears to deliver the same semantics\n(and hence failure modes) as the Spark Direct OutputCommitter, but which\ndoes not need any change in either Spark *or* the Hadoop committers. In contrast,\nthe committers proposed here combines changing the Hadoop MR committers for\nease of pluggability, and offers a new committer exclusively for S3, one\nstrongly dependent upon and tightly integrated with the S3A Filesystem.\n\nThe simplicity of the Stocator committer is something to appreciate.\n\n## Background: The S3 multi-part PUT mechanism\n\nIn the [S3 REST API](http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html),\nmultipart uploads allow clients to upload a series of \"Parts\" of a file,\nthen commit the upload with a final call.\n\n1. Caller initiates a multipart request, including the destination bucket, key\nand metadata.\n\n POST bucket.s3.aws.com/path?uploads\n\n An `UploadId` is returned\n\n1. Caller uploads one or more parts.\n\n PUT bucket.s3.aws.com/path?partNumber=PartNumber&uploadId=UploadId\n\n The part number is used to declare the ordering of the PUTs; they\n can be uploaded in parallel and out of order.\n All parts *excluding the final part* must be 5MB or larger.\n Every upload completes with an etag returned\n\n1. Caller completes the operation\n\n POST /ObjectName?uploadId=UploadId\n \n (number)(Tag)\n ...\n \n\n This final call lists the etags of all uploaded parts and the actual ordering\n of the parts within the object.\n\nThe completion operation is apparently `O(1)`; presumably the PUT requests\nhave already uploaded the data to the server(s) which will eventually be\nserving up the data for the final path. All that is needed to complete\nthe upload is to construct an object by linking together the files in\nthe server's local filesystem and update an entry the index table of the\nobject store.\n\nIn the S3A client, all PUT calls in the sequence and the final commit are\ninitiated by the same process. *This does not have to be the case*.\nIt is that fact, that a different process may perform different parts\nof the upload, which make this algorithm viable.\n\n\n## The Netfix \"Staging\" committer\n\nRyan Blue, of Netflix, has submitted an alternate committer, one which has a\nnumber of appealing features\n\n* Doesn't have any requirements of the destination object store,\n* Known to work.\n\nThe final point is not to be underestimated, es not even\na need for a consistency layer.\n* Overall a simpler design. Especially given the need to\nbe resilient to the various failure modes which may arise.\n\n\nThe committer writes task outputs to a temporary directory on the local FS.\nTask outputs are directed to the local FS by `getTaskAttemptPath` and `getWorkPath`.\nOn task commit, the committer enumerates files in the task attempt directory (ignoring hidden files).\nEach file is uploaded to S3 using the [multi-part upload API](http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html),\n\nThe information needed to commit the upload is saved into HDFS and committed\nvia that protocol: when the job commits, the pending uploads of the successful\ntasks are all committed.\n\n\n### Commit logic\n\nThe core algorithm is as follows:\n\n1. The destination directory for output (e.g. `FileOutputFormat` and subclasses)\nis a local `file://` reference.\n1. Task commit initiates the multipart PUT to the destination object store.\n1. A list of every pending PUT for task is persisted to a single file\nwithin a consistent, cluster-wide filesystem. For Netflix, that is HDFS.\n1. The Standard `FileOutputCommitter` (algorithm 1) is used to manage the commit/abort of these\nfiles. That is: it copies only those lists of files to commit from successful tasks\ninto a (transient) job commit directory.\n1. The S3 job committer reads the pending file list for every task committed\nin HDFS, and completes those put requests.\n\nBy using `FileOutputCommmitter` to manage the propagation of the lists of files\nto commit, the existing commit algorithm implicitly becomes that defining which\nfiles will be committed at the end of the job.\n\n\nThe Netflix contribution has Hadoop `OutputCommitter` implementations for S3.\n\nThere are 3 main classes:\n* `S3MultipartOutputCommitter` is a base committer class that handles commit logic. This should not be used directly.\n* `S3DirectoryOutputCommitter` for writing unpartitioned data to S3 with conflict resolution.\n* `S3PartitionedOutputCommitter` for writing partitioned data to S3 with conflict resolution.\n\nCallers should use `S3DirectoryOutputCommitter` for single-directory outputs,\nor `S3PartitionedOutputCommitter` for partitioned data.\n\n\nThese S3 committers work by writing task outputs to a temporary directory on the local FS.\nTask outputs are directed to the local FS by `getTaskAttemptPath` and `getWorkPath`.\n\n\n### Conflict Resolution\n\nThe single-directory and partitioned committers handle conflict resolution by\nchecking whether target paths exist in S3 before uploading any data.\nThere are three conflict resolution modes, controlled by setting `fs.s3a.committer.staging.conflict-mode`:\n\n* `fail`: Fail a task if an output directory or partition already exists. (Default)\n* `append`: Upload data files without checking whether directories or partitions already exist.\n* `replace`: If an output directory exists, delete it so the new data replaces the current content.\n\nThe partitioned committer enforces the conflict mode when a conflict is detected with output data, not before the job runs.\nConflict resolution differs from an output mode because it does not enforce the mode when there is no conflict.\nFor example, overwriting a partition should remove all sub-partitions and data it contains, whether or not new output is created.\nConflict resolution will only replace partitions that have output data.\n\nWhen the conflict mode is `replace`, conflicting directories are removed during\njob commit. Data is only deleted if all tasks have completed successfully.\n\nA UUID that identifies a write is added to filenames that are uploaded to S3.\nThis allows rolling back data from jobs that fail during job commit (see failure cases below) and avoids\nfile-level conflicts when appending data to existing directories.\n\n\n*Note* the checks for existence are made via `S3AFileSystem.getFileStatus()` requests of the destination paths.\nUnless the view of the S3 store is consistent, it may be that a newly-deleted object\nis still discovered in the probe, so a commit fail, even when there is no longer any actual conflict.\n\n### Performance\n\nCompared to the previous proposal, henceforth the \"magic\" committer, this\ncommitter, the \"staging committer\", adds the extra overhead of uploading\neach file at the end of every task. This is an `O(data)` operation; it can be\nparallelized, but is bounded by the bandwidth from compute node to S3, as\nwell as the write/IOP capacity of the destination shard of S3. If many tasks\ncomplete at or near the same time, there may be a peak of bandwidth load\nslowing down the upload.\n\nTime to commit will be the same, and, given the Netflix committer has already\nimplemented the parallelization logic here, a time of `O(files/threads)`.\n\n### Resilience\n\nThere's already a lot of code in the task and job commits to handle failure.\n\nAny failure in a commit triggers a best-effort abort/revert of the commit\nactions for a task/job.\n\nTask commits delegate to the `FileOutputCommitter` to ensure that only one task's\noutput reaches the job commit.\n\nSimilarly, if a task is aborted, temporary output on the local FS is removed.\n\nIf a task dies while the committer is running, it is possible for data to be\neft on the local FS or as unfinished parts in S3.\nUnfinished upload parts in S3 are not visible to table readers and are cleaned\nup following the rules in the target bucket's life-cycle policy.\n\nFailures during job commit are handled by deleting any files that have already\nbeen completed and aborting the remaining uploads.\nBecause uploads are completed individually, the files that are deleted were visible to readers.\n\nIf the process dies while the job committer is running, there are two possible failures:\n\n1. Some directories that would be replaced have been deleted, but no new data is visible.\n2. Some new data is visible but is not complete, and all replaced directories have been removed.\n Only complete files are visible.\n\nIf the process dies during job commit, cleaning up is a manual process.\nFile names include a UUID for each write so that files can be identified and removed.\n\n\n**Failure during task execution**\n\nAll data is written to local temporary files; these need to be cleaned up.\n\nThe job must ensure that the local (pending) data is purged.\n\n**Failure during task commit**\n\n\nA process failure during the upload process will result in the\nlist of pending multipart PUTs to *not* be persisted to the cluster filesystem.\nThis window is smaller than the entire task execution, but still potentially\nsignificant, at least for large uploads.\n\nPer-file persistence, or incremental overwrites of the upload list may\nreduce the problems here, but there would still be a small risk of\nan outstanding multipart upload not being recorded\n\n**Explicit Task abort before task commit**\n\nTask will delete all local data; no uploads will be initiated.\n\n**Failure to communicate with S3 during data upload**\n\nIf an upload fails, tasks will:\n\n* Retry using the retry policies implemented in the S3AFileSystem classes\nand the AWS libraries.\n* Eventually: attempt to abort outstanding multipart uploads already initiated.\n* Remove temporary files on the local FS.\n\n\n**Explicit Job Abort**\n\nAll in-progress tasks are aborted and cleaned up. The pending commit data\nof all completed tasks can be loaded, outstanding multipart PUT requests aborted.\n\n\nThis is done by\n\n1. Listing all local files, with a best effort read attempt followed by\nan abort of all successfully read files.\n1. List and abort all pending multipart uploads.\n\nBecause of action #2, action #1 is superfluous. It is retained so as to leave\nopen the option of making action #2 a configurable option -which would be\nrequired to handle the use case of >1 partitioned commit running simultaneously/\n\n**Job Driver failure before Job Commit**\n\nBecause the local data is managed with the v1 commit algorithm, the\nsecond attempt of the job will recover all the outstanding commit data\nof the first attempt; those tasks will not be rerun.\n\nThis also ensures that on a job abort, the individual tasks' .pendingset\nfiles can be read and used to initiate the abort of those uploads.\nThat is: a recovered job can clean up the pending writes of the previous job\n\nIf the query engine does not support multiple job attempts, then the\npending commit data will not be recovered; an explicit abort operation will\nneed to be initiated (we will add a CLI command for this), or the S3 bucket\nmust be configured to automatically delete the pending request.\n\n**Job Driver failure during Job Commit**\n\nThose uploads already executed by a failed job commit will persist; those\nyet to execute will remain outstanding.\n\nThe committer currently declares itself as non-recoverable, but that\nmay not actually hold, as the recovery process could be one of:\n\n1. Enumerate all job commits from the .pendingset files (*:= Commits*).\n1. List all outstanding uploads under the destination path (*:= Outstandings*)..\n1. List all written files (for better consistency, via a GET call on the known\nfilenames)\n1. Identify all files which are yet to be completed (*Commits - Written*)/\n1. Verify that the set of pend","hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md":"\n\n# Hadoop-Aliyun module: Integration with Aliyun Web Services\n\n\n\n## Overview\n\nThe `hadoop-aliyun` module provides support for Aliyun integration with\n[Aliyun Object Storage Service (Aliyun OSS)](https://www.aliyun.com/product/oss).\nThe generated JAR file, `hadoop-aliyun.jar` also declares a transitive\ndependency on all external artifacts which are needed for this support — enabling\ndownstream applications to easily use this support.\n\nTo make it part of Apache Hadoop's default classpath, simply make sure\nthat HADOOP_OPTIONAL_TOOLS in hadoop-env.sh has 'hadoop-aliyun' in the list.\n\n### Features\n\n* Read and write data stored in Aliyun OSS.\n* Present a hierarchical file system view by implementing the standard Hadoop\n[`FileSystem`](../api/org/apache/hadoop/fs/FileSystem.html) interface.\n* Can act as a source of data in a MapReduce job, or a sink.\n\n### Warning #1: Object Stores are not filesystems.\n\nAliyun OSS is an example of \"an object store\". In order to achieve scalability\nand especially high availability, Aliyun OSS has relaxed some of the constraints\nwhich classic \"POSIX\" filesystems promise.\n\n\n\nSpecifically\n\n1. Atomic operations: `delete()` and `rename()` are implemented by recursive\nfile-by-file operations. They take time at least proportional to the number of files,\nduring which time partial updates may be visible. `delete()` and `rename()`\ncan not guarantee atomicity. If the operations are interrupted, the filesystem\nis left in an intermediate state.\n2. File owner and group are persisted, but the permissions model is not enforced.\nAuthorization occurs at the level of the entire Aliyun account via\n[Aliyun Resource Access Management (Aliyun RAM)](https://www.aliyun.com/product/ram).\n3. Directory last access time is not tracked.\n4. The append operation is not supported.\n\n### Warning #2: Directory last access time is not tracked,\nFeatures of Hadoop relying on this can have unexpected behaviour. E.g. the\nAggregatedLogDeletionService of YARN will not remove the appropriate log files.\n\n### Warning #3: Your Aliyun credentials are valuable\n\nYour Aliyun credentials not only pay for services, they offer read and write\naccess to the data. Anyone with the account can not only read your datasets\n— they can delete them.\n\nDo not inadvertently share these credentials through means such as\n1. Checking in to SCM any configuration files containing the secrets.\n2. Logging them to a console, as they invariably end up being seen.\n3. Defining filesystem URIs with the credentials in the URL, such as\n`oss://accessKeyId:accessKeySecret@directory/file`. They will end up in\nlogs and error messages.\n4. Including the secrets in bug reports.\n\nIf you do any of these: change your credentials immediately!\n\n### Warning #4: The Aliyun OSS client provided by Aliyun E-MapReduce are different from this implementation\n\nSpecifically: on Aliyun E-MapReduce, `oss://` is also supported but with\na different implementation. If you are using Aliyun E-MapReduce,\nfollow these instructions —and be aware that all issues related to Aliyun\nOSS integration in E-MapReduce can only be addressed by Aliyun themselves:\nplease raise your issues with them.\n\n## OSS\n\n### Authentication properties\n\n \n fs.oss.accessKeyId\n Aliyun access key ID\n \n\n \n fs.oss.accessKeySecret\n Aliyun access key secret\n \n\n \n fs.oss.credentials.provider\n \n Class name of a credentials provider that implements\n com.aliyun.oss.common.auth.CredentialsProvider. Omit if using access/secret keys\n or another authentication mechanism. The specified class must provide an\n accessible constructor accepting java.net.URI and\n org.apache.hadoop.conf.Configuration, or an accessible default constructor.\n \n \n\n### Other properties\n\n \n fs.oss.endpoint\n Aliyun OSS endpoint to connect to. An up-to-date list is\n provided in the Aliyun OSS Documentation.\n \n \n\n \n fs.oss.assumed.role.arn\n \n Role ARN for the role to be assumed.\n Required if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.sts.endpoint\n \n STS Token Service endpoint.\n Required if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.session.name\n \n \n Session name for the assumed role, must be valid characters\n according to Aliyun API. It is optional, will be generated by\n oss java sdk if it is empty.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.session.duration\n \n \n Duration of assumed roles before it is expired. Default is 30 minutes.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.sts.expiredFactor\n \n \n Sts token will be refreshed after (expiredFactor * duration) seconds.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.proxy.host\n Hostname of the (optinal) proxy server for Aliyun OSS connection\n \n\n \n fs.oss.proxy.port\n Proxy server port\n \n\n \n fs.oss.proxy.username\n Username for authenticating with proxy server\n \n\n \n fs.oss.proxy.password\n Password for authenticating with proxy server.\n \n\n \n fs.oss.proxy.domain\n Domain for authenticating with proxy server.\n \n\n \n fs.oss.proxy.workstation\n Workstation for authenticating with proxy server.\n \n\n \n fs.oss.attempts.maximum\n 20\n How many times we should retry commands on transient errors.\n \n\n \n fs.oss.connection.establish.timeout\n 50000\n Connection setup timeout in milliseconds.\n \n\n \n fs.oss.connection.timeout\n 200000\n Socket connection timeout in milliseconds.\n \n\n \n fs.oss.paging.maximum\n 1000\n How many keys to request from Aliyun OSS when doing directory listings at a time.\n \n \n\n \n fs.oss.multipart.upload.size\n 10485760\n Size of each of multipart pieces in bytes.\n \n\n \n fs.oss.multipart.upload.threshold\n 20971520\n Minimum size in bytes before we start a multipart uploads or copy.\n \n\n \n fs.oss.multipart.download.size\n 102400/value>\n Size in bytes in each request from ALiyun OSS.\n \n\n \n fs.oss.buffer.dir\n Comma separated list of directories to buffer OSS data before uploading to Aliyun OSS\n \n\n \n fs.oss.acl.default\n \n Set a canned ACL for bucket. Value may be private, public-read, public-read-write.\n \n \n\n \n fs.oss.server-side-encryption-algorithm\n \n Specify a server-side encryption algorithm for oss: file system.\n Unset by default, and the only other currently allowable value is AES256.\n \n \n\n \n fs.oss.connection.maximum\n 32\n Number of simultaneous connections to oss.\n \n\n \n fs.oss.connection.secure.enabled\n true\n Connect to oss over ssl or not, true by default.\n \n\n## Testing the hadoop-aliyun Module\n\nTo test `oss://` filesystem client, two files which pass in authentication\ndetails to the test runner are needed.\n\n1. `auth-keys.xml`\n2. `core-site.xml`\n\nThose two configuration files must be put into\n`hadoop-tools/hadoop-aliyun/src/test/resources`.\n\n### `core-site.xml`\n\nThis file pre-exists and sources the configurations created in `auth-keys.xml`.\n\nFor most cases, no modification is needed, unless a specific, non-default property\nneeds to be set during the testing.\n\n### `auth-keys.xml`\n\nThis file triggers the testing of Aliyun OSS module. Without this file,\n*none of the tests in this module will be executed*\n\nIt contains the access key Id/secret and proxy information that are needed to\nconnect to Aliyun OSS, and an OSS bucket URL should be also provided.\n\n1. `test.fs.oss.name` : the URL of the bucket for Aliyun OSS tests\n\nThe contents of the bucket will be cleaned during the testing process, so\ndo not use the bucket for any purpose other than testing.\n\n### Run Hadoop contract tests\nCreate file `contract-test-options.xml` under `/test/resources`. If a\nspecific file `fs.contract.test.fs.oss` test path is not defined, those\ntests will be skipped. Credentials are also needed to run any of those\ntests, they can be copied from `auth-keys.xml` or through direct\nXInclude inclusion. Here is an example of `contract-test-options.xml`:\n\n \n \n \n\n \n\n \n fs.contract.test.fs.oss\n oss://spark-tests\n \n\n \n fs.oss.impl\n org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem\n \n\n \n fs.oss.endpoint\n oss-cn-hangzhou.aliyuncs.com\n \n\n \n fs.oss.buffer.dir\n /tmp/oss\n \n\n \n fs.oss.multipart.download.size\n 102400\n \n \n","hadoop-tools/hadoop-resourceestimator/README.md":"Resource Estimator Service\n==========================\n\nResource Estimator Service can parse the history logs of production jobs, extract their resource consumption skylines in the past runs and predict their resource requirements for the new run.\n\n## Current Status\n\n * Support [Hadoop YARN ResourceManager](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) logs.\n * In-memory store for parsed history resource skyline and estimation.\n * A [Linear Programming](https://github.com/optimatika/ojAlgo) based estimator.\n * Provides REST interface to parse logs, query history store and estimations.\n\n## Upcoming features\n\n * UI to query history and edit and save estimations.\n * Persisent store implementation for store (either DB based or distributed key-value like HBase).\n * Integrate directly with the [Hadoop YARN Reservation System](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ReservationSystem.html) to make a recurring reservation based on the estimated resources.\n\nRefer to the [design document](https://issues.apache.org/jira/secure/attachment/12886714/ResourceEstimator-design-v1.pdf) for more details.\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md":"\n\n# Maximizing Performance when working with the S3A Connector\n\n\n\n\n## Introduction\n\nS3 is slower to work with than HDFS, even on virtual clusters running on\nAmazon EC2.\n\nThat's because its a very different system, as you can see:\n\n\n| Feature | HDFS | S3 through the S3A connector |\n|---------|------|------------------------------|\n| communication | RPC | HTTP GET/PUT/HEAD/LIST/COPY requests |\n| data locality | local storage | remote S3 servers |\n| replication | multiple datanodes | asynchronous after upload |\n| consistency | consistent data and listings | eventual consistent for listings, deletes and updates |\n| bandwidth | best: local IO, worst: datacenter network | bandwidth between servers and S3 |\n| latency | low | high, especially for \"low cost\" directory operations |\n| rename | fast, atomic | slow faked rename through COPY & DELETE|\n| delete | fast, atomic | fast for a file, slow & non-atomic for directories |\n| writing| incremental | in blocks; not visible until the writer is closed |\n| reading | seek() is fast | seek() is slow and expensive |\n| IOPs | limited only by hardware | callers are throttled to shards in an s3 bucket |\n| Security | Posix user+group; ACLs | AWS Roles and policies |\n\nFrom a performance perspective, key points to remember are:\n\n* S3 throttles bucket access across all callers: adding workers can make things worse.\n* EC2 VMs have network IO throttled based on the VM type.\n* Directory rename and copy operations take *much* longer the more objects and data there is.\nThe slow performance of `rename()` surfaces during the commit phase of jobs,\napplications like `DistCP`, and elsewhere.\n* seek() calls when reading a file can force new HTTP requests.\nThis can make reading columnar Parquet/ORC data expensive.\n\nOverall, although the S3A connector makes S3 look like a file system,\nit isn't, and some attempts to preserve the metaphor are \"aggressively suboptimal\".\n\nTo make most efficient use of S3, care is needed.\n\n## Speeding up directory listing operations through S3Guard\n\n[S3Guard](s3guard.html) provides significant speedups for operations which\nlist files a lot. This includes the setup of all queries against data:\nMapReduce, Hive and Spark, as well as DistCP.\n\n\nExperiment with using it to see what speedup it delivers.\n\n\n## Improving data input performance through fadvise\n\nThe S3A Filesystem client supports the notion of input policies, similar\nto that of the Posix `fadvise()` API call. This tunes the behavior of the S3A\nclient to optimise HTTP GET requests for the different use cases.\n\n### fadvise `sequential`\n\nRead through the file, possibly with some short forward seeks.\n\nThe whole document is requested in a single HTTP request; forward seeks\nwithin the readahead range are supported by skipping over the intermediate\ndata.\n\nThis delivers maximum sequential throughput —but with very expensive\nbackward seeks.\n\nApplications reading a file in bulk (DistCP, any copy operations) should use\nsequential access, as should those reading data from gzipped `.gz` files.\nBecause the \"normal\" fadvise policy starts off in sequential IO mode,\nthere is rarely any need to explicit request this policy.\n\n### fadvise `random`\n\nOptimised for random IO, specifically the Hadoop `PositionedReadable`\noperations —though `seek(offset); read(byte_buffer)` also benefits.\n\nRather than ask for the whole file, the range of the HTTP request is\nset to that that of the length of data desired in the `read` operation\n(Rounded up to the readahead value set in `setReadahead()` if necessary).\n\nBy reducing the cost of closing existing HTTP requests, this is\nhighly efficient for file IO accessing a binary file\nthrough a series of `PositionedReadable.read()` and `PositionedReadable.readFully()`\ncalls. Sequential reading of a file is expensive, as now many HTTP requests must\nbe made to read through the file: there's a delay between each GET operation.\n\n\nRandom IO is best for IO with seek-heavy characteristics:\n\n* Data is read using the `PositionedReadable` API.\n* Long distance (many MB) forward seeks\n* Backward seeks as likely as forward seeks.\n* Little or no use of single character `read()` calls or small `read(buffer)`\ncalls.\n* Applications running close to the S3 data store. That is: in EC2 VMs in\nthe same datacenter as the S3 instance.\n\nThe desired fadvise policy must be set in the configuration option\n`fs.s3a.experimental.input.fadvise` when the filesystem instance is created.\nThat is: it can only be set on a per-filesystem basis, not on a per-file-read\nbasis.\n\n```xml\n\n fs.s3a.experimental.input.fadvise\n random\n \n Policy for reading files.\n Values: 'random', 'sequential' or 'normal'\n \n\n```\n\n[HDFS-2744](https://issues.apache.org/jira/browse/HDFS-2744),\n*Extend FSDataInputStream to allow fadvise* proposes adding a public API\nto set fadvise policies on input streams. Once implemented,\nthis will become the supported mechanism used for configuring the input IO policy.\n\n### fadvise `normal` (default)\n\nThe `normal` policy starts off reading a file in `sequential` mode,\nbut if the caller seeks backwards in the stream, it switches from\nsequential to `random`.\n\nThis policy essentially recognizes the initial read pattern of columnar\nstorage formats (e.g. Apache ORC and Apache Parquet), which seek to the end\nof a file, read in index data and then seek backwards to selectively read\ncolumns. The first seeks may be be expensive compared to the random policy,\nhowever the overall process is much less expensive than either sequentially\nreading through a file with the `random` policy, or reading columnar data\nwith the `sequential` policy.\n\n\n## Committing Work in MapReduce and Spark\n\nHadoop MapReduce, Apache Hive and Apache Spark all write their work\nto HDFS and similar filesystems.\nWhen using S3 as a destination, this is slow because of the way `rename()`\nis mimicked with copy and delete.\n\nIf committing output takes a long time, it is because you are using the standard\n`FileOutputCommitter`. If you are doing this on any S3 endpoint which lacks\nlist consistency (Amazon S3 without [S3Guard](s3guard.html)), this committer\nis at risk of losing data!\n\n*Your problem may appear to be performance, but that is a symptom\nof the underlying problem: the way S3A fakes rename operations means that\nthe rename cannot be safely be used in output-commit algorithms.*\n\nFix: Use one of the dedicated [S3A Committers](committers.md).\n\n## Options to Tune\n\n### Thread and connection pool sizes.\n\nEach S3A client interacting with a single bucket, as a single user, has its\nown dedicated pool of open HTTP 1.1 connections alongside a pool of threads used\nfor upload and copy operations.\nThe default pool sizes are intended to strike a balance between performance\nand memory/thread use.\n\nYou can have a larger pool of (reused) HTTP connections and threads\nfor parallel IO (especially uploads) by setting the properties\n\n\n| property | meaning | default |\n|----------|---------|---------|\n| `fs.s3a.threads.max`| Threads in the AWS transfer manager| 10 |\n| `fs.s3a.connection.maximum`| Maximum number of HTTP connections | 10|\n\nWe recommend using larger values for processes which perform\na lot of IO: `DistCp`, Spark Workers and similar.\n\n```xml\n\n fs.s3a.threads.max\n 20\n\n\n fs.s3a.connection.maximum\n 20\n\n```\n\nBe aware, however, that processes which perform many parallel queries\nmay consume large amounts of resources if each query is working with\na different set of s3 buckets, or are acting on behalf of different users.\n\n### For large data uploads, tune the block size: `fs.s3a.block.size`\n\nWhen uploading data, it is uploaded in blocks set by the option\n`fs.s3a.block.size`; default value \"32M\" for 32 Megabytes.\n\nIf a larger value is used, then more data is buffered before the upload\nbegins:\n\n```xml\n\n fs.s3a.block.size\n 128M\n\n```\n\nThis means that fewer PUT/POST requests are made of S3 to upload data,\nwhich reduces the likelihood that S3 will throttle the client(s)\n\n### Maybe: Buffer Write Data in Memory\n\nWhen large files are being uploaded, blocks are saved to disk and then\nqueued for uploading, with multiple threads uploading different blocks\nin parallel.\n\nThe blocks can be buffered in memory by setting the option\n`fs.s3a.fast.upload.buffer` to `bytebuffer`, or, for on-heap storage\n`array`.\n\n1. Switching to in memory-IO reduces disk IO, and can be faster if the bandwidth\nto the S3 store is so high that the disk IO becomes the bottleneck.\nThis can have a tangible benefit when working with on-premise S3-compatible\nobject stores with very high bandwidth to servers.\n\nIt is very easy to run out of memory when buffering to it; the option\n`fs.s3a.fast.upload.active.blocks\"` exists to tune how many active blocks\na single output stream writing to S3 may have queued at a time.\n\nAs the size of each buffered block is determined by the value of `fs.s3a.block.size`,\nthe larger the block size, the more likely you will run out of memory.\n\n## DistCP\n\nDistCP can be slow, especially if the parameters and options for the operation\nare not tuned for working with S3.\n\nTo exacerbate the issue, DistCP invariably puts heavy load against the\nbucket being worked with, which will cause S3 to throttle requests.\nIt will throttle: directory operations, uploads of new data, and delete operations,\namongst other things\n\n### DistCP: Options to Tune\n\n* `-numListstatusThreads ` : set to something higher than the default (1).\n* `-bandwidth ` : use to limit the upload bandwidth per worker\n* `-m ` : limit the number of mappers, hence the load on the S3 bucket.\n\nAdding more maps with the `-m` option does not guarantee better performance;\nit may just increase the amount of throttling which takes place.\nA smaller number of maps with a higher bandwidth per map can be more efficient.\n\n### DistCP: Options to Avoid.\n\nDistCp's `-atomic` option copies up data into a directory, then renames\nit into place, which is the where the copy takes place. This is a performance\nkiller.\n\n* Do not use the `-atomic` option.\n* The `-append` operation is not supported on S3; avoid.\n* `-p` S3 does not have a POSIX-style permission model; this will fail.\n\n\n### DistCP: Parameters to Tune\n\n1. As discussed [earlier](#pooling), use large values for\n`fs.s3a.threads.max` and `fs.s3a.connection.maximum`.\n\n1. Make sure that the bucket is using `sequential` or `normal` fadvise seek policies,\nthat is, `fs.s3a.experimental.fadvise` is not set to `random`\n\n1. Perform listings in parallel by setting `-numListstatusThreads`\nto a higher number. Make sure that `fs.s3a.connection.maximum`\nis equal to or greater than the value used.\n\n1. If using `-delete`, set `fs.trash.interval` to 0 to avoid the deleted\nobjects from being copied to a trash directory.\n\n*DO NOT* switch `fs.s3a.fast.upload.buffer` to buffer in memory.\nIf one distcp mapper runs out of memory it will fail,\nand that runs the risk of failing the entire job.\nIt is safer to keep the default value, `disk`.\n\nWhat is potentially useful is uploading in bigger blocks; this is more\nefficient in terms of HTTP connection use, and reduce the IOP rate against\nthe S3 bucket/shard.\n\n```xml\n\n fs.s3a.threads.max\n 20\n\n\n\n fs.s3a.connection.maximum\n 30\n \n Make greater than both fs.s3a.threads.max and -numListstatusThreads\n \n\n\n\n fs.s3a.experimental.fadvise\n normal\n\n\n\n fs.s3a.block.size\n 128M\n\n\n\n fs.s3a.fast.upload.buffer\n disk\n\n\n\n fs.trash.interval\n 0\n\n```\n\n## hadoop shell commands `fs -rm`\n\nThe `hadoop fs -rm` command can rename the file under `.Trash` rather than\ndeleting it. Use `-skipTrash` to eliminate that step.\n\n\nThis can be set in the property `fs.trash.interval`; while the default is 0,\nmost HDFS deployments have it set to a non-zero value to reduce the risk of\ndata loss.\n\n```xml\n\n fs.trash.interval\n 0\n\n```\n\n\n## Improving S3 load-balancing behavior\n\nAmazon S3 uses a set of front-end servers to provide access to the underlying data.\nThe choice of which front-end server to use is handled via load-balancing DNS\nservice: when the IP address of an S3 bucket is looked up, the choice of which\nIP address to return to the client is made based on the the current load\nof the front-end servers.\n\nOver time, the load across the front-end changes, so those servers considered\n\"lightly loaded\" will change. If the DNS value is cached for any length of time,\nyour application may end up talking to an overloaded server. Or, in the case\nof failures, trying to talk to a server that is no longer there.\n\nAnd by default, for historical security reasons in the era of applets,\nthe DNS TTL of a JVM is \"infinity\".\n\nTo work with AWS better, set the DNS time-to-live of an application which\nworks with S3 to something lower.\nSee [AWS documentation](http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-jvm-ttl.html).\n\n## Troubleshooting network performance\n\nAn example of this is covered in [HADOOP-13871](https://issues.apache.org/jira/browse/HADOOP-13871).\n\n1. For public data, use `curl`:\n\n curl -O https://landsat-pds.s3.amazonaws.com/scene_list.gz\n1. Use `nettop` to monitor a processes connections.\n\n\n## Throttling\n\nWhen many requests are made of a specific S3 bucket (or shard inside it),\nS3 will respond with a 503 \"throttled\" response.\nThrottling can be recovered from, provided overall load decreases.\nFurthermore, because it is sent before any changes are made to the object store,\nis inherently idempotent. For this reason, the client will always attempt to\nretry throttled requests.\n\nThe limit of the number of times a throttled request can be retried,\nand the exponential interval increase between attempts, can be configured\nindependently of the other retry limits.\n\n```xml\n\n fs.s3a.retry.throttle.limit\n 20\n \n Number of times to retry any throttled request.\n \n\n\n\n fs.s3a.retry.throttle.interval\n 500ms\n \n Interval between retry attempts on throttled requests.\n \n\n```\n\nIf a client is failing due to `AWSServiceThrottledException` failures,\nincreasing the interval and limit *may* address this. However, it\nit is a sign of AWS services being overloaded by the sheer number of clients\nand rate of requests. Spreading data across different buckets, and/or using\na more balanced directory structure may be beneficial.\nConsult [the AWS documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).\n\nReading or writing data encrypted with SSE-KMS forces S3 to make calls of\nthe AWS KMS Key Management Service, which comes with its own\n[Request Rate Limits](http://docs.aws.amazon.com/kms/latest/developerguide/limits.html).\nThese default to 1200/second for an account, across all keys and all uses of\nthem, which, for S3 means: across all buckets with data encrypted with SSE-KMS.\n\n### Tips to Keep Throttling down\n\nIf you are seeing a lot of throttling responses on a large scale\noperation like a `distcp` copy, *reduce* the number of processes trying\nto work with the bucket (for distcp: reduce the number of mappers with the\n`-m` option).\n\nIf you are reading or writing lists of files, if you can randomize\nthe list so they are not processed in a simple sorted order, you may\nreduce load on a specific shard of S3 data, so potentially increase throughput.\n\nAn S3 Bucket is throttled by requests coming from all\nsimultaneous clients. Different applications and jobs may interfere with\neach other: consider that when troubleshooting.\nPartitioning data into different buckets may help isolate load here.\n\nIf you are using data encrypted with SSE-KMS, then the\nwill also apply: these are stricter than the S3 numbers.\nIf you believe that you are reaching these limits, you may be able to\nget them increased.\nConsult [the KMS Rate Limit documentation](http://docs.aws.amazon.com/kms/latest/developerguide/limits.html).\n\n### S3Guard and Throttling\n\n\nS3Guard uses DynamoDB for directory and file lookups;\nit is rate limited to the amount of (guaranteed) IO purchased for a\ntable.\n\nTo see the allocated capacity of a bucket, the `hadoop s3guard bucket-info s3a://bucket`\ncommand will print out the allocated capacity.\n\n\nIf significant throttling events/rate is observed here, the pre-allocated\nIOPs can be increased with the `hadoop s3guard set-capacity` command, or\nthrough the AWS Console. Throttling events in S3Guard are noted in logs, and\nalso in the S3A metrics `s3guard_metadatastore_throttle_rate` and\n`s3guard_metadatastore_throttled`.\n\nIf you are using DistCP for a large backup to/from a S3Guarded bucket, it is\nactually possible to increase the capacity for the duration of the operation.\n\n\n## Best Practises for Code\n\nHere are some best practises if you are writing applications to work with\nS3 or any other object store through the Hadoop APIs.\n\nUse `listFiles(path, recursive)` over `listStatus(path)`.\nThe recursive `listFiles()` call can enumerate all dependents of a path\nin a single LIST call, irrespective of how deep the path is.\nIn contrast, any directory tree-walk implemented in the client is issuing\nmultiple HTTP requests to scan each directory, all the way down.\n\nCache the outcome of `getFileStats()`, rather than repeatedly ask for it.\nThat includes using `isFile()`, `isDirectory()`, which are simply wrappers\naround `getFileStatus()`.\n\nDon't immediately look for a file with a `getFileStatus()` or listing call\nafter creating it, or try to read it immediately.\nThis is where eventual consistency problems surface: the data may not yet be visible.\n\nRely on `FileNotFoundException` being raised if the source of an operation is\nmissing, rather than implementing your own probe for the file before\nconditionally calling the operation.\n\n### `rename()`\n\nAvoid any algorithm which uploads data into a temporary file and then uses\n`rename()` to commit it into place with a final path.\nOn HDFS this offers a fast commit operation.\nWith S3, Wasb and other object stores, you can write straight to the destination,\nknowing that the file isn't visible until you close the write: the write itself\nis atomic.\n\nThe `rename()` operation may return `false` if the source is missing; this\nis a weakness in the API. Consider a check before calling rename, and if/when\na new rename() call is made public, switch to it.\n\n\n### `delete(path, recursive)`\n\nKeep in mind that `delete(path, recursive)` is a no-op if the path does not exist, so\nthere's no need to have a check for the path existing before you call it.\n\n`delete()` is often used as a cleanup operation.\nWith an object store this is slow, and may cause problems if the caller\nexpects an immediate response. For example, a thread may block so long\nthat other liveness checks start to fail.\nConsider spawning off an executor thread to do these background cleanup operations.\n","hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md":"\n\n# Testing the S3A filesystem client and its features, including S3Guard\n\n\n\nThis module includes both unit tests, which can run in isolation without\nconnecting to the S3 service, and integration tests, which require a working\nconnection to S3 to interact with a bucket. Unit test suites follow the naming\nconvention `Test*.java`. Integration tests follow the naming convention\n`ITest*.java`.\n\nDue to eventual consistency, integration tests may fail without reason.\nTransient failures, which no longer occur upon rerunning the test, should thus\nbe ignored.\n\n## Policy for submitting patches which affect the `hadoop-aws` module.\n\nThe Apache Jenkins infrastructure does not run any S3 integration tests,\ndue to the need to keep credentials secure.\n\n### The submitter of any patch is required to run all the integration tests and declare which S3 region/implementation they used.\n\nThis is important: **patches which do not include this declaration will be ignored**\n\nThis policy has proven to be the only mechanism to guarantee full regression\ntesting of code changes. Why the declaration of region? Two reasons\n\n1. It helps us identify regressions which only surface against specific endpoints\nor third-party implementations of the S3 protocol.\n1. It forces the submitters to be more honest about their testing. It's easy\nto lie, \"yes, I tested this\". To say \"yes, I tested this against S3 US-west\"\nis a more specific lie and harder to make. And, if you get caught out: you\nlose all credibility with the project.\n\nYou don't need to test from a VM within the AWS infrastructure; with the\n`-Dparallel=tests` option the non-scale tests complete in under ten minutes.\nBecause the tests clean up after themselves, they are also designed to be low\ncost. It's neither hard nor expensive to run the tests; if you can't,\nthere's no guarantee your patch works. The reviewers have enough to do, and\ndon't have the time to do these tests, especially as every failure will simply\nmake for a slow iterative development.\n\nPlease: run the tests. And if you don't, we are sorry for declining your\npatch, but we have to.\n\n\n### What if there's an intermittent failure of a test?\n\nSome of the tests do fail intermittently, especially in parallel runs.\nIf this happens, try to run the test on its own to see if the test succeeds.\n\nIf it still fails, include this fact in your declaration. We know some tests\nare intermittently unreliable.\n\n### What if the tests are timing out or failing over my network connection?\n\nThe tests and the S3A client are designed to be configurable for different\ntimeouts. If you are seeing problems and this configuration isn't working,\nthat's a sign of the configuration mechanism isn't complete. If it's happening\nin the production code, that could be a sign of a problem which may surface\nover long-haul connections. Please help us identify and fix these problems\n— especially as you are the one best placed to verify the fixes work.\n\n## Setting up the tests\n\nTo integration test the S3* filesystem clients, you need to provide\n`auth-keys.xml` which passes in authentication details to the test runner.\n\nIt is a Hadoop XML configuration file, which must be placed into\n`hadoop-tools/hadoop-aws/src/test/resources`.\n\n### File `core-site.xml`\n\nThis file pre-exists and sources the configurations created\nunder `auth-keys.xml`.\n\nFor most purposes you will not need to edit this file unless you\nneed to apply a specific, non-default property change during the tests.\n\n### File `auth-keys.xml`\n\nThe presence of this file triggers the testing of the S3 classes.\n\nWithout this file, *none of the integration tests in this module will be\nexecuted*.\n\nThe XML file must contain all the ID/key information needed to connect\neach of the filesystem clients to the object stores, and a URL for\neach filesystem for its testing.\n\n1. `test.fs.s3a.name` : the URL of the bucket for S3a tests\n1. `fs.contract.test.fs.s3a` : the URL of the bucket for S3a filesystem contract tests\n\n\nThe contents of the bucket will be destroyed during the test process:\ndo not use the bucket for any purpose other than testing. Furthermore, for\ns3a, all in-progress multi-part uploads to the bucket will be aborted at the\nstart of a test (by forcing `fs.s3a.multipart.purge=true`) to clean up the\ntemporary state of previously failed tests.\n\nExample:\n\n```xml\n\n\n \n test.fs.s3a.name\n s3a://test-aws-s3a/\n \n\n \n fs.contract.test.fs.s3a\n ${test.fs.s3a.name}\n \n\n \n fs.s3a.access.key\n AWS access key ID. Omit for IAM role-based authentication.\n DONOTCOMMITTHISKEYTOSCM\n \n\n \n fs.s3a.secret.key\n AWS secret key. Omit for IAM role-based authentication.\n DONOTEVERSHARETHISSECRETKEY!\n \n\n \n test.sts.endpoint\n Specific endpoint to use for STS requests.\n sts.amazonaws.com\n \n\n\n```\n\n### Configuring S3a Encryption\n\nFor S3a encryption tests to run correctly, the\n`fs.s3a.server-side-encryption.key` must be configured in the s3a contract xml\nfile with a AWS KMS encryption key arn as this value is different for each AWS\nKMS.\n\nExample:\n\n```xml\n\n fs.s3a.server-side-encryption.key\n arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n```\n\nYou can also force all the tests to run with a specific SSE encryption method\nby configuring the property `fs.s3a.server-side-encryption-algorithm` in the s3a\ncontract file.\n\n## Running the Tests\n\nAfter completing the configuration, execute the test run through Maven.\n\n```bash\nmvn clean verify\n```\n\nIt's also possible to execute multiple test suites in parallel by passing the\n`parallel-tests` property on the command line. The tests spend most of their\ntime blocked on network I/O with the S3 service, so running in parallel tends to\ncomplete full test runs faster.\n\n```bash\nmvn -Dparallel-tests clean verify\n```\n\nSome tests must run with exclusive access to the S3 bucket, so even with the\n`parallel-tests` property, several test suites will run in serial in a separate\nMaven execution step after the parallel tests.\n\nBy default, `parallel-tests` runs 4 test suites concurrently. This can be tuned\nby passing the `testsThreadCount` property.\n\n```bash\nmvn -Dparallel-tests -DtestsThreadCount=8 clean verify\n```\n\nTo run just unit tests, which do not require S3 connectivity or AWS credentials,\nuse any of the above invocations, but switch the goal to `test` instead of\n`verify`.\n\n```bash\nmvn clean test\n\nmvn -Dparallel-tests clean test\n\nmvn -Dparallel-tests -DtestsThreadCount=8 clean test\n```\n\nTo run only a specific named subset of tests, pass the `test` property for unit\ntests or the `it.test` property for integration tests.\n\n```bash\nmvn clean test -Dtest=TestS3AInputPolicies\n\nmvn clean verify -Dit.test=ITestS3AFileContextStatistics -Dtest=none\n\nmvn clean verify -Dtest=TestS3A* -Dit.test=ITestS3A*\n```\n\nNote that when running a specific subset of tests, the patterns passed in `test`\nand `it.test` override the configuration of which tests need to run in isolation\nin a separate serial phase (mentioned above). This can cause unpredictable\nresults, so the recommendation is to avoid passing `parallel-tests` in\ncombination with `test` or `it.test`. If you know that you are specifying only\ntests that can run safely in parallel, then it will work. For wide patterns,\nlike `ITestS3A*` shown above, it may cause unpredictable test failures.\n\n### Testing against different regions\n\nS3A can connect to different regions —the tests support this. Simply\ndefine the target region in `auth-keys.xml`.\n\n```xml\n\n fs.s3a.endpoint\n s3.eu-central-1.amazonaws.com\n\n```\nThis is used for all tests expect for scale tests using a Public CSV.gz file\n(see below)\n\n### CSV Data Tests\n\nThe `TestS3AInputStreamPerformance` tests require read access to a multi-MB\ntext file. The default file for these tests is one published by amazon,\n[s3a://landsat-pds.s3.amazonaws.com/scene_list.gz](http://landsat-pds.s3.amazonaws.com/scene_list.gz).\nThis is a gzipped CSV index of other files which amazon serves for open use.\n\nThe path to this object is set in the option `fs.s3a.scale.test.csvfile`,\n\n```xml\n\n fs.s3a.scale.test.csvfile\n s3a://landsat-pds/scene_list.gz\n\n```\n\n1. If the option is not overridden, the default value is used. This\nis hosted in Amazon's US-east datacenter.\n1. If `fs.s3a.scale.test.csvfile` is empty, tests which require it will be skipped.\n1. If the data cannot be read for any reason then the test will fail.\n1. If the property is set to a different path, then that data must be readable\nand \"sufficiently\" large.\n\n(the reason the space or newline is needed is to add \"an empty entry\"; an empty\n`` would be considered undefined and pick up the default)\n\nOf using a test file in an S3 region requiring a different endpoint value\nset in `fs.s3a.endpoint`, a bucket-specific endpoint must be defined.\nFor the default test dataset, hosted in the `landsat-pds` bucket, this is:\n\n```xml\n\n fs.s3a.bucket.landsat-pds.endpoint\n s3.amazonaws.com\n The endpoint for s3a://landsat-pds URLs\n\n```\n\n## Viewing Integration Test Reports\n\n\nIntegration test results and logs are stored in `target/failsafe-reports/`.\nAn HTML report can be generated during site generation, or with the `surefire-report`\nplugin:\n\n```bash\nmvn surefire-report:failsafe-report-only\n```\n## Scale Tests\n\nThere are a set of tests designed to measure the scalability and performance\nat scale of the S3A tests, *Scale Tests*. Tests include: creating\nand traversing directory trees, uploading large files, renaming them,\ndeleting them, seeking through the files, performing random IO, and others.\nThis makes them a foundational part of the benchmarking.\n\nBy their very nature they are slow. And, as their execution time is often\nlimited by bandwidth between the computer running the tests and the S3 endpoint,\nparallel execution does not speed these tests up.\n\n***Note: Running scale tests with -Ds3guard and -Ddynamo requires that\nyou use a private, testing-only DynamoDB table.*** The tests do disruptive\nthings such as deleting metadata and setting the provisioned throughput\nto very low values.\n\n### Enabling the Scale Tests\n\nThe tests are enabled if the `scale` property is set in the maven build\nthis can be done regardless of whether or not the parallel test profile\nis used\n\n```bash\nmvn verify -Dscale\n\nmvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8\n```\n\nThe most bandwidth intensive tests (those which upload data) always run\nsequentially; those which are slow due to HTTPS setup costs or server-side\nactions are included in the set of parallelized tests.\n\n\n### Tuning scale options from Maven\n\n\nSome of the tests can be tuned from the maven build or from the\nconfiguration file used to run the tests.\n\n```bash\nmvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dfs.s3a.scale.test.huge.filesize=128M\n```\n\nThe algorithm is\n\n1. The value is queried from the configuration file, using a default value if\nit is not set.\n1. The value is queried from the JVM System Properties, where it is passed\ndown by maven.\n1. If the system property is null, an empty string, or it has the value `unset`,\nthen the configuration value is used. The `unset` option is used to\n[work round a quirk in maven property propagation](http://stackoverflow.com/questions/7773134/null-versus-empty-arguments-in-maven).\n\nOnly a few properties can be set this way; more will be added.\n\n| Property | Meaning |\n|-----------|-------------|\n| `fs.s3a.scale.test.timeout`| Timeout in seconds for scale tests |\n| `fs.s3a.scale.test.huge.filesize`| Size for huge file uploads |\n| `fs.s3a.scale.test.huge.huge.partitionsize`| Size for partitions in huge file uploads |\n\nThe file and partition sizes are numeric values with a k/m/g/t/p suffix depending\non the desired size. For example: 128M, 128m, 2G, 2G, 4T or even 1P.\n\n### Scale test configuration options\n\nSome scale tests perform multiple operations (such as creating many directories).\n\nThe exact number of operations to perform is configurable in the option\n`scale.test.operation.count`\n\n```xml\n\n scale.test.operation.count\n 10\n\n```\n\nLarger values generate more load, and are recommended when testing locally,\nor in batch runs.\n\nSmaller values results in faster test runs, especially when the object\nstore is a long way away.\n\nOperations which work on directories have a separate option: this controls\nthe width and depth of tests creating recursive directories. Larger\nvalues create exponentially more directories, with consequent performance\nimpact.\n\n```xml\n\n scale.test.directory.count\n 2\n\n```\n\nDistCp tests targeting S3A support a configurable file size. The default is\n10 MB, but the configuration value is expressed in KB so that it can be tuned\nsmaller to achieve faster test runs.\n\n```xml\n\n scale.test.distcp.file.size.kb\n 10240\n\n```\n\nS3A specific scale test properties are\n\n*`fs.s3a.scale.test.huge.filesize`: size in MB for \"Huge file tests\".*\n\nThe Huge File tests validate S3A's ability to handle large files —the property\n`fs.s3a.scale.test.huge.filesize` declares the file size to use.\n\n```xml\n\n fs.s3a.scale.test.huge.filesize\n 200M\n\n```\n\nAmazon S3 handles files larger than 5GB differently than smaller ones.\nSetting the huge filesize to a number greater than that) validates support\nfor huge files.\n\n```xml\n\n fs.s3a.scale.test.huge.filesize\n 6G\n\n```\n\nTests at this scale are slow: they are best executed from hosts running in\nthe cloud infrastructure where the S3 endpoint is based.\nOtherwise, set a large timeout in `fs.s3a.scale.test.timeout`\n\n```xml\n\n fs.s3a.scale.test.timeout\n 432000\n\n```\n\nThe tests are executed in an order to only clean up created files after\nthe end of all the tests. If the tests are interrupted, the test data will remain.\n\n\n## Testing against non AWS S3 endpoints.\n\nThe S3A filesystem is designed to work with storage endpoints which implement\nthe S3 protocols to the extent that the amazon S3 SDK is capable of talking\nto it. We encourage testing against other filesystems and submissions of patches\nwhich address issues. In particular, we encourage testing of Hadoop release\ncandidates, as these third-party endpoints get even less testing than the\nS3 endpoint itself.\n\n\n### Disabling the encryption tests\n\nIf the endpoint doesn't support server-side-encryption, these will fail. They\ncan be turned off.\n\n```xml\n\n test.fs.s3a.encryption.enabled\n false\n\n```\n\nEncryption is only used for those specific test suites with `Encryption` in\ntheir classname.\n\n### Configuring the CSV file read tests**\n\nTo test on alternate infrastructures supporting\nthe same APIs, the option `fs.s3a.scale.test.csvfile` must either be\nset to \" \", or an object of at least 10MB is uploaded to the object store, and\nthe `fs.s3a.scale.test.csvfile` option set to its path.\n\n```xml\n\n fs.s3a.scale.test.csvfile\n \n\n```\n\n(yes, the space is necessary. The Hadoop `Configuration` class treats an empty\nvalue as \"do not override the default\").\n\n\n### Testing Session Credentials\n\nThe test `TestS3ATemporaryCredentials` requests a set of temporary\ncredentials from the STS service, then uses them to authenticate with S3.\n\nIf an S3 implementation does not support STS, then the functional test\ncases must be disabled:\n\n```xml\n\n test.fs.s3a.sts.enabled\n false\n\n```\nThese tests request a temporary set of credentials from the STS service endpoint.\nAn alternate endpoint may be defined in `test.fs.s3a.sts.endpoint`.\n\n```xml\n\n test.fs.s3a.sts.endpoint\n https://sts.example.org/\n\n```\nThe default is \"\"; meaning \"use the amazon default value\".\n\n\n## Debugging Test failures\n\nLogging at debug level is the standard way to provide more diagnostics output;\nafter setting this rerun the tests\n\n```properties\nlog4j.logger.org.apache.hadoop.fs.s3a=DEBUG\n```\n\nThere are also some logging options for debug logging of the AWS client\n```properties\nlog4j.logger.com.amazonaws=DEBUG\nlog4j.logger.com.amazonaws.http.conn.ssl=INFO\nlog4j.logger.com.amazonaws.internal=INFO\n```\n\nThere is also the option of enabling logging on a bucket; this could perhaps\nbe used to diagnose problems from that end. This isn't something actively\nused, but remains an option. If you are forced to debug this way, consider\nsetting the `fs.s3a.user.agent.prefix` to a unique prefix for a specific\ntest run, which will enable the specific log entries to be more easily\nlocated.\n\n## Adding new tests\n\nNew tests are always welcome. Bear in mind that we need to keep costs\nand test time down, which is done by\n\n* Not duplicating tests.\n* Being efficient in your use of Hadoop API calls.\n* Isolating large/slow tests into the \"scale\" test group.\n* Designing all tests to execute in parallel (where possible).\n* Adding new probes and predicates into existing tests, albeit carefully.\n\n*No duplication*: if an operation is tested elsewhere, don't repeat it. This\napplies as much for metadata operations as it does for bulk IO. If a new\ntest case is added which completely obsoletes an existing test, it is OK\nto cut the previous one —after showing that coverage is not worsened.\n\n*Efficient*: prefer the `getFileStatus()` and examining the results, rather than\ncall to `exists()`, `isFile()`, etc.\n\n*Isolating Scale tests*. Any S3A test doing large amounts of IO MUST extend the\nclass `S3AScaleTestBase`, so only running if `scale` is defined on a build,\nsupporting test timeouts configurable by the user. Scale tests should also\nsupport configurability as to the actual size of objects/number of operations,\nso that behavior at different scale can be verified.\n\n*Designed for parallel execution*. A key need here is for each test suite to work\non isolated parts of the filesystem. Subclasses of `AbstractS3ATestBase`\nSHOULD use the `path()` method, with a base path of the test suite name, to\nbuild isolated paths. Tests MUST NOT assume that they have exclusive access\nto a bucket.\n\n*Extending existing tests where appropriate*. This recommendation goes\nagainst normal testing best practise of \"test one thing per method\".\nBecause it is so slow to create directory trees or upload large files, we do\nnot have that luxury. All the tests against real S3 endpoints are integration\ntests where sharing test setup and teardown saves time and money.\n\nA standard way to do this is to extend existing tests with some extra predicates,\nrather than write new tests. When doing this, make sure that the new predicates\nfail with meaningful diagnostics, so any new problems can be easily debugged\nfrom test logs.\n\n\n## Requirements of new Tests\n\n\nThis is what we expect from new tests; they're an extension of the normal\nHadoop requirements, based on the need to work with remote servers whose\nuse requires the presence of secret credentials, where tests may be slow,\nand where finding out why something failed from nothing but the test output\nis critical.\n\n### Subclasses Existing Shared Base Classes\n\nExtend `AbstractS3ATestBase` or `AbstractSTestS3AHugeFiles` unless justifiable.\nThese set things up for testing against the object stores, provide good threadnames,\nhelp generate isolated paths, and for `AbstractSTestS3AHugeFiles` subclasses,\nonly run if `-Dscale` is set.\n\nKey features of `AbstractS3ATestBase`\n\n* `getFileSystem()` returns the S3A Filesystem bonded to the contract test Filesystem\ndefined in `fs.s3a.contract.test`\n* will automatically skip all tests if that URL is unset.\n* Extends `AbstractFSContractTestBase` and `Assert` for all their methods.\n\nHaving shared base classes may help reduce future maintenance too. Please\nuse them/\n\n### Secure\n\nDon't ever log credentials. The credential tests go out of their way to\nnot provide meaningful logs or assertion messages precisely to avoid this.\n\n### Efficient of Time and Money\n\nThis means efficient in test setup/teardown, and, ideally, making use of\nexisting public datasets to save setup time and tester cost.\n\nStrategies of particular note are:\n\n1. `ITestS3ADirectoryPerformance`: a single test case sets up the directory\ntree then performs different list operations, measuring the time taken.\n1. `AbstractSTestS3AHugeFiles`: marks the test suite as\n`@FixMethodOrder(MethodSorters.NAME_ASCENDING)` then orders the test cases such\nthat each test case expects the previous test to have completed (here: uploaded a file,\nrenamed a file, ...). This provides for independent tests in the reports, yet still\npermits an ordered sequence of operations. Do note the use of `Assume.assume()`\nto detect when the preconditions for a single test case are not met, hence,\nthe tests become skipped, rather than fail with a trace which is really a false alarm.\n\nThe ordered test case mechanism of `AbstractSTestS3AHugeFiles` is probably\nthe most elegant way of chaining test setup/teardown.\n\nRegarding reusing existing data, we tend to use the landsat archive of\nAWS US-East for our testing of input stream operations. This doesn't work\nagainst other regions, or with third party S3 implementations. Thus the\nURL can be overridden for testing elsewhere.\n\n\n### Works With Other S3 Endpoints\n\nDon't assume AWS S3 US-East only, do allow for working with external S3 implementations.\nThose may be behind the latest S3 API features, not support encryption, session\nAPIs, etc.\n\nThey won't have the same CSV test files as some of the input tests rely on.\nLook at `ITestS3AInputStreamPerformance` to see how tests can be written\nto support the declaration of a specific large test file on alternate filesystems.\n\n\n### Works Over Long-haul Links\n\nAs well as making file size and operation counts scalable, this includes\nmaking test timeouts adequate. The Scale tests make this configurable; it's\nhard coded to ten minutes in `AbstractS3ATestBase()`; subclasses can\nchange this by overriding `getTestTimeoutMillis()`.\n\nEqually importantly: support proxies, as some testers need them.\n\n\n### Provides Diagnostics and timing information\n\n1. Give threads useful names.\n1. Create logs, log things. Know that the `S3AFileSystem` and its input\nand output streams *all* provide useful statistics in their {{toString()}}\ncalls; logging them is useful on its own.\n1. you can use `AbstractS3ATestBase.describe(format-stringm, args)` here.; it\nadds some newlines so as to be easier to spot.\n1. Use `ContractTestUtils.NanoTimer` to measure the duration of operations,\nand log the output.\n\n### Fails Meaningfully\n\nThe `ContractTestUtils` class contains a whole set of assertions for making\nstatements about the expected state of a filesystem, e.g.\n`assertPathExists(FS, path)`, `assertPathDoesNotExists(FS, path)`, and others.\nThese do their best to provide meaningful diagnostics on failures (e.g. directory\nlistings, file status, ...), so help make failures easier to understand.\n\nAt the very least, do not use `assertTrue()` or `assertFalse()` without\nincluding error messages.\n\n### Sets up its filesystem and checks for those settings\n\nTests can overrun `createConfiguration()` to add new options to the configuration\nfile for the S3A Filesystem instance used in their tests.\n\nHowever, filesystem caching may mean that a test suite may get a cached\ninstance created with an different configuration. For tests which don't need\nspecific configurations caching is good: it reduces test setup time.\n\nFor those tests which do need unique options (encryption, magic files),\nthings can break, and they will do so in hard-to-replicate ways.\n\nUse `S3ATestUtils.disableFilesystemCaching(conf)` to disable caching when\nmodifying the config. As an example from `AbstractTestS3AEncryption`:\n\n```java\n@Override\nprotected Configuration createConfiguration() {\n Configuration conf = super.createConfiguration();\n S3ATestUtils.disableFilesystemCaching(conf);\n conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM,\n getSSEAlgorithm().getMethod());\n return conf;\n}\n```\n\nThen verify in the setup method or test cases that their filesystem actually has\nthe desired feature (`fs.getConf().getProperty(...)`). This not only\ncatches filesystem reuse problems, it catches the situation where the\nfilesystem configuration in `auth-keys.xml` has explicit per-bucket settings\nwhich override the test suite's general option settings.\n\n### Cleans Up Afterwards\n\nKeeps costs down.\n\n1. Do not only cleanup if a test case completes successfully; test suite\nteardown must do it.\n1. That teardown code must check for the filesystem and other fields being\nnull before the cleanup. Why? If test setup fails, the teardown methods still\nget called.\n\n### Works Reliably\n\nWe really appreciate this — you will too.\n\n### Runs in parallel unless this is unworkable.\n\nTests must be designed to run in parallel with other tests, all working\nwith the same shared S3 bucket. This means\n\n* Uses relative and JVM-fork-unique paths provided by the method\n `AbstractFSContractTestBase.path(String filepath)`.\n* Doesn't manipulate the root directory or make assertions about its contents\n(for example: delete its contents and assert that it is now empty).\n* Doesn't have a specific requirement of all active clients of the bucket\n(example: SSE-C tests which require all files, even directory markers,\nto be encrypted with the same key).\n* Doesn't use so much bandwidth that all other tests will be starved of IO and\nstart timing out (e.g. the scale tests).\n\nTests such as these can only be run as sequential tests. When adding one,\nexclude it in the POM file. from the parallel failsafe run and add to the\nsequential one afterwards. The IO heavy ones must also be subclasses of\n`S3AScaleTestBase` and so only run if the system/maven property\n`fs.s3a.scale.test.enabled` is true.\n\n## Individual test cases can be run in an IDE\n\nThis is invaluable for debugging test failures.\n\n### Keeping AWS Costs down\n\nMost of the base S3 tests are designed to use public AWS data\n(the landsat-pds bucket) for read IO, so you don't have to pay for bytes\ndownloaded or long term storage costs. The scale tests do work with more data\nso will cost more as well as generally take more time to execute.\n\nYou are however billed for\n\n1. Data left in S3 after test runs.\n2. DynamoDB capacity reserved by S3Guard tables.\n3. HTTP operations on files (HEAD, LIST, GET).\n4. In-progress multipart uploads from bulk IO or S3A committer tests.\n5. Encryption/decryption using AWS KMS keys.\n\nThe GET/decrypt costs are incurred on each partial read of a file,\nso random IO can cost more than sequential IO; the speedup of queries with\ncolumnar data usually justifies this.\n\nThe DynamoDB costs come from the number of entries stores and the allocated capacity.\n\nHow to keep costs down\n\n* Don't run the scale tests with large datasets; keep `fs.s3a.scale.test.huge.filesize` unset, or a few MB (minimum: 5).\n* Remove all files in the filesystem. The root tests usually do this, but\nit can be manually done:\n\n hadoop fs -rm -r -f -skipTrash s3a://test-bucket/\\*\n* Abort all outstanding uploads:\n\n hadoop s3guard uploads -abort -force s3a://test-bucket/\n* If you don't need it, destroy the S3Guard DDB table.\n\n hadoop s3guard destroy s3a://hwdev-steve-ireland-new/\n\nThe S3Guard tests will automatically create the Dynamo DB table in runs with\n`-Ds3guard -Ddynamodb` set; default capacity of these buckets\ntests is very small; it keeps costs down at the expense of IO performance\nand, for test runs in or near the S3/DDB stores, throttling events.\n\nIf you want to manage capacity, use `s3guard set-capacity` to increase it\n(performance) or decrease it (costs).\nFor remote `hadoop-aws` test runs, the read/write capacities of \"10\" each should suffice;\nincrease it if parallel test run logs warn of throttling.\n\nTip: for agility, use DynamoDB autoscaling, setting the minimum to something very low (e.g 5 units), the maximum to the largest amount you are willing to pay.\nThis will automatically reduce capacity when you are not running tests against\nthe bucket, slowly increase it over multiple test runs, if the load justifies it.\n\n## Tips\n\n### How to keep your credentials really safe\n\nAlthough the `auth-keys.xml` file is marked as ignored in git and subversion,\nit is still in your source tree, and there's always that risk that it may\ncreep out.\n\nYou can avoid this by keeping your keys outside the source tree and\nusing an absolute XInclude reference to it.\n\n```xml\n\n\n \n\n\n```\n\n# Failure Injection\n\n**Warning do not enable any type of failure injection in production. The\nfollowing settings are for testing only.**\n\nOne of the challenges with S3A integration tests is the fact that S3 is an\neventually-consistent storage system. In practice, we rarely see delays in\nvisibility of recently created objects both in listings (`listStatus()`) and\nwhen getting a single file's metadata (`getFileStatus()`). Since this behavior\nis rare and non-deterministic, thorough integration testing is challenging.\n\nTo address this, S3A supports a shim layer on top of the `AmazonS3Client`\nclass which artificially delays certain paths from appearing in listings.\nThis is implemented in the class `InconsistentAmazonS3Client`.\n\n## Simulating List Inconsistencies\n\n### Enabling the InconsistentAmazonS3CClient\n\nThere are two ways of enabling the `InconsistentAmazonS3Client`: at\nconfig-time, or programmatically. For an example of programmatic test usage,\nsee `ITestS3GuardListConsistency`.\n\nTo enable the fault-injecting client via configuration, switch the\nS3A client to use the \"Inconsistent S3 Client Factory\" when connecting to\nS3:\n\n```xml\n\n fs.s3a.s3.client.factory.impl\n org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory\n\n```\n\nThe inconsistent client works by:\n\n1. Choosing which objects will be \"inconsistent\" at the time the object is\ncreated or deleted.\n2. When `listObjects()` is called, any keys that we have marked as\ninconsistent above will not be returned in the results (until the\nconfigured delay has elapsed). Similarly, deleted items may be *added* to\nmissing results to delay the visibility of the delete.\n\nThere are two ways of choosing which keys (filenames) will be affected: By\nsubstring, and by random probability.\n\n```xml\n\n fs.s3a.failinject.inconsistency.key.substring\n DELAY_LISTING_ME\n\n\n\n fs.s3a.failinject.inconsistency.probability\n 1.0\n\n```\n\nBy default, any object which has the substring \"DELAY_LISTING_ME\" in its key\nwill subject to delayed visibility. For example, the path\n`s3a://my-bucket/test/DELAY_LISTING_ME/file.txt` would match this condition.\nTo match all keys use the value \"\\*\" (a single asterisk). This is a special\nvalue: *We don't support arbitrary wildcards.*\n\nThe default probability of delaying an object is 1.0. This means that *all*\nkeys that match the substring will get delayed visibility. Note that we take\nthe logical *and* of the two conditions (substring matches *and* probability\nrandom chance occurs). Here are some example configurations:\n\n```\n| substring | probability | behavior |\n|-----------|-------------|--------------------------------------------|\n| | 0.001 | An empty tag in .xml config will |\n| | | be interpreted as unset and revert to the |\n| | | default value, \"DELAY_LISTING_ME\" |\n| | | |\n| * | 0.001 | 1/1000 chance of *any* key being delayed. |\n| | | |\n| delay | 0.01 | 1/100 chance of any key containing \"delay\" |\n| | | |\n| delay | 1.0 | All keys containing substring \"delay\" .. |\n```\n\nYou can also configure how long you want the delay in visibility to last.\nThe default is 5000 milliseconds (five seconds).\n\n```xml\n\n fs.s3a.failinject.inconsistency.msec\n 5000\n\n```\n\nFuture versions of this client will introduce new failure modes,\nwith simulation of S3 throttling exceptions the next feature under\ndevelopment.\n\n### Limitations of Inconsistency Injection\n\nAlthough `InconsistentAmazonS3Client` can delay the visibility of an object\nor parent directory, it does not prevent the key of that object from\nappearing in all prefix searches. For example, if we create the following\nobject with the default configuration above, in an otherwise empty bucket:\n\n```\ns3a://bucket/a/b/c/DELAY_LISTING_ME\n```\n\nThen the following paths will still be visible as directories (ignoring\npossible real-world inconsistencies):\n\n```\ns3a://bucket/a\ns3a://bucket/a/b\n```\n\nWhereas `getFileStatus()` on the following *will* be subject to delayed\nvisibility (`FileNotFoundException` until delay has elapsed):\n\n```\ns3a://bucket/a/b/c\ns3a://bucket/a/b/c/DELAY_LISTING_ME\n```\n\nIn real-life S3 inconsistency, however, we expect that all the above paths\n(including `a` and `b`) will be subject to delayed visibility.\n\n### Using the `InconsistentAmazonS3CClient` in downstream integration tests\n\nThe inconsistent client is shipped in the `hadoop-aws` JAR, so it can\nbe used in applications which work with S3 to see how they handle\ninconsistent directory listings.\n\n## Testing S3Guard\n\n[S3Guard](./s3guard.html) is an extension to S3A which adds consistent metadata\nlistings to the S3A client. As it is part of S3A, it also needs to be tested.\n\nThe basic strategy for testing S3Guard correctness consists of:\n\n1. MetadataStore Contract tests.\n\n The MetadataStore contract tests are inspired by the Hadoop FileSystem and\n `FileContext` contract tests. Each implementation of the `MetadataStore` interface\n subclasses the `MetadataStoreTestBase` class and customizes it to initialize\n their MetadataStore. This test ensures that the different implementations\n all satisfy the semantics of the MetadataStore API.\n\n2. Running existing S3A unit and integration tests with S3Guard enabled.\n\n You can run the S3A integration tests on top of S3Guard by configuring your\n `MetadataStore` in your\n `hadoop-tools/hadoop-aws/src/test/resources/core-site.xml` or\n `hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml` files.\n Next run the S3A integration tests as outlined in the *Running the Tests* section\n of the [S3A documentation](./index.html)\n\n3. Running fault-injection tests that test S3Guard's consistency features.\n\n The `ITestS3GuardListConsistency` uses failure injection to ensure\n that list consistency logic is correct even when the underlying storage is\n eventually consistent.\n\n The integration test adds a shim above the Amazon S3 Client layer that injects\n delays in object visibility.\n\n All of these tests will be run if you follow the steps listed in step 2 above.\n\n No charges are incurred for using this store, and its consistency\n guarantees are that of the underlying object store instance. \n\n## Testing S3A with S3Guard Enabled\n\nAll the S3A tests which work with a private repository can be configured to\nrun with S3Guard by using the `s3guard` profile. When set, this will run\nall the tests with local memory for the metadata set to \"non-authoritative\" mode.\n\n```bash\nmvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard\n```\n\nWhen the `s3guard` profile is enabled, following profiles can be specified:\n\n* `dynamo`: use an AWS-hosted DynamoDB table; creating the table if it does\n not exist. You will have to pay the bills for DynamoDB web service.\n* `dynamodblocal`: use an in-memory DynamoDBLocal server instead of real AWS\n DynamoDB web service; launch the server and creating the table.\n You won't be charged bills for using DynamoDB in test. As it runs in-JVM,\n the table isn't shared across other tests running in parallel.\n* `non-auth`: treat the S3Guard metadata as authoritative.\n\n```bash\nmvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Ddynamo -Dauth\n```\n\nWhen experimenting with options, it is usually best to run a single test suite\nat a time until the operations appear to be working.\n\n```bash\nmvn -T 1C verify -Dtest=skip -Dit.test=ITestS3AMiscOperations -Ds3guard -Ddynamo\n```\n\n### Notes\n\n1. If the `s3guard` profile is not set, then the S3Guard properties are those\nof the test configuration set in `contract-test-options.xml` or `auth-keys.xml`\n\nIf the `s3guard` profile *is* set,\n1. The S3Guard options from maven (the dynamo and authoritative flags)\n overwrite any previously set in the configuration files.\n1. DynamoDB will be configured to create any missing tables.\n\n\n### Scale Testing MetadataStore Directly\n\nThere are some scale tests that exercise Metadata Store implementations\ndirectly. These ensure that S3Guard is are robust to things like DynamoDB\nthrottling, and compare performance for different implementations. These\nare included in the scale tests executed when `-Dscale` is passed to\nthe maven command line.\n\nThe two S3Guard scale tests are `ITestDynamoDBMetadataStoreScale` and\n`ITestLocalMetadataStoreScale`.\n\nTo run these tests, your DynamoDB table needs to be of limited capacity;\nthe values in `ITestDynamoDBMetadataStoreScale` currently require a read capacity\nof 10 or less. a write capacity of 15 or more.\n\nThe following settings allow us to run `ITestDynamoDBMetadataStoreScale` with\nartificially low read and write capacity provisioned, so we can judge the\neffects of being throttled by the DynamoDB service:\n\n```xml\n\n scale.test.operation.count\n 10\n\n\n scale.test.directory.count\n 3\n\n\n fs.s3a.scale.test.enabled\n true\n\n\n fs.s3a.s3guard.ddb.table\n my-scale-test\n\n\n fs.s3a.s3guard.ddb.table.create\n true\n\n\n fs.s3a.s3guard.ddb.table.capacity.read\n 5\n\n\n fs.s3a.s3guard.ddb.table.capacity.write\n 5\n\n```\n\nThese tests verify that the invoked operations can trigger retries in the\nS3Guard code, rather than just in the AWS SDK level, so showing that if\nSDK operations fail, they get retried. They also verify that the filesystem\nstatistics are updated to record that throttling took place.\n\n*Do not panic if these tests fail to detect throttling!*\n\nThese tests are unreliable as they need certain conditions to be met\nto repeatedly fail:\n\n1. You must have a low-enough latency connection to the DynamoDB store that,\nfor the capacity allocated, you can overload it.\n1. The AWS Console can give you a view of what is happening here.\n1. Running a single test on its own is less likely to trigger an overload\nthan trying to run the whole test suite.\n1. And running the test suite more than once, back-to-back, can also help\noverload the cluster.\n1. Stepping through with a debugger will reduce load, so may not trigger\nfailures.\n\nIf the tests fail, it *probably* just means you aren't putting enough load\non the table.\n\nThese tests do not verify that the entire set of DynamoDB calls made\nduring the use of a S3Guarded S3A filesystem are wrapped by retry logic.\n\n*The best way to verify resilience is to run the entire `hadoop-aws` test suite,\nor even a real application, with throttling enabled.\n\n### Testing only: Local Metadata Store\n\nThere is an in-memory Metadata Store for testing.\n\n```xml\n\n fs.s3a.metadatastore.impl\n org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore\n\n```\n\nThis is not for use in production.\n\n## Testing Assumed Roles\n\nTests for the AWS Assumed Role credential provider require an assumed\nrole to request.\n\nIf this role is not set, the tests which require it will be skipped.\n\nTo run the tests in `ITestAssumeRole`, you need:\n\n1. A role in your AWS account will full read and write access rights to\nthe S3 bucket used in the tests, and ideally DynamoDB, for S3Guard.\nIf your bucket is set up by default to use S3Guard, the role must have access\nto that service.\n\n1. Your IAM User to have the permissions to adopt that role.\n\n1. The role ARN must be set in `fs.s3a.assumed.role.arn`.\n\n```xml\n\n fs.s3a.assumed.role.arn\n arn:aws:iam::9878543210123:role/role-s3-restricted\n\n```\n\nThe tests assume the role with different subsets of permissions and verify\nthat the S3A client (mostly) works when the caller has only write access\nto part of the directory tree.\n\nYou can also run the entire test suite in an assumed role, a more\nthorough test, by switching to the credentials provider.\n\n```xml\n\n fs.s3a.aws.credentials.provider\n org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n\n```\n\nThe usual credentials needed to log in to the bucket will be used, but now\nthe credentials used to interact with S3 and DynamoDB will be temporary\nrole credentials, rather than the full credentials.\n\n## Qualifying an AWS SDK Update\n\nUpdating the AWS SDK is something which does need to be done regularly,\nbut is rarely without complications, major or minor.\n\nAssume that the version of the SDK will remain constant for an X.Y release,\nexcluding security fixes, so it's good to have an update before each release\n— as long as that update works doesn't trigger any regressions.\n\n\n1. Don't make this a last minute action.\n1. The upgrade patch should focus purely on the SDK update, so it can be cherry\npicked and reverted easily.\n1. Do not mix in an SDK update with any other piece of work, for the same reason.\n1. Plan for an afternoon's work, including before/after testing, log analysis\nand any manual tests.\n1. Make sure all the integration tests are running (including s3guard, ARN, encryption, scale)\n *before you start the upgrade*.\n1. Create a JIRA for updating the SDK. Don't include the version (yet),\nas it may take a couple of SDK updates before it is ready.\n1. Identify the latest AWS SDK [available for download](https://aws.amazon.com/sdk-for-java/).\n1. Create a private git branch of trunk for JIRA, and in\n `hadoop-project/pom.xml` update the `aws-java-sdk.version` to the new SDK version.\n1. Do a clean build and rerun all the `hadoop-aws` tests, with and without the `-Ds3guard -Ddynamodb` options.\n This includes the `-Pscale` set, with a role defined for the assumed role tests.\n in `fs.s3a.assumed.role.arn` for testing assumed roles,\n and `fs.s3a.server-side-encryption.key` for encryption, for full coverage.\n If you can, scale up the scale tests.\n1. Create the site with `mvn site -DskipTests`; look in `target/site` for the report.\n1. Review *every single `-output.txt` file in `hadoop-tools/hadoop-aws/target/failsafe-reports`,\n paying particular attention to\n `org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance-output.txt`,\n as that is where changes in stream close/abort logic will surface.\n1. Run `mvn install` to install the artifacts, then in\n `hadoop-cloud-storage-project/hadoop-cloud-storage` run\n `mvn dependency:tree -Dverbose > target/dependencies.txt`.\n Examine the `target/dependencies.txt` file to verify that no new\n artifacts have unintentionally been declared as dependencies\n of the shaded `aws-java-sdk-bundle` artifact.\n\n### Basic command line regression testing\n\nWe need a run through of the CLI to see if there have been changes there\nwhich cause problems, especially whether new log messages have surfaced,\nor whether some packaging change breaks that CLI\n\nFrom the root of the project, create a command line release `mvn package -Pdist -DskipTests -Dmaven.javadoc.skip=true -DskipShade`;\n\n1. Change into the `hadoop/dist/target/hadoop-x.y.z-SNAPSHOT` dir.\n1. Copy a `core-site.xml` file into `etc/hadoop`.\n1. Set the `HADOOP_OPTIONAL_TOOLS` env var on the command line or `~/.hadoop-env`.\n\n```bash\nexport HADOOP_OPTIONAL_TOOLS=\"hadoop-aws\"\n```\n\nRun some basic s3guard commands as well as file operations.\n\n```bash\nexport BUCKET=s3a://example-bucket-name\n\nbin/hadoop s3guard bucket-info $BUCKET\nbin/hadoop s3guard set-capacity $BUCKET\nbin/hadoop s3guard set-capacity -read 15 -write 15 $BUCKET\nbin/hadoop s3guard uploads $BUCKET\nbin/hadoop s3guard diff $BUCKET/\nbin/hadoop s3guard prune -minutes 10 $BUCKET/\nbin/hadoop s3guard import $BUCKET/\nbin/hadoop fs -ls $BUCKET/\nbin/hadoop fs -ls $BUCKET/file\nbin/hadoop fs -rm -R -f $BUCKET/dir-no-trailing\nbin/hadoop fs -rm -R -f $BUCKET/dir-trailing/\nbin/hadoop fs -rm $BUCKET/\nbin/hadoop fs -touchz $BUCKET/file\n# expect I/O error as root dir is not empty\nbin/hadoop fs -rm -r $BUCKET/\nbin/hadoop fs -rm -r $BUCKET/\\*\n# now success\nbin/hadoop fs -rm -r $BUCKET/\n\nbin/hadoop fs -mkdir $BUCKET/dir-no-trailing\n# fails with S3Guard\nbin/hadoop fs -mkdir $BUCKET/dir-trailing/\nbin/hadoop fs -touchz $BUCKET/file\nbin/hadoop fs -ls $BUCKET/\nbin/hadoop fs -mv $BUCKET/file $BUCKET/file2\n# expect \"No such file or directory\"\nbin/hadoop fs -stat $BUCKET/file\nbin/hadoop fs -stat $BUCKET/file2\nbin/hadoop fs -mv $BUCKET/file2 $BUCKET/dir-no-trailing\nbin/hadoop fs -stat $BUCKET/dir-no-trailing/file2\n# treated the same as the file stat\nbin/hadoop fs -stat $BUCKET/dir-no-trailing/file2/\nbin/hadoop fs -ls $BUCKET/dir-no-trailing/file2/\nbin/hadoop fs -ls $BUCKET/dir-no-trailing\n# expect a \"0\" here:\nbin/hadoop fs -test -d $BUCKET/dir-no-trailing && echo $?\n# expect a \"1\" here:\nbin/hadoop fs -test -d $BUCKET/dir-no-trailing/file2 && echo $?\n# will return NONE unless bucket has checksums enabled\nbin/hadoop fs -checksum $BUCKET/dir-no-trailing/file2\n# expect \"etag\" + a long string\nbin/hadoop fs -D fs.s3a.etag.checksum.enabled=true -checksum $BUCKET/dir-no-trailing/file2\n```\n\n### Other tests\n\n* Whatever applications you have which use S3A: build and run them before the upgrade,\nThen see if complete successfully in roughly the same time once the upgrade is applied.\n* Test any third-party endpoints you have access to.\n* Try different regions (especially a v4 only region), and encryption settings.\n* Any performance tests you have can identify slowdowns, which can be a sign\n of changed behavior in the SDK (especially on stream reads and writes).\n* If you can, try to test in an environment where a proxy is needed to talk\nto AWS services.\n* Try and get other people, especially anyone with their own endpoints,\n apps or different deployment environments, to run their own tests.\n\n### Dealing with Deprecated APIs and New Features\n\nA Jenkins run should tell you if there are new deprecations.\nIf so, you should think about how to deal with them.\n\nMoving to methods and APIs which weren't in the previous SDK release makes it\nharder to roll back if there is a problem; but there may be good reasons\nfor the deprecation.\n\nAt the same time, there may be good reasons for staying with the old code.\n\n* AWS have embraced the builder pattern for new operations; note that objects\nconstructed this way often have their (existing) setter methods disabled; this\nmay break existing code.\n* New versions of S3 calls (list v2, bucket existence checks, bulk operations)\nmay be better than the previous HTTP operations & APIs, but they may not work with\nthird-party endpoints, so can only be adopted if made optional, which then\nadds a new configuration option (with docs, testing, ...). A change like that\nmust be done in its own patch, with its new tests which compare the old\nvs new operations.\n\n### Committing the patch\n\nWhen the patch is committed: update the JIRA to the version number actually\nused; use that title in the commit message.\n\nBe prepared to roll-back, re-iterate or code your way out of a regression.\n\nThere may be some problem which surfaces with wider use, which can get\nfixed in a new AWS release, "},"items":[{"name":"index.md","path":"hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md","rawUrl":"https://raw.githubusercontent.com/apache/hadoop/HEAD/hadoop-tools/hadoop-aliyun/src/site/markdown/tools/hadoop-aliyun/index.md","title":"Hadoop-aliyun Skill","category":"anthropic-skill","format":"markdown","content":"\n\n# Hadoop-Aliyun module: Integration with Aliyun Web Services\n\n\n\n## Overview\n\nThe `hadoop-aliyun` module provides support for Aliyun integration with\n[Aliyun Object Storage Service (Aliyun OSS)](https://www.aliyun.com/product/oss).\nThe generated JAR file, `hadoop-aliyun.jar` also declares a transitive\ndependency on all external artifacts which are needed for this support — enabling\ndownstream applications to easily use this support.\n\nTo make it part of Apache Hadoop's default classpath, simply make sure\nthat HADOOP_OPTIONAL_TOOLS in hadoop-env.sh has 'hadoop-aliyun' in the list.\n\n### Features\n\n* Read and write data stored in Aliyun OSS.\n* Present a hierarchical file system view by implementing the standard Hadoop\n[`FileSystem`](../api/org/apache/hadoop/fs/FileSystem.html) interface.\n* Can act as a source of data in a MapReduce job, or a sink.\n\n### Warning #1: Object Stores are not filesystems.\n\nAliyun OSS is an example of \"an object store\". In order to achieve scalability\nand especially high availability, Aliyun OSS has relaxed some of the constraints\nwhich classic \"POSIX\" filesystems promise.\n\n\n\nSpecifically\n\n1. Atomic operations: `delete()` and `rename()` are implemented by recursive\nfile-by-file operations. They take time at least proportional to the number of files,\nduring which time partial updates may be visible. `delete()` and `rename()`\ncan not guarantee atomicity. If the operations are interrupted, the filesystem\nis left in an intermediate state.\n2. File owner and group are persisted, but the permissions model is not enforced.\nAuthorization occurs at the level of the entire Aliyun account via\n[Aliyun Resource Access Management (Aliyun RAM)](https://www.aliyun.com/product/ram).\n3. Directory last access time is not tracked.\n4. The append operation is not supported.\n\n### Warning #2: Directory last access time is not tracked,\nFeatures of Hadoop relying on this can have unexpected behaviour. E.g. the\nAggregatedLogDeletionService of YARN will not remove the appropriate log files.\n\n### Warning #3: Your Aliyun credentials are valuable\n\nYour Aliyun credentials not only pay for services, they offer read and write\naccess to the data. Anyone with the account can not only read your datasets\n— they can delete them.\n\nDo not inadvertently share these credentials through means such as\n1. Checking in to SCM any configuration files containing the secrets.\n2. Logging them to a console, as they invariably end up being seen.\n3. Defining filesystem URIs with the credentials in the URL, such as\n`oss://accessKeyId:accessKeySecret@directory/file`. They will end up in\nlogs and error messages.\n4. Including the secrets in bug reports.\n\nIf you do any of these: change your credentials immediately!\n\n### Warning #4: The Aliyun OSS client provided by Aliyun E-MapReduce are different from this implementation\n\nSpecifically: on Aliyun E-MapReduce, `oss://` is also supported but with\na different implementation. If you are using Aliyun E-MapReduce,\nfollow these instructions —and be aware that all issues related to Aliyun\nOSS integration in E-MapReduce can only be addressed by Aliyun themselves:\nplease raise your issues with them.\n\n## OSS\n\n### Authentication properties\n\n \n fs.oss.accessKeyId\n Aliyun access key ID\n \n\n \n fs.oss.accessKeySecret\n Aliyun access key secret\n \n\n \n fs.oss.credentials.provider\n \n Class name of a credentials provider that implements\n com.aliyun.oss.common.auth.CredentialsProvider. Omit if using access/secret keys\n or another authentication mechanism. The specified class must provide an\n accessible constructor accepting java.net.URI and\n org.apache.hadoop.conf.Configuration, or an accessible default constructor.\n \n \n\n### Other properties\n\n \n fs.oss.endpoint\n Aliyun OSS endpoint to connect to. An up-to-date list is\n provided in the Aliyun OSS Documentation.\n \n \n\n \n fs.oss.assumed.role.arn\n \n Role ARN for the role to be assumed.\n Required if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.sts.endpoint\n \n STS Token Service endpoint.\n Required if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.session.name\n \n \n Session name for the assumed role, must be valid characters\n according to Aliyun API. It is optional, will be generated by\n oss java sdk if it is empty.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.session.duration\n \n \n Duration of assumed roles before it is expired. Default is 30 minutes.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.sts.expiredFactor\n \n \n Sts token will be refreshed after (expiredFactor * duration) seconds.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.proxy.host\n Hostname of the (optinal) proxy server for Aliyun OSS connection\n \n\n \n fs.oss.proxy.port\n Proxy server port\n \n\n \n fs.oss.proxy.username\n Username for authenticating with proxy server\n \n\n \n fs.oss.proxy.password\n Password for authenticating with proxy server.\n \n\n \n fs.oss.proxy.domain\n Domain for authenticating with proxy server.\n \n\n \n fs.oss.proxy.workstation\n Workstation for authenticating with proxy server.\n \n\n \n fs.oss.attempts.maximum\n 20\n How many times we should retry commands on transient errors.\n \n\n \n fs.oss.connection.establish.timeout\n 50000\n Connection setup timeout in milliseconds.\n \n\n \n fs.oss.connection.timeout\n 200000\n Socket connection timeout in milliseconds.\n \n\n \n fs.oss.paging.maximum\n 1000\n How many keys to request from Aliyun OSS when doing directory listings at a time.\n \n \n\n \n fs.oss.multipart.upload.size\n 10485760\n Size of each of multipart pieces in bytes.\n \n\n \n fs.oss.multipart.upload.threshold\n 20971520\n Minimum size in bytes before we start a multipart uploads or copy.\n \n\n \n fs.oss.multipart.download.size\n 102400/value>\n Size in bytes in each request from ALiyun OSS.\n \n\n \n fs.oss.buffer.dir\n Comma separated list of directories to buffer OSS data before uploading to Aliyun OSS\n \n\n \n fs.oss.acl.default\n \n Set a canned ACL for bucket. Value may be private, public-read, public-read-write.\n \n \n\n \n fs.oss.server-side-encryption-algorithm\n \n Specify a server-side encryption algorithm for oss: file system.\n Unset by default, and the only other currently allowable value is AES256.\n \n \n\n \n fs.oss.connection.maximum\n 32\n Number of simultaneous connections to oss.\n \n\n \n fs.oss.connection.secure.enabled\n true\n Connect to oss over ssl or not, true by default.\n \n\n## Testing the hadoop-aliyun Module\n\nTo test `oss://` filesystem client, two files which pass in authentication\ndetails to the test runner are needed.\n\n1. `auth-keys.xml`\n2. `core-site.xml`\n\nThose two configuration files must be put into\n`hadoop-tools/hadoop-aliyun/src/test/resources`.\n\n### `core-site.xml`\n\nThis file pre-exists and sources the configurations created in `auth-keys.xml`.\n\nFor most cases, no modification is needed, unless a specific, non-default property\nneeds to be set during the testing.\n\n### `auth-keys.xml`\n\nThis file triggers the testing of Aliyun OSS module. Without this file,\n*none of the tests in this module will be executed*\n\nIt contains the access key Id/secret and proxy information that are needed to\nconnect to Aliyun OSS, and an OSS bucket URL should be also provided.\n\n1. `test.fs.oss.name` : the URL of the bucket for Aliyun OSS tests\n\nThe contents of the bucket will be cleaned during the testing process, so\ndo not use the bucket for any purpose other than testing.\n\n### Run Hadoop contract tests\nCreate file `contract-test-options.xml` under `/test/resources`. If a\nspecific file `fs.contract.test.fs.oss` test path is not defined, those\ntests will be skipped. Credentials are also needed to run any of those\ntests, they can be copied from `auth-keys.xml` or through direct\nXInclude inclusion. Here is an example of `contract-test-options.xml`:\n\n \n \n \n\n \n\n \n fs.contract.test.fs.oss\n oss://spark-tests\n \n\n \n fs.oss.impl\n org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem\n \n\n \n fs.oss.endpoint\n oss-cn-hangzhou.aliyuncs.com\n \n\n \n fs.oss.buffer.dir\n /tmp/oss\n \n\n \n fs.oss.multipart.download.size\n 102400\n \n \n","isInternal":false,"tokens":2796,"sizeBytes":12091},{"name":"assumed_roles.md","path":"hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md","rawUrl":"https://raw.githubusercontent.com/apache/hadoop/HEAD/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/assumed_roles.md","title":"Hadoop-aws Skill","category":"anthropic-skill","format":"markdown","content":"\n\n# Working with IAM Assumed Roles\n\n\n\nAWS [\"IAM Assumed Roles\"](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)\nallows applications to change the AWS role with which to authenticate with AWS services.\nThe assumed roles can have different rights from the main user login.\n\nThe S3A connector supports assumed roles for authentication with AWS.\nA full set of login credentials must be provided, which will be used\nto obtain the assumed role and refresh it regularly.\nBy using per-filesystem configuration, it is possible to use different\nassumed roles for different buckets.\n\n*IAM Assumed Roles are unlikely to be supported by third-party systems\nsupporting the S3 APIs.*\n\n## Using IAM Assumed Roles\n\n### Before You Begin\n\nThis document assumes you know about IAM Assumed roles, what they\nare, how to configure their policies, etc.\n\n* You need a role to assume, and know its \"ARN\".\n* You need a pair of long-lived IAM User credentials, not the root account set.\n* Have the AWS CLI installed, and test that it works there.\n* Give the role access to S3, and, if using S3Guard, to DynamoDB.\n* For working with data encrypted with SSE-KMS, the role must\nhave access to the appropriate KMS keys.\n\nTrying to learn how IAM Assumed Roles work by debugging stack traces from\nthe S3A client is \"suboptimal\".\n\n### How the S3A connector support IAM Assumed Roles.\n\nTo use assumed roles, the client must be configured to use the\n*Assumed Role Credential Provider*, `org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider`,\nin the configuration option `fs.s3a.aws.credentials.provider`.\n\nThis AWS Credential provider will read in the `fs.s3a.assumed.role` options needed to connect to the\nSecurity Token Service [Assumed Role API](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html),\nfirst authenticating with the full credentials, then assuming the specific role\nspecified. It will then refresh this login at the configured rate of\n`fs.s3a.assumed.role.session.duration`\n\nTo authenticate with the [AWS STS service](https://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html)\nboth for the initial credential retrieval\nand for background refreshes, a different credential provider must be\ncreated, one which uses long-lived credentials (secret keys, environment variables).\nShort lived credentials (e.g other session tokens, EC2 instance credentials) cannot be used.\n\nA list of providers can be set in `s.s3a.assumed.role.credentials.provider`;\nif unset the standard `BasicAWSCredentialsProvider` credential provider is used,\nwhich uses `fs.s3a.access.key` and `fs.s3a.secret.key`.\n\nNote: although you can list other AWS credential providers in to the\nAssumed Role Credential Provider, it can only cause confusion.\n\n### Configuring Assumed Roles\n\nTo use assumed roles, the S3A client credentials provider must be set to\nthe `AssumedRoleCredentialProvider`, and `fs.s3a.assumed.role.arn` to\nthe previously created ARN.\n\n```xml\n\n fs.s3a.aws.credentials.provider\n org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n\n\n\n fs.s3a.assumed.role.arn\n arn:aws:iam::90066806600238:role/s3-restricted\n\n```\n\nThe STS service itself needs the caller to be authenticated, *which can\nonly be done with a set of long-lived credentials*.\nThis means the normal `fs.s3a.access.key` and `fs.s3a.secret.key`\npair, environment variables, or some other supplier of long-lived secrets.\n\nThe default is the `fs.s3a.access.key` and `fs.s3a.secret.key` pair.\nIf you wish to use a different authentication mechanism, set it in the property\n`fs.s3a.assumed.role.credentials.provider`.\n\n```xml\n\n fs.s3a.assumed.role.credentials.provider\n org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\n\n```\n\nRequirements for long-lived credentials notwithstanding, this option takes the\nsame values as `fs.s3a.aws.credentials.provider`.\n\nThe safest way to manage AWS secrets is via\n[Hadoop Credential Providers](index.html#hadoop_credential_providers).\n\n### Assumed Role Configuration Options\n\nHere are the full set of configuration options.\n\n```xml\n\n fs.s3a.assumed.role.arn\n \n \n AWS ARN for the role to be assumed.\n Required if the fs.s3a.aws.credentials.provider contains\n org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider\n \n\n\n\n fs.s3a.assumed.role.session.name\n \n \n Session name for the assumed role, must be valid characters according to\n the AWS APIs.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n If not set, one is generated from the current Hadoop/Kerberos username.\n \n\n\n\n fs.s3a.assumed.role.policy\n \n \n JSON policy to apply to the role.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.session.duration\n 30m\n \n Duration of assumed roles before a refresh is attempted.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n Range: 15m to 1h\n \n\n\n\n fs.s3a.assumed.role.sts.endpoint\n \n \n AWS Security Token Service Endpoint. If unset, uses the default endpoint.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.sts.endpoint.region\n us-west-1\n \n AWS Security Token Service Endpoint's region;\n Needed if fs.s3a.assumed.role.sts.endpoint points to an endpoint\n other than the default one and the v4 signature is used.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.credentials.provider\n org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\n \n List of credential providers to authenticate with the STS endpoint and\n retrieve short-lived role credentials.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n If unset, uses \"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\".\n \n\n```\n\n## Restricting S3A operations through AWS Policies\n\nThe S3A client needs to be granted specific permissions in order\nto work with a bucket.\nHere is a non-normative list of the permissions which must be granted\nfor FileSystem operations to work.\n\n*Disclaimer* The specific set of actions which the S3A connector needs\nwill change over time.\n\nAs more operations are added to the S3A connector, and as the\nmeans by which existing operations are implemented change, the\nAWS actions which are required by the client will change.\n\nThese lists represent the minimum actions to which the client's principal\nmust have in order to work with a bucket.\n\n\n### Read Access Permissions\n\nPermissions which must be granted when reading from a bucket:\n\n\n```\ns3:Get*\ns3:ListBucket\n```\n\nWhen using S3Guard, the client needs the appropriate\nDynamoDB access permissions\n\nTo use SSE-KMS encryption, the client needs the\nSSE-KMS Permissions to access the\nKMS key(s).\n\n### Write Access Permissions\n\nThese permissions must all be granted for write access:\n\n```\ns3:Get*\ns3:Delete*\ns3:Put*\ns3:ListBucket\ns3:ListBucketMultipartUploads\ns3:AbortMultipartUpload\n```\n\n### SSE-KMS Permissions\n\nWhen to read data encrypted using SSE-KMS, the client must have\n `kms:Decrypt` permission for the specific key a file was encrypted with.\n\n```\nkms:Decrypt\n```\n\nTo write data using SSE-KMS, the client must have all the following permissions.\n\n```\nkms:Decrypt\nkms:GenerateDataKey\n```\n\nThis includes renaming: renamed files are encrypted with the encryption key\nof the current S3A client; it must decrypt the source file first.\n\nIf the caller doesn't have these permissions, the operation will fail with an\n`AccessDeniedException`: the S3 Store does not provide the specifics of\nthe cause of the failure.\n\n### S3Guard Permissions\n\nTo use S3Guard, all clients must have a subset of the\n[AWS DynamoDB Permissions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/api-permissions-reference.html).\n\nTo work with buckets protected with S3Guard, the client must have\nall the following rights on the DynamoDB Table used to protect that bucket.\n\n```\ndynamodb:BatchGetItem\ndynamodb:BatchWriteItem\ndynamodb:DeleteItem\ndynamodb:DescribeTable\ndynamodb:GetItem\ndynamodb:PutItem\ndynamodb:Query\ndynamodb:UpdateItem\n```\n\nThis is true, *even if the client only has read access to the data*.\n\nFor the `hadoop s3guard` table management commands, _extra_ permissions are required:\n\n```\ndynamodb:CreateTable\ndynamodb:DescribeLimits\ndynamodb:DeleteTable\ndynamodb:Scan\ndynamodb:TagResource\ndynamodb:UntagResource\ndynamodb:UpdateTable\n```\n\nWithout these permissions, tables cannot be created, destroyed or have their IO capacity\nchanged through the `s3guard set-capacity` call.\nThe `dynamodb:Scan` permission is needed for `s3guard prune`\n\nThe `dynamodb:CreateTable` permission is needed by a client it tries to\ncreate the DynamoDB table on startup, that is\n`fs.s3a.s3guard.ddb.table.create` is `true` and the table does not already exist.\n\n### Mixed Permissions in a single S3 Bucket\n\nMixing permissions down the \"directory tree\" is limited\nonly to the extent of supporting writeable directories under\nread-only parent paths.\n\n*Disclaimer:* When a client lacks write access up the entire\ndirectory tree, there are no guarantees of consistent filesystem\nviews or operations.\n\nParticular troublespots are \"directory markers\" and\nfailures of non-atomic operations, particularly `rename()` and `delete()`.\n\nA directory marker such as `/users/` will not be deleted if the user `alice`\ncreates a directory `/users/alice` *and* she only has access to `/users/alice`.\n\nWhen a path or directory is deleted, the parent directory may not exist afterwards.\nIn the example above, if `alice` deletes `/users/alice` and there are no\nother entries under `/users/alice`, then the directory marker `/users/` cannot\nbe created. The directory `/users` will not exist in listings,\n`getFileStatus(\"/users\")` or similar.\n\nRename will fail if it cannot delete the items it has just copied, that is\n`rename(read-only-source, writeable-dest)` will fail —but only after\nperforming the COPY of the data.\nEven though the operation failed, for a single file copy, the destination\nfile will exist.\nFor a directory copy, only a partial copy of the source data may take place\nbefore the permission failure is raised.\n\n\n*S3Guard*: if [S3Guard](s3guard.html) is used to manage the directory listings,\nthen after partial failures of rename/copy the DynamoDB tables can get out of sync.\n\n### Example: Read access to the base, R/W to the path underneath\n\nThis example has the base bucket read only, and a directory underneath,\n`/users/alice/` granted full R/W access.\n\n```json\n{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"4\",\n \"Effect\" : \"Allow\",\n \"Action\" : [\n \"s3:ListBucket\",\n \"s3:ListBucketMultipartUploads\",\n \"s3:Get*\"\n ],\n \"Resource\" : \"arn:aws:s3:::example-bucket/*\"\n }, {\n \"Sid\" : \"5\",\n \"Effect\" : \"Allow\",\n \"Action\" : [\n \"s3:Get*\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:AbortMultipartUpload\",\n \"s3:ListMultipartUploadParts\" ],\n \"Resource\" : [\n \"arn:aws:s3:::example-bucket/users/alice/*\",\n \"arn:aws:s3:::example-bucket/users/alice\",\n \"arn:aws:s3:::example-bucket/users/alice/\"\n ]\n } ]\n}\n```\n\nNote how three resources are provided to represent the path `/users/alice`\n\n| Path | Matches |\n|-------|----------|\n| `/users/alice` | Any file `alice` created under `/users` |\n| `/users/alice/` | The directory marker `alice/` created under `/users` |\n| `/users/alice/*` | All files and directories under the path `/users/alice` |\n\nNote that the resource `arn:aws:s3:::example-bucket/users/alice*` cannot\nbe used to refer to all of these paths, because it would also cover\nadjacent paths like `/users/alice2` and `/users/alicebob`.\n\n\n## Troubleshooting Assumed Roles\n\n1. Make sure the role works and the user trying to enter it can do so from AWS\nthe command line before trying to use the S3A client.\n1. Try to access the S3 bucket with reads and writes from the AWS CLI.\n1. With the Hadoop configuration set too use the role,\n try to read data from the `hadoop fs` CLI:\n`hadoop fs -ls -p s3a://bucket/`\n1. With the hadoop CLI, try to create a new directory with a request such as\n`hadoop fs -mkdirs -p s3a://bucket/path/p1/`\n\n\n### IOException: \"Unset property fs.s3a.assumed.role.arn\"\n\nThe Assumed Role Credential Provider is enabled, but `fs.s3a.assumed.role.arn` is unset.\n\n```\njava.io.IOException: Unset property fs.s3a.assumed.role.arn\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:76)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n```\n\n### \"Not authorized to perform sts:AssumeRole\"\n\nThis can arise if the role ARN set in `fs.s3a.assumed.role.arn` is invalid\nor one to which the caller has no access.\n\n```\njava.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Not authorized to perform sts:AssumeRole (Service: AWSSecurityTokenService; Status Code: 403;\n Error Code: AccessDenied; Request ID: aad4e59a-f4b0-11e7-8c78-f36aaa9457f6):AccessDenied\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n```\n\n### \"Roles may not be assumed by root accounts\"\n\nYou can't assume a role with the root account of an AWS account;\nyou need to create a new user and give it the permission to change into\nthe role.\n\n```\njava.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;\n Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2):\n No AWS Credentials provided by AssumedRoleCredentialProvider :\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService;\n Status Code: 403; Error Code: AccessDenied; Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n ... 22 more\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;\n Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n```\n\n### \"Assume Role session duration should be in the range of 15min - 1Hr\"\n\nThe value of `fs.s3a.assumed.role.session.duration` is out of range.\n\n```\njava.lang.IllegalArgumentException: Assume Role session duration should be in the range of 15min\n- 1Hr\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$Builder.withRoleSessionDurationSeconds(STSAssumeRoleSessionCredentialsProvider.java:437)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:86)\n```\n\n\n### `MalformedPolicyDocumentException` \"The policy is not in the valid JSON format\"\n\n\nThe policy set in `fs.s3a.assumed.role.policy` is not valid according to the\nAWS specification of Role Policies.\n\n```\nrg.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n The policy is not in the valid JSON format. (Service: AWSSecurityTokenService; Status Code: 400;\n Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c):\n MalformedPolicyDocument: The policy is not in the valid JSON format.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\nCaused by: com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n The policy is not in the valid JSON format.\n (Service: AWSSecurityTokenService; Status Code: 400;\n Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 19 more\n```\n\n### `MalformedPolicyDocumentException` \"Syntax errors in policy\"\n\nThe policy set in `fs.s3a.assumed.role.policy` is not valid JSON.\n\n```\norg.apache.hadoop.fs.s3a.AWSBadRequestException:\nInstantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n Syntax errors in policy. (Service: AWSSecurityTokenService;\n Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45):\n MalformedPolicyDocument: Syntax errors in policy.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 19 more\n```\n\n### `IOException`: \"AssumedRoleCredentialProvider cannot be in fs.s3a.assumed.role.credentials.provider\"\n\nYou can't use the Assumed Role Credential Provider as the provider in\n`fs.s3a.assumed.role.credentials.provider`.\n\n```\njava.io.IOException: AssumedRoleCredentialProvider cannot be in fs.s3a.assumed.role.credentials.provider\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:86)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n```\n\n### `AWSBadRequestException`: \"not a valid key=value pair\"\n\n\nThere's an space or other typo in the `fs.s3a.access.key` or `fs.s3a.secret.key` values used for the\ninner authentication which is breaking signature creation.\n\n```\n org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header:\n 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code:\n IncompleteSignature; Request ID: c4a8841d-f556-11e7-99f9-af005a829416):IncompleteSignature:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign)\n in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date,\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature;\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign)\n in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date,\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature;\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 25 more\n```\n\n### `AccessDeniedException/InvalidClientTokenId`: \"The security token included in the request is invalid\"\n\nThe credentials used to authenticate with the AWS Security Token Service are invalid.\n\n```\n[ERROR] Failures:\n[ERROR] java.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n The security token included in the request is invalid.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId;\n Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7):InvalidClientTokenId\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\nThe security token included in the request is invalid.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId;\n Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 25 more\n```\n\n### `AWSSecurityTokenServiceExceptiond`: \"Member must satisfy regular expression pattern: `[\\w+=,.@-]*`\"\n\n\nThe session name, as set in `fs.s3a.assumed.role.session.name` must match the wildcard `[\\w+=,.@-]*`.\n\nIf the property is unset, it is extracted from the current username and then sanitized to\nmatch these constraints.\nIf set explicitly, it must be valid.\n\n```\norg.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint: Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n Request ID: 7c437acb-f55d-11e7-9ad8-3b5e4f701c20):ValidationError:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint: Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint:\n Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:135)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:124)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 26 more\n```\n\n\n### `java.nio.file.AccessDeniedException` within a FileSystem API call\n\nIf an operation fails with an `AccessDeniedException`, then the role does not have\nthe permission for the S3 Operation invoked during the call.\n\n```\njava.nio.file.AccessDeniedException: s3a://bucket/readonlyDir:\n rename(s3a://bucket/readonlyDir, s3a://bucket/renameDest)\n on s3a://bucket/readonlyDir:\n com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied\n (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1;\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=),\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=:AccessDenied\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:216)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:143)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:853)\n ...\nCaused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied\n (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1;\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=),\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229)\n at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176)\n at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2066)\n at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2052)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObject$7(S3AFileSystem.java:1338)\n at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:314)\n at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:280)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObject(S3AFileSystem.java:1334)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1657)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:1046)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:851)\n```\n\nThis is the policy restriction behaving as intended: the caller is trying to\nperform an action which is forbidden.\n\n1. If a policy has been set in `fs.s3a.assumed.role.policy` then it must declare *all*\npermissions which the caller is allowed to perform. The existing role policies\nact as an outer constraint on what the caller can perform, but are not inherited.\n\n1. If the policy for a bucket is set up with complex rules on different paths,\ncheck the path for the operation.\n\n1. The policy may have omitted one or more actions which are required.\nMake sure that all the read and write permissions are allowed for any bucket/path\nto which data is being written to, and read permissions for all\nbuckets read from.\n\nIf the bucket is using SSE-KMS to encrypt data:\n\n1. The caller must have the `kms:Decrypt` permission to read the data.\n1. The caller needs `kms:Decrypt` and `kms:GenerateDataKey`.\n\nWithout permissions, the request fails *and there is no explicit message indicating\nthat this is an encryption-key issue*.\n\n### `AccessDeniedException` + `AmazonDynamoDBException`\n\n```\njava.nio.file.AccessDeniedException: bucket1:\n com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:\n User: arn:aws:sts::980678866538:assumed-role/s3guard-test-role/test is not authorized to perform:\n dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-1:980678866538:table/bucket1\n (Service: AmazonDynamoDBv2; Status Code: 400;\n```\n\nThe caller is trying to access an S3 bucket which uses S3Guard, but the caller\nlacks the relevant DynamoDB access permissions.\n\nThe `dynamodb:DescribeTable` operation is the first one used in S3Guard to access,\nthe DynamoDB table, so it is often the first to fail. It can be a sign\nthat the role has no permissions at all to access the table named in the exception,\nor just that this specific permission has been omitted.\n\nIf the role policy requested for the assumed role didn't ask for any DynamoDB\npermissions, this is where all attempts to work with a S3Guarded bucket will\nfail. Check the value of `fs.s3a.assumed.role.policy`\n","isInternal":false,"tokens":12464,"sizeBytes":49743},{"name":"committer_architecture.md","path":"hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committer_architecture.md","rawUrl":"https://raw.githubusercontent.com/apache/hadoop/HEAD/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committer_architecture.md","title":"Hadoop-aws Skill","category":"anthropic-skill","format":"markdown","content":"\n\n# S3A Committers: Architecture and Implementation\n\n\n\nThis document covers the architecture and implementation details of the S3A committers.\n\nFor information on using the committers, see [the S3A Committers](./committer.html).\n\n\n## Problem: Efficient, reliable commits of work to consistent S3 buckets\n\n\nThe standard commit algorithms (the `FileOutputCommitter` and its v1 and v2 algorithms)\nrely on directory rename being an `O(1)` atomic operation: callers output their\nwork to temporary directories in the destination filesystem, then\nrename these directories to the final destination as way of committing work.\nThis is the perfect solution for committing work against any filesystem with\nconsistent listing operations and where the `FileSystem.rename()` command\nis an atomic `O(1)` operation.\n\nUsing rename allows individual tasks to work in temporary directories, with the\nrename as the atomic operation can be used to explicitly commit tasks and\nultimately the entire job. Because the cost of the rename is low, it can be\nperformed during task and job commits with minimal delays. Note that HDFS\nwill lock the namenode metadata during the rename operation, so all rename() calls\nwill be serialized. However, as they only update the metadata of two directory\nentries, the duration of the lock is low.\n\nIn contrast to a \"real\" filesystem, Amazon's S3A object store, similar to\nmost others, does not support `rename()` at all. A hash operation on the filename\ndetermines the location of of the data —there is no separate metadata to change.\nTo mimic renaming, the Hadoop S3A client has to copy the data to a new object\nwith the destination filename, then delete the original entry. This copy\ncan be executed server-side, but as it does not complete until the in-cluster\ncopy has completed, it takes time proportional to the amount of data.\n\nThe rename overhead is the most visible issue, but it is not the most dangerous.\nThat is the fact that path listings have no consistency guarantees, and may\nlag the addition or deletion of files.\nIf files are not listed, the commit operation will *not* copy them, and\nso they will not appear in the final output.\n\nThe solution to this problem is closely coupled to the S3 protocol itself:\ndelayed completion of multi-part PUT operations\n\nThat is: tasks write all data as multipart uploads, *but delay the final\ncommit action until until the final, single job commit action.* Only that\ndata committed in the job commit action will be made visible; work from speculative\nand failed tasks will not be instantiated. As there is no rename, there is no\ndelay while data is copied from a temporary directory to the final directory.\nThe duration of the commit will be the time needed to determine which commit operations\nto construct, and to execute them.\n\n\n## Terminology\n\n* *Job*: a potentially parallelized query/operation to execute. The execution\nof a job: the division of work into tasks and the management of their completion,\nis generally executed in a single process.\n\nThe output of a Job is made visible to other stages in a larger operation\nsequence or other applications if the job *completes successfully*.\n\n* *Job Driver*. Not sure quite what term to use here. Whatever process schedules\ntask execution, tracks success/failures and, determines when all the work has been\nprocessed and then commits the output. It may also determine that a job\nhas failed and cannot be recovered, in which case the job is aborted.\nIn MR and Tez, this is inside the YARN application master.\nIn Spark it is the driver, which can run in the AM, the YARN client, or other\nplaces (e.g Livy?).\n\n* *Final directory*: the directory into which the output of a job is placed\nso as to be visible.\n\n* *Task* a single operation within a job, on a single process, one which generates\none or more files.\nAfter a successful job completion, the data MUST be visible in the final directory.\nA task completes successfully if it generates all the output it expects to without\nfailing in some way (error in processing; network/process failure).\n\n* *Job Context* an instance of the class `org.apache.hadoop.mapreduce.JobContext`,\nwhich provides a read-only view of the Job for the Job Driver and tasks.\n\n* *Task Attempt Context* an instance of the class\n`org.apache.hadoop.mapreduce.TaskAttemptContext extends JobContext, Progressable`,\nwhich provides operations for tasks, such as getting and setting status,\nprogress and counter values.\n\n* *Task Working Directory*: a directory for exclusive access by a single task,\ninto which uncommitted work may be placed.\n\n* *Task Commit* The act of taking the output of a task, as found in the\nTask Working Directory, and making it visible in the final directory.\nThis is traditionally implemented via a `FileSystem.rename()` call.\n\n It is useful to differentiate between a *task-side commit*: an operation performed\n in the task process after its work, and a *driver-side task commit*, in which\n the Job driver performs the commit operation. Any task-side commit work will\n be performed across the cluster, and may take place off the critical part for\n job execution. However, unless the commit protocol requires all tasks to await\n a signal from the job driver, task-side commits cannot instantiate their output\n in the final directory. They may be used to promote the output of a successful\n task into a state ready for the job commit, addressing speculative execution\n and failures.\n\n* *Job Commit* The act of taking all successfully completed tasks of a job,\nand committing them. This process is generally non-atomic; as it is often\na serialized operation at the end of a job, its performance can be a bottleneck.\n\n* *Task Abort* To cancel a task such that its data is not committed.\n\n* *Job Abort* To cancel all work in a job: no task's work is committed.\n\n* *Speculative Task Execution/ \"Speculation\"* Running multiple tasks against the same\ninput dataset in parallel, with the first task which completes being the one\nwhich is considered successful. Its output SHALL be committed; the other task\nSHALL be aborted. There's a requirement that a task can be executed in parallel,\nand that the output of a task MUST NOT BE visible until the job is committed,\nat the behest of the Job driver. There is the expectation that the output\nSHOULD BE the same on each task, though that MAY NOT be the case. What matters\nis if any instance of a speculative task is committed, the output MUST BE\nconsidered valid.\n\nThere is an expectation that the Job Driver and tasks can communicate: if a task\nperforms any operations itself during the task commit phase, it shall only do\nthis when instructed by the Job Driver. Similarly, if a task is unable to\ncommunicate its final status to the Job Driver, it MUST NOT commit is work.\nThis is very important when working with S3, as some network partitions could\nisolate a task from the Job Driver, while the task retains access to S3.\n\n## The execution workflow\n\n\n**setup**:\n\n* A job is created, assigned a Job ID (YARN?).\n* For each attempt, an attempt ID is created, to build the job attempt ID.\n* `Driver`: a `JobContext` is created/configured\n* A committer instance is instantiated with the `JobContext`; `setupJob()` invoked.\n\n\n## The `FileOutputCommitter`\n\nThe standard commit protocols are implemented in\n `org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter`.\n\nThere are two algorithms, the \"v1\" designed to address failures and restarts\nof the MapReduce application master. The V2 algorithm cannot recover from failure\nexcept by re-executing the entire job. It does, however, propagate all its work\nto the output directory in the task commit. When working with object stores which\nmimic `rename()` by copy and delete, it is more efficient due to the reduced\nnumber of listings, copies and deletes, and, because these are executed in\ntask commits, eliminates the final `O(data)` pause at the end of all work.\nIt is still highly inefficient, but the inefficiencies are less visible as\nlarge pauses in execution.\n\n\nNotes\n\n* The v1 algorithm was implemented to handle MapReduce AM restarts, it was\n not used in Hadoop 1.x, whose JobTracker could not recover from failures.\n Historically then, it is the second version of a file commit algorithm.\n* Because the renames are considered to be fast, there is no logging\nof renames being in progress, or their duration.\n* Speculative execution is supported by having every task attempt write its\nuncommitted data to a task attempt directory. When a task is ready to commit,\nit must notify that job driver that it is ready to commit -the job driver\nwill then commit or abort the task.\n\n## Hadoop MR Commit algorithm \"1\"\n\n\nThe \"v1\" commit algorithm is the default commit algorithm in Hadoop 2.x;\nit was implemented as part of [MAPREDUCE-2702](https://issues.apache.org/jira/browse/MAPREDUCE-2702).\n\nThis algorithm is designed to handle a failure and restart of the Job driver,\nwith the restarted job driver only rerunning the incomplete tasks; the\noutput of the completed tasks is recovered for commitment when the restarted\njob completes.\n\nThere is a cost: the time to commit by recursive listing all files in all\ncommitted task directories, and renaming this.\n\nAs this is performed sequentially, time to commit is `O(files)`, which is\ngenerally `O(tasks)`.\n\n\n\n```python\n#Job Attempt Path is `$dest/_temporary/$appAttemptId/`\n\njobAttemptPath = '$dest/_temporary/$appAttemptId/'\n\n# Task Attempt Path is `$dest/_temporary/$appAttemptId/_temporary/$taskAttemptID`\ntaskAttemptPath = '$dest/_temporary/$appAttemptId/_temporary/$taskAttemptID'\n\n#Task committed path is `$dest/_temporary/$appAttemptId/$taskAttemptID`\ntaskCommittedPath = '$dest/_temporary/$appAttemptId/$taskAttemptID'\n```\n\nTasks write in/under the task attempt path.\n\n### Job Setup\n\n```python\nfs.mkdir(jobAttemptPath)\n```\n\n### Task Setup\n\nNone: directories are created on demand.\n\n\n### Task Commit\n\nRename task attempt path to task committed path.\n\n```python\ndef needsTaskCommit(fs, jobAttemptPath, taskAttemptPath, dest):\n return fs.exists(taskAttemptPath)\n\n\ndef commitTask(fs, jobAttemptPath, taskAttemptPath, dest):\n if fs.exists(taskAttemptPath) :\n fs.delete(taskCommittedPath, recursive=True)\n fs.rename(taskAttemptPath, taskCommittedPath)\n```\n\nOn a genuine filesystem this is an `O(1)` directory rename.\n\nOn an object store with a mimiced rename, it is `O(data)` for the copy,\nalong with overhead for listing and deleting all files (For S3, that's\n`(1 + files/500)` lists, and the same number of delete calls.\n\n\n### Task Abort\n\nDelete task attempt path.\n\n```python\ndef abortTask(fs, jobAttemptPath, taskAttemptPath, dest):\n fs.delete(taskAttemptPath, recursive=True)\n```\n\nOn a genuine filesystem this is an `O(1)` operation. On an object store,\nproportional to the time to list and delete files, usually in batches.\n\n\n### Job Commit\n\nMerge all files/directories in all task committed paths into final destination path.\nOptionally; create 0-byte `_SUCCESS` file in destination path.\n\n```python\ndef commitJob(fs, jobAttemptDir, dest):\n for committedTask in fs.listFiles(jobAttemptDir):\n mergePathsV1(fs, committedTask, dest)\n fs.touch(\"$dest/_SUCCESS\")\n```\n\n(See below for details on `mergePaths()`)\n\n\nA failure during job commit cannot be recovered from except by re-executing\nthe entire query:\n\n```python\ndef isCommitJobRepeatable() :\n return False\n```\n\nAccordingly, it is a failure point in the protocol. With a low number of files\nand fast rename/list algorithms, the window of vulnerability is low. At\nscale, the vulnerability increases. It could actually be reduced through\nparallel execution of the renaming of of committed tasks.\n\n\n### Job Abort\n\nDelete all data under job attempt path.\n\n```python\ndef abortJob(fs, jobAttemptDir, dest):\n fs.delete(jobAttemptDir, recursive = True)\n```\n\n### Job Cleanup\n\n```python\ndef cleanupJob(fs, dest):\n fs.delete('$dest/_temporary', recursive = True)\n```\n\n\n### Job Recovery Before `commitJob()`\n\nFor all committers, the recovery process takes place in the application\nmaster.\n1. The job history file of the previous attempt is loaded and scanned\nto determine which tasks were recorded as having succeeded.\n1. For each successful task, the job committer has its `recoverTask()` method\ninvoked with a `TaskAttemptContext` built from the previous attempt's details.\n1. If the method does not raise an exception, it is considered to have been\nrecovered, and not to be re-executed.\n1. All other tasks are queued for execution.\n\nFor the v1 committer, task recovery is straightforward.\nThe directory of the committed task from the previous attempt is\nmoved under the directory of the current application attempt.\n\n```python\ndef recoverTask(tac):\n oldAttemptId = appAttemptId - 1\n fs.rename('$dest/_temporary/oldAttemptId/${tac.taskId}',\n '$dest/_temporary/appAttemptId/${tac.taskId}')\n```\n\nThis significantly improves time to recover from Job driver (here MR AM) failure.\nThe only lost work is that of all tasks in progress -those which had generated\ndata but were not yet committed.\n\nOnly the failure of the job driver requires a job restart, not an individual\ntask. Therefore the probability of this happening is independent of the number\nof tasks executed in parallel, instead simply due to the duration of the query.\n\nThe longer the task, the higher the risk of failure, the more value there is\nin recovering the work in progress.\n\nFast queries not only have a lower risk of failure, they can recover from\nfailure simply by rerunning the entire job. This is implicitly the strategy\nin Spark, which does not attempt to recover any in-progress jobs. The faster\nyour queries, the simpler your recovery strategy needs to be.\n\n### Job Recovery During `commitJob()`\n\nThis is not possible; a failure during job commit requires the entire job\nto be re-executed after cleaning up the destination directory.\n\n### `mergePaths(FileSystem fs, FileStatus src, Path dest)` Algorithm\n\n`mergePaths()` is the core algorithm to merge data; it is somewhat confusing\nas the implementation mixes the strategies for both algorithms across\ntwo co-recursive routines, `mergePaths()` and `renameOrMerge()`.\n\n\nHere the two algorithms have been split, and one of the co-recursive methods\ninlined.\n\n```python\ndef mergePathsV1(fs, src, dest) :\n if fs.exists(dest) :\n toStat = fs.getFileStatus(dest)\n else:\n toStat = None\n\n if src.isFile :\n if not toStat is None :\n fs.delete(dest, recursive = True)\n fs.rename(src.getPath, dest)\n else :\n # src is directory, choose action on dest type\n if not toStat is None :\n if not toStat.isDirectory :\n # Destination exists and is not a directory\n fs.delete(dest)\n fs.rename(src.getPath(), dest)\n else :\n # Destination exists and is a directory\n # merge all children under destination directory\n for child in fs.listStatus(src.getPath) :\n mergePathsV1(fs, child, dest + child.getName)\n else :\n # destination does not exist\n fs.rename(src.getPath(), dest)\n```\n\n## The v2 Commit Algorithm\n\n\nThe v2 algorithm directly commits task output into the destination directory.\nIt is essentially a re-implementation of the Hadoop 1.x commit algorithm.\n\n1. During execution, intermediate data becomes visible.\n1. On a failure, all output must be deleted and the job restarted.\n\nIt implements `mergePaths` differently, as shown below.\n\n```python\ndef mergePathsV2(fs, src, dest) :\n if fs.exists(dest) :\n toStat = fs.getFileStatus(dest)\n else:\n toStat = None\n\n if src.isFile :\n if not toStat is None :\n fs.delete(dest, recursive = True)\n fs.rename(src.getPath, dest)\n else :\n # destination is directory, choose action on source type\n if src.isDirectory :\n if not toStat is None :\n if not toStat.isDirectory :\n # Destination exists and is not a directory\n fs.delete(dest)\n fs.mkdirs(dest) #\n for child in fs.listStatus(src.getPath) : # HERE\n mergePathsV2(fs, child, dest + child.getName) #\n\n else :\n # Destination exists and is a directory\n # merge all children under destination directory\n for child in fs.listStatus(src.getPath) :\n mergePathsV2(fs, child, dest + child.getName)\n else :\n # destination does not exist\n fs.mkdirs(dest) #\n for child in fs.listStatus(src.getPath) : # HERE\n mergePathsV2(fs, child, dest + child.getName) #\n```\n\nBoth recurse down any source directory tree, and commit single files\nby renaming the files.\n\nA a key difference is that the v1 algorithm commits a source directory to\nvia a directory rename, which is traditionally an `O(1)` operation.\n\nIn contrast, the v2 algorithm lists all direct children of a source directory\nand recursively calls `mergePath()` on them, ultimately renaming the individual\nfiles. As such, the number of renames it performs equals the number of source\n*files*, rather than the number of source *directories*; the number of directory\nlistings being `O(depth(src))` , where `depth(path)` is a function returning the\ndepth of directories under the given path.\n\nOn a normal filesystem, the v2 merge algorithm is potentially more expensive\nthan the v1 algorithm. However, as the merging only takes place in task commit,\nit is potentially less of a bottleneck in the entire execution process.\n\nOn an object store, it is suboptimal not just from its expectation that `rename()`\nis an `O(1)` operation, but from its expectation that a recursive tree walk is\nan efficient way to enumerate and act on a tree of data. If the algorithm was\nswitched to using `FileSystem.listFiles(path, recursive)` for a single call to\nenumerate all children under a path, then the listing operation would be significantly\nfaster, at least on a deep or wide tree. However, for any realistic dataset,\nthe size of the output files is likely to be the main cause of delays. That\nis, if the cost of `mergePathsV2` is `O(depth(src)) + O(data))`, then\ngenerally the `O(data)` value will be more significant than the `depth(src)`.\n\nThere is one key exception: tests which work on small amounts of data yet try\nto generate realistic output directory structures. In these tests the cost\nof listing directories and calling `getFileStatus()` could exceed that of the copy\ncalls. This is why small-scale tests of the commit algorithms against object stores\nmust be considered significantly misleading.\n\n### v2 Task Commit\n\nRename task attempt path to task committed path.\n\n```python\ndef needsTaskCommit(fs, jobAttemptPath, taskAttemptPath, dest):\n return fs.exists(taskAttemptPath)\n\ndef commitTask(fs, jobAttemptPath, taskAttemptPath, dest):\n if fs.exists(taskAttemptPath) :\n mergePathsV2(fs. taskAttemptPath, dest)\n```\n\n### v2 Task Abort\n\nDelete task attempt path.\n\n```python\ndef abortTask(fs, jobAttemptPath, taskAttemptPath, dest):\n fs.delete(taskAttemptPath, recursive=True)\n```\n\nCost: `O(1)` for normal filesystems, `O(files)` for object stores.\n\n\n### v2 Job Commit\n\nAs all the task output is already completed, all that is needed is\nto touch the `_SUCCESS` marker.\n\n```python\ndef commitJob(fs, jobAttemptDir, dest):\n fs.touch(\"$dest/_SUCCESS\")\n```\nCost: `O(1)`\n\nA failure during job abort is implicitly repeatable\n\n```python\ndef isCommitJobRepeatable() :\n return True\n```\n\n### v2 Job Abort\n\nDelete all data under job attempt path.\n\n```python\ndef abortJob(fs, jobAttemptDir, dest):\n fs.delete(jobAttemptDir, recursive=True)\n```\n\nCost: `O(1)` for normal filesystems, `O(files)` for object stores.\n\n### v2 Task Recovery\n\nAs no data is written to the destination directory, a task can be cleaned up\nby deleting the task attempt directory.\n\n### v2 Job Recovery Before `commitJob()`\n\n\nBecause the data has been renamed into the destination directory, all tasks\nrecorded as having being committed have no recovery needed at all:\n\n```python\ndef recoverTask(tac):\n```\n\nAll active and queued tasks are scheduled for execution.\n\nThere is a weakness here, the same one on a failure during `commitTask()`:\nit is only safe to repeat a task which failed during that commit operation\nif the name of all generated files are constant across all task attempts.\n\nIf the Job AM fails while a task attempt has been instructed to commit,\nand that commit is not recorded as having completed, the state of that\nin-progress task is unknown...really it isn't be safe to recover the\njob at this point.\n\n\n### v2 Job Recovery During `commitJob()`\n\nThis is straightforward: `commitJob()` is re-invoked.\n\n## How MapReduce uses the committer in a task\n\nMapReduce runs each Mapper or Reducer in its own container; it\ngets its own process. Implicitly, this also means that each task gets\nits own instance of every filesystem.\n\nThe work is choreographed in `org.apache.hadoop.mapred.Task`, for which there\nare specific subclassing for the Map (`MapTask`) and reduce (`ReduceTask`).\nThere are also cleanup tasks whose role is simpler: clean things up,\nand a Job setup task which runs `OutputCommittre.setupJob`. That is: even\nthe job setup phase is run in a Task.\n\nMapTask uses the committer to write the output of all the mappers into\nthe filesystem, ready for the reducers. Each partition writes its data\nas a `MapFile`, which is actually two `SequenceFile` files in a directory,:\nthe `data` file of all Key-Value\noutput, and `index` which contains an index of some of the keys in the file.\n\nThis is all written to the local filesystem.\n\nThe `ReduceTask` does the final write to the destination filesystem.\n\nBecause the Map phase uses a committer to commits intermediate work,\nany plug in committer supplied to a process\nthrough any extension mechanism *must* work with the output generated by a mapper.\nThe staging committers do only if unique filenames are disabled, but as they\nand the magic committers are only meant to be used for the final output of work,\nit is somewhat moot. What is important is to be able to use different committers\nfor the map phase as the final reduce. This is implicit if separate committers\nare defined for different filesystems: the new committer can be defined for\nthe final destination FS, while `file://` can retain the default\n`FileOutputCommitter`.\n\n### Task Setup\n\n`Task.initialize()`: read in the configuration, instantiate the `JobContextImpl`\nand `TaskAttemptContextImpl` instances bonded to the current job & task.\n\n### Task Ccommit\n\nAfter the work is completed, `Task.done()` is invoked. Which is essentially the following\ncodepath:\n\n```java\nif (committer.needsTaskCommit(taskContext)) {\n // get permission to commit from AM\n int retries = MAX_RETRIES;\n while(true) {\n try {\n umbilical.commitPending(taskId, taskStatus);\n break;\n } catch(IOException ie) {\n if (--retries == 0) {\n // FAIL WITHOUT CALLING ABORT\n System.exit(67);\n }\n }\n }\n // commit the work\n try {\n committer.commitTask(taskContext);\n } catch (IOException iee) {\n // failure: abort\n try {\n committer.abortTask(taskContext);\n } catch (IOException ioe) {\n LOG.warn(\"Failure cleaning up: \" +\n StringUtils.stringifyException(ioe));\n }\n throw iee;\n }\n}\n```\n\nThat is: if and only if there is data to write, the Task requests clearance\nto do so from the AM. This ensures that speculative work is not committed if\nanother task has written it, *and that even non-speculative work is not committed\nif the task has lost contact with the AM*.\n\nThat is: task output is not committed if the Job's AM has failed or\nis in a separate network partition from the Task.\n\nIf permission is granted, the commit call is invoked.\nIf this doesn't work, `abortTask()` is called; all\nfailures there are logged and swallowed.\n\n\nThis method actually appears to be limited in that it does not guarantee\nthat `committer.abortTask()` is always called: if the `umbilical.commitPending()` calls\nfail repeatedly and the Task process aborts, `committer.abortTask()` is not called.\nIf this is due to a network partition, we can hope that the AM invokes\n`committer.abortTask()`, and that if it is an AM failure then a restarted AM can\nclean up its previous attempts. For the classic FileOutputCommitter, listing and\ndeleting the previous attempt's data is straightforward. However, for S3 committers\nusing Multipart Upload as the means of uploading uncommitted data, it is critical\nto ensure that pending uploads are always aborted. This can be done by\n\n* Making sure that all task-side failure branches in `Task.done()` call `committer.abortTask()`.\n* Having job commit & abort cleaning up all pending multipart writes to the same directory\ntree. That is: require that no other jobs are writing to the same tree, and so\nlist all pending operations and cancel them.\n* Add a CLI command to verify that there are no commits under a path; returning\na non-zero exit code if there are. This can be used in testing.\n* Add a CLI command to purge commits under a path. This can be used in workflows,\nif felt necessary, and in test runs to isolate the output of different tests.\n* Document a recommended purge-pending-commit timeout for the system\n* Add probes in the integration tests to verify that pending uploads are never\npresent after a write.\n\n\n\n### How MapReduce uses the committer in the Application Master\n\nThe AM uses a committer to set up and commit the job. To support failure\nand recovery of the AM, `OutputCommitter.isRecoverySupported()` is used\nto declare whether all the output of successful tasks can be used in the final\njob, or the entire job needs to be reset and repeated.\n`OutputCommitter.isCommitJobRepeatable()` addresses the other question:\ncan a committer recover from a failure of the commit process itself.\n\nA committer is created in the Application Master, and handed to an instance\nof `org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler`,\nwhich then manages the job-level lifecycle.\n\nIf the MRv1 API is used, the committer is chosen from the value of\n`\"mapred.output.committer.class\"`; in the MRv2 API the output format\nis instantiated, then asked for a committer using a task and task attempt ID of\n0. A committer is obtained from the output format via a call to\n`Committer.getOutputCommitter(taskContext)`, again using the task attempt context\nwith the (job, 0, 0) task and attempt IDs. That is: even for the job committers,\na task context is always passed in to the `OutputFormat` when requesting a committer.\n*It is critical for all implementations of `OutputCommitter.abortTask()` to\nbe able to execute from the AM, rather than the container and host running\nthe task. Furthermore, all information needed for the abort (paths, filesystem instances\n&c) *must* be retrieved from the `TaskAttemptContext` passed to the method,\nrather than relying on fields initiated from the context passed to the constructor.\n\n\n#### AM: Job setup: `OutputCommitter.setupJob()`\n\nThis is initiated in `org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.StartTransition`.\nIt is queued for asynchronous execution in `org.apache.hadoop.mapreduce.v2.app.MRAppMaster.startJobs()`,\nwhich is invoked when the service is started. Thus: the job is set up when the\nAM is started.\n\n\n#### AM: Job Commit: `OutputCommitter.commitJob()`\n\nIn the \"default\"/cluster filesystem, the `CommitterEventHandler` uses data in\nthe staging area defined in `yarn.app.mapreduce.am.staging-dir`\n(default `/tmp/hadoop-yarn/staging/$user/.staging`), in a subdirectory\nnamed from the job ID.\n\nThree paths are built up for using the filesystem for tracking the state\nof the commit process, with a goal of making the commit operation recoverable,\nwhere supported.\n\n\n| name | role |\n|------|--------|\n| `COMMIT_STARTED` | mark the commencement of the job commit |\n| `COMMIT_SUCCESS` | mark the successful completion of the job commit |\n| `COMMIT_FAIL` | mark the failure of the job commit |\n\nThese markers are used to manage job restart/failure of they happen during\nthe job commit itself.\n\nWhen an AM starts up, it looks in its staging area for these files. as a way\nto determine the previous state of the job. If there are no `COMMIT_` marker files,\nthe job is considered not to have attempted to commit itself yet.\n\n\nThe presence of `COMMIT_SUCCESS` or `COMMIT_FAIL` are taken as evidence\nthat the previous job completed successfully or unsuccessfully; the AM\nthen completes with a success/failure error code, without attempting to rerun\nthe job.\n\nIf `COMMIT_STARTED` exists but not either of the completion markers, then,\nif the committer declares that its job commit operation is repeatable\n(`Committer.isCommitJobRepeatable(jobContext) == true`), then an attempt\nis made to recommit the job, deleting the `COMMIT_STARTED` and commencing\nthe commit process again.\n\nThese `COMMIT_STARTED` files are simply 0-byte files, but are created\nwith the overwrite bit only set to true if the job commit is considered\nrepeatable:\n\n```java\nprivate void touchz(Path p, boolean overwrite) throws IOException {\n fs.create(p, overwrite).close();\n}\n```\nThat is: the atomicity of the `create(path, overwrite=false)` on the cluster\nfilesystem is used to guarantee that only one process will attempt to commit\na specific job.\n\n```java\nboolean commitJobIsRepeatable = committer.isCommitJobRepeatable(\n event.getJobContext());\ntry {\n touchz(startCommitFile, commitJobIsRepeatable);\n waitForValidCommitWindow();\n committer.commitJob(event.getJobContext());\n touchz(endCommitSuccessFile, commitJobIsRepeatable);\n} catch (Exception e) {\n touchz(endCommitFailureFile, commitJobIsRepeatable);\n}\n```\n\nThe `waitForValidCommitWindow()` operation is important: it declares that\nthe committer must not commit unless there has been communication with the YARN\nResource Manager with in `yarn.app.mapreduce.am.job.committer.commit-window` milliseconds\n(default: 10,000). It does this by waiting until the next heartbeat it received.\nThere's a possible bug here: if the interval is set too small the thread may\npermanently spin waiting a callback within the window. Ignoring that, this algorithm\nguarantees that\n\n1. As only one call can create a file with `overwrite=false`,\n only one process's attempt to commit a non-repeatable job will proceed\n\n1. Only a process with contact with the YARN within the configured window\n may commit a job.\n\n1. If the AM is partitioned from the rest of the network, provided that its clock\nis monotonically increasing at the same rate as the rest of the cluster, then\nthe rest of the cluster can be confident that\n`yarn.app.mapreduce.am.ob.committer.commit-window` milliseconds after the AM\nsuccessfully heartbeated to the YARN RM, then the output of this job attempt\nwill *never* be committed. This permits 1 job to run simultaneously, but\nhelps ensure that only one of them will attempt to commit.\n\n1. Provided YARN heartbeats are only sent to the AM which successfully created\nthe `COMMIT_STARTED` file, it will initiate the commit operation.\n\nPossible issues with this algorithm:\n\n* The `COMMIT_STARTED` file is created before waiting to get a heartbeat. It\nmay be that this AM has lost contact with YARN, but doesn't know it yet. When the\nYARN liveness protocols eventually time out, the AM will correctly terminate,\nbut as the `COMMIT_STARTED` file has been created at this point, no other launched\nAM will be able to commit.\n\n* If two committers attempt to create `COMMIT_STARTED` files on a no-repeatable\ncommit, one will succeed, wait for a heartbeat then attempt a (possibly slow) commit.\nThe second committer will fail, and will *immediately* create a `COMMIT_FAILED` file.\nAs a result, the state of the staging area will imply that the commit has failed,\nwhen really it is in progress, and that only the second process failed.\n\nIt would seem safer to address this through\n1. Waiting for a heartbeat before creating the `COMMIT_STARTED` file.\n1. Maybe: not creating the `COMMIT_FAILED` file if the failure happens when\ntrying to create the `COMMIT_STARTED` file. That is: only a process which\nsuccessfully created the `COMMIT_STARTED` file may indicate that a commit has failed.\n\n\n### AM: Cancelling job commit\n\nThe thread performing the commit is interrupted; the `CommitterEventHandler`\nawaits for it to finish. (set in `yarn.app.mapreduce.am.job.committer.cancel-timeout`\nas milliseconds).\n\n### AM: Task Abort\n\nThe AM may call the `OutputCommitter.taskAbort()` with a task attempt context,\nwhen handling the failure/loss of a container. That is: on container failure,\nthe task abort operation is executed in the AM, using the AM's committer.\nThis avoids the need to create a new container, and means that the \"best-effort\"\ntask abort does cope with container failures.\n\nA partition between the AM and the task container means that this AM-executed\ntask abort may take place while a task in the partitioned container is still\nexecuting. Unless output writing operations will fail after the abort operation,\nthe partitioned task may not become aware of the partition until it's own task\ncommit sequence in `Task.done()`, when `talkToAMTGetPermissionToCommit()`\n\n# Requirements of an S3A Committer\n\n1. Support an eventually consistent S3 object store as a reliable direct\ndestination of work through the S3A filesystem client.\n1. Efficient: implies no rename, and a minimal amount of delay in the job driver's\ntask and job commit phases,\n1. Support task failure and speculation.\n1. Can be used by existing code: Hadoop MapReduce, Spark, Hive.\n1. Retrofittable to existing subclasses of FileOutputFormat and/or compatible\nwith committers which expect a specific FileOutputFormat.\n1. Clean up uncommitted data from all task attempts, all previous attempts of\nthe job, and any previous incompleted jobs.\n1. Security: not to permit privilege escalation from other users with\nwrite access to the same file system(s).\n\n\n## Features of S3 and the S3A Client\n\n\nA core problem is that\n[object stores are not filesystems](../../../hadoop-project-dist/hadoop-common/filesystem/introduction.html);\nhow `rename()` has been emulated in the S3A client means that both the existing\nMR committer algorithms have significant performance problems.\n\n1. Single-object renames are implemented as a copy and delete sequence.\n1. COPY is atomic, but overwrites cannot be prevented.\n1. Amazon S3 is eventually consistent on listings, deletes and updates.\n1. Amazon S3 has create consistency, however, the negative response of a HEAD/GET\nperformed on a path before an object was created can be cached, unintentionally\ncreating a create inconsistency. The S3A client library does perform such a check,\non `create()` and `rename()` to check the state of the destination path, and\nso, whether the operation is permitted.\n1. Multi-object renames are sequential or parallel single object COPY+DELETE operations:\nnon atomic, `O(data)` and, on failure, can leave the filesystem in an unknown\nstate.\n1. There is a PUT operation, capable of uploading 5GB of data in one HTTP request.\n1. The PUT operation is atomic, but there is no PUT-no-overwrite option.\n1. There is a multipart POST/PUT sequence for uploading larger amounts of data\nin a sequence of PUT requests.\n\n\nThe Hadoop S3A Filesystem client supports PUT and multipart PUT for uploading\ndata, with the `S3ABlockOutputStream` of HADOOP-13560 uploading written data\nas parts of a multipart PUT once the threshold set in the configuration\nparameter `fs.s3a.multipart.size` (default: 100MB).\n\n[S3Guard](./s3guard.html) adds an option of consistent view of the filesystem\nto all processes using the shared DynamoDB table as the authoritative store of\nmetadata. Some S3-compatible object stores are fully consistent; the\nproposed algorithm is designed to work with such object stores without the\nneed for any DynamoDB tables.\n\n## Related work: Spark's `DirectOutputCommitter`\n\nOne implementation to look at is the\n[`DirectOutputCommitter` of Spark 1.6](https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/DirectParquetOutputCommitter.scala).\n\nThis implements a zero rename commit by subclassing the `ParquetOutputCommitter`\nand then\n\n1. Returning the destination directory as the task working directory.\n1. Subclassing all the task commit/abort operations to be no-ops.\n\nWith the working directory as the destination directory, there is no need\nto move/rename the task output on a successful commit. However, it is flawed.\nThere is no notion of \"committing\" or \"aborting\" a task, hence no ability to\nhandle speculative execution or failures. This is why the committer\nwas removed from Spark 2 [SPARK-10063](https://issues.apache.org/jira/browse/SPARK-10063)\n\nThere is also the issue that work-in-progress data is visible; this may or may\nnot be a problem.\n\n## Related work: IBM's \"Stocator\" committer\n\nIBM's [Stocator](https://github.com/SparkTC/stocator) can transform indirect\nwrites of V1/V2 committers into direct writes to the destination directory.\n\nHpw does it do this? It's a special Hadoop `FileSystem` implementation which\nrecognizes writes to `_temporary` paths and translate them to writes to the\nbase directory. As well as translating the write operation, it also supports\na `getFileStatus()` call on the original path, returning details on the file\nat the final destination. This allows for committing applications to verify\nthe creation/existence/size of the written files (in contrast to the magic\ncommitter covered below).\n\nThe FS targets Openstack Swift, though other object stores are supportable through\ndifferent backends.\n\nThis solution is innovative in that it appears to deliver the same semantics\n(and hence failure modes) as the Spark Direct OutputCommitter, but which\ndoes not need any change in either Spark *or* the Hadoop committers. In contrast,\nthe committers proposed here combines changing the Hadoop MR committers for\nease of pluggability, and offers a new committer exclusively for S3, one\nstrongly dependent upon and tightly integrated with the S3A Filesystem.\n\nThe simplicity of the Stocator committer is something to appreciate.\n\n## Background: The S3 multi-part PUT mechanism\n\nIn the [S3 REST API](http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html),\nmultipart uploads allow clients to upload a series of \"Parts\" of a file,\nthen commit the upload with a final call.\n\n1. Caller initiates a multipart request, including the destination bucket, key\nand metadata.\n\n POST bucket.s3.aws.com/path?uploads\n\n An `UploadId` is returned\n\n1. Caller uploads one or more parts.\n\n PUT bucket.s3.aws.com/path?partNumber=PartNumber&uploadId=UploadId\n\n The part number is used to declare the ordering of the PUTs; they\n can be uploaded in parallel and out of order.\n All parts *excluding the final part* must be 5MB or larger.\n Every upload completes with an etag returned\n\n1. Caller completes the operation\n\n POST /ObjectName?uploadId=UploadId\n \n (number)(Tag)\n ...\n \n\n This final call lists the etags of all uploaded parts and the actual ordering\n of the parts within the object.\n\nThe completion operation is apparently `O(1)`; presumably the PUT requests\nhave already uploaded the data to the server(s) which will eventually be\nserving up the data for the final path. All that is needed to complete\nthe upload is to construct an object by linking together the files in\nthe server's local filesystem and update an entry the index table of the\nobject store.\n\nIn the S3A client, all PUT calls in the sequence and the final commit are\ninitiated by the same process. *This does not have to be the case*.\nIt is that fact, that a different process may perform different parts\nof the upload, which make this algorithm viable.\n\n\n## The Netfix \"Staging\" committer\n\nRyan Blue, of Netflix, has submitted an alternate committer, one which has a\nnumber of appealing features\n\n* Doesn't have any requirements of the destination object store,\n* Known to work.\n\nThe final point is not to be underestimated, es not even\na need for a consistency layer.\n* Overall a simpler design. Especially given the need to\nbe resilient to the various failure modes which may arise.\n\n\nThe committer writes task outputs to a temporary directory on the local FS.\nTask outputs are directed to the local FS by `getTaskAttemptPath` and `getWorkPath`.\nOn task commit, the committer enumerates files in the task attempt directory (ignoring hidden files).\nEach file is uploaded to S3 using the [multi-part upload API](http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html),\n\nThe information needed to commit the upload is saved into HDFS and committed\nvia that protocol: when the job commits, the pending uploads of the successful\ntasks are all committed.\n\n\n### Commit logic\n\nThe core algorithm is as follows:\n\n1. The destination directory for output (e.g. `FileOutputFormat` and subclasses)\nis a local `file://` reference.\n1. Task commit initiates the multipart PUT to the destination object store.\n1. A list of every pending PUT for task is persisted to a single file\nwithin a consistent, cluster-wide filesystem. For Netflix, that is HDFS.\n1. The Standard `FileOutputCommitter` (algorithm 1) is used to manage the commit/abort of these\nfiles. That is: it copies only those lists of files to commit from successful tasks\ninto a (transient) job commit directory.\n1. The S3 job committer reads the pending file list for every task committed\nin HDFS, and completes those put requests.\n\nBy using `FileOutputCommmitter` to manage the propagation of the lists of files\nto commit, the existing commit algorithm implicitly becomes that defining which\nfiles will be committed at the end of the job.\n\n\nThe Netflix contribution has Hadoop `OutputCommitter` implementations for S3.\n\nThere are 3 main classes:\n* `S3MultipartOutputCommitter` is a base committer class that handles commit logic. This should not be used directly.\n* `S3DirectoryOutputCommitter` for writing unpartitioned data to S3 with conflict resolution.\n* `S3PartitionedOutputCommitter` for writing partitioned data to S3 with conflict resolution.\n\nCallers should use `S3DirectoryOutputCommitter` for single-directory outputs,\nor `S3PartitionedOutputCommitter` for partitioned data.\n\n\nThese S3 committers work by writing task outputs to a temporary directory on the local FS.\nTask outputs are directed to the local FS by `getTaskAttemptPath` and `getWorkPath`.\n\n\n### Conflict Resolution\n\nThe single-directory and partitioned committers handle conflict resolution by\nchecking whether target paths exist in S3 before uploading any data.\nThere are three conflict resolution modes, controlled by setting `fs.s3a.committer.staging.conflict-mode`:\n\n* `fail`: Fail a task if an output directory or partition already exists. (Default)\n* `append`: Upload data files without checking whether directories or partitions already exist.\n* `replace`: If an output directory exists, delete it so the new data replaces the current content.\n\nThe partitioned committer enforces the conflict mode when a conflict is detected with output data, not before the job runs.\nConflict resolution differs from an output mode because it does not enforce the mode when there is no conflict.\nFor example, overwriting a partition should remove all sub-partitions and data it contains, whether or not new output is created.\nConflict resolution will only replace partitions that have output data.\n\nWhen the conflict mode is `replace`, conflicting directories are removed during\njob commit. Data is only deleted if all tasks have completed successfully.\n\nA UUID that identifies a write is added to filenames that are uploaded to S3.\nThis allows rolling back data from jobs that fail during job commit (see failure cases below) and avoids\nfile-level conflicts when appending data to existing directories.\n\n\n*Note* the checks for existence are made via `S3AFileSystem.getFileStatus()` requests of the destination paths.\nUnless the view of the S3 store is consistent, it may be that a newly-deleted object\nis still discovered in the probe, so a commit fail, even when there is no longer any actual conflict.\n\n### Performance\n\nCompared to the previous proposal, henceforth the \"magic\" committer, this\ncommitter, the \"staging committer\", adds the extra overhead of uploading\neach file at the end of every task. This is an `O(data)` operation; it can be\nparallelized, but is bounded by the bandwidth from compute node to S3, as\nwell as the write/IOP capacity of the destination shard of S3. If many tasks\ncomplete at or near the same time, there may be a peak of bandwidth load\nslowing down the upload.\n\nTime to commit will be the same, and, given the Netflix committer has already\nimplemented the parallelization logic here, a time of `O(files/threads)`.\n\n### Resilience\n\nThere's already a lot of code in the task and job commits to handle failure.\n\nAny failure in a commit triggers a best-effort abort/revert of the commit\nactions for a task/job.\n\nTask commits delegate to the `FileOutputCommitter` to ensure that only one task's\noutput reaches the job commit.\n\nSimilarly, if a task is aborted, temporary output on the local FS is removed.\n\nIf a task dies while the committer is running, it is possible for data to be\neft on the local FS or as unfinished parts in S3.\nUnfinished upload parts in S3 are not visible to table readers and are cleaned\nup following the rules in the target bucket's life-cycle policy.\n\nFailures during job commit are handled by deleting any files that have already\nbeen completed and aborting the remaining uploads.\nBecause uploads are completed individually, the files that are deleted were visible to readers.\n\nIf the process dies while the job committer is running, there are two possible failures:\n\n1. Some directories that would be replaced have been deleted, but no new data is visible.\n2. Some new data is visible but is not complete, and all replaced directories have been removed.\n Only complete files are visible.\n\nIf the process dies during job commit, cleaning up is a manual process.\nFile names include a UUID for each write so that files can be identified and removed.\n\n\n**Failure during task execution**\n\nAll data is written to local temporary files; these need to be cleaned up.\n\nThe job must ensure that the local (pending) data is purged.\n\n**Failure during task commit**\n\n\nA process failure during the upload process will result in the\nlist of pending multipart PUTs to *not* be persisted to the cluster filesystem.\nThis window is smaller than the entire task execution, but still potentially\nsignificant, at least for large uploads.\n\nPer-file persistence, or incremental overwrites of the upload list may\nreduce the problems here, but there would still be a small risk of\nan outstanding multipart upload not being recorded\n\n**Explicit Task abort before task commit**\n\nTask will delete all local data; no uploads will be initiated.\n\n**Failure to communicate with S3 during data upload**\n\nIf an upload fails, tasks will:\n\n* Retry using the retry policies implemented in the S3AFileSystem classes\nand the AWS libraries.\n* Eventually: attempt to abort outstanding multipart uploads already initiated.\n* Remove temporary files on the local FS.\n\n\n**Explicit Job Abort**\n\nAll in-progress tasks are aborted and cleaned up. The pending commit data\nof all completed tasks can be loaded, outstanding multipart PUT requests aborted.\n\n\nThis is done by\n\n1. Listing all local files, with a best effort read attempt followed by\nan abort of all successfully read files.\n1. List and abort all pending multipart uploads.\n\nBecause of action #2, action #1 is superfluous. It is retained so as to leave\nopen the option of making action #2 a configurable option -which would be\nrequired to handle the use case of >1 partitioned commit running simultaneously/\n\n**Job Driver failure before Job Commit**\n\nBecause the local data is managed with the v1 commit algorithm, the\nsecond attempt of the job will recover all the outstanding commit data\nof the first attempt; those tasks will not be rerun.\n\nThis also ensures that on a job abort, the individual tasks' .pendingset\nfiles can be read and used to initiate the abort of those uploads.\nThat is: a recovered job can clean up the pending writes of the previous job\n\nIf the query engine does not support multiple job attempts, then the\npending commit data will not be recovered; an explicit abort operation will\nneed to be initiated (we will add a CLI command for this), or the S3 bucket\nmust be configured to automatically delete the pending request.\n\n**Job Driver failure during Job Commit**\n\nThose uploads already executed by a failed job commit will persist; those\nyet to execute will remain outstanding.\n\nThe committer currently declares itself as non-recoverable, but that\nmay not actually hold, as the recovery process could be one of:\n\n1. Enumerate all job commits from the .pendingset files (*:= Commits*).\n1. List all outstanding uploads under the destination path (*:= Outstandings*)..\n1. List all written files (for better consistency, via a GET call on the known\nfilenames)\n1. Identify all files which are yet to be completed (*Commits - Written*)/\n1. Verify that the set of pend","isInternal":false,"tokens":11161,"sizeBytes":50002},{"name":"committers.md","path":"hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md","rawUrl":"https://raw.githubusercontent.com/apache/hadoop/HEAD/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/committers.md","title":"Hadoop-aws Skill","category":"anthropic-skill","format":"markdown","content":"\n\n# Committing work to S3 with the \"S3A Committers\"\n\n\n\nThis page covers the S3A Committers, which can commit work directly\nto an S3 object store.\n\nThese committers are designed to solve a fundamental problem which\nthe standard committers of work cannot do to S3: consistent, high performance,\nand reliable commitment of output to S3.\n\nFor details on their internal design, see\n[S3A Committers: Architecture and Implementation](./committer_architecture.html).\n\n\n## Introduction: The Commit Problem\n\n\nApache Hadoop MapReduce (and behind the scenes, Apache Spark) often write\nthe output of their work to filesystems\n\nNormally, Hadoop uses the `FileOutputFormatCommitter` to manage the\npromotion of files created in a single task attempt to the final output of\na query. This is done in a way to handle failures of tasks and jobs, and to\nsupport speculative execution. It does that by listing directories and renaming\ntheir content into the final destination when tasks and then jobs are committed.\n\nThis has some key requirement of the underlying filesystem:\n\n1. When you list a directory, you see all the files which have been created in it,\nand no files which are not in it (i.e. have been deleted).\n1. When you rename a directory, it is an `O(1)` atomic transaction. No other\nprocess across the cluster may rename a file or directory to the same path.\nIf the rename fails for any reason, either the data is at the original location,\nor it is at the destination, -in which case the rename actually succeeded.\n\n**The S3 object store and the `s3a://` filesystem client cannot meet these requirements.*\n\n1. Amazon S3 has inconsistent directory listings unless S3Guard is enabled.\n1. The S3A mimics `rename()` by copying files and then deleting the originals.\nThis can fail partway through, and there is nothing to prevent any other process\nin the cluster attempting a rename at the same time.\n\nAs a result,\n\n* Files my not be listed, hence not renamed into place.\n* Deleted files may still be discovered, confusing the rename process to the point\nof failure.\n* If a rename fails, the data is left in an unknown state.\n* If more than one process attempts to commit work simultaneously, the output\ndirectory may contain the results of both processes: it is no longer an exclusive\noperation.\n*. While S3Guard may deliver the listing consistency, commit time is still\nproportional to the amount of data created. It still can't handle task failure.\n\n**Using the \"classic\" `FileOutputCommmitter` to commit work to Amazon S3 risks\nloss or corruption of generated data**\n\n\nTo address these problems there is now explicit support in the `hadop-aws`\nmodule for committing work to Amazon S3 via the S3A filesystem client,\n*the S3A Committers*\n\n\nFor safe, as well as high-performance output of work to S3,\nwe need use \"a committer\" explicitly written to work with S3, treating it as\nan object store with special features.\n\n\n### Background : Hadoop's \"Commit Protocol\"\n\nHow exactly is work written to its final destination? That is accomplished by\na \"commit protocol\" between the workers and the job manager.\n\nThis protocol is implemented in Hadoop MapReduce, with a similar but extended\nversion in Apache Spark:\n\n1. A \"Job\" is the entire query, with inputs to outputs\n1. The \"Job Manager\" is the process in charge of choreographing the execution\nof the job. It may perform some of the actual computation too.\n1. The job has \"workers\", which are processes which work the actual data\nand write the results.\n1. Workers execute \"Tasks\", which are fractions of the job, a job whose\ninput has been *partitioned* into units of work which can be executed independently.\n1. The Job Manager directs workers to execute \"tasks\", usually trying to schedule\nthe work close to the data (if the filesystem provides locality information).\n1. Workers can fail: the Job manager needs to detect this and reschedule their active tasks.\n1. Workers can also become separated from the Job Manager, a \"network partition\".\nIt is (provably) impossible for the Job Manager to distinguish a running-but-unreachable\nworker from a failed one.\n1. The output of a failed task must not be visible; this is to avoid its\ndata getting into the final output.\n1. Multiple workers can be instructed to evaluate the same partition of the work;\nthis \"speculation\" delivers speedup as it can address the \"straggler problem\".\nWhen multiple workers are working on the same data, only one worker is allowed\nto write the final output.\n1. The entire job may fail (often from the failure of the Job Manager (MR Master, Spark Driver, ...)).\n1, The network may partition, with workers isolated from each other or\nthe process managing the entire commit.\n1. Restarted jobs may recover from a failure by reusing the output of all\ncompleted tasks (MapReduce with the \"v1\" algorithm), or just by rerunning everything\n(The \"v2\" algorithm and Spark).\n\n\nWhat is \"the commit protocol\" then? It is the requirements on workers as to\nwhen their data is made visible, where, for a filesystem, \"visible\" means \"can\nbe seen in the destination directory of the query.\"\n\n* There is a destination directory of work, \"the output directory.\"\n* The final output of tasks must be in this directory *or paths underneath it*.\n* The intermediate output of a task must not be visible in the destination directory.\nThat is: they must not write directly to the destination.\n* The final output of a task *may* be visible under the destination.\n* The Job Manager makes the decision as to whether a task's data is to be \"committed\",\nbe it directly to the final directory or to some intermediate store..\n* Individual workers communicate with the Job manager to manage the commit process:\nwhether the output is to be *committed* or *aborted*\n* When a worker commits the output of a task, it somehow promotes its intermediate work to becoming\nfinal.\n* When a worker aborts a task's output, that output must not become visible\n(i.e. it is not committed).\n* Jobs themselves may be committed/aborted (the nature of \"when\" is not covered here).\n* After a Job is committed, all its work must be visible.\n* And a file `_SUCCESS` may be written to the output directory.\n* After a Job is aborted, all its intermediate data is lost.\n* Jobs may also fail. When restarted, the successor job must be able to clean up\nall the intermediate and committed work of its predecessor(s).\n* Task and Job processes measure the intervals between communications with their\nApplication Master and YARN respectively.\nWhen the interval has grown too large they must conclude\nthat the network has partitioned and that they must abort their work.\n\n\nThat's \"essentially\" it. When working with HDFS and similar filesystems,\ndirectory `rename()` is the mechanism used to commit the work of tasks and\njobs.\n* Tasks write data to task attempt directories under the directory `_temporary`\nunderneath the final destination directory.\n* When a task is committed, these files are renamed to the destination directory\n(v2 algorithm) or a job attempt directory under `_temporary` (v1 algorithm).\n* When a job is committed, for the v2 algorithm the `_SUCCESS` file is created,\nand the `_temporary` deleted.\n* For the v1 algorithm, when a job is committed, all the tasks committed under\nthe job attempt directory will have their output renamed into the destination\ndirectory.\n* The v2 algorithm recovers from failure by deleting the destination directory\nand restarting the job.\n* The v1 algorithm recovers from failure by discovering all committed tasks\nwhose output is in the job attempt directory, *and only rerunning all uncommitted tasks*.\n\n\nNone of this algorithm works safely or swiftly when working with \"raw\" AWS S3 storage:\n* Directory listing can be inconsistent: the tasks and jobs may not list all work to\nbe committed.\n* Renames go from being fast, atomic operations to slow operations which can fail partway through.\n\nThis then is the problem which the S3A committers address:\n\n*How to safely and reliably commit work to Amazon S3 or compatible object store*\n\n\n## Meet the S3A Commmitters\n\nSince Hadoop 3.1, the S3A FileSystem has been accompanied by classes\ndesigned to integrate with the Hadoop and Spark job commit protocols, classes\nwhich interact with the S3A filesystem to reliably commit work work to S3:\n*The S3A Committers*\n\nThe underlying architecture of this process is very complex, and\ncovered in [the committer architecture documentation](./committer_architecture.html).\n\nThe key concept to know of is S3's \"Multipart Upload\" mechanism. This allows\nan S3 client to write data to S3 in multiple HTTP POST requests, only completing\nthe write operation with a final POST to complete the upload; this final POST\nconsisting of a short list of the etags of the uploaded blocks.\nThis multipart upload mechanism is already automatically used when writing large\namounts of data to S3; an implementation detail of the S3A output stream.\n\nThe S3A committers make explicit use of this multipart upload (\"MPU\") mechanism:\n\n1. The individual *tasks* in a job write their data to S3 as POST operations\nwithin multipart uploads, yet do not issue the final POST to complete the upload.\n1. The multipart uploads are committed in the job commit process.\n\nThere are two different S3A committer types, *staging*\nand *magic*. The committers primarily vary in how data is written during task execution,\nhow the pending commit information is passed to the job manager, and in how\nconflict with existing files is resolved.\n\n\n| feature | staging | magic |\n|--------|---------|---|\n| task output destination | local disk | S3A *without completing the write* |\n| task commit process | upload data from disk to S3 | list all pending uploads on s3 and write details to job attempt directory |\n| task abort process | delete local disk data | list all pending uploads and abort them |\n| job commit | list & complete pending uploads | list & complete pending uploads |\n\nThe other metric is \"maturity\". There, the fact that the Staging committers\nare based on Netflix's production code counts in its favor.\n\n\n### The Staging Committer\n\nThis is based on work from Netflix. It \"stages\" data into the local filesystem.\nIt also requires the cluster to have HDFS, so that\n\nTasks write to URLs with `file://` schemas. When a task is committed,\nits files are listed, uploaded to S3 as incompleted Multipart Uploads.\nThe information needed to complete the uploads is saved to HDFS where\nit is committed through the standard \"v1\" commit algorithm.\n\nWhen the Job is committed, the Job Manager reads the lists of pending writes from its\nHDFS Job destination directory and completes those uploads.\n\nCancelling a task is straightforward: the local directory is deleted with\nits staged data. Cancelling a job is achieved by reading in the lists of\npending writes from the HDFS job attempt directory, and aborting those\nuploads. For extra safety, all outstanding multipart writes to the destination directory\nare aborted.\n\nThe staging committer comes in two slightly different forms, with slightly\ndifferent conflict resolution policies:\n\n\n* **Directory**: the entire directory tree of data is written or overwritten,\nas normal.\n\n* **Partitioned**: special handling of partitioned directory trees of the form\n`YEAR=2017/MONTH=09/DAY=19`: conflict resolution is limited to the partitions\nbeing updated.\n\n\nThe Partitioned Committer is intended to allow jobs updating a partitioned\ndirectory tree to restrict the conflict resolution to only those partition\ndirectories containing new data. It is intended for use with Apache Spark\nonly.\n\n\n## Conflict Resolution in the Staging Committers\n\nThe Staging committers offer the ability to replace the conflict policy\nof the execution engine with policy designed to work with the tree of data.\nThis is based on the experience and needs of Netflix, where efficiently adding\nnew data to an existing partitioned directory tree is a common operation.\n\n```xml\n\n fs.s3a.committer.staging.conflict-mode\n fail\n \n Staging committer conflict resolution policy: {@value}.\n Supported: fail, append, replace.\n \n\n```\n\n**replace** : when the job is committed (and not before), delete files in\ndirectories into which new data will be written.\n\n**fail**: when there are existing files in the destination, fail the job.\n\n**append**: Add new data to the directories at the destination; overwriting\nany with the same name. Reliable use requires unique names for generated files,\nwhich the committers generate\nby default.\n\nThe difference between the two staging committers are as follows:\n\nThe Directory Committer uses the entire directory tree for conflict resolution.\nIf any file exists at the destination it will fail in job setup; if the resolution\nmechanism is \"replace\" then all existing files will be deleted.\n\nThe partitioned committer calculates the partitions into which files are added,\nthe final directories in the tree, and uses that in its conflict resolution\nprocess:\n\n\n**replace** : delete all data in the destination partition before committing\nthe new files.\n\n**fail**: fail if there is data in the destination partition, ignoring the state\nof any parallel partitions.\n\n**append**: add the new data.\n\nIt's intended for use in Apache Spark Dataset operations, rather\nthan Hadoop's original MapReduce engine, and only in jobs\nwhere adding new data to an existing dataset is the desired goal.\n\nPrerequisites for successful work\n\n1. The output is written into partitions via `PARTITIONED BY` or `partitionedBy()`\ninstructions.\n2. There is no data written directly to the root path (all files there are\nignored; it's implicitly \"append\").\n\nHere's an example in Spark, assuming that `sourceDataset` is a dataset\nwhose columns include \"year\" and \"month\":\n\n```scala\nsourceDataset\n .write\n .partitionBy(\"year\", \"month\")\n .mode(SaveMode.Append)\n .opt(\"fs.s3a.committer.name\", \"partitioned\")\n .opt(\"fs.s3a.committer.staging.conflict-mode\", \"replace\")\n .format(\"orc\")\n .save(\"s3a://examples/statistics\")\n```\n\n\n### The Magic Committer\n\nThe \"Magic\" committer does its work through \"magic\" in the filesystem:\nattempts to write to specific \"magic\" paths are interpreted as writes\nto a parent directory *which are not to be completed*. When the output stream\nis closed, the information needed to complete the write is saved in the magic\ndirectory. The task committer saves the list of these to a directory for the\njob committers use, or, if aborting, lists the pending writes and aborts them.\n\nThe job committer reads in the list of pending commits, and commits them as\nthe Staging Committer does.\n\nCompared to the Staging Committer, the Magic Committer offers faster write\ntimes: output is uploaded to S3 as it is written, rather than in the\ntask commit.\n\nHowever, it has extra requirements of the filesystem\n\n1. It requires a consistent object store, which for Amazon S3,\nmeans that [S3Guard](./s3guard.html) must be enabled. For third-party stores,\nconsult the documentation.\n1. The S3A client must be configured to recognize interactions\nwith the magic directories and treat them specially.\n\n\nIt's also not been field tested to the extent of Netflix's committer; consider\nit the least mature of the committers.\n\n\n#### Which Committer to Use?\n\n1. If you want to create or update existing partitioned data trees in Spark, use thee\nPartitioned Committer. Make sure you have enough hard disk capacity for all staged data.\nDo not use it in other situations.\n\n1. If you know that your object store is consistent, or that the processes\nwriting data use S3Guard, use the Magic Committer for higher performance\nwriting of large amounts of data.\n\n1. Otherwise: use the directory committer, making sure you have enough\nhard disk capacity for all staged data.\n\nPut differently: start with the Directory Committer.\n\n## Switching to an S3A Committer\n\nTo use an S3A committer, the property `mapreduce.outputcommitter.factory.scheme.s3a`\nmust be set to the S3A committer factory, `org.apache.hadoop.fs.s3a.commit.staging.S3ACommitterFactory`.\nThis is done in `mapred-default.xml`\n\n```xml\n\n mapreduce.outputcommitter.factory.scheme.s3a\n org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory\n \n The committer factory to use when writing data to S3A filesystems.\n \n\n```\n\nWhat is missing is an explicit choice of committer to use in the property\n`fs.s3a.committer.name`; so the classic (and unsafe) file committer is used.\n\n| `fs.s3a.committer.name` | Committer |\n|--------|---------|\n| `directory` | directory staging committer |\n| `partitioned` | partition staging committer (for use in Spark only) |\n| `magic` | the \"magic\" committer |\n| `file` | the original and unsafe File committer; (default) |\n\n\n\n## Using the Directory and Partitioned Staging Committers\n\nGenerated files are initially written to a local directory underneath one of the temporary\ndirectories listed in `fs.s3a.buffer.dir`.\n\n\nThe staging committer needs a path in the cluster filesystem\n(e.g. HDFS). This must be declared in `fs.s3a.committer.staging.tmp.path`.\n\nTemporary files are saved in HDFS (or other cluster filesystem) under the path\n`${fs.s3a.committer.staging.tmp.path}/${user}` where `user` is the name of the user running the job.\nThe default value of `fs.s3a.committer.staging.tmp.path` is `tmp/staging`,\nWhich will be converted at run time to a path under the current user's home directory,\nessentially `~/tmp/staging`\n so the temporary directory\n\nThe application attempt ID is used to create a unique path under this directory,\nresulting in a path `~/tmp/staging/${user}/${application-attempt-id}/` under which\nsummary data of each task's pending commits are managed using the standard\n`FileOutputFormat` committer.\n\nWhen a task is committed the data is uploaded under the destination directory.\nThe policy of how to react if the destination exists is defined by\nthe `fs.s3a.committer.staging.conflict-mode` setting.\n\n| `fs.s3a.committer.staging.conflict-mode` | Meaning |\n| -----------------------------------------|---------|\n| `fail` | Fail if the destination directory exists |\n| `replace` | Delete all existing files before committing the new data |\n| `append` | Add the new files to the existing directory tree |\n\n\n## The \"Partitioned\" Staging Committer\n\nThis committer an extension of the \"Directory\" committer which has a special conflict resolution\npolicy designed to support operations which insert new data into a directory tree structured\nusing Hive's partitioning strategy: different levels of the tree represent different columns.\n\nFor example, log data could be partitioned by `YEAR` and then by `MONTH`, with different\nentries underneath.\n\n```\nlogs/YEAR=2017/MONTH=01/\n log-20170101.avro\n log-20170102.avro\n ...\n log-20170131.avro\n\nlogs/YEAR=2017/MONTH=02/\n log-20170201.avro\n log-20170202.avro\n ...\n log-20170227.avro\n\nlogs/YEAR=2017/MONTH=03/\nlogs/YEAR=2017/MONTH=04/\n```\n\nA partitioned structure like this allows for queries using Hive or Spark to filter out\nfiles which do not contain relevant data.\n\nWhat the partitioned committer does is, where the tooling permits, allows callers\nto add data to an existing partitioned layout*.\n\nMore specifically, it does this by having a conflict resolution options which\nonly act on individual partitions, rather than across the entire output tree.\n\n| `fs.s3a.committer.staging.conflict-mode` | Meaning |\n| -----------------------------------------|---------|\n| `fail` | Fail if the destination partition(s) exist |\n| `replace` | Delete the existing data partitions before committing the new data |\n| `append` | Add the new data to the existing partitions |\n\n\nAs an example, if a job was writing the file\n`logs/YEAR=2017/MONTH=02/log-20170228.avro`, then with a policy of `fail`,\nthe job would fail. With a policy of `replace`, then entire directory\n`logs/YEAR=2017/MONTH=02/` would be deleted before the new file `log-20170228.avro`\nwas written. With the policy of `append`, the new file would be added to\nthe existing set of files.\n\n\n### Notes\n\n1. A deep partition tree can itself be a performance problem in S3 and the s3a client,\nor, more specifically. a problem with applications which use recursive directory tree\nwalks to work with data.\n\n1. The outcome if you have more than one job trying simultaneously to write data\nto the same destination with any policy other than \"append\" is undefined.\n\n1. In the `append` operation, there is no check for conflict with file names.\nIf, in the example above, the file `log-20170228.avro` already existed,\nit would be overridden. Set `fs.s3a.committer.staging.unique-filenames` to `true`\nto ensure that a UUID is included in every filename to avoid this.\n\n\n## Using the Magic committer\n\nThis is less mature than the Staging Committer, but promises higher\nperformance.\n\n### FileSystem client setup\n\n1. Use a *consistent* S3 object store. For Amazon S3, this means enabling\n[S3Guard](./s3guard.html). For S3-compatible filesystems, consult the filesystem\ndocumentation to see if it is consistent, hence compatible \"out of the box\".\n1. Turn the magic on by `fs.s3a.committer.magic.enabled\"`\n\n```xml\n\n fs.s3a.committer.magic.enabled\n \n Enable support in the filesystem for the S3 \"Magic\" committer.\n \n true\n\n```\n\n*Do not use the Magic Committer on an inconsistent S3 object store. For\nAmazon S3, that means S3Guard must *always* be enabled.\n\n\n### Enabling the committer\n\n```xml\n\n fs.s3a.committer.name\n magic\n\n\n```\n\nConflict management is left to the execution engine itself.\n\n## Committer Configuration Options\n\n\n| Option | Magic | Directory | Partitioned | Meaning | Default |\n|--------|-------|-----------|-------------|---------|---------|\n| `mapreduce.fileoutputcommitter.marksuccessfuljobs` | X | X | X | Write a `_SUCCESS` file at the end of each job | `true` |\n| `fs.s3a.committer.threads` | X | X | X | Number of threads in committers for parallel operations on files. | 8 |\n| `fs.s3a.committer.staging.conflict-mode` | | X | X | Conflict resolution: `fail`, `abort` or `overwrite`| `fail` |\n| `fs.s3a.committer.staging.unique-filenames` | | X | X | Generate unique filenames | `true` |\n\n| `fs.s3a.committer.magic.enabled` | X | | | Enable \"magic committer\" support in the filesystem | `false` |\n\n\n\n\n| Option | Magic | Directory | Partitioned | Meaning | Default |\n|--------|-------|-----------|-------------|---------|---------|\n| `fs.s3a.buffer.dir` | X | X | X | Local filesystem directory for data being written and/or staged. | |\n| `fs.s3a.committer.staging.tmp.path` | | X | X | Path in the cluster filesystem for temporary data | `tmp/staging` |\n\n\n```xml\n\n fs.s3a.committer.name\n file\n \n Committer to create for output to S3A, one of:\n \"file\", \"directory\", \"partitioned\", \"magic\".\n \n\n\n\n fs.s3a.committer.magic.enabled\n false\n \n Enable support in the filesystem for the S3 \"Magic\" committer.\n When working with AWS S3, S3Guard must be enabled for the destination\n bucket, as consistent metadata listings are required.\n \n\n\n\n fs.s3a.committer.threads\n 8\n \n Number of threads in committers for parallel operations on files\n (upload, commit, abort, delete...)\n \n\n\n\n fs.s3a.committer.staging.tmp.path\n tmp/staging\n \n Path in the cluster filesystem for temporary data.\n This is for HDFS, not the local filesystem.\n It is only for the summary data of each file, not the actual\n data being committed.\n Using an unqualified path guarantees that the full path will be\n generated relative to the home directory of the user creating the job,\n hence private (assuming home directory permissions are secure).\n \n\n\n\n fs.s3a.committer.staging.unique-filenames\n true\n \n Option for final files to have a unique name through job attempt info,\n or the value of fs.s3a.committer.staging.uuid\n When writing data with the \"append\" conflict option, this guarantees\n that new data will not overwrite any existing data.\n \n\n\n\n fs.s3a.committer.staging.conflict-mode\n fail\n \n Staging committer conflict resolution policy.\n Supported: \"fail\", \"append\", \"replace\".\n \n\n\n\n s.s3a.committer.staging.abort.pending.uploads\n true\n \n Should the staging committers abort all pending uploads to the destination\n directory?\n\n Changing this if more than one partitioned committer is\n writing to the same destination tree simultaneously; otherwise\n the first job to complete will cancel all outstanding uploads from the\n others. However, it may lead to leaked outstanding uploads from failed\n tasks. If disabled, configure the bucket lifecycle to remove uploads\n after a time period, and/or set up a workflow to explicitly delete\n entries. Otherwise there is a risk that uncommitted uploads may run up\n bills.\n \n\n\n\n mapreduce.outputcommitter.factory.scheme.s3a\n org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory\n \n The committer factory to use when writing data to S3A filesystems.\n If mapreduce.outputcommitter.factory.class is set, it will\n override this property.\n\n (This property is set in mapred-default.xml)\n \n\n\n```\n\n\n## Troubleshooting\n\n### `Filesystem does not have support for 'magic' committer`\n\n```\norg.apache.hadoop.fs.s3a.commit.PathCommitException: `s3a://landsat-pds': Filesystem does not have support for 'magic' committer enabled\nin configuration option fs.s3a.committer.magic.enabled\n```\n\nThe Job is configured to use the magic committer, but the S3A bucket has not been explicitly\ndeclared as supporting it.\n\nThe destination bucket **must** be declared as supporting the magic committer.\n\nThis can be done for those buckets which are known to be consistent, either\nbecause [S3Guard](s3guard.html) is used to provide consistency,\nor because the S3-compatible filesystem is known to be strongly consistent.\n\n```xml\n\n fs.s3a.bucket.landsat-pds.committer.magic.enabled\n true\n\n```\n\n*IMPORTANT*: only enable the magic committer against object stores which\noffer consistent listings. By default, Amazon S3 does not do this -which is\nwhy the option `fs.s3a.committer.magic.enabled` is disabled by default.\n\n\nTip: you can verify that a bucket supports the magic committer through the\n`hadoop s3guard bucket-info` command:\n\n\n```\n> hadoop s3guard bucket-info -magic s3a://landsat-pds/\n\nFilesystem s3a://landsat-pds\nLocation: us-west-2\nFilesystem s3a://landsat-pds is not using S3Guard\nThe \"magic\" committer is not supported\n\nS3A Client\n Endpoint: fs.s3a.endpoint=(unset)\n Encryption: fs.s3a.server-side-encryption-algorithm=none\n Input seek policy: fs.s3a.experimental.input.fadvise=normal\n2017-09-27 19:18:57,917 INFO util.ExitUtil: Exiting with status 46: 46: The magic committer is not enabled for s3a://landsat-pds\n```\n\n## Error message: \"File being created has a magic path, but the filesystem has magic file support disabled:\n\nA file is being written to a path which is used for \"magic\" files,\nfiles which are actually written to a different destination than their stated path\n*but the filesystem doesn't support \"magic\" files*\n\nThis message should not appear through the committer itself —it will\nfail with the error message in the previous section, but may arise\nif other applications are attempting to create files under the path `/__magic/`.\n\nMake sure the filesystem meets the requirements of the magic committer\n(a consistent S3A filesystem through S3Guard or the S3 service itself),\nand set the `fs.s3a.committer.magic.enabled` flag to indicate that magic file\nwrites are supported.\n\n\n### `FileOutputCommitter` appears to be still used (from logs or delays in commits)\n\nThe Staging committers use the original `FileOutputCommitter` to manage\nthe propagation of commit information: do not worry if it the logs show `FileOutputCommitter`\nwork with data in the cluster filesystem (e.g. HDFS).\n\nOne way to make sure that the `FileOutputCommitter` is not being used to write\nthe data to S3 is to set the option `mapreduce.fileoutputcommitter.algorithm.version`\nto a value such as \"10\". Because the only supported algorithms are \"1\" and \"2\",\nany erroneously created `FileOutputCommitter` will raise an exception in its constructor\nwhen instantiated:\n\n```\njava.io.IOException: Only 1 or 2 algorithm version is supported\nat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:130)\nat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:104)\nat org.apache.parquet.hadoop.ParquetOutputCommitter.(ParquetOutputCommitter.java:42)\nat org.apache.parquet.hadoop.ParquetOutputFormat.getOutputCommitter(ParquetOutputFormat.java:395)\nat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupCommitter(HadoopMapReduceCommitProtocol.scala:67)\nat com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol.setupCommitter(PathOutputCommitProtocol.scala:62)\nat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:124)\nat com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol.setupJob(PathOutputCommitProtocol.scala:152)\nat org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:175)\nat org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:145)\n```\n\nWhile that will not make the problem go away, it will at least make\nthe failure happen at the start of a job.\n\n(Setting this option will not interfere with the Staging Committers' use of HDFS,\nas it explicitly sets the algorithm to \"2\" for that part of its work).\n\nThe other way to check which committer to use is to examine the `_SUCCESS` file.\nIf it is 0-bytes long, the classic `FileOutputCommitter` committed the job.\nThe S3A committers all write a non-empty JSON file; the `committer` field lists\nthe committer used.\n\n\n*Common causes*\n\n1. The property `fs.s3a.committer.name` is set to \"file\". Fix: change.\n1. The job has overridden the property `mapreduce.outputcommitter.factory.class`\nwith a new factory class for all committers. This takes priority over\nall committers registered for the s3a:// schema.\n1. The property `mapreduce.outputcommitter.factory.scheme.s3a` is unset.\n1. The output format has overridden `FileOutputFormat.getOutputCommitter()`\nand is returning its own committer -one which is a subclass of `FileOutputCommitter`.\n\nThat final cause. *the output format is returning its own committer*, is not\neasily fixed; it may be that the custom committer performs critical work\nduring its lifecycle, and contains assumptions about the state of the written\ndata during task and job commit (i.e. it is in the destination filesystem).\nConsult with the authors/maintainers of the output format\nto see whether it would be possible to integrate with the new committer factory\nmechanism and object-store-specific commit algorithms.\n\nParquet is a special case here: its committer does no extra work\nother than add the option to read all newly-created files then write a schema\nsummary. The Spark integration has explicit handling for Parquet to enable it\nto support the new committers, removing this (slow on S3) option.\n\nIf you have subclassed `FileOutputCommitter` and want to move to the\nfactory model, please get in touch.\n\n\n## Job/Task fails with PathExistsException: Destination path exists and committer conflict resolution mode is \"fail\"\n\nThis surfaces when either of two conditions are met.\n\n1. The Directory committer is used with `fs.s3a.committer.staging.conflict-mode` set to\n`fail` and the output/destination directory exists.\nThe job will fail in the driver during job setup.\n1. The Partitioned Committer is used with `fs.s3a.committer.staging.conflict-mode` set to\n`fail` and one of the partitions. The specific task(s) generating conflicting data will fail\nduring task commit, which will cause the entire job to fail.\n\nIf you are trying to write data and want write conflicts to be rejected, this is the correct\nbehavior: there was data at the destination so the job was aborted.\n\n## Staging committer task fails with IOException: No space left on device\n\nThere's not enough space on the local hard disk (real or virtual)\nto store all the uncommitted data of the active tasks on that host.\nBecause the staging committers write all output to the local disk\nand only upload the data on task commits, enough local temporary\nstorage is needed to store all output generated by all uncommitted\ntasks running on the single host. Small EC2 VMs may run out of disk.\n\n1. Make sure that `fs.s3a.buffer.dir` includes a temporary directory on\nevery available hard disk; this spreads load better.\n\n1. Add more disk space. In EC2: request instances with more local storage.\nThere is no need for EMR storage; this is just for temporary data.\n\n1. Purge the directories listed in `fs.s3a.buffer.dir` of old data.\nFailed tasks may not clean up all old files.\n\n1. Reduce the number of worker threads/process in the host.\n\n1. Consider partitioning the job into more tasks. This *may* result in more tasks\ngenerating less data each.\n\n1. Use the magic committer. This only needs enough disk storage to buffer\nblocks of the currently being written file during their upload process,\nso can use a lot less disk space.\n","isInternal":false,"tokens":8031,"sizeBytes":34671},{"name":"encryption.md","path":"hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/encryption.md","rawUrl":"https://raw.githubusercontent.com/apache/hadoop/HEAD/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/encryption.md","title":"Hadoop-aws Skill","category":"anthropic-skill","format":"markdown","content":"\n\n\n# Working with Encrypted S3 Data\n\n\n\n\n## Introduction\n\nThe S3A filesystem client supports Amazon S3's Server Side Encryption\nfor at-rest data encryption.\nYou should to read up on the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html)\nfor S3 Server Side Encryption for up to date information on the encryption mechanisms.\n\n\n\nWhen configuring an encryption method in the `core-site.xml`, this will apply cluster wide.\nAny new file written will be encrypted with this encryption configuration.\nWhen the S3A client reads a file, S3 will attempt to decrypt it using the mechanism\nand keys with which the file was encrypted.\n\n* It is **NOT** advised to mix and match encryption types in a bucket\n* It is much simpler and safer to encrypt with just one type and key per bucket.\n* You can use AWS bucket policies to mandate encryption rules for a bucket.\n* You can use S3A per-bucket configuration to ensure that S3A clients use encryption\npolicies consistent with the mandated rules.\n* You can use S3 Default Encryption to encrypt data without needing to\nset anything in the client.\n* Changing the encryption options on the client does not change how existing\nfiles were encrypted, except when the files are renamed.\n* For all mechanisms other than SSE-C, clients do not need any configuration\noptions set in order to read encrypted data: it is all automatically handled\nin S3 itself.\n\n## How data is encrypted\n\nAWS S3 supports server-side encryption inside the storage system itself.\nWhen an S3 client uploading data requests data to be encrypted, then an encryption key is used\nto encrypt the data as it saved to S3. It remains encrypted on S3 until deleted:\nclients cannot change the encryption attributes of an object once uploaded.\n\nThe Amazon AWS SDK also offers client-side encryption, in which all the encoding\nand decoding of data is performed on the client. This is *not* supported by\nthe S3A client.\n\nThe server-side \"SSE\" encryption is performed with symmetric AES256 encryption;\nS3 offers different mechanisms for actually defining the key to use.\n\n\nThere are four key management mechanisms, which in order of simplicity of use,\nare:\n\n* S3 Default Encryption\n* SSE-S3: an AES256 key is generated in S3, and saved alongside the data.\n* SSE-KMS: an AES256 key is generated in S3, and encrypted with a secret key provided\nby Amazon's Key Management Service, a key referenced by name in the uploading client.\n* SSE-C : the client specifies an actual base64 encoded AES-256 key to be used\nto encrypt and decrypt the data.\n\n\n## S3 Default Encryption\n\nThis feature allows the administrators of the AWS account to set the \"default\"\nencryption policy on a bucket -the encryption to use if the client does\nnot explicitly declare an encryption algorithm.\n\n[S3 Default Encryption for S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html)\n\nThis supports SSE-S3 and SSE-KMS.\n\nThere is no need to set anything up in the client: do it in the AWS console.\n\n\n## SSE-S3 Amazon S3-Managed Encryption Keys\n\nIn SSE-S3, all keys and secrets are managed inside S3. This is the simplest encryption mechanism.\nThere is no extra cost for storing data with this option.\n\n\n### Enabling SSE-S3\n\nTo write S3-SSE encrypted files, the value of\n`fs.s3a.server-side-encryption-algorithm` must be set to that of\nthe encryption mechanism used in `core-site`; currently only `AES256` is supported.\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n AES256\n\n```\n\nOnce set, all new data will be stored encrypted. There is no need to set this property when downloading data — the data will be automatically decrypted when read using\nthe Amazon S3-managed key.\n\nTo learn more, refer to\n[Protecting Data Using Server-Side Encryption with Amazon S3-Managed Encryption Keys (SSE-S3) in AWS documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html).\n\n\n### SSE-KMS: Amazon S3-KMS Managed Encryption Keys\n\n\nAmazon offers a pay-per-use key management service, [AWS KMS](https://aws.amazon.com/documentation/kms/).\nThis service can be used to encrypt data on S3 by defining \"customer master keys\", CMKs,\nwhich can be centrally managed and assigned to specific roles and IAM accounts.\n\nThe AWS KMS [can be used encrypt data on S3uploaded data](http://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html).\n\n> The AWS KMS service is **not** related to the Key Management Service built into Hadoop (*Hadoop KMS*). The *Hadoop KMS* primarily focuses on\n managing keys for *HDFS Transparent Encryption*. Similarly, HDFS encryption is unrelated to S3 data encryption.\n\nWhen uploading data encrypted with SSE-KMS, the sequence is as follows.\n\n1. The S3A client must declare a specific CMK in the property `fs.s3a.server-side-encryption.key`, or leave\nit blank to use the default configured for that region.\n\n1. The S3A client uploads all the data as normal, now including encryption information.\n\n1. The S3 service encrypts the data with a symmetric key unique to the new object.\n\n1. The S3 service retrieves the chosen CMK key from the KMS service, and, if the user has\nthe right to use it, uses it to encrypt the object-specific key.\n\n\nWhen downloading SSE-KMS encrypted data, the sequence is as follows\n\n1. The S3A client issues an HTTP GET request to read the data.\n1. S3 sees that the data was encrypted with SSE-KMS, and looks up the specific key in the KMS service\n1. If and only if the requesting user has been granted permission to use the CMS key does\nthe KMS service provide S3 with the key.\n1. As a result, S3 will only decode the data if the user has been granted access to the key.\n\n\nKMS keys can be managed by an organization's administrators in AWS, including\nhaving access permissions assigned and removed from specific users, groups, and IAM roles.\nOnly those \"principals\" with granted rights to a key may access it,\nhence only they may encrypt data with the key, *and decrypt data encrypted with it*.\nThis allows KMS to be used to provide a cryptographically secure access control mechanism for data stores on S3.\n\n\nEach KMS server is region specific, and accordingly, so is each CMK configured.\nA CMK defined in one region cannot be used with an S3 bucket in a different region.\n\n\nNotes\n\n* Callers are charged for every use of a key, both for encrypting the data in uploads\n and for decrypting it when reading it back.\n* Random-access IO on files may result in multiple GET requests of an object during a read\nsequence (especially for columnar data), so may require more than one key retrieval to process a single file,\n* The KMS service is throttled: too many requests may cause requests to fail.\n* As well as incurring charges, heavy I/O *may* reach IO limits for a customer. If those limits are reached,\nthey can be increased through the AWS console.\n\n\n### Enabling SSE-KMS\n\nTo enable SSE-KMS, the property `fs.s3a.server-side-encryption-algorithm` must be set to `SSE-KMS` in `core-site`:\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n SSE-KMS\n\n```\n\nThe ID of the specific key used to encrypt the data should also be set in the property `fs.s3a.server-side-encryption.key`:\n\n```xml\n\n fs.s3a.server-side-encryption.key\n arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n```\n\nOrganizations may define a default key in the Amazon KMS; if a default key is set,\nthen it will be used whenever SSE-KMS encryption is chosen and the value of `fs.s3a.server-side-encryption.key` is empty.\n\n### the S3A `fs.s3a.encryption.key` key only affects created files\n\nWith SSE-KMS, the S3A client option `fs.s3a.server-side-encryption.key` sets the\nkey to be used when new files are created. When reading files, this key,\nand indeed the value of `fs.s3a.server-side-encryption-algorithme` is ignored:\nS3 will attempt to retrieve the key and decrypt the file based on the create-time settings.\n\nThis means that\n\n* There's no need to configure any client simply reading data.\n* It is possible for a client to read data encrypted with one KMS key, and\nwrite it with another.\n\n\n## SSE-C: Server side encryption with a client-supplied key.\n\nIn SSE-C, the client supplies the secret key needed to read and write data.\nEvery client trying to read or write data must be configured with the same\nsecret key.\n\n\nSSE-C integration with Hadoop is still stabilizing; issues related to it are still surfacing.\nIt is already clear that SSE-C with a common key must be used exclusively within\na bucket if it is to be used at all. This is the only way to ensure that path and\ndirectory listings do not fail with \"Bad Request\" errors.\n\n### Enabling SSE-C\n\nTo use SSE-C, the configuration option `fs.s3a.server-side-encryption-algorithm`\nmust be set to `SSE-C`, and a base-64 encoding of the key placed in\n`fs.s3a.server-side-encryption.key`.\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n SSE-C\n\n\n\n fs.s3a.server-side-encryption.key\n SGVscCwgSSdtIHRyYXBwZWQgaW5zaWRlIGEgYmFzZS02NC1jb2RlYyE=\n\n```\n\nAll clients must share this same key.\n\n### The `fs.s3a.encryption.key` value is used to read and write data\n\nWith SSE-C, the S3A client option `fs.s3a.server-side-encryption.key` sets the\nkey to be used for both reading *and* writing data.\n\nWhen reading any file written with SSE-C, the same key must be set\nin the property `fs.s3a.server-side-encryption.key`.\n\nThis is unlike SSE-S3 and SSE-KMS, where the information needed to\ndecode data is kept in AWS infrastructure.\n\n\n### SSE-C Warning\n\nYou need to fully understand how SSE-C works in the S3\nenvironment before using this encryption type. Please refer to the Server Side\nEncryption documentation available from AWS. SSE-C is only recommended for\nadvanced users with advanced encryption use cases. Failure to properly manage\nencryption keys can cause data loss. Currently, the AWS S3 API(and thus S3A)\nonly supports one encryption key and cannot support decrypting objects during\nmoves under a previous key to a new destination. It is **NOT** advised to use\nmultiple encryption keys in a bucket, and is recommended to use one key per\nbucket and to not change this key. This is due to when a request is made to S3,\nthe actual encryption key must be provided to decrypt the object and access the\nmetadata. Since only one encryption key can be provided at a time, S3A will not\npass the correct encryption key to decrypt the data.\n\n\n## Encryption best practises\n\n\n### Mandate encryption through policies\n\nBecause it is up to the clients to enable encryption on new objects, all clients\nmust be correctly configured in order to guarantee that data is encrypted.\n\n\nTo mandate that all data uploaded to a bucket is encrypted,\nyou can set a [bucket policy](https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/)\ndeclaring that clients must provide encryption information with all data uploaded.\n\n\n* Mandating an encryption mechanism on newly uploaded data does not encrypt existing data; existing data will retain whatever encryption (if any) applied at the time of creation*\n\nHere is a policy to mandate `SSE-S3/AES265` encryption on all data uploaded to a bucket. This covers uploads as well as the copy operations which take place when file/directory rename operations are mimicked.\n\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Id\": \"EncryptionPolicy\",\n \"Statement\": [\n {\n \"Sid\": \"RequireEncryptionHeaderOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"Null\": {\n \"s3:x-amz-server-side-encryption\": true\n }\n }\n },\n {\n \"Sid\": \"RequireAESEncryptionOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"StringNotEquals\": {\n \"s3:x-amz-server-side-encryption\": \"AES256\"\n }\n }\n }\n ]\n}\n```\n\nTo use SSE-KMS, a different restriction must be defined:\n\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Id\": \"EncryptionPolicy\",\n \"Statement\": [\n {\n \"Sid\": \"RequireEncryptionHeaderOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"Null\": {\n \"s3:x-amz-server-side-encryption\": true\n }\n }\n },\n {\n \"Sid\": \"RequireKMSEncryptionOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"StringNotEquals\": {\n \"s3:x-amz-server-side-encryption\": \"SSE-KMS\"\n }\n }\n }\n ]\n}\n```\n\nTo use one of these policies:\n\n1. Replace `BUCKET` with the specific name of the bucket being secured.\n1. Locate the bucket in the AWS console [S3 section](https://console.aws.amazon.com/s3/home).\n1. Select the \"Permissions\" tab.\n1. Select the \"Bucket Policy\" tab in the permissions section.\n1. Paste the edited policy into the form.\n1. Save the policy.\n\n### Use S3a per-bucket configuration to control encryption settings\n\nIn an organisation which has embraced S3 encryption, different buckets inevitably have\ndifferent encryption policies, such as different keys for SSE-KMS encryption.\nIn particular, as different keys need to be named for different regions, unless\nyou rely on the administrator-managed \"default\" key for each S3 region, you\nwill need unique keys.\n\nS3A's per-bucket configuration enables this.\n\n\nHere, for example, are settings for a bucket in London, `london-stats`:\n\n\n```xml\n\n fs.s3a.bucket.london-stats.server-side-encryption-algorithm\n AES256\n\n```\n\nThis requests SSE-S; if matched with a bucket policy then all data will\nbe encrypted as it is uploaded.\n\n\nA different bucket can use a different policy\n(here SSE-KMS) and, when necessary, declare a key.\n\nHere is an example bucket in S3 Ireland, which uses SSE-KMS and\na KMS key hosted in the AWS-KMS service in the same region.\n\n\n```xml\n\n fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm\n SSE-KMS\n\n\n\n fs.s3a.bucket.ireland-dev.server-side-encryption.key\n arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n\n```\n\nAgain the appropriate bucket policy can be used to guarantee that all callers\nwill use SSE-KMS; they can even mandate the name of the key used to encrypt\nthe data, so guaranteeing that access to thee data can be read by everyone\ngranted access to that key, and nobody without access to it.\n\n\n### Use rename() to encrypt files with new keys\n\nThe encryption of an object is set when it is uploaded. If you want to encrypt\nan unencrypted file, or change the SEE-KMS key of a file, the only way to do\nso is by copying the object.\n\nHow can you do that from Hadoop? With `rename()`.\n\nThe S3A client mimics a real filesystem's' rename operation by copying all the\nsource files to the destination paths, then deleting the old ones.\n\nNote: this does not work for SSE-C, because you cannot set a different key\nfor reading as for writing, and you must supply that key for reading. There\nyou need to copy one bucket to a different bucket, one with a different key.\nUse `distCp`for this, with per-bucket encryption policies.\n\n\n## Troubleshooting Encryption\n\nThe [troubleshooting](./troubleshooting_s3a.html) document covers\nstack traces which may surface when working with encrypted data.\n","isInternal":false,"tokens":4136,"sizeBytes":16807},{"name":"performance.md","path":"hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md","rawUrl":"https://raw.githubusercontent.com/apache/hadoop/HEAD/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/performance.md","title":"Hadoop-aws Skill","category":"anthropic-skill","format":"markdown","content":"\n\n# Maximizing Performance when working with the S3A Connector\n\n\n\n\n## Introduction\n\nS3 is slower to work with than HDFS, even on virtual clusters running on\nAmazon EC2.\n\nThat's because its a very different system, as you can see:\n\n\n| Feature | HDFS | S3 through the S3A connector |\n|---------|------|------------------------------|\n| communication | RPC | HTTP GET/PUT/HEAD/LIST/COPY requests |\n| data locality | local storage | remote S3 servers |\n| replication | multiple datanodes | asynchronous after upload |\n| consistency | consistent data and listings | eventual consistent for listings, deletes and updates |\n| bandwidth | best: local IO, worst: datacenter network | bandwidth between servers and S3 |\n| latency | low | high, especially for \"low cost\" directory operations |\n| rename | fast, atomic | slow faked rename through COPY & DELETE|\n| delete | fast, atomic | fast for a file, slow & non-atomic for directories |\n| writing| incremental | in blocks; not visible until the writer is closed |\n| reading | seek() is fast | seek() is slow and expensive |\n| IOPs | limited only by hardware | callers are throttled to shards in an s3 bucket |\n| Security | Posix user+group; ACLs | AWS Roles and policies |\n\nFrom a performance perspective, key points to remember are:\n\n* S3 throttles bucket access across all callers: adding workers can make things worse.\n* EC2 VMs have network IO throttled based on the VM type.\n* Directory rename and copy operations take *much* longer the more objects and data there is.\nThe slow performance of `rename()` surfaces during the commit phase of jobs,\napplications like `DistCP`, and elsewhere.\n* seek() calls when reading a file can force new HTTP requests.\nThis can make reading columnar Parquet/ORC data expensive.\n\nOverall, although the S3A connector makes S3 look like a file system,\nit isn't, and some attempts to preserve the metaphor are \"aggressively suboptimal\".\n\nTo make most efficient use of S3, care is needed.\n\n## Speeding up directory listing operations through S3Guard\n\n[S3Guard](s3guard.html) provides significant speedups for operations which\nlist files a lot. This includes the setup of all queries against data:\nMapReduce, Hive and Spark, as well as DistCP.\n\n\nExperiment with using it to see what speedup it delivers.\n\n\n## Improving data input performance through fadvise\n\nThe S3A Filesystem client supports the notion of input policies, similar\nto that of the Posix `fadvise()` API call. This tunes the behavior of the S3A\nclient to optimise HTTP GET requests for the different use cases.\n\n### fadvise `sequential`\n\nRead through the file, possibly with some short forward seeks.\n\nThe whole document is requested in a single HTTP request; forward seeks\nwithin the readahead range are supported by skipping over the intermediate\ndata.\n\nThis delivers maximum sequential throughput —but with very expensive\nbackward seeks.\n\nApplications reading a file in bulk (DistCP, any copy operations) should use\nsequential access, as should those reading data from gzipped `.gz` files.\nBecause the \"normal\" fadvise policy starts off in sequential IO mode,\nthere is rarely any need to explicit request this policy.\n\n### fadvise `random`\n\nOptimised for random IO, specifically the Hadoop `PositionedReadable`\noperations —though `seek(offset); read(byte_buffer)` also benefits.\n\nRather than ask for the whole file, the range of the HTTP request is\nset to that that of the length of data desired in the `read` operation\n(Rounded up to the readahead value set in `setReadahead()` if necessary).\n\nBy reducing the cost of closing existing HTTP requests, this is\nhighly efficient for file IO accessing a binary file\nthrough a series of `PositionedReadable.read()` and `PositionedReadable.readFully()`\ncalls. Sequential reading of a file is expensive, as now many HTTP requests must\nbe made to read through the file: there's a delay between each GET operation.\n\n\nRandom IO is best for IO with seek-heavy characteristics:\n\n* Data is read using the `PositionedReadable` API.\n* Long distance (many MB) forward seeks\n* Backward seeks as likely as forward seeks.\n* Little or no use of single character `read()` calls or small `read(buffer)`\ncalls.\n* Applications running close to the S3 data store. That is: in EC2 VMs in\nthe same datacenter as the S3 instance.\n\nThe desired fadvise policy must be set in the configuration option\n`fs.s3a.experimental.input.fadvise` when the filesystem instance is created.\nThat is: it can only be set on a per-filesystem basis, not on a per-file-read\nbasis.\n\n```xml\n\n fs.s3a.experimental.input.fadvise\n random\n \n Policy for reading files.\n Values: 'random', 'sequential' or 'normal'\n \n\n```\n\n[HDFS-2744](https://issues.apache.org/jira/browse/HDFS-2744),\n*Extend FSDataInputStream to allow fadvise* proposes adding a public API\nto set fadvise policies on input streams. Once implemented,\nthis will become the supported mechanism used for configuring the input IO policy.\n\n### fadvise `normal` (default)\n\nThe `normal` policy starts off reading a file in `sequential` mode,\nbut if the caller seeks backwards in the stream, it switches from\nsequential to `random`.\n\nThis policy essentially recognizes the initial read pattern of columnar\nstorage formats (e.g. Apache ORC and Apache Parquet), which seek to the end\nof a file, read in index data and then seek backwards to selectively read\ncolumns. The first seeks may be be expensive compared to the random policy,\nhowever the overall process is much less expensive than either sequentially\nreading through a file with the `random` policy, or reading columnar data\nwith the `sequential` policy.\n\n\n## Committing Work in MapReduce and Spark\n\nHadoop MapReduce, Apache Hive and Apache Spark all write their work\nto HDFS and similar filesystems.\nWhen using S3 as a destination, this is slow because of the way `rename()`\nis mimicked with copy and delete.\n\nIf committing output takes a long time, it is because you are using the standard\n`FileOutputCommitter`. If you are doing this on any S3 endpoint which lacks\nlist consistency (Amazon S3 without [S3Guard](s3guard.html)), this committer\nis at risk of losing data!\n\n*Your problem may appear to be performance, but that is a symptom\nof the underlying problem: the way S3A fakes rename operations means that\nthe rename cannot be safely be used in output-commit algorithms.*\n\nFix: Use one of the dedicated [S3A Committers](committers.md).\n\n## Options to Tune\n\n### Thread and connection pool sizes.\n\nEach S3A client interacting with a single bucket, as a single user, has its\nown dedicated pool of open HTTP 1.1 connections alongside a pool of threads used\nfor upload and copy operations.\nThe default pool sizes are intended to strike a balance between performance\nand memory/thread use.\n\nYou can have a larger pool of (reused) HTTP connections and threads\nfor parallel IO (especially uploads) by setting the properties\n\n\n| property | meaning | default |\n|----------|---------|---------|\n| `fs.s3a.threads.max`| Threads in the AWS transfer manager| 10 |\n| `fs.s3a.connection.maximum`| Maximum number of HTTP connections | 10|\n\nWe recommend using larger values for processes which perform\na lot of IO: `DistCp`, Spark Workers and similar.\n\n```xml\n\n fs.s3a.threads.max\n 20\n\n\n fs.s3a.connection.maximum\n 20\n\n```\n\nBe aware, however, that processes which perform many parallel queries\nmay consume large amounts of resources if each query is working with\na different set of s3 buckets, or are acting on behalf of different users.\n\n### For large data uploads, tune the block size: `fs.s3a.block.size`\n\nWhen uploading data, it is uploaded in blocks set by the option\n`fs.s3a.block.size`; default value \"32M\" for 32 Megabytes.\n\nIf a larger value is used, then more data is buffered before the upload\nbegins:\n\n```xml\n\n fs.s3a.block.size\n 128M\n\n```\n\nThis means that fewer PUT/POST requests are made of S3 to upload data,\nwhich reduces the likelihood that S3 will throttle the client(s)\n\n### Maybe: Buffer Write Data in Memory\n\nWhen large files are being uploaded, blocks are saved to disk and then\nqueued for uploading, with multiple threads uploading different blocks\nin parallel.\n\nThe blocks can be buffered in memory by setting the option\n`fs.s3a.fast.upload.buffer` to `bytebuffer`, or, for on-heap storage\n`array`.\n\n1. Switching to in memory-IO reduces disk IO, and can be faster if the bandwidth\nto the S3 store is so high that the disk IO becomes the bottleneck.\nThis can have a tangible benefit when working with on-premise S3-compatible\nobject stores with very high bandwidth to servers.\n\nIt is very easy to run out of memory when buffering to it; the option\n`fs.s3a.fast.upload.active.blocks\"` exists to tune how many active blocks\na single output stream writing to S3 may have queued at a time.\n\nAs the size of each buffered block is determined by the value of `fs.s3a.block.size`,\nthe larger the block size, the more likely you will run out of memory.\n\n## DistCP\n\nDistCP can be slow, especially if the parameters and options for the operation\nare not tuned for working with S3.\n\nTo exacerbate the issue, DistCP invariably puts heavy load against the\nbucket being worked with, which will cause S3 to throttle requests.\nIt will throttle: directory operations, uploads of new data, and delete operations,\namongst other things\n\n### DistCP: Options to Tune\n\n* `-numListstatusThreads ` : set to something higher than the default (1).\n* `-bandwidth ` : use to limit the upload bandwidth per worker\n* `-m ` : limit the number of mappers, hence the load on the S3 bucket.\n\nAdding more maps with the `-m` option does not guarantee better performance;\nit may just increase the amount of throttling which takes place.\nA smaller number of maps with a higher bandwidth per map can be more efficient.\n\n### DistCP: Options to Avoid.\n\nDistCp's `-atomic` option copies up data into a directory, then renames\nit into place, which is the where the copy takes place. This is a performance\nkiller.\n\n* Do not use the `-atomic` option.\n* The `-append` operation is not supported on S3; avoid.\n* `-p` S3 does not have a POSIX-style permission model; this will fail.\n\n\n### DistCP: Parameters to Tune\n\n1. As discussed [earlier](#pooling), use large values for\n`fs.s3a.threads.max` and `fs.s3a.connection.maximum`.\n\n1. Make sure that the bucket is using `sequential` or `normal` fadvise seek policies,\nthat is, `fs.s3a.experimental.fadvise` is not set to `random`\n\n1. Perform listings in parallel by setting `-numListstatusThreads`\nto a higher number. Make sure that `fs.s3a.connection.maximum`\nis equal to or greater than the value used.\n\n1. If using `-delete`, set `fs.trash.interval` to 0 to avoid the deleted\nobjects from being copied to a trash directory.\n\n*DO NOT* switch `fs.s3a.fast.upload.buffer` to buffer in memory.\nIf one distcp mapper runs out of memory it will fail,\nand that runs the risk of failing the entire job.\nIt is safer to keep the default value, `disk`.\n\nWhat is potentially useful is uploading in bigger blocks; this is more\nefficient in terms of HTTP connection use, and reduce the IOP rate against\nthe S3 bucket/shard.\n\n```xml\n\n fs.s3a.threads.max\n 20\n\n\n\n fs.s3a.connection.maximum\n 30\n \n Make greater than both fs.s3a.threads.max and -numListstatusThreads\n \n\n\n\n fs.s3a.experimental.fadvise\n normal\n\n\n\n fs.s3a.block.size\n 128M\n\n\n\n fs.s3a.fast.upload.buffer\n disk\n\n\n\n fs.trash.interval\n 0\n\n```\n\n## hadoop shell commands `fs -rm`\n\nThe `hadoop fs -rm` command can rename the file under `.Trash` rather than\ndeleting it. Use `-skipTrash` to eliminate that step.\n\n\nThis can be set in the property `fs.trash.interval`; while the default is 0,\nmost HDFS deployments have it set to a non-zero value to reduce the risk of\ndata loss.\n\n```xml\n\n fs.trash.interval\n 0\n\n```\n\n\n## Improving S3 load-balancing behavior\n\nAmazon S3 uses a set of front-end servers to provide access to the underlying data.\nThe choice of which front-end server to use is handled via load-balancing DNS\nservice: when the IP address of an S3 bucket is looked up, the choice of which\nIP address to return to the client is made based on the the current load\nof the front-end servers.\n\nOver time, the load across the front-end changes, so those servers considered\n\"lightly loaded\" will change. If the DNS value is cached for any length of time,\nyour application may end up talking to an overloaded server. Or, in the case\nof failures, trying to talk to a server that is no longer there.\n\nAnd by default, for historical security reasons in the era of applets,\nthe DNS TTL of a JVM is \"infinity\".\n\nTo work with AWS better, set the DNS time-to-live of an application which\nworks with S3 to something lower.\nSee [AWS documentation](http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-jvm-ttl.html).\n\n## Troubleshooting network performance\n\nAn example of this is covered in [HADOOP-13871](https://issues.apache.org/jira/browse/HADOOP-13871).\n\n1. For public data, use `curl`:\n\n curl -O https://landsat-pds.s3.amazonaws.com/scene_list.gz\n1. Use `nettop` to monitor a processes connections.\n\n\n## Throttling\n\nWhen many requests are made of a specific S3 bucket (or shard inside it),\nS3 will respond with a 503 \"throttled\" response.\nThrottling can be recovered from, provided overall load decreases.\nFurthermore, because it is sent before any changes are made to the object store,\nis inherently idempotent. For this reason, the client will always attempt to\nretry throttled requests.\n\nThe limit of the number of times a throttled request can be retried,\nand the exponential interval increase between attempts, can be configured\nindependently of the other retry limits.\n\n```xml\n\n fs.s3a.retry.throttle.limit\n 20\n \n Number of times to retry any throttled request.\n \n\n\n\n fs.s3a.retry.throttle.interval\n 500ms\n \n Interval between retry attempts on throttled requests.\n \n\n```\n\nIf a client is failing due to `AWSServiceThrottledException` failures,\nincreasing the interval and limit *may* address this. However, it\nit is a sign of AWS services being overloaded by the sheer number of clients\nand rate of requests. Spreading data across different buckets, and/or using\na more balanced directory structure may be beneficial.\nConsult [the AWS documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).\n\nReading or writing data encrypted with SSE-KMS forces S3 to make calls of\nthe AWS KMS Key Management Service, which comes with its own\n[Request Rate Limits](http://docs.aws.amazon.com/kms/latest/developerguide/limits.html).\nThese default to 1200/second for an account, across all keys and all uses of\nthem, which, for S3 means: across all buckets with data encrypted with SSE-KMS.\n\n### Tips to Keep Throttling down\n\nIf you are seeing a lot of throttling responses on a large scale\noperation like a `distcp` copy, *reduce* the number of processes trying\nto work with the bucket (for distcp: reduce the number of mappers with the\n`-m` option).\n\nIf you are reading or writing lists of files, if you can randomize\nthe list so they are not processed in a simple sorted order, you may\nreduce load on a specific shard of S3 data, so potentially increase throughput.\n\nAn S3 Bucket is throttled by requests coming from all\nsimultaneous clients. Different applications and jobs may interfere with\neach other: consider that when troubleshooting.\nPartitioning data into different buckets may help isolate load here.\n\nIf you are using data encrypted with SSE-KMS, then the\nwill also apply: these are stricter than the S3 numbers.\nIf you believe that you are reaching these limits, you may be able to\nget them increased.\nConsult [the KMS Rate Limit documentation](http://docs.aws.amazon.com/kms/latest/developerguide/limits.html).\n\n### S3Guard and Throttling\n\n\nS3Guard uses DynamoDB for directory and file lookups;\nit is rate limited to the amount of (guaranteed) IO purchased for a\ntable.\n\nTo see the allocated capacity of a bucket, the `hadoop s3guard bucket-info s3a://bucket`\ncommand will print out the allocated capacity.\n\n\nIf significant throttling events/rate is observed here, the pre-allocated\nIOPs can be increased with the `hadoop s3guard set-capacity` command, or\nthrough the AWS Console. Throttling events in S3Guard are noted in logs, and\nalso in the S3A metrics `s3guard_metadatastore_throttle_rate` and\n`s3guard_metadatastore_throttled`.\n\nIf you are using DistCP for a large backup to/from a S3Guarded bucket, it is\nactually possible to increase the capacity for the duration of the operation.\n\n\n## Best Practises for Code\n\nHere are some best practises if you are writing applications to work with\nS3 or any other object store through the Hadoop APIs.\n\nUse `listFiles(path, recursive)` over `listStatus(path)`.\nThe recursive `listFiles()` call can enumerate all dependents of a path\nin a single LIST call, irrespective of how deep the path is.\nIn contrast, any directory tree-walk implemented in the client is issuing\nmultiple HTTP requests to scan each directory, all the way down.\n\nCache the outcome of `getFileStats()`, rather than repeatedly ask for it.\nThat includes using `isFile()`, `isDirectory()`, which are simply wrappers\naround `getFileStatus()`.\n\nDon't immediately look for a file with a `getFileStatus()` or listing call\nafter creating it, or try to read it immediately.\nThis is where eventual consistency problems surface: the data may not yet be visible.\n\nRely on `FileNotFoundException` being raised if the source of an operation is\nmissing, rather than implementing your own probe for the file before\nconditionally calling the operation.\n\n### `rename()`\n\nAvoid any algorithm which uploads data into a temporary file and then uses\n`rename()` to commit it into place with a final path.\nOn HDFS this offers a fast commit operation.\nWith S3, Wasb and other object stores, you can write straight to the destination,\nknowing that the file isn't visible until you close the write: the write itself\nis atomic.\n\nThe `rename()` operation may return `false` if the source is missing; this\nis a weakness in the API. Consider a check before calling rename, and if/when\na new rename() call is made public, switch to it.\n\n\n### `delete(path, recursive)`\n\nKeep in mind that `delete(path, recursive)` is a no-op if the path does not exist, so\nthere's no need to have a check for the path existing before you call it.\n\n`delete()` is often used as a cleanup operation.\nWith an object store this is slow, and may cause problems if the caller\nexpects an immediate response. For example, a thread may block so long\nthat other liveness checks start to fail.\nConsider spawning off an executor thread to do these background cleanup operations.\n","isInternal":false,"tokens":4866,"sizeBytes":20404},{"name":"testing.md","path":"hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md","rawUrl":"https://raw.githubusercontent.com/apache/hadoop/HEAD/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/testing.md","title":"Hadoop-aws Skill","category":"anthropic-skill","format":"markdown","content":"\n\n# Testing the S3A filesystem client and its features, including S3Guard\n\n\n\nThis module includes both unit tests, which can run in isolation without\nconnecting to the S3 service, and integration tests, which require a working\nconnection to S3 to interact with a bucket. Unit test suites follow the naming\nconvention `Test*.java`. Integration tests follow the naming convention\n`ITest*.java`.\n\nDue to eventual consistency, integration tests may fail without reason.\nTransient failures, which no longer occur upon rerunning the test, should thus\nbe ignored.\n\n## Policy for submitting patches which affect the `hadoop-aws` module.\n\nThe Apache Jenkins infrastructure does not run any S3 integration tests,\ndue to the need to keep credentials secure.\n\n### The submitter of any patch is required to run all the integration tests and declare which S3 region/implementation they used.\n\nThis is important: **patches which do not include this declaration will be ignored**\n\nThis policy has proven to be the only mechanism to guarantee full regression\ntesting of code changes. Why the declaration of region? Two reasons\n\n1. It helps us identify regressions which only surface against specific endpoints\nor third-party implementations of the S3 protocol.\n1. It forces the submitters to be more honest about their testing. It's easy\nto lie, \"yes, I tested this\". To say \"yes, I tested this against S3 US-west\"\nis a more specific lie and harder to make. And, if you get caught out: you\nlose all credibility with the project.\n\nYou don't need to test from a VM within the AWS infrastructure; with the\n`-Dparallel=tests` option the non-scale tests complete in under ten minutes.\nBecause the tests clean up after themselves, they are also designed to be low\ncost. It's neither hard nor expensive to run the tests; if you can't,\nthere's no guarantee your patch works. The reviewers have enough to do, and\ndon't have the time to do these tests, especially as every failure will simply\nmake for a slow iterative development.\n\nPlease: run the tests. And if you don't, we are sorry for declining your\npatch, but we have to.\n\n\n### What if there's an intermittent failure of a test?\n\nSome of the tests do fail intermittently, especially in parallel runs.\nIf this happens, try to run the test on its own to see if the test succeeds.\n\nIf it still fails, include this fact in your declaration. We know some tests\nare intermittently unreliable.\n\n### What if the tests are timing out or failing over my network connection?\n\nThe tests and the S3A client are designed to be configurable for different\ntimeouts. If you are seeing problems and this configuration isn't working,\nthat's a sign of the configuration mechanism isn't complete. If it's happening\nin the production code, that could be a sign of a problem which may surface\nover long-haul connections. Please help us identify and fix these problems\n— especially as you are the one best placed to verify the fixes work.\n\n## Setting up the tests\n\nTo integration test the S3* filesystem clients, you need to provide\n`auth-keys.xml` which passes in authentication details to the test runner.\n\nIt is a Hadoop XML configuration file, which must be placed into\n`hadoop-tools/hadoop-aws/src/test/resources`.\n\n### File `core-site.xml`\n\nThis file pre-exists and sources the configurations created\nunder `auth-keys.xml`.\n\nFor most purposes you will not need to edit this file unless you\nneed to apply a specific, non-default property change during the tests.\n\n### File `auth-keys.xml`\n\nThe presence of this file triggers the testing of the S3 classes.\n\nWithout this file, *none of the integration tests in this module will be\nexecuted*.\n\nThe XML file must contain all the ID/key information needed to connect\neach of the filesystem clients to the object stores, and a URL for\neach filesystem for its testing.\n\n1. `test.fs.s3a.name` : the URL of the bucket for S3a tests\n1. `fs.contract.test.fs.s3a` : the URL of the bucket for S3a filesystem contract tests\n\n\nThe contents of the bucket will be destroyed during the test process:\ndo not use the bucket for any purpose other than testing. Furthermore, for\ns3a, all in-progress multi-part uploads to the bucket will be aborted at the\nstart of a test (by forcing `fs.s3a.multipart.purge=true`) to clean up the\ntemporary state of previously failed tests.\n\nExample:\n\n```xml\n\n\n \n test.fs.s3a.name\n s3a://test-aws-s3a/\n \n\n \n fs.contract.test.fs.s3a\n ${test.fs.s3a.name}\n \n\n \n fs.s3a.access.key\n AWS access key ID. Omit for IAM role-based authentication.\n DONOTCOMMITTHISKEYTOSCM\n \n\n \n fs.s3a.secret.key\n AWS secret key. Omit for IAM role-based authentication.\n DONOTEVERSHARETHISSECRETKEY!\n \n\n \n test.sts.endpoint\n Specific endpoint to use for STS requests.\n sts.amazonaws.com\n \n\n\n```\n\n### Configuring S3a Encryption\n\nFor S3a encryption tests to run correctly, the\n`fs.s3a.server-side-encryption.key` must be configured in the s3a contract xml\nfile with a AWS KMS encryption key arn as this value is different for each AWS\nKMS.\n\nExample:\n\n```xml\n\n fs.s3a.server-side-encryption.key\n arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n```\n\nYou can also force all the tests to run with a specific SSE encryption method\nby configuring the property `fs.s3a.server-side-encryption-algorithm` in the s3a\ncontract file.\n\n## Running the Tests\n\nAfter completing the configuration, execute the test run through Maven.\n\n```bash\nmvn clean verify\n```\n\nIt's also possible to execute multiple test suites in parallel by passing the\n`parallel-tests` property on the command line. The tests spend most of their\ntime blocked on network I/O with the S3 service, so running in parallel tends to\ncomplete full test runs faster.\n\n```bash\nmvn -Dparallel-tests clean verify\n```\n\nSome tests must run with exclusive access to the S3 bucket, so even with the\n`parallel-tests` property, several test suites will run in serial in a separate\nMaven execution step after the parallel tests.\n\nBy default, `parallel-tests` runs 4 test suites concurrently. This can be tuned\nby passing the `testsThreadCount` property.\n\n```bash\nmvn -Dparallel-tests -DtestsThreadCount=8 clean verify\n```\n\nTo run just unit tests, which do not require S3 connectivity or AWS credentials,\nuse any of the above invocations, but switch the goal to `test` instead of\n`verify`.\n\n```bash\nmvn clean test\n\nmvn -Dparallel-tests clean test\n\nmvn -Dparallel-tests -DtestsThreadCount=8 clean test\n```\n\nTo run only a specific named subset of tests, pass the `test` property for unit\ntests or the `it.test` property for integration tests.\n\n```bash\nmvn clean test -Dtest=TestS3AInputPolicies\n\nmvn clean verify -Dit.test=ITestS3AFileContextStatistics -Dtest=none\n\nmvn clean verify -Dtest=TestS3A* -Dit.test=ITestS3A*\n```\n\nNote that when running a specific subset of tests, the patterns passed in `test`\nand `it.test` override the configuration of which tests need to run in isolation\nin a separate serial phase (mentioned above). This can cause unpredictable\nresults, so the recommendation is to avoid passing `parallel-tests` in\ncombination with `test` or `it.test`. If you know that you are specifying only\ntests that can run safely in parallel, then it will work. For wide patterns,\nlike `ITestS3A*` shown above, it may cause unpredictable test failures.\n\n### Testing against different regions\n\nS3A can connect to different regions —the tests support this. Simply\ndefine the target region in `auth-keys.xml`.\n\n```xml\n\n fs.s3a.endpoint\n s3.eu-central-1.amazonaws.com\n\n```\nThis is used for all tests expect for scale tests using a Public CSV.gz file\n(see below)\n\n### CSV Data Tests\n\nThe `TestS3AInputStreamPerformance` tests require read access to a multi-MB\ntext file. The default file for these tests is one published by amazon,\n[s3a://landsat-pds.s3.amazonaws.com/scene_list.gz](http://landsat-pds.s3.amazonaws.com/scene_list.gz).\nThis is a gzipped CSV index of other files which amazon serves for open use.\n\nThe path to this object is set in the option `fs.s3a.scale.test.csvfile`,\n\n```xml\n\n fs.s3a.scale.test.csvfile\n s3a://landsat-pds/scene_list.gz\n\n```\n\n1. If the option is not overridden, the default value is used. This\nis hosted in Amazon's US-east datacenter.\n1. If `fs.s3a.scale.test.csvfile` is empty, tests which require it will be skipped.\n1. If the data cannot be read for any reason then the test will fail.\n1. If the property is set to a different path, then that data must be readable\nand \"sufficiently\" large.\n\n(the reason the space or newline is needed is to add \"an empty entry\"; an empty\n`` would be considered undefined and pick up the default)\n\nOf using a test file in an S3 region requiring a different endpoint value\nset in `fs.s3a.endpoint`, a bucket-specific endpoint must be defined.\nFor the default test dataset, hosted in the `landsat-pds` bucket, this is:\n\n```xml\n\n fs.s3a.bucket.landsat-pds.endpoint\n s3.amazonaws.com\n The endpoint for s3a://landsat-pds URLs\n\n```\n\n## Viewing Integration Test Reports\n\n\nIntegration test results and logs are stored in `target/failsafe-reports/`.\nAn HTML report can be generated during site generation, or with the `surefire-report`\nplugin:\n\n```bash\nmvn surefire-report:failsafe-report-only\n```\n## Scale Tests\n\nThere are a set of tests designed to measure the scalability and performance\nat scale of the S3A tests, *Scale Tests*. Tests include: creating\nand traversing directory trees, uploading large files, renaming them,\ndeleting them, seeking through the files, performing random IO, and others.\nThis makes them a foundational part of the benchmarking.\n\nBy their very nature they are slow. And, as their execution time is often\nlimited by bandwidth between the computer running the tests and the S3 endpoint,\nparallel execution does not speed these tests up.\n\n***Note: Running scale tests with -Ds3guard and -Ddynamo requires that\nyou use a private, testing-only DynamoDB table.*** The tests do disruptive\nthings such as deleting metadata and setting the provisioned throughput\nto very low values.\n\n### Enabling the Scale Tests\n\nThe tests are enabled if the `scale` property is set in the maven build\nthis can be done regardless of whether or not the parallel test profile\nis used\n\n```bash\nmvn verify -Dscale\n\nmvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8\n```\n\nThe most bandwidth intensive tests (those which upload data) always run\nsequentially; those which are slow due to HTTPS setup costs or server-side\nactions are included in the set of parallelized tests.\n\n\n### Tuning scale options from Maven\n\n\nSome of the tests can be tuned from the maven build or from the\nconfiguration file used to run the tests.\n\n```bash\nmvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dfs.s3a.scale.test.huge.filesize=128M\n```\n\nThe algorithm is\n\n1. The value is queried from the configuration file, using a default value if\nit is not set.\n1. The value is queried from the JVM System Properties, where it is passed\ndown by maven.\n1. If the system property is null, an empty string, or it has the value `unset`,\nthen the configuration value is used. The `unset` option is used to\n[work round a quirk in maven property propagation](http://stackoverflow.com/questions/7773134/null-versus-empty-arguments-in-maven).\n\nOnly a few properties can be set this way; more will be added.\n\n| Property | Meaning |\n|-----------|-------------|\n| `fs.s3a.scale.test.timeout`| Timeout in seconds for scale tests |\n| `fs.s3a.scale.test.huge.filesize`| Size for huge file uploads |\n| `fs.s3a.scale.test.huge.huge.partitionsize`| Size for partitions in huge file uploads |\n\nThe file and partition sizes are numeric values with a k/m/g/t/p suffix depending\non the desired size. For example: 128M, 128m, 2G, 2G, 4T or even 1P.\n\n### Scale test configuration options\n\nSome scale tests perform multiple operations (such as creating many directories).\n\nThe exact number of operations to perform is configurable in the option\n`scale.test.operation.count`\n\n```xml\n\n scale.test.operation.count\n 10\n\n```\n\nLarger values generate more load, and are recommended when testing locally,\nor in batch runs.\n\nSmaller values results in faster test runs, especially when the object\nstore is a long way away.\n\nOperations which work on directories have a separate option: this controls\nthe width and depth of tests creating recursive directories. Larger\nvalues create exponentially more directories, with consequent performance\nimpact.\n\n```xml\n\n scale.test.directory.count\n 2\n\n```\n\nDistCp tests targeting S3A support a configurable file size. The default is\n10 MB, but the configuration value is expressed in KB so that it can be tuned\nsmaller to achieve faster test runs.\n\n```xml\n\n scale.test.distcp.file.size.kb\n 10240\n\n```\n\nS3A specific scale test properties are\n\n*`fs.s3a.scale.test.huge.filesize`: size in MB for \"Huge file tests\".*\n\nThe Huge File tests validate S3A's ability to handle large files —the property\n`fs.s3a.scale.test.huge.filesize` declares the file size to use.\n\n```xml\n\n fs.s3a.scale.test.huge.filesize\n 200M\n\n```\n\nAmazon S3 handles files larger than 5GB differently than smaller ones.\nSetting the huge filesize to a number greater than that) validates support\nfor huge files.\n\n```xml\n\n fs.s3a.scale.test.huge.filesize\n 6G\n\n```\n\nTests at this scale are slow: they are best executed from hosts running in\nthe cloud infrastructure where the S3 endpoint is based.\nOtherwise, set a large timeout in `fs.s3a.scale.test.timeout`\n\n```xml\n\n fs.s3a.scale.test.timeout\n 432000\n\n```\n\nThe tests are executed in an order to only clean up created files after\nthe end of all the tests. If the tests are interrupted, the test data will remain.\n\n\n## Testing against non AWS S3 endpoints.\n\nThe S3A filesystem is designed to work with storage endpoints which implement\nthe S3 protocols to the extent that the amazon S3 SDK is capable of talking\nto it. We encourage testing against other filesystems and submissions of patches\nwhich address issues. In particular, we encourage testing of Hadoop release\ncandidates, as these third-party endpoints get even less testing than the\nS3 endpoint itself.\n\n\n### Disabling the encryption tests\n\nIf the endpoint doesn't support server-side-encryption, these will fail. They\ncan be turned off.\n\n```xml\n\n test.fs.s3a.encryption.enabled\n false\n\n```\n\nEncryption is only used for those specific test suites with `Encryption` in\ntheir classname.\n\n### Configuring the CSV file read tests**\n\nTo test on alternate infrastructures supporting\nthe same APIs, the option `fs.s3a.scale.test.csvfile` must either be\nset to \" \", or an object of at least 10MB is uploaded to the object store, and\nthe `fs.s3a.scale.test.csvfile` option set to its path.\n\n```xml\n\n fs.s3a.scale.test.csvfile\n \n\n```\n\n(yes, the space is necessary. The Hadoop `Configuration` class treats an empty\nvalue as \"do not override the default\").\n\n\n### Testing Session Credentials\n\nThe test `TestS3ATemporaryCredentials` requests a set of temporary\ncredentials from the STS service, then uses them to authenticate with S3.\n\nIf an S3 implementation does not support STS, then the functional test\ncases must be disabled:\n\n```xml\n\n test.fs.s3a.sts.enabled\n false\n\n```\nThese tests request a temporary set of credentials from the STS service endpoint.\nAn alternate endpoint may be defined in `test.fs.s3a.sts.endpoint`.\n\n```xml\n\n test.fs.s3a.sts.endpoint\n https://sts.example.org/\n\n```\nThe default is \"\"; meaning \"use the amazon default value\".\n\n\n## Debugging Test failures\n\nLogging at debug level is the standard way to provide more diagnostics output;\nafter setting this rerun the tests\n\n```properties\nlog4j.logger.org.apache.hadoop.fs.s3a=DEBUG\n```\n\nThere are also some logging options for debug logging of the AWS client\n```properties\nlog4j.logger.com.amazonaws=DEBUG\nlog4j.logger.com.amazonaws.http.conn.ssl=INFO\nlog4j.logger.com.amazonaws.internal=INFO\n```\n\nThere is also the option of enabling logging on a bucket; this could perhaps\nbe used to diagnose problems from that end. This isn't something actively\nused, but remains an option. If you are forced to debug this way, consider\nsetting the `fs.s3a.user.agent.prefix` to a unique prefix for a specific\ntest run, which will enable the specific log entries to be more easily\nlocated.\n\n## Adding new tests\n\nNew tests are always welcome. Bear in mind that we need to keep costs\nand test time down, which is done by\n\n* Not duplicating tests.\n* Being efficient in your use of Hadoop API calls.\n* Isolating large/slow tests into the \"scale\" test group.\n* Designing all tests to execute in parallel (where possible).\n* Adding new probes and predicates into existing tests, albeit carefully.\n\n*No duplication*: if an operation is tested elsewhere, don't repeat it. This\napplies as much for metadata operations as it does for bulk IO. If a new\ntest case is added which completely obsoletes an existing test, it is OK\nto cut the previous one —after showing that coverage is not worsened.\n\n*Efficient*: prefer the `getFileStatus()` and examining the results, rather than\ncall to `exists()`, `isFile()`, etc.\n\n*Isolating Scale tests*. Any S3A test doing large amounts of IO MUST extend the\nclass `S3AScaleTestBase`, so only running if `scale` is defined on a build,\nsupporting test timeouts configurable by the user. Scale tests should also\nsupport configurability as to the actual size of objects/number of operations,\nso that behavior at different scale can be verified.\n\n*Designed for parallel execution*. A key need here is for each test suite to work\non isolated parts of the filesystem. Subclasses of `AbstractS3ATestBase`\nSHOULD use the `path()` method, with a base path of the test suite name, to\nbuild isolated paths. Tests MUST NOT assume that they have exclusive access\nto a bucket.\n\n*Extending existing tests where appropriate*. This recommendation goes\nagainst normal testing best practise of \"test one thing per method\".\nBecause it is so slow to create directory trees or upload large files, we do\nnot have that luxury. All the tests against real S3 endpoints are integration\ntests where sharing test setup and teardown saves time and money.\n\nA standard way to do this is to extend existing tests with some extra predicates,\nrather than write new tests. When doing this, make sure that the new predicates\nfail with meaningful diagnostics, so any new problems can be easily debugged\nfrom test logs.\n\n\n## Requirements of new Tests\n\n\nThis is what we expect from new tests; they're an extension of the normal\nHadoop requirements, based on the need to work with remote servers whose\nuse requires the presence of secret credentials, where tests may be slow,\nand where finding out why something failed from nothing but the test output\nis critical.\n\n### Subclasses Existing Shared Base Classes\n\nExtend `AbstractS3ATestBase` or `AbstractSTestS3AHugeFiles` unless justifiable.\nThese set things up for testing against the object stores, provide good threadnames,\nhelp generate isolated paths, and for `AbstractSTestS3AHugeFiles` subclasses,\nonly run if `-Dscale` is set.\n\nKey features of `AbstractS3ATestBase`\n\n* `getFileSystem()` returns the S3A Filesystem bonded to the contract test Filesystem\ndefined in `fs.s3a.contract.test`\n* will automatically skip all tests if that URL is unset.\n* Extends `AbstractFSContractTestBase` and `Assert` for all their methods.\n\nHaving shared base classes may help reduce future maintenance too. Please\nuse them/\n\n### Secure\n\nDon't ever log credentials. The credential tests go out of their way to\nnot provide meaningful logs or assertion messages precisely to avoid this.\n\n### Efficient of Time and Money\n\nThis means efficient in test setup/teardown, and, ideally, making use of\nexisting public datasets to save setup time and tester cost.\n\nStrategies of particular note are:\n\n1. `ITestS3ADirectoryPerformance`: a single test case sets up the directory\ntree then performs different list operations, measuring the time taken.\n1. `AbstractSTestS3AHugeFiles`: marks the test suite as\n`@FixMethodOrder(MethodSorters.NAME_ASCENDING)` then orders the test cases such\nthat each test case expects the previous test to have completed (here: uploaded a file,\nrenamed a file, ...). This provides for independent tests in the reports, yet still\npermits an ordered sequence of operations. Do note the use of `Assume.assume()`\nto detect when the preconditions for a single test case are not met, hence,\nthe tests become skipped, rather than fail with a trace which is really a false alarm.\n\nThe ordered test case mechanism of `AbstractSTestS3AHugeFiles` is probably\nthe most elegant way of chaining test setup/teardown.\n\nRegarding reusing existing data, we tend to use the landsat archive of\nAWS US-East for our testing of input stream operations. This doesn't work\nagainst other regions, or with third party S3 implementations. Thus the\nURL can be overridden for testing elsewhere.\n\n\n### Works With Other S3 Endpoints\n\nDon't assume AWS S3 US-East only, do allow for working with external S3 implementations.\nThose may be behind the latest S3 API features, not support encryption, session\nAPIs, etc.\n\nThey won't have the same CSV test files as some of the input tests rely on.\nLook at `ITestS3AInputStreamPerformance` to see how tests can be written\nto support the declaration of a specific large test file on alternate filesystems.\n\n\n### Works Over Long-haul Links\n\nAs well as making file size and operation counts scalable, this includes\nmaking test timeouts adequate. The Scale tests make this configurable; it's\nhard coded to ten minutes in `AbstractS3ATestBase()`; subclasses can\nchange this by overriding `getTestTimeoutMillis()`.\n\nEqually importantly: support proxies, as some testers need them.\n\n\n### Provides Diagnostics and timing information\n\n1. Give threads useful names.\n1. Create logs, log things. Know that the `S3AFileSystem` and its input\nand output streams *all* provide useful statistics in their {{toString()}}\ncalls; logging them is useful on its own.\n1. you can use `AbstractS3ATestBase.describe(format-stringm, args)` here.; it\nadds some newlines so as to be easier to spot.\n1. Use `ContractTestUtils.NanoTimer` to measure the duration of operations,\nand log the output.\n\n### Fails Meaningfully\n\nThe `ContractTestUtils` class contains a whole set of assertions for making\nstatements about the expected state of a filesystem, e.g.\n`assertPathExists(FS, path)`, `assertPathDoesNotExists(FS, path)`, and others.\nThese do their best to provide meaningful diagnostics on failures (e.g. directory\nlistings, file status, ...), so help make failures easier to understand.\n\nAt the very least, do not use `assertTrue()` or `assertFalse()` without\nincluding error messages.\n\n### Sets up its filesystem and checks for those settings\n\nTests can overrun `createConfiguration()` to add new options to the configuration\nfile for the S3A Filesystem instance used in their tests.\n\nHowever, filesystem caching may mean that a test suite may get a cached\ninstance created with an different configuration. For tests which don't need\nspecific configurations caching is good: it reduces test setup time.\n\nFor those tests which do need unique options (encryption, magic files),\nthings can break, and they will do so in hard-to-replicate ways.\n\nUse `S3ATestUtils.disableFilesystemCaching(conf)` to disable caching when\nmodifying the config. As an example from `AbstractTestS3AEncryption`:\n\n```java\n@Override\nprotected Configuration createConfiguration() {\n Configuration conf = super.createConfiguration();\n S3ATestUtils.disableFilesystemCaching(conf);\n conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM,\n getSSEAlgorithm().getMethod());\n return conf;\n}\n```\n\nThen verify in the setup method or test cases that their filesystem actually has\nthe desired feature (`fs.getConf().getProperty(...)`). This not only\ncatches filesystem reuse problems, it catches the situation where the\nfilesystem configuration in `auth-keys.xml` has explicit per-bucket settings\nwhich override the test suite's general option settings.\n\n### Cleans Up Afterwards\n\nKeeps costs down.\n\n1. Do not only cleanup if a test case completes successfully; test suite\nteardown must do it.\n1. That teardown code must check for the filesystem and other fields being\nnull before the cleanup. Why? If test setup fails, the teardown methods still\nget called.\n\n### Works Reliably\n\nWe really appreciate this — you will too.\n\n### Runs in parallel unless this is unworkable.\n\nTests must be designed to run in parallel with other tests, all working\nwith the same shared S3 bucket. This means\n\n* Uses relative and JVM-fork-unique paths provided by the method\n `AbstractFSContractTestBase.path(String filepath)`.\n* Doesn't manipulate the root directory or make assertions about its contents\n(for example: delete its contents and assert that it is now empty).\n* Doesn't have a specific requirement of all active clients of the bucket\n(example: SSE-C tests which require all files, even directory markers,\nto be encrypted with the same key).\n* Doesn't use so much bandwidth that all other tests will be starved of IO and\nstart timing out (e.g. the scale tests).\n\nTests such as these can only be run as sequential tests. When adding one,\nexclude it in the POM file. from the parallel failsafe run and add to the\nsequential one afterwards. The IO heavy ones must also be subclasses of\n`S3AScaleTestBase` and so only run if the system/maven property\n`fs.s3a.scale.test.enabled` is true.\n\n## Individual test cases can be run in an IDE\n\nThis is invaluable for debugging test failures.\n\n### Keeping AWS Costs down\n\nMost of the base S3 tests are designed to use public AWS data\n(the landsat-pds bucket) for read IO, so you don't have to pay for bytes\ndownloaded or long term storage costs. The scale tests do work with more data\nso will cost more as well as generally take more time to execute.\n\nYou are however billed for\n\n1. Data left in S3 after test runs.\n2. DynamoDB capacity reserved by S3Guard tables.\n3. HTTP operations on files (HEAD, LIST, GET).\n4. In-progress multipart uploads from bulk IO or S3A committer tests.\n5. Encryption/decryption using AWS KMS keys.\n\nThe GET/decrypt costs are incurred on each partial read of a file,\nso random IO can cost more than sequential IO; the speedup of queries with\ncolumnar data usually justifies this.\n\nThe DynamoDB costs come from the number of entries stores and the allocated capacity.\n\nHow to keep costs down\n\n* Don't run the scale tests with large datasets; keep `fs.s3a.scale.test.huge.filesize` unset, or a few MB (minimum: 5).\n* Remove all files in the filesystem. The root tests usually do this, but\nit can be manually done:\n\n hadoop fs -rm -r -f -skipTrash s3a://test-bucket/\\*\n* Abort all outstanding uploads:\n\n hadoop s3guard uploads -abort -force s3a://test-bucket/\n* If you don't need it, destroy the S3Guard DDB table.\n\n hadoop s3guard destroy s3a://hwdev-steve-ireland-new/\n\nThe S3Guard tests will automatically create the Dynamo DB table in runs with\n`-Ds3guard -Ddynamodb` set; default capacity of these buckets\ntests is very small; it keeps costs down at the expense of IO performance\nand, for test runs in or near the S3/DDB stores, throttling events.\n\nIf you want to manage capacity, use `s3guard set-capacity` to increase it\n(performance) or decrease it (costs).\nFor remote `hadoop-aws` test runs, the read/write capacities of \"10\" each should suffice;\nincrease it if parallel test run logs warn of throttling.\n\nTip: for agility, use DynamoDB autoscaling, setting the minimum to something very low (e.g 5 units), the maximum to the largest amount you are willing to pay.\nThis will automatically reduce capacity when you are not running tests against\nthe bucket, slowly increase it over multiple test runs, if the load justifies it.\n\n## Tips\n\n### How to keep your credentials really safe\n\nAlthough the `auth-keys.xml` file is marked as ignored in git and subversion,\nit is still in your source tree, and there's always that risk that it may\ncreep out.\n\nYou can avoid this by keeping your keys outside the source tree and\nusing an absolute XInclude reference to it.\n\n```xml\n\n\n \n\n\n```\n\n# Failure Injection\n\n**Warning do not enable any type of failure injection in production. The\nfollowing settings are for testing only.**\n\nOne of the challenges with S3A integration tests is the fact that S3 is an\neventually-consistent storage system. In practice, we rarely see delays in\nvisibility of recently created objects both in listings (`listStatus()`) and\nwhen getting a single file's metadata (`getFileStatus()`). Since this behavior\nis rare and non-deterministic, thorough integration testing is challenging.\n\nTo address this, S3A supports a shim layer on top of the `AmazonS3Client`\nclass which artificially delays certain paths from appearing in listings.\nThis is implemented in the class `InconsistentAmazonS3Client`.\n\n## Simulating List Inconsistencies\n\n### Enabling the InconsistentAmazonS3CClient\n\nThere are two ways of enabling the `InconsistentAmazonS3Client`: at\nconfig-time, or programmatically. For an example of programmatic test usage,\nsee `ITestS3GuardListConsistency`.\n\nTo enable the fault-injecting client via configuration, switch the\nS3A client to use the \"Inconsistent S3 Client Factory\" when connecting to\nS3:\n\n```xml\n\n fs.s3a.s3.client.factory.impl\n org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory\n\n```\n\nThe inconsistent client works by:\n\n1. Choosing which objects will be \"inconsistent\" at the time the object is\ncreated or deleted.\n2. When `listObjects()` is called, any keys that we have marked as\ninconsistent above will not be returned in the results (until the\nconfigured delay has elapsed). Similarly, deleted items may be *added* to\nmissing results to delay the visibility of the delete.\n\nThere are two ways of choosing which keys (filenames) will be affected: By\nsubstring, and by random probability.\n\n```xml\n\n fs.s3a.failinject.inconsistency.key.substring\n DELAY_LISTING_ME\n\n\n\n fs.s3a.failinject.inconsistency.probability\n 1.0\n\n```\n\nBy default, any object which has the substring \"DELAY_LISTING_ME\" in its key\nwill subject to delayed visibility. For example, the path\n`s3a://my-bucket/test/DELAY_LISTING_ME/file.txt` would match this condition.\nTo match all keys use the value \"\\*\" (a single asterisk). This is a special\nvalue: *We don't support arbitrary wildcards.*\n\nThe default probability of delaying an object is 1.0. This means that *all*\nkeys that match the substring will get delayed visibility. Note that we take\nthe logical *and* of the two conditions (substring matches *and* probability\nrandom chance occurs). Here are some example configurations:\n\n```\n| substring | probability | behavior |\n|-----------|-------------|--------------------------------------------|\n| | 0.001 | An empty tag in .xml config will |\n| | | be interpreted as unset and revert to the |\n| | | default value, \"DELAY_LISTING_ME\" |\n| | | |\n| * | 0.001 | 1/1000 chance of *any* key being delayed. |\n| | | |\n| delay | 0.01 | 1/100 chance of any key containing \"delay\" |\n| | | |\n| delay | 1.0 | All keys containing substring \"delay\" .. |\n```\n\nYou can also configure how long you want the delay in visibility to last.\nThe default is 5000 milliseconds (five seconds).\n\n```xml\n\n fs.s3a.failinject.inconsistency.msec\n 5000\n\n```\n\nFuture versions of this client will introduce new failure modes,\nwith simulation of S3 throttling exceptions the next feature under\ndevelopment.\n\n### Limitations of Inconsistency Injection\n\nAlthough `InconsistentAmazonS3Client` can delay the visibility of an object\nor parent directory, it does not prevent the key of that object from\nappearing in all prefix searches. For example, if we create the following\nobject with the default configuration above, in an otherwise empty bucket:\n\n```\ns3a://bucket/a/b/c/DELAY_LISTING_ME\n```\n\nThen the following paths will still be visible as directories (ignoring\npossible real-world inconsistencies):\n\n```\ns3a://bucket/a\ns3a://bucket/a/b\n```\n\nWhereas `getFileStatus()` on the following *will* be subject to delayed\nvisibility (`FileNotFoundException` until delay has elapsed):\n\n```\ns3a://bucket/a/b/c\ns3a://bucket/a/b/c/DELAY_LISTING_ME\n```\n\nIn real-life S3 inconsistency, however, we expect that all the above paths\n(including `a` and `b`) will be subject to delayed visibility.\n\n### Using the `InconsistentAmazonS3CClient` in downstream integration tests\n\nThe inconsistent client is shipped in the `hadoop-aws` JAR, so it can\nbe used in applications which work with S3 to see how they handle\ninconsistent directory listings.\n\n## Testing S3Guard\n\n[S3Guard](./s3guard.html) is an extension to S3A which adds consistent metadata\nlistings to the S3A client. As it is part of S3A, it also needs to be tested.\n\nThe basic strategy for testing S3Guard correctness consists of:\n\n1. MetadataStore Contract tests.\n\n The MetadataStore contract tests are inspired by the Hadoop FileSystem and\n `FileContext` contract tests. Each implementation of the `MetadataStore` interface\n subclasses the `MetadataStoreTestBase` class and customizes it to initialize\n their MetadataStore. This test ensures that the different implementations\n all satisfy the semantics of the MetadataStore API.\n\n2. Running existing S3A unit and integration tests with S3Guard enabled.\n\n You can run the S3A integration tests on top of S3Guard by configuring your\n `MetadataStore` in your\n `hadoop-tools/hadoop-aws/src/test/resources/core-site.xml` or\n `hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml` files.\n Next run the S3A integration tests as outlined in the *Running the Tests* section\n of the [S3A documentation](./index.html)\n\n3. Running fault-injection tests that test S3Guard's consistency features.\n\n The `ITestS3GuardListConsistency` uses failure injection to ensure\n that list consistency logic is correct even when the underlying storage is\n eventually consistent.\n\n The integration test adds a shim above the Amazon S3 Client layer that injects\n delays in object visibility.\n\n All of these tests will be run if you follow the steps listed in step 2 above.\n\n No charges are incurred for using this store, and its consistency\n guarantees are that of the underlying object store instance. \n\n## Testing S3A with S3Guard Enabled\n\nAll the S3A tests which work with a private repository can be configured to\nrun with S3Guard by using the `s3guard` profile. When set, this will run\nall the tests with local memory for the metadata set to \"non-authoritative\" mode.\n\n```bash\nmvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard\n```\n\nWhen the `s3guard` profile is enabled, following profiles can be specified:\n\n* `dynamo`: use an AWS-hosted DynamoDB table; creating the table if it does\n not exist. You will have to pay the bills for DynamoDB web service.\n* `dynamodblocal`: use an in-memory DynamoDBLocal server instead of real AWS\n DynamoDB web service; launch the server and creating the table.\n You won't be charged bills for using DynamoDB in test. As it runs in-JVM,\n the table isn't shared across other tests running in parallel.\n* `non-auth`: treat the S3Guard metadata as authoritative.\n\n```bash\nmvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Ddynamo -Dauth\n```\n\nWhen experimenting with options, it is usually best to run a single test suite\nat a time until the operations appear to be working.\n\n```bash\nmvn -T 1C verify -Dtest=skip -Dit.test=ITestS3AMiscOperations -Ds3guard -Ddynamo\n```\n\n### Notes\n\n1. If the `s3guard` profile is not set, then the S3Guard properties are those\nof the test configuration set in `contract-test-options.xml` or `auth-keys.xml`\n\nIf the `s3guard` profile *is* set,\n1. The S3Guard options from maven (the dynamo and authoritative flags)\n overwrite any previously set in the configuration files.\n1. DynamoDB will be configured to create any missing tables.\n\n\n### Scale Testing MetadataStore Directly\n\nThere are some scale tests that exercise Metadata Store implementations\ndirectly. These ensure that S3Guard is are robust to things like DynamoDB\nthrottling, and compare performance for different implementations. These\nare included in the scale tests executed when `-Dscale` is passed to\nthe maven command line.\n\nThe two S3Guard scale tests are `ITestDynamoDBMetadataStoreScale` and\n`ITestLocalMetadataStoreScale`.\n\nTo run these tests, your DynamoDB table needs to be of limited capacity;\nthe values in `ITestDynamoDBMetadataStoreScale` currently require a read capacity\nof 10 or less. a write capacity of 15 or more.\n\nThe following settings allow us to run `ITestDynamoDBMetadataStoreScale` with\nartificially low read and write capacity provisioned, so we can judge the\neffects of being throttled by the DynamoDB service:\n\n```xml\n\n scale.test.operation.count\n 10\n\n\n scale.test.directory.count\n 3\n\n\n fs.s3a.scale.test.enabled\n true\n\n\n fs.s3a.s3guard.ddb.table\n my-scale-test\n\n\n fs.s3a.s3guard.ddb.table.create\n true\n\n\n fs.s3a.s3guard.ddb.table.capacity.read\n 5\n\n\n fs.s3a.s3guard.ddb.table.capacity.write\n 5\n\n```\n\nThese tests verify that the invoked operations can trigger retries in the\nS3Guard code, rather than just in the AWS SDK level, so showing that if\nSDK operations fail, they get retried. They also verify that the filesystem\nstatistics are updated to record that throttling took place.\n\n*Do not panic if these tests fail to detect throttling!*\n\nThese tests are unreliable as they need certain conditions to be met\nto repeatedly fail:\n\n1. You must have a low-enough latency connection to the DynamoDB store that,\nfor the capacity allocated, you can overload it.\n1. The AWS Console can give you a view of what is happening here.\n1. Running a single test on its own is less likely to trigger an overload\nthan trying to run the whole test suite.\n1. And running the test suite more than once, back-to-back, can also help\noverload the cluster.\n1. Stepping through with a debugger will reduce load, so may not trigger\nfailures.\n\nIf the tests fail, it *probably* just means you aren't putting enough load\non the table.\n\nThese tests do not verify that the entire set of DynamoDB calls made\nduring the use of a S3Guarded S3A filesystem are wrapped by retry logic.\n\n*The best way to verify resilience is to run the entire `hadoop-aws` test suite,\nor even a real application, with throttling enabled.\n\n### Testing only: Local Metadata Store\n\nThere is an in-memory Metadata Store for testing.\n\n```xml\n\n fs.s3a.metadatastore.impl\n org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore\n\n```\n\nThis is not for use in production.\n\n## Testing Assumed Roles\n\nTests for the AWS Assumed Role credential provider require an assumed\nrole to request.\n\nIf this role is not set, the tests which require it will be skipped.\n\nTo run the tests in `ITestAssumeRole`, you need:\n\n1. A role in your AWS account will full read and write access rights to\nthe S3 bucket used in the tests, and ideally DynamoDB, for S3Guard.\nIf your bucket is set up by default to use S3Guard, the role must have access\nto that service.\n\n1. Your IAM User to have the permissions to adopt that role.\n\n1. The role ARN must be set in `fs.s3a.assumed.role.arn`.\n\n```xml\n\n fs.s3a.assumed.role.arn\n arn:aws:iam::9878543210123:role/role-s3-restricted\n\n```\n\nThe tests assume the role with different subsets of permissions and verify\nthat the S3A client (mostly) works when the caller has only write access\nto part of the directory tree.\n\nYou can also run the entire test suite in an assumed role, a more\nthorough test, by switching to the credentials provider.\n\n```xml\n\n fs.s3a.aws.credentials.provider\n org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n\n```\n\nThe usual credentials needed to log in to the bucket will be used, but now\nthe credentials used to interact with S3 and DynamoDB will be temporary\nrole credentials, rather than the full credentials.\n\n## Qualifying an AWS SDK Update\n\nUpdating the AWS SDK is something which does need to be done regularly,\nbut is rarely without complications, major or minor.\n\nAssume that the version of the SDK will remain constant for an X.Y release,\nexcluding security fixes, so it's good to have an update before each release\n— as long as that update works doesn't trigger any regressions.\n\n\n1. Don't make this a last minute action.\n1. The upgrade patch should focus purely on the SDK update, so it can be cherry\npicked and reverted easily.\n1. Do not mix in an SDK update with any other piece of work, for the same reason.\n1. Plan for an afternoon's work, including before/after testing, log analysis\nand any manual tests.\n1. Make sure all the integration tests are running (including s3guard, ARN, encryption, scale)\n *before you start the upgrade*.\n1. Create a JIRA for updating the SDK. Don't include the version (yet),\nas it may take a couple of SDK updates before it is ready.\n1. Identify the latest AWS SDK [available for download](https://aws.amazon.com/sdk-for-java/).\n1. Create a private git branch of trunk for JIRA, and in\n `hadoop-project/pom.xml` update the `aws-java-sdk.version` to the new SDK version.\n1. Do a clean build and rerun all the `hadoop-aws` tests, with and without the `-Ds3guard -Ddynamodb` options.\n This includes the `-Pscale` set, with a role defined for the assumed role tests.\n in `fs.s3a.assumed.role.arn` for testing assumed roles,\n and `fs.s3a.server-side-encryption.key` for encryption, for full coverage.\n If you can, scale up the scale tests.\n1. Create the site with `mvn site -DskipTests`; look in `target/site` for the report.\n1. Review *every single `-output.txt` file in `hadoop-tools/hadoop-aws/target/failsafe-reports`,\n paying particular attention to\n `org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance-output.txt`,\n as that is where changes in stream close/abort logic will surface.\n1. Run `mvn install` to install the artifacts, then in\n `hadoop-cloud-storage-project/hadoop-cloud-storage` run\n `mvn dependency:tree -Dverbose > target/dependencies.txt`.\n Examine the `target/dependencies.txt` file to verify that no new\n artifacts have unintentionally been declared as dependencies\n of the shaded `aws-java-sdk-bundle` artifact.\n\n### Basic command line regression testing\n\nWe need a run through of the CLI to see if there have been changes there\nwhich cause problems, especially whether new log messages have surfaced,\nor whether some packaging change breaks that CLI\n\nFrom the root of the project, create a command line release `mvn package -Pdist -DskipTests -Dmaven.javadoc.skip=true -DskipShade`;\n\n1. Change into the `hadoop/dist/target/hadoop-x.y.z-SNAPSHOT` dir.\n1. Copy a `core-site.xml` file into `etc/hadoop`.\n1. Set the `HADOOP_OPTIONAL_TOOLS` env var on the command line or `~/.hadoop-env`.\n\n```bash\nexport HADOOP_OPTIONAL_TOOLS=\"hadoop-aws\"\n```\n\nRun some basic s3guard commands as well as file operations.\n\n```bash\nexport BUCKET=s3a://example-bucket-name\n\nbin/hadoop s3guard bucket-info $BUCKET\nbin/hadoop s3guard set-capacity $BUCKET\nbin/hadoop s3guard set-capacity -read 15 -write 15 $BUCKET\nbin/hadoop s3guard uploads $BUCKET\nbin/hadoop s3guard diff $BUCKET/\nbin/hadoop s3guard prune -minutes 10 $BUCKET/\nbin/hadoop s3guard import $BUCKET/\nbin/hadoop fs -ls $BUCKET/\nbin/hadoop fs -ls $BUCKET/file\nbin/hadoop fs -rm -R -f $BUCKET/dir-no-trailing\nbin/hadoop fs -rm -R -f $BUCKET/dir-trailing/\nbin/hadoop fs -rm $BUCKET/\nbin/hadoop fs -touchz $BUCKET/file\n# expect I/O error as root dir is not empty\nbin/hadoop fs -rm -r $BUCKET/\nbin/hadoop fs -rm -r $BUCKET/\\*\n# now success\nbin/hadoop fs -rm -r $BUCKET/\n\nbin/hadoop fs -mkdir $BUCKET/dir-no-trailing\n# fails with S3Guard\nbin/hadoop fs -mkdir $BUCKET/dir-trailing/\nbin/hadoop fs -touchz $BUCKET/file\nbin/hadoop fs -ls $BUCKET/\nbin/hadoop fs -mv $BUCKET/file $BUCKET/file2\n# expect \"No such file or directory\"\nbin/hadoop fs -stat $BUCKET/file\nbin/hadoop fs -stat $BUCKET/file2\nbin/hadoop fs -mv $BUCKET/file2 $BUCKET/dir-no-trailing\nbin/hadoop fs -stat $BUCKET/dir-no-trailing/file2\n# treated the same as the file stat\nbin/hadoop fs -stat $BUCKET/dir-no-trailing/file2/\nbin/hadoop fs -ls $BUCKET/dir-no-trailing/file2/\nbin/hadoop fs -ls $BUCKET/dir-no-trailing\n# expect a \"0\" here:\nbin/hadoop fs -test -d $BUCKET/dir-no-trailing && echo $?\n# expect a \"1\" here:\nbin/hadoop fs -test -d $BUCKET/dir-no-trailing/file2 && echo $?\n# will return NONE unless bucket has checksums enabled\nbin/hadoop fs -checksum $BUCKET/dir-no-trailing/file2\n# expect \"etag\" + a long string\nbin/hadoop fs -D fs.s3a.etag.checksum.enabled=true -checksum $BUCKET/dir-no-trailing/file2\n```\n\n### Other tests\n\n* Whatever applications you have which use S3A: build and run them before the upgrade,\nThen see if complete successfully in roughly the same time once the upgrade is applied.\n* Test any third-party endpoints you have access to.\n* Try different regions (especially a v4 only region), and encryption settings.\n* Any performance tests you have can identify slowdowns, which can be a sign\n of changed behavior in the SDK (especially on stream reads and writes).\n* If you can, try to test in an environment where a proxy is needed to talk\nto AWS services.\n* Try and get other people, especially anyone with their own endpoints,\n apps or different deployment environments, to run their own tests.\n\n### Dealing with Deprecated APIs and New Features\n\nA Jenkins run should tell you if there are new deprecations.\nIf so, you should think about how to deal with them.\n\nMoving to methods and APIs which weren't in the previous SDK release makes it\nharder to roll back if there is a problem; but there may be good reasons\nfor the deprecation.\n\nAt the same time, there may be good reasons for staying with the old code.\n\n* AWS have embraced the builder pattern for new operations; note that objects\nconstructed this way often have their (existing) setter methods disabled; this\nmay break existing code.\n* New versions of S3 calls (list v2, bucket existence checks, bulk operations)\nmay be better than the previous HTTP operations & APIs, but they may not work with\nthird-party endpoints, so can only be adopted if made optional, which then\nadds a new configuration option (with docs, testing, ...). A change like that\nmust be done in its own patch, with its new tests which compare the old\nvs new operations.\n\n### Committing the patch\n\nWhen the patch is committed: update the JIRA to the version number actually\nused; use that title in the commit message.\n\nBe prepared to roll-back, re-iterate or code your way out of a regression.\n\nThere may be some problem which surfaces with wider use, which can get\nfixed in a new AWS release, ","isInternal":false,"tokens":12113,"sizeBytes":50006},{"name":"README.md","path":"hadoop-tools/hadoop-resourceestimator/README.md","rawUrl":"https://raw.githubusercontent.com/apache/hadoop/HEAD/hadoop-tools/hadoop-resourceestimator/README.md","title":"hadoop-resourceestimator Documentation","category":"plugin-manifest","format":"markdown","content":"Resource Estimator Service\n==========================\n\nResource Estimator Service can parse the history logs of production jobs, extract their resource consumption skylines in the past runs and predict their resource requirements for the new run.\n\n## Current Status\n\n * Support [Hadoop YARN ResourceManager](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) logs.\n * In-memory store for parsed history resource skyline and estimation.\n * A [Linear Programming](https://github.com/optimatika/ojAlgo) based estimator.\n * Provides REST interface to parse logs, query history store and estimations.\n\n## Upcoming features\n\n * UI to query history and edit and save estimations.\n * Persisent store implementation for store (either DB based or distributed key-value like HBase).\n * Integrate directly with the [Hadoop YARN Reservation System](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ReservationSystem.html) to make a recurring reservation based on the estimated resources.\n\nRefer to the [design document](https://issues.apache.org/jira/secure/attachment/12886714/ResourceEstimator-design-v1.pdf) for more details.\n","isInternal":false,"tokens":253,"sizeBytes":1168}],"systemPromptSnippet":"\n\n\n\n# Hadoop-Aliyun module: Integration with Aliyun Web Services\n\n\n\n## Overview\n\nThe `hadoop-aliyun` module provides support for Aliyun integration with\n[Aliyun Object Storage Service (Aliyun OSS)](https://www.aliyun.com/product/oss).\nThe generated JAR file, `hadoop-aliyun.jar` also declares a transitive\ndependency on all external artifacts which are needed for this support — enabling\ndownstream applications to easily use this support.\n\nTo make it part of Apache Hadoop's default classpath, simply make sure\nthat HADOOP_OPTIONAL_TOOLS in hadoop-env.sh has 'hadoop-aliyun' in the list.\n\n### Features\n\n* Read and write data stored in Aliyun OSS.\n* Present a hierarchical file system view by implementing the standard Hadoop\n[`FileSystem`](../api/org/apache/hadoop/fs/FileSystem.html) interface.\n* Can act as a source of data in a MapReduce job, or a sink.\n\n### Warning #1: Object Stores are not filesystems.\n\nAliyun OSS is an example of \"an object store\". In order to achieve scalability\nand especially high availability, Aliyun OSS has relaxed some of the constraints\nwhich classic \"POSIX\" filesystems promise.\n\n\n\nSpecifically\n\n1. Atomic operations: `delete()` and `rename()` are implemented by recursive\nfile-by-file operations. They take time at least proportional to the number of files,\nduring which time partial updates may be visible. `delete()` and `rename()`\ncan not guarantee atomicity. If the operations are interrupted, the filesystem\nis left in an intermediate state.\n2. File owner and group are persisted, but the permissions model is not enforced.\nAuthorization occurs at the level of the entire Aliyun account via\n[Aliyun Resource Access Management (Aliyun RAM)](https://www.aliyun.com/product/ram).\n3. Directory last access time is not tracked.\n4. The append operation is not supported.\n\n### Warning #2: Directory last access time is not tracked,\nFeatures of Hadoop relying on this can have unexpected behaviour. E.g. the\nAggregatedLogDeletionService of YARN will not remove the appropriate log files.\n\n### Warning #3: Your Aliyun credentials are valuable\n\nYour Aliyun credentials not only pay for services, they offer read and write\naccess to the data. Anyone with the account can not only read your datasets\n— they can delete them.\n\nDo not inadvertently share these credentials through means such as\n1. Checking in to SCM any configuration files containing the secrets.\n2. Logging them to a console, as they invariably end up being seen.\n3. Defining filesystem URIs with the credentials in the URL, such as\n`oss://accessKeyId:accessKeySecret@directory/file`. They will end up in\nlogs and error messages.\n4. Including the secrets in bug reports.\n\nIf you do any of these: change your credentials immediately!\n\n### Warning #4: The Aliyun OSS client provided by Aliyun E-MapReduce are different from this implementation\n\nSpecifically: on Aliyun E-MapReduce, `oss://` is also supported but with\na different implementation. If you are using Aliyun E-MapReduce,\nfollow these instructions —and be aware that all issues related to Aliyun\nOSS integration in E-MapReduce can only be addressed by Aliyun themselves:\nplease raise your issues with them.\n\n## OSS\n\n### Authentication properties\n\n \n fs.oss.accessKeyId\n Aliyun access key ID\n \n\n \n fs.oss.accessKeySecret\n Aliyun access key secret\n \n\n \n fs.oss.credentials.provider\n \n Class name of a credentials provider that implements\n com.aliyun.oss.common.auth.CredentialsProvider. Omit if using access/secret keys\n or another authentication mechanism. The specified class must provide an\n accessible constructor accepting java.net.URI and\n org.apache.hadoop.conf.Configuration, or an accessible default constructor.\n \n \n\n### Other properties\n\n \n fs.oss.endpoint\n Aliyun OSS endpoint to connect to. An up-to-date list is\n provided in the Aliyun OSS Documentation.\n \n \n\n \n fs.oss.assumed.role.arn\n \n Role ARN for the role to be assumed.\n Required if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.sts.endpoint\n \n STS Token Service endpoint.\n Required if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.session.name\n \n \n Session name for the assumed role, must be valid characters\n according to Aliyun API. It is optional, will be generated by\n oss java sdk if it is empty.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.session.duration\n \n \n Duration of assumed roles before it is expired. Default is 30 minutes.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.assumed.role.sts.expiredFactor\n \n \n Sts token will be refreshed after (expiredFactor * duration) seconds.\n Only used if the fs.oss.credentials.provider is\n org.apache.hadoop.fs.aliyun.oss.AssumedRoleCredentialProvider.\n \n \n\n \n fs.oss.proxy.host\n Hostname of the (optinal) proxy server for Aliyun OSS connection\n \n\n \n fs.oss.proxy.port\n Proxy server port\n \n\n \n fs.oss.proxy.username\n Username for authenticating with proxy server\n \n\n \n fs.oss.proxy.password\n Password for authenticating with proxy server.\n \n\n \n fs.oss.proxy.domain\n Domain for authenticating with proxy server.\n \n\n \n fs.oss.proxy.workstation\n Workstation for authenticating with proxy server.\n \n\n \n fs.oss.attempts.maximum\n 20\n How many times we should retry commands on transient errors.\n \n\n \n fs.oss.connection.establish.timeout\n 50000\n Connection setup timeout in milliseconds.\n \n\n \n fs.oss.connection.timeout\n 200000\n Socket connection timeout in milliseconds.\n \n\n \n fs.oss.paging.maximum\n 1000\n How many keys to request from Aliyun OSS when doing directory listings at a time.\n \n \n\n \n fs.oss.multipart.upload.size\n 10485760\n Size of each of multipart pieces in bytes.\n \n\n \n fs.oss.multipart.upload.threshold\n 20971520\n Minimum size in bytes before we start a multipart uploads or copy.\n \n\n \n fs.oss.multipart.download.size\n 102400/value>\n Size in bytes in each request from ALiyun OSS.\n \n\n \n fs.oss.buffer.dir\n Comma separated list of directories to buffer OSS data before uploading to Aliyun OSS\n \n\n \n fs.oss.acl.default\n \n Set a canned ACL for bucket. Value may be private, public-read, public-read-write.\n \n \n\n \n fs.oss.server-side-encryption-algorithm\n \n Specify a server-side encryption algorithm for oss: file system.\n Unset by default, and the only other currently allowable value is AES256.\n \n \n\n \n fs.oss.connection.maximum\n 32\n Number of simultaneous connections to oss.\n \n\n \n fs.oss.connection.secure.enabled\n true\n Connect to oss over ssl or not, true by default.\n \n\n## Testing the hadoop-aliyun Module\n\nTo test `oss://` filesystem client, two files which pass in authentication\ndetails to the test runner are needed.\n\n1. `auth-keys.xml`\n2. `core-site.xml`\n\nThose two configuration files must be put into\n`hadoop-tools/hadoop-aliyun/src/test/resources`.\n\n### `core-site.xml`\n\nThis file pre-exists and sources the configurations created in `auth-keys.xml`.\n\nFor most cases, no modification is needed, unless a specific, non-default property\nneeds to be set during the testing.\n\n### `auth-keys.xml`\n\nThis file triggers the testing of Aliyun OSS module. Without this file,\n*none of the tests in this module will be executed*\n\nIt contains the access key Id/secret and proxy information that are needed to\nconnect to Aliyun OSS, and an OSS bucket URL should be also provided.\n\n1. `test.fs.oss.name` : the URL of the bucket for Aliyun OSS tests\n\nThe contents of the bucket will be cleaned during the testing process, so\ndo not use the bucket for any purpose other than testing.\n\n### Run Hadoop contract tests\nCreate file `contract-test-options.xml` under `/test/resources`. If a\nspecific file `fs.contract.test.fs.oss` test path is not defined, those\ntests will be skipped. Credentials are also needed to run any of those\ntests, they can be copied from `auth-keys.xml` or through direct\nXInclude inclusion. Here is an example of `contract-test-options.xml`:\n\n \n \n \n\n \n\n \n fs.contract.test.fs.oss\n oss://spark-tests\n \n\n \n fs.oss.impl\n org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem\n \n\n \n fs.oss.endpoint\n oss-cn-hangzhou.aliyuncs.com\n \n\n \n fs.oss.buffer.dir\n /tmp/oss\n \n\n \n fs.oss.multipart.download.size\n 102400\n \n \n\n\n\n\n# Working with IAM Assumed Roles\n\n\n\nAWS [\"IAM Assumed Roles\"](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html)\nallows applications to change the AWS role with which to authenticate with AWS services.\nThe assumed roles can have different rights from the main user login.\n\nThe S3A connector supports assumed roles for authentication with AWS.\nA full set of login credentials must be provided, which will be used\nto obtain the assumed role and refresh it regularly.\nBy using per-filesystem configuration, it is possible to use different\nassumed roles for different buckets.\n\n*IAM Assumed Roles are unlikely to be supported by third-party systems\nsupporting the S3 APIs.*\n\n## Using IAM Assumed Roles\n\n### Before You Begin\n\nThis document assumes you know about IAM Assumed roles, what they\nare, how to configure their policies, etc.\n\n* You need a role to assume, and know its \"ARN\".\n* You need a pair of long-lived IAM User credentials, not the root account set.\n* Have the AWS CLI installed, and test that it works there.\n* Give the role access to S3, and, if using S3Guard, to DynamoDB.\n* For working with data encrypted with SSE-KMS, the role must\nhave access to the appropriate KMS keys.\n\nTrying to learn how IAM Assumed Roles work by debugging stack traces from\nthe S3A client is \"suboptimal\".\n\n### How the S3A connector support IAM Assumed Roles.\n\nTo use assumed roles, the client must be configured to use the\n*Assumed Role Credential Provider*, `org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider`,\nin the configuration option `fs.s3a.aws.credentials.provider`.\n\nThis AWS Credential provider will read in the `fs.s3a.assumed.role` options needed to connect to the\nSecurity Token Service [Assumed Role API](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html),\nfirst authenticating with the full credentials, then assuming the specific role\nspecified. It will then refresh this login at the configured rate of\n`fs.s3a.assumed.role.session.duration`\n\nTo authenticate with the [AWS STS service](https://docs.aws.amazon.com/STS/latest/APIReference/Welcome.html)\nboth for the initial credential retrieval\nand for background refreshes, a different credential provider must be\ncreated, one which uses long-lived credentials (secret keys, environment variables).\nShort lived credentials (e.g other session tokens, EC2 instance credentials) cannot be used.\n\nA list of providers can be set in `s.s3a.assumed.role.credentials.provider`;\nif unset the standard `BasicAWSCredentialsProvider` credential provider is used,\nwhich uses `fs.s3a.access.key` and `fs.s3a.secret.key`.\n\nNote: although you can list other AWS credential providers in to the\nAssumed Role Credential Provider, it can only cause confusion.\n\n### Configuring Assumed Roles\n\nTo use assumed roles, the S3A client credentials provider must be set to\nthe `AssumedRoleCredentialProvider`, and `fs.s3a.assumed.role.arn` to\nthe previously created ARN.\n\n```xml\n\n fs.s3a.aws.credentials.provider\n org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n\n\n\n fs.s3a.assumed.role.arn\n arn:aws:iam::90066806600238:role/s3-restricted\n\n```\n\nThe STS service itself needs the caller to be authenticated, *which can\nonly be done with a set of long-lived credentials*.\nThis means the normal `fs.s3a.access.key` and `fs.s3a.secret.key`\npair, environment variables, or some other supplier of long-lived secrets.\n\nThe default is the `fs.s3a.access.key` and `fs.s3a.secret.key` pair.\nIf you wish to use a different authentication mechanism, set it in the property\n`fs.s3a.assumed.role.credentials.provider`.\n\n```xml\n\n fs.s3a.assumed.role.credentials.provider\n org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\n\n```\n\nRequirements for long-lived credentials notwithstanding, this option takes the\nsame values as `fs.s3a.aws.credentials.provider`.\n\nThe safest way to manage AWS secrets is via\n[Hadoop Credential Providers](index.html#hadoop_credential_providers).\n\n### Assumed Role Configuration Options\n\nHere are the full set of configuration options.\n\n```xml\n\n fs.s3a.assumed.role.arn\n \n \n AWS ARN for the role to be assumed.\n Required if the fs.s3a.aws.credentials.provider contains\n org.apache.hadoop.fs.s3a.AssumedRoleCredentialProvider\n \n\n\n\n fs.s3a.assumed.role.session.name\n \n \n Session name for the assumed role, must be valid characters according to\n the AWS APIs.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n If not set, one is generated from the current Hadoop/Kerberos username.\n \n\n\n\n fs.s3a.assumed.role.policy\n \n \n JSON policy to apply to the role.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.session.duration\n 30m\n \n Duration of assumed roles before a refresh is attempted.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n Range: 15m to 1h\n \n\n\n\n fs.s3a.assumed.role.sts.endpoint\n \n \n AWS Security Token Service Endpoint. If unset, uses the default endpoint.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.sts.endpoint.region\n us-west-1\n \n AWS Security Token Service Endpoint's region;\n Needed if fs.s3a.assumed.role.sts.endpoint points to an endpoint\n other than the default one and the v4 signature is used.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n \n\n\n\n fs.s3a.assumed.role.credentials.provider\n org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\n \n List of credential providers to authenticate with the STS endpoint and\n retrieve short-lived role credentials.\n Only used if AssumedRoleCredentialProvider is the AWS credential provider.\n If unset, uses \"org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider\".\n \n\n```\n\n## Restricting S3A operations through AWS Policies\n\nThe S3A client needs to be granted specific permissions in order\nto work with a bucket.\nHere is a non-normative list of the permissions which must be granted\nfor FileSystem operations to work.\n\n*Disclaimer* The specific set of actions which the S3A connector needs\nwill change over time.\n\nAs more operations are added to the S3A connector, and as the\nmeans by which existing operations are implemented change, the\nAWS actions which are required by the client will change.\n\nThese lists represent the minimum actions to which the client's principal\nmust have in order to work with a bucket.\n\n\n### Read Access Permissions\n\nPermissions which must be granted when reading from a bucket:\n\n\n```\ns3:Get*\ns3:ListBucket\n```\n\nWhen using S3Guard, the client needs the appropriate\nDynamoDB access permissions\n\nTo use SSE-KMS encryption, the client needs the\nSSE-KMS Permissions to access the\nKMS key(s).\n\n### Write Access Permissions\n\nThese permissions must all be granted for write access:\n\n```\ns3:Get*\ns3:Delete*\ns3:Put*\ns3:ListBucket\ns3:ListBucketMultipartUploads\ns3:AbortMultipartUpload\n```\n\n### SSE-KMS Permissions\n\nWhen to read data encrypted using SSE-KMS, the client must have\n `kms:Decrypt` permission for the specific key a file was encrypted with.\n\n```\nkms:Decrypt\n```\n\nTo write data using SSE-KMS, the client must have all the following permissions.\n\n```\nkms:Decrypt\nkms:GenerateDataKey\n```\n\nThis includes renaming: renamed files are encrypted with the encryption key\nof the current S3A client; it must decrypt the source file first.\n\nIf the caller doesn't have these permissions, the operation will fail with an\n`AccessDeniedException`: the S3 Store does not provide the specifics of\nthe cause of the failure.\n\n### S3Guard Permissions\n\nTo use S3Guard, all clients must have a subset of the\n[AWS DynamoDB Permissions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/api-permissions-reference.html).\n\nTo work with buckets protected with S3Guard, the client must have\nall the following rights on the DynamoDB Table used to protect that bucket.\n\n```\ndynamodb:BatchGetItem\ndynamodb:BatchWriteItem\ndynamodb:DeleteItem\ndynamodb:DescribeTable\ndynamodb:GetItem\ndynamodb:PutItem\ndynamodb:Query\ndynamodb:UpdateItem\n```\n\nThis is true, *even if the client only has read access to the data*.\n\nFor the `hadoop s3guard` table management commands, _extra_ permissions are required:\n\n```\ndynamodb:CreateTable\ndynamodb:DescribeLimits\ndynamodb:DeleteTable\ndynamodb:Scan\ndynamodb:TagResource\ndynamodb:UntagResource\ndynamodb:UpdateTable\n```\n\nWithout these permissions, tables cannot be created, destroyed or have their IO capacity\nchanged through the `s3guard set-capacity` call.\nThe `dynamodb:Scan` permission is needed for `s3guard prune`\n\nThe `dynamodb:CreateTable` permission is needed by a client it tries to\ncreate the DynamoDB table on startup, that is\n`fs.s3a.s3guard.ddb.table.create` is `true` and the table does not already exist.\n\n### Mixed Permissions in a single S3 Bucket\n\nMixing permissions down the \"directory tree\" is limited\nonly to the extent of supporting writeable directories under\nread-only parent paths.\n\n*Disclaimer:* When a client lacks write access up the entire\ndirectory tree, there are no guarantees of consistent filesystem\nviews or operations.\n\nParticular troublespots are \"directory markers\" and\nfailures of non-atomic operations, particularly `rename()` and `delete()`.\n\nA directory marker such as `/users/` will not be deleted if the user `alice`\ncreates a directory `/users/alice` *and* she only has access to `/users/alice`.\n\nWhen a path or directory is deleted, the parent directory may not exist afterwards.\nIn the example above, if `alice` deletes `/users/alice` and there are no\nother entries under `/users/alice`, then the directory marker `/users/` cannot\nbe created. The directory `/users` will not exist in listings,\n`getFileStatus(\"/users\")` or similar.\n\nRename will fail if it cannot delete the items it has just copied, that is\n`rename(read-only-source, writeable-dest)` will fail —but only after\nperforming the COPY of the data.\nEven though the operation failed, for a single file copy, the destination\nfile will exist.\nFor a directory copy, only a partial copy of the source data may take place\nbefore the permission failure is raised.\n\n\n*S3Guard*: if [S3Guard](s3guard.html) is used to manage the directory listings,\nthen after partial failures of rename/copy the DynamoDB tables can get out of sync.\n\n### Example: Read access to the base, R/W to the path underneath\n\nThis example has the base bucket read only, and a directory underneath,\n`/users/alice/` granted full R/W access.\n\n```json\n{\n \"Version\" : \"2012-10-17\",\n \"Statement\" : [ {\n \"Sid\" : \"4\",\n \"Effect\" : \"Allow\",\n \"Action\" : [\n \"s3:ListBucket\",\n \"s3:ListBucketMultipartUploads\",\n \"s3:Get*\"\n ],\n \"Resource\" : \"arn:aws:s3:::example-bucket/*\"\n }, {\n \"Sid\" : \"5\",\n \"Effect\" : \"Allow\",\n \"Action\" : [\n \"s3:Get*\",\n \"s3:PutObject\",\n \"s3:DeleteObject\",\n \"s3:AbortMultipartUpload\",\n \"s3:ListMultipartUploadParts\" ],\n \"Resource\" : [\n \"arn:aws:s3:::example-bucket/users/alice/*\",\n \"arn:aws:s3:::example-bucket/users/alice\",\n \"arn:aws:s3:::example-bucket/users/alice/\"\n ]\n } ]\n}\n```\n\nNote how three resources are provided to represent the path `/users/alice`\n\n| Path | Matches |\n|-------|----------|\n| `/users/alice` | Any file `alice` created under `/users` |\n| `/users/alice/` | The directory marker `alice/` created under `/users` |\n| `/users/alice/*` | All files and directories under the path `/users/alice` |\n\nNote that the resource `arn:aws:s3:::example-bucket/users/alice*` cannot\nbe used to refer to all of these paths, because it would also cover\nadjacent paths like `/users/alice2` and `/users/alicebob`.\n\n\n## Troubleshooting Assumed Roles\n\n1. Make sure the role works and the user trying to enter it can do so from AWS\nthe command line before trying to use the S3A client.\n1. Try to access the S3 bucket with reads and writes from the AWS CLI.\n1. With the Hadoop configuration set too use the role,\n try to read data from the `hadoop fs` CLI:\n`hadoop fs -ls -p s3a://bucket/`\n1. With the hadoop CLI, try to create a new directory with a request such as\n`hadoop fs -mkdirs -p s3a://bucket/path/p1/`\n\n\n### IOException: \"Unset property fs.s3a.assumed.role.arn\"\n\nThe Assumed Role Credential Provider is enabled, but `fs.s3a.assumed.role.arn` is unset.\n\n```\njava.io.IOException: Unset property fs.s3a.assumed.role.arn\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:76)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n```\n\n### \"Not authorized to perform sts:AssumeRole\"\n\nThis can arise if the role ARN set in `fs.s3a.assumed.role.arn` is invalid\nor one to which the caller has no access.\n\n```\njava.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Not authorized to perform sts:AssumeRole (Service: AWSSecurityTokenService; Status Code: 403;\n Error Code: AccessDenied; Request ID: aad4e59a-f4b0-11e7-8c78-f36aaa9457f6):AccessDenied\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n```\n\n### \"Roles may not be assumed by root accounts\"\n\nYou can't assume a role with the root account of an AWS account;\nyou need to create a new user and give it the permission to change into\nthe role.\n\n```\njava.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;\n Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2):\n No AWS Credentials provided by AssumedRoleCredentialProvider :\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts. (Service: AWSSecurityTokenService;\n Status Code: 403; Error Code: AccessDenied; Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n ... 22 more\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n Roles may not be assumed by root accounts.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied;\n Request ID: e86dfd8f-e758-11e7-88e7-ad127c04b5e2)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n```\n\n### \"Assume Role session duration should be in the range of 15min - 1Hr\"\n\nThe value of `fs.s3a.assumed.role.session.duration` is out of range.\n\n```\njava.lang.IllegalArgumentException: Assume Role session duration should be in the range of 15min\n- 1Hr\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$Builder.withRoleSessionDurationSeconds(STSAssumeRoleSessionCredentialsProvider.java:437)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:86)\n```\n\n\n### `MalformedPolicyDocumentException` \"The policy is not in the valid JSON format\"\n\n\nThe policy set in `fs.s3a.assumed.role.policy` is not valid according to the\nAWS specification of Role Policies.\n\n```\nrg.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n The policy is not in the valid JSON format. (Service: AWSSecurityTokenService; Status Code: 400;\n Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c):\n MalformedPolicyDocument: The policy is not in the valid JSON format.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\nCaused by: com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n The policy is not in the valid JSON format.\n (Service: AWSSecurityTokenService; Status Code: 400;\n Error Code: MalformedPolicyDocument; Request ID: baf8cb62-f552-11e7-9768-9df3b384e40c)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 19 more\n```\n\n### `MalformedPolicyDocumentException` \"Syntax errors in policy\"\n\nThe policy set in `fs.s3a.assumed.role.policy` is not valid JSON.\n\n```\norg.apache.hadoop.fs.s3a.AWSBadRequestException:\nInstantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.MalformedPolicyDocumentException:\n Syntax errors in policy. (Service: AWSSecurityTokenService;\n Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45):\n MalformedPolicyDocument: Syntax errors in policy.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: MalformedPolicyDocument;\n Request ID: 24a281e8-f553-11e7-aa91-a96becfb4d45)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 19 more\n```\n\n### `IOException`: \"AssumedRoleCredentialProvider cannot be in fs.s3a.assumed.role.credentials.provider\"\n\nYou can't use the Assumed Role Credential Provider as the provider in\n`fs.s3a.assumed.role.credentials.provider`.\n\n```\njava.io.IOException: AssumedRoleCredentialProvider cannot be in fs.s3a.assumed.role.credentials.provider\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:86)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n```\n\n### `AWSBadRequestException`: \"not a valid key=value pair\"\n\n\nThere's an space or other typo in the `fs.s3a.access.key` or `fs.s3a.secret.key` values used for the\ninner authentication which is breaking signature creation.\n\n```\n org.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n on : com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign) in Authorization header:\n 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date.\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code:\n IncompleteSignature; Request ID: c4a8841d-f556-11e7-99f9-af005a829416):IncompleteSignature:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign)\n in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date,\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature;\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 'valid/20180109/us-east-1/sts/aws4_request' not a valid key=value pair (missing equal-sign)\n in Authorization header: 'AWS4-HMAC-SHA256 Credential=not valid/20180109/us-east-1/sts/aws4_request,\n SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;host;user-agent;x-amz-date,\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: IncompleteSignature;\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 25 more\n```\n\n### `AccessDeniedException/InvalidClientTokenId`: \"The security token included in the request is invalid\"\n\nThe credentials used to authenticate with the AWS Security Token Service are invalid.\n\n```\n[ERROR] Failures:\n[ERROR] java.nio.file.AccessDeniedException: : Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on :\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n The security token included in the request is invalid.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId;\n Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7):InvalidClientTokenId\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:215)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\nThe security token included in the request is invalid.\n (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId;\n Request ID: 74aa7f8a-f557-11e7-850c-33d05b3658d7)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:127)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:116)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 25 more\n```\n\n### `AWSSecurityTokenServiceExceptiond`: \"Member must satisfy regular expression pattern: `[\\w+=,.@-]*`\"\n\n\nThe session name, as set in `fs.s3a.assumed.role.session.name` must match the wildcard `[\\w+=,.@-]*`.\n\nIf the property is unset, it is extracted from the current username and then sanitized to\nmatch these constraints.\nIf set explicitly, it must be valid.\n\n```\norg.apache.hadoop.fs.s3a.AWSBadRequestException: Instantiate org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider on\n com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint: Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n Request ID: 7c437acb-f55d-11e7-9ad8-3b5e4f701c20):ValidationError:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint: Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:209)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:616)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:520)\n at org.apache.hadoop.fs.s3a.DefaultS3ClientFactory.createS3Client(DefaultS3ClientFactory.java:52)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:252)\n at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3354)\n at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:474)\n at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)\n\nCaused by: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:\n 1 validation error detected: Value 'Session Names cannot Hava Spaces!' at 'roleSessionName'\n failed to satisfy constraint:\n Member must satisfy regular expression pattern: [\\w+=,.@-]*\n (Service: AWSSecurityTokenService; Status Code: 400; Error Code: ValidationError;\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)\n at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:321)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:37)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:76)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:73)\n at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:256)\n at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:212)\n at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:153)\n at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:299)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.getCredentials(AssumedRoleCredentialProvider.java:135)\n at org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider.(AssumedRoleCredentialProvider.java:124)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)\n at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n at java.lang.reflect.Constructor.newInstance(Constructor.java:423)\n at org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:583)\n ... 26 more\n```\n\n\n### `java.nio.file.AccessDeniedException` within a FileSystem API call\n\nIf an operation fails with an `AccessDeniedException`, then the role does not have\nthe permission for the S3 Operation invoked during the call.\n\n```\njava.nio.file.AccessDeniedException: s3a://bucket/readonlyDir:\n rename(s3a://bucket/readonlyDir, s3a://bucket/renameDest)\n on s3a://bucket/readonlyDir:\n com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied\n (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1;\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=),\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=:AccessDenied\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:216)\n at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:143)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:853)\n ...\nCaused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied\n (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2805F2ABF5246BB1;\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=),\n S3 Extended Request ID: iEXDVzjIyRbnkAc40MS8Sjv+uUQNvERRcqLsJsy9B0oyrjHLdkRKwJ/phFfA17Kjn483KSlyJNw=\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1638)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1303)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1055)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)\n at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)\n at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)\n at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4229)\n at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4176)\n at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2066)\n at com.amazonaws.services.s3.AmazonS3Client.deleteObject(AmazonS3Client.java:2052)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObject$7(S3AFileSystem.java:1338)\n at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:314)\n at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:280)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObject(S3AFileSystem.java:1334)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:1657)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:1046)\n at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:851)\n```\n\nThis is the policy restriction behaving as intended: the caller is trying to\nperform an action which is forbidden.\n\n1. If a policy has been set in `fs.s3a.assumed.role.policy` then it must declare *all*\npermissions which the caller is allowed to perform. The existing role policies\nact as an outer constraint on what the caller can perform, but are not inherited.\n\n1. If the policy for a bucket is set up with complex rules on different paths,\ncheck the path for the operation.\n\n1. The policy may have omitted one or more actions which are required.\nMake sure that all the read and write permissions are allowed for any bucket/path\nto which data is being written to, and read permissions for all\nbuckets read from.\n\nIf the bucket is using SSE-KMS to encrypt data:\n\n1. The caller must have the `kms:Decrypt` permission to read the data.\n1. The caller needs `kms:Decrypt` and `kms:GenerateDataKey`.\n\nWithout permissions, the request fails *and there is no explicit message indicating\nthat this is an encryption-key issue*.\n\n### `AccessDeniedException` + `AmazonDynamoDBException`\n\n```\njava.nio.file.AccessDeniedException: bucket1:\n com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException:\n User: arn:aws:sts::980678866538:assumed-role/s3guard-test-role/test is not authorized to perform:\n dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-1:980678866538:table/bucket1\n (Service: AmazonDynamoDBv2; Status Code: 400;\n```\n\nThe caller is trying to access an S3 bucket which uses S3Guard, but the caller\nlacks the relevant DynamoDB access permissions.\n\nThe `dynamodb:DescribeTable` operation is the first one used in S3Guard to access,\nthe DynamoDB table, so it is often the first to fail. It can be a sign\nthat the role has no permissions at all to access the table named in the exception,\nor just that this specific permission has been omitted.\n\nIf the role policy requested for the assumed role didn't ask for any DynamoDB\npermissions, this is where all attempts to work with a S3Guarded bucket will\nfail. Check the value of `fs.s3a.assumed.role.policy`\n\n\n\n\n# S3A Committers: Architecture and Implementation\n\n\n\nThis document covers the architecture and implementation details of the S3A committers.\n\nFor information on using the committers, see [the S3A Committers](./committer.html).\n\n\n## Problem: Efficient, reliable commits of work to consistent S3 buckets\n\n\nThe standard commit algorithms (the `FileOutputCommitter` and its v1 and v2 algorithms)\nrely on directory rename being an `O(1)` atomic operation: callers output their\nwork to temporary directories in the destination filesystem, then\nrename these directories to the final destination as way of committing work.\nThis is the perfect solution for committing work against any filesystem with\nconsistent listing operations and where the `FileSystem.rename()` command\nis an atomic `O(1)` operation.\n\nUsing rename allows individual tasks to work in temporary directories, with the\nrename as the atomic operation can be used to explicitly commit tasks and\nultimately the entire job. Because the cost of the rename is low, it can be\nperformed during task and job commits with minimal delays. Note that HDFS\nwill lock the namenode metadata during the rename operation, so all rename() calls\nwill be serialized. However, as they only update the metadata of two directory\nentries, the duration of the lock is low.\n\nIn contrast to a \"real\" filesystem, Amazon's S3A object store, similar to\nmost others, does not support `rename()` at all. A hash operation on the filename\ndetermines the location of of the data —there is no separate metadata to change.\nTo mimic renaming, the Hadoop S3A client has to copy the data to a new object\nwith the destination filename, then delete the original entry. This copy\ncan be executed server-side, but as it does not complete until the in-cluster\ncopy has completed, it takes time proportional to the amount of data.\n\nThe rename overhead is the most visible issue, but it is not the most dangerous.\nThat is the fact that path listings have no consistency guarantees, and may\nlag the addition or deletion of files.\nIf files are not listed, the commit operation will *not* copy them, and\nso they will not appear in the final output.\n\nThe solution to this problem is closely coupled to the S3 protocol itself:\ndelayed completion of multi-part PUT operations\n\nThat is: tasks write all data as multipart uploads, *but delay the final\ncommit action until until the final, single job commit action.* Only that\ndata committed in the job commit action will be made visible; work from speculative\nand failed tasks will not be instantiated. As there is no rename, there is no\ndelay while data is copied from a temporary directory to the final directory.\nThe duration of the commit will be the time needed to determine which commit operations\nto construct, and to execute them.\n\n\n## Terminology\n\n* *Job*: a potentially parallelized query/operation to execute. The execution\nof a job: the division of work into tasks and the management of their completion,\nis generally executed in a single process.\n\nThe output of a Job is made visible to other stages in a larger operation\nsequence or other applications if the job *completes successfully*.\n\n* *Job Driver*. Not sure quite what term to use here. Whatever process schedules\ntask execution, tracks success/failures and, determines when all the work has been\nprocessed and then commits the output. It may also determine that a job\nhas failed and cannot be recovered, in which case the job is aborted.\nIn MR and Tez, this is inside the YARN application master.\nIn Spark it is the driver, which can run in the AM, the YARN client, or other\nplaces (e.g Livy?).\n\n* *Final directory*: the directory into which the output of a job is placed\nso as to be visible.\n\n* *Task* a single operation within a job, on a single process, one which generates\none or more files.\nAfter a successful job completion, the data MUST be visible in the final directory.\nA task completes successfully if it generates all the output it expects to without\nfailing in some way (error in processing; network/process failure).\n\n* *Job Context* an instance of the class `org.apache.hadoop.mapreduce.JobContext`,\nwhich provides a read-only view of the Job for the Job Driver and tasks.\n\n* *Task Attempt Context* an instance of the class\n`org.apache.hadoop.mapreduce.TaskAttemptContext extends JobContext, Progressable`,\nwhich provides operations for tasks, such as getting and setting status,\nprogress and counter values.\n\n* *Task Working Directory*: a directory for exclusive access by a single task,\ninto which uncommitted work may be placed.\n\n* *Task Commit* The act of taking the output of a task, as found in the\nTask Working Directory, and making it visible in the final directory.\nThis is traditionally implemented via a `FileSystem.rename()` call.\n\n It is useful to differentiate between a *task-side commit*: an operation performed\n in the task process after its work, and a *driver-side task commit*, in which\n the Job driver performs the commit operation. Any task-side commit work will\n be performed across the cluster, and may take place off the critical part for\n job execution. However, unless the commit protocol requires all tasks to await\n a signal from the job driver, task-side commits cannot instantiate their output\n in the final directory. They may be used to promote the output of a successful\n task into a state ready for the job commit, addressing speculative execution\n and failures.\n\n* *Job Commit* The act of taking all successfully completed tasks of a job,\nand committing them. This process is generally non-atomic; as it is often\na serialized operation at the end of a job, its performance can be a bottleneck.\n\n* *Task Abort* To cancel a task such that its data is not committed.\n\n* *Job Abort* To cancel all work in a job: no task's work is committed.\n\n* *Speculative Task Execution/ \"Speculation\"* Running multiple tasks against the same\ninput dataset in parallel, with the first task which completes being the one\nwhich is considered successful. Its output SHALL be committed; the other task\nSHALL be aborted. There's a requirement that a task can be executed in parallel,\nand that the output of a task MUST NOT BE visible until the job is committed,\nat the behest of the Job driver. There is the expectation that the output\nSHOULD BE the same on each task, though that MAY NOT be the case. What matters\nis if any instance of a speculative task is committed, the output MUST BE\nconsidered valid.\n\nThere is an expectation that the Job Driver and tasks can communicate: if a task\nperforms any operations itself during the task commit phase, it shall only do\nthis when instructed by the Job Driver. Similarly, if a task is unable to\ncommunicate its final status to the Job Driver, it MUST NOT commit is work.\nThis is very important when working with S3, as some network partitions could\nisolate a task from the Job Driver, while the task retains access to S3.\n\n## The execution workflow\n\n\n**setup**:\n\n* A job is created, assigned a Job ID (YARN?).\n* For each attempt, an attempt ID is created, to build the job attempt ID.\n* `Driver`: a `JobContext` is created/configured\n* A committer instance is instantiated with the `JobContext`; `setupJob()` invoked.\n\n\n## The `FileOutputCommitter`\n\nThe standard commit protocols are implemented in\n `org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter`.\n\nThere are two algorithms, the \"v1\" designed to address failures and restarts\nof the MapReduce application master. The V2 algorithm cannot recover from failure\nexcept by re-executing the entire job. It does, however, propagate all its work\nto the output directory in the task commit. When working with object stores which\nmimic `rename()` by copy and delete, it is more efficient due to the reduced\nnumber of listings, copies and deletes, and, because these are executed in\ntask commits, eliminates the final `O(data)` pause at the end of all work.\nIt is still highly inefficient, but the inefficiencies are less visible as\nlarge pauses in execution.\n\n\nNotes\n\n* The v1 algorithm was implemented to handle MapReduce AM restarts, it was\n not used in Hadoop 1.x, whose JobTracker could not recover from failures.\n Historically then, it is the second version of a file commit algorithm.\n* Because the renames are considered to be fast, there is no logging\nof renames being in progress, or their duration.\n* Speculative execution is supported by having every task attempt write its\nuncommitted data to a task attempt directory. When a task is ready to commit,\nit must notify that job driver that it is ready to commit -the job driver\nwill then commit or abort the task.\n\n## Hadoop MR Commit algorithm \"1\"\n\n\nThe \"v1\" commit algorithm is the default commit algorithm in Hadoop 2.x;\nit was implemented as part of [MAPREDUCE-2702](https://issues.apache.org/jira/browse/MAPREDUCE-2702).\n\nThis algorithm is designed to handle a failure and restart of the Job driver,\nwith the restarted job driver only rerunning the incomplete tasks; the\noutput of the completed tasks is recovered for commitment when the restarted\njob completes.\n\nThere is a cost: the time to commit by recursive listing all files in all\ncommitted task directories, and renaming this.\n\nAs this is performed sequentially, time to commit is `O(files)`, which is\ngenerally `O(tasks)`.\n\n\n\n```python\n#Job Attempt Path is `$dest/_temporary/$appAttemptId/`\n\njobAttemptPath = '$dest/_temporary/$appAttemptId/'\n\n# Task Attempt Path is `$dest/_temporary/$appAttemptId/_temporary/$taskAttemptID`\ntaskAttemptPath = '$dest/_temporary/$appAttemptId/_temporary/$taskAttemptID'\n\n#Task committed path is `$dest/_temporary/$appAttemptId/$taskAttemptID`\ntaskCommittedPath = '$dest/_temporary/$appAttemptId/$taskAttemptID'\n```\n\nTasks write in/under the task attempt path.\n\n### Job Setup\n\n```python\nfs.mkdir(jobAttemptPath)\n```\n\n### Task Setup\n\nNone: directories are created on demand.\n\n\n### Task Commit\n\nRename task attempt path to task committed path.\n\n```python\ndef needsTaskCommit(fs, jobAttemptPath, taskAttemptPath, dest):\n return fs.exists(taskAttemptPath)\n\n\ndef commitTask(fs, jobAttemptPath, taskAttemptPath, dest):\n if fs.exists(taskAttemptPath) :\n fs.delete(taskCommittedPath, recursive=True)\n fs.rename(taskAttemptPath, taskCommittedPath)\n```\n\nOn a genuine filesystem this is an `O(1)` directory rename.\n\nOn an object store with a mimiced rename, it is `O(data)` for the copy,\nalong with overhead for listing and deleting all files (For S3, that's\n`(1 + files/500)` lists, and the same number of delete calls.\n\n\n### Task Abort\n\nDelete task attempt path.\n\n```python\ndef abortTask(fs, jobAttemptPath, taskAttemptPath, dest):\n fs.delete(taskAttemptPath, recursive=True)\n```\n\nOn a genuine filesystem this is an `O(1)` operation. On an object store,\nproportional to the time to list and delete files, usually in batches.\n\n\n### Job Commit\n\nMerge all files/directories in all task committed paths into final destination path.\nOptionally; create 0-byte `_SUCCESS` file in destination path.\n\n```python\ndef commitJob(fs, jobAttemptDir, dest):\n for committedTask in fs.listFiles(jobAttemptDir):\n mergePathsV1(fs, committedTask, dest)\n fs.touch(\"$dest/_SUCCESS\")\n```\n\n(See below for details on `mergePaths()`)\n\n\nA failure during job commit cannot be recovered from except by re-executing\nthe entire query:\n\n```python\ndef isCommitJobRepeatable() :\n return False\n```\n\nAccordingly, it is a failure point in the protocol. With a low number of files\nand fast rename/list algorithms, the window of vulnerability is low. At\nscale, the vulnerability increases. It could actually be reduced through\nparallel execution of the renaming of of committed tasks.\n\n\n### Job Abort\n\nDelete all data under job attempt path.\n\n```python\ndef abortJob(fs, jobAttemptDir, dest):\n fs.delete(jobAttemptDir, recursive = True)\n```\n\n### Job Cleanup\n\n```python\ndef cleanupJob(fs, dest):\n fs.delete('$dest/_temporary', recursive = True)\n```\n\n\n### Job Recovery Before `commitJob()`\n\nFor all committers, the recovery process takes place in the application\nmaster.\n1. The job history file of the previous attempt is loaded and scanned\nto determine which tasks were recorded as having succeeded.\n1. For each successful task, the job committer has its `recoverTask()` method\ninvoked with a `TaskAttemptContext` built from the previous attempt's details.\n1. If the method does not raise an exception, it is considered to have been\nrecovered, and not to be re-executed.\n1. All other tasks are queued for execution.\n\nFor the v1 committer, task recovery is straightforward.\nThe directory of the committed task from the previous attempt is\nmoved under the directory of the current application attempt.\n\n```python\ndef recoverTask(tac):\n oldAttemptId = appAttemptId - 1\n fs.rename('$dest/_temporary/oldAttemptId/${tac.taskId}',\n '$dest/_temporary/appAttemptId/${tac.taskId}')\n```\n\nThis significantly improves time to recover from Job driver (here MR AM) failure.\nThe only lost work is that of all tasks in progress -those which had generated\ndata but were not yet committed.\n\nOnly the failure of the job driver requires a job restart, not an individual\ntask. Therefore the probability of this happening is independent of the number\nof tasks executed in parallel, instead simply due to the duration of the query.\n\nThe longer the task, the higher the risk of failure, the more value there is\nin recovering the work in progress.\n\nFast queries not only have a lower risk of failure, they can recover from\nfailure simply by rerunning the entire job. This is implicitly the strategy\nin Spark, which does not attempt to recover any in-progress jobs. The faster\nyour queries, the simpler your recovery strategy needs to be.\n\n### Job Recovery During `commitJob()`\n\nThis is not possible; a failure during job commit requires the entire job\nto be re-executed after cleaning up the destination directory.\n\n### `mergePaths(FileSystem fs, FileStatus src, Path dest)` Algorithm\n\n`mergePaths()` is the core algorithm to merge data; it is somewhat confusing\nas the implementation mixes the strategies for both algorithms across\ntwo co-recursive routines, `mergePaths()` and `renameOrMerge()`.\n\n\nHere the two algorithms have been split, and one of the co-recursive methods\ninlined.\n\n```python\ndef mergePathsV1(fs, src, dest) :\n if fs.exists(dest) :\n toStat = fs.getFileStatus(dest)\n else:\n toStat = None\n\n if src.isFile :\n if not toStat is None :\n fs.delete(dest, recursive = True)\n fs.rename(src.getPath, dest)\n else :\n # src is directory, choose action on dest type\n if not toStat is None :\n if not toStat.isDirectory :\n # Destination exists and is not a directory\n fs.delete(dest)\n fs.rename(src.getPath(), dest)\n else :\n # Destination exists and is a directory\n # merge all children under destination directory\n for child in fs.listStatus(src.getPath) :\n mergePathsV1(fs, child, dest + child.getName)\n else :\n # destination does not exist\n fs.rename(src.getPath(), dest)\n```\n\n## The v2 Commit Algorithm\n\n\nThe v2 algorithm directly commits task output into the destination directory.\nIt is essentially a re-implementation of the Hadoop 1.x commit algorithm.\n\n1. During execution, intermediate data becomes visible.\n1. On a failure, all output must be deleted and the job restarted.\n\nIt implements `mergePaths` differently, as shown below.\n\n```python\ndef mergePathsV2(fs, src, dest) :\n if fs.exists(dest) :\n toStat = fs.getFileStatus(dest)\n else:\n toStat = None\n\n if src.isFile :\n if not toStat is None :\n fs.delete(dest, recursive = True)\n fs.rename(src.getPath, dest)\n else :\n # destination is directory, choose action on source type\n if src.isDirectory :\n if not toStat is None :\n if not toStat.isDirectory :\n # Destination exists and is not a directory\n fs.delete(dest)\n fs.mkdirs(dest) #\n for child in fs.listStatus(src.getPath) : # HERE\n mergePathsV2(fs, child, dest + child.getName) #\n\n else :\n # Destination exists and is a directory\n # merge all children under destination directory\n for child in fs.listStatus(src.getPath) :\n mergePathsV2(fs, child, dest + child.getName)\n else :\n # destination does not exist\n fs.mkdirs(dest) #\n for child in fs.listStatus(src.getPath) : # HERE\n mergePathsV2(fs, child, dest + child.getName) #\n```\n\nBoth recurse down any source directory tree, and commit single files\nby renaming the files.\n\nA a key difference is that the v1 algorithm commits a source directory to\nvia a directory rename, which is traditionally an `O(1)` operation.\n\nIn contrast, the v2 algorithm lists all direct children of a source directory\nand recursively calls `mergePath()` on them, ultimately renaming the individual\nfiles. As such, the number of renames it performs equals the number of source\n*files*, rather than the number of source *directories*; the number of directory\nlistings being `O(depth(src))` , where `depth(path)` is a function returning the\ndepth of directories under the given path.\n\nOn a normal filesystem, the v2 merge algorithm is potentially more expensive\nthan the v1 algorithm. However, as the merging only takes place in task commit,\nit is potentially less of a bottleneck in the entire execution process.\n\nOn an object store, it is suboptimal not just from its expectation that `rename()`\nis an `O(1)` operation, but from its expectation that a recursive tree walk is\nan efficient way to enumerate and act on a tree of data. If the algorithm was\nswitched to using `FileSystem.listFiles(path, recursive)` for a single call to\nenumerate all children under a path, then the listing operation would be significantly\nfaster, at least on a deep or wide tree. However, for any realistic dataset,\nthe size of the output files is likely to be the main cause of delays. That\nis, if the cost of `mergePathsV2` is `O(depth(src)) + O(data))`, then\ngenerally the `O(data)` value will be more significant than the `depth(src)`.\n\nThere is one key exception: tests which work on small amounts of data yet try\nto generate realistic output directory structures. In these tests the cost\nof listing directories and calling `getFileStatus()` could exceed that of the copy\ncalls. This is why small-scale tests of the commit algorithms against object stores\nmust be considered significantly misleading.\n\n### v2 Task Commit\n\nRename task attempt path to task committed path.\n\n```python\ndef needsTaskCommit(fs, jobAttemptPath, taskAttemptPath, dest):\n return fs.exists(taskAttemptPath)\n\ndef commitTask(fs, jobAttemptPath, taskAttemptPath, dest):\n if fs.exists(taskAttemptPath) :\n mergePathsV2(fs. taskAttemptPath, dest)\n```\n\n### v2 Task Abort\n\nDelete task attempt path.\n\n```python\ndef abortTask(fs, jobAttemptPath, taskAttemptPath, dest):\n fs.delete(taskAttemptPath, recursive=True)\n```\n\nCost: `O(1)` for normal filesystems, `O(files)` for object stores.\n\n\n### v2 Job Commit\n\nAs all the task output is already completed, all that is needed is\nto touch the `_SUCCESS` marker.\n\n```python\ndef commitJob(fs, jobAttemptDir, dest):\n fs.touch(\"$dest/_SUCCESS\")\n```\nCost: `O(1)`\n\nA failure during job abort is implicitly repeatable\n\n```python\ndef isCommitJobRepeatable() :\n return True\n```\n\n### v2 Job Abort\n\nDelete all data under job attempt path.\n\n```python\ndef abortJob(fs, jobAttemptDir, dest):\n fs.delete(jobAttemptDir, recursive=True)\n```\n\nCost: `O(1)` for normal filesystems, `O(files)` for object stores.\n\n### v2 Task Recovery\n\nAs no data is written to the destination directory, a task can be cleaned up\nby deleting the task attempt directory.\n\n### v2 Job Recovery Before `commitJob()`\n\n\nBecause the data has been renamed into the destination directory, all tasks\nrecorded as having being committed have no recovery needed at all:\n\n```python\ndef recoverTask(tac):\n```\n\nAll active and queued tasks are scheduled for execution.\n\nThere is a weakness here, the same one on a failure during `commitTask()`:\nit is only safe to repeat a task which failed during that commit operation\nif the name of all generated files are constant across all task attempts.\n\nIf the Job AM fails while a task attempt has been instructed to commit,\nand that commit is not recorded as having completed, the state of that\nin-progress task is unknown...really it isn't be safe to recover the\njob at this point.\n\n\n### v2 Job Recovery During `commitJob()`\n\nThis is straightforward: `commitJob()` is re-invoked.\n\n## How MapReduce uses the committer in a task\n\nMapReduce runs each Mapper or Reducer in its own container; it\ngets its own process. Implicitly, this also means that each task gets\nits own instance of every filesystem.\n\nThe work is choreographed in `org.apache.hadoop.mapred.Task`, for which there\nare specific subclassing for the Map (`MapTask`) and reduce (`ReduceTask`).\nThere are also cleanup tasks whose role is simpler: clean things up,\nand a Job setup task which runs `OutputCommittre.setupJob`. That is: even\nthe job setup phase is run in a Task.\n\nMapTask uses the committer to write the output of all the mappers into\nthe filesystem, ready for the reducers. Each partition writes its data\nas a `MapFile`, which is actually two `SequenceFile` files in a directory,:\nthe `data` file of all Key-Value\noutput, and `index` which contains an index of some of the keys in the file.\n\nThis is all written to the local filesystem.\n\nThe `ReduceTask` does the final write to the destination filesystem.\n\nBecause the Map phase uses a committer to commits intermediate work,\nany plug in committer supplied to a process\nthrough any extension mechanism *must* work with the output generated by a mapper.\nThe staging committers do only if unique filenames are disabled, but as they\nand the magic committers are only meant to be used for the final output of work,\nit is somewhat moot. What is important is to be able to use different committers\nfor the map phase as the final reduce. This is implicit if separate committers\nare defined for different filesystems: the new committer can be defined for\nthe final destination FS, while `file://` can retain the default\n`FileOutputCommitter`.\n\n### Task Setup\n\n`Task.initialize()`: read in the configuration, instantiate the `JobContextImpl`\nand `TaskAttemptContextImpl` instances bonded to the current job & task.\n\n### Task Ccommit\n\nAfter the work is completed, `Task.done()` is invoked. Which is essentially the following\ncodepath:\n\n```java\nif (committer.needsTaskCommit(taskContext)) {\n // get permission to commit from AM\n int retries = MAX_RETRIES;\n while(true) {\n try {\n umbilical.commitPending(taskId, taskStatus);\n break;\n } catch(IOException ie) {\n if (--retries == 0) {\n // FAIL WITHOUT CALLING ABORT\n System.exit(67);\n }\n }\n }\n // commit the work\n try {\n committer.commitTask(taskContext);\n } catch (IOException iee) {\n // failure: abort\n try {\n committer.abortTask(taskContext);\n } catch (IOException ioe) {\n LOG.warn(\"Failure cleaning up: \" +\n StringUtils.stringifyException(ioe));\n }\n throw iee;\n }\n}\n```\n\nThat is: if and only if there is data to write, the Task requests clearance\nto do so from the AM. This ensures that speculative work is not committed if\nanother task has written it, *and that even non-speculative work is not committed\nif the task has lost contact with the AM*.\n\nThat is: task output is not committed if the Job's AM has failed or\nis in a separate network partition from the Task.\n\nIf permission is granted, the commit call is invoked.\nIf this doesn't work, `abortTask()` is called; all\nfailures there are logged and swallowed.\n\n\nThis method actually appears to be limited in that it does not guarantee\nthat `committer.abortTask()` is always called: if the `umbilical.commitPending()` calls\nfail repeatedly and the Task process aborts, `committer.abortTask()` is not called.\nIf this is due to a network partition, we can hope that the AM invokes\n`committer.abortTask()`, and that if it is an AM failure then a restarted AM can\nclean up its previous attempts. For the classic FileOutputCommitter, listing and\ndeleting the previous attempt's data is straightforward. However, for S3 committers\nusing Multipart Upload as the means of uploading uncommitted data, it is critical\nto ensure that pending uploads are always aborted. This can be done by\n\n* Making sure that all task-side failure branches in `Task.done()` call `committer.abortTask()`.\n* Having job commit & abort cleaning up all pending multipart writes to the same directory\ntree. That is: require that no other jobs are writing to the same tree, and so\nlist all pending operations and cancel them.\n* Add a CLI command to verify that there are no commits under a path; returning\na non-zero exit code if there are. This can be used in testing.\n* Add a CLI command to purge commits under a path. This can be used in workflows,\nif felt necessary, and in test runs to isolate the output of different tests.\n* Document a recommended purge-pending-commit timeout for the system\n* Add probes in the integration tests to verify that pending uploads are never\npresent after a write.\n\n\n\n### How MapReduce uses the committer in the Application Master\n\nThe AM uses a committer to set up and commit the job. To support failure\nand recovery of the AM, `OutputCommitter.isRecoverySupported()` is used\nto declare whether all the output of successful tasks can be used in the final\njob, or the entire job needs to be reset and repeated.\n`OutputCommitter.isCommitJobRepeatable()` addresses the other question:\ncan a committer recover from a failure of the commit process itself.\n\nA committer is created in the Application Master, and handed to an instance\nof `org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler`,\nwhich then manages the job-level lifecycle.\n\nIf the MRv1 API is used, the committer is chosen from the value of\n`\"mapred.output.committer.class\"`; in the MRv2 API the output format\nis instantiated, then asked for a committer using a task and task attempt ID of\n0. A committer is obtained from the output format via a call to\n`Committer.getOutputCommitter(taskContext)`, again using the task attempt context\nwith the (job, 0, 0) task and attempt IDs. That is: even for the job committers,\na task context is always passed in to the `OutputFormat` when requesting a committer.\n*It is critical for all implementations of `OutputCommitter.abortTask()` to\nbe able to execute from the AM, rather than the container and host running\nthe task. Furthermore, all information needed for the abort (paths, filesystem instances\n&c) *must* be retrieved from the `TaskAttemptContext` passed to the method,\nrather than relying on fields initiated from the context passed to the constructor.\n\n\n#### AM: Job setup: `OutputCommitter.setupJob()`\n\nThis is initiated in `org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.StartTransition`.\nIt is queued for asynchronous execution in `org.apache.hadoop.mapreduce.v2.app.MRAppMaster.startJobs()`,\nwhich is invoked when the service is started. Thus: the job is set up when the\nAM is started.\n\n\n#### AM: Job Commit: `OutputCommitter.commitJob()`\n\nIn the \"default\"/cluster filesystem, the `CommitterEventHandler` uses data in\nthe staging area defined in `yarn.app.mapreduce.am.staging-dir`\n(default `/tmp/hadoop-yarn/staging/$user/.staging`), in a subdirectory\nnamed from the job ID.\n\nThree paths are built up for using the filesystem for tracking the state\nof the commit process, with a goal of making the commit operation recoverable,\nwhere supported.\n\n\n| name | role |\n|------|--------|\n| `COMMIT_STARTED` | mark the commencement of the job commit |\n| `COMMIT_SUCCESS` | mark the successful completion of the job commit |\n| `COMMIT_FAIL` | mark the failure of the job commit |\n\nThese markers are used to manage job restart/failure of they happen during\nthe job commit itself.\n\nWhen an AM starts up, it looks in its staging area for these files. as a way\nto determine the previous state of the job. If there are no `COMMIT_` marker files,\nthe job is considered not to have attempted to commit itself yet.\n\n\nThe presence of `COMMIT_SUCCESS` or `COMMIT_FAIL` are taken as evidence\nthat the previous job completed successfully or unsuccessfully; the AM\nthen completes with a success/failure error code, without attempting to rerun\nthe job.\n\nIf `COMMIT_STARTED` exists but not either of the completion markers, then,\nif the committer declares that its job commit operation is repeatable\n(`Committer.isCommitJobRepeatable(jobContext) == true`), then an attempt\nis made to recommit the job, deleting the `COMMIT_STARTED` and commencing\nthe commit process again.\n\nThese `COMMIT_STARTED` files are simply 0-byte files, but are created\nwith the overwrite bit only set to true if the job commit is considered\nrepeatable:\n\n```java\nprivate void touchz(Path p, boolean overwrite) throws IOException {\n fs.create(p, overwrite).close();\n}\n```\nThat is: the atomicity of the `create(path, overwrite=false)` on the cluster\nfilesystem is used to guarantee that only one process will attempt to commit\na specific job.\n\n```java\nboolean commitJobIsRepeatable = committer.isCommitJobRepeatable(\n event.getJobContext());\ntry {\n touchz(startCommitFile, commitJobIsRepeatable);\n waitForValidCommitWindow();\n committer.commitJob(event.getJobContext());\n touchz(endCommitSuccessFile, commitJobIsRepeatable);\n} catch (Exception e) {\n touchz(endCommitFailureFile, commitJobIsRepeatable);\n}\n```\n\nThe `waitForValidCommitWindow()` operation is important: it declares that\nthe committer must not commit unless there has been communication with the YARN\nResource Manager with in `yarn.app.mapreduce.am.job.committer.commit-window` milliseconds\n(default: 10,000). It does this by waiting until the next heartbeat it received.\nThere's a possible bug here: if the interval is set too small the thread may\npermanently spin waiting a callback within the window. Ignoring that, this algorithm\nguarantees that\n\n1. As only one call can create a file with `overwrite=false`,\n only one process's attempt to commit a non-repeatable job will proceed\n\n1. Only a process with contact with the YARN within the configured window\n may commit a job.\n\n1. If the AM is partitioned from the rest of the network, provided that its clock\nis monotonically increasing at the same rate as the rest of the cluster, then\nthe rest of the cluster can be confident that\n`yarn.app.mapreduce.am.ob.committer.commit-window` milliseconds after the AM\nsuccessfully heartbeated to the YARN RM, then the output of this job attempt\nwill *never* be committed. This permits 1 job to run simultaneously, but\nhelps ensure that only one of them will attempt to commit.\n\n1. Provided YARN heartbeats are only sent to the AM which successfully created\nthe `COMMIT_STARTED` file, it will initiate the commit operation.\n\nPossible issues with this algorithm:\n\n* The `COMMIT_STARTED` file is created before waiting to get a heartbeat. It\nmay be that this AM has lost contact with YARN, but doesn't know it yet. When the\nYARN liveness protocols eventually time out, the AM will correctly terminate,\nbut as the `COMMIT_STARTED` file has been created at this point, no other launched\nAM will be able to commit.\n\n* If two committers attempt to create `COMMIT_STARTED` files on a no-repeatable\ncommit, one will succeed, wait for a heartbeat then attempt a (possibly slow) commit.\nThe second committer will fail, and will *immediately* create a `COMMIT_FAILED` file.\nAs a result, the state of the staging area will imply that the commit has failed,\nwhen really it is in progress, and that only the second process failed.\n\nIt would seem safer to address this through\n1. Waiting for a heartbeat before creating the `COMMIT_STARTED` file.\n1. Maybe: not creating the `COMMIT_FAILED` file if the failure happens when\ntrying to create the `COMMIT_STARTED` file. That is: only a process which\nsuccessfully created the `COMMIT_STARTED` file may indicate that a commit has failed.\n\n\n### AM: Cancelling job commit\n\nThe thread performing the commit is interrupted; the `CommitterEventHandler`\nawaits for it to finish. (set in `yarn.app.mapreduce.am.job.committer.cancel-timeout`\nas milliseconds).\n\n### AM: Task Abort\n\nThe AM may call the `OutputCommitter.taskAbort()` with a task attempt context,\nwhen handling the failure/loss of a container. That is: on container failure,\nthe task abort operation is executed in the AM, using the AM's committer.\nThis avoids the need to create a new container, and means that the \"best-effort\"\ntask abort does cope with container failures.\n\nA partition between the AM and the task container means that this AM-executed\ntask abort may take place while a task in the partitioned container is still\nexecuting. Unless output writing operations will fail after the abort operation,\nthe partitioned task may not become aware of the partition until it's own task\ncommit sequence in `Task.done()`, when `talkToAMTGetPermissionToCommit()`\n\n# Requirements of an S3A Committer\n\n1. Support an eventually consistent S3 object store as a reliable direct\ndestination of work through the S3A filesystem client.\n1. Efficient: implies no rename, and a minimal amount of delay in the job driver's\ntask and job commit phases,\n1. Support task failure and speculation.\n1. Can be used by existing code: Hadoop MapReduce, Spark, Hive.\n1. Retrofittable to existing subclasses of FileOutputFormat and/or compatible\nwith committers which expect a specific FileOutputFormat.\n1. Clean up uncommitted data from all task attempts, all previous attempts of\nthe job, and any previous incompleted jobs.\n1. Security: not to permit privilege escalation from other users with\nwrite access to the same file system(s).\n\n\n## Features of S3 and the S3A Client\n\n\nA core problem is that\n[object stores are not filesystems](../../../hadoop-project-dist/hadoop-common/filesystem/introduction.html);\nhow `rename()` has been emulated in the S3A client means that both the existing\nMR committer algorithms have significant performance problems.\n\n1. Single-object renames are implemented as a copy and delete sequence.\n1. COPY is atomic, but overwrites cannot be prevented.\n1. Amazon S3 is eventually consistent on listings, deletes and updates.\n1. Amazon S3 has create consistency, however, the negative response of a HEAD/GET\nperformed on a path before an object was created can be cached, unintentionally\ncreating a create inconsistency. The S3A client library does perform such a check,\non `create()` and `rename()` to check the state of the destination path, and\nso, whether the operation is permitted.\n1. Multi-object renames are sequential or parallel single object COPY+DELETE operations:\nnon atomic, `O(data)` and, on failure, can leave the filesystem in an unknown\nstate.\n1. There is a PUT operation, capable of uploading 5GB of data in one HTTP request.\n1. The PUT operation is atomic, but there is no PUT-no-overwrite option.\n1. There is a multipart POST/PUT sequence for uploading larger amounts of data\nin a sequence of PUT requests.\n\n\nThe Hadoop S3A Filesystem client supports PUT and multipart PUT for uploading\ndata, with the `S3ABlockOutputStream` of HADOOP-13560 uploading written data\nas parts of a multipart PUT once the threshold set in the configuration\nparameter `fs.s3a.multipart.size` (default: 100MB).\n\n[S3Guard](./s3guard.html) adds an option of consistent view of the filesystem\nto all processes using the shared DynamoDB table as the authoritative store of\nmetadata. Some S3-compatible object stores are fully consistent; the\nproposed algorithm is designed to work with such object stores without the\nneed for any DynamoDB tables.\n\n## Related work: Spark's `DirectOutputCommitter`\n\nOne implementation to look at is the\n[`DirectOutputCommitter` of Spark 1.6](https://github.com/apache/spark/blob/branch-1.6/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/DirectParquetOutputCommitter.scala).\n\nThis implements a zero rename commit by subclassing the `ParquetOutputCommitter`\nand then\n\n1. Returning the destination directory as the task working directory.\n1. Subclassing all the task commit/abort operations to be no-ops.\n\nWith the working directory as the destination directory, there is no need\nto move/rename the task output on a successful commit. However, it is flawed.\nThere is no notion of \"committing\" or \"aborting\" a task, hence no ability to\nhandle speculative execution or failures. This is why the committer\nwas removed from Spark 2 [SPARK-10063](https://issues.apache.org/jira/browse/SPARK-10063)\n\nThere is also the issue that work-in-progress data is visible; this may or may\nnot be a problem.\n\n## Related work: IBM's \"Stocator\" committer\n\nIBM's [Stocator](https://github.com/SparkTC/stocator) can transform indirect\nwrites of V1/V2 committers into direct writes to the destination directory.\n\nHpw does it do this? It's a special Hadoop `FileSystem` implementation which\nrecognizes writes to `_temporary` paths and translate them to writes to the\nbase directory. As well as translating the write operation, it also supports\na `getFileStatus()` call on the original path, returning details on the file\nat the final destination. This allows for committing applications to verify\nthe creation/existence/size of the written files (in contrast to the magic\ncommitter covered below).\n\nThe FS targets Openstack Swift, though other object stores are supportable through\ndifferent backends.\n\nThis solution is innovative in that it appears to deliver the same semantics\n(and hence failure modes) as the Spark Direct OutputCommitter, but which\ndoes not need any change in either Spark *or* the Hadoop committers. In contrast,\nthe committers proposed here combines changing the Hadoop MR committers for\nease of pluggability, and offers a new committer exclusively for S3, one\nstrongly dependent upon and tightly integrated with the S3A Filesystem.\n\nThe simplicity of the Stocator committer is something to appreciate.\n\n## Background: The S3 multi-part PUT mechanism\n\nIn the [S3 REST API](http://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html),\nmultipart uploads allow clients to upload a series of \"Parts\" of a file,\nthen commit the upload with a final call.\n\n1. Caller initiates a multipart request, including the destination bucket, key\nand metadata.\n\n POST bucket.s3.aws.com/path?uploads\n\n An `UploadId` is returned\n\n1. Caller uploads one or more parts.\n\n PUT bucket.s3.aws.com/path?partNumber=PartNumber&uploadId=UploadId\n\n The part number is used to declare the ordering of the PUTs; they\n can be uploaded in parallel and out of order.\n All parts *excluding the final part* must be 5MB or larger.\n Every upload completes with an etag returned\n\n1. Caller completes the operation\n\n POST /ObjectName?uploadId=UploadId\n \n (number)(Tag)\n ...\n \n\n This final call lists the etags of all uploaded parts and the actual ordering\n of the parts within the object.\n\nThe completion operation is apparently `O(1)`; presumably the PUT requests\nhave already uploaded the data to the server(s) which will eventually be\nserving up the data for the final path. All that is needed to complete\nthe upload is to construct an object by linking together the files in\nthe server's local filesystem and update an entry the index table of the\nobject store.\n\nIn the S3A client, all PUT calls in the sequence and the final commit are\ninitiated by the same process. *This does not have to be the case*.\nIt is that fact, that a different process may perform different parts\nof the upload, which make this algorithm viable.\n\n\n## The Netfix \"Staging\" committer\n\nRyan Blue, of Netflix, has submitted an alternate committer, one which has a\nnumber of appealing features\n\n* Doesn't have any requirements of the destination object store,\n* Known to work.\n\nThe final point is not to be underestimated, es not even\na need for a consistency layer.\n* Overall a simpler design. Especially given the need to\nbe resilient to the various failure modes which may arise.\n\n\nThe committer writes task outputs to a temporary directory on the local FS.\nTask outputs are directed to the local FS by `getTaskAttemptPath` and `getWorkPath`.\nOn task commit, the committer enumerates files in the task attempt directory (ignoring hidden files).\nEach file is uploaded to S3 using the [multi-part upload API](http://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html),\n\nThe information needed to commit the upload is saved into HDFS and committed\nvia that protocol: when the job commits, the pending uploads of the successful\ntasks are all committed.\n\n\n### Commit logic\n\nThe core algorithm is as follows:\n\n1. The destination directory for output (e.g. `FileOutputFormat` and subclasses)\nis a local `file://` reference.\n1. Task commit initiates the multipart PUT to the destination object store.\n1. A list of every pending PUT for task is persisted to a single file\nwithin a consistent, cluster-wide filesystem. For Netflix, that is HDFS.\n1. The Standard `FileOutputCommitter` (algorithm 1) is used to manage the commit/abort of these\nfiles. That is: it copies only those lists of files to commit from successful tasks\ninto a (transient) job commit directory.\n1. The S3 job committer reads the pending file list for every task committed\nin HDFS, and completes those put requests.\n\nBy using `FileOutputCommmitter` to manage the propagation of the lists of files\nto commit, the existing commit algorithm implicitly becomes that defining which\nfiles will be committed at the end of the job.\n\n\nThe Netflix contribution has Hadoop `OutputCommitter` implementations for S3.\n\nThere are 3 main classes:\n* `S3MultipartOutputCommitter` is a base committer class that handles commit logic. This should not be used directly.\n* `S3DirectoryOutputCommitter` for writing unpartitioned data to S3 with conflict resolution.\n* `S3PartitionedOutputCommitter` for writing partitioned data to S3 with conflict resolution.\n\nCallers should use `S3DirectoryOutputCommitter` for single-directory outputs,\nor `S3PartitionedOutputCommitter` for partitioned data.\n\n\nThese S3 committers work by writing task outputs to a temporary directory on the local FS.\nTask outputs are directed to the local FS by `getTaskAttemptPath` and `getWorkPath`.\n\n\n### Conflict Resolution\n\nThe single-directory and partitioned committers handle conflict resolution by\nchecking whether target paths exist in S3 before uploading any data.\nThere are three conflict resolution modes, controlled by setting `fs.s3a.committer.staging.conflict-mode`:\n\n* `fail`: Fail a task if an output directory or partition already exists. (Default)\n* `append`: Upload data files without checking whether directories or partitions already exist.\n* `replace`: If an output directory exists, delete it so the new data replaces the current content.\n\nThe partitioned committer enforces the conflict mode when a conflict is detected with output data, not before the job runs.\nConflict resolution differs from an output mode because it does not enforce the mode when there is no conflict.\nFor example, overwriting a partition should remove all sub-partitions and data it contains, whether or not new output is created.\nConflict resolution will only replace partitions that have output data.\n\nWhen the conflict mode is `replace`, conflicting directories are removed during\njob commit. Data is only deleted if all tasks have completed successfully.\n\nA UUID that identifies a write is added to filenames that are uploaded to S3.\nThis allows rolling back data from jobs that fail during job commit (see failure cases below) and avoids\nfile-level conflicts when appending data to existing directories.\n\n\n*Note* the checks for existence are made via `S3AFileSystem.getFileStatus()` requests of the destination paths.\nUnless the view of the S3 store is consistent, it may be that a newly-deleted object\nis still discovered in the probe, so a commit fail, even when there is no longer any actual conflict.\n\n### Performance\n\nCompared to the previous proposal, henceforth the \"magic\" committer, this\ncommitter, the \"staging committer\", adds the extra overhead of uploading\neach file at the end of every task. This is an `O(data)` operation; it can be\nparallelized, but is bounded by the bandwidth from compute node to S3, as\nwell as the write/IOP capacity of the destination shard of S3. If many tasks\ncomplete at or near the same time, there may be a peak of bandwidth load\nslowing down the upload.\n\nTime to commit will be the same, and, given the Netflix committer has already\nimplemented the parallelization logic here, a time of `O(files/threads)`.\n\n### Resilience\n\nThere's already a lot of code in the task and job commits to handle failure.\n\nAny failure in a commit triggers a best-effort abort/revert of the commit\nactions for a task/job.\n\nTask commits delegate to the `FileOutputCommitter` to ensure that only one task's\noutput reaches the job commit.\n\nSimilarly, if a task is aborted, temporary output on the local FS is removed.\n\nIf a task dies while the committer is running, it is possible for data to be\neft on the local FS or as unfinished parts in S3.\nUnfinished upload parts in S3 are not visible to table readers and are cleaned\nup following the rules in the target bucket's life-cycle policy.\n\nFailures during job commit are handled by deleting any files that have already\nbeen completed and aborting the remaining uploads.\nBecause uploads are completed individually, the files that are deleted were visible to readers.\n\nIf the process dies while the job committer is running, there are two possible failures:\n\n1. Some directories that would be replaced have been deleted, but no new data is visible.\n2. Some new data is visible but is not complete, and all replaced directories have been removed.\n Only complete files are visible.\n\nIf the process dies during job commit, cleaning up is a manual process.\nFile names include a UUID for each write so that files can be identified and removed.\n\n\n**Failure during task execution**\n\nAll data is written to local temporary files; these need to be cleaned up.\n\nThe job must ensure that the local (pending) data is purged.\n\n**Failure during task commit**\n\n\nA process failure during the upload process will result in the\nlist of pending multipart PUTs to *not* be persisted to the cluster filesystem.\nThis window is smaller than the entire task execution, but still potentially\nsignificant, at least for large uploads.\n\nPer-file persistence, or incremental overwrites of the upload list may\nreduce the problems here, but there would still be a small risk of\nan outstanding multipart upload not being recorded\n\n**Explicit Task abort before task commit**\n\nTask will delete all local data; no uploads will be initiated.\n\n**Failure to communicate with S3 during data upload**\n\nIf an upload fails, tasks will:\n\n* Retry using the retry policies implemented in the S3AFileSystem classes\nand the AWS libraries.\n* Eventually: attempt to abort outstanding multipart uploads already initiated.\n* Remove temporary files on the local FS.\n\n\n**Explicit Job Abort**\n\nAll in-progress tasks are aborted and cleaned up. The pending commit data\nof all completed tasks can be loaded, outstanding multipart PUT requests aborted.\n\n\nThis is done by\n\n1. Listing all local files, with a best effort read attempt followed by\nan abort of all successfully read files.\n1. List and abort all pending multipart uploads.\n\nBecause of action #2, action #1 is superfluous. It is retained so as to leave\nopen the option of making action #2 a configurable option -which would be\nrequired to handle the use case of >1 partitioned commit running simultaneously/\n\n**Job Driver failure before Job Commit**\n\nBecause the local data is managed with the v1 commit algorithm, the\nsecond attempt of the job will recover all the outstanding commit data\nof the first attempt; those tasks will not be rerun.\n\nThis also ensures that on a job abort, the individual tasks' .pendingset\nfiles can be read and used to initiate the abort of those uploads.\nThat is: a recovered job can clean up the pending writes of the previous job\n\nIf the query engine does not support multiple job attempts, then the\npending commit data will not be recovered; an explicit abort operation will\nneed to be initiated (we will add a CLI command for this), or the S3 bucket\nmust be configured to automatically delete the pending request.\n\n**Job Driver failure during Job Commit**\n\nThose uploads already executed by a failed job commit will persist; those\nyet to execute will remain outstanding.\n\nThe committer currently declares itself as non-recoverable, but that\nmay not actually hold, as the recovery process could be one of:\n\n1. Enumerate all job commits from the .pendingset files (*:= Commits*).\n1. List all outstanding uploads under the destination path (*:= Outstandings*)..\n1. List all written files (for better consistency, via a GET call on the known\nfilenames)\n1. Identify all files which are yet to be completed (*Commits - Written*)/\n1. Verify that the set of pend\n\n\n\n\n# Committing work to S3 with the \"S3A Committers\"\n\n\n\nThis page covers the S3A Committers, which can commit work directly\nto an S3 object store.\n\nThese committers are designed to solve a fundamental problem which\nthe standard committers of work cannot do to S3: consistent, high performance,\nand reliable commitment of output to S3.\n\nFor details on their internal design, see\n[S3A Committers: Architecture and Implementation](./committer_architecture.html).\n\n\n## Introduction: The Commit Problem\n\n\nApache Hadoop MapReduce (and behind the scenes, Apache Spark) often write\nthe output of their work to filesystems\n\nNormally, Hadoop uses the `FileOutputFormatCommitter` to manage the\npromotion of files created in a single task attempt to the final output of\na query. This is done in a way to handle failures of tasks and jobs, and to\nsupport speculative execution. It does that by listing directories and renaming\ntheir content into the final destination when tasks and then jobs are committed.\n\nThis has some key requirement of the underlying filesystem:\n\n1. When you list a directory, you see all the files which have been created in it,\nand no files which are not in it (i.e. have been deleted).\n1. When you rename a directory, it is an `O(1)` atomic transaction. No other\nprocess across the cluster may rename a file or directory to the same path.\nIf the rename fails for any reason, either the data is at the original location,\nor it is at the destination, -in which case the rename actually succeeded.\n\n**The S3 object store and the `s3a://` filesystem client cannot meet these requirements.*\n\n1. Amazon S3 has inconsistent directory listings unless S3Guard is enabled.\n1. The S3A mimics `rename()` by copying files and then deleting the originals.\nThis can fail partway through, and there is nothing to prevent any other process\nin the cluster attempting a rename at the same time.\n\nAs a result,\n\n* Files my not be listed, hence not renamed into place.\n* Deleted files may still be discovered, confusing the rename process to the point\nof failure.\n* If a rename fails, the data is left in an unknown state.\n* If more than one process attempts to commit work simultaneously, the output\ndirectory may contain the results of both processes: it is no longer an exclusive\noperation.\n*. While S3Guard may deliver the listing consistency, commit time is still\nproportional to the amount of data created. It still can't handle task failure.\n\n**Using the \"classic\" `FileOutputCommmitter` to commit work to Amazon S3 risks\nloss or corruption of generated data**\n\n\nTo address these problems there is now explicit support in the `hadop-aws`\nmodule for committing work to Amazon S3 via the S3A filesystem client,\n*the S3A Committers*\n\n\nFor safe, as well as high-performance output of work to S3,\nwe need use \"a committer\" explicitly written to work with S3, treating it as\nan object store with special features.\n\n\n### Background : Hadoop's \"Commit Protocol\"\n\nHow exactly is work written to its final destination? That is accomplished by\na \"commit protocol\" between the workers and the job manager.\n\nThis protocol is implemented in Hadoop MapReduce, with a similar but extended\nversion in Apache Spark:\n\n1. A \"Job\" is the entire query, with inputs to outputs\n1. The \"Job Manager\" is the process in charge of choreographing the execution\nof the job. It may perform some of the actual computation too.\n1. The job has \"workers\", which are processes which work the actual data\nand write the results.\n1. Workers execute \"Tasks\", which are fractions of the job, a job whose\ninput has been *partitioned* into units of work which can be executed independently.\n1. The Job Manager directs workers to execute \"tasks\", usually trying to schedule\nthe work close to the data (if the filesystem provides locality information).\n1. Workers can fail: the Job manager needs to detect this and reschedule their active tasks.\n1. Workers can also become separated from the Job Manager, a \"network partition\".\nIt is (provably) impossible for the Job Manager to distinguish a running-but-unreachable\nworker from a failed one.\n1. The output of a failed task must not be visible; this is to avoid its\ndata getting into the final output.\n1. Multiple workers can be instructed to evaluate the same partition of the work;\nthis \"speculation\" delivers speedup as it can address the \"straggler problem\".\nWhen multiple workers are working on the same data, only one worker is allowed\nto write the final output.\n1. The entire job may fail (often from the failure of the Job Manager (MR Master, Spark Driver, ...)).\n1, The network may partition, with workers isolated from each other or\nthe process managing the entire commit.\n1. Restarted jobs may recover from a failure by reusing the output of all\ncompleted tasks (MapReduce with the \"v1\" algorithm), or just by rerunning everything\n(The \"v2\" algorithm and Spark).\n\n\nWhat is \"the commit protocol\" then? It is the requirements on workers as to\nwhen their data is made visible, where, for a filesystem, \"visible\" means \"can\nbe seen in the destination directory of the query.\"\n\n* There is a destination directory of work, \"the output directory.\"\n* The final output of tasks must be in this directory *or paths underneath it*.\n* The intermediate output of a task must not be visible in the destination directory.\nThat is: they must not write directly to the destination.\n* The final output of a task *may* be visible under the destination.\n* The Job Manager makes the decision as to whether a task's data is to be \"committed\",\nbe it directly to the final directory or to some intermediate store..\n* Individual workers communicate with the Job manager to manage the commit process:\nwhether the output is to be *committed* or *aborted*\n* When a worker commits the output of a task, it somehow promotes its intermediate work to becoming\nfinal.\n* When a worker aborts a task's output, that output must not become visible\n(i.e. it is not committed).\n* Jobs themselves may be committed/aborted (the nature of \"when\" is not covered here).\n* After a Job is committed, all its work must be visible.\n* And a file `_SUCCESS` may be written to the output directory.\n* After a Job is aborted, all its intermediate data is lost.\n* Jobs may also fail. When restarted, the successor job must be able to clean up\nall the intermediate and committed work of its predecessor(s).\n* Task and Job processes measure the intervals between communications with their\nApplication Master and YARN respectively.\nWhen the interval has grown too large they must conclude\nthat the network has partitioned and that they must abort their work.\n\n\nThat's \"essentially\" it. When working with HDFS and similar filesystems,\ndirectory `rename()` is the mechanism used to commit the work of tasks and\njobs.\n* Tasks write data to task attempt directories under the directory `_temporary`\nunderneath the final destination directory.\n* When a task is committed, these files are renamed to the destination directory\n(v2 algorithm) or a job attempt directory under `_temporary` (v1 algorithm).\n* When a job is committed, for the v2 algorithm the `_SUCCESS` file is created,\nand the `_temporary` deleted.\n* For the v1 algorithm, when a job is committed, all the tasks committed under\nthe job attempt directory will have their output renamed into the destination\ndirectory.\n* The v2 algorithm recovers from failure by deleting the destination directory\nand restarting the job.\n* The v1 algorithm recovers from failure by discovering all committed tasks\nwhose output is in the job attempt directory, *and only rerunning all uncommitted tasks*.\n\n\nNone of this algorithm works safely or swiftly when working with \"raw\" AWS S3 storage:\n* Directory listing can be inconsistent: the tasks and jobs may not list all work to\nbe committed.\n* Renames go from being fast, atomic operations to slow operations which can fail partway through.\n\nThis then is the problem which the S3A committers address:\n\n*How to safely and reliably commit work to Amazon S3 or compatible object store*\n\n\n## Meet the S3A Commmitters\n\nSince Hadoop 3.1, the S3A FileSystem has been accompanied by classes\ndesigned to integrate with the Hadoop and Spark job commit protocols, classes\nwhich interact with the S3A filesystem to reliably commit work work to S3:\n*The S3A Committers*\n\nThe underlying architecture of this process is very complex, and\ncovered in [the committer architecture documentation](./committer_architecture.html).\n\nThe key concept to know of is S3's \"Multipart Upload\" mechanism. This allows\nan S3 client to write data to S3 in multiple HTTP POST requests, only completing\nthe write operation with a final POST to complete the upload; this final POST\nconsisting of a short list of the etags of the uploaded blocks.\nThis multipart upload mechanism is already automatically used when writing large\namounts of data to S3; an implementation detail of the S3A output stream.\n\nThe S3A committers make explicit use of this multipart upload (\"MPU\") mechanism:\n\n1. The individual *tasks* in a job write their data to S3 as POST operations\nwithin multipart uploads, yet do not issue the final POST to complete the upload.\n1. The multipart uploads are committed in the job commit process.\n\nThere are two different S3A committer types, *staging*\nand *magic*. The committers primarily vary in how data is written during task execution,\nhow the pending commit information is passed to the job manager, and in how\nconflict with existing files is resolved.\n\n\n| feature | staging | magic |\n|--------|---------|---|\n| task output destination | local disk | S3A *without completing the write* |\n| task commit process | upload data from disk to S3 | list all pending uploads on s3 and write details to job attempt directory |\n| task abort process | delete local disk data | list all pending uploads and abort them |\n| job commit | list & complete pending uploads | list & complete pending uploads |\n\nThe other metric is \"maturity\". There, the fact that the Staging committers\nare based on Netflix's production code counts in its favor.\n\n\n### The Staging Committer\n\nThis is based on work from Netflix. It \"stages\" data into the local filesystem.\nIt also requires the cluster to have HDFS, so that\n\nTasks write to URLs with `file://` schemas. When a task is committed,\nits files are listed, uploaded to S3 as incompleted Multipart Uploads.\nThe information needed to complete the uploads is saved to HDFS where\nit is committed through the standard \"v1\" commit algorithm.\n\nWhen the Job is committed, the Job Manager reads the lists of pending writes from its\nHDFS Job destination directory and completes those uploads.\n\nCancelling a task is straightforward: the local directory is deleted with\nits staged data. Cancelling a job is achieved by reading in the lists of\npending writes from the HDFS job attempt directory, and aborting those\nuploads. For extra safety, all outstanding multipart writes to the destination directory\nare aborted.\n\nThe staging committer comes in two slightly different forms, with slightly\ndifferent conflict resolution policies:\n\n\n* **Directory**: the entire directory tree of data is written or overwritten,\nas normal.\n\n* **Partitioned**: special handling of partitioned directory trees of the form\n`YEAR=2017/MONTH=09/DAY=19`: conflict resolution is limited to the partitions\nbeing updated.\n\n\nThe Partitioned Committer is intended to allow jobs updating a partitioned\ndirectory tree to restrict the conflict resolution to only those partition\ndirectories containing new data. It is intended for use with Apache Spark\nonly.\n\n\n## Conflict Resolution in the Staging Committers\n\nThe Staging committers offer the ability to replace the conflict policy\nof the execution engine with policy designed to work with the tree of data.\nThis is based on the experience and needs of Netflix, where efficiently adding\nnew data to an existing partitioned directory tree is a common operation.\n\n```xml\n\n fs.s3a.committer.staging.conflict-mode\n fail\n \n Staging committer conflict resolution policy: {@value}.\n Supported: fail, append, replace.\n \n\n```\n\n**replace** : when the job is committed (and not before), delete files in\ndirectories into which new data will be written.\n\n**fail**: when there are existing files in the destination, fail the job.\n\n**append**: Add new data to the directories at the destination; overwriting\nany with the same name. Reliable use requires unique names for generated files,\nwhich the committers generate\nby default.\n\nThe difference between the two staging committers are as follows:\n\nThe Directory Committer uses the entire directory tree for conflict resolution.\nIf any file exists at the destination it will fail in job setup; if the resolution\nmechanism is \"replace\" then all existing files will be deleted.\n\nThe partitioned committer calculates the partitions into which files are added,\nthe final directories in the tree, and uses that in its conflict resolution\nprocess:\n\n\n**replace** : delete all data in the destination partition before committing\nthe new files.\n\n**fail**: fail if there is data in the destination partition, ignoring the state\nof any parallel partitions.\n\n**append**: add the new data.\n\nIt's intended for use in Apache Spark Dataset operations, rather\nthan Hadoop's original MapReduce engine, and only in jobs\nwhere adding new data to an existing dataset is the desired goal.\n\nPrerequisites for successful work\n\n1. The output is written into partitions via `PARTITIONED BY` or `partitionedBy()`\ninstructions.\n2. There is no data written directly to the root path (all files there are\nignored; it's implicitly \"append\").\n\nHere's an example in Spark, assuming that `sourceDataset` is a dataset\nwhose columns include \"year\" and \"month\":\n\n```scala\nsourceDataset\n .write\n .partitionBy(\"year\", \"month\")\n .mode(SaveMode.Append)\n .opt(\"fs.s3a.committer.name\", \"partitioned\")\n .opt(\"fs.s3a.committer.staging.conflict-mode\", \"replace\")\n .format(\"orc\")\n .save(\"s3a://examples/statistics\")\n```\n\n\n### The Magic Committer\n\nThe \"Magic\" committer does its work through \"magic\" in the filesystem:\nattempts to write to specific \"magic\" paths are interpreted as writes\nto a parent directory *which are not to be completed*. When the output stream\nis closed, the information needed to complete the write is saved in the magic\ndirectory. The task committer saves the list of these to a directory for the\njob committers use, or, if aborting, lists the pending writes and aborts them.\n\nThe job committer reads in the list of pending commits, and commits them as\nthe Staging Committer does.\n\nCompared to the Staging Committer, the Magic Committer offers faster write\ntimes: output is uploaded to S3 as it is written, rather than in the\ntask commit.\n\nHowever, it has extra requirements of the filesystem\n\n1. It requires a consistent object store, which for Amazon S3,\nmeans that [S3Guard](./s3guard.html) must be enabled. For third-party stores,\nconsult the documentation.\n1. The S3A client must be configured to recognize interactions\nwith the magic directories and treat them specially.\n\n\nIt's also not been field tested to the extent of Netflix's committer; consider\nit the least mature of the committers.\n\n\n#### Which Committer to Use?\n\n1. If you want to create or update existing partitioned data trees in Spark, use thee\nPartitioned Committer. Make sure you have enough hard disk capacity for all staged data.\nDo not use it in other situations.\n\n1. If you know that your object store is consistent, or that the processes\nwriting data use S3Guard, use the Magic Committer for higher performance\nwriting of large amounts of data.\n\n1. Otherwise: use the directory committer, making sure you have enough\nhard disk capacity for all staged data.\n\nPut differently: start with the Directory Committer.\n\n## Switching to an S3A Committer\n\nTo use an S3A committer, the property `mapreduce.outputcommitter.factory.scheme.s3a`\nmust be set to the S3A committer factory, `org.apache.hadoop.fs.s3a.commit.staging.S3ACommitterFactory`.\nThis is done in `mapred-default.xml`\n\n```xml\n\n mapreduce.outputcommitter.factory.scheme.s3a\n org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory\n \n The committer factory to use when writing data to S3A filesystems.\n \n\n```\n\nWhat is missing is an explicit choice of committer to use in the property\n`fs.s3a.committer.name`; so the classic (and unsafe) file committer is used.\n\n| `fs.s3a.committer.name` | Committer |\n|--------|---------|\n| `directory` | directory staging committer |\n| `partitioned` | partition staging committer (for use in Spark only) |\n| `magic` | the \"magic\" committer |\n| `file` | the original and unsafe File committer; (default) |\n\n\n\n## Using the Directory and Partitioned Staging Committers\n\nGenerated files are initially written to a local directory underneath one of the temporary\ndirectories listed in `fs.s3a.buffer.dir`.\n\n\nThe staging committer needs a path in the cluster filesystem\n(e.g. HDFS). This must be declared in `fs.s3a.committer.staging.tmp.path`.\n\nTemporary files are saved in HDFS (or other cluster filesystem) under the path\n`${fs.s3a.committer.staging.tmp.path}/${user}` where `user` is the name of the user running the job.\nThe default value of `fs.s3a.committer.staging.tmp.path` is `tmp/staging`,\nWhich will be converted at run time to a path under the current user's home directory,\nessentially `~/tmp/staging`\n so the temporary directory\n\nThe application attempt ID is used to create a unique path under this directory,\nresulting in a path `~/tmp/staging/${user}/${application-attempt-id}/` under which\nsummary data of each task's pending commits are managed using the standard\n`FileOutputFormat` committer.\n\nWhen a task is committed the data is uploaded under the destination directory.\nThe policy of how to react if the destination exists is defined by\nthe `fs.s3a.committer.staging.conflict-mode` setting.\n\n| `fs.s3a.committer.staging.conflict-mode` | Meaning |\n| -----------------------------------------|---------|\n| `fail` | Fail if the destination directory exists |\n| `replace` | Delete all existing files before committing the new data |\n| `append` | Add the new files to the existing directory tree |\n\n\n## The \"Partitioned\" Staging Committer\n\nThis committer an extension of the \"Directory\" committer which has a special conflict resolution\npolicy designed to support operations which insert new data into a directory tree structured\nusing Hive's partitioning strategy: different levels of the tree represent different columns.\n\nFor example, log data could be partitioned by `YEAR` and then by `MONTH`, with different\nentries underneath.\n\n```\nlogs/YEAR=2017/MONTH=01/\n log-20170101.avro\n log-20170102.avro\n ...\n log-20170131.avro\n\nlogs/YEAR=2017/MONTH=02/\n log-20170201.avro\n log-20170202.avro\n ...\n log-20170227.avro\n\nlogs/YEAR=2017/MONTH=03/\nlogs/YEAR=2017/MONTH=04/\n```\n\nA partitioned structure like this allows for queries using Hive or Spark to filter out\nfiles which do not contain relevant data.\n\nWhat the partitioned committer does is, where the tooling permits, allows callers\nto add data to an existing partitioned layout*.\n\nMore specifically, it does this by having a conflict resolution options which\nonly act on individual partitions, rather than across the entire output tree.\n\n| `fs.s3a.committer.staging.conflict-mode` | Meaning |\n| -----------------------------------------|---------|\n| `fail` | Fail if the destination partition(s) exist |\n| `replace` | Delete the existing data partitions before committing the new data |\n| `append` | Add the new data to the existing partitions |\n\n\nAs an example, if a job was writing the file\n`logs/YEAR=2017/MONTH=02/log-20170228.avro`, then with a policy of `fail`,\nthe job would fail. With a policy of `replace`, then entire directory\n`logs/YEAR=2017/MONTH=02/` would be deleted before the new file `log-20170228.avro`\nwas written. With the policy of `append`, the new file would be added to\nthe existing set of files.\n\n\n### Notes\n\n1. A deep partition tree can itself be a performance problem in S3 and the s3a client,\nor, more specifically. a problem with applications which use recursive directory tree\nwalks to work with data.\n\n1. The outcome if you have more than one job trying simultaneously to write data\nto the same destination with any policy other than \"append\" is undefined.\n\n1. In the `append` operation, there is no check for conflict with file names.\nIf, in the example above, the file `log-20170228.avro` already existed,\nit would be overridden. Set `fs.s3a.committer.staging.unique-filenames` to `true`\nto ensure that a UUID is included in every filename to avoid this.\n\n\n## Using the Magic committer\n\nThis is less mature than the Staging Committer, but promises higher\nperformance.\n\n### FileSystem client setup\n\n1. Use a *consistent* S3 object store. For Amazon S3, this means enabling\n[S3Guard](./s3guard.html). For S3-compatible filesystems, consult the filesystem\ndocumentation to see if it is consistent, hence compatible \"out of the box\".\n1. Turn the magic on by `fs.s3a.committer.magic.enabled\"`\n\n```xml\n\n fs.s3a.committer.magic.enabled\n \n Enable support in the filesystem for the S3 \"Magic\" committer.\n \n true\n\n```\n\n*Do not use the Magic Committer on an inconsistent S3 object store. For\nAmazon S3, that means S3Guard must *always* be enabled.\n\n\n### Enabling the committer\n\n```xml\n\n fs.s3a.committer.name\n magic\n\n\n```\n\nConflict management is left to the execution engine itself.\n\n## Committer Configuration Options\n\n\n| Option | Magic | Directory | Partitioned | Meaning | Default |\n|--------|-------|-----------|-------------|---------|---------|\n| `mapreduce.fileoutputcommitter.marksuccessfuljobs` | X | X | X | Write a `_SUCCESS` file at the end of each job | `true` |\n| `fs.s3a.committer.threads` | X | X | X | Number of threads in committers for parallel operations on files. | 8 |\n| `fs.s3a.committer.staging.conflict-mode` | | X | X | Conflict resolution: `fail`, `abort` or `overwrite`| `fail` |\n| `fs.s3a.committer.staging.unique-filenames` | | X | X | Generate unique filenames | `true` |\n\n| `fs.s3a.committer.magic.enabled` | X | | | Enable \"magic committer\" support in the filesystem | `false` |\n\n\n\n\n| Option | Magic | Directory | Partitioned | Meaning | Default |\n|--------|-------|-----------|-------------|---------|---------|\n| `fs.s3a.buffer.dir` | X | X | X | Local filesystem directory for data being written and/or staged. | |\n| `fs.s3a.committer.staging.tmp.path` | | X | X | Path in the cluster filesystem for temporary data | `tmp/staging` |\n\n\n```xml\n\n fs.s3a.committer.name\n file\n \n Committer to create for output to S3A, one of:\n \"file\", \"directory\", \"partitioned\", \"magic\".\n \n\n\n\n fs.s3a.committer.magic.enabled\n false\n \n Enable support in the filesystem for the S3 \"Magic\" committer.\n When working with AWS S3, S3Guard must be enabled for the destination\n bucket, as consistent metadata listings are required.\n \n\n\n\n fs.s3a.committer.threads\n 8\n \n Number of threads in committers for parallel operations on files\n (upload, commit, abort, delete...)\n \n\n\n\n fs.s3a.committer.staging.tmp.path\n tmp/staging\n \n Path in the cluster filesystem for temporary data.\n This is for HDFS, not the local filesystem.\n It is only for the summary data of each file, not the actual\n data being committed.\n Using an unqualified path guarantees that the full path will be\n generated relative to the home directory of the user creating the job,\n hence private (assuming home directory permissions are secure).\n \n\n\n\n fs.s3a.committer.staging.unique-filenames\n true\n \n Option for final files to have a unique name through job attempt info,\n or the value of fs.s3a.committer.staging.uuid\n When writing data with the \"append\" conflict option, this guarantees\n that new data will not overwrite any existing data.\n \n\n\n\n fs.s3a.committer.staging.conflict-mode\n fail\n \n Staging committer conflict resolution policy.\n Supported: \"fail\", \"append\", \"replace\".\n \n\n\n\n s.s3a.committer.staging.abort.pending.uploads\n true\n \n Should the staging committers abort all pending uploads to the destination\n directory?\n\n Changing this if more than one partitioned committer is\n writing to the same destination tree simultaneously; otherwise\n the first job to complete will cancel all outstanding uploads from the\n others. However, it may lead to leaked outstanding uploads from failed\n tasks. If disabled, configure the bucket lifecycle to remove uploads\n after a time period, and/or set up a workflow to explicitly delete\n entries. Otherwise there is a risk that uncommitted uploads may run up\n bills.\n \n\n\n\n mapreduce.outputcommitter.factory.scheme.s3a\n org.apache.hadoop.fs.s3a.commit.S3ACommitterFactory\n \n The committer factory to use when writing data to S3A filesystems.\n If mapreduce.outputcommitter.factory.class is set, it will\n override this property.\n\n (This property is set in mapred-default.xml)\n \n\n\n```\n\n\n## Troubleshooting\n\n### `Filesystem does not have support for 'magic' committer`\n\n```\norg.apache.hadoop.fs.s3a.commit.PathCommitException: `s3a://landsat-pds': Filesystem does not have support for 'magic' committer enabled\nin configuration option fs.s3a.committer.magic.enabled\n```\n\nThe Job is configured to use the magic committer, but the S3A bucket has not been explicitly\ndeclared as supporting it.\n\nThe destination bucket **must** be declared as supporting the magic committer.\n\nThis can be done for those buckets which are known to be consistent, either\nbecause [S3Guard](s3guard.html) is used to provide consistency,\nor because the S3-compatible filesystem is known to be strongly consistent.\n\n```xml\n\n fs.s3a.bucket.landsat-pds.committer.magic.enabled\n true\n\n```\n\n*IMPORTANT*: only enable the magic committer against object stores which\noffer consistent listings. By default, Amazon S3 does not do this -which is\nwhy the option `fs.s3a.committer.magic.enabled` is disabled by default.\n\n\nTip: you can verify that a bucket supports the magic committer through the\n`hadoop s3guard bucket-info` command:\n\n\n```\n> hadoop s3guard bucket-info -magic s3a://landsat-pds/\n\nFilesystem s3a://landsat-pds\nLocation: us-west-2\nFilesystem s3a://landsat-pds is not using S3Guard\nThe \"magic\" committer is not supported\n\nS3A Client\n Endpoint: fs.s3a.endpoint=(unset)\n Encryption: fs.s3a.server-side-encryption-algorithm=none\n Input seek policy: fs.s3a.experimental.input.fadvise=normal\n2017-09-27 19:18:57,917 INFO util.ExitUtil: Exiting with status 46: 46: The magic committer is not enabled for s3a://landsat-pds\n```\n\n## Error message: \"File being created has a magic path, but the filesystem has magic file support disabled:\n\nA file is being written to a path which is used for \"magic\" files,\nfiles which are actually written to a different destination than their stated path\n*but the filesystem doesn't support \"magic\" files*\n\nThis message should not appear through the committer itself —it will\nfail with the error message in the previous section, but may arise\nif other applications are attempting to create files under the path `/__magic/`.\n\nMake sure the filesystem meets the requirements of the magic committer\n(a consistent S3A filesystem through S3Guard or the S3 service itself),\nand set the `fs.s3a.committer.magic.enabled` flag to indicate that magic file\nwrites are supported.\n\n\n### `FileOutputCommitter` appears to be still used (from logs or delays in commits)\n\nThe Staging committers use the original `FileOutputCommitter` to manage\nthe propagation of commit information: do not worry if it the logs show `FileOutputCommitter`\nwork with data in the cluster filesystem (e.g. HDFS).\n\nOne way to make sure that the `FileOutputCommitter` is not being used to write\nthe data to S3 is to set the option `mapreduce.fileoutputcommitter.algorithm.version`\nto a value such as \"10\". Because the only supported algorithms are \"1\" and \"2\",\nany erroneously created `FileOutputCommitter` will raise an exception in its constructor\nwhen instantiated:\n\n```\njava.io.IOException: Only 1 or 2 algorithm version is supported\nat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:130)\nat org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:104)\nat org.apache.parquet.hadoop.ParquetOutputCommitter.(ParquetOutputCommitter.java:42)\nat org.apache.parquet.hadoop.ParquetOutputFormat.getOutputCommitter(ParquetOutputFormat.java:395)\nat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupCommitter(HadoopMapReduceCommitProtocol.scala:67)\nat com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol.setupCommitter(PathOutputCommitProtocol.scala:62)\nat org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.setupJob(HadoopMapReduceCommitProtocol.scala:124)\nat com.hortonworks.spark.cloud.commit.PathOutputCommitProtocol.setupJob(PathOutputCommitProtocol.scala:152)\nat org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:175)\nat org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:145)\n```\n\nWhile that will not make the problem go away, it will at least make\nthe failure happen at the start of a job.\n\n(Setting this option will not interfere with the Staging Committers' use of HDFS,\nas it explicitly sets the algorithm to \"2\" for that part of its work).\n\nThe other way to check which committer to use is to examine the `_SUCCESS` file.\nIf it is 0-bytes long, the classic `FileOutputCommitter` committed the job.\nThe S3A committers all write a non-empty JSON file; the `committer` field lists\nthe committer used.\n\n\n*Common causes*\n\n1. The property `fs.s3a.committer.name` is set to \"file\". Fix: change.\n1. The job has overridden the property `mapreduce.outputcommitter.factory.class`\nwith a new factory class for all committers. This takes priority over\nall committers registered for the s3a:// schema.\n1. The property `mapreduce.outputcommitter.factory.scheme.s3a` is unset.\n1. The output format has overridden `FileOutputFormat.getOutputCommitter()`\nand is returning its own committer -one which is a subclass of `FileOutputCommitter`.\n\nThat final cause. *the output format is returning its own committer*, is not\neasily fixed; it may be that the custom committer performs critical work\nduring its lifecycle, and contains assumptions about the state of the written\ndata during task and job commit (i.e. it is in the destination filesystem).\nConsult with the authors/maintainers of the output format\nto see whether it would be possible to integrate with the new committer factory\nmechanism and object-store-specific commit algorithms.\n\nParquet is a special case here: its committer does no extra work\nother than add the option to read all newly-created files then write a schema\nsummary. The Spark integration has explicit handling for Parquet to enable it\nto support the new committers, removing this (slow on S3) option.\n\nIf you have subclassed `FileOutputCommitter` and want to move to the\nfactory model, please get in touch.\n\n\n## Job/Task fails with PathExistsException: Destination path exists and committer conflict resolution mode is \"fail\"\n\nThis surfaces when either of two conditions are met.\n\n1. The Directory committer is used with `fs.s3a.committer.staging.conflict-mode` set to\n`fail` and the output/destination directory exists.\nThe job will fail in the driver during job setup.\n1. The Partitioned Committer is used with `fs.s3a.committer.staging.conflict-mode` set to\n`fail` and one of the partitions. The specific task(s) generating conflicting data will fail\nduring task commit, which will cause the entire job to fail.\n\nIf you are trying to write data and want write conflicts to be rejected, this is the correct\nbehavior: there was data at the destination so the job was aborted.\n\n## Staging committer task fails with IOException: No space left on device\n\nThere's not enough space on the local hard disk (real or virtual)\nto store all the uncommitted data of the active tasks on that host.\nBecause the staging committers write all output to the local disk\nand only upload the data on task commits, enough local temporary\nstorage is needed to store all output generated by all uncommitted\ntasks running on the single host. Small EC2 VMs may run out of disk.\n\n1. Make sure that `fs.s3a.buffer.dir` includes a temporary directory on\nevery available hard disk; this spreads load better.\n\n1. Add more disk space. In EC2: request instances with more local storage.\nThere is no need for EMR storage; this is just for temporary data.\n\n1. Purge the directories listed in `fs.s3a.buffer.dir` of old data.\nFailed tasks may not clean up all old files.\n\n1. Reduce the number of worker threads/process in the host.\n\n1. Consider partitioning the job into more tasks. This *may* result in more tasks\ngenerating less data each.\n\n1. Use the magic committer. This only needs enough disk storage to buffer\nblocks of the currently being written file during their upload process,\nso can use a lot less disk space.\n\n\n\n\n# Working with Encrypted S3 Data\n\n\n\n\n## Introduction\n\nThe S3A filesystem client supports Amazon S3's Server Side Encryption\nfor at-rest data encryption.\nYou should to read up on the [AWS documentation](https://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html)\nfor S3 Server Side Encryption for up to date information on the encryption mechanisms.\n\n\n\nWhen configuring an encryption method in the `core-site.xml`, this will apply cluster wide.\nAny new file written will be encrypted with this encryption configuration.\nWhen the S3A client reads a file, S3 will attempt to decrypt it using the mechanism\nand keys with which the file was encrypted.\n\n* It is **NOT** advised to mix and match encryption types in a bucket\n* It is much simpler and safer to encrypt with just one type and key per bucket.\n* You can use AWS bucket policies to mandate encryption rules for a bucket.\n* You can use S3A per-bucket configuration to ensure that S3A clients use encryption\npolicies consistent with the mandated rules.\n* You can use S3 Default Encryption to encrypt data without needing to\nset anything in the client.\n* Changing the encryption options on the client does not change how existing\nfiles were encrypted, except when the files are renamed.\n* For all mechanisms other than SSE-C, clients do not need any configuration\noptions set in order to read encrypted data: it is all automatically handled\nin S3 itself.\n\n## How data is encrypted\n\nAWS S3 supports server-side encryption inside the storage system itself.\nWhen an S3 client uploading data requests data to be encrypted, then an encryption key is used\nto encrypt the data as it saved to S3. It remains encrypted on S3 until deleted:\nclients cannot change the encryption attributes of an object once uploaded.\n\nThe Amazon AWS SDK also offers client-side encryption, in which all the encoding\nand decoding of data is performed on the client. This is *not* supported by\nthe S3A client.\n\nThe server-side \"SSE\" encryption is performed with symmetric AES256 encryption;\nS3 offers different mechanisms for actually defining the key to use.\n\n\nThere are four key management mechanisms, which in order of simplicity of use,\nare:\n\n* S3 Default Encryption\n* SSE-S3: an AES256 key is generated in S3, and saved alongside the data.\n* SSE-KMS: an AES256 key is generated in S3, and encrypted with a secret key provided\nby Amazon's Key Management Service, a key referenced by name in the uploading client.\n* SSE-C : the client specifies an actual base64 encoded AES-256 key to be used\nto encrypt and decrypt the data.\n\n\n## S3 Default Encryption\n\nThis feature allows the administrators of the AWS account to set the \"default\"\nencryption policy on a bucket -the encryption to use if the client does\nnot explicitly declare an encryption algorithm.\n\n[S3 Default Encryption for S3 Buckets](https://docs.aws.amazon.com/AmazonS3/latest/dev/bucket-encryption.html)\n\nThis supports SSE-S3 and SSE-KMS.\n\nThere is no need to set anything up in the client: do it in the AWS console.\n\n\n## SSE-S3 Amazon S3-Managed Encryption Keys\n\nIn SSE-S3, all keys and secrets are managed inside S3. This is the simplest encryption mechanism.\nThere is no extra cost for storing data with this option.\n\n\n### Enabling SSE-S3\n\nTo write S3-SSE encrypted files, the value of\n`fs.s3a.server-side-encryption-algorithm` must be set to that of\nthe encryption mechanism used in `core-site`; currently only `AES256` is supported.\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n AES256\n\n```\n\nOnce set, all new data will be stored encrypted. There is no need to set this property when downloading data — the data will be automatically decrypted when read using\nthe Amazon S3-managed key.\n\nTo learn more, refer to\n[Protecting Data Using Server-Side Encryption with Amazon S3-Managed Encryption Keys (SSE-S3) in AWS documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html).\n\n\n### SSE-KMS: Amazon S3-KMS Managed Encryption Keys\n\n\nAmazon offers a pay-per-use key management service, [AWS KMS](https://aws.amazon.com/documentation/kms/).\nThis service can be used to encrypt data on S3 by defining \"customer master keys\", CMKs,\nwhich can be centrally managed and assigned to specific roles and IAM accounts.\n\nThe AWS KMS [can be used encrypt data on S3uploaded data](http://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html).\n\n> The AWS KMS service is **not** related to the Key Management Service built into Hadoop (*Hadoop KMS*). The *Hadoop KMS* primarily focuses on\n managing keys for *HDFS Transparent Encryption*. Similarly, HDFS encryption is unrelated to S3 data encryption.\n\nWhen uploading data encrypted with SSE-KMS, the sequence is as follows.\n\n1. The S3A client must declare a specific CMK in the property `fs.s3a.server-side-encryption.key`, or leave\nit blank to use the default configured for that region.\n\n1. The S3A client uploads all the data as normal, now including encryption information.\n\n1. The S3 service encrypts the data with a symmetric key unique to the new object.\n\n1. The S3 service retrieves the chosen CMK key from the KMS service, and, if the user has\nthe right to use it, uses it to encrypt the object-specific key.\n\n\nWhen downloading SSE-KMS encrypted data, the sequence is as follows\n\n1. The S3A client issues an HTTP GET request to read the data.\n1. S3 sees that the data was encrypted with SSE-KMS, and looks up the specific key in the KMS service\n1. If and only if the requesting user has been granted permission to use the CMS key does\nthe KMS service provide S3 with the key.\n1. As a result, S3 will only decode the data if the user has been granted access to the key.\n\n\nKMS keys can be managed by an organization's administrators in AWS, including\nhaving access permissions assigned and removed from specific users, groups, and IAM roles.\nOnly those \"principals\" with granted rights to a key may access it,\nhence only they may encrypt data with the key, *and decrypt data encrypted with it*.\nThis allows KMS to be used to provide a cryptographically secure access control mechanism for data stores on S3.\n\n\nEach KMS server is region specific, and accordingly, so is each CMK configured.\nA CMK defined in one region cannot be used with an S3 bucket in a different region.\n\n\nNotes\n\n* Callers are charged for every use of a key, both for encrypting the data in uploads\n and for decrypting it when reading it back.\n* Random-access IO on files may result in multiple GET requests of an object during a read\nsequence (especially for columnar data), so may require more than one key retrieval to process a single file,\n* The KMS service is throttled: too many requests may cause requests to fail.\n* As well as incurring charges, heavy I/O *may* reach IO limits for a customer. If those limits are reached,\nthey can be increased through the AWS console.\n\n\n### Enabling SSE-KMS\n\nTo enable SSE-KMS, the property `fs.s3a.server-side-encryption-algorithm` must be set to `SSE-KMS` in `core-site`:\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n SSE-KMS\n\n```\n\nThe ID of the specific key used to encrypt the data should also be set in the property `fs.s3a.server-side-encryption.key`:\n\n```xml\n\n fs.s3a.server-side-encryption.key\n arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n```\n\nOrganizations may define a default key in the Amazon KMS; if a default key is set,\nthen it will be used whenever SSE-KMS encryption is chosen and the value of `fs.s3a.server-side-encryption.key` is empty.\n\n### the S3A `fs.s3a.encryption.key` key only affects created files\n\nWith SSE-KMS, the S3A client option `fs.s3a.server-side-encryption.key` sets the\nkey to be used when new files are created. When reading files, this key,\nand indeed the value of `fs.s3a.server-side-encryption-algorithme` is ignored:\nS3 will attempt to retrieve the key and decrypt the file based on the create-time settings.\n\nThis means that\n\n* There's no need to configure any client simply reading data.\n* It is possible for a client to read data encrypted with one KMS key, and\nwrite it with another.\n\n\n## SSE-C: Server side encryption with a client-supplied key.\n\nIn SSE-C, the client supplies the secret key needed to read and write data.\nEvery client trying to read or write data must be configured with the same\nsecret key.\n\n\nSSE-C integration with Hadoop is still stabilizing; issues related to it are still surfacing.\nIt is already clear that SSE-C with a common key must be used exclusively within\na bucket if it is to be used at all. This is the only way to ensure that path and\ndirectory listings do not fail with \"Bad Request\" errors.\n\n### Enabling SSE-C\n\nTo use SSE-C, the configuration option `fs.s3a.server-side-encryption-algorithm`\nmust be set to `SSE-C`, and a base-64 encoding of the key placed in\n`fs.s3a.server-side-encryption.key`.\n\n```xml\n\n fs.s3a.server-side-encryption-algorithm\n SSE-C\n\n\n\n fs.s3a.server-side-encryption.key\n SGVscCwgSSdtIHRyYXBwZWQgaW5zaWRlIGEgYmFzZS02NC1jb2RlYyE=\n\n```\n\nAll clients must share this same key.\n\n### The `fs.s3a.encryption.key` value is used to read and write data\n\nWith SSE-C, the S3A client option `fs.s3a.server-side-encryption.key` sets the\nkey to be used for both reading *and* writing data.\n\nWhen reading any file written with SSE-C, the same key must be set\nin the property `fs.s3a.server-side-encryption.key`.\n\nThis is unlike SSE-S3 and SSE-KMS, where the information needed to\ndecode data is kept in AWS infrastructure.\n\n\n### SSE-C Warning\n\nYou need to fully understand how SSE-C works in the S3\nenvironment before using this encryption type. Please refer to the Server Side\nEncryption documentation available from AWS. SSE-C is only recommended for\nadvanced users with advanced encryption use cases. Failure to properly manage\nencryption keys can cause data loss. Currently, the AWS S3 API(and thus S3A)\nonly supports one encryption key and cannot support decrypting objects during\nmoves under a previous key to a new destination. It is **NOT** advised to use\nmultiple encryption keys in a bucket, and is recommended to use one key per\nbucket and to not change this key. This is due to when a request is made to S3,\nthe actual encryption key must be provided to decrypt the object and access the\nmetadata. Since only one encryption key can be provided at a time, S3A will not\npass the correct encryption key to decrypt the data.\n\n\n## Encryption best practises\n\n\n### Mandate encryption through policies\n\nBecause it is up to the clients to enable encryption on new objects, all clients\nmust be correctly configured in order to guarantee that data is encrypted.\n\n\nTo mandate that all data uploaded to a bucket is encrypted,\nyou can set a [bucket policy](https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/)\ndeclaring that clients must provide encryption information with all data uploaded.\n\n\n* Mandating an encryption mechanism on newly uploaded data does not encrypt existing data; existing data will retain whatever encryption (if any) applied at the time of creation*\n\nHere is a policy to mandate `SSE-S3/AES265` encryption on all data uploaded to a bucket. This covers uploads as well as the copy operations which take place when file/directory rename operations are mimicked.\n\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Id\": \"EncryptionPolicy\",\n \"Statement\": [\n {\n \"Sid\": \"RequireEncryptionHeaderOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"Null\": {\n \"s3:x-amz-server-side-encryption\": true\n }\n }\n },\n {\n \"Sid\": \"RequireAESEncryptionOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"StringNotEquals\": {\n \"s3:x-amz-server-side-encryption\": \"AES256\"\n }\n }\n }\n ]\n}\n```\n\nTo use SSE-KMS, a different restriction must be defined:\n\n\n```json\n{\n \"Version\": \"2012-10-17\",\n \"Id\": \"EncryptionPolicy\",\n \"Statement\": [\n {\n \"Sid\": \"RequireEncryptionHeaderOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"Null\": {\n \"s3:x-amz-server-side-encryption\": true\n }\n }\n },\n {\n \"Sid\": \"RequireKMSEncryptionOnPut\",\n \"Effect\": \"Deny\",\n \"Principal\": \"*\",\n \"Action\": [\n \"s3:PutObject\"\n ],\n \"Resource\": \"arn:aws:s3:::BUCKET/*\",\n \"Condition\": {\n \"StringNotEquals\": {\n \"s3:x-amz-server-side-encryption\": \"SSE-KMS\"\n }\n }\n }\n ]\n}\n```\n\nTo use one of these policies:\n\n1. Replace `BUCKET` with the specific name of the bucket being secured.\n1. Locate the bucket in the AWS console [S3 section](https://console.aws.amazon.com/s3/home).\n1. Select the \"Permissions\" tab.\n1. Select the \"Bucket Policy\" tab in the permissions section.\n1. Paste the edited policy into the form.\n1. Save the policy.\n\n### Use S3a per-bucket configuration to control encryption settings\n\nIn an organisation which has embraced S3 encryption, different buckets inevitably have\ndifferent encryption policies, such as different keys for SSE-KMS encryption.\nIn particular, as different keys need to be named for different regions, unless\nyou rely on the administrator-managed \"default\" key for each S3 region, you\nwill need unique keys.\n\nS3A's per-bucket configuration enables this.\n\n\nHere, for example, are settings for a bucket in London, `london-stats`:\n\n\n```xml\n\n fs.s3a.bucket.london-stats.server-side-encryption-algorithm\n AES256\n\n```\n\nThis requests SSE-S; if matched with a bucket policy then all data will\nbe encrypted as it is uploaded.\n\n\nA different bucket can use a different policy\n(here SSE-KMS) and, when necessary, declare a key.\n\nHere is an example bucket in S3 Ireland, which uses SSE-KMS and\na KMS key hosted in the AWS-KMS service in the same region.\n\n\n```xml\n\n fs.s3a.bucket.ireland-dev.server-side-encryption-algorithm\n SSE-KMS\n\n\n\n fs.s3a.bucket.ireland-dev.server-side-encryption.key\n arn:aws:kms:eu-west-1:98067faff834c:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n\n```\n\nAgain the appropriate bucket policy can be used to guarantee that all callers\nwill use SSE-KMS; they can even mandate the name of the key used to encrypt\nthe data, so guaranteeing that access to thee data can be read by everyone\ngranted access to that key, and nobody without access to it.\n\n\n### Use rename() to encrypt files with new keys\n\nThe encryption of an object is set when it is uploaded. If you want to encrypt\nan unencrypted file, or change the SEE-KMS key of a file, the only way to do\nso is by copying the object.\n\nHow can you do that from Hadoop? With `rename()`.\n\nThe S3A client mimics a real filesystem's' rename operation by copying all the\nsource files to the destination paths, then deleting the old ones.\n\nNote: this does not work for SSE-C, because you cannot set a different key\nfor reading as for writing, and you must supply that key for reading. There\nyou need to copy one bucket to a different bucket, one with a different key.\nUse `distCp`for this, with per-bucket encryption policies.\n\n\n## Troubleshooting Encryption\n\nThe [troubleshooting](./troubleshooting_s3a.html) document covers\nstack traces which may surface when working with encrypted data.\n\n\n\n\n# Maximizing Performance when working with the S3A Connector\n\n\n\n\n## Introduction\n\nS3 is slower to work with than HDFS, even on virtual clusters running on\nAmazon EC2.\n\nThat's because its a very different system, as you can see:\n\n\n| Feature | HDFS | S3 through the S3A connector |\n|---------|------|------------------------------|\n| communication | RPC | HTTP GET/PUT/HEAD/LIST/COPY requests |\n| data locality | local storage | remote S3 servers |\n| replication | multiple datanodes | asynchronous after upload |\n| consistency | consistent data and listings | eventual consistent for listings, deletes and updates |\n| bandwidth | best: local IO, worst: datacenter network | bandwidth between servers and S3 |\n| latency | low | high, especially for \"low cost\" directory operations |\n| rename | fast, atomic | slow faked rename through COPY & DELETE|\n| delete | fast, atomic | fast for a file, slow & non-atomic for directories |\n| writing| incremental | in blocks; not visible until the writer is closed |\n| reading | seek() is fast | seek() is slow and expensive |\n| IOPs | limited only by hardware | callers are throttled to shards in an s3 bucket |\n| Security | Posix user+group; ACLs | AWS Roles and policies |\n\nFrom a performance perspective, key points to remember are:\n\n* S3 throttles bucket access across all callers: adding workers can make things worse.\n* EC2 VMs have network IO throttled based on the VM type.\n* Directory rename and copy operations take *much* longer the more objects and data there is.\nThe slow performance of `rename()` surfaces during the commit phase of jobs,\napplications like `DistCP`, and elsewhere.\n* seek() calls when reading a file can force new HTTP requests.\nThis can make reading columnar Parquet/ORC data expensive.\n\nOverall, although the S3A connector makes S3 look like a file system,\nit isn't, and some attempts to preserve the metaphor are \"aggressively suboptimal\".\n\nTo make most efficient use of S3, care is needed.\n\n## Speeding up directory listing operations through S3Guard\n\n[S3Guard](s3guard.html) provides significant speedups for operations which\nlist files a lot. This includes the setup of all queries against data:\nMapReduce, Hive and Spark, as well as DistCP.\n\n\nExperiment with using it to see what speedup it delivers.\n\n\n## Improving data input performance through fadvise\n\nThe S3A Filesystem client supports the notion of input policies, similar\nto that of the Posix `fadvise()` API call. This tunes the behavior of the S3A\nclient to optimise HTTP GET requests for the different use cases.\n\n### fadvise `sequential`\n\nRead through the file, possibly with some short forward seeks.\n\nThe whole document is requested in a single HTTP request; forward seeks\nwithin the readahead range are supported by skipping over the intermediate\ndata.\n\nThis delivers maximum sequential throughput —but with very expensive\nbackward seeks.\n\nApplications reading a file in bulk (DistCP, any copy operations) should use\nsequential access, as should those reading data from gzipped `.gz` files.\nBecause the \"normal\" fadvise policy starts off in sequential IO mode,\nthere is rarely any need to explicit request this policy.\n\n### fadvise `random`\n\nOptimised for random IO, specifically the Hadoop `PositionedReadable`\noperations —though `seek(offset); read(byte_buffer)` also benefits.\n\nRather than ask for the whole file, the range of the HTTP request is\nset to that that of the length of data desired in the `read` operation\n(Rounded up to the readahead value set in `setReadahead()` if necessary).\n\nBy reducing the cost of closing existing HTTP requests, this is\nhighly efficient for file IO accessing a binary file\nthrough a series of `PositionedReadable.read()` and `PositionedReadable.readFully()`\ncalls. Sequential reading of a file is expensive, as now many HTTP requests must\nbe made to read through the file: there's a delay between each GET operation.\n\n\nRandom IO is best for IO with seek-heavy characteristics:\n\n* Data is read using the `PositionedReadable` API.\n* Long distance (many MB) forward seeks\n* Backward seeks as likely as forward seeks.\n* Little or no use of single character `read()` calls or small `read(buffer)`\ncalls.\n* Applications running close to the S3 data store. That is: in EC2 VMs in\nthe same datacenter as the S3 instance.\n\nThe desired fadvise policy must be set in the configuration option\n`fs.s3a.experimental.input.fadvise` when the filesystem instance is created.\nThat is: it can only be set on a per-filesystem basis, not on a per-file-read\nbasis.\n\n```xml\n\n fs.s3a.experimental.input.fadvise\n random\n \n Policy for reading files.\n Values: 'random', 'sequential' or 'normal'\n \n\n```\n\n[HDFS-2744](https://issues.apache.org/jira/browse/HDFS-2744),\n*Extend FSDataInputStream to allow fadvise* proposes adding a public API\nto set fadvise policies on input streams. Once implemented,\nthis will become the supported mechanism used for configuring the input IO policy.\n\n### fadvise `normal` (default)\n\nThe `normal` policy starts off reading a file in `sequential` mode,\nbut if the caller seeks backwards in the stream, it switches from\nsequential to `random`.\n\nThis policy essentially recognizes the initial read pattern of columnar\nstorage formats (e.g. Apache ORC and Apache Parquet), which seek to the end\nof a file, read in index data and then seek backwards to selectively read\ncolumns. The first seeks may be be expensive compared to the random policy,\nhowever the overall process is much less expensive than either sequentially\nreading through a file with the `random` policy, or reading columnar data\nwith the `sequential` policy.\n\n\n## Committing Work in MapReduce and Spark\n\nHadoop MapReduce, Apache Hive and Apache Spark all write their work\nto HDFS and similar filesystems.\nWhen using S3 as a destination, this is slow because of the way `rename()`\nis mimicked with copy and delete.\n\nIf committing output takes a long time, it is because you are using the standard\n`FileOutputCommitter`. If you are doing this on any S3 endpoint which lacks\nlist consistency (Amazon S3 without [S3Guard](s3guard.html)), this committer\nis at risk of losing data!\n\n*Your problem may appear to be performance, but that is a symptom\nof the underlying problem: the way S3A fakes rename operations means that\nthe rename cannot be safely be used in output-commit algorithms.*\n\nFix: Use one of the dedicated [S3A Committers](committers.md).\n\n## Options to Tune\n\n### Thread and connection pool sizes.\n\nEach S3A client interacting with a single bucket, as a single user, has its\nown dedicated pool of open HTTP 1.1 connections alongside a pool of threads used\nfor upload and copy operations.\nThe default pool sizes are intended to strike a balance between performance\nand memory/thread use.\n\nYou can have a larger pool of (reused) HTTP connections and threads\nfor parallel IO (especially uploads) by setting the properties\n\n\n| property | meaning | default |\n|----------|---------|---------|\n| `fs.s3a.threads.max`| Threads in the AWS transfer manager| 10 |\n| `fs.s3a.connection.maximum`| Maximum number of HTTP connections | 10|\n\nWe recommend using larger values for processes which perform\na lot of IO: `DistCp`, Spark Workers and similar.\n\n```xml\n\n fs.s3a.threads.max\n 20\n\n\n fs.s3a.connection.maximum\n 20\n\n```\n\nBe aware, however, that processes which perform many parallel queries\nmay consume large amounts of resources if each query is working with\na different set of s3 buckets, or are acting on behalf of different users.\n\n### For large data uploads, tune the block size: `fs.s3a.block.size`\n\nWhen uploading data, it is uploaded in blocks set by the option\n`fs.s3a.block.size`; default value \"32M\" for 32 Megabytes.\n\nIf a larger value is used, then more data is buffered before the upload\nbegins:\n\n```xml\n\n fs.s3a.block.size\n 128M\n\n```\n\nThis means that fewer PUT/POST requests are made of S3 to upload data,\nwhich reduces the likelihood that S3 will throttle the client(s)\n\n### Maybe: Buffer Write Data in Memory\n\nWhen large files are being uploaded, blocks are saved to disk and then\nqueued for uploading, with multiple threads uploading different blocks\nin parallel.\n\nThe blocks can be buffered in memory by setting the option\n`fs.s3a.fast.upload.buffer` to `bytebuffer`, or, for on-heap storage\n`array`.\n\n1. Switching to in memory-IO reduces disk IO, and can be faster if the bandwidth\nto the S3 store is so high that the disk IO becomes the bottleneck.\nThis can have a tangible benefit when working with on-premise S3-compatible\nobject stores with very high bandwidth to servers.\n\nIt is very easy to run out of memory when buffering to it; the option\n`fs.s3a.fast.upload.active.blocks\"` exists to tune how many active blocks\na single output stream writing to S3 may have queued at a time.\n\nAs the size of each buffered block is determined by the value of `fs.s3a.block.size`,\nthe larger the block size, the more likely you will run out of memory.\n\n## DistCP\n\nDistCP can be slow, especially if the parameters and options for the operation\nare not tuned for working with S3.\n\nTo exacerbate the issue, DistCP invariably puts heavy load against the\nbucket being worked with, which will cause S3 to throttle requests.\nIt will throttle: directory operations, uploads of new data, and delete operations,\namongst other things\n\n### DistCP: Options to Tune\n\n* `-numListstatusThreads ` : set to something higher than the default (1).\n* `-bandwidth ` : use to limit the upload bandwidth per worker\n* `-m ` : limit the number of mappers, hence the load on the S3 bucket.\n\nAdding more maps with the `-m` option does not guarantee better performance;\nit may just increase the amount of throttling which takes place.\nA smaller number of maps with a higher bandwidth per map can be more efficient.\n\n### DistCP: Options to Avoid.\n\nDistCp's `-atomic` option copies up data into a directory, then renames\nit into place, which is the where the copy takes place. This is a performance\nkiller.\n\n* Do not use the `-atomic` option.\n* The `-append` operation is not supported on S3; avoid.\n* `-p` S3 does not have a POSIX-style permission model; this will fail.\n\n\n### DistCP: Parameters to Tune\n\n1. As discussed [earlier](#pooling), use large values for\n`fs.s3a.threads.max` and `fs.s3a.connection.maximum`.\n\n1. Make sure that the bucket is using `sequential` or `normal` fadvise seek policies,\nthat is, `fs.s3a.experimental.fadvise` is not set to `random`\n\n1. Perform listings in parallel by setting `-numListstatusThreads`\nto a higher number. Make sure that `fs.s3a.connection.maximum`\nis equal to or greater than the value used.\n\n1. If using `-delete`, set `fs.trash.interval` to 0 to avoid the deleted\nobjects from being copied to a trash directory.\n\n*DO NOT* switch `fs.s3a.fast.upload.buffer` to buffer in memory.\nIf one distcp mapper runs out of memory it will fail,\nand that runs the risk of failing the entire job.\nIt is safer to keep the default value, `disk`.\n\nWhat is potentially useful is uploading in bigger blocks; this is more\nefficient in terms of HTTP connection use, and reduce the IOP rate against\nthe S3 bucket/shard.\n\n```xml\n\n fs.s3a.threads.max\n 20\n\n\n\n fs.s3a.connection.maximum\n 30\n \n Make greater than both fs.s3a.threads.max and -numListstatusThreads\n \n\n\n\n fs.s3a.experimental.fadvise\n normal\n\n\n\n fs.s3a.block.size\n 128M\n\n\n\n fs.s3a.fast.upload.buffer\n disk\n\n\n\n fs.trash.interval\n 0\n\n```\n\n## hadoop shell commands `fs -rm`\n\nThe `hadoop fs -rm` command can rename the file under `.Trash` rather than\ndeleting it. Use `-skipTrash` to eliminate that step.\n\n\nThis can be set in the property `fs.trash.interval`; while the default is 0,\nmost HDFS deployments have it set to a non-zero value to reduce the risk of\ndata loss.\n\n```xml\n\n fs.trash.interval\n 0\n\n```\n\n\n## Improving S3 load-balancing behavior\n\nAmazon S3 uses a set of front-end servers to provide access to the underlying data.\nThe choice of which front-end server to use is handled via load-balancing DNS\nservice: when the IP address of an S3 bucket is looked up, the choice of which\nIP address to return to the client is made based on the the current load\nof the front-end servers.\n\nOver time, the load across the front-end changes, so those servers considered\n\"lightly loaded\" will change. If the DNS value is cached for any length of time,\nyour application may end up talking to an overloaded server. Or, in the case\nof failures, trying to talk to a server that is no longer there.\n\nAnd by default, for historical security reasons in the era of applets,\nthe DNS TTL of a JVM is \"infinity\".\n\nTo work with AWS better, set the DNS time-to-live of an application which\nworks with S3 to something lower.\nSee [AWS documentation](http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-jvm-ttl.html).\n\n## Troubleshooting network performance\n\nAn example of this is covered in [HADOOP-13871](https://issues.apache.org/jira/browse/HADOOP-13871).\n\n1. For public data, use `curl`:\n\n curl -O https://landsat-pds.s3.amazonaws.com/scene_list.gz\n1. Use `nettop` to monitor a processes connections.\n\n\n## Throttling\n\nWhen many requests are made of a specific S3 bucket (or shard inside it),\nS3 will respond with a 503 \"throttled\" response.\nThrottling can be recovered from, provided overall load decreases.\nFurthermore, because it is sent before any changes are made to the object store,\nis inherently idempotent. For this reason, the client will always attempt to\nretry throttled requests.\n\nThe limit of the number of times a throttled request can be retried,\nand the exponential interval increase between attempts, can be configured\nindependently of the other retry limits.\n\n```xml\n\n fs.s3a.retry.throttle.limit\n 20\n \n Number of times to retry any throttled request.\n \n\n\n\n fs.s3a.retry.throttle.interval\n 500ms\n \n Interval between retry attempts on throttled requests.\n \n\n```\n\nIf a client is failing due to `AWSServiceThrottledException` failures,\nincreasing the interval and limit *may* address this. However, it\nit is a sign of AWS services being overloaded by the sheer number of clients\nand rate of requests. Spreading data across different buckets, and/or using\na more balanced directory structure may be beneficial.\nConsult [the AWS documentation](http://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html).\n\nReading or writing data encrypted with SSE-KMS forces S3 to make calls of\nthe AWS KMS Key Management Service, which comes with its own\n[Request Rate Limits](http://docs.aws.amazon.com/kms/latest/developerguide/limits.html).\nThese default to 1200/second for an account, across all keys and all uses of\nthem, which, for S3 means: across all buckets with data encrypted with SSE-KMS.\n\n### Tips to Keep Throttling down\n\nIf you are seeing a lot of throttling responses on a large scale\noperation like a `distcp` copy, *reduce* the number of processes trying\nto work with the bucket (for distcp: reduce the number of mappers with the\n`-m` option).\n\nIf you are reading or writing lists of files, if you can randomize\nthe list so they are not processed in a simple sorted order, you may\nreduce load on a specific shard of S3 data, so potentially increase throughput.\n\nAn S3 Bucket is throttled by requests coming from all\nsimultaneous clients. Different applications and jobs may interfere with\neach other: consider that when troubleshooting.\nPartitioning data into different buckets may help isolate load here.\n\nIf you are using data encrypted with SSE-KMS, then the\nwill also apply: these are stricter than the S3 numbers.\nIf you believe that you are reaching these limits, you may be able to\nget them increased.\nConsult [the KMS Rate Limit documentation](http://docs.aws.amazon.com/kms/latest/developerguide/limits.html).\n\n### S3Guard and Throttling\n\n\nS3Guard uses DynamoDB for directory and file lookups;\nit is rate limited to the amount of (guaranteed) IO purchased for a\ntable.\n\nTo see the allocated capacity of a bucket, the `hadoop s3guard bucket-info s3a://bucket`\ncommand will print out the allocated capacity.\n\n\nIf significant throttling events/rate is observed here, the pre-allocated\nIOPs can be increased with the `hadoop s3guard set-capacity` command, or\nthrough the AWS Console. Throttling events in S3Guard are noted in logs, and\nalso in the S3A metrics `s3guard_metadatastore_throttle_rate` and\n`s3guard_metadatastore_throttled`.\n\nIf you are using DistCP for a large backup to/from a S3Guarded bucket, it is\nactually possible to increase the capacity for the duration of the operation.\n\n\n## Best Practises for Code\n\nHere are some best practises if you are writing applications to work with\nS3 or any other object store through the Hadoop APIs.\n\nUse `listFiles(path, recursive)` over `listStatus(path)`.\nThe recursive `listFiles()` call can enumerate all dependents of a path\nin a single LIST call, irrespective of how deep the path is.\nIn contrast, any directory tree-walk implemented in the client is issuing\nmultiple HTTP requests to scan each directory, all the way down.\n\nCache the outcome of `getFileStats()`, rather than repeatedly ask for it.\nThat includes using `isFile()`, `isDirectory()`, which are simply wrappers\naround `getFileStatus()`.\n\nDon't immediately look for a file with a `getFileStatus()` or listing call\nafter creating it, or try to read it immediately.\nThis is where eventual consistency problems surface: the data may not yet be visible.\n\nRely on `FileNotFoundException` being raised if the source of an operation is\nmissing, rather than implementing your own probe for the file before\nconditionally calling the operation.\n\n### `rename()`\n\nAvoid any algorithm which uploads data into a temporary file and then uses\n`rename()` to commit it into place with a final path.\nOn HDFS this offers a fast commit operation.\nWith S3, Wasb and other object stores, you can write straight to the destination,\nknowing that the file isn't visible until you close the write: the write itself\nis atomic.\n\nThe `rename()` operation may return `false` if the source is missing; this\nis a weakness in the API. Consider a check before calling rename, and if/when\na new rename() call is made public, switch to it.\n\n\n### `delete(path, recursive)`\n\nKeep in mind that `delete(path, recursive)` is a no-op if the path does not exist, so\nthere's no need to have a check for the path existing before you call it.\n\n`delete()` is often used as a cleanup operation.\nWith an object store this is slow, and may cause problems if the caller\nexpects an immediate response. For example, a thread may block so long\nthat other liveness checks start to fail.\nConsider spawning off an executor thread to do these background cleanup operations.\n\n\n\n\n# Testing the S3A filesystem client and its features, including S3Guard\n\n\n\nThis module includes both unit tests, which can run in isolation without\nconnecting to the S3 service, and integration tests, which require a working\nconnection to S3 to interact with a bucket. Unit test suites follow the naming\nconvention `Test*.java`. Integration tests follow the naming convention\n`ITest*.java`.\n\nDue to eventual consistency, integration tests may fail without reason.\nTransient failures, which no longer occur upon rerunning the test, should thus\nbe ignored.\n\n## Policy for submitting patches which affect the `hadoop-aws` module.\n\nThe Apache Jenkins infrastructure does not run any S3 integration tests,\ndue to the need to keep credentials secure.\n\n### The submitter of any patch is required to run all the integration tests and declare which S3 region/implementation they used.\n\nThis is important: **patches which do not include this declaration will be ignored**\n\nThis policy has proven to be the only mechanism to guarantee full regression\ntesting of code changes. Why the declaration of region? Two reasons\n\n1. It helps us identify regressions which only surface against specific endpoints\nor third-party implementations of the S3 protocol.\n1. It forces the submitters to be more honest about their testing. It's easy\nto lie, \"yes, I tested this\". To say \"yes, I tested this against S3 US-west\"\nis a more specific lie and harder to make. And, if you get caught out: you\nlose all credibility with the project.\n\nYou don't need to test from a VM within the AWS infrastructure; with the\n`-Dparallel=tests` option the non-scale tests complete in under ten minutes.\nBecause the tests clean up after themselves, they are also designed to be low\ncost. It's neither hard nor expensive to run the tests; if you can't,\nthere's no guarantee your patch works. The reviewers have enough to do, and\ndon't have the time to do these tests, especially as every failure will simply\nmake for a slow iterative development.\n\nPlease: run the tests. And if you don't, we are sorry for declining your\npatch, but we have to.\n\n\n### What if there's an intermittent failure of a test?\n\nSome of the tests do fail intermittently, especially in parallel runs.\nIf this happens, try to run the test on its own to see if the test succeeds.\n\nIf it still fails, include this fact in your declaration. We know some tests\nare intermittently unreliable.\n\n### What if the tests are timing out or failing over my network connection?\n\nThe tests and the S3A client are designed to be configurable for different\ntimeouts. If you are seeing problems and this configuration isn't working,\nthat's a sign of the configuration mechanism isn't complete. If it's happening\nin the production code, that could be a sign of a problem which may surface\nover long-haul connections. Please help us identify and fix these problems\n— especially as you are the one best placed to verify the fixes work.\n\n## Setting up the tests\n\nTo integration test the S3* filesystem clients, you need to provide\n`auth-keys.xml` which passes in authentication details to the test runner.\n\nIt is a Hadoop XML configuration file, which must be placed into\n`hadoop-tools/hadoop-aws/src/test/resources`.\n\n### File `core-site.xml`\n\nThis file pre-exists and sources the configurations created\nunder `auth-keys.xml`.\n\nFor most purposes you will not need to edit this file unless you\nneed to apply a specific, non-default property change during the tests.\n\n### File `auth-keys.xml`\n\nThe presence of this file triggers the testing of the S3 classes.\n\nWithout this file, *none of the integration tests in this module will be\nexecuted*.\n\nThe XML file must contain all the ID/key information needed to connect\neach of the filesystem clients to the object stores, and a URL for\neach filesystem for its testing.\n\n1. `test.fs.s3a.name` : the URL of the bucket for S3a tests\n1. `fs.contract.test.fs.s3a` : the URL of the bucket for S3a filesystem contract tests\n\n\nThe contents of the bucket will be destroyed during the test process:\ndo not use the bucket for any purpose other than testing. Furthermore, for\ns3a, all in-progress multi-part uploads to the bucket will be aborted at the\nstart of a test (by forcing `fs.s3a.multipart.purge=true`) to clean up the\ntemporary state of previously failed tests.\n\nExample:\n\n```xml\n\n\n \n test.fs.s3a.name\n s3a://test-aws-s3a/\n \n\n \n fs.contract.test.fs.s3a\n ${test.fs.s3a.name}\n \n\n \n fs.s3a.access.key\n AWS access key ID. Omit for IAM role-based authentication.\n DONOTCOMMITTHISKEYTOSCM\n \n\n \n fs.s3a.secret.key\n AWS secret key. Omit for IAM role-based authentication.\n DONOTEVERSHARETHISSECRETKEY!\n \n\n \n test.sts.endpoint\n Specific endpoint to use for STS requests.\n sts.amazonaws.com\n \n\n\n```\n\n### Configuring S3a Encryption\n\nFor S3a encryption tests to run correctly, the\n`fs.s3a.server-side-encryption.key` must be configured in the s3a contract xml\nfile with a AWS KMS encryption key arn as this value is different for each AWS\nKMS.\n\nExample:\n\n```xml\n\n fs.s3a.server-side-encryption.key\n arn:aws:kms:us-west-2:360379543683:key/071a86ff-8881-4ba0-9230-95af6d01ca01\n\n```\n\nYou can also force all the tests to run with a specific SSE encryption method\nby configuring the property `fs.s3a.server-side-encryption-algorithm` in the s3a\ncontract file.\n\n## Running the Tests\n\nAfter completing the configuration, execute the test run through Maven.\n\n```bash\nmvn clean verify\n```\n\nIt's also possible to execute multiple test suites in parallel by passing the\n`parallel-tests` property on the command line. The tests spend most of their\ntime blocked on network I/O with the S3 service, so running in parallel tends to\ncomplete full test runs faster.\n\n```bash\nmvn -Dparallel-tests clean verify\n```\n\nSome tests must run with exclusive access to the S3 bucket, so even with the\n`parallel-tests` property, several test suites will run in serial in a separate\nMaven execution step after the parallel tests.\n\nBy default, `parallel-tests` runs 4 test suites concurrently. This can be tuned\nby passing the `testsThreadCount` property.\n\n```bash\nmvn -Dparallel-tests -DtestsThreadCount=8 clean verify\n```\n\nTo run just unit tests, which do not require S3 connectivity or AWS credentials,\nuse any of the above invocations, but switch the goal to `test` instead of\n`verify`.\n\n```bash\nmvn clean test\n\nmvn -Dparallel-tests clean test\n\nmvn -Dparallel-tests -DtestsThreadCount=8 clean test\n```\n\nTo run only a specific named subset of tests, pass the `test` property for unit\ntests or the `it.test` property for integration tests.\n\n```bash\nmvn clean test -Dtest=TestS3AInputPolicies\n\nmvn clean verify -Dit.test=ITestS3AFileContextStatistics -Dtest=none\n\nmvn clean verify -Dtest=TestS3A* -Dit.test=ITestS3A*\n```\n\nNote that when running a specific subset of tests, the patterns passed in `test`\nand `it.test` override the configuration of which tests need to run in isolation\nin a separate serial phase (mentioned above). This can cause unpredictable\nresults, so the recommendation is to avoid passing `parallel-tests` in\ncombination with `test` or `it.test`. If you know that you are specifying only\ntests that can run safely in parallel, then it will work. For wide patterns,\nlike `ITestS3A*` shown above, it may cause unpredictable test failures.\n\n### Testing against different regions\n\nS3A can connect to different regions —the tests support this. Simply\ndefine the target region in `auth-keys.xml`.\n\n```xml\n\n fs.s3a.endpoint\n s3.eu-central-1.amazonaws.com\n\n```\nThis is used for all tests expect for scale tests using a Public CSV.gz file\n(see below)\n\n### CSV Data Tests\n\nThe `TestS3AInputStreamPerformance` tests require read access to a multi-MB\ntext file. The default file for these tests is one published by amazon,\n[s3a://landsat-pds.s3.amazonaws.com/scene_list.gz](http://landsat-pds.s3.amazonaws.com/scene_list.gz).\nThis is a gzipped CSV index of other files which amazon serves for open use.\n\nThe path to this object is set in the option `fs.s3a.scale.test.csvfile`,\n\n```xml\n\n fs.s3a.scale.test.csvfile\n s3a://landsat-pds/scene_list.gz\n\n```\n\n1. If the option is not overridden, the default value is used. This\nis hosted in Amazon's US-east datacenter.\n1. If `fs.s3a.scale.test.csvfile` is empty, tests which require it will be skipped.\n1. If the data cannot be read for any reason then the test will fail.\n1. If the property is set to a different path, then that data must be readable\nand \"sufficiently\" large.\n\n(the reason the space or newline is needed is to add \"an empty entry\"; an empty\n`` would be considered undefined and pick up the default)\n\nOf using a test file in an S3 region requiring a different endpoint value\nset in `fs.s3a.endpoint`, a bucket-specific endpoint must be defined.\nFor the default test dataset, hosted in the `landsat-pds` bucket, this is:\n\n```xml\n\n fs.s3a.bucket.landsat-pds.endpoint\n s3.amazonaws.com\n The endpoint for s3a://landsat-pds URLs\n\n```\n\n## Viewing Integration Test Reports\n\n\nIntegration test results and logs are stored in `target/failsafe-reports/`.\nAn HTML report can be generated during site generation, or with the `surefire-report`\nplugin:\n\n```bash\nmvn surefire-report:failsafe-report-only\n```\n## Scale Tests\n\nThere are a set of tests designed to measure the scalability and performance\nat scale of the S3A tests, *Scale Tests*. Tests include: creating\nand traversing directory trees, uploading large files, renaming them,\ndeleting them, seeking through the files, performing random IO, and others.\nThis makes them a foundational part of the benchmarking.\n\nBy their very nature they are slow. And, as their execution time is often\nlimited by bandwidth between the computer running the tests and the S3 endpoint,\nparallel execution does not speed these tests up.\n\n***Note: Running scale tests with -Ds3guard and -Ddynamo requires that\nyou use a private, testing-only DynamoDB table.*** The tests do disruptive\nthings such as deleting metadata and setting the provisioned throughput\nto very low values.\n\n### Enabling the Scale Tests\n\nThe tests are enabled if the `scale` property is set in the maven build\nthis can be done regardless of whether or not the parallel test profile\nis used\n\n```bash\nmvn verify -Dscale\n\nmvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8\n```\n\nThe most bandwidth intensive tests (those which upload data) always run\nsequentially; those which are slow due to HTTPS setup costs or server-side\nactions are included in the set of parallelized tests.\n\n\n### Tuning scale options from Maven\n\n\nSome of the tests can be tuned from the maven build or from the\nconfiguration file used to run the tests.\n\n```bash\nmvn verify -Dparallel-tests -Dscale -DtestsThreadCount=8 -Dfs.s3a.scale.test.huge.filesize=128M\n```\n\nThe algorithm is\n\n1. The value is queried from the configuration file, using a default value if\nit is not set.\n1. The value is queried from the JVM System Properties, where it is passed\ndown by maven.\n1. If the system property is null, an empty string, or it has the value `unset`,\nthen the configuration value is used. The `unset` option is used to\n[work round a quirk in maven property propagation](http://stackoverflow.com/questions/7773134/null-versus-empty-arguments-in-maven).\n\nOnly a few properties can be set this way; more will be added.\n\n| Property | Meaning |\n|-----------|-------------|\n| `fs.s3a.scale.test.timeout`| Timeout in seconds for scale tests |\n| `fs.s3a.scale.test.huge.filesize`| Size for huge file uploads |\n| `fs.s3a.scale.test.huge.huge.partitionsize`| Size for partitions in huge file uploads |\n\nThe file and partition sizes are numeric values with a k/m/g/t/p suffix depending\non the desired size. For example: 128M, 128m, 2G, 2G, 4T or even 1P.\n\n### Scale test configuration options\n\nSome scale tests perform multiple operations (such as creating many directories).\n\nThe exact number of operations to perform is configurable in the option\n`scale.test.operation.count`\n\n```xml\n\n scale.test.operation.count\n 10\n\n```\n\nLarger values generate more load, and are recommended when testing locally,\nor in batch runs.\n\nSmaller values results in faster test runs, especially when the object\nstore is a long way away.\n\nOperations which work on directories have a separate option: this controls\nthe width and depth of tests creating recursive directories. Larger\nvalues create exponentially more directories, with consequent performance\nimpact.\n\n```xml\n\n scale.test.directory.count\n 2\n\n```\n\nDistCp tests targeting S3A support a configurable file size. The default is\n10 MB, but the configuration value is expressed in KB so that it can be tuned\nsmaller to achieve faster test runs.\n\n```xml\n\n scale.test.distcp.file.size.kb\n 10240\n\n```\n\nS3A specific scale test properties are\n\n*`fs.s3a.scale.test.huge.filesize`: size in MB for \"Huge file tests\".*\n\nThe Huge File tests validate S3A's ability to handle large files —the property\n`fs.s3a.scale.test.huge.filesize` declares the file size to use.\n\n```xml\n\n fs.s3a.scale.test.huge.filesize\n 200M\n\n```\n\nAmazon S3 handles files larger than 5GB differently than smaller ones.\nSetting the huge filesize to a number greater than that) validates support\nfor huge files.\n\n```xml\n\n fs.s3a.scale.test.huge.filesize\n 6G\n\n```\n\nTests at this scale are slow: they are best executed from hosts running in\nthe cloud infrastructure where the S3 endpoint is based.\nOtherwise, set a large timeout in `fs.s3a.scale.test.timeout`\n\n```xml\n\n fs.s3a.scale.test.timeout\n 432000\n\n```\n\nThe tests are executed in an order to only clean up created files after\nthe end of all the tests. If the tests are interrupted, the test data will remain.\n\n\n## Testing against non AWS S3 endpoints.\n\nThe S3A filesystem is designed to work with storage endpoints which implement\nthe S3 protocols to the extent that the amazon S3 SDK is capable of talking\nto it. We encourage testing against other filesystems and submissions of patches\nwhich address issues. In particular, we encourage testing of Hadoop release\ncandidates, as these third-party endpoints get even less testing than the\nS3 endpoint itself.\n\n\n### Disabling the encryption tests\n\nIf the endpoint doesn't support server-side-encryption, these will fail. They\ncan be turned off.\n\n```xml\n\n test.fs.s3a.encryption.enabled\n false\n\n```\n\nEncryption is only used for those specific test suites with `Encryption` in\ntheir classname.\n\n### Configuring the CSV file read tests**\n\nTo test on alternate infrastructures supporting\nthe same APIs, the option `fs.s3a.scale.test.csvfile` must either be\nset to \" \", or an object of at least 10MB is uploaded to the object store, and\nthe `fs.s3a.scale.test.csvfile` option set to its path.\n\n```xml\n\n fs.s3a.scale.test.csvfile\n \n\n```\n\n(yes, the space is necessary. The Hadoop `Configuration` class treats an empty\nvalue as \"do not override the default\").\n\n\n### Testing Session Credentials\n\nThe test `TestS3ATemporaryCredentials` requests a set of temporary\ncredentials from the STS service, then uses them to authenticate with S3.\n\nIf an S3 implementation does not support STS, then the functional test\ncases must be disabled:\n\n```xml\n\n test.fs.s3a.sts.enabled\n false\n\n```\nThese tests request a temporary set of credentials from the STS service endpoint.\nAn alternate endpoint may be defined in `test.fs.s3a.sts.endpoint`.\n\n```xml\n\n test.fs.s3a.sts.endpoint\n https://sts.example.org/\n\n```\nThe default is \"\"; meaning \"use the amazon default value\".\n\n\n## Debugging Test failures\n\nLogging at debug level is the standard way to provide more diagnostics output;\nafter setting this rerun the tests\n\n```properties\nlog4j.logger.org.apache.hadoop.fs.s3a=DEBUG\n```\n\nThere are also some logging options for debug logging of the AWS client\n```properties\nlog4j.logger.com.amazonaws=DEBUG\nlog4j.logger.com.amazonaws.http.conn.ssl=INFO\nlog4j.logger.com.amazonaws.internal=INFO\n```\n\nThere is also the option of enabling logging on a bucket; this could perhaps\nbe used to diagnose problems from that end. This isn't something actively\nused, but remains an option. If you are forced to debug this way, consider\nsetting the `fs.s3a.user.agent.prefix` to a unique prefix for a specific\ntest run, which will enable the specific log entries to be more easily\nlocated.\n\n## Adding new tests\n\nNew tests are always welcome. Bear in mind that we need to keep costs\nand test time down, which is done by\n\n* Not duplicating tests.\n* Being efficient in your use of Hadoop API calls.\n* Isolating large/slow tests into the \"scale\" test group.\n* Designing all tests to execute in parallel (where possible).\n* Adding new probes and predicates into existing tests, albeit carefully.\n\n*No duplication*: if an operation is tested elsewhere, don't repeat it. This\napplies as much for metadata operations as it does for bulk IO. If a new\ntest case is added which completely obsoletes an existing test, it is OK\nto cut the previous one —after showing that coverage is not worsened.\n\n*Efficient*: prefer the `getFileStatus()` and examining the results, rather than\ncall to `exists()`, `isFile()`, etc.\n\n*Isolating Scale tests*. Any S3A test doing large amounts of IO MUST extend the\nclass `S3AScaleTestBase`, so only running if `scale` is defined on a build,\nsupporting test timeouts configurable by the user. Scale tests should also\nsupport configurability as to the actual size of objects/number of operations,\nso that behavior at different scale can be verified.\n\n*Designed for parallel execution*. A key need here is for each test suite to work\non isolated parts of the filesystem. Subclasses of `AbstractS3ATestBase`\nSHOULD use the `path()` method, with a base path of the test suite name, to\nbuild isolated paths. Tests MUST NOT assume that they have exclusive access\nto a bucket.\n\n*Extending existing tests where appropriate*. This recommendation goes\nagainst normal testing best practise of \"test one thing per method\".\nBecause it is so slow to create directory trees or upload large files, we do\nnot have that luxury. All the tests against real S3 endpoints are integration\ntests where sharing test setup and teardown saves time and money.\n\nA standard way to do this is to extend existing tests with some extra predicates,\nrather than write new tests. When doing this, make sure that the new predicates\nfail with meaningful diagnostics, so any new problems can be easily debugged\nfrom test logs.\n\n\n## Requirements of new Tests\n\n\nThis is what we expect from new tests; they're an extension of the normal\nHadoop requirements, based on the need to work with remote servers whose\nuse requires the presence of secret credentials, where tests may be slow,\nand where finding out why something failed from nothing but the test output\nis critical.\n\n### Subclasses Existing Shared Base Classes\n\nExtend `AbstractS3ATestBase` or `AbstractSTestS3AHugeFiles` unless justifiable.\nThese set things up for testing against the object stores, provide good threadnames,\nhelp generate isolated paths, and for `AbstractSTestS3AHugeFiles` subclasses,\nonly run if `-Dscale` is set.\n\nKey features of `AbstractS3ATestBase`\n\n* `getFileSystem()` returns the S3A Filesystem bonded to the contract test Filesystem\ndefined in `fs.s3a.contract.test`\n* will automatically skip all tests if that URL is unset.\n* Extends `AbstractFSContractTestBase` and `Assert` for all their methods.\n\nHaving shared base classes may help reduce future maintenance too. Please\nuse them/\n\n### Secure\n\nDon't ever log credentials. The credential tests go out of their way to\nnot provide meaningful logs or assertion messages precisely to avoid this.\n\n### Efficient of Time and Money\n\nThis means efficient in test setup/teardown, and, ideally, making use of\nexisting public datasets to save setup time and tester cost.\n\nStrategies of particular note are:\n\n1. `ITestS3ADirectoryPerformance`: a single test case sets up the directory\ntree then performs different list operations, measuring the time taken.\n1. `AbstractSTestS3AHugeFiles`: marks the test suite as\n`@FixMethodOrder(MethodSorters.NAME_ASCENDING)` then orders the test cases such\nthat each test case expects the previous test to have completed (here: uploaded a file,\nrenamed a file, ...). This provides for independent tests in the reports, yet still\npermits an ordered sequence of operations. Do note the use of `Assume.assume()`\nto detect when the preconditions for a single test case are not met, hence,\nthe tests become skipped, rather than fail with a trace which is really a false alarm.\n\nThe ordered test case mechanism of `AbstractSTestS3AHugeFiles` is probably\nthe most elegant way of chaining test setup/teardown.\n\nRegarding reusing existing data, we tend to use the landsat archive of\nAWS US-East for our testing of input stream operations. This doesn't work\nagainst other regions, or with third party S3 implementations. Thus the\nURL can be overridden for testing elsewhere.\n\n\n### Works With Other S3 Endpoints\n\nDon't assume AWS S3 US-East only, do allow for working with external S3 implementations.\nThose may be behind the latest S3 API features, not support encryption, session\nAPIs, etc.\n\nThey won't have the same CSV test files as some of the input tests rely on.\nLook at `ITestS3AInputStreamPerformance` to see how tests can be written\nto support the declaration of a specific large test file on alternate filesystems.\n\n\n### Works Over Long-haul Links\n\nAs well as making file size and operation counts scalable, this includes\nmaking test timeouts adequate. The Scale tests make this configurable; it's\nhard coded to ten minutes in `AbstractS3ATestBase()`; subclasses can\nchange this by overriding `getTestTimeoutMillis()`.\n\nEqually importantly: support proxies, as some testers need them.\n\n\n### Provides Diagnostics and timing information\n\n1. Give threads useful names.\n1. Create logs, log things. Know that the `S3AFileSystem` and its input\nand output streams *all* provide useful statistics in their {{toString()}}\ncalls; logging them is useful on its own.\n1. you can use `AbstractS3ATestBase.describe(format-stringm, args)` here.; it\nadds some newlines so as to be easier to spot.\n1. Use `ContractTestUtils.NanoTimer` to measure the duration of operations,\nand log the output.\n\n### Fails Meaningfully\n\nThe `ContractTestUtils` class contains a whole set of assertions for making\nstatements about the expected state of a filesystem, e.g.\n`assertPathExists(FS, path)`, `assertPathDoesNotExists(FS, path)`, and others.\nThese do their best to provide meaningful diagnostics on failures (e.g. directory\nlistings, file status, ...), so help make failures easier to understand.\n\nAt the very least, do not use `assertTrue()` or `assertFalse()` without\nincluding error messages.\n\n### Sets up its filesystem and checks for those settings\n\nTests can overrun `createConfiguration()` to add new options to the configuration\nfile for the S3A Filesystem instance used in their tests.\n\nHowever, filesystem caching may mean that a test suite may get a cached\ninstance created with an different configuration. For tests which don't need\nspecific configurations caching is good: it reduces test setup time.\n\nFor those tests which do need unique options (encryption, magic files),\nthings can break, and they will do so in hard-to-replicate ways.\n\nUse `S3ATestUtils.disableFilesystemCaching(conf)` to disable caching when\nmodifying the config. As an example from `AbstractTestS3AEncryption`:\n\n```java\n@Override\nprotected Configuration createConfiguration() {\n Configuration conf = super.createConfiguration();\n S3ATestUtils.disableFilesystemCaching(conf);\n conf.set(Constants.SERVER_SIDE_ENCRYPTION_ALGORITHM,\n getSSEAlgorithm().getMethod());\n return conf;\n}\n```\n\nThen verify in the setup method or test cases that their filesystem actually has\nthe desired feature (`fs.getConf().getProperty(...)`). This not only\ncatches filesystem reuse problems, it catches the situation where the\nfilesystem configuration in `auth-keys.xml` has explicit per-bucket settings\nwhich override the test suite's general option settings.\n\n### Cleans Up Afterwards\n\nKeeps costs down.\n\n1. Do not only cleanup if a test case completes successfully; test suite\nteardown must do it.\n1. That teardown code must check for the filesystem and other fields being\nnull before the cleanup. Why? If test setup fails, the teardown methods still\nget called.\n\n### Works Reliably\n\nWe really appreciate this — you will too.\n\n### Runs in parallel unless this is unworkable.\n\nTests must be designed to run in parallel with other tests, all working\nwith the same shared S3 bucket. This means\n\n* Uses relative and JVM-fork-unique paths provided by the method\n `AbstractFSContractTestBase.path(String filepath)`.\n* Doesn't manipulate the root directory or make assertions about its contents\n(for example: delete its contents and assert that it is now empty).\n* Doesn't have a specific requirement of all active clients of the bucket\n(example: SSE-C tests which require all files, even directory markers,\nto be encrypted with the same key).\n* Doesn't use so much bandwidth that all other tests will be starved of IO and\nstart timing out (e.g. the scale tests).\n\nTests such as these can only be run as sequential tests. When adding one,\nexclude it in the POM file. from the parallel failsafe run and add to the\nsequential one afterwards. The IO heavy ones must also be subclasses of\n`S3AScaleTestBase` and so only run if the system/maven property\n`fs.s3a.scale.test.enabled` is true.\n\n## Individual test cases can be run in an IDE\n\nThis is invaluable for debugging test failures.\n\n### Keeping AWS Costs down\n\nMost of the base S3 tests are designed to use public AWS data\n(the landsat-pds bucket) for read IO, so you don't have to pay for bytes\ndownloaded or long term storage costs. The scale tests do work with more data\nso will cost more as well as generally take more time to execute.\n\nYou are however billed for\n\n1. Data left in S3 after test runs.\n2. DynamoDB capacity reserved by S3Guard tables.\n3. HTTP operations on files (HEAD, LIST, GET).\n4. In-progress multipart uploads from bulk IO or S3A committer tests.\n5. Encryption/decryption using AWS KMS keys.\n\nThe GET/decrypt costs are incurred on each partial read of a file,\nso random IO can cost more than sequential IO; the speedup of queries with\ncolumnar data usually justifies this.\n\nThe DynamoDB costs come from the number of entries stores and the allocated capacity.\n\nHow to keep costs down\n\n* Don't run the scale tests with large datasets; keep `fs.s3a.scale.test.huge.filesize` unset, or a few MB (minimum: 5).\n* Remove all files in the filesystem. The root tests usually do this, but\nit can be manually done:\n\n hadoop fs -rm -r -f -skipTrash s3a://test-bucket/\\*\n* Abort all outstanding uploads:\n\n hadoop s3guard uploads -abort -force s3a://test-bucket/\n* If you don't need it, destroy the S3Guard DDB table.\n\n hadoop s3guard destroy s3a://hwdev-steve-ireland-new/\n\nThe S3Guard tests will automatically create the Dynamo DB table in runs with\n`-Ds3guard -Ddynamodb` set; default capacity of these buckets\ntests is very small; it keeps costs down at the expense of IO performance\nand, for test runs in or near the S3/DDB stores, throttling events.\n\nIf you want to manage capacity, use `s3guard set-capacity` to increase it\n(performance) or decrease it (costs).\nFor remote `hadoop-aws` test runs, the read/write capacities of \"10\" each should suffice;\nincrease it if parallel test run logs warn of throttling.\n\nTip: for agility, use DynamoDB autoscaling, setting the minimum to something very low (e.g 5 units), the maximum to the largest amount you are willing to pay.\nThis will automatically reduce capacity when you are not running tests against\nthe bucket, slowly increase it over multiple test runs, if the load justifies it.\n\n## Tips\n\n### How to keep your credentials really safe\n\nAlthough the `auth-keys.xml` file is marked as ignored in git and subversion,\nit is still in your source tree, and there's always that risk that it may\ncreep out.\n\nYou can avoid this by keeping your keys outside the source tree and\nusing an absolute XInclude reference to it.\n\n```xml\n\n\n \n\n\n```\n\n# Failure Injection\n\n**Warning do not enable any type of failure injection in production. The\nfollowing settings are for testing only.**\n\nOne of the challenges with S3A integration tests is the fact that S3 is an\neventually-consistent storage system. In practice, we rarely see delays in\nvisibility of recently created objects both in listings (`listStatus()`) and\nwhen getting a single file's metadata (`getFileStatus()`). Since this behavior\nis rare and non-deterministic, thorough integration testing is challenging.\n\nTo address this, S3A supports a shim layer on top of the `AmazonS3Client`\nclass which artificially delays certain paths from appearing in listings.\nThis is implemented in the class `InconsistentAmazonS3Client`.\n\n## Simulating List Inconsistencies\n\n### Enabling the InconsistentAmazonS3CClient\n\nThere are two ways of enabling the `InconsistentAmazonS3Client`: at\nconfig-time, or programmatically. For an example of programmatic test usage,\nsee `ITestS3GuardListConsistency`.\n\nTo enable the fault-injecting client via configuration, switch the\nS3A client to use the \"Inconsistent S3 Client Factory\" when connecting to\nS3:\n\n```xml\n\n fs.s3a.s3.client.factory.impl\n org.apache.hadoop.fs.s3a.InconsistentS3ClientFactory\n\n```\n\nThe inconsistent client works by:\n\n1. Choosing which objects will be \"inconsistent\" at the time the object is\ncreated or deleted.\n2. When `listObjects()` is called, any keys that we have marked as\ninconsistent above will not be returned in the results (until the\nconfigured delay has elapsed). Similarly, deleted items may be *added* to\nmissing results to delay the visibility of the delete.\n\nThere are two ways of choosing which keys (filenames) will be affected: By\nsubstring, and by random probability.\n\n```xml\n\n fs.s3a.failinject.inconsistency.key.substring\n DELAY_LISTING_ME\n\n\n\n fs.s3a.failinject.inconsistency.probability\n 1.0\n\n```\n\nBy default, any object which has the substring \"DELAY_LISTING_ME\" in its key\nwill subject to delayed visibility. For example, the path\n`s3a://my-bucket/test/DELAY_LISTING_ME/file.txt` would match this condition.\nTo match all keys use the value \"\\*\" (a single asterisk). This is a special\nvalue: *We don't support arbitrary wildcards.*\n\nThe default probability of delaying an object is 1.0. This means that *all*\nkeys that match the substring will get delayed visibility. Note that we take\nthe logical *and* of the two conditions (substring matches *and* probability\nrandom chance occurs). Here are some example configurations:\n\n```\n| substring | probability | behavior |\n|-----------|-------------|--------------------------------------------|\n| | 0.001 | An empty tag in .xml config will |\n| | | be interpreted as unset and revert to the |\n| | | default value, \"DELAY_LISTING_ME\" |\n| | | |\n| * | 0.001 | 1/1000 chance of *any* key being delayed. |\n| | | |\n| delay | 0.01 | 1/100 chance of any key containing \"delay\" |\n| | | |\n| delay | 1.0 | All keys containing substring \"delay\" .. |\n```\n\nYou can also configure how long you want the delay in visibility to last.\nThe default is 5000 milliseconds (five seconds).\n\n```xml\n\n fs.s3a.failinject.inconsistency.msec\n 5000\n\n```\n\nFuture versions of this client will introduce new failure modes,\nwith simulation of S3 throttling exceptions the next feature under\ndevelopment.\n\n### Limitations of Inconsistency Injection\n\nAlthough `InconsistentAmazonS3Client` can delay the visibility of an object\nor parent directory, it does not prevent the key of that object from\nappearing in all prefix searches. For example, if we create the following\nobject with the default configuration above, in an otherwise empty bucket:\n\n```\ns3a://bucket/a/b/c/DELAY_LISTING_ME\n```\n\nThen the following paths will still be visible as directories (ignoring\npossible real-world inconsistencies):\n\n```\ns3a://bucket/a\ns3a://bucket/a/b\n```\n\nWhereas `getFileStatus()` on the following *will* be subject to delayed\nvisibility (`FileNotFoundException` until delay has elapsed):\n\n```\ns3a://bucket/a/b/c\ns3a://bucket/a/b/c/DELAY_LISTING_ME\n```\n\nIn real-life S3 inconsistency, however, we expect that all the above paths\n(including `a` and `b`) will be subject to delayed visibility.\n\n### Using the `InconsistentAmazonS3CClient` in downstream integration tests\n\nThe inconsistent client is shipped in the `hadoop-aws` JAR, so it can\nbe used in applications which work with S3 to see how they handle\ninconsistent directory listings.\n\n## Testing S3Guard\n\n[S3Guard](./s3guard.html) is an extension to S3A which adds consistent metadata\nlistings to the S3A client. As it is part of S3A, it also needs to be tested.\n\nThe basic strategy for testing S3Guard correctness consists of:\n\n1. MetadataStore Contract tests.\n\n The MetadataStore contract tests are inspired by the Hadoop FileSystem and\n `FileContext` contract tests. Each implementation of the `MetadataStore` interface\n subclasses the `MetadataStoreTestBase` class and customizes it to initialize\n their MetadataStore. This test ensures that the different implementations\n all satisfy the semantics of the MetadataStore API.\n\n2. Running existing S3A unit and integration tests with S3Guard enabled.\n\n You can run the S3A integration tests on top of S3Guard by configuring your\n `MetadataStore` in your\n `hadoop-tools/hadoop-aws/src/test/resources/core-site.xml` or\n `hadoop-tools/hadoop-aws/src/test/resources/auth-keys.xml` files.\n Next run the S3A integration tests as outlined in the *Running the Tests* section\n of the [S3A documentation](./index.html)\n\n3. Running fault-injection tests that test S3Guard's consistency features.\n\n The `ITestS3GuardListConsistency` uses failure injection to ensure\n that list consistency logic is correct even when the underlying storage is\n eventually consistent.\n\n The integration test adds a shim above the Amazon S3 Client layer that injects\n delays in object visibility.\n\n All of these tests will be run if you follow the steps listed in step 2 above.\n\n No charges are incurred for using this store, and its consistency\n guarantees are that of the underlying object store instance. \n\n## Testing S3A with S3Guard Enabled\n\nAll the S3A tests which work with a private repository can be configured to\nrun with S3Guard by using the `s3guard` profile. When set, this will run\nall the tests with local memory for the metadata set to \"non-authoritative\" mode.\n\n```bash\nmvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard\n```\n\nWhen the `s3guard` profile is enabled, following profiles can be specified:\n\n* `dynamo`: use an AWS-hosted DynamoDB table; creating the table if it does\n not exist. You will have to pay the bills for DynamoDB web service.\n* `dynamodblocal`: use an in-memory DynamoDBLocal server instead of real AWS\n DynamoDB web service; launch the server and creating the table.\n You won't be charged bills for using DynamoDB in test. As it runs in-JVM,\n the table isn't shared across other tests running in parallel.\n* `non-auth`: treat the S3Guard metadata as authoritative.\n\n```bash\nmvn -T 1C verify -Dparallel-tests -DtestsThreadCount=6 -Ds3guard -Ddynamo -Dauth\n```\n\nWhen experimenting with options, it is usually best to run a single test suite\nat a time until the operations appear to be working.\n\n```bash\nmvn -T 1C verify -Dtest=skip -Dit.test=ITestS3AMiscOperations -Ds3guard -Ddynamo\n```\n\n### Notes\n\n1. If the `s3guard` profile is not set, then the S3Guard properties are those\nof the test configuration set in `contract-test-options.xml` or `auth-keys.xml`\n\nIf the `s3guard` profile *is* set,\n1. The S3Guard options from maven (the dynamo and authoritative flags)\n overwrite any previously set in the configuration files.\n1. DynamoDB will be configured to create any missing tables.\n\n\n### Scale Testing MetadataStore Directly\n\nThere are some scale tests that exercise Metadata Store implementations\ndirectly. These ensure that S3Guard is are robust to things like DynamoDB\nthrottling, and compare performance for different implementations. These\nare included in the scale tests executed when `-Dscale` is passed to\nthe maven command line.\n\nThe two S3Guard scale tests are `ITestDynamoDBMetadataStoreScale` and\n`ITestLocalMetadataStoreScale`.\n\nTo run these tests, your DynamoDB table needs to be of limited capacity;\nthe values in `ITestDynamoDBMetadataStoreScale` currently require a read capacity\nof 10 or less. a write capacity of 15 or more.\n\nThe following settings allow us to run `ITestDynamoDBMetadataStoreScale` with\nartificially low read and write capacity provisioned, so we can judge the\neffects of being throttled by the DynamoDB service:\n\n```xml\n\n scale.test.operation.count\n 10\n\n\n scale.test.directory.count\n 3\n\n\n fs.s3a.scale.test.enabled\n true\n\n\n fs.s3a.s3guard.ddb.table\n my-scale-test\n\n\n fs.s3a.s3guard.ddb.table.create\n true\n\n\n fs.s3a.s3guard.ddb.table.capacity.read\n 5\n\n\n fs.s3a.s3guard.ddb.table.capacity.write\n 5\n\n```\n\nThese tests verify that the invoked operations can trigger retries in the\nS3Guard code, rather than just in the AWS SDK level, so showing that if\nSDK operations fail, they get retried. They also verify that the filesystem\nstatistics are updated to record that throttling took place.\n\n*Do not panic if these tests fail to detect throttling!*\n\nThese tests are unreliable as they need certain conditions to be met\nto repeatedly fail:\n\n1. You must have a low-enough latency connection to the DynamoDB store that,\nfor the capacity allocated, you can overload it.\n1. The AWS Console can give you a view of what is happening here.\n1. Running a single test on its own is less likely to trigger an overload\nthan trying to run the whole test suite.\n1. And running the test suite more than once, back-to-back, can also help\noverload the cluster.\n1. Stepping through with a debugger will reduce load, so may not trigger\nfailures.\n\nIf the tests fail, it *probably* just means you aren't putting enough load\non the table.\n\nThese tests do not verify that the entire set of DynamoDB calls made\nduring the use of a S3Guarded S3A filesystem are wrapped by retry logic.\n\n*The best way to verify resilience is to run the entire `hadoop-aws` test suite,\nor even a real application, with throttling enabled.\n\n### Testing only: Local Metadata Store\n\nThere is an in-memory Metadata Store for testing.\n\n```xml\n\n fs.s3a.metadatastore.impl\n org.apache.hadoop.fs.s3a.s3guard.LocalMetadataStore\n\n```\n\nThis is not for use in production.\n\n## Testing Assumed Roles\n\nTests for the AWS Assumed Role credential provider require an assumed\nrole to request.\n\nIf this role is not set, the tests which require it will be skipped.\n\nTo run the tests in `ITestAssumeRole`, you need:\n\n1. A role in your AWS account will full read and write access rights to\nthe S3 bucket used in the tests, and ideally DynamoDB, for S3Guard.\nIf your bucket is set up by default to use S3Guard, the role must have access\nto that service.\n\n1. Your IAM User to have the permissions to adopt that role.\n\n1. The role ARN must be set in `fs.s3a.assumed.role.arn`.\n\n```xml\n\n fs.s3a.assumed.role.arn\n arn:aws:iam::9878543210123:role/role-s3-restricted\n\n```\n\nThe tests assume the role with different subsets of permissions and verify\nthat the S3A client (mostly) works when the caller has only write access\nto part of the directory tree.\n\nYou can also run the entire test suite in an assumed role, a more\nthorough test, by switching to the credentials provider.\n\n```xml\n\n fs.s3a.aws.credentials.provider\n org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider\n\n```\n\nThe usual credentials needed to log in to the bucket will be used, but now\nthe credentials used to interact with S3 and DynamoDB will be temporary\nrole credentials, rather than the full credentials.\n\n## Qualifying an AWS SDK Update\n\nUpdating the AWS SDK is something which does need to be done regularly,\nbut is rarely without complications, major or minor.\n\nAssume that the version of the SDK will remain constant for an X.Y release,\nexcluding security fixes, so it's good to have an update before each release\n— as long as that update works doesn't trigger any regressions.\n\n\n1. Don't make this a last minute action.\n1. The upgrade patch should focus purely on the SDK update, so it can be cherry\npicked and reverted easily.\n1. Do not mix in an SDK update with any other piece of work, for the same reason.\n1. Plan for an afternoon's work, including before/after testing, log analysis\nand any manual tests.\n1. Make sure all the integration tests are running (including s3guard, ARN, encryption, scale)\n *before you start the upgrade*.\n1. Create a JIRA for updating the SDK. Don't include the version (yet),\nas it may take a couple of SDK updates before it is ready.\n1. Identify the latest AWS SDK [available for download](https://aws.amazon.com/sdk-for-java/).\n1. Create a private git branch of trunk for JIRA, and in\n `hadoop-project/pom.xml` update the `aws-java-sdk.version` to the new SDK version.\n1. Do a clean build and rerun all the `hadoop-aws` tests, with and without the `-Ds3guard -Ddynamodb` options.\n This includes the `-Pscale` set, with a role defined for the assumed role tests.\n in `fs.s3a.assumed.role.arn` for testing assumed roles,\n and `fs.s3a.server-side-encryption.key` for encryption, for full coverage.\n If you can, scale up the scale tests.\n1. Create the site with `mvn site -DskipTests`; look in `target/site` for the report.\n1. Review *every single `-output.txt` file in `hadoop-tools/hadoop-aws/target/failsafe-reports`,\n paying particular attention to\n `org.apache.hadoop.fs.s3a.scale.ITestS3AInputStreamPerformance-output.txt`,\n as that is where changes in stream close/abort logic will surface.\n1. Run `mvn install` to install the artifacts, then in\n `hadoop-cloud-storage-project/hadoop-cloud-storage` run\n `mvn dependency:tree -Dverbose > target/dependencies.txt`.\n Examine the `target/dependencies.txt` file to verify that no new\n artifacts have unintentionally been declared as dependencies\n of the shaded `aws-java-sdk-bundle` artifact.\n\n### Basic command line regression testing\n\nWe need a run through of the CLI to see if there have been changes there\nwhich cause problems, especially whether new log messages have surfaced,\nor whether some packaging change breaks that CLI\n\nFrom the root of the project, create a command line release `mvn package -Pdist -DskipTests -Dmaven.javadoc.skip=true -DskipShade`;\n\n1. Change into the `hadoop/dist/target/hadoop-x.y.z-SNAPSHOT` dir.\n1. Copy a `core-site.xml` file into `etc/hadoop`.\n1. Set the `HADOOP_OPTIONAL_TOOLS` env var on the command line or `~/.hadoop-env`.\n\n```bash\nexport HADOOP_OPTIONAL_TOOLS=\"hadoop-aws\"\n```\n\nRun some basic s3guard commands as well as file operations.\n\n```bash\nexport BUCKET=s3a://example-bucket-name\n\nbin/hadoop s3guard bucket-info $BUCKET\nbin/hadoop s3guard set-capacity $BUCKET\nbin/hadoop s3guard set-capacity -read 15 -write 15 $BUCKET\nbin/hadoop s3guard uploads $BUCKET\nbin/hadoop s3guard diff $BUCKET/\nbin/hadoop s3guard prune -minutes 10 $BUCKET/\nbin/hadoop s3guard import $BUCKET/\nbin/hadoop fs -ls $BUCKET/\nbin/hadoop fs -ls $BUCKET/file\nbin/hadoop fs -rm -R -f $BUCKET/dir-no-trailing\nbin/hadoop fs -rm -R -f $BUCKET/dir-trailing/\nbin/hadoop fs -rm $BUCKET/\nbin/hadoop fs -touchz $BUCKET/file\n# expect I/O error as root dir is not empty\nbin/hadoop fs -rm -r $BUCKET/\nbin/hadoop fs -rm -r $BUCKET/\\*\n# now success\nbin/hadoop fs -rm -r $BUCKET/\n\nbin/hadoop fs -mkdir $BUCKET/dir-no-trailing\n# fails with S3Guard\nbin/hadoop fs -mkdir $BUCKET/dir-trailing/\nbin/hadoop fs -touchz $BUCKET/file\nbin/hadoop fs -ls $BUCKET/\nbin/hadoop fs -mv $BUCKET/file $BUCKET/file2\n# expect \"No such file or directory\"\nbin/hadoop fs -stat $BUCKET/file\nbin/hadoop fs -stat $BUCKET/file2\nbin/hadoop fs -mv $BUCKET/file2 $BUCKET/dir-no-trailing\nbin/hadoop fs -stat $BUCKET/dir-no-trailing/file2\n# treated the same as the file stat\nbin/hadoop fs -stat $BUCKET/dir-no-trailing/file2/\nbin/hadoop fs -ls $BUCKET/dir-no-trailing/file2/\nbin/hadoop fs -ls $BUCKET/dir-no-trailing\n# expect a \"0\" here:\nbin/hadoop fs -test -d $BUCKET/dir-no-trailing && echo $?\n# expect a \"1\" here:\nbin/hadoop fs -test -d $BUCKET/dir-no-trailing/file2 && echo $?\n# will return NONE unless bucket has checksums enabled\nbin/hadoop fs -checksum $BUCKET/dir-no-trailing/file2\n# expect \"etag\" + a long string\nbin/hadoop fs -D fs.s3a.etag.checksum.enabled=true -checksum $BUCKET/dir-no-trailing/file2\n```\n\n### Other tests\n\n* Whatever applications you have which use S3A: build and run them before the upgrade,\nThen see if complete successfully in roughly the same time once the upgrade is applied.\n* Test any third-party endpoints you have access to.\n* Try different regions (especially a v4 only region), and encryption settings.\n* Any performance tests you have can identify slowdowns, which can be a sign\n of changed behavior in the SDK (especially on stream reads and writes).\n* If you can, try to test in an environment where a proxy is needed to talk\nto AWS services.\n* Try and get other people, especially anyone with their own endpoints,\n apps or different deployment environments, to run their own tests.\n\n### Dealing with Deprecated APIs and New Features\n\nA Jenkins run should tell you if there are new deprecations.\nIf so, you should think about how to deal with them.\n\nMoving to methods and APIs which weren't in the previous SDK release makes it\nharder to roll back if there is a problem; but there may be good reasons\nfor the deprecation.\n\nAt the same time, there may be good reasons for staying with the old code.\n\n* AWS have embraced the builder pattern for new operations; note that objects\nconstructed this way often have their (existing) setter methods disabled; this\nmay break existing code.\n* New versions of S3 calls (list v2, bucket existence checks, bulk operations)\nmay be better than the previous HTTP operations & APIs, but they may not work with\nthird-party endpoints, so can only be adopted if made optional, which then\nadds a new configuration option (with docs, testing, ...). A change like that\nmust be done in its own patch, with its new tests which compare the old\nvs new operations.\n\n### Committing the patch\n\nWhen the patch is committed: update the JIRA to the version number actually\nused; use that title in the commit message.\n\nBe prepared to roll-back, re-iterate or code your way out of a regression.\n\nThere may be some problem which surfaces with wider use, which can get\nfixed in a new AWS release,\n\n\nResource Estimator Service\n==========================\n\nResource Estimator Service can parse the history logs of production jobs, extract their resource consumption skylines in the past runs and predict their resource requirements for the new run.\n\n## Current Status\n\n * Support [Hadoop YARN ResourceManager](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html) logs.\n * In-memory store for parsed history resource skyline and estimation.\n * A [Linear Programming](https://github.com/optimatika/ojAlgo) based estimator.\n * Provides REST interface to parse logs, query history store and estimations.\n\n## Upcoming features\n\n * UI to query history and edit and save estimations.\n * Persisent store implementation for store (either DB based or distributed key-value like HBase).\n * Integrate directly with the [Hadoop YARN Reservation System](http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ReservationSystem.html) to make a recurring reservation based on the estimated resources.\n\nRefer to the [design document](https://issues.apache.org/jira/secure/attachment/12886714/ResourceEstimator-design-v1.pdf) for more details.\n"}