Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EC2 region and AMI parameter can block startup #11

Closed
stsievert opened this issue Sep 20, 2015 · 2 comments
Closed

EC2 region and AMI parameter can block startup #11

stsievert opened this issue Sep 20, 2015 · 2 comments

Comments

@stsievert
Copy link
Member

I ran a set of commands that according to the readme.md/wiki should work. However, I ran into an issue that took several hours to resolve (described below). Long story short, the DEFAULT_REGION and DEFAULT_AMI defined in ec2/next_ec2.py have to be aligned with the EC2 defaults. The readme/wiki should indicate these changes, perhaps under a "Troubleshooting" wiki page.

export AWS_ACCESS_KEY_ID=<key>
export AWS_SECRET_ACCESS_KEY=<secret>
export KEY_FILE=/Users/scott/Classes/security/SSkey
export KEY_PAIR=SSkey

cd /Users/scott/Desktop/NEXT/ec2
source activate py27
#python next_ec2.py --key-pair=$KEY_PAIR --identity-file=$KEY_FILE createbucket testing2
export AWS_BUCKET_NAME=<bucket>
python next_ec2.py --key-pair=$KEY_PAIR --identity-file=$KEY_FILE launch testing2

This last command produced the following output:

> python next_ec2.py --key-pair=$KEY_PAIR --identity-file=$KEY_FILE launch testing2
ERROR:boto:400 Bad Request
ERROR:boto:<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidKeyPair.NotFound</Code><Message>The key pair 'SSkey' does not exist</Message></Error></Errors><RequestID>907c6818-8cbb-49fa-a912-2a5557cf7b11</RequestID></Response>
Traceback (most recent call last):
  File "next_ec2.py", line 1210, in <module>
    main()
  File "next_ec2.py", line 1202, in main
    real_main()
  File "next_ec2.py", line 1009, in real_main
    (master_nodes, slave_nodes) = launch_cluster(conn, opts, cluster_name)
  File "next_ec2.py", line 519, in launch_cluster
    user_data=user_data_content)
  File "/Users/scott/anaconda/envs/py27/lib/python2.7/site-packages/boto/ec2/image.py", line 329, in run
    tenancy=tenancy, dry_run=dry_run)
  File "/Users/scott/anaconda/envs/py27/lib/python2.7/site-packages/boto/ec2/connection.py", line 973, in run_instances
    verb='POST')
  File "/Users/scott/anaconda/envs/py27/lib/python2.7/site-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidKeyPair.NotFound</Code><Message>The key pair 'SSkey' does not exist</Message></Error></Errors><RequestID>907c6818-8cbb-49fa-a912-2a5557cf7b11</RequestID></Response

which eventually led to a spark-users mailing list which indicated that Spark only supports regions us-east. Because next_ec2.py uses a Spark startup script, I saw my default EC2 region as shown in this image:

screen shot 2015-09-20 at 5 19 24 pm

(changing this changes the default keys/etc)

In ec2/next_ec2.py I then defined

DEFAULT_REGION = 'us-east-1'
DEFAULT_AMI = 'ami-d05e75b8'  # Ubuntu Server 14.04 LTS

which is different from the defaults and could launch a NEXT EC2 instance.

This issue can also be resolved by changing your default region to Oregon, the default region for the next_ec2.py script. That is, if this issue encountered, you can just change your region like below instead of editing the source:

screen shot 2015-10-01 at 11 59 10 am

@stsievert stsievert added the bug label Sep 24, 2015
stsievert pushed a commit that referenced this issue Sep 25, 2015
address #11, also includes syntax highlighting
@lalitkumarj
Copy link
Member

So we need to decide how we want to deal with this. Since AMI's are tied to regions, we should have some logic that if a region is specified, an AMI must be specified as well.

In addition, let's try to try catch and connections with boto and put slightly more informative error messages. For example "Your key file did not work. Are you sure you are in the us-west region?" or something like that.

@stsievert
Copy link
Member Author

We mention this in the docs (and have the new AMI launch for the GUI). I'm comfortable closing this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants