Spark单节点提交多个Application的问题

所有的Spark Application都有一个SparkContext，SparkContext内置一个WebUI用来跟踪任务执行的细节。

WebUI默认的端口号为4040。

如果在同一个节点上已经有一个Spark Application正在运行了，那么在运行的时候会出现端口冲突的问题：

```
14/12/30 15:28:05 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: 地址已在使用
java.net.BindException: 地址已在使用
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
	at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
	at org.eclipse.jetty.server.Server.doStart(Server.java:293)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:195)
	at org.apache.spark.ui.JettyUtils$$anonfun$4.apply(JettyUtils.scala:205)
	at org.apache.spark.ui.JettyUtils$$anonfun$4.apply(JettyUtils.scala:205)
	at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1504)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1495)
	at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:205)
	at org.apache.spark.ui.WebUI.bind(WebUI.scala:102)
	at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:234)
	at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:234)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:234)
	at Transfer$.main(Transfer.scala:14)
	at Transfer.main(Transfer.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
14/12/30 15:28:05 WARN AbstractLifeCycle: FAILED org.eclipse.jetty.server.Server@41627ba1: java.net.BindException: 地址已在使用
java.net.BindException: 地址已在使用
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:444)
	at sun.nio.ch.Net.bind(Net.java:436)
	at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
	at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
	at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
	at org.eclipse.jetty.server.Server.doStart(Server.java:293)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
	at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$$connect$1(JettyUtils.scala:195)
	at org.apache.spark.ui.JettyUtils$$anonfun$4.apply(JettyUtils.scala:205)
	at org.apache.spark.ui.JettyUtils$$anonfun$4.apply(JettyUtils.scala:205)
	at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1504)
	at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
	at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1495)
	at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:205)
	at org.apache.spark.ui.WebUI.bind(WebUI.scala:102)
	at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:234)
	at org.apache.spark.SparkContext$$anonfun$10.apply(SparkContext.scala:234)
	at scala.Option.foreach(Option.scala:236)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:234)
	at Transfer$.main(Transfer.scala:14)
	at Transfer.main(Transfer.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:329)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
14/12/30 15:28:05 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
```

不过好在Spark考虑到了这个问题，如果4040端口已经在用了，他会：`Attempting port 4041`。

如果4041也被占用的话，就一次使用后面的端口，4042...

Spark单节点提交多个Application的问题

请我喝咖啡