{"id":3536,"date":"2019-04-17T19:19:47","date_gmt":"2019-04-17T15:19:47","guid":{"rendered":"https:\/\/netpoint-dc.com\/blog\/?p=3536"},"modified":"2021-08-25T05:44:24","modified_gmt":"2021-08-25T01:44:24","slug":"sborka-docker-spark-job-server-dlya-pyspark","status":"publish","type":"post","link":"https:\/\/netpoint-dc.com\/blog\/sborka-docker-spark-job-server-dlya-pyspark\/","title":{"rendered":"\u0421\u0431\u043e\u0440\u043a\u0430, \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0430 \u0438 \u0437\u0430\u043f\u0443\u0441\u043a Spark Job Server \u0434\u043b\u044f PySpark \u0441 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435\u043c Docker"},"content":{"rendered":"\n<p>\u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043c\u044b \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c, \u043a\u0430\u043a \u0441\u043e\u0431\u0440\u0430\u0442\u044c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c Spark Job Server \u0432 \u0432\u0438\u0434\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f \u0434\u043b\u044f Docker. Spark Job Server &#8212; \u043f\u043e\u043b\u0435\u0437\u043d\u043e\u0435 \u0434\u043e\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u0435 Spark, \u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u0437\u0430\u0433\u0440\u0443\u0436\u0430\u0442\u044c \u0438 \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0442\u044c \u0437\u0430\u0434\u0430\u043d\u0438\u044f Spark \u0447\u0435\u0440\u0435\u0437 Rest API. <\/p>\n\n\n\n<p>Spark Job Server \u0432\u0445\u043e\u0434\u0438\u0442 \u0432 \u0441\u043e\u0441\u0442\u0430\u0432 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u044b\u0445 \u0434\u0438\u0441\u0442\u0440\u0438\u0431\u0443\u0442\u0438\u0432\u043e\u0432 \u0434\u043b\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445, \u0442\u0430\u043a\u0438\u0445 \u043a\u0430\u043a Cloudera CDH5 \u0438\u043b\u0438 Hortonworks, \u043e\u0434\u043d\u0430\u043a\u043e, \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u043f\u043e\u043b\u043d\u043e\u0446\u0435\u043d\u043d\u043e\u0433\u043e \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 \u0438\u0437\u0431\u044b\u0442\u043e\u0447\u043d\u0430 \u0434\u043b\u044f \u0446\u0435\u043b\u0435\u0439 \u043e\u0431\u0443\u0447\u0435\u043d\u0438\u044f, \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438 \u0438\u043b\u0438 \u043c\u0430\u043b\u044b\u0445 \u0432\u044b\u0447\u0438\u0441\u043b\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0445 \u0437\u0430\u0434\u0430\u0447.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"533\" src=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/apache_spark_logo.svg_-1024x533.png\" alt=\"\" class=\"wp-image-3537\" srcset=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/apache_spark_logo.svg_-1024x533.png 1024w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/apache_spark_logo.svg_-300x156.png 300w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/apache_spark_logo.svg_-768x400.png 768w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/apache_spark_logo.svg_-624x325.png 624w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/apache_spark_logo.svg_.png 1200w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>\u0412 \u0434\u0430\u043d\u043d\u043e\u043c \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432\u0435 \u0432\u044b \u0443\u0437\u043d\u0430\u0435\u0442\u0435 \u043a\u0430\u043a \u0440\u0430\u0437\u0432\u0435\u0440\u043d\u0443\u0442\u044c \u0433\u043e\u0442\u043e\u0432\u0443\u044e \u0441\u0440\u0435\u0434\u0443 \u0434\u043b\u044f \u0440\u0430\u0437\u0440\u0430\u0431\u043e\u0442\u043a\u0438, \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u0438 \u043f\u0440\u043e\u0434\u0443\u043a\u0442\u043e\u0432\u043e\u0433\u043e \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f Apache Spark \u0431\u0435\u0437 \u0432\u043d\u0435\u0448\u043d\u0438\u0445 \u0437\u0430\u0432\u0438\u0441\u0438\u043c\u043e\u0441\u0442\u0435\u0439 \u0441 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435\u043c Spark Job Server.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">\u0421\u0431\u043e\u0440\u043a\u0430 Spark Job Server \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e SBT \u0438 Docker<\/h2>\n\n\n\n<p>\u041c\u044b \u0431\u0443\u0434\u0435\u043c \u0441\u043e\u0431\u0438\u0440\u0430\u0442\u044c \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440 Spark Job Server (SJS), \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u044f\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u0441\u0430\u043c\u043e\u0434\u043e\u0441\u0442\u0430\u0442\u043e\u0447\u043d\u044b\u043c \u0438 \u043c\u043e\u0436\u0435\u0442 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u043e \u0432\u0441\u0442\u0440\u043e\u0435\u043d\u043d\u044b\u043c \u0441\u0435\u0440\u0432\u0435\u0440\u043e\u043c Spark \u0431\u0435\u0437 \u0432\u043d\u0435\u0448\u043d\u0438\u0445 \u0437\u0430\u0432\u0438\u0441\u0438\u043c\u043e\u0441\u0442\u0435\u0439, \u043a\u0440\u043e\u043c\u0435 \u0442\u043e\u0433\u043e, \u0432\u044b \u043c\u043e\u0436\u0435\u0442\u0435 \u0438\u0437\u043c\u0435\u043d\u0438\u0442\u044c \u0444\u0430\u0439\u043b \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 \u0438 \u043f\u043e\u0434\u043a\u043b\u044e\u0447\u0438\u0442\u044c Spark Job Server \u043a \u0441\u0443\u0449\u0435\u0441\u0442\u0432\u0443\u044e\u0449\u0435\u043c\u0443 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0443 Spark.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 Docker<\/h3>\n\n\n\n<p>\u0414\u043b\u044f \u0443\u0441\u043f\u0435\u0448\u043d\u043e\u0433\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u0434\u0430\u043b\u044c\u043d\u0435\u0439\u0448\u0438\u0445 \u0434\u0435\u0439\u0441\u0442\u0432\u0438\u0439 \u0432\u0430\u043c \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u044c Docker \u0434\u043b\u044f \u0432\u0430\u0448\u0435\u0439 \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u043e\u0439 \u0441\u0438\u0441\u0442\u0435\u043c\u044b. \u0415\u0441\u043b\u0438 \u0443 \u0432\u0430\u0441 \u0443\u0436\u0435 \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d \u0438 \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d Docker, \u043f\u0440\u043e\u0441\u0442\u043e \u043f\u0440\u043e\u043f\u0443\u0441\u0442\u0438\u0442\u0435 \u044d\u0442\u043e\u0442 \u0448\u0430\u0433. \u0415\u0441\u043b\u0438 Docker \u043d\u0435 \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043b\u0435\u043d, \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0435 \u0435\u0433\u043e \u043f\u043e \u043e\u0434\u043d\u043e\u043c\u0443 \u0438\u0437 \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432: <a rel=\"noreferrer noopener\" href=\"https:\/\/netpoint-dc.com\/blog\/ustanovka-docker-v-ubuntu-18-04\/\" target=\"_blank\">Ubuntu 16.04 \u0438 18.04<\/a>, <a rel=\"noreferrer noopener\" href=\"https:\/\/netpoint-dc.com\/blog\/ustanovka-docker-ce-v-debian-9-stretch-or-debian-10-buster\/\" target=\"_blank\">Debian 9 \u0438 10<\/a>, <a rel=\"noreferrer noopener\" href=\"https:\/\/netpoint-dc.com\/blog\/ustanovka-docker-ce-v-centos-7\/\" target=\"_blank\">Centos 7<\/a> \u0438 \u043f\u0440\u0438\u0441\u0442\u0443\u043f\u0430\u0439\u0442\u0435 \u043a \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0435\u043c\u0443 \u0448\u0430\u0433\u0443.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 Scala Build Tool<\/h3>\n\n\n\n<p>\u0414\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u043f\u043e\u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f \u0443\u0442\u0438\u043b\u0438\u0442\u0430 SBT, \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043a\u043e\u0442\u043e\u0440\u043e\u0439 \u043c\u044b \u0431\u0443\u0434\u0435\u043c \u0441\u043e\u0431\u0438\u0440\u0430\u0442\u044c \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440 Docker. <\/p>\n\n\n\n<p>\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 SBT \u0434\u043b\u044f Ubuntu \/ Debian \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0430 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u043c\u0438 \u043a\u043e\u043c\u0430\u043d\u0434\u0430\u043c\u0438:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">echo \"deb https:\/\/dl.bintray.com\/sbt\/debian \/\" | sudo tee -a \/etc\/apt\/sources.list.d\/sbt.list\nsudo apt-key adv --keyserver hkp:\/\/keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823\nsudo apt-get update\nsudo apt-get install sbt<\/pre>\n\n\n\n<p>\u0423\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u0434\u043b\u044f CentOS \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u0442\u0441\u044f, \u043a\u0430\u043a \u043e\u043f\u0438\u0441\u0430\u043d\u043e \u043d\u0438\u0436\u0435:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">curl https:\/\/bintray.com\/sbt\/rpm\/rpm > bintray-sbt-rpm.repo\nsudo mv bintray-sbt-rpm.repo \/etc\/yum.repos.d\/\nsudo yum install sbt<\/pre>\n\n\n\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u0441\u043a\u0430\u0447\u0430\u0435\u043c \u0438\u0441\u0445\u043e\u0434\u043d\u044b\u0435 \u043a\u043e\u0434\u044b SJS \u0441 GitHub:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">git clone https:\/\/github.com\/spark-jobserver\/spark-jobserver.git\ncd spark-jobserver\/<\/pre>\n\n\n\n<p>\u0421\u0431\u043e\u0440\u043a\u0430 \u0431\u0430\u0437\u043e\u0432\u043e\u0433\u043e \u043e\u0431\u0440\u0430\u0437\u0430 Docker \u0434\u043b\u044f SJS \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0435\u0442\u0441\u044f \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u044b <strong>sbt docker<\/strong>. \u0415\u0441\u043b\u0438 \u0434\u043e\u0441\u0442\u0443\u043f \u043a Docker \u0434\u043e\u0441\u0442\u0443\u043f\u0435\u043d \u0442\u043e\u043b\u044c\u043a\u043e \u0447\u0435\u0440\u0435\u0437 sudo, \u0437\u0430\u043f\u0443\u0441\u043a\u0430\u0439\u0442\u0435 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0443\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u0443, \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u044f sudo:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">sbt docker job-server-tests\/package<\/pre>\n\n\n\n<p>\u0421\u0431\u043e\u0440\u043a\u0430 \u0437\u0430\u0439\u043c\u0435\u0442 \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043f\u0440\u043e\u0434\u043e\u043b\u0436\u0438\u0442\u0435\u043b\u044c\u043d\u043e\u0435 \u0432\u0440\u0435\u043c\u044f, \u0438\u0442\u043e\u0433\u043e\u043c \u0447\u0435\u0433\u043e \u0441\u0442\u0430\u043d\u0435\u0442 \u0433\u0435\u043d\u0435\u0440\u0430\u0446\u0438\u044f \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">$ docker images | grep spark-jobserver\nvelvia\/spark-jobserver      0.9.1-SNAPSHOT.mesos-1.0.0.spark-2.3.2.scala-2.11.jdk-8-jdk   3ee46d478a7a        5 days ago           1.41GB<\/pre>\n\n\n\n<p>\u041e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435 \u043d\u0430 \u0432\u0435\u0440\u0441\u0438\u044e Spark \u0432 \u0438\u043c\u0435\u043d\u0438 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430, \u0432 \u043d\u0430\u0448\u0435\u043c \u0441\u043b\u0443\u0447\u0430\u0435 &#8212; 2.3.2. \u042d\u0442\u043e \u043d\u0430\u043c \u043f\u043e\u043d\u0430\u0434\u043e\u0431\u0438\u0442\u0441\u044f \u043f\u0440\u0438 \u0434\u043e\u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0435 \u0432 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440 \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u044b\u0445 \u0437\u0430\u0432\u0438\u0441\u0438\u043c\u043e\u0441\u0442\u0435\u0439 \u0434\u043b\u044f PySpark.<\/p>\n\n\n\n<p>\u0415\u0441\u043b\u0438 \u0432\u044b \u043f\u043b\u0430\u043d\u0438\u0440\u0443\u0435\u0442\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440 \u0434\u043b\u044f \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u0437\u0430\u0434\u0430\u043d\u0438\u0439 Spark, \u0440\u0435\u0430\u043b\u0438\u0437\u043e\u0432\u0430\u043d\u043d\u044b\u0445 \u043d\u0430 Scala \u0438\u043b\u0438 Java, \u043c\u043e\u0436\u043d\u043e \u043f\u0435\u0440\u0435\u0439\u0442\u0438 \u0441\u0440\u0430\u0437\u0443 \u043a \u0440\u0430\u0437\u0434\u0435\u043b\u0443 &#171;\u0417\u0430\u043f\u0443\u0441\u043a Spark Job Server&#187;. \u0414\u0430\u043b\u0435\u0435 \u043c\u044b \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c \u043a\u0430\u043a \u043d\u0430 \u043e\u0441\u043d\u043e\u0432\u0430\u043d\u0438\u0438 \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u043d\u043e\u0433\u043e \u043e\u0431\u0440\u0430\u0437\u0430 \u0441\u043e\u0431\u0440\u0430\u0442\u044c \u043f\u043e\u0434\u0445\u043e\u0434\u044f\u0449\u0438\u0439 \u0434\u043b\u044f \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f \u0441\u043e\u0432\u043c\u0435\u0441\u0442\u043d\u043e \u0441 PySpark.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">\u0421\u0431\u043e\u0440\u043a\u0430 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430 \u0434\u043b\u044f PySpark<\/h3>\n\n\n\n<p>\u041c\u044b \u0431\u0443\u0434\u0435\u043c \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u043f\u043e\u043b\u0443\u0447\u0435\u043d\u043d\u044b\u0439 \u043e\u0431\u0440\u0430\u0437 \u043a\u0430\u043a \u043e\u0441\u043d\u043e\u0432\u0443 \u0434\u043b\u044f \u0433\u0435\u043d\u0435\u0440\u0430\u0446\u0438\u0438 \u0446\u0435\u043b\u0435\u0432\u043e\u0433\u043e \u043e\u0431\u0440\u0430\u0437\u0430, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u0443\u0434\u0435\u0442 \u043f\u043e\u0434\u0434\u0435\u0440\u0436\u0438\u0432\u0430\u0442\u044c PySpark.<\/p>\n\n\n\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u043a\u0430\u0442\u0430\u043b\u043e\u0433 \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u043d\u043e\u0432\u043e\u0433\u043e \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430, \u043d\u0430\u0437\u043e\u0432\u0435\u043c \u0435\u0433\u043e spark-jobserver-pyspark:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">cd .. &amp;&amp; mkdir spark-jobserver-pyspark<\/pre>\n\n\n\n<p>\u041f\u043e\u043c\u0435\u0441\u0442\u0438\u043c \u0432 \u044d\u0442\u043e\u0442 \u043a\u0430\u0442\u0430\u043b\u043e\u0433 \u043d\u043e\u0432\u044b\u0439 Dockerfile, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0431\u0443\u0434\u0435\u0442 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u0434\u043b\u044f \u043f\u043e\u0441\u0442\u0440\u043e\u0435\u043d\u0438\u044f \u0446\u0435\u043b\u0435\u0432\u043e\u0433\u043e \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">cat > Dockerfile\nfrom velvia\/spark-jobserver:0.9.1-SNAPSHOT.mesos-1.0.0.spark-2.3.2.scala-2.11.jdk-8-jdk\nRUN apt-get install -y python-pip &amp;&amp; pip --no-cache-dir install pyhocon pyspark==2.3.2 py4j\n\nCtrl+D<\/pre>\n\n\n\n<p>\u041e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435 \u043d\u0430 \u0442\u043e \u043a\u0430\u043a\u0430\u044f \u0432\u0435\u0440\u0441\u0438\u044f \u0443\u043a\u0430\u0437\u0430\u043d\u0430 \u0434\u043b\u044f PySpark, \u0432\u044b \u0434\u043e\u043b\u0436\u043d\u044b \u0443\u043a\u0430\u0437\u0430\u0442\u044c \u0432\u0435\u0440\u0441\u0438\u044e, \u043a\u043e\u0442\u043e\u0440\u0430\u044f \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0443\u0435\u0442 \u0432\u0435\u0440\u0441\u0438\u0438 Spark \u0432 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0435 SJS (\u043f\u0435\u0440\u0432\u0430\u044f \u0441\u0442\u0440\u043e\u043a\u0430 \u043b\u0438\u0441\u0442\u0438\u043d\u0433\u0430).<\/p>\n\n\n\n<p>\u0415\u0441\u043b\u0438 \u0432\u0430\u043c \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u0434\u043e\u0431\u0430\u0432\u0438\u0442\u044c \u0432 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440 \u0434\u043e\u043f\u043e\u043b\u043d\u0438\u0442\u0435\u043b\u044c\u043d\u044b\u0435 \u0444\u0430\u0439\u043b\u044b, \u0431\u0438\u0431\u043b\u0438\u043e\u0442\u0435\u043a\u0438 \u0438 \u0442.\u043f., \u0432\u044b \u043c\u043e\u0436\u0435\u0442\u0435 \u0434\u043e\u043f\u0438\u0441\u0430\u0442\u044c \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u044b\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 \u0432 Dockerfile.<\/p>\n\n\n\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u0432\u044b \u043c\u043e\u0436\u0435\u0442\u0435 \u0441\u043e\u0431\u0440\u0430\u0442\u044c \u0444\u0438\u043d\u0430\u043b\u044c\u043d\u044b\u0439 \u043e\u0431\u0440\u0430\u0437:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">docker build -t pyspark-job-server:2.3.2 .<\/pre>\n\n\n\n<p>\u0418\u0442\u043e\u0433\u043e\u043c \u0441\u0431\u043e\u0440\u043a\u0438 \u0434\u043e\u043b\u0436\u0435\u043d \u0441\u0442\u0430\u0442\u044c \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u0434\u0430\u043b\u0435\u0435 \u0431\u0443\u0434\u0435\u0442 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u0422\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0435 \u0440\u0430\u0431\u043e\u0442\u043e\u0441\u043f\u043e\u0441\u043e\u0431\u043d\u043e\u0441\u0442\u0438 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430<\/h2>\n\n\n\n<p>\u0414\u043b\u044f \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430 \u043c\u044b \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u043c \u0435\u0433\u043e \u0438 \u0432\u044b\u043f\u043e\u043b\u043d\u0438\u043c \u0442\u0435\u0441\u0442\u043e\u0432\u043e\u0435 \u0437\u0430\u0434\u0430\u043d\u0438\u0435 Spark:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">docker run --name sjs -d --rm -p 8090:8090 spark-job-server<\/pre>\n\n\n\n<p>\u0422\u0435\u043f\u0435\u0440\u044c, \u0435\u0441\u043b\u0438 \u0432\u044b \u043f\u0435\u0440\u0435\u0439\u0434\u0435\u0442\u0435 \u0432 \u0431\u0440\u0430\u0443\u0437\u0435\u0440\u0435 \u043f\u043e \u0430\u0434\u0440\u0435\u0441\u0443 http:\/\/localhost:8090\/, \u0442\u043e \u0441\u043c\u043e\u0436\u0435\u0442\u0435 \u0432\u0438\u0434\u0435\u0442\u044c \u043f\u0430\u043d\u0435\u043b\u044c \u043c\u043e\u043d\u0438\u0442\u043e\u0440\u0438\u043d\u0433\u0430 SJS:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-09-42-35.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-09-42-35-1024x576.png\" alt=\"\" class=\"wp-image-3565\" srcset=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-09-42-35-1024x576.png 1024w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-09-42-35-300x169.png 300w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-09-42-35-768x432.png 768w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-09-42-35-624x351.png 624w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-09-42-35.png 1920w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p>\u041f\u0430\u043d\u0435\u043b\u044c \u0434\u043e\u0432\u043e\u043b\u044c\u043d\u043e \u043f\u0440\u043e\u0441\u0442\u0430\u044f \u0438 \u043f\u0440\u0435\u0434\u043d\u0430\u0437\u043d\u0430\u0447\u0435\u043d\u0430 \u0434\u043b\u044f \u043f\u0440\u043e\u0441\u043c\u043e\u0442\u0440\u0430 \u0437\u0430\u0434\u0430\u0447, \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442\u043e\u0432, \u0437\u0430\u0433\u0440\u0443\u0436\u0435\u043d\u043d\u044b\u0445 \u0444\u0430\u0439\u043b\u043e\u0432. \u0422\u0430\u043a \u0436\u0435 \u043f\u0430\u043d\u0435\u043b\u044c \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u043f\u0440\u0438\u043d\u0443\u0434\u0438\u0442\u0435\u043b\u044c\u043d\u043e \u0437\u0430\u0432\u0435\u0440\u0448\u0438\u0442\u044c \u0437\u0430\u0434\u0430\u0447\u0443 \u0438\u043b\u0438 \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442 (kill). \u0412\u0441\u0435 \u0432\u0437\u0430\u0438\u043c\u043e\u0434\u0435\u0439\u0441\u0442\u0432\u0438\u0435 \u0441 SJS \u043e\u0441\u0443\u0449\u0435\u0441\u0442\u0432\u043b\u044f\u0435\u0442\u0441\u044f \u043f\u043e\u0441\u0440\u0435\u0434\u0441\u0442\u0432\u043e\u043c REST API. <\/p>\n\n\n\n<p>\u0414\u0432\u0430 \u0432\u0430\u0436\u043d\u044b\u0445 \u043c\u043e\u043c\u0435\u043d\u0442\u0430:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>\u0415\u0441\u043b\u0438 \u0441\u0435\u0440\u0432\u0435\u0440, \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u0432\u044b \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0435 SJS \u0434\u043e\u0441\u0442\u0443\u043f\u0435\u043d \u0438\u0437\u0432\u043d\u0435, \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u043f\u0440\u0438 \u0437\u0430\u043f\u0443\u0441\u043a\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c <strong>-p 127.0.0.1:8090:8090<\/strong> \u0432\u043c\u0435\u0441\u0442\u043e <strong>-p 8090:8090<\/strong>, \u0430 \u0434\u043e\u0441\u0442\u0443\u043f \u043a \u0441\u0435\u0440\u0432\u0438\u0441\u0443 \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0442\u044c \u0447\u0435\u0440\u0435\u0437 Nginx \u0441 \u0430\u0443\u0442\u0435\u043d\u0442\u0438\u0444\u0438\u043a\u0430\u0446\u0438\u0435\u0439, \u043d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u0447\u0435\u0440\u0435\u0437 \u0431\u0430\u0437\u043e\u0432\u0443\u044e \u0430\u0443\u0442\u0435\u043d\u0442\u0438\u0444\u0438\u043a\u0430\u0446\u0438\u044e HTTP. \u0422\u0430\u043a \u0436\u0435 \u043b\u0443\u0447\u0448\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0437\u0430\u0449\u0438\u0449\u0435\u043d\u043d\u044b\u0439 \u043f\u0440\u043e\u0442\u043e\u043a\u043e\u043b HTTPS. \u0412 \u043d\u0430\u0448\u0438\u0445 \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432\u0430\u0445 \u0432\u044b \u0441\u043c\u043e\u0436\u0435\u0442\u0435 \u043d\u0430\u0439\u0442\u0438 \u043a\u0430\u043a \u044d\u0442\u043e \u0441\u0434\u0435\u043b\u0430\u0442\u044c \u0434\u043b\u044f <a rel=\"noreferrer noopener\" aria-label=\"Ubuntu (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/netpoint-dc.com\/blog\/nastroika-proxy-nginx-s-sertifikatom-let-s-encrypt-v-ubuntu-18-04-bionic-ili-debian-9-stretch\/\" target=\"_blank\">Ubuntu \u0438 Debian<\/a>, <a rel=\"noreferrer noopener\" aria-label=\"CentOS (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/netpoint-dc.com\/blog\/nastroika-balansiruyushego-proxy-nginx-s-sertificatom-letss-encrypt-v-centos-7\/\" target=\"_blank\">CentOS<\/a> \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0431\u0435\u0441\u043f\u043b\u0430\u0442\u043d\u043e\u0433\u043e \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u0430 Let&#8217;s Encrypt.<\/li><li>SJS \u043f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e \u0445\u0440\u0430\u043d\u0438\u0442 \u0441\u0432\u043e\u0438 \u0434\u0430\u043d\u043d\u044b\u0435 \u0432\u043e \u0432\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0435\u043c\u043e\u0439 \u0421\u0423\u0411\u0414 <a rel=\"noreferrer noopener\" aria-label=\"H2 (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/www.h2database.com\/html\/main.html\" target=\"_blank\">H2<\/a>, \u043a\u043e\u0442\u043e\u0440\u0430\u044f  \u043d\u0435 \u043f\u043e\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442 \u0441\u0442\u0430\u0431\u0438\u043b\u044c\u043d\u0443\u044e \u0440\u0430\u0431\u043e\u0442\u0443 \u0438 \u043c\u044b \u043d\u0435 \u0440\u0435\u043a\u043e\u043c\u0435\u043d\u0434\u0443\u0435\u043c \u0435\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0435 \u0437\u0430 \u043f\u0440\u0435\u0434\u0435\u043b\u0430\u043c\u0438 \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f, \u0434\u0430\u043b\u0435\u0435 \u043c\u044b \u0440\u0430\u0441\u0441\u043a\u0430\u0436\u0435\u043c \u043a\u0430\u043a \u0437\u0430\u043c\u0435\u043d\u0438\u0442\u044c H2 \u043d\u0430 MariaDB.<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">\u041f\u043e\u0434\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435 \u0421\u0423\u0411\u0414 MariaDB<\/h2>\n\n\n\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u044c \u0438 \u043d\u0430\u0441\u0442\u0440\u043e\u0438\u0442\u044c MariaDB \u0432 \u0432\u0438\u0434\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f Docker \u043f\u043e <a rel=\"noreferrer noopener\" aria-label=\"\u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/netpoint-dc.com\/blog\/ustanovka-i-nastroika-mysql-mariadb-v-vide-docker-dlya-produktovogo-ispolzovaniya\/\" target=\"_blank\">\u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438<\/a> \u0432 \u043d\u0430\u0448\u0435\u0439 \u0441\u0442\u0430\u0442\u044c\u0435. \u0415\u0441\u043b\u0438 \u043b\u0435\u043d\u044c \u0447\u0438\u0442\u0430\u0442\u044c \u0432\u0441\u0435 \u043f\u043e\u0434\u0440\u044f\u0434, \u043c\u043e\u0436\u043d\u043e \u0432\u044b\u043f\u043e\u043b\u043d\u0438\u0442\u044c \u043f\u0440\u043e\u0441\u0442\u043e \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0443\u044e \u043a\u043e\u043c\u0430\u043d\u0434\u0443:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">docker run -d --restart=always --name mariadb_1 \\\n    -v \/opt\/mariadb\/data:\/var\/lib\/mysql \\\n    -v \/opt\/mariadb\/etc:\/etc\/mysql\/conf.d \\\n    -v \/opt\/mariadb\/logs:\/var\/lib\/mysql\/logs \\\n    -e MYSQL_ROOT_PASSWORD=secret \\\n    -p 127.0.0.1:3306:3306 mariadb:10.3<\/pre>\n\n\n\n<p>\u0414\u043e\u0436\u0434\u0438\u0442\u0435\u0441\u044c \u043e\u043a\u043e\u043d\u0447\u0430\u043d\u0438\u044f \u0438\u043d\u0438\u0446\u0438\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u0438 MariaDB (5-10 \u0441\u0435\u043a\u0443\u043d\u0434). \u0422\u0435\u043f\u0435\u0440\u044c \u0441\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0431\u0430\u0437\u0443 \u0434\u0430\u043d\u043d\u044b\u0445 \u0434\u043b\u044f SJS:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">docker run -it --rm --link mariadb_1:mysql mariadb:10.3 mysql -hmysql -uroot -psecret\n\nmysql> CREATE USER 'jobserver'@'%' IDENTIFIED BY 'secret';\nmysql> CREATE DATABASE spark_jobserver;\nmysql> GRANT ALL ON spark_jobserver.* TO 'jobserver'@'%';\nmysql> FLUSH PRIVILEGES;\n\nCTRL^D<\/pre>\n\n\n\n<p>\u041f\u0440\u043e\u0432\u0435\u0440\u044c\u0442\u0435 \u0441\u043e\u0435\u0434\u0438\u043d\u0435\u043d\u0438\u0435 \u0441 \u0441\u043e\u0437\u0434\u0430\u043d\u043d\u043e\u0439 \u0411\u0414 \u0438\u0437 \u043f\u043e\u0434 \u043d\u043e\u0432\u043e\u0433\u043e \u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u0435\u043b\u044f:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">docker run -it --rm --link mariadb_1:mysql mariadb:10.3 mysql -hmysql -ujobserver -psecret spark_jobserver -e 'SELECT VERSION();'<\/pre>\n\n\n\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u0441\u043a\u043e\u043f\u0438\u0440\u0443\u0435\u043c \u0444\u0430\u0439\u043b \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u0438 \u0434\u043b\u044f SJS \u0438\u0437 \u043a\u043e\u043d\u0442\u0435\u0439\u043d\u0435\u0440\u0430 \u0432 \u0444\u0430\u0439\u043b\u043e\u0432\u0443\u044e \u0441\u0438\u0441\u0442\u0435\u043c\u0443 \u0441\u0435\u0440\u0432\u0435\u0440\u0430:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">docker cp sjs:app\/docker.conf .<\/pre>\n\n\n\n<p>\u041e\u0441\u0442\u0430\u043d\u043e\u0432\u0438\u0442\u0435 \u0441\u0435\u0440\u0432\u0435\u0440 SJS, \u043f\u043e\u043a\u0430 \u0447\u0442\u043e \u043e\u043d \u043d\u0435 \u0442\u0440\u0435\u0431\u0443\u0435\u0442\u0441\u044f, \u0432 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u0439 \u0440\u0430\u0437 \u043c\u044b \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u043c \u0435\u0433\u043e \u0443\u0436\u0435 \u0441 \u0431\u044d\u043a\u0435\u043d\u0434\u043e\u043c MySQL:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">docker stop sjs<\/pre>\n\n\n\n<p>\u041c\u043e\u0434\u0438\u0444\u0438\u0446\u0438\u0440\u0443\u0435\u043c \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0439 \u0444\u0430\u0439\u043b <strong>docker.conf<\/strong> \u0434\u043b\u044f <a rel=\"noreferrer noopener\" aria-label=\"\u0440\u0430\u0431\u043e\u0442\u044b (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/github.com\/spark-jobserver\/spark-jobserver#configuring-spark-jobserver-mysql-database-backend\" target=\"_blank\">\u0440\u0430\u0431\u043e\u0442\u044b<\/a> c MySQL. \u0412 \u043d\u043e\u0432\u044b\u0445 \u0432\u0435\u0440\u0441\u0438\u044f\u0445 SJS \u0432 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u043e\u043c \u0444\u0430\u0439\u043b\u0435 \u043c\u043e\u0433\u0443\u0442 \u0431\u044b\u0442\u044c \u043d\u043e\u0432\u044b\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440\u044b, \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u0443\u0431\u0435\u0434\u0438\u0442\u0435\u0441\u044c, \u0447\u0442\u043e \u043e\u043d\u0438 \u0431\u0443\u0434\u0443\u0442 \u043f\u0435\u0440\u0435\u043d\u0435\u0441\u0435\u043d\u044b \u0432 \u0446\u0435\u043b\u0435\u0432\u043e\u0439 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0439 \u0444\u0430\u0439\u043b. \u041d\u0430 \u043c\u043e\u043c\u0435\u043d\u0442 \u043f\u0443\u0431\u043b\u0438\u043a\u0430\u0446\u0438\u0438 \u043a\u043e\u043d\u0444\u0438\u0433\u0443\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0439 \u0444\u0430\u0439\u043b \u0441\u043e\u0434\u0435\u0440\u0436\u0438\u0442 \u0441\u043b\u0435\u0434\u0443\u044e\u0449\u0438\u0435 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440\u044b:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">spark {\n  master = \"local[*]\"\n  master = ${?SPARK_MASTER}\n  submit.deployMode = \"client\"\n  job-number-cpus = 4\n  jobserver {\n    port = 8090\n    jobdao = spark.jobserver.io.JobSqlDAO\n    context-creation-timeout = 60000\n    context-per-jvm = true\n    # ----------------------\n    # \u043d\u0430\u0441\u0442\u0440\u043e\u0439\u043a\u0438 MYSQL\n    sqldao {\n      slick-driver = slick.driver.MySQLDriver\n      jdbc-driver = com.mysql.jdbc.Driver\n      jdbc {\n        url = \"jdbc:mysql:\/\/mysql\/spark_jobserver\"\n        user = \"jobserver\"\n        password = \"secret\"\n      }\n      dbcp {\n        maxactive = 20\n        maxidle = 10\n        initialsize = 10\n      }\n    }\n  }\n  context-settings {\n    num-cpu-cores = 4          \n    memory-per-node = 8192m\n    passthrough {\n    }\n  }\n  home = \"\/spark\"\n}\ndeploy {\n  manager-start-cmd = \"app\/manager_start.sh\"\n}\n# ---------------------\n# \u0441\u043e\u0437\u0434\u0430\u043d\u0438\u0435 \u0442\u0430\u0431\u043b\u0438\u0446 MySQL\nflyway.locations=\"db\/mysql\/migration\"\n<\/pre>\n\n\n\n<p>\u041e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435 \u043d\u0430 \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440\u044b <strong>spark.job-number-cpus<\/strong>, <strong>spark.context-settings<\/strong> &#8212; \u043e\u043d\u0438 \u0434\u043e\u043b\u0436\u043d\u044b \u0431\u044b\u0442\u044c \u043d\u0430\u0441\u0442\u0440\u043e\u0435\u043d\u044b \u0434\u043b\u044f \u0441\u043e\u043e\u0442\u0432\u0435\u0442\u0441\u0442\u0432\u0438\u044f \u0432\u0430\u0448\u0435\u043c\u0443 \u0443\u0437\u043b\u0443, \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u0432\u044b \u0437\u0430\u043f\u0443\u0441\u043a\u0430\u0435\u0442\u0435 SJS. <\/p>\n\n\n\n<p>\u0422\u0430\u043a \u0436\u0435, \u043e\u0431\u0440\u0430\u0442\u0438\u0442\u0435 \u0432\u043d\u0438\u043c\u0430\u043d\u0438\u0435, \u0447\u0442\u043e \u0432\u043c\u0435\u0441\u0442\u043e <strong>local[*]<\/strong> \u0434\u043b\u044f \u043f\u0430\u0440\u0430\u043c\u0435\u0442\u0440\u0430 <strong>spark.master<\/strong> \u043c\u043e\u0436\u043d\u043e \u0443\u043a\u0430\u0437\u0430\u0442\u044c \u0430\u0434\u0440\u0435\u0441 \u0434\u0435\u0439\u0441\u0442\u0432\u0443\u044e\u0449\u0435\u0433\u043e \u043c\u0430\u0441\u0442\u0435\u0440\u0430 Spark, \u0432 \u0442\u043e\u043c \u0447\u0438\u0441\u043b\u0435 \u0438 \u0447\u0435\u0440\u0435\u0437 \u043f\u0435\u0440\u0435\u043c\u0435\u043d\u043d\u0443\u044e docker <strong>-e SPARK_MASTER=URI<\/strong>. \u041f\u043e \u0443\u043c\u043e\u043b\u0447\u0430\u043d\u0438\u044e, \u0431\u0443\u0434\u0435\u0442 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c\u0441\u044f \u0432\u0441\u0442\u0440\u0430\u0438\u0432\u0430\u0435\u043c\u044b\u0439 \u0441\u0435\u0440\u0432\u0435\u0440 Spark, \u0447\u0442\u043e \u0432\u043f\u043e\u043b\u043d\u0435 \u043f\u043e\u0434\u0445\u043e\u0434\u0438\u0442 \u043f\u0440\u0438 \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u0438 \u0432\u044b\u0447\u0438\u0441\u043b\u0435\u043d\u0438\u0439 \u043d\u0430 \u043e\u0434\u043d\u043e\u043c \u0443\u0437\u043b\u0435.<\/p>\n\n\n\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u043c\u043e\u0436\u043d\u043e \u0437\u0430\u043f\u0443\u0441\u0442\u0438\u0442\u044c SJS \u0441 \u0431\u044d\u043a\u0435\u043d\u0434\u043e\u043c MySQL:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">docker run --restart=always --link mariadb_1:mysql \\\n      -v $(pwd)\/docker.conf:\/app\/docker.conf \\\n      -v $(pwd)\/data:\/tmp\/data \\\n      --name sjs -d \\\n      -p 8090:8090 spark-job-server<\/pre>\n\n\n\n<p>\u0415\u0441\u043b\u0438 \u0441\u0435\u0440\u0432\u0435\u0440, \u043d\u0430 \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u0432\u044b \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442\u0435 SJS \u0434\u043e\u0441\u0442\u0443\u043f\u0435\u043d \u0438\u0437\u0432\u043d\u0435, \u043d\u0435\u043e\u0431\u0445\u043e\u0434\u0438\u043c\u043e \u043f\u0440\u0438 \u0437\u0430\u043f\u0443\u0441\u043a\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c <strong>-p 127.0.0.1:8090:8090<\/strong> \u0432\u043c\u0435\u0441\u0442\u043e <strong>-p 8090:8090<\/strong>, \u0430 \u0434\u043e\u0441\u0442\u0443\u043f \u043a \u0441\u0435\u0440\u0432\u0438\u0441\u0443 \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u0442\u044c \u0447\u0435\u0440\u0435\u0437 Nginx \u0441 \u0430\u0443\u0442\u0435\u043d\u0442\u0438\u0444\u0438\u043a\u0430\u0446\u0438\u0435\u0439, \u043d\u0430\u043f\u0440\u0438\u043c\u0435\u0440, \u0447\u0435\u0440\u0435\u0437 \u0431\u0430\u0437\u043e\u0432\u0443\u044e \u0430\u0443\u0442\u0435\u043d\u0442\u0438\u0444\u0438\u043a\u0430\u0446\u0438\u044e HTTP. \u0422\u0430\u043a \u0436\u0435 \u043b\u0443\u0447\u0448\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c \u0437\u0430\u0449\u0438\u0449\u0435\u043d\u043d\u044b\u0439 \u043f\u0440\u043e\u0442\u043e\u043a\u043e\u043b HTTPS. \u0412 \u043d\u0430\u0448\u0438\u0445 \u0440\u0443\u043a\u043e\u0432\u043e\u0434\u0441\u0442\u0432\u0430\u0445 \u0432\u044b \u0441\u043c\u043e\u0436\u0435\u0442\u0435 \u043d\u0430\u0439\u0442\u0438 \u043a\u0430\u043a \u044d\u0442\u043e \u0441\u0434\u0435\u043b\u0430\u0442\u044c \u0434\u043b\u044f <a rel=\"noreferrer noopener\" href=\"https:\/\/netpoint-dc.com\/blog\/nastroika-proxy-nginx-s-sertifikatom-let-s-encrypt-v-ubuntu-18-04-bionic-ili-debian-9-stretch\/\" target=\"_blank\">Ubuntu \u0438 Debian<\/a>, <a rel=\"noreferrer noopener\" href=\"https:\/\/netpoint-dc.com\/blog\/nastroika-balansiruyushego-proxy-nginx-s-sertificatom-letss-encrypt-v-centos-7\/\" target=\"_blank\">CentOS<\/a> \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e \u0431\u0435\u0441\u043f\u043b\u0430\u0442\u043d\u043e\u0433\u043e \u0441\u0435\u0440\u0442\u0438\u0444\u0438\u043a\u0430\u0442\u0430 Let&#8217;s Encrypt. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u0417\u0430\u043f\u0443\u0441\u043a \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f Spark \u0432 Spark Job Server<\/h2>\n\n\n\n<p>\u0414\u043b\u044f \u0434\u0430\u043b\u044c\u043d\u0435\u0439\u0448\u0435\u0433\u043e \u0442\u0435\u0441\u0442\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f SJS \u0437\u0430\u0433\u0440\u0443\u0437\u0438\u043c \u0442\u0435\u0441\u0442\u043e\u0432\u044b\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f Spark, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u043f\u043e\u0441\u0442\u0430\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u0432 \u0441\u043e\u0441\u0442\u0430\u0432\u0435 SJS:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">JOBFILE=$(find ..\/spark-jobserver\/job-server-tests\/target\/ -name 'job-server-tests*.jar')\n\ncurl -X POST localhost:8090\/binaries\/test -H 'Content-Type: application\/java-archive' --data-binary @$JOBFILE<\/pre>\n\n\n\n<p>\u0417\u0430\u043f\u0443\u0441\u0442\u0438\u043c \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0435 \u043f\u043e\u0434\u0441\u0447\u0435\u0442\u0430 \u0441\u043b\u043e\u0432:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">curl -d \"input.string = a b c a b see\" \"localhost:8090\/jobs?appName=test&amp;classPath=spark.jobserver.WordCountExample\"\n\n{\n  \"duration\": \"Job not done yet\",\n  \"classPath\": \"spark.jobserver.WordCountExample\",\n  \"startTime\": \"2019-04-17T14:44:08.435Z\",\n  \"context\": \"78c96fd8-222f-4a5a-af24-10171c42065b-spark.jobserver.WordCountExample\",\n  \"status\": \"STARTED\",\n  \"jobId\": \"8a99ed4e-8a54-4ab0-b61e-fbe2bb05bda8\",\n  \"contextId\": \"f2b0dbea-d95f-4190-9b37-a87a055d1acc\"\n}<\/pre>\n\n\n\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u043c\u043e\u0436\u043d\u043e \u043e\u0442\u0441\u043b\u0435\u0436\u0438\u0432\u0430\u0442\u044c \u0441\u043e\u0441\u0442\u043e\u044f\u043d\u0438\u0435 \u0437\u0430\u0434\u0430\u0447\u0438 \u043f\u043e JobID, \u043a\u0430\u043a \u0442\u043e\u043b\u044c\u043a\u043e \u0437\u0430\u0434\u0430\u0447\u0430 \u0437\u0430\u0432\u0435\u0440\u0448\u0438\u0442\u0441\u044f, \u0432\u044b \u0441\u043c\u043e\u0436\u0435\u0442\u0435 \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 \u0435\u0435 \u0432\u044b\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">curl localhost:8090\/jobs\/8a99ed4e-8a54-4ab0-b61e-fbe2bb05bda8\n{\n  \"duration\": \"1.0 secs\",\n  \"classPath\": \"spark.jobserver.WordCountExample\",\n  \"startTime\": \"2019-04-17T14:44:08.000Z\",\n  \"context\": \"78c96fd8-222f-4a5a-af24-10171c42065b-spark.jobserver.WordCountExample\",\n  \"result\": {\n    \"a\": 2,\n    \"b\": 2,\n    \"see\": 1,\n    \"c\": 1\n  },\n  \"status\": \"FINISHED\",\n  \"jobId\": \"8a99ed4e-8a54-4ab0-b61e-fbe2bb05bda8\",\n  \"contextId\": \"f2b0dbea-d95f-4190-9b37-a87a055d1acc\"\n}<\/pre>\n\n\n\n<p>\u042d\u0442\u043e\u0442 \u0436\u0435 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 \u043c\u043e\u0436\u043d\u043e \u0443\u0432\u0438\u0434\u0435\u0442\u044c \u0438 \u0447\u0435\u0440\u0435\u0437 \u0431\u0440\u0430\u0443\u0437\u0435\u0440:<\/p>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-27.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-27-1024x576.png\" alt=\"\" class=\"wp-image-3570\" srcset=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-27-1024x576.png 1024w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-27-300x169.png 300w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-27-768x432.png 768w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-27-624x351.png 624w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-27.png 1920w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<figure class=\"wp-block-image\"><a href=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-25.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-25-1024x576.png\" alt=\"\" class=\"wp-image-3571\" srcset=\"https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-25-1024x576.png 1024w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-25-300x169.png 300w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-25-768x432.png 768w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-25-624x351.png 624w, https:\/\/netpoint-dc.com\/blog\/wp-content\/uploads\/2019\/04\/snimok-jekrana-ot-2019-04-17-10-47-25.png 1920w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">\u0417\u0430\u043f\u0443\u0441\u043a \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f PySpark<\/h2>\n\n\n\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0435 SJS \u0434\u043b\u044f Spark \u043f\u043e <a rel=\"noreferrer noopener\" aria-label=\"\u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/github.com\/spark-jobserver\/spark-jobserver\/blob\/master\/doc\/python.md\" target=\"_blank\">\u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438<\/a>:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">mkdir job\ncat > job\/__init__.py\n\nfrom sparkjobserver.api import SparkJob, build_problems\n\nclass WordCountSparkJob(SparkJob):\n\n    def validate(self, context, runtime, config):\n        if config.get('input.strings', None):\n            return config.get('input.strings')\n        else:\n            return build_problems([\"config 'input.strings' not found\"])\n\n    def run_job(self, context, runtime, data):\n        return context._sc.parallelize(data).countByValue()\n\nCtrl^D\n<\/pre>\n\n\n\n<p>\u0421\u043a\u043e\u043f\u0438\u0440\u0443\u0435\u043c \u0437\u0430\u0432\u0438\u0441\u0438\u043c\u043e\u0441\u0442\u0438 \u0434\u043b\u044f \u0432\u0437\u0430\u0438\u043c\u043e\u0434\u0435\u0439\u0441\u0442\u0432\u0438\u044f \u0441 API SJS \u0432 \u043f\u0440\u043e\u0435\u043a\u0442:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">cp -R ..\/spark-jobserver\/job-server-python\/src\/python\/sparkjobserver .<\/pre>\n\n\n\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u0444\u0430\u0439\u043b \u0434\u043b\u044f \u0441\u0431\u043e\u0440\u043a\u0438 \u043f\u0430\u043a\u0435\u0442\u0430 Python:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">cat > setup.py\n \nfrom setuptools import find_packages, setup\n\nsetup(\n        name='job',\n        packages=['job','sparkjobserver'],\n        include_package_data=True,\n)\n\nCtrl^D<\/pre>\n\n\n\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u043f\u0430\u043a\u0435\u0442 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">python setup.py bdist_egg\n\nrunning bdist_egg\nrunning egg_info\nwriting job.egg-info\/PKG-INFO\nwriting top-level names to job.egg-info\/top_level.txt\nwriting dependency_links to job.egg-info\/dependency_links.txt\nreading manifest file 'job.egg-info\/SOURCES.txt'\nwriting manifest file 'job.egg-info\/SOURCES.txt'\ninstalling library code to build\/bdist.linux-x86_64\/egg\nrunning install_lib\nrunning build_py\ncreating build\/bdist.linux-x86_64\/egg\ncreating build\/bdist.linux-x86_64\/egg\/sparkjobserver\ncopying build\/lib.linux-x86_64-2.7\/sparkjobserver\/subprocess.py -> build\/bdist.linux-x86_64\/egg\/sparkjobserver\ncopying build\/lib.linux-x86_64-2.7\/sparkjobserver\/api.py -> build\/bdist.linux-x86_64\/egg\/sparkjobserver\ncopying build\/lib.linux-x86_64-2.7\/sparkjobserver\/__init__.py -> build\/bdist.linux-x86_64\/egg\/sparkjobserver\ncreating build\/bdist.linux-x86_64\/egg\/job\ncopying build\/lib.linux-x86_64-2.7\/job\/__init__.py -> build\/bdist.linux-x86_64\/egg\/job\nbyte-compiling build\/bdist.linux-x86_64\/egg\/sparkjobserver\/subprocess.py to subprocess.pyc\nbyte-compiling build\/bdist.linux-x86_64\/egg\/sparkjobserver\/api.py to api.pyc\nbyte-compiling build\/bdist.linux-x86_64\/egg\/sparkjobserver\/__init__.py to __init__.pyc\nbyte-compiling build\/bdist.linux-x86_64\/egg\/job\/__init__.py to __init__.pyc\ncreating build\/bdist.linux-x86_64\/egg\/EGG-INFO\ncopying job.egg-info\/PKG-INFO -> build\/bdist.linux-x86_64\/egg\/EGG-INFO\ncopying job.egg-info\/SOURCES.txt -> build\/bdist.linux-x86_64\/egg\/EGG-INFO\ncopying job.egg-info\/dependency_links.txt -> build\/bdist.linux-x86_64\/egg\/EGG-INFO\ncopying job.egg-info\/top_level.txt -> build\/bdist.linux-x86_64\/egg\/EGG-INFO\nzip_safe flag not set; analyzing archive contents...\ncreating 'dist\/job-0.0.0-py2.7.egg' and adding 'build\/bdist.linux-x86_64\/egg' to it\nremoving 'build\/bdist.linux-x86_64\/egg' (and everything under it)<\/pre>\n\n\n\n<p>\u0417\u0430\u0433\u0440\u0443\u0437\u0438\u043c \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u0435 PySpark \u043d\u0430 \u0441\u0435\u0440\u0432\u0435\u0440:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">JOBFILE=$(find -name *.egg)\n\ncurl --data-binary @$JOBFILE -H 'Content-Type: application\/python-archive' localhost:8090\/binaries\/my_py_job<\/pre>\n\n\n\n<p>\u0421\u043e\u0437\u0434\u0430\u0434\u0438\u043c \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442 \u0434\u043b\u044f \u0438\u0441\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u044f \u0437\u0430\u0434\u0430\u0447\u0438:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"false\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">curl -X POST \"localhost:8090\/contexts\/py-context?context-factory=spark.jobserver.python.PythonSessionContextFactory\"\n\n{\n  \"status\": \"SUCCESS\",\n  \"result\": \"Context initialized\"\n}<\/pre>\n\n\n\n<p>\u0417\u0430\u043f\u0443\u0441\u0442\u0438\u043c \u0437\u0430\u0434\u0430\u0447\u0443:<\/p>\n\n\n\n<pre class=\"EnlighterJSRAW\" data-enlighter-language=\"generic\" data-enlighter-theme=\"\" data-enlighter-highlight=\"\" data-enlighter-linenumbers=\"\" data-enlighter-lineoffset=\"\" data-enlighter-title=\"\" data-enlighter-group=\"\">curl-d 'input.strings = [\"a\", \"b\", \"a\", \"b\" ]' \"localhost:8090\/jobs?appName=my_py_job&amp;classPath=job.WordCountSparkJob&amp;context=py-context\"\n{\n  \"duration\": \"Job not done yet\",\n  \"classPath\": \"job.WordCountSparkJob\",\n  \"startTime\": \"2019-04-17T15:17:58.397Z\",\n  \"context\": \"py-context\",\n  \"status\": \"STARTED\",\n  \"jobId\": \"932bc1d3-011e-459b-9618-111832a1ef87\",\n  \"contextId\": \"555eb5ba-9951-4917-982b-8b944cc782dc\"\n}\n\ncurl localhost:8090\/jobs\/932bc1d3-011e-459b-9618-111832a1ef87\n\n{\n  \"duration\": \"1.0 secs\",\n  \"classPath\": \"job.WordCountSparkJob\",\n  \"startTime\": \"2019-04-17T15:17:58.000Z\",\n  \"context\": \"py-context\",\n  \"result\": {\n    \"a\": 2,\n    \"b\": 2\n  },\n  \"status\": \"FINISHED\",\n  \"jobId\": \"932bc1d3-011e-459b-9618-111832a1ef87\",\n  \"contextId\": \"555eb5ba-9951-4917-982b-8b944cc782dc\"\n}<\/pre>\n\n\n\n<p>\u0422\u0435\u043f\u0435\u0440\u044c \u0432\u044b \u043c\u043e\u0436\u0435\u0442\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c SJS \u0434\u043b\u044f \u0440\u0430\u0431\u043e\u0442\u044b \u0441\u043e Spark \u0441 \u043f\u043e\u043c\u043e\u0449\u044c\u044e REST API. \u0414\u0430\u043b\u044c\u043d\u0435\u0439\u0448\u0438\u0435 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438 \u043f\u043e \u0440\u0430\u0431\u043e\u0442\u0435 \u0441 PySpark \u0438 SJS \u043c\u043e\u0436\u043d\u043e \u043d\u0430\u0439\u0442\u0438 \u0432 <a rel=\"noreferrer noopener\" aria-label=\"\u043f\u0440\u043e\u0444\u0438\u043b\u044c\u043d\u043e\u0439 (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/github.com\/spark-jobserver\/spark-jobserver\/blob\/master\/doc\/python.md\" target=\"_blank\">\u043f\u0440\u043e\u0444\u0438\u043b\u044c\u043d\u043e\u0439<\/a> \u0441\u0442\u0430\u0442\u044c\u0435. \u0415\u0441\u043b\u0438 \u0432\u044b \u043f\u043b\u0430\u043d\u0438\u0440\u0443\u0435\u0442\u0435 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c Scala, \u0432\u0430\u043c \u0441\u043b\u0435\u0434\u0443\u0435\u0442 \u043e\u0431\u0440\u0430\u0442\u0438\u0442\u044c\u0441\u044f \u043a <a rel=\"noreferrer noopener\" aria-label=\"\u043f\u0440\u0438\u043c\u0435\u0440\u0430\u043c (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/github.com\/spark-jobserver\/spark-jobserver\/tree\/master\/job-server-tests\/src\/main\/scala\/spark\/jobserver\" target=\"_blank\">\u043f\u0440\u0438\u043c\u0435\u0440\u0430\u043c<\/a> \u0434\u043b\u044f Scala, \u0435\u0441\u043b\u0438 Java &#8212; \u043a <a rel=\"noreferrer noopener\" aria-label=\"\u043f\u0440\u0438\u043c\u0435\u0440\u0430\u043c (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/github.com\/spark-jobserver\/spark-jobserver\/tree\/master\/job-server-tests\/src\/main\/java\/spark\/jobserver\" target=\"_blank\">\u043f\u0440\u0438\u043c\u0435\u0440\u0430\u043c<\/a> \u0434\u043b\u044f Java.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">\u0417\u0430\u043a\u043b\u044e\u0447\u0435\u043d\u0438\u0435<\/h2>\n\n\n\n<p> Spark Job Server &#8212; \u043c\u043e\u0449\u043d\u044b\u0439 \u0438\u043d\u0441\u0442\u0440\u0443\u043c\u0435\u043d\u0442, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u0440\u0430\u0431\u043e\u0442\u0430\u0442\u044c \u0441 \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u043e\u043c \u0438\u043b\u0438 \u043b\u043e\u043a\u0430\u043b\u044c\u043d\u044b\u043c \u0432\u044b\u0447\u0438\u0441\u043b\u0438\u0442\u0435\u043b\u0435\u043c Spark \u044d\u0444\u0444\u0435\u043a\u0442\u0438\u0432\u043d\u043e. \u0423\u043d\u0438\u043a\u0430\u043b\u044c\u043d\u044b\u0435 \u0432\u043e\u0437\u043c\u043e\u0436\u043d\u043e\u0441\u0442\u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044f <a rel=\"noreferrer noopener\" aria-label=\"\u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u044b\u0445 \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442\u043e\u0432 (\u043e\u0442\u043a\u0440\u043e\u0435\u0442\u0441\u044f \u0432 \u043d\u043e\u0432\u043e\u0439 \u0432\u043a\u043b\u0430\u0434\u043a\u0435)\" href=\"https:\/\/bitworks.software\/2019-04-15-how-to-run-low-latency-jobs-with-spark.html\" target=\"_blank\">\u043f\u043e\u0441\u0442\u043e\u044f\u043d\u043d\u044b\u0445 \u043a\u043e\u043d\u0442\u0435\u043a\u0441\u0442\u043e\u0432<\/a> Spark \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0442 \u0437\u0430\u043f\u0443\u0441\u043a\u0430\u0442\u044c \u0440\u043e\u0434\u0441\u0442\u0432\u0435\u043d\u043d\u044b\u0435 \u0437\u0430\u0434\u0430\u0447\u0438 \u0431\u044b\u0441\u0442\u0440\u043e \u0441 \u0443\u043c\u0435\u043d\u044c\u0448\u0435\u043d\u043d\u044b\u043c\u0438 \u043d\u0430\u043a\u043b\u0430\u0434\u043d\u044b\u043c\u0438 \u0440\u0430\u0441\u0445\u043e\u0434\u0430\u043c\u0438.<\/p>\n\n\n\n<p>\u0417\u0430 \u0434\u0430\u043b\u044c\u043d\u0435\u0439\u0448\u0438\u043c\u0438 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u044f\u043c\u0438 \u043f\u043e \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u044e SJS \u043c\u044b \u0440\u0435\u043a\u043e\u043c\u0435\u043d\u0434\u0443\u0435\u043c \u043e\u0431\u0440\u0430\u0442\u0438\u0442\u044c\u0441\u044f \u043a \u043e\u0444\u0438\u0446\u0438\u0430\u043b\u044c\u043d\u043e\u0439 <a href=\"https:\/\/github.com\/spark-jobserver\/spark-jobserver\">\u0434\u043e\u043a\u0443\u043c\u0435\u043d\u0442\u0430\u0446\u0438\u0438<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u0412 \u044d\u0442\u043e\u0439 \u0441\u0442\u0430\u0442\u044c\u0435 \u043c\u044b \u0440\u0430\u0441\u0441\u043c\u043e\u0442\u0440\u0438\u043c, \u043a\u0430\u043a \u0441\u043e\u0431\u0440\u0430\u0442\u044c \u0438 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u0442\u044c Spark Job Server \u0432 \u0432\u0438\u0434\u0435 \u043f\u0440\u0438\u043b\u043e\u0436\u0435\u043d\u0438\u044f \u0434\u043b\u044f Docker. Spark Job Server &#8212; \u043f\u043e\u043b\u0435\u0437\u043d\u043e\u0435 \u0434\u043e\u043f\u043e\u043b\u043d\u0435\u043d\u0438\u0435 Spark, \u043a\u043e\u0442\u043e\u0440\u043e\u0435 \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u0435\u0442 \u0437\u0430\u0433\u0440\u0443\u0436\u0430\u0442\u044c \u0438 \u0432\u044b\u043f\u043e\u043b\u043d\u044f\u0442\u044c \u0437\u0430\u0434\u0430\u043d\u0438\u044f Spark \u0447\u0435\u0440\u0435\u0437 Rest API. Spark Job Server \u0432\u0445\u043e\u0434\u0438\u0442 \u0432 \u0441\u043e\u0441\u0442\u0430\u0432 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u044b\u0445 \u0434\u0438\u0441\u0442\u0440\u0438\u0431\u0443\u0442\u0438\u0432\u043e\u0432 \u0434\u043b\u044f \u0431\u043e\u043b\u044c\u0448\u0438\u0445 \u0434\u0430\u043d\u043d\u044b\u0445, \u0442\u0430\u043a\u0438\u0445 \u043a\u0430\u043a Cloudera CDH5 \u0438\u043b\u0438 Hortonworks, \u043e\u0434\u043d\u0430\u043a\u043e, \u0443\u0441\u0442\u0430\u043d\u043e\u0432\u043a\u0430 \u043f\u043e\u043b\u043d\u043e\u0446\u0435\u043d\u043d\u043e\u0433\u043e \u043a\u043b\u0430\u0441\u0442\u0435\u0440\u0430 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,17],"tags":[],"class_list":["post-3536","post","type-post","status-publish","format-standard","hentry","category-novosti-i-obyavlenia","category-helpful-information"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/posts\/3536","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/comments?post=3536"}],"version-history":[{"count":27,"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/posts\/3536\/revisions"}],"predecessor-version":[{"id":3647,"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/posts\/3536\/revisions\/3647"}],"wp:attachment":[{"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/media?parent=3536"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/categories?post=3536"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/netpoint-dc.com\/blog\/wp-json\/wp\/v2\/tags?post=3536"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}