apache spark - API compatibility between scala and python? -

I've read more than a dozen pages of documents, and it seems that:

< Li>

1. The API is fully implemented in Python (I should not learn anything)

2. Interactive mode completely Also works as a shell shell and troubleshooting is equally easy

3. Python modules such as numpy will still be imported (no crippled dragon environment)

   What a Area at least the area that will make it impossible?

   In recent Spark release (1.0+), we implemented all the missing PySpark features listed below is. Some new features are still missing, such as Python binding for graphics, but other APIs have acquired similarities (including a pilot python apc for spark streaming).

   My first answers have been rediscovered again:

   the original answer of SPARC 0.9

   after my original answer (given below this answer) A lot has changed in seven months from:

   • Spark 0.7.3 has fixed the issue "Jvm flare with a big pile" issue.
   • Added support for rotation 0.8.1 (), sample (), and sort ().
   • , the main missing features in PySpark are as follows:
     • Support for reading and writing non-text input formats, such as Handop Sequence File
     • .
     • Cygwin support (though Pyspark works fine under Windows Powerhell or cmd.exe).
     • L>

       Although we have made several performance improvements, there is a performance difference between Spark Scala and the Python API. Spark User mailing list is discussing its current performance.

       If you search for any missing feature in Piespark, please open our new ticket.

       Spark 0.7.2:

       There is a list of its PySpark features. Like Spark 0.7.2, PySpark is currently missing support for (.), Sorting (), and persistence at different storage levels. This feature is also lacking in some feature methods added to the Scala API.

       The Java API was synchronized with Scala API when it was released, but since then many RDD methods have been added and all of them have been added to Java wrapper sections. There is talk about keeping Java API up-to-date. In that thread, I remembered that a technique has been suggested to automatically find the features, so it's a matter of taking time to request and drag someone.

       About the display, PySpark is going to be slow, part of the performance difference with the Scarka spark large differences with forking processes is born from a strange JVM issue, but it should be done to fix it needed. Other hurdles come from the serialization: Now, PySpark users do not need to explicitly register serialisers for their objects (currently we use binary cPickle plus some batch optimization) In the past, I used the user-customized serializer Has seen in adding support for you which will allow you to specify the type of your objects and thus Use Esh serials will be faster; I hope to start working on it at some point again.

       PySpark is implemented using a regular cPython interpreter, so libraries like numpy should work fine (this will not be if PySpark was written in Jyothi).

       Getting started with PySpark is very easy; To run and run the pyspark interpreter should be sufficient to test it on your personal computer and let you evaluate its interactive features. If you want to use Ipianthon, you can use IPYTHON = 1 ./pyspark to launch Pyspark with an IPython shell in your shell.