QuantCluster is a set of plugins for MIT's StarCluster useful for scalable quantitative finance research with Amazon’s Elastic Compute Cloud (EC2). It provides reproducible parallel computing environments that can be scaled to accommodate large data sets.
The quant_cluster plugin itself allows the user to specify software to install via apt-get, pip, and pip+git on every node of the cluster. The other plugins are for configuring Hadoop, Hive, and HBase on a StarCluster.
The following packages are added to the base StarCluster images with make_image.sh:
- Zipline - Algorithmic trading library.
- swigibpy - Third party Interactive Brokers Python API generated from TWS C++ API using SWIG.
- Celery - Fault tolerant distributed task queue for live strategy execution and data processing.
- Apache Hadoop - Framework for writing massively distributed map/reduce tasks.
- Apache Hive - Data warehouse querying and managing large datasets residing in distributed storage.
- Apache HBase - Random, realtime read/write access to big financial data.
Additionally, the following are already on the base StarCluster images:
- IPython - An advanced interactive shell for Python with notebooks.
- NumPy - Fast array and numerical library for Python (compiled against OpenBLAS).
- SciPy - Scientific algorithms library for Python (compiled against OpenBLAS).
- Pandas - Data analysis tools for Python.
- OpenMPI - Library used for writing/running parallel applications.
- OpenMP - Library for shared memory multiprocessing.
- OpenBLAS - Multi-threaded BLAS library.
- Open Grid Scheduler - Queuing system for scheduling jobs on the cluster and handling load balancing.
- NFS - Network File System for sharing folders across the cluster.
pip install quantclusterOlder versions are on the Python Package Index page. Documentation can be found at StarCluster site.