Deployment

Since Starsim is implemented in pure Python, it can easily be deployed on the cloud. Here we describe some different approaches for doing this.

Virtual machine

One of the most common approaches is to run Starsim on a single large virtual machine (VM). By default, ss.MultiSim (and ss.parallel()) will use all available cores. If your script already makes use of these, then you don’t need to make any more changes:

import sciris as sc
import starsim as ss

base_pars = sc.objdict(
    n_agents = 10e3,
    diseases = sc.objdict(
        type = 'sis',
        beta = 0.1,
    ),
    networks = 'random',
    rand_seed = 1,
    verbose = False,
)

# Generate sims in serial
sims = sc.autolist() # Can also just use []
for i in range(10):
    pars = base_pars.copy()
    pars.diseases.beta *= sc.perturb()
    pars.rand_seed = i
    sim = ss.Sim(pars)
    sims += sim

# Run in parallel
msim = ss.parallel(sims)
msim.plot(legend=False)

Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd526fb060>:
Traceback (most recent call last):
  File "/home/runner/work/starsim/starsim/starsim/time.py", line 261, in _rebuild
    out = unpickling_func(*args)
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd527ebf10>:
Traceback (most recent call last):
  File "/home/runner/work/starsim/starsim/starsim/time.py", line 261, in _rebuild
    out = unpickling_func(*args)
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd5253bd80>:
Traceback (most recent call last):
  File "/home/runner/work/starsim/starsim/starsim/time.py", line 261, in _rebuild
    out = unpickling_func(*args)
BufferError: Existing exports of data: object cannot be re-sized

Figure(768x576)

Note that this example uses sc.objdict() rather than dict() – either work, but it means you can use pars.diseases.beta rather than pars['diseases']['beta']. You could also create full Starsim objects (e.g. diseases = ss.SIS() and then modify pars.diseases.pars.beta).

In some cases, creating the sim is itself a time-consuming step (especially if hundreds or thousands are being generated). In this case, you can write a make_sim() function and parallelize that too:

def make_sim(i, pars):
    pars.diseases.beta *= sc.perturb() # Don't need to copy pars since implicitly copied via the pickling process
    pars.rand_seed = i
    sim = ss.Sim(pars)
    return sim

sims = sc.parallelize(make_sim, range(10), pars=base_pars, serial=False)
msim = ss.parallel(sims)
msim.plot(legend=False)

Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd52420400>:
Traceback (most recent call last):
  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
    self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd52238d10>:
Traceback (most recent call last):
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd52420400>  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
:
    Traceback (most recent call last):
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump

    BufferErrorself.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps: 
Existing exports of data: object cannot be re-sizedBufferError
: Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51682570>Existing exports of data: object cannot be re-sized
:
Traceback (most recent call last):
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd52238d10>:
  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
Traceback (most recent call last):
      File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps    
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumpsBufferError
: BufferErrorExisting exports of data: object cannot be re-sized: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51682c50>
:
Traceback (most recent call last):
  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51682570>:
    Traceback (most recent call last):
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump

    self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumpsBufferError: 
BufferErrorExisting exports of data: object cannot be re-sized: 
Existing exports of data: object cannot be re-sizedException ignored while finalizing file <_io.BytesIO object at 0x7fcd51682e80>
:
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51682c50>Traceback (most recent call last):
:
  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
Traceback (most recent call last):
  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
        self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumpsself.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps

BufferErrorBufferError: : Existing exports of data: object cannot be re-sizedExisting exports of data: object cannot be re-sized

Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51682e80>:
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683420>Traceback (most recent call last):
:
  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
Traceback (most recent call last):
      File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps
    BufferErrorself.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps: 
Existing exports of data: object cannot be re-sizedBufferError
: Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683420>Existing exports of data: object cannot be re-sized:

Traceback (most recent call last):
  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd516836a0>    :
Traceback (most recent call last):
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump

    BufferErrorself.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps: 
Existing exports of data: object cannot be re-sized
BufferErrorException ignored while finalizing file <_io.BytesIO object at 0x7fcd516836a0>: :
Existing exports of data: object cannot be re-sizedTraceback (most recent call last):

  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683920>    :
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumpsTraceback (most recent call last):

  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
BufferError    self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps: 
Existing exports of data: object cannot be re-sized
BufferErrorException ignored while finalizing file <_io.BytesIO object at 0x7fcd51683920>: :
Existing exports of data: object cannot be re-sizedTraceback (most recent call last):

  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683ba0>    :
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumpsTraceback (most recent call last):

BufferError  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
    : self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumpsExisting exports of data: object cannot be re-sized

Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683ba0>BufferError: :
Existing exports of data: object cannot be re-sizedTraceback (most recent call last):

  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683e20>    :
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumpsTraceback (most recent call last):
  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump

    BufferErrorself.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps: 
Existing exports of data: object cannot be re-sizedBufferError
: Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683e20>Existing exports of data: object cannot be re-sized:

Traceback (most recent call last):
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd516cc0e0>:
  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
Traceback (most recent call last):
      File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps    
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumpsBufferError
: BufferErrorExisting exports of data: object cannot be re-sized: 
Existing exports of data: object cannot be re-sizedException ignored while finalizing file <_io.BytesIO object at 0x7fcd516cc0e0>:

Traceback (most recent call last):
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd516cc360>  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
:
    Traceback (most recent call last):
self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump

    BufferErrorself.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps: 
Existing exports of data: object cannot be re-sized
BufferErrorException ignored while finalizing file <_io.BytesIO object at 0x7fcd516cc360>: :
Existing exports of data: object cannot be re-sizedTraceback (most recent call last):

  File "/home/runner/work/starsim/starsim/starsim/distributions.py", line 477, in jump
    self.bitgen.state = self.bitgen.jumped(jumps=jumps).state # Now take "jumps" number of jumps
BufferError: Existing exports of data: object cannot be re-sized

Figure(768x576)

Note that parallelizing the build process pickles and unpickles the sims, which can be an expensive operation. make_sim() functions can often get quite complicated, so it’s often good software engineering practice to separate them out anyway. You can use the serial=True argument of Sciris’ sc.parallelize() function (which is what ss.parallel() calls under the hood) in order to run in serial, to see if it’s the same speed or faster.

While the traditional way to run on a VM is via SSH and terminal, it is also possible to run remotely via VS Code (and Cursor etc.), PyCharm, or Spyder. You can also run a Jupyter server on the VM and access it that way (we like The Littlest JupyterHub).

Dask and Coiled

Adapting the examples above, we can fairly easily make Starsim simulations run using other popular tools such as Dask and Joblib. Here’s a Dask example:

import dask
import dask.distributed as dd
import numpy as np
import starsim as ss


def run_sim(index, beta):
    """ Run a standard simulation """
    label = f'Sim {index}, beta={beta:n}'
    sis = ss.SIS(beta=beta)
    sim = ss.Sim(label=label, networks='random', diseases=sis, rand_seed=index, verbose=False)
    sim.run()
    sim.shrink() # Remove People and other states to make pickling faster
    return sim


if __name__ == '__main__':

    # Run settings
    n = 8
    n_workers = 4
    betas = 0.1*np.sort(np.random.random(n))

    # Create and queue the Dask jobs
    client = dd.Client(n_workers=n_workers)
    queued = []
    for i,beta in enumerate(betas):
        run = dask.delayed(run_sim)(i, beta)
        queued.append(run)

    # Run and process the simulations
    sims = list(dask.compute(*queued))
    msim = ss.MultiSim(sims)
    msim.plot()

Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd52420400>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd52238d10>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51682570>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51682c50>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51682e80>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683420>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd516836a0>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683920>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683ba0>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51683e20>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd516cc0e0>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd516cc360>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd516ccdb0>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51e8e390>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51dac040>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51c95d50>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51b87bf0>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51ce0e50>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd51a959e0>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd519d5080>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd50f45440>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized
Exception ignored while finalizing file <_io.BytesIO object at 0x7fcd50d34e00>:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap_external>", line 511, in _compile_bytecode
BufferError: Existing exports of data: object cannot be re-sized

Figure(768x576)

Coiled, which is a paid service by Dask that allows auto-scaling across clusters, has a similar syntax:

import sciris as sc
import starsim as ss
import coiled
import dask.distributed as dd

# Parameters
n_workers = 50
n = 1000

def run_sim(seed):
    sim = ss.Sim(n_agents=100e3, dur=100, diseases='sis', networks='random', rand_seed=seed)
    sim.run().shrink()
    return sim

# Set up cluster
cluster = coiled.Cluster(n_workers=n_workers, workspace="<your_coiled_workspace>")
client = cluster.get_client()

# Set up futures
futures = []
for seed in range(n):
    future = client.submit(run_sim, seed)
    futures.append(future)

# Run
sims = client.gather(futures)

# Plot
msim = ss.MultiSim(sims)
msim.plot()

(Note: You will need a Coiled subscription to run this example.)

Interactive dashboards

Another common desire is to make interactive dashboards. There are many ways to do this, including Shiny for Python, Voila, and Panel, but the simplest is probably Streamlit:

import streamlit as st
import starsim as ss

def run_sim(beta, n_agents):
    sis = ss.SIS(beta=beta)
    sim = ss.Sim(
        n_agents = n_agents,
        diseases = sis,
        networks = 'random',
    )
    sim.run()
    sim.label = f'Beta={beta:n} • Agents={n_agents:,} • Time={sim.timer.total:0.1f} s'
    return sim

# Create the Streamlit interface
st.title('SIS Dashboard')
beta = st.slider('Transmission rate (beta)', 0.0, 1.0, 0.1)
n_agents = st.slider('Number of agents', 1_000, 100_000, 10_000)

# Run simulation and plot results
sim = run_sim(beta, n_agents)
fig = sim.diseases.sis.plot()
fig.suptitle(sim.label)
st.pyplot(fig)

This example is saved in this folder as streamlit.py, and (after pip install streamlit) can be run with streamlit run streamlit.py. This should give something like this:

Streamlit example