Application of Multiprocessing

In this tutorial, we demonstrate how multiprocessing can be used to run custom experimental simulations that are difficult to implement using the ExperimentManager API alone.

Problem Description

We consider two crop rotation systems:

Maize → Wheat → Soybean
Maize → Wheat

Our objective is to evaluate specific planting date combinations, where wheat is planted before soybean. For example, wheat may be terminated on 26th April, followed by soybean planting on 1st May.

Simulation Design

The experiment evaluates multiple planting-date scenarios:

Soybean planting dates are spaced every 5 days
A total of 11 soybean planting dates are considered
For each soybean planting date, the corresponding wheat termination date is adjusted accordingly
Each planting-date combination is simulated for both rotation systems

This results in a total of:

\[2 \times 11 = 22\]

independent simulation runs.

Why Multiprocessing?

Although these simulations could be executed sequentially using a simple loop, they are independent tasks and therefore well suited to a multiprocessing approach. Using multiprocessing:

Reduces overall execution time
Demonstrates how custom simulation workflows can be scaled
Provides flexibility beyond what is easily achievable with the ExperimentManager API

In this tutorial, multiprocessing is used primarily for demonstration purposes, showing how multiple simulation jobs can be dispatched and executed concurrently.

Import the necessary libraries

import os
from pathlib import Path
from loguru import logger
from apsimNGpy.core.config import set_apsim_bin_path
from dotenv import load_dotenv

Set up the workspace

Base_DIR = Path(__file__).parent
plots = Base_DIR / 'Plots'
plots.mkdir(exist_ok=True)
out_apsimx = Base_DIR / 'output'
out_apsimx.mkdir(exist_ok=True)

Create arrays for planting and termination dates

soybean_PD = [
 "01-may",
 "06-may",
 "11-may",
 "16-may",
 "21-may",
 "26-may",
 "31-may",
 "05-jun",
 "10-jun",
 "15-jun",
 "29-jun",]

 wheat_termination_date = [
     "26-apr",
     "01-may",
     "06-may",
     "11-may",
     "16-may",
     "21-may",
     "26-may",
     "31-may",
     "05-jun",
     "10-jun",
     "24-jun",
 ]

Zip them up for iteration since are correlated

td_plt = dict(zip(soybean_PD, wheat_termination_date))

def create_jobs(base_file):
  index =0
  for crop in ['Maize, Wheat', 'Maize, Wheat, Soybean']:

      for idx, (soy_planting, rye_termination) in enumerate(td_plt.items()):
          index += 1
          jj = {
              "model": base_file,
              "ID": index,
              "inputs": [
                  {
                      "path": ".Simulations.P3051.Field1.SowSoy",
                      "StartDate": soy_planting,
                  },
                  {
                      "path": ".Simulations.P3051.Field1.HarvestWheat",
                      "Date": rye_termination,
                  },
                  {
                      'path': '.Simulations.P3051.Field1.Simple Rotation',
                      'Crops': crop
                  }
              ],
          }
          yield jj

base_file = Base_DIR / 'APSIMX' / 'cc_cover.apsimx'

The base file has been configured with a rotation manager. If you require this simple rotation manager, please raise an issue on GitHub and we will be happy to share it.

if __name__ == '__main__':
    # Assumes that the APSIM binary path has been set
    from apsimNGpy.core.mult_cores import MultiCoreManager
    edited_model_path= Base_DIR / 'APSIMX'/'cc_cover_edited.apsimx'
    jobs = create_jobs(edited_model_path)
    manager = MultiCoreManager()
    subset_columns = ['AGB', 'SoyYield']
    manager.run_all_jobs(jobs, ncore=-10, subset_columns=subset_columns)
    df = manager.results
    base_path = Base_DIR / 'Results'
    for tables in df.source_table.unique():
        tb_df = df[df.source_table == tables]
        tb_df.to_csv(base_path/f'{tables}.csv', index=False)
    df.to_csv(Base_DIR / 'Results/simulated.csv', index=False)
    wheat_r= df[df['source_table']=='WheatR']