Assessing both wave height and wind speed at the same time
import modelskill as ms
Below, the observations will take the default quantity names from the eum type of the item. Alternatively, the user can give another quantity name by providing the quantity
argument.
# wave height
o1 = ms.PointObservation('../tests/testdata/SW/HKNA_Hm0.dfs0', item=0, x=4.2420, y=52.6887, name="HKNA_Hm0")
o2 = ms.PointObservation("../tests/testdata/SW/eur_Hm0.dfs0", item=0, x=3.2760, y=51.9990, name="EPL_Hm0")
o3 = ms.TrackObservation("../tests/testdata/SW/Alti_c2_Dutch.dfs0", item=3, name="c2_Hm0")
# wind speed
wind1 = ms.PointObservation('../tests/testdata/SW/HKNA_wind.dfs0', item=0, x=4.2420, y=52.6887, name="HKNA_wind")
wind2 = ms.PointObservation('../tests/testdata/SW/F16_wind.dfs0', item=0, x=4.01222, y=54.1167, name="F16_wind")
wind3 = ms.TrackObservation("../tests/testdata/SW/Alti_c2_Dutch.dfs0", item=2, name="c2_wind")
o1.quantity
Quantity(name='Significant wave height', unit='meter')
Two different simulation runs are used to create four model results:
fn1 = "../tests/testdata/SW/HKZN_local_2017_DutchCoast.dfsu"
mr1Hm0 = ms.model_result(fn1, name='SW_1', item='Sign. Wave Height')
mr1WS = ms.model_result(fn1, name='SW_1', item='Wind speed')
fn2 = "../tests/testdata/SW/HKZN_local_2017_DutchCoast_v2.dfsu"
mr2Hm0 = ms.model_result(fn2, name='SW_2', item='Sign. Wave Height')
mr2WS = ms.model_result(fn2, name='SW_2', item='Wind speed')
mr1WS
<DfsuModelResult> 'SW_1'
We connect the observation item and model item by refering to the item name in the ModelResult. Item number can also be used.
cc1 = ms.match([o1, o2, o3], [mr1Hm0, mr2Hm0])
cc2 = ms.match([wind1, wind2, wind3], [mr1WS, mr2WS])
cc = cc1 + cc2
cc
<ComparerCollection> Comparer: HKNA_Hm0 Comparer: EPL_Hm0 Comparer: c2_Hm0 Comparer: HKNA_wind Comparer: F16_wind Comparer: c2_wind
cc.n_quantities
2
cc.quantity_names
['Significant wave height', 'Wind speed']
Now that the result has been extracted, we can do analysis. Multiple quantities means an extra level in the multi-index of the skill dataframe.
sk = cc.skill()
sk.round(3)
n | bias | rmse | urmse | mae | cc | si | r2 | |||
---|---|---|---|---|---|---|---|---|---|---|
model | observation | quantity | ||||||||
SW_1 | EPL_Hm0 | Significant wave height | 67 | -0.067 | 0.224 | 0.213 | 0.189 | 0.970 | 0.083 | 0.933 |
F16_wind | Wind speed | 67 | 2.102 | 2.775 | 1.812 | 2.197 | 0.825 | 0.141 | 0.230 | |
HKNA_Hm0 | Significant wave height | 386 | -0.194 | 0.352 | 0.293 | 0.252 | 0.971 | 0.094 | 0.905 | |
HKNA_wind | Wind speed | 277 | -0.881 | 1.276 | 0.923 | 1.023 | 0.963 | 0.066 | 0.861 | |
c2_Hm0 | Significant wave height | 113 | -0.001 | 0.352 | 0.352 | 0.295 | 0.974 | 0.128 | 0.900 | |
c2_wind | Wind speed | 113 | 0.409 | 0.638 | 0.490 | 0.506 | 0.960 | 0.051 | 0.867 | |
SW_2 | EPL_Hm0 | Significant wave height | 67 | -0.000 | 0.232 | 0.232 | 0.198 | 0.970 | 0.090 | 0.927 |
F16_wind | Wind speed | 67 | 2.102 | 2.775 | 1.812 | 2.197 | 0.825 | 0.141 | 0.230 | |
HKNA_Hm0 | Significant wave height | 386 | -0.100 | 0.293 | 0.275 | 0.214 | 0.971 | 0.089 | 0.934 | |
HKNA_wind | Wind speed | 277 | -0.881 | 1.276 | 0.923 | 1.023 | 0.963 | 0.066 | 0.861 | |
c2_Hm0 | Significant wave height | 113 | 0.081 | 0.430 | 0.422 | 0.357 | 0.974 | 0.153 | 0.850 | |
c2_wind | Wind speed | 113 | 0.409 | 0.638 | 0.490 | 0.506 | 0.960 | 0.051 | 0.867 |
sk = cc.sel(quantity='Significant wave height').skill()
sk.style()
n | bias | rmse | urmse | mae | cc | si | r2 | ||
---|---|---|---|---|---|---|---|---|---|
model | observation | ||||||||
SW_1 | EPL_Hm0 | 67 | -0.067 | 0.224 | 0.213 | 0.189 | 0.970 | 0.083 | 0.933 |
HKNA_Hm0 | 386 | -0.194 | 0.352 | 0.293 | 0.252 | 0.971 | 0.094 | 0.905 | |
c2_Hm0 | 113 | -0.001 | 0.352 | 0.352 | 0.295 | 0.974 | 0.128 | 0.900 | |
SW_2 | EPL_Hm0 | 67 | -0.000 | 0.232 | 0.232 | 0.198 | 0.970 | 0.090 | 0.927 |
HKNA_Hm0 | 386 | -0.100 | 0.293 | 0.275 | 0.214 | 0.971 | 0.089 | 0.934 | |
c2_Hm0 | 113 | 0.081 | 0.430 | 0.422 | 0.357 | 0.974 | 0.153 | 0.850 |
sk.sel(observation='c2_Hm0').style(columns='rmse')
observation | n | bias | rmse | urmse | mae | cc | si | r2 | |
---|---|---|---|---|---|---|---|---|---|
model | |||||||||
SW_1 | c2_Hm0 | 113 | -0.001 | 0.352 | 0.352 | 0.295 | 0.974 | 0.128 | 0.900 |
SW_2 | c2_Hm0 | 113 | 0.081 | 0.430 | 0.422 | 0.357 | 0.974 | 0.153 | 0.850 |
sk['rmse'].plot.line(ylabel='Hm0 RMSE [m]');
cc.sel(model=1).plot.scatter();
The mean_skill()
method will return a weighted average of the skill score per model and quantity. You can get the "normal" mean_skill (per model) by selecting a specific quantity either by id or name.
sk = cc.mean_skill()
sk
n | bias | rmse | urmse | mae | cc | si | r2 | ||
---|---|---|---|---|---|---|---|---|---|
model | quantity | ||||||||
SW_1 | Significant wave height | 566 | -0.087356 | 0.309119 | 0.286247 | 0.244979 | 0.971791 | 0.101595 | 0.912468 |
Wind speed | 457 | 0.543226 | 1.563076 | 1.075053 | 1.241847 | 0.915916 | 0.086039 | 0.652388 | |
SW_2 | Significant wave height | 566 | -0.006398 | 0.318593 | 0.310086 | 0.256618 | 0.971791 | 0.110653 | 0.903722 |
Wind speed | 457 | 0.543226 | 1.563076 | 1.075053 | 1.241847 | 0.915916 | 0.086039 | 0.652388 |
sk['si'].plot.bar(title='scatter index');
cc.sel(model='SW_2').mean_skill()
n | bias | rmse | urmse | mae | cc | si | r2 | |
---|---|---|---|---|---|---|---|---|
quantity | ||||||||
Significant wave height | 566 | -0.006398 | 0.318593 | 0.310086 | 0.256618 | 0.971791 | 0.110653 | 0.903722 |
Wind speed | 457 | 0.543226 | 1.563076 | 1.075053 | 1.241847 | 0.915916 | 0.086039 | 0.652388 |
cc.sel(quantity='Significant wave height').mean_skill().style(columns=[])
n | bias | rmse | urmse | mae | cc | si | r2 | |
---|---|---|---|---|---|---|---|---|
model | ||||||||
SW_1 | 566 | -0.087 | 0.309 | 0.286 | 0.245 | 0.972 | 0.102 | 0.912 |
SW_2 | 566 | -0.006 | 0.319 | 0.310 | 0.257 | 0.972 | 0.111 | 0.904 |
Note: The score is a single metric for each model, aggregating all quantities.
cc.score()
{'SW_1': 0.9360973356120318, 'SW_2': 0.9408344041671327}
To get the score for a specific varibable, it is possible to filter using the .sel
method.
cc.sel(quantity='Wind speed').score()
{'SW_1': 1.5630757282647778, 'SW_2': 1.5630757282647778}