Here we describe a very simple TransmogrifAI workflow for predicting survivors in the often-cited Titanic dataset. The code for building and applying the Titanic model can be found here: Titanic Code, and the data can be found here: Titanic Data.

First we need to load transmogrifai and Spark Mllib jars

In [1]:
import $ivy.`org.apache.spark::spark-sql:2.3.3`
import $ivy.`org.apache.spark::spark-mllib:2.3.3`
import $ivy.`sh.almond::almond-spark:0.5.0`
import $ivy.`com.salesforce.transmogrifai::transmogrifai-core:0.5.1`

import com.salesforce.op._
import com.salesforce.op.features._
import com.salesforce.op.features.types._
import com.salesforce.op.evaluators.Evaluators
Out[1]:
import $ivy.$                                  

import $ivy.$                                    

import $ivy.$                              

import $ivy.$                                                       


import com.salesforce.op._

import com.salesforce.op.features._

import com.salesforce.op.features.types._

import com.salesforce.op.evaluators.Evaluators

We also want avoid too extensive logging and long outputs in our notebook.

In [2]:
import org.apache.log4j.{Level, Logger}
Logger.getLogger("org.apache.spark").setLevel(Level.WARN)
Logger.getLogger("breeze").setLevel(Level.WARN)
Logger.getLogger("com.salesforce.op").setLevel(Level.WARN)

repl.pprinter() = repl.pprinter().copy(defaultHeight = 10)
Out[2]:
import org.apache.log4j.{Level, Logger}

Instantiate a SparkSession.

In [3]:
import org.apache.spark.sql._

implicit val spark = {
  NotebookSparkSession.builder()
    .progress(false)
    .master("local[*]")
    .appName("TitanicPrediction")
    .getOrCreate()
}
Loading spark-stubs
Getting spark JARs
log4j:WARN No appenders could be found for logger (org.eclipse.jetty.util.log).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Creating SparkSession
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/04/04 23:29:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Out[3]:
import org.apache.spark.sql._


spark: SparkSession = [email protected]

Let us create a case class to describe the schema for the data:

In [4]:
// Needed for now for case classes defined withing Ammonite. Won't be necessary in future versions of Spark.
// See https://github.com/alexarchambault/ammonite-spark/issues/19 and https://github.com/apache/spark/pull/23607
org.apache.spark.sql.catalyst.encoders.OuterScopes.addOuterScope(this);
case class Passenger(
  id: Int,
  survived: Int,
  pClass: Option[Int],
  name: Option[String],
  sex: Option[String],
  age: Option[Double],
  sibSp: Option[Int],
  parCh: Option[Int],
  ticket: Option[String],
  fare: Option[Double],
  cabin: Option[String],
  embarked: Option[String]
)
// Required to make sure the String representation of the case class doesn't change in later cells.
implicit val passengerTypeTag = scala.reflect.runtime.universe.weakTypeTag[Passenger]
Out[4]:
defined class Passenger
passengerTypeTag: reflect.runtime.package.universe.WeakTypeTag[Passenger] = TypeTag[Helper.this.Passenger]

We then define the set of raw features that we would like to extract from the data. The raw features are defined using FeatureBuilders, and are strongly typed. TransmogrifAI supports the following basic feature types: Text, Numeric, Vector, List , Set, Map. In addition it supports many specific feature types which extend these base types: Email extends Text; Integral, Real and Binary extend Numeric; Currency and Percentage extend Real. For a complete view of the types supported see the Type Hierarchy and Automatic Feature Engineering section in the Documentation.

Basic FeatureBuilders will be created for you if you use the TransmogrifAI CLI to bootstrap your project as described here. However, it is often useful to edit this code to customize feature generation and take full advantage of the Feature types available (selecting the appropriate type will improve automatic feature engineering steps).

When defining raw features, specify the extract logic to be applied to the raw data, and also annotate the features as either predictor or response variables via the FeatureBuilders:

In [5]:
val survived = FeatureBuilder.RealNN[Passenger].extract(_.survived.toRealNN).asResponse
val pClass = FeatureBuilder.PickList[Passenger].extract(_.pClass.map(_.toString).toPickList).asPredictor
val name = FeatureBuilder.Text[Passenger].extract(_.name.toText).asPredictor
val sex = FeatureBuilder.PickList[Passenger].extract(_.sex.map(_.toString).toPickList).asPredictor
val age = FeatureBuilder.Real[Passenger].extract(_.age.toReal).asPredictor
val sibSp = FeatureBuilder.Integral[Passenger].extract(_.sibSp.toIntegral).asPredictor
val parCh = FeatureBuilder.Integral[Passenger].extract(_.parCh.toIntegral).asPredictor
val ticket = FeatureBuilder.PickList[Passenger].extract(_.ticket.map(_.toString).toPickList).asPredictor
val fare = FeatureBuilder.Real[Passenger].extract(_.fare.toReal).asPredictor
val cabin = FeatureBuilder.PickList[Passenger].extract(_.cabin.map(_.toString).toPickList).asPredictor
val embarked = FeatureBuilder.PickList[Passenger].extract(_.embarked.map(_.toString).toPickList).asPredictor
Out[5]:
survived: Feature[RealNN] = Feature(
  "survived",
  true,
  FeatureGeneratorStage_000000000001,
  List(),
  "RealNN_000000000001",
  List()
)
pClass: Feature[PickList] = Feature(
  "pClass",
  false,
  FeatureGeneratorStage_000000000002,
  List(),
  "PickList_000000000002",
  List()
)
name: Feature[Text] = Feature(
  "name",
  false,
  FeatureGeneratorStage_000000000003,
  List(),
  "Text_000000000003",
  List()
)
sex: Feature[PickList] = Feature(
  "sex",
  false,
  FeatureGeneratorStage_000000000004,
  List(),
  "PickList_000000000004",
  List()
)
age: Feature[Real] = Feature(
  "age",
  false,
  FeatureGeneratorStage_000000000005,
  List(),
  "Real_000000000005",
  List()
)
sibSp: Feature[Integral] = Feature(
  "sibSp",
  false,
  FeatureGeneratorStage_000000000006,
  List(),
  "Integral_000000000006",
  List()
)
parCh: Feature[Integral] = Feature(
  "parCh",
  false,
  FeatureGeneratorStage_000000000007,
  List(),
  "Integral_000000000007",
  List()
)
ticket: Feature[PickList] = Feature(
  "ticket",
  false,
  FeatureGeneratorStage_000000000008,
  List(),
  "PickList_000000000008",
  List()
)
fare: Feature[Real] = Feature(
  "fare",
  false,
  FeatureGeneratorStage_000000000009,
  List(),
  "Real_000000000009",
  List()
)
cabin: Feature[PickList] = Feature(
  "cabin",
  false,
  FeatureGeneratorStage_00000000000a,
  List(),
  "PickList_00000000000a",
  List()
)
embarked: Feature[PickList] = Feature(
  "embarked",
  false,
  FeatureGeneratorStage_00000000000b,
  List(),
  "PickList_00000000000b",
  List()
)

Now that the raw features have been defined, we go ahead and define how we would like to manipulate them via Stages (Transformers and Estimators). A TransmogrifAI Stage takes one or more Features, and returns a new Feature. TransmogrifAI provides numerous handy short cuts for specifying common feature manipulations. For basic arithmetic operations, you can just use “+”, “-“, “*” and “/”. In addition, shortcuts like “normalize”, “pivot” and “map” are also available.

In [6]:
val familySize = sibSp + parCh + 1
val estimatedCostOfTickets = familySize * fare
val pivotedSex = sex.pivot()
val normedAge = age.fillMissingWithMean().zNormalize()
val ageGroup = age.map[PickList](_.value.map(v => if (v > 18) "adult" else "child").toPickList)
Out[6]:
familySize: FeatureLike[Real] = Feature(
  "parCh-sibSp_2-stagesApplied_Real_00000000000d",
  false,
  UnaryLambdaTransformer_00000000000d,
  WrappedArray(
    Feature(
      "parCh-sibSp_1-stagesApplied_Real_00000000000c",
      false,
      BinaryLambdaTransformer_00000000000c,
...
estimatedCostOfTickets: FeatureLike[Real] = Feature(
  "fare-parCh-sibSp_3-stagesApplied_Real_00000000000e",
  false,
  BinaryLambdaTransformer_00000000000e,
  WrappedArray(
    Feature(
      "parCh-sibSp_2-stagesApplied_Real_00000000000d",
      false,
      UnaryLambdaTransformer_00000000000d,
...
pivotedSex: FeatureLike[OPVector] = Feature(
  "sex_1-stagesApplied_OPVector_00000000000f",
  false,
  OpSetVectorizer_00000000000f,
  WrappedArray(
    Feature(
      "sex",
      false,
      FeatureGeneratorStage_000000000004,
...
normedAge: FeatureLike[RealNN] = Feature(
  "age_2-stagesApplied_RealNN_000000000011",
  false,
  OpScalarStandardScaler_000000000011,
  WrappedArray(
    Feature(
      "age_1-stagesApplied_RealNN_000000000010",
      false,
      FillMissingWithMean_000000000010,
...
ageGroup: FeatureLike[PickList] = Feature(
  "age_1-stagesApplied_PickList_000000000012",
  false,
  UnaryLambdaTransformer_000000000012,
  WrappedArray(
    Feature(
      "age",
      false,
      FeatureGeneratorStage_000000000005,
...

See Creating Shortcuts for Transformers and Estimators for more documentation on how shortcuts for stages can be created. We now define a Feature of type Vector, that is a vector representation of all the features we would like to use as predictors in our workflow.

In [7]:
val passengerFeatures = Seq(
  pClass, name, age, sibSp, parCh, ticket,
  cabin, embarked, familySize, estimatedCostOfTickets,
  pivotedSex, ageGroup
).transmogrify()
Out[7]:
passengerFeatures: FeatureLike[OPVector] = Feature(
  "age-cabin-embarked-fare-name-pClass-parCh-sex-sibSp-ticket_10-stagesApplied_OPVector_000000000017",
  false,
  VectorsCombiner_000000000017,
  WrappedArray(
    Feature(
      "parCh-sibSp_1-stagesApplied_OPVector_000000000013",
      false,
...

The .transmogrify() shortcut is a special AutoML Estimator that applies a default set of transformations to all the specified inputs and combines them into a single vector. This is in essence the automatic feature engineering Stage of TransmogrifAI. This stage can be discarded in favor of hand-tuned feature engineering and manual vector creation followed by combination using the VectorsCombiner Transformer (short-hand Seq(....).combine()) if the user desires to have complete control over feature engineering.

The next stage applies another powerful AutoML Estimator — the SanityChecker. The SanityChecker applies a variety of statistical tests to the data based on Feature types and discards predictors that are indicative of label leakage or that show little to no predictive power. This is in essence the automatic feature selection Stage of TransmogrifAI:

In [8]:
val sanityCheck = true
val finalFeatures = if (sanityCheck) survived.sanityCheck(passengerFeatures) else passengerFeatures
Out[8]:
sanityCheck: Boolean = true
finalFeatures: FeatureLike[OPVector] = Feature(
  "age-cabin-embarked-fare-name-pClass-parCh-sex-sibSp-survived-ticket_11-stagesApplied_OPVector_000000000018",
  false,
  SanityChecker_000000000018,
  WrappedArray(
    Feature(
      "survived",
      true,
...

Finally, the OpLogisticRegression Estimator is applied to derive a new triplet of Features which are essentially probabilities and predictions returned by the logistic regression algorithm:

In [9]:
import com.salesforce.op.stages.impl.classification.BinaryClassificationModelSelector
import com.salesforce.op.stages.impl.classification.BinaryClassificationModelsToTry._

val prediction = BinaryClassificationModelSelector.withTrainValidationSplit(
  modelTypesToUse = Seq(OpLogisticRegression)
).setInput(survived, finalFeatures).getOutput()
Out[9]:
import com.salesforce.op.stages.impl.classification.BinaryClassificationModelSelector

import com.salesforce.op.stages.impl.classification.BinaryClassificationModelsToTry._


prediction: FeatureLike[Prediction] = Feature(
  "age-cabin-embarked-fare-name-pClass-parCh-sex-sibSp-survived-ticket_12-stagesApplied_Prediction_000000000023",
  true,
  ModelSelector_000000000023,
  WrappedArray(
    Feature(
      "survived",
      true,
...
In [10]:
val evaluator = Evaluators.BinaryClassification().setLabelCol(survived).setPredictionCol(prediction)
Out[10]:
evaluator: evaluators.OpBinaryClassificationEvaluator = OpBinaryClassificationEvaluator_000000000024

Notice that everything we’ve done so far has been purely at the level of definitions. We have defined how we would like to extract our raw features from data of type Passenger, and we have defined how we would like to manipulate them. In order to actually manifest the data described by these features, we need to add them to a workflow and attach a data source to the workflow.

In [11]:
import spark.implicits._ // Needed for Encoders for the Passenger case class
import com.salesforce.op.readers.DataReaders

val trainFilePath = "datasets/TitanicDataset/TitanicPassengersTrainData.csv"
// Define a way to read data into our Passenger class from our CSV file
val trainDataReader = DataReaders.Simple.csvCase[Passenger](
  path = Option(trainFilePath),
  key = _.id.toString
)
Out[11]:
import spark.implicits._ // Needed for Encoders for the Passenger case class

import com.salesforce.op.readers.DataReaders


trainFilePath: String = "datasets/TitanicDataset/TitanicPassengersTrainData.csv"
trainDataReader: readers.CSVProductReader[Passenger] = [email protected]
In [12]:
val workflow = new OpWorkflow()
  .setResultFeatures(survived, prediction)
  .setReader(trainDataReader)
Out[12]:
workflow: OpWorkflow = [email protected]

When we now call train on this workflow, it automatically computes and executes the entire DAG of Stages needed to compute the features survived, prediction, rawPrediction, and prob, fitting all the estimators on the training data in the process. Calling score on the fitted workflow then transforms the underlying training data to produce a DataFrame with the all the features manifested. The score method can optionally be passed an evaluator that produces metrics.

In [13]:
val fittedWorkflow = workflow.train()
println("Summary:\n" + fittedWorkflow.summaryPretty())
19/04/04 23:30:08 WARN PearsonCorrelation: Pearson correlation matrix contains NaN values.
19/04/04 23:30:08 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
19/04/04 23:30:08 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
Summary:
Evaluated OpLogisticRegression model using Train Validation Split and area under precision-recall metric.
Evaluated 8 OpLogisticRegression models with area under precision-recall metric between [0.7258579350211893, 0.8158704337161141].
+--------------------------------------------------------+
|         Selected Model - OpLogisticRegression          |
+--------------------------------------------------------+
| Model Param      | Value                               |
+------------------+-------------------------------------+
| aggregationDepth | 2                                   |
| elasticNetParam  | 0.5                                 |
| family           | auto                                |
| fitIntercept     | true                                |
| maxIter          | 50                                  |
| modelType        | OpLogisticRegression                |
| name             | OpLogisticRegression_00000000001c_7 |
| regParam         | 0.2                                 |
| standardization  | true                                |
| tol              | 1.0E-6                              |
| uid              | OpLogisticRegression_00000000001c   |
+------------------+-------------------------------------+
+------------------------------------------------------------------------+
|                        Model Evaluation Metrics                        |
+------------------------------------------------------------------------+
| Metric Name                 | Training Set Value  | Hold Out Set Value |
+-----------------------------+---------------------+--------------------+
| area under ROC              | 0.8454730495722972  | 0.8350683148991541 |
| area under precision-recall | 0.8250338838777734  | 0.7387030767861703 |
| error                       | 0.21508034610630408 | 0.1951219512195122 |
| f1                          | 0.71                | 0.7142857142857143 |
| false negative              | 100.0               | 9.0                |
| false positive              | 74.0                | 7.0                |
| precision                   | 0.7421602787456446  | 0.7407407407407407 |
| recall                      | 0.6805111821086262  | 0.6896551724137931 |
| true negative               | 422.0               | 46.0               |
| true positive               | 213.0               | 20.0               |
+-----------------------------+---------------------+--------------------+
+-------------------------------------------------------------------------------+
|                              Top Model Insights                               |
+-------------------------------------------------------------------------------+
| Top Positive Correlations                              |    Correlation Value |
+--------------------------------------------------------+----------------------+
| sex(sex = Female)                                      |   0.5408676879258657 |
| name                                                   |  0.33895808575240116 |
| cabin(cabin = other)                                   |   0.3142575853287733 |
| pClass(pClass = 1)                                     |   0.2829126682258749 |
| embarked(embarked = C)                                 |  0.19616299633692874 |
| age(age_1-stagesApplied_PickList_000000000012 = Child) |  0.11682855241012707 |
| pClass(pClass = 2)                                     |   0.1162019601454126 |
| parCh                                                  |  0.11141839753358806 |
| fare                                                   |  0.11141839753358806 |
| sibSp                                                  |  0.11141839753358806 |
| embarked(embarked = null)                              |  0.06266815370525775 |
| age(age_1-stagesApplied_PickList_000000000012 = Adult) | 0.003828185766210828 |
| embarked(embarked = Q)                                 | -0.01423449991531148 |
| age                                                    | -0.07606557384744762 |
| age(age = null)                                        |  -0.1116519063188936 |
+--------------------------------------------------------+----------------------+
+------------------------------------------------------------------------------+
| Top Negative Correlations                             |    Correlation Value |
+-------------------------------------------------------+----------------------+
| sex(sex = Male)                                       |  -0.5408676879258653 |
| name                                                  |  -0.5320277005898669 |
| pClass(pClass = 3)                                    | -0.33865161163109314 |
| cabin(cabin = null)                                   | -0.31425758532877285 |
| embarked(embarked = S)                                | -0.17044508720497387 |
| age(age_1-stagesApplied_PickList_000000000012 = null) |  -0.1116519063188936 |
| sibSp                                                 | -0.04103199324784826 |
| parCh                                                 | 0.014173619278117376 |
+-------------------------------------------------------+----------------------+
+----------------------------------------------------------------------------------------+
| Top Contributions                                                |  Contribution Value |
+------------------------------------------------------------------+---------------------+
| name                                                             | 0.43686641838492407 |
| sex(sex = Male)                                                  | 0.41412991086601963 |
| sex(sex = Female)                                                |  0.4127484203214077 |
| pClass(pClass = 3)                                               | 0.21862002466078745 |
| cabin(cabin = null)                                              | 0.08833226022369427 |
| cabin(cabin = other)                                             | 0.08794630568886878 |
| pClass(pClass = null)                                            |                 0.0 |
| pClass(pClass = other)                                           |                 0.0 |
| pClass(pClass = 2)                                               |                 0.0 |
| pClass(pClass = 1)                                               |                 0.0 |
| parCh(fare-parCh-sibSp_3-stagesApplied_Real_00000000000e = null) |                 0.0 |
| parCh                                                            |                 0.0 |
| parCh(parCh-sibSp_2-stagesApplied_Real_00000000000d = null)      |                 0.0 |
| parCh(parCh = null)                                              |                 0.0 |
| embarked(embarked = null)                                        |                 0.0 |
+------------------------------------------------------------------+---------------------+
+--------------------------------------------------------------------------+
| Top CramersV                                       |            CramersV |
+----------------------------------------------------+---------------------+
| sex                                                |  0.5408676879258657 |
| pClass                                             | 0.34975140363576573 |
| cabin                                              | 0.31425758532877324 |
| embarked                                           | 0.20797606343621572 |
| age_1-stagesApplied_PickList_000000000012          |  0.1469944783048929 |
| age                                                |  0.1116519063188937 |
| parCh                                              |                 0.0 |
| ticket                                             |                 0.0 |
| name                                               |                 0.0 |
| fare-parCh-sibSp_3-stagesApplied_Real_00000000000e |                 0.0 |
| parCh-sibSp_2-stagesApplied_Real_00000000000d      |                 0.0 |
| sibSp                                              |                 0.0 |
+----------------------------------------------------+---------------------+
Out[13]:
fittedWorkflow: OpWorkflowModel = [email protected]

After model has been fitted we use scoreAndEvaluate() function to evaluate the metrics

In [14]:
println("Scoring the model:\n=================")
val (dataframe, metrics) = fittedWorkflow.scoreAndEvaluate(evaluator = evaluator)

println("Transformed dataframe columns:\n--------------------------")
dataframe.columns.foreach(println)

println("Metrics:\n------------")
println(metrics)
Scoring the model:
=================
Transformed dataframe columns:
--------------------------
key
survived
age-cabin-embarked-fare-name-pClass-parCh-sex-sibSp-survived-ticket_12-stagesApplied_Prediction_000000000023
Metrics:
------------
{
  "Precision" : 0.7420382165605095,
  "Recall" : 0.6812865497076024,
  "F1" : 0.7103658536585367,
  "AuROC" : 0.8442463170677148,
  "AuPR" : 0.8175464116852524,
  "Error" : 0.2132435465768799,
  "TP" : 233.0,
  "TN" : 468.0,
  "FP" : 81.0,
  "FN" : 109.0,
  "thresholds" : [ 0.6337362389928601, 0.5919422915099323, 0.581676424427552, 0.53827013167045, 0.4307972287346466, 0.38820165599662904, 0.3781943782287164, 0.33771406493189465, 0.3283922601119171, 0.29075001557187263, 0.2820965571896635, 0.24780299079760443, 0.20938927696917828 ],
  "precisionByThreshold" : [ 0.9560439560439561, 0.9470588235294117, 0.9375, 0.7420382165605095, 0.7423312883435583, 0.7217391304347827, 0.7225433526011561, 0.696236559139785, 0.6355748373101953, 0.5334507042253521, 0.5305410122164049, 0.38288288288288286, 0.3838383838383838 ],
  "recallByThreshold" : [ 0.2543859649122807, 0.47076023391812866, 0.4824561403508772, 0.6812865497076024, 0.7076023391812866, 0.7280701754385965, 0.7309941520467836, 0.7573099415204678, 0.8567251461988304, 0.8859649122807017, 0.8888888888888888, 0.9941520467836257, 1.0 ],
  "falsePositiveRateByThreshold" : [ 0.007285974499089253, 0.01639344262295082, 0.020036429872495445, 0.14754098360655737, 0.15300546448087432, 0.17486338797814208, 0.17486338797814208, 0.2058287795992714, 0.30601092896174864, 0.48269581056466304, 0.4899817850637523, 0.9981785063752276, 1.0 ]
}
Out[14]:
dataframe: DataFrame = [key: string, survived: double ... 1 more field]
metrics: evaluators.EvaluationMetrics = BinaryClassificationMetrics(
  0.7420382165605095,
  0.6812865497076024,
  0.7103658536585367,
  0.8442463170677148,
  0.8175464116852524,
  0.2132435465768799,
  233.0,
  468.0,
...